Re: Async I/O on IRIX 6.x?

On 14 Sep 98, at 13:47, Alex Rousskov <rousskov@nlanr.net> wrote:

> > In this sense using OS provided threads should be highly preferred to
> > other pthreads implementations
>
> What are "OS provided threads"? Are pthreads on IRIX "OS provided"? How do I
> tell if pthreads on XXX are "OS provided", in general?

 I haven't seen a single IRIX so I can't tell you. But I'd think that if
 libthreads is located on your distr CD, it is OS provided. Another criteria
 should be to see "multithreaded kernel bla-bla-bla" somewhere in docs.
 They (vendor) should be pretty proud it ;)

> > Stew has designed ingenious async-io while avoiding mutexes. At a time
> > we had quite a arguing about whether to write academic threaded code
> > with zillions of mutexes or designing mutual exclusions algoritmically.
>
> As far as I understand you ended up doing something in between, right? That
> is, no mutexes and no guaranteed mutual exclusions?

 Nope. (btw, I've contributed not a single line in this async-io stuff, so all
 credit goes to Stew, and he did it all his way) mutex and cond_signal seems
 to be the only portable and fast way to pass data between threads. I guess
 Stew just didn't bother with it much more, as it did work for him.

> Also, by "zillions of mutexes", you mean "2", I guess. :)

 nope. To give you a relation, consider: Solaris kernel has few thousands of
 internal mutexes, (not counting VM page, cache, and file system locks). Just
 to syncronise kernel threads in such a way that to avoid all possible mutex
 contentions. This is one reason why it (kernel) scales relatively well on
 multi-CPU boxes.

> > So, to be
> > pedantic, we might say that on any OS that has ever heard of threads,
> > multiple IO operations on the same FD is MT-Safe as each file should be
> > protected by its own mutex,
>
> If so, the fact that we want to avoid mutexes sounds strange. If OS already
> protects every FD with a mutex, our added overhead should be minimal, unless
> OS has access to zero-overhead mutexes and we do not. What am I missing here?
>
> I recall that successfully locking (no blocking) a mutex is a very cheap
> operation. Are my recollections correct?

 Any mutex is (near)zero-overhead if it is unlocked. Take a look at any mutex
 implementation source. It is basically a cpu memory-barrier instruction that
 exchanges byte (word/double) in some memory for "1" in one op. Then it test
 to see if there already was "1". If there was "0", we have locked a mutex
 and can almost return immediately. If there was "1" then there is contention
 and libthread must initiate thread switch, first saving all current state, etc.

 Avoiding mutexes gives benefit if you can avoid them, just because you avoid
 extra code to run, you give it a really good thought thus avoiding any (most)
 contentions already in design. But it imposes limitations.

 What makes mutex expensive, is unlock with some threads blocked on it. If there
 are no threads blocked on mutex, lib threads _could_ return immediately. But if
 there are threads blocked on the same mutex, lib must unblock some other thread
 before returning. As threads are still implemented in user space lib, there is
 just OS support for them, the lib must still negotiate with OS on what to do next.
 I don't know what goes on to unblock one thread, but basically lib needs to
 find one thread eligible to run and place it on the run queue somehow, etc.
 Also, mutex operations are "cancellation points" where thread lib would look
 at threads scheduling anyway, for eg. non-kernel threads are scheduled by threads
 lib itself, not kernel. So it might need to traverse lots of code before
 returning.
 And this is why lib "_could_ return immediately" - it might switch threads instead.

 Another expensive call is cond_signal, as it prepares some thread to be unblocked.
 After subsequent mutex unlock, there would be most _probably_ a thread switch.

 So IMHO, few of general MT guidelines would be "don't make mutex crowdy", avoid
 ping-pong between threads by mutexes, prefer 1 mutex per data struct rather than
 1 mutex per subroutine. Of course, mutex is quite large, so there must be some
 sane balance.

 ----------------------------------------------------------------------
  Andres Kroonmaa mail: andre@online.ee
  Network Manager
  Organization: MicroLink Online Tel: 6308 909
  Tallinn, Sakala 19 Pho: +372 6308 909
  Estonia, EE0001 http://www.online.ee Fax: +372 6308 901
 ----------------------------------------------------------------------
Received on Tue Jul 29 2003 - 13:15:54 MDT