squid-2 and client-side delay-pool _writes_

I've been planning out the implementation of client-side delay pool
writes for a while and I think I'm coming close to a final design.

I'd just like to clarify a few things about the implementation of the
read-side delay pools trickery before I rush off to provide write-side
hooks.

As far as I can tell, the throughput is delayed by the fd read handler
limiting the FD_READ_METHOD() size; but the -callback- is delayed if
there's a read defer handler registered and it notes there's 0 bytes
to read. That way the handler isn't initially called if the pool is
completely drained.

FDs which are delayed are put onto the slowfds list which is dequeued
randomly to give some fairness to clients able to quickly drain pools;
the slowfds list is dequeued once a second and the callbacks are
called. I'm guessing the callbacks are called there because the only
reason they made it onto the slowfds list in the first place is that
the fd -was- ready but the bucket was empty and thus no data could be
read.

Now, for the delay-enabled write side, I was going to ignore trying to
hack up the comm code entirely and instead put the bulk of the logic
in comm_write and commHandleWrite routines. If commHandleWrite is
called but the bucket is empty the FD is added into another slowfds
array (just for write slowfds) which gets similarly randomly dequeued
once a second. It can then do the trickery required and the comm layer
stays mostly untouched. The write slowfds array should be able to
exist outside of the event core and inside comm.c -
FDs are added to it from commHandleWrite; its a scheduled event which
runs once a second; and all it needs to really do is call
commHandleWrite again and see if any further data can be written.

If that works out successful then I'll probably look at modifying the
read side of things in cacheboy with the introduction of a sensible
comm_read() - put all of the delay logic in there, strip out deferred
reads entirely and delete whatever mess is left. Note that a sensible
comm_read() is a completely different discussion for a later time -
I'm just trying to determine whether its feasible to move the delay
pools out of the event notification and simplify the comm code in the
medium term.

Comments please? My timeline for all of this is pretty tight now. I'd
appreciate feedback from anyone who has had their fingers deep in the
delay pools IO pie in the past.

Thanks,

Adrian
Received on Mon Jul 14 2008 - 16:30:21 MDT