Re: Squid3 with multiple ICAP Services

On Wed, 2007-09-19 at 17:02 -0400, Richard Bishop wrote:

> As far as I can see, all ICAP transactions themselves (once we have
> indentified a suitable service from the candidates in squid.conf) are
> fired as asynchronous events. Events are then fired in the order they
> are pushed onto the events list, with a callback being executed at the
> respective time.

You are technically correct, but are thinking at a very low-level that
would not get you far enough. Events or asynchronous calls is just a
mechanism to implement non-blocking actions such as waiting for fresh
ICAP OPTIONS or starting communication with an ICAP service.

> So, as I understand it, adding multiple ICAP services is simply a
> matter of adding several events onto the events list one after
> another, each having a respective argument for the desired service.
> These will then be processed in order, each adapting the request /
> response (or leaving it untouched).

I am afraid the actual implementation is a lot more complex and has
little to do with events. Three areas would need to be changed:

1. The ICAP configuration classes (e.g., ICAPClass and ICAPAccessCheck)
and code would need to support selection of multiple services. It is not
yet clear whether such selection should be static (once per HTTP
transaction) or dynamic (select next ICAP service after the previous
service has finished). Perhaps the choice should even be configurable.

2. The ICAPLauncher class needs to be enhanced to support iteration over
ICAP services. I am not sure whether that iteration should be supported
directly by that class. The alternative is to split ICAPLauncher into
two classes: one that handles ICAP transaction retries (the current
code) and one that handles ICAP service iteration (the new code). If the
class is split, some glue will be needed to make the split invisible to
the core/calling code.

Besides handling bypass and aborts, the iteration code should correctly
pass adapted HTTP bodies from one service to another via BodyPipe. It is
probably important to support ICAP service "pipeline" where the next
service can start working on the HTTP message still being adapted by the
previous service (where possible). This implies that the code will have
to deal with multiple ICAP services working on the same HTTP transaction
at the same time.

3. Squid.conf changes to better support multiple services. The current
(Squid2) interface has limited functionality and is rather difficult to
understand for an average administrator. It should be significantly
improved while support for chaining is added. As you already pointed
out, I have outlined possible changes in
http://www.squid-cache.org/mail-archive/squid-dev/200610/0051.html
Most, if not all, of those points should still be valid.

> Have I understood this all correctly or is there some massive amount
> of work involved that I've missed?

It would be much simpler if handling of each ICAP service would be an
isolated event. Unfortunately, the output of one service is the input of
another; headers, bodies, errors, and other events needed to be properly
propagated.

Alex.
Received on Mon Sep 24 2007 - 10:21:27 MDT