RE: Introduction and Question

> -----Original Message-----
> From: Bryan Hopkins [mailto:bwh@stwing.upenn.edu]
> Sent: Friday, 19 July 2002 12:24 AM

> Before this "done" section, nothing is sent to the client,
> since I diverted the
> result to the transcoding module. So for the
> proof-of-concept prototype, all of
> the writing back to the client's file descriptor takes place
> in this "done"
> section.

This is a design issue. Rather than answering your question on
re-queuing the Complete call, I'm going to critique your design, and
offer a more effective one.

The issue is that if you send nothing to the client until your
transcoding is complete, bad things may happen. I.e. the client may time
out - say for example you are recompressing a gz file into a bz2 file,
with the upstream source several Mb in size, and a slow link in the way.
A second is server resources, you will increase maximal memory use
significantly if all concurrent objects need to be fully stored in RAM
until the connection completes.

As a bit of back ground: I've coded up a generic content modification
framework for squid 2.4-5 ish in the past. It's currently broken, due to
the need to redesign the innards of the request pipeline to address
content-length modification semantics within squid. That's been mostly
(perhaps completely) done in HEAD, meaning I may be able to pick up on
the framework in the future.

For your specific project, I presume you are interested in transcoding
on each and every client request (ie not caching the resulting
transcoded page). This makes a complexity vs efficiency tradeoff.

What you need to do is insert a new nonblocking module in the call
chain. It's hardcoded in the current sources.

storeClientCopy uses clientSendMoreData as the callback completion
routine for reading data from the cache, and clientSendMoreData passes
clientWriteBodyComplete or clientWriteComplete as the callback
completion routine for comm_write.

What you need to do is have a new function (clientTranscodeSendMoreData)
passed to storeClientCopy, which pulls of data and sends it to your
transcoder, and which also operates as a callback routine, calling the
passed routine (which will be clientSendMoreData) when the data is
available.

This allows you to keep the non-blocking callback 'stream' of data in
flow. If you need to perform blocking operations, then ensure the the
select loop will call you when it completes the blocking operation.

Your routine will look similar to clientSendMoreData, but without most
of the corner cases, unless you need to worry about error generation or
http header creation.

Rob
Received on Thu Jul 18 2002 - 16:36:25 MDT