Re: Introduction and Question

===
----- Original Message -----
From: "Bryan Hopkins" <bwh@stwing.upenn.edu>
To: "Robert Collins" <robert.collins@syncretize.net>
Cc: <bwh@stwing.upenn.edu>; <squid-dev@squid-cache.org>
Sent: Friday, July 19, 2002 11:08 AM

> I really have no choice but to send nothing until transcoding is
complete.
> Perhaps a little more detail on the project would help. An HTTP
object
> will be transcoded if and and only if it contains a set of extra HTTP
> headers in the reply object that we defined for the project. Two of
those
> headers define an applet to be downloaded by the proxy to be used for
the
> transcoding. Another contains the arguments for said applet.
Security
> issues aside, this applet can do whatever it likes to the object, and
its
> functionality is totally unknown to the proxy. So if the applet's
> function is to invert the file so that the last byte is first, that
has to
> be facilitated. The applet also has the right to modify the headers.
> Either way, there is going to be a significant lag from the download
of
> the applet if it has not already been cached, because squid certainly
has
> to wait for the entirety of that to arrive before it can run it, and
it
> can't start sending a reply before the transcoder has been run. As I
> said, at the current stage this is just a proof-of-concept prototype
to
> examine the possibilities for this computing model. Its in no way
meant
> to be deployable in the real world. Timeout being a real issue, we
have
> in our architecture an allowance that if a transcoding instance is
taking
> too long, it is aborted and untouched data is returned to the client.

Ok.

> This sounds like a cleaner overall design from the squid end, although
> I'm not sure how much is gained in terms of performance by it taking
> into account my comments above regarding the delays that will be
incurred.
> For the real implementation of the project (should the prototype
> prove to demonstrate value-addition in terms of functionality), this
> sounds like the proper way to do things in terms of execution flow in
> terms of interfacing with squid.

The performance gained is immense. It's not a per-request performance
gain, it's the concurrent execution of IO required by squid's design.

> Given the state of the Programmers Guide, for a proof-of-concept
> prototype like this I just went with the simplest way to make it
> work ASAP (as per my marching orders :) ). The bulk of my work has
> been in the transcoder process(es) with which the modified squid
> communicates. Since this (synchronization with the transcoder) is
> the last issue keeping me from having a working click-and-play system
in
> the lab, I could just block squid until the transcoding is complete,
but I
> hoped to avoid that because of the awful performance penalty. Going
> forward after I've bought some time with the prototype, I totally
> agree that the flow you recommend is a better way to go, but for right
now
> I'd appreciate the cheapest, dirtiest, quickfix you could recommend.
:)

What I've suggested *is* the cheapest dirtiest and quickest fix. It's ~
four functions.
1) A callback for the StoreClientCopy function. This queues the data to
the transcoder.
2) A callback for comm_write. This requests more data from
StoreClientCopy.
3) A callback for comm_read. This reads data from the transcoder and
performs the clientSendMoreData callback.
4) A function for clientSendMoreData to call that kicks everything off.

The reason this is pretty much the cheapest approach is that it allows
the existing callback method to provide the syncronisation to the client
of how much data you have pushed out of the transcoding process. If you
generate the data incrementally, or all at once, this will still handle
it all seamlessly.

Rob
Received on Thu Jul 18 2002 - 21:20:09 MDT