Re: Squid Patches for Transfer-Encoding/TE (against 2.3devel2)

Hand that man a cigar.

D

Patrick McManus wrote:
>
> Folks,
>
> As some of you might know, I'm very interested in the hop-to-hop
> semantics of HTTP/1.1 as the basis for deploying new features into the
> network.. the deployed basis of clients and servers makes it
> challenging to try new techniques end-to-end; but as HTTP is defined
> hop to hop we can implement these changes over a large proportion of a
> transfer just by going from proxy to proxy.
>
> I'm in the process of conducting a study to try out the effects of a
> number of different transfer-encodings for this purpose and am using
> squid as the base proxy. Of course squid (2.3devel2) has no support
> for transfer-encoding, or TE (or making http/1.1 requests for that
> matter)..
>
> so, I created support for transfer-encoding, and TE.. I also
> implemented a fairly generic mechanism for including your own
> transfer-encodings, and implemented a few. (I implemented chunked,
> gzip, deflate, and a couple called jones and biff for illustrative
> purposes.. the last two are just byte swappers but they show how the
> framework operates.. the compression formats are unfortunately done
> with an AppliedTheory library that I can't share with you, but I left
> the code in there as I think the zlib stuff is an almost drop in
> replacement and can be adapted easily)..
>
> I'm making these patches available to anybody who's interested at:
>
> http://proximate.appliedtheory.com/transfer-encoding/
>
> here's the basic api..
>
> /* Transfer Encoding Filters: They work like this..
>
> filter (a,b,c,d,e)
> a - is a character pointer to the data to be filtered
> b - is length of a
> c - is address of a character pointer where the resulting buffer
> should be stored
> d - pointer to integer to store length of c in
> e - pointer to a general data pointer that the filter may use to
> maintain state, it begins life as NULL.
>
> the return value is a 4 bit mask
> 0x01 - c is a newly allocated buffer and should be freed when the
> calling function has completed the filter
> 0x02 - reserved internally for chunking
> 0x04 - reserved internally for chunking
> 0x08 - set this bit if your function was called with input, but did
> not produce any out (for instance if you're buffering it in the
> context provided by e.) prevents squid from thinking EOF
> has been reached.
>
> call sequence
> first call: *e is NULL but a is not and b>0
> body calls : a is not null, b>0 and value of *e determined by
> previous calls
> last call: b==0.. good time to clean up *e if you've stored stuff
> there.. you may produce output if necessary, but
> last call will be repeated.
> */
>
> and they are installed using the new_xlat() function in
> client_side.c which there are several examples of..
>
> Also, the patches are against 2.3devel2 which had a couple bugs for me
> out of the box.. they were trivial and I fixed them, but those fixes
> are in the patch too, so I should enumerate them:
>
> * dns_internal.c: parsing problem with extra \n at the end of my
> resolv.conf..
> * strListisMember() ignored the length argument it parsed, and
> indeed would only return true if the the argument it was searching
> for was the last one in a list (i.e. list="x,y"
> strListisMember(list,"x") would be false but
> strListisMember(list,"y") would be true.. fixed.. also added a
> strListIsMember_q that works the same way but allows the list
> members to have q values and returns the qvalue * 1000.
>
> There are a few caveats however, both wrt specification and quality of
> implementation..
>
> specification: rfc 2616 section 3.6 prohibits the sending of
> transfer-encodings to 1.0 clients which the squid<->squid hop is.. In
> my implementation (which is only running privately for the purposes of
> study and doesn't claim to be compliant to anything) I will send a
> transfer-encoding to a client if that encoding is explicitly mentioned
> on the TE line with a positive q value.. (which should be safe as both
> the te todo and chunked need to be explicit on that line so the client
> is capable)
>
> because 2616 also requires that any response with a non-identity
> tr-enc be chunked and no content-length be present I *did* implement
> those requirements.. which means squid can now both send and receive
> chunked encodings while preserving persistent connections.. though it
> will only send chunked either to a 1.1 or higher client or a 1.0
> client with it on the TE header. (and because chunked is required to do
> more interesting things, it always puts it in it's requests.. though
> if it made 1.1 requests this would be superfluous.)
>
> related thought: given that chunked is in there now, what is lacking
> from the MUST list to make HTTP/1.1 requests?
>
> quality of implementation: There are a few things that aren't suitable
> for production code, but the basis of what needs to be there is in, so
> I thought I'd pass it on and y'all could consider if it should be
> dressed up.
>
> * selection of what objects to apply te's to is hardcoded to be
> text/* and *postscript* in the C.
>
> * new te's are just whacked into client_side.c, it lacks a good
> configure mechanism
>
> * squid.conf should be used to decide what TE's to announce on a
> request.. hardcoded in http.c for the moment.
>
> * no thought given to how an object should be stored.. right now it
> is stored with any t-e's it is received with..
>
> * when sending out new responses a very naive "strip to identity,
> rebuild to desired output" set of filters is constructed.. this
> can no doubt be optimized..
>
> all that not withstanding, it's a very handy framework for what I'm
> trying to do, so under normal opensource tenets I'll make the patches
> available.
>
> I'm very interested in dialogue and feedback, though I'm not on
> squid-dev@.. I could request that be changed if convenient for folks.
>
> -Patrick
>
> --
> Patrick R. McManus - AppliedTheory Corportation - Software Engineering
> http://pat.appliedtheory.com/~mcmanus Lead Developer
> mcmanus@AppliedTheory.com 'Prince of Pollywood' Standards, today!
> *** - You Kill Nostalgia, Xenophobic Fears. It's Now or Neverland. - ***
Received on Tue Jul 29 2003 - 13:15:59 MDT