A youtube and windows update cache thing.

Since the development of StoreID there were some advances in couple
areas and it seems like the idea of 206 responses caching can be moved
on to the next level.

Based on my work with GrreasySpoon for youtube (mentiond here:
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator#Implementing_ICAP_solution)
I took it to the next level and I am now trying to proof with almost the
same way another thing.

The idea is to proof that caching partial content as a stand alone
object in squid is better then trying to cache the full object each time
it's being requested.
This approach is basically meant to work only for windows updates which
tries to do all updates in 206 chunks.
They now do use a lot of SSL traffic in order to secure their clients
but still I have seen non SSL updates.

How will it be done?
in youtube case we have used only the url to get info on the object that
the client requests.

Squid in a way can cache everything if there was someguy inside the
system who actually will tell us how everything works but we are not there.

So in a case of 206 requests we can use:
1. URL
2. request HEADERS
3. response HEADERS

We have used the url and are using in a specific way specific vary headers.
We can now try a new approach and use more request HEADERS in order to
define the chunks of the requests.
pros:
more cache.

cons:
more disk space(we want it since it's cheap)
more RAM for objects db
more CPU

With today computer monsters I think this approach is a very nice one.
Please share your opinion about using more resources to make cache more
effective.
Is it good? for who is it good? Bandwidth vs Power consumption pros and
cons?

The idea is to use this kind of script:
https://github.com/elico/squid-helpers/blob/master/squid_helpers/youtubetwist/youtube_to_internal.req.server.java

and add extraction of request range headers which will result in the
StoreID in squid for the object as the url + a custom fragment which
squid will use to define the range of the request.

I am looking for people to test the idea and anyone who can do that is
welcome to help me test it.

Eliezer
Received on Sat May 25 2013 - 00:18:11 MDT