Re: "lookahead caching"

It has been discussed a number of times back and forth, but no final
view yet. As you say, on-demand prefetch only reduces latency on the
expence of bandwidth.

Issues:
a) It is not certain that the user-agent will request the "lookahead
objects".

b) Even if the user-agent requests them, it is not certain that the
request will be compatible with the fake one used to prefetch the
object. There might for example be variance in cookies and similar
stateful aspects.

c) Many web-sites does not allow automated retreival, and this is on the
border.

If one builds such a method, it must also be coupled with a "blacklist"
of sites (whole or partial) that cannot be prefetched, and should make
use of "robots.txt".

--
Henrik Nordstrom
Squid hacker
Brian Szymanski wrote:
> 
> hi,
> 
> i was wondering if anyone's ever considered the idea of adding
> "lookahead caching" to squid. that is, for speed (definitely not
> bandwidth reduction purposes), do something like the following:
> 
> whenever a new page is added to the cache, try to download any page
> linked from that page and put it in a (smaller) different cache called
> the transient cache. when a page in the transient cache is viewed, it
> moves to the stable cache. the stable cache is managed in squid's
> standard fashion. the transient cache can be managed in an LRU fashion
> indexed on the date that this webpage's (most recent) parent was viewed.
> 
> that way, the typical users experience of clicking on some page, reading
> for awhile, and then clicking on one of the links will be greatly
> accelerated. this idea is basically what a product called peak net.jet
> was doing a couple of years back (it doesn't look like they're still in
> business though), but only with netscape's builtin cache.
> 
> so my question are as follows:
>     would this be difficult to implement?
>     would it violate netiquette to implement this? (imagine a single
> user on a cable modem sucking around 10 times as much bandwidth on the
> net)
>     any other thoughts?
> 
> thanks for reading...
> brian szymanski
> bks10@cornell.edu