Re: client-store interface

On Fri, 2003-10-31 at 10:57, Henrik Nordstrom wrote:
> On Fri, 31 Oct 2003, Robert Collins wrote:
>
> > > * store merge logics when there is multiple data feeds (different clients
> > > requesting different not yet cached ranges)
> >
> > Heh, I actually consider this a reason to feed into one store object
> > rather than a new one.
>
> before we continue this we really need to have the discussion clearly
> separated in
>
> * client-side
> * server-side
> * store
> * broker
>
> it is quite evident that both of us are very unclear on the split between
> broker and store.

yah.

> > Sure. The issue I have with your design is that it makes the
> > server/store drive everything which is exactly the problem we had before
> > where the client side drove everything. Downgrading is esaier if the
> > client side can drive it's needs. Teaching the store about freeable
> > ranges in random access needs to be done - in either approach.
>
> Why I propose what I do is because there is a high risk that we will be
> receiving reply content in a different order than asked for. When
> designing for processing retrieved ranged HTTP data it must be assumed
> that from time to time the returned data will be compatible with what was
> asked for but not in the exact same order. The most probable cases (save
> for for non-ranged response) is that the ranges have been simplified or
> sorted in the response.
>
> This brings the issue of receiving a fully valid reply, but which does not
> fulfill our requirements in order to be able to construct a suitable reply
> to the client if it is the client who decides on what it wants to see.
> This is not so much of a problem when not doing range merging, but when
> doing range merging it is a more difficult problem as some times it may
> not be evident from start that there is a problem.

In which case we need to hold off telling the client anything until we
have enough info to see there won't be a consistency problem.

> If you look closely at what I propose the result is in short that on cache
> misses (full or partial) the client side has no control over the order of
> the returned data. It is up to the server side (ours, or origin server) to
> make sure the content arrives in an order compatible with the request.

mmm, probably, haven't looked at all the corner cases.

> only on cache hits can the client side have control, but in order to unify
> the two cases this control is isolated to just telling at start what is
> wanted.

> My ideas in more detail:
>
>
> Client -> Broker
>
> I want these ranges.
> I accept/don't accept multi-part responses. (can be deduced from the
> wanted ranges)

Right, easily handed in via the request->range pointer.

> Broker -> store
>
> Random read/write, as cache-able data flows via the broker.
>
> non-cache-able data never hits the store at all.
>
> Broker -> Server side (miss, full or partial)
>
> Client wants these ranges
> Clients accepts / don't accept multi-part responses. (can be deduced from
> the wanted ranges)
> I have these entities (and ranges thereof) in the store which may be
> compatible with the request. This information is used for constructing
> If-Range, If-None-Match etc.

How do you propose to handle entities the client has that we don't?

> Related Note: In some cases two server round-trips may be required by
> the server side to satisfy the request, first If-None-Match to find
> the correct reply entity, then followed by If-Range if the entity is not
> complete.
>
> Server side -> Broker
>
> Reply entity headers
> Ranges of entity body
>
> Broker -> Client
>
> Entity headers
> Ranges of entity body. In case of cache misses the ranges are returned
> in the exact same order as received from upstream (server side).

which may break on merging for single part replies.

> About the only thing I have not yet set my mind on is the tiny detail
> where to do range merging. This is either done server-side or in the
> broker. My initial preference is for the server-side as it makes it easier
> to deal with corner cases, but more efficient in the broker as the parts
> who are cache hits don't need to be looped back and also allows for store
> merging in place without creating a new store object.

I think the broker is the place.

> > Yeah. Theres other uses for the notification - to implement the arrival
> > of headers, and for the transactions needed for 1XX Continue support.
>
> Headers should be given by a completely different "read" operation in my
> opinion. The whole forwarding and store paths should be strict about
> separation between headers and body type content. This is true both for
> initial headers and trailer header data, and the presence of trailer
> header data further emphasizes the reasons why the store needs to separate
> the two cleanly.

Agreed.

Rob

-- 
GPG key available at: <http://members.aardvark.net.au/lifeless/keys.txt>.