Re: [RFC] new Squid status codes

On 03/10/2014 03:19 PM, Amos Jeffries wrote:

> Okay. Is this better?

[TCP_TUNNEL]
> - traffic for which Squid is not able to act as a proxy. The
> transaction payload/body was tunneled as raw binary to the server.

>>>> TCP_RELAY
>>>>
>>>> - for requests which Squid serves without even considering the stored
>>>> content.

The above definitions still overlap (besides other problems).

For the tunneling case, how about the following:

TCP_TUNNEL_MISS: Squid established a tunnel between client and the next
transfer hop by shoveling opaque bytes between two TCP connections, one
with the client and one with the next hop. Such a tunnel ends transfer
protocol processing on the connection. [ The next hop is determined by
the transfer protocol. For HTTP CONNECT requests, it is either the
origin server identified in the CONNECT request (DIRECT hierarchy code)
or a cache peer (a peering hierarchy code such as DEFAULT_PARENT). ]

AFAICT, the above both provides the "tunnel" outcome information you
want and distinguishes between direct tunnels and tunnels through/with
peers.

The text in [brackets] should probably be moved to apply to all statuses
because it applies to all transactions that may use peers, not just tunnels.

For the relaying case, how about these two (one new and one old,
adjusted to avoid overlap):

TCP_RELAY_MISS: The request was forwarded to the next hop without
looking up the requested object in the cache and without updating the
cache state when handling the response. No tunnel was established as the
result of this transaction (see TCP_TUNNEL_MISS). TCP_RELAY_MISSes do
not affect Squid cache state and cache tuning does not affect the number
of TCP_RELAY_MISSes.

TCP_MISS: The request was forwarded to the next hop after looking up the
requested object in the cache. The response object delivered came from
the next hop.

We can drop the _MISS suffix for new codes (a separate decision!). If we
use that suffix, most old log analysis scripts should continue to
provide hit ratios as expected by most admins. If we do not, the outcome
would be less predictable. HIT/MISS is a transaction flag that (for most
hit ratio definitions) applies to all [compound] transactions. I think
we should continue adding it.

>>> The above two definitions overlap. Please adjust to make them disjoint
>>> if you think both are needed, keeping in mind that:
>>>
>>> * Whether the request went direct or to a peer is already covered by the
>>> "hierarchy code" field and should not be repeated in the "Squid result
>>> code" (a.k.a. "Squid request status" and "Squid status code", depending
>>> on where one looks) that you propose to expand.
>>>
>>> * Request method such as CONNECT is also logged separately.

> CONNECT could be either tunneled or relayed.

Yes, as well as other things (e.g., rejected or asked to be
authenticated). We already have access.log fields to detect that
difference AFAICT. The important thing here is that a tunnel was
established. Whether that tunnel is with a peer or origin server is
secondary information (that, again, we already have access to via the
hierarchy codes).

> In the case of RELAY:
> - the traffic with server/peer is in a protocol which Squid supports
> (HTTP/FTP/Gopher/ICY).

The initial traffic uses the transfer protocol indeed. However, when/if
the tunnel is established, the traffic becomes opaque to this Squid, and
that is the important information that what you want to log (AFAICT).

> - the inability to cache is defined by the specification for the method.
> - note RELAY is not limited to CONNECT, it could be from TRACE/PUT/POST
> or any one of the extension methods.

IIRC, PUT affects the cache state. If your motivation is to provide a
simple way to identify caching-related traffic, PUT should be excluded
from TCP_RELAY_MISS. In my definition above, it is excluded.

> In the case of TUNNEL
> - the traffic with server/peer is in unknown protocol taken from inside
> a CONNECT (or equivalent) message verbatim rather than gateway like
> FTP/Gopher/etc.

Yes, and the same can be said of the final outcome of a CONNECT request
"relayed" to the peer.

> So RELAY could be seen if the peer was another proxy and the CONNECT
> being passed on as-is. TUNNEL if the peer were an origin and CONNECT
> being delivered there un-wrapped.

AFAICT, the difference between different kind of peers is already
recorded in the hierarchy code. If there is some additional information
that you want to convey, it is not yet clear base on your proposed
definitions for the new status codes. Thus, I do not know whether my
attempt to refine those definitions addresses that additional need.

>>> * The _MISS suffix is appropriate in both cases (using the current
>>> definition of _MISS at http://wiki.squid-cache.org/SquidFaq/SquidLogs):
>>> "The response object delivered was the network response object".
>>>
>
> Yes these are sub-types of MISS. I was thinking RELAY to be used most
> when the RFC prevents cache storage being relevant or checked. We can
> drop the definition line below...

Dropped.

> If you are looking for that kind of misconfiguration then a TCP_FORCED_*
> would be useful. Would you like that added?

No, thank you. I was just explaining that the first version of your
definition did not quite match the stated intent. I am not asking for
more status codes at this time (and do not object to new ones being
added). I only insist that the ones we add are well defined.

>>>> TCP_SHARED_*
>>>>
>>>> - to indicate collapsed forwarding on the request. Similar in
>>>> principle
>>>> to the TCP_CLIENT_* and TCP_REFRESH_* labels indicating client forced
>>>> something to happen or revalidation took place.
>>>> Mainly so people can measure the difference between reguler HIT/MISS
>>>> and SHARED_HIT/MISS to determine if collapsed forwarding is worth it.
>>>
>>> I agree that _SHARED_ (and adding sharing stats to the Cache Manager)
>>> can be useful.
>>>
>>> Please adjust the definition to clarify that _SHARED_ is going to be
>>> used for all transactions that _started_ collapsed and not just those
>>> that ended collapsed. For example. a request that was initially
>>> collapsed but for which things did not work out (e.g., the master
>>> transaction on which we collapsed got an uncachable response, and Squid
>>> had to send that request to the origin server in isolation after all)
>>> would most likely be logged as TCP_SHARED_MISS.
>>>
>>> This SHARED tag will _not_ be used for the "initial" transaction that
>>> others collapsed on, right? Again, it would be nice if the definition of
>>> the _SHARED_ tag made that clear[er].
>
> Yes that was the idea exactly.
>
> TCP_SHARED_HIT being when request #2 is collapsed into request #1
> TCP_SHARED_MISS being when request #3 is held for collapsing, but is
> forced into forwarding with extra lag.
>
> Similar for collapsed revalidations and TCP_SHARED_REFRESH_*.

Glad we agree. Please adjust the definition to clarify the above.

Thank you,

Alex.
Received on Tue Mar 11 2014 - 15:05:27 MDT