Re: Storing of information

On 11/02/2012 15:43, Amos Jeffries wrote:
> On 11/02/2012 8:34 a.m., Pieter De Wit wrote:
>> Hi Guys,
>>
>> So I saw on the mailing list the question about quotas came up again.
>> I thought I would give it a shot (planning etc) and I was wondering,
>> does "Squid" have a way to store data ?
>>
>> In Quota you will have Bandwidth (bytes, not per second) that has to
>> be checked/adjusted and updated to disk. Instead of writing my own
>> routines to store this in my "own" format, it would be nice to have
>> something that grows with Squid.
>
> Discard the idea of disk formats for quota control. You are dealing
> with individual packet read/writes at this level. Everything of
> importance needs to be in RAM. Squid has a client_db memory cache
> which stores statistics and details about each unique client. Several
> transaction state controllers (ConnStateData, TunnelStateData,
> *ServerData) already interact with that for per-client bandwidth
> reporting.
>
> At the more abstracted level, semi-accurate quotas would need the
> client_db to be backed up on disk or somewhere periodically. (oh yay,
> yet another event to cause "squid keeps pausing" bugs).
>
> Or alternatively, the design I worked out a years or so ago uses a
> helper process to query some database or system managing client
> quotas. There is a lot of interest in RADIUS and ActiveDirectory
> backends controlling this type of thing, and a helper interface is
> much more flexible than pre-determined config formats. You can query
> this helper on each new client to receive a quota value which gets
> used up then re-checked. We still get some slowdown from the lookups,
> but it is up to the admin to configure how much quota bytes the helper
> requests each cycle and thus how much overhead they add.
>
> Amos
>
Hi Amos,

Thanks - I second the idea of a helper process, I like the "milter" type
idea where this can run on another box, away from Squid. I have been
messing around with ICAP to do this, what are your thoughts on that ?
IMHO, the time part of quotas should be handled by ACL's, it is
impossible for Squid to know how long someone has been on the net.

Back to the "sync" of client_db. I agree that the working set of the DB
should be in memory, but would a threaded approach slow squid down "that
much" ? I am thinking along the lines of a pthread that just sits there
and lazy writes dirty objects to disk. A quick lock of the global mutex,
copy the object, unlock mutex, write object, rinse repeat. There will
also have to be a shutdown hook...the helper process is looking pretty
good round about now :)

For some reason, version control systems and I have never been friends.
I can't get a copy of the source code using BZR (using the instructions
on the Squid wiki), so I will build against the latest .tar.gz that I
can find. I also suspect this will be new files and very little
modifying of current files.

Cheers,

Pieter
Received on Sat Feb 11 2012 - 04:50:28 MST