Re: Squid & recovery of metadata

Web server manager wrote:

> I could not see any way to get Squid to load the available metadata from
> swap.state but *also* then scan the cache directories as a background
> activity to find any "lost" files, mislaid because their existence could not
> be recorded in swap.state.
>
> I had to delete swap.state, which forced it to scan the cache directories,
> *but* that in turn nearly led to further problem/potential problems.

Yes, Squids "from cache rebuild" is way far from optimal. A lot can be
improved there. One idea I have is to let Squid start up as if the cache
was empty, and then have a background task rebuild the cache metadata
independently.

Related to this, Squid should monitor the amount of free disk space and
not only it's accounted use of the disk and stop storing objects if
there is to little space left (won't help Solaris fragmentation
thought). Most OS:es have statfs / statvfs to query filesystem usage.

> Alternatively or in addition, perhaps Squid could - as a background
> activity - check the content of cache directories (slowly, one complete scan
> per week or even per month would suffice except when there was a known
> problem) and if it found files that it did not already know about, extract
> their metadata and add them back to the master metadata.

Squid is supposed to already have a background task (storeDirClean)
cleaning one directory each 15 seconds. See store_clean.c. This task
walks the entire L0/L1/L2 directory tree (L0 == cache_dir), so unless
your cache_dir L1 parameter is somewhat in range it will spend a lot of
time checking unused directories until it wraps and starts from the
beginning.

--
Henrik Nordstrom
Spare time Squid hacker