Re: squid-fs some very basic info

On Fri, Sep 11, 1998 at 01:37:42AM +0200, Oskar Pearson wrote:

> I was reading linux-kernel the other day and saw a problem where
> someone had changed the block size, and could no longer make very
> large files on that device.

That might have been me - I was having trouble creating >17GB files
on my Alpha because my block size was (accidentally) 1K. Using a 4K
block size makes more sense, and I should have been doing that all
along. A 4K block since means the ext2fs triple-indirection limit is
around 170TB, which is presently more disk that I have available
right now.

8K pages make even more sense of the alpha, but then the FS isn't
going to be portable to architectures with 4k pages (well, unless we
forego mmap.

> I just realized that the number of 'indirect' block pointers could
> be reduced on Linux's ext2 filesystem by specifying a larger block
> size, and this has an if the squid-fs stuff seems worthwhile.

I don't think squidFS need to work this way. We don't necessarily
want to support sparse files (although doing so might be a nice hack
for partial range storage).

> If we have a really large file (on a filesystem, not on an actual
> /dev entry...) a very large block size would reduce the number of
> indirect blocks that we need. People should re-format their cache
> store filesystem to have a very large block size.

Ahh.. but if you average object size is say 10.5K, and your using 4K
blocks - are you not then wasting 2.5K/object is storage compared to
0.5K with 1K blocks?

2K pages gives us 1 loss of 1.5K/object. Unless you have a very busy
squid, using 4K blocks over 2K or 1K isn't going to be noticeable
(well, I don't think so).

> I don't know if putting Squid-FS on a /dev/ entry is such a good
> idea.

I think it probably is... having raw device access and a simplified
filesystem structure for squidFS might perhaps increase effective
disk-IO by a factor of two or more, but it needs to be done right.

Right now, there is a pretty clear idea of what features are required
to make a decent squidFS, the only thing that bothers me is that any
assumptions we may make also need to work for the `web of tomorrow',
hence my earlier questions on this list about ranges and partial
object storage.

Partial object storage to me seems akward and problematic to support
efficiently. Since its an apparently rare event (0.1% or so), we
could store the hunks as individual objects with a chaining object
(or is this too ugly a hack?).

-cw
Received on Tue Jul 29 2003 - 13:15:53 MDT