Re: Chunked memory allocator

Henrik Nordström wrote:
> It has come to light that the chunked allocator have a couple of
> fundamental design issues.
>
> The primary is that free space management becomes overly complex,
> causing significant performance issues.
>
> The second is that you can not really disable the use of pools by
> configuration, even if we claim this in squid.conf. The only way to
> control this runtime is by setting the undocumented(?) MEMPOOLS
> environment variable or by running under valgrind. I suspect the main
> effect of trying to disable or tune down usage of pools via squid.conf
> today is only significantly worsened performance.
>
> As I see it we basically have two choices
>
> a) Remove the memory pools config parameters from MemPools & squid.conf
> + related "impossible" free space reclaim code.
>
> b) Revert to the older much simpler pooled design.
>

I recall Adrian arguing for dropping the old allocator on grounds that
malloc was more efficient these days. If thats true then (a) would be
better.

> The main benefit of the chunked allocator is that it can pack data more
> efficiently than malloc() taking benefit of our typed allocations.
>
> In terms of fragmentation reduction there is very limited benefits
>
> And there is obvious drawbacks in free space reclaim efficiency.
>
> Regards
> Henrik
>

Sounds like the packing is grounds for making a sane chunked allocator.
Perhapse designed in place of the existing, or a complete re-design
using the lessons learned from this one.

Or is anyone aware of an off-the-shelf implementation of a chunked
allocator we can gain the same typed allocation benefits from?

As usual, there is a decided lack of current benchmark testing on these
things.

Right now in light of the issues that keep cropping up my vote for
squid-3.1 is for (a). Dropping of the impossible settings in squid.conf,
with real default-disabled mempools. Plus some clear documentation of
the environment variable to turn them on. Or would it be better to make
it a command-line option for startup?

For squid-3.2;
  I have not yet taken a good close look at the SMP project
implementation details yet, but the final architecture design allowed
for the master process to use a regular inefficient allocator in its
binary and the child instance binaries to use more efficient ones.

On other slightly related memory issues... some users who need good
malloc have been trying to build squid with talloc or the google malloc
implementation. I'd like to see such alternative allocators easy to
plug in. If it already is that way then we lack documentation about how
to do it.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.3