r13295: cache_mem + rock store = "memCopy: could not find start of [x,y) in memory"

While testing the large rock store with trunk revision 13295 I
stumbled upon at the following nasty bug:

<cache.log>
2014/03/02 05:33:24 kid1| memCopy: could not find start of [374,32768)
in memory.
2014/03/02 05:33:24 kid1| mem_hdr::debugDump: lowest offset: 0 highest
offset + 1: 10156823.
2014/03/02 05:33:24 kid1| mem_hdr::debugDump: Current available data
is: [0,374) - .
2014/03/02 05:33:24 kid1| Closing HTTP port 10.240.254.52:8081
2014/03/02 05:33:24 kid1| storeDirWriteCleanLogs: Starting...
2014/03/02 05:33:24 kid1| Finished. Wrote 0 entries.
2014/03/02 05:33:24 kid1| Took 0.00 seconds ( 0.00 entries/sec).
FATAL: Squid has attempted to read data from memory that is not
present. This is an indication of of (pre-3.0) code that hasn't been
updated to deal with sparse objects in memory. Squid should
coredump.allowing to review the cause. Immediately preceding this
message is a dump of the available data in the format [start,end). The
[ means from the value, the ) means up to the value. I.e. [1,5) means
that there are 4 bytes of data, at offsets 1,2,3,4.

Squid Cache (Version 3.HEAD.13295): Terminated abnormally.
</cache.log>

It doesn't happen when browsing solo, but the moment I redirect 150
mbps of real user traffic Squid crashes immediately and keeps crashing
with this same message again and again.

I managed to isolate the reason for crashing - cache_mem != 0. Here's
my simple test config:

<squid.conf>
http_port 10.240.254.52:8081 tproxy
http_access allow all
cache allow all
cache_mem 0 MB
debug_options ALL,1
coredump_dir /var/log/squid3
shutdown_lifetime 0
cache_dir rock /test 100 max-size=32753
</squid.conf>

With this config, Squid works OK. Changing cache_mem to anything
different then zero makes Squid crashing . Same config with aufs works
OK even with cache_mem enabled. Squid 3.4.3 also works OK with
cache_mem != 0, so it has to be a large rock related issue.

Unfortunately, no core dumps are available for unknown reason. kill -3
<pid of kid1> generates core file in /var/log/squid3, but the crash -
doesn't. Tried running squid -C - didn't help - still no core dump
generated during to the crash.

Ubuntu 12.04.04 Server 64-bit.

Configure options: '--prefix=/usr'
'--libexecdir=${prefix}/lib/squid3' '--sysconfdir=/etc/squid3'
'--localstatedir=/var' '--sharedstatedir=${localstatedir}/run/squid'
'--with-pidfile=${localstatedir}/run/squid3.pid'
'--with-logdir=${localstatedir}/log/squid3'
'--datadir=${prefix}/share/squid3' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--enable-inline'
'--enable-storeio=aufs,rock' '--enable-removal-policies=lru,heap'
'--disable-delay-pools' '--enable-esi' '--disable-ecap'
'--disable-wccp' '--disable-wccpv2' '--enable-snmp' '--enable-eui'
'--enable-htcp' '--disable-ssl' '--disable-cache-digests'
'--enable-http-violations' '--enable-linux-netfilter'
'--disable-follow-x-forwarded-for' '--disable-auth'
'--disable-auth-basic' '--disable-auth-ntlm'
'--disable-auth-negotiate' '--disable-auth-digest'
'--enable-log-daemon-helpers=file'
'--enable-external-acl-helpers=none' '--disable-url-rewrite-helpers'
'--enable-storeid-rewrite-helpers' '--disable-ipv6' '--enable-zph-qos'
'--disable-translation' '--disable-auto-locale' '--with-included-ltdl'
'--with-default-user=proxy' '--with-large-files'
'--without-krb5-config' '--with-filedescriptors=524288'
'--enable-underscores' --enable-ltdl-convenience

If you need any further clarification, please let me know.

Best,
Niki
Received on Mon Mar 03 2014 - 23:04:27 MST