Re: squid test suite question

On 18/02/11 09:56, Ryan wrote:
> Dear Squid developers,
>
> I'm not sure if this is the appropriate mailing list to send to. If
> not, sorry for the spam and please tell me where can I get help.

This is the right place.

>
> I plan to do some coverage statistics of squid based on the in-house test suite.
> But the released version does not come with a test suite directory.
> The repository version has one,
> but seems mostly unit testing. Is that all the testing that has been
> done before release?

Unit-tests are pretty much all of what is automatically tested on a
systematic basis. They form the core basis of what we test BUT are still
very patchy despite attempts to cover more of the code. Help wanted to
extend them.

We have an audit process that code of any significance submitted needs
to pass at least two developer eyeballs before it enters even the main
testing process.

  *** At this point the code is deemed suitable for commit to the 3.HEAD
release branch and begins its formal testing. (this is being changed at
present to have a pre-commit stage as well).

 From 3.HEAD we have continuous integration testing through
Hudson/Jenkins at build.squid-cache.org which tests the full auto-tools
testing sequence using the test-suite scripts. The tests include whether
bootstrap works, basic compile, package, and the unit tests. They are
run for several permutations of the ./configure options (where the
option affects code being tested) every hour if the code has changed.
This is done on a range of different operating systems and compilers.

We also have certain structures built into the low-level code to fail to
build with unsafe functions (like sprintf or strcat with their buffer
overflow problems). And daily maintenance code updates to enforce some
good programming practices and retain consistent code style.

It is usually only at this point that its deemed suitable for
*potential* porting down to the beta branch depending on the maintainers
(myself) knowledge of the situation and possibly some additional manual
testing (below).

Each branch is run through the full BuildFarm level of testing *again*
after commit to ensure the porting did not corrupt anything.

On top of this there is an additional single build test run during the
tarball generation. Largely redundant now.

** At this point we have the daily tarball and rsync gets updated for
beta testers to begin their manual checks on the new behaviour.

Patches which have all the above and some extra delay for the beta
testes to catch behaviour problems are potentials for porting down
further into the production branch. Where they face the testing process
all over again before the daily production release tarball goes out.

Additional to the units and in-house automated stuff:

At irregular periods we also manually run valgrind memory leak checks,
cppcheck static analysis. Or other tools we find that may provide useful
checks. Before their systems got broke for our project we also used to
participate in Coverity scan project.
  IRcache, The Measurement Factory and other institutions also run
testing systems and scans which are not open to the general public (or
even ourselves at times) but provide feedback on what needs fixing.

On top of all that machine testing we have people in various networks
attempting to run the 3.HEAD and/or the beta code for personal LAN
through to production network use. The squid project websites are among
these.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.11
   Beta testers wanted for 3.2.0.5