> Hello,
>
> When working on SourceLayout I got a few false errors from
> build-test.sh when it could not find some string in the make output. The
> attached patch fixes that by relying on make exit code rather than some
> magic string some make targets may print in the middle of the make
> process.
Target string is logged by the master make file _after_ all make targets
are completed successfully. Anything else is a bug, either in code or in
TestBed.
More Major Warning Bells are going off in my head on that Alex. It's
particularly important to catch the bugs in TestBed this brings out.
Yes I have seen the message occuring several times in the midst of a
distcheck run. However thay only signify successful distcheck unit tests.
Only the final output at the end signifies a successful test.
PS, I've committed a fix to trunk that now catches the magic string again
properly. It was due to the sudden absence of additional automake
debug/stderr info in the log. Where that info has been dropped to is a
minor bug which we can accept but really needs fixing itself.
> The build is still considered failed if error strings are found
> in the log.
>
More Major Warning Bells are going off in my head on that Alex.
The first design attempt used this enumerating-badness method. Far too
many false-positives were getting through. I'd rather face the odd
false-negative like this than hobble the testing.
The nature of the false-positive is that we cannot determine what exactly
a problem is before the testbed catches it. So its blind luck if we catch
any at all. The $errors scan is just to provide some quick additional
hints what the problem might be.
> I was also annoyed on how the test would quit on the first error,
> wasting a lot of test time if the first error gets unnoticed. The
> attached patch adds a --keep-going option, similar to "make -k". The
> option is off by default so I am not changing the default behavior here.
Okay. This is acceptable. IMO its just as wasteful of time resource to run
3x long tests when running one is sufficient to catch an error. That old
CS adage of 'fix the first error. the rest are probably caused by it'.
>
> Finally, I made test spec names processing a little bit more flexible.
> It is now possible to explicitly specify multiple tests, similar to
> "make all check distcheck".
Way cool. This will solve some wetware problems :)
Hmm, on that 'make all check distcheck'....
Do you know if doing that is faster than make all && make check && make
distcheck? Maybe we should be doing that to speed things up a little.
>
> More detailed comments are below. Please review.
>
> Thank you,
>
> Alex.
> bb:approve
Checked. Certain parts are great. The whole is not at this point.
Amos
bb:reject
>
> ------- change log -----------
> Support multiple test spec arguments. If at least one test fails, the
> script
> exits with a non-zero code (but possibly not immediately, see
> --keep-going).
>
> Each test spec is a test config file name or a well-known config name
> (no path
> or extension!). If no specs are given, all known test specs are used (as
> before). The same happens if the only test spec given is 'all'. The
> following are now equivalent:
>
> ./test-builds.sh
> ./test-builds.sh all
> ./test-builds.sh btlayer-00-default btlayer-01-minimal
> btlayer-02-maximus
> ./test-builds.sh test-suite/buildtests/layer-*
>
> You can mix file names and spec names, but not the 'all' macro: There is
> currently no support for using 'all' together with other test cases.
>
> Tolerate individual test errors if --keep-going is specified. This helps
> when
> one wants to find more errors than just the first one, especially when
> tests
> are long and are running without a human watching.
>
> When detecting test failures, rely on test-suite/buildtest.sh exit
> status code
> rather than on the presence of error-like strings in the log file.
>
> Added and polished comments.
> -----------------------------------
>
Received on Mon Mar 09 2009 - 22:01:24 MDT
This archive was generated by hypermail 2.2.0 : Tue Mar 10 2009 - 12:00:05 MDT