Summary

Test scripts that don't properly report exit status

Metadata

Description

The eggs listed bellow don't properly report their exit code to the test infrastructure.

Please, make them exit 0 when all tests pass or a non-zero value when one or more tests fail, so salmonella can properly indicate the tests status. Hint: for those using the test egg, you just need (test-exit) at the end of the test script.

If you test script relies on errors (or assert) to test egg features, you can assume it is ok (I'm sorry for the false alert), since on error the test script will exit 70 (which is ok for the test infrastructure).

I took a look at the last 20 lines of the run.scm files (or at the files they include/load) in the trunk dir for each egg (yeah, it's not exactly a pleasant work). If you don't intend to make a new release of your egg(s) to fix bugs or to add new features soon, please consider making a new release just to fix the tests, so we can have a more accurate idea about the shape of the set of eggs. Also consider that egg tests help to test the compiler too.

- discount (Stephen Eilert) - lookup-table (Kon Lovett) - mailbox (Felix Winkelmann, Kon Lovett) - mw (Tony Sidaway. Uses testeez, which has no way to exit on failure?) - mw-core (Tony Sidaway. Uses testeez, which has no way to exit on failure?) - numspell (Felix Winkelmann(?) uses testeez) - operations (Felix Winkelmann) - specialized-io (Jeronimo Pellegrini) - srfi-45 (Kon Lovett) - statistics (Peter Lane. Uses srfi-78, which has no test-exit equivalent) - synch (Kon Lovett) - system (Felix Winkelmann) - tinyclos (Felix Winkelmann. Uses testeez and is deprecated) - yelp (Derrell Piper)

The following eggs have tests that aren't really automated. If they can't be automated or you don't feel like doing that, please remove your eggs from the list:

- kanren (Alex Shinn) - lazyffi (Felix Winkelmann) - openal (Felix Winkelmann) - progress-indicators (Felix Winkelmann) - remote-mailbox (Kon Lovett) - xosd (Felix Winkelmann)

Changes and comments

[2011-07-02 21:55:07 UTC] felix wrote:

I updated advice, ftl, udp, defstruct, simple-sha1 and coops.

The following eggs don't use test, use assert or perform just simple code execution examples:

big-chicken, bind, dollar, fast-generic, fps, heap-o-rama, htmlprag, inline, loop, protobj, s11n, spock, sparse-vectors, stalin, z3.

[2011-07-05 04:37:57 UTC] iraikov changed description

[2011-07-12 04:20:23 UTC] iraikov changed description

[2011-07-12 04:24:55 UTC] iraikov changed description

[2011-07-12 08:20:22 UTC] kiatoa wrote:

logpro updated with (test-exit)

[2011-07-13 20:52:42 UTC] sjamaan changed description

[2011-07-13 20:52:42 UTC] sjamaan wrote:

Thank you for working through these things, Mario!

I've updated numbers (which is still not a great testsuite, but more-or-less works as expected now)

prometheus is also an egg that just uses assert and has a few examples.

I've removed these from the list as well as those Felix listed.

[2011-07-13 21:17:11 UTC] ckeen changed description

[2011-07-13 21:18:58 UTC] ckeen changed description

[2011-07-14 04:40:59 UTC] iraikov changed description

[2011-07-14 04:40:59 UTC] iraikov wrote:

Added test-exit for csv and digraph, and assert for datatype

[2011-07-14 04:48:57 UTC] iraikov changed description

[2011-07-14 04:48:57 UTC] iraikov wrote:

Removing mpi and memcached from the list, as their test scripts have been already fixed.

[2011-07-16 06:21:23 UTC] iraikov changed description

[2011-07-16 06:21:23 UTC] iraikov wrote:

Added test-exit to the test scripts of getopt-long and format-textdiff

[2011-07-22 17:02:50 UTC] iraikov changed description

[2011-07-22 17:02:50 UTC] iraikov wrote:

Added test-exit to the test scripts of filepath, graph-bfs, graph-dfs, graph-cycles

[2011-08-04 03:41:33 UTC] iraikov changed description

[2011-08-04 03:41:33 UTC] iraikov wrote:

Added test-exit to the test scripts for internet-message, graph-scc, iexpr

[2011-09-20 03:15:51 UTC] iraikov changed description

[2011-09-20 03:15:51 UTC] iraikov wrote:

Fixed test scripts of internet-message, lexgen, json-abnf, object-graph

[2011-10-05 08:11:51 UTC] iraikov changed description

[2011-10-05 08:11:51 UTC] iraikov wrote:

Fixed test scripts of mime, rb-tree, sfht, smtp.

[2011-10-06 09:54:31 UTC] felix removed milestone 4.8.0

[2011-11-03 03:15:52 UTC] iraikov changed description

[2011-11-03 03:15:52 UTC] iraikov wrote:

Using assert in treap, typeclass, tiny-prolog

[2012-05-08 19:54:05 UTC] sjamaan changed description

[2012-05-08 19:54:05 UTC] sjamaan wrote:

coops-utils has been modified in the meanwhile, it appears: http://tests.call-cc.org/master/linux/x86/2012/05/08/salmonella-report/test/coops-utils.html

[2012-05-08 19:55:48 UTC] sjamaan changed description

[2012-05-08 19:55:48 UTC] sjamaan wrote:

quaternoins has a hand-rolled test system which calls "error" when a test fails; this should cause exit status 1

[2013-01-30 20:30:57 UTC] sjamaan changed description

[2013-01-30 20:30:57 UTC] sjamaan wrote:

I've updated the master list with author annotations so we know who to contact.

I've removed a couple of eggs that had already been fixed:

- logpro (Matthew Welland) - ripemd (Kon Lovett) - sha2 (Aaron D. Gifford and Daishi Kato) - string-utils (Kon Lovett) - tiger-hash (Kon Lovett)

Also some eggs that have custom hand-rolled scripts which correctly exit with a nonzero status on error:

- bindings (Juergen Lorenz) - contracts (Juergen Lorenz) - expand-full (?, Kon Lovett) - packedobjects (John PT Moore) - sassy (Jonathan Kraut, this egg has a handful of tests that don't exit, but the majority do) - snowdigest (Felix Winkelmann) - srfi-25 (Jussi Piitulainen, Felix Winkelmann) - srfi-42 (Ivan Raikov) - tuples (Juergen Lorenz)

[2013-01-31 00:11:38 UTC] syn changed description

[2013-01-31 00:11:38 UTC] syn wrote:

Fixed char-set-literals, couchdb and netstring.

[2013-02-01 20:38:31 UTC] sjamaan changed description

[2013-02-20 02:31:10 UTC] kon wrote:

Fixed

- condition-utils (Kon Lovett) - dsssl-utils (Kon Lovett) - err5rs-arithmetic (Kon Lovett) UNRELEASED - format (Felix Winkelmann, Kon Lovett) - list-utils (Kon Lovett)

[2013-02-20 20:43:29 UTC] sjamaan changed description

[2013-02-20 20:43:29 UTC] sjamaan wrote:

Thanks Kon! I've removed these eggs from the master list so we can more clearly see which ones still need fixing

[2013-02-23 18:43:41 UTC] kon wrote:

Fixed (the rest of the low hanging fruit)

- macosx (this cannot be completely automated. since auto testing w/ non-macosx box anyway ...) - mathh - procedure-decoration - srfi-19 - srfi-27 - srfi-101 - stack - symbol-utils - uuid-ossp

[2013-02-25 09:26:25 UTC] sjamaan changed description

[2013-02-25 09:26:25 UTC] sjamaan wrote:

Updated description to reflect #25

[2013-08-15 18:57:14 UTC] sjamaan changed description

[2013-08-15 18:57:14 UTC] sjamaan wrote:

Fixed sexp-diff (I'm sure Felix won't mind)

[2013-08-18 21:15:48 UTC] sjamaan changed description

[2013-08-18 21:15:48 UTC] sjamaan wrote:

Fixed sxml-templates (and notified Klutometis)

[2013-11-12 20:19:28 UTC] sjamaan changed description

[2013-11-12 20:19:28 UTC] sjamaan wrote:

Noticed that yasos now uses simple-tests instead of testeez. Notified Juergen Lorenz about the fact that simple-tests has no way of indicating exit status.

[2013-11-14 20:50:08 UTC] sjamaan changed description

[2013-11-14 20:50:08 UTC] sjamaan wrote:

Removed yasos from the list. Juergen assured me that the new version of simple-test will call exit, as long as compound-test is used. Yasos does this.

[2014-01-09 20:34:35 UTC] mario changed description

[2014-01-09 20:34:35 UTC] mario wrote:

fmt has been fixed in version 0.802

[2014-05-02 17:18:59 UTC] chust wrote:

modular-arithmetic, elliptic-curves and pandora no longer have that bug since a while. pandora may have failed to install due to missing dependencies, but that should also be fixed now.

[2014-05-02 17:26:49 UTC] sjamaan changed description

[2014-05-02 17:26:49 UTC] sjamaan wrote:

Thanks for checking, Thomas. I've updated the description so your eggs are no longer listed.

[2015-09-11 20:21:23 UTC] sjamaan changed status from new to closed

[2015-09-11 20:21:23 UTC] sjamaan set resolution to wontfix

[2015-09-11 20:21:23 UTC] sjamaan wrote:

The remaining eggs are not likely to get fixed soon. We'll just ignore them and as soon as CHICKEN 5 eggs start appearing with missing exit statuses in tests, aggressively file new tickets for those eggs.