Opened 11 years ago

Last modified 7 years ago

#630 closed defect

Test scripts that don't properly report exit status — at Version 19

Reported by: Mario Domenech Goulart Owned by:
Priority: major Milestone:
Component: extensions Version: 4.7.x
Keywords: test infrastructure Cc:
Estimated difficulty:

Description (last modified by sjamaan)

The eggs listed bellow don't properly report their exit code to the test infrastructure.

Please, make them exit 0 when all tests pass or a non-zero value when one or more tests fail, so salmonella can properly indicate the tests status. Hint: for those using the test egg, you just need (test-exit) at the end of the test script.

If you test script relies on errors (or assert) to test egg features, you can assume it is ok (I'm sorry for the false alert), since on error the test script will exit 70 (which is ok for the test infrastructure).

I took a look at the last 20 lines of the run.scm files (or at the files they include/load) in the trunk dir for each egg (yeah, it's not exactly a pleasant work). If you don't intend to make a new release of your egg(s) to fix bugs or to add new features soon, please consider making a new release just to fix the tests, so we can have a more accurate idea about the shape of the set of eggs. Also consider that egg tests help to test the compiler too.

bindings
char-set-literals
condition-utils
contracts
couchdb
directory-utils
discount
dsssl-utils
elliptic-curves
err5rs-arithmetic
expand-full
fmt
format
kanren
lazy-ffi
list-utils
logpro
lookup-table
macosx
mailbox
mathh
modular-arithmetic
mw-core
mw
netstring
numspell
openal
operations
packedobjects
pandora
procedure-decoration
progress-indicators
remote-mailbox
ripemd
sassy
sexp-diff
sha2
snowdigest
specialized-io
srfi-101
srfi-19
srfi-25
srfi-27
srfi-42
srfi-45
stack
statistics
string-utils
sxml-templates
symbol-utils
synch
system
tigher-hash
tinyclos
tuples
uuid-ossp
xosd
yasos
yelp

Change History (19)

comment:1 Changed 11 years ago by felix winkelmann

I updated advice, ftl, udp, defstruct, simple-sha1 and coops.

The following eggs don't use test, use assert or perform just simple code execution examples:

big-chicken, bind, dollar, fast-generic, fps, heap-o-rama, htmlprag, inline, loop, protobj, s11n,
spock, sparse-vectors, stalin, z3.

comment:2 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

comment:3 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

comment:4 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

comment:5 Changed 11 years ago by Matt Welland

logpro updated with (test-exit)

comment:6 Changed 11 years ago by sjamaan

Description: modified (diff)

Thank you for working through these things, Mario!

I've updated numbers (which is still not a great testsuite, but more-or-less works as expected now)

prometheus is also an egg that just uses assert and has a few examples.

I've removed these from the list as well as those Felix listed.

comment:7 Changed 11 years ago by Christian Kellermann

Description: modified (diff)

comment:8 Changed 11 years ago by Christian Kellermann

Description: modified (diff)

comment:9 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Added test-exit for csv and digraph, and assert for datatype

comment:10 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Removing mpi and memcached from the list, as their test scripts have been already fixed.

comment:11 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Added test-exit to the test scripts of getopt-long and format-textdiff

comment:12 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Added test-exit to the test scripts of filepath, graph-bfs, graph-dfs, graph-cycles

comment:13 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Added test-exit to the test scripts for internet-message, graph-scc, iexpr

comment:14 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Fixed test scripts of internet-message, lexgen, json-abnf, object-graph

comment:15 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Fixed test scripts of mime, rb-tree, sfht, smtp.

comment:16 Changed 11 years ago by felix winkelmann

Milestone: 4.8.0

comment:17 Changed 11 years ago by Ivan Raikov

Description: modified (diff)

Using assert in treap, typeclass, tiny-prolog

comment:18 Changed 10 years ago by sjamaan

Description: modified (diff)

comment:19 Changed 10 years ago by sjamaan

Description: modified (diff)

quaternoins has a hand-rolled test system which calls "error" when a test fails; this should cause exit status 1

Note: See TracTickets for help on using tickets.