(test-exit) returns 0 code for 'error in group outside of tests'

[Caolan McMahon]

I'd expect the example below to exit with code 1, instead it exits with code 0:

(use test)

(test-group "A"
  (abort "error"))

(test-group "B"
  (test-assert #t))

(test-exit)

$ csi -s run.scm 

Warning: error in group outside of tests

Error: error
1 test completed in 0.0 seconds.
1 error (100%).
0 out of 1 (0%) tests passed.
-- done testing A ------------------------------------------------------------


-- testing B -----------------------------------------------------------------
#t ................................................................... [ PASS]
1 test completed in 0.0 seconds.
1 out of 1 (100%) test passed.
-- done testing B ------------------------------------------------------------

$ echo $?
0

[Caolan McMahon]

And here is my suggested fix:

--- test/test.scm	2017-04-04 14:11:51.645769430 +0100
+++ test-edit/test.scm	2017-04-04 14:10:56.045335979 +0100
@@ -88,6 +88,7 @@
                        (e ()
                           (warning "error in group outside of tests")
                           (print-error-message e)
+                          (test-failure-count (+ 1 (test-failure-count)))
                           (test-group-inc! (current-test-group) 'count)
                           (test-group-inc! (current-test-group) 'ERROR)))
        (test-end name)

[sjamaan]

Perhaps a better fix would involve adding a new errors-outside-test-count parameter? Because increasing test-failure-count could result in the impression that there are more tests than actually exist, if the error happens after the last test.

Also, in custom reporters it might be helpful to know that there were errors outside the tests, so the test count is unreliable (because it wouldn't get around to running the remaining tests that follow the code that triggered the error.

[Alex Shinn]

a separate count might be better, but we only use the current count for test-exit right now, and I'm busy :P