Opened 3 years ago

Closed 3 years ago

#1358 closed defect (fixed)

(test-exit) returns 0 code for 'error in group outside of tests'

Reported by: Caolan McMahon Owned by: Alex Shinn
Priority: major Milestone: someday
Component: extensions Version: 4.12.0
Keywords: Cc:
Estimated difficulty:

Description

I'd expect the example below to exit with code 1, instead it exits with code 0:

(use test)

(test-group "A"
  (abort "error"))

(test-group "B"
  (test-assert #t))

(test-exit)
$ csi -s run.scm 

Warning: error in group outside of tests

Error: error
1 test completed in 0.0 seconds.
1 error (100%).
0 out of 1 (0%) tests passed.
-- done testing A ------------------------------------------------------------


-- testing B -----------------------------------------------------------------
#t ................................................................... [ PASS]
1 test completed in 0.0 seconds.
1 out of 1 (100%) test passed.
-- done testing B ------------------------------------------------------------

$ echo $?
0

Change History (4)

comment:1 Changed 3 years ago by Caolan McMahon

And here is my suggested fix:

--- test/test.scm	2017-04-04 14:11:51.645769430 +0100
+++ test-edit/test.scm	2017-04-04 14:10:56.045335979 +0100
@@ -88,6 +88,7 @@
                        (e ()
                           (warning "error in group outside of tests")
                           (print-error-message e)
+                          (test-failure-count (+ 1 (test-failure-count)))
                           (test-group-inc! (current-test-group) 'count)
                           (test-group-inc! (current-test-group) 'ERROR)))
        (test-end name)

comment:2 Changed 3 years ago by sjamaan

Perhaps a better fix would involve adding a new errors-outside-test-count parameter? Because increasing test-failure-count could result in the impression that there are more tests than actually exist, if the error happens after the last test.

Also, in custom reporters it might be helpful to know that there were errors outside the tests, so the test count is unreliable (because it wouldn't get around to running the remaining tests that follow the code that triggered the error.

comment:3 Changed 3 years ago by Alex Shinn

a separate count might be better, but we only use the current count for test-exit right now, and I'm busy :P

comment:4 Changed 3 years ago by Alex Shinn

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.