Opened 4 months ago

Closed 4 months ago

#1358 closed defect (fixed)

(test-exit) returns 0 code for 'error in group outside of tests'

Reported by: caolan Owned by: ashinn
Priority: major Milestone: someday
Component: extensions Version: 4.12.0
Keywords: Cc:
Estimated difficulty:

Description

I'd expect the example below to exit with code 1, instead it exits with code 0:

(use test)

(test-group "A"
  (abort "error"))

(test-group "B"
  (test-assert #t))

(test-exit)
$ csi -s run.scm 

Warning: error in group outside of tests

Error: error
1 test completed in 0.0 seconds.
1 error (100%).
0 out of 1 (0%) tests passed.
-- done testing A ------------------------------------------------------------


-- testing B -----------------------------------------------------------------
#t ................................................................... [ PASS]
1 test completed in 0.0 seconds.
1 out of 1 (100%) test passed.
-- done testing B ------------------------------------------------------------

$ echo $?
0

Change History (4)

comment:1 Changed 4 months ago by caolan

And here is my suggested fix:

--- test/test.scm	2017-04-04 14:11:51.645769430 +0100
+++ test-edit/test.scm	2017-04-04 14:10:56.045335979 +0100
@@ -88,6 +88,7 @@
                        (e ()
                           (warning "error in group outside of tests")
                           (print-error-message e)
+                          (test-failure-count (+ 1 (test-failure-count)))
                           (test-group-inc! (current-test-group) 'count)
                           (test-group-inc! (current-test-group) 'ERROR)))
        (test-end name)

comment:2 Changed 4 months ago by sjamaan

Perhaps a better fix would involve adding a new errors-outside-test-count parameter? Because increasing test-failure-count could result in the impression that there are more tests than actually exist, if the error happens after the last test.

Also, in custom reporters it might be helpful to know that there were errors outside the tests, so the test count is unreliable (because it wouldn't get around to running the remaining tests that follow the code that triggered the error.

comment:3 Changed 4 months ago by ashinn

a separate count might be better, but we only use the current count for test-exit right now, and I'm busy :P

comment:4 Changed 4 months ago by ashinn

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.