Summary
(test-exit) returns 0 code for 'error in group outside of tests'
Metadata
- Id: 280b3a532041b16f182bced020df9505a18f03fe
- Trac id: 1358
- Type: defect
- Reporter: caolan
- Owner: ashinn
- Cc:
- Status: closed
- Component: extensions
- Estimated difficulty:
- Resolution: fixed
- Priority: major
- Milestone: someday
- Version: 4.12.0
- Changetime: 2017-04-04 15:53:13 UTC
- Created: 2017-04-04 15:06:15 UTC
- Keywords:
Description
I'd expect the example below to exit with code 1, instead it exits with code 0:
(use test) (test-group "A" (abort "error")) (test-group "B" (test-assert #t)) (test-exit) $ csi -s run.scm Warning: error in group outside of tests Error: error 1 test completed in 0.0 seconds. 1 error (100%). 0 out of 1 (0%) tests passed. -- done testing A ------------------------------------------------------------ -- testing B ----------------------------------------------------------------- #t ................................................................... [ PASS] 1 test completed in 0.0 seconds. 1 out of 1 (100%) test passed. -- done testing B ------------------------------------------------------------ $ echo $? 0
Changes and comments
[2017-04-04 15:12:56 UTC] caolan wrote:
And here is my suggested fix:
--- test/test.scm 2017-04-04 14:11:51.645769430 +0100
+++ test-edit/test.scm 2017-04-04 14:10:56.045335979 +0100
@@ -88,6 +88,7 @@
(e ()
(warning "error in group outside of tests")
(print-error-message e)
+ (test-failure-count (+ 1 (test-failure-count)))
(test-group-inc! (current-test-group) 'count)
(test-group-inc! (current-test-group) 'ERROR)))
(test-end name)
[2017-04-04 15:21:55 UTC] sjamaan wrote:
Perhaps a better fix would involve adding a new errors-outside-test-count parameter? Because increasing test-failure-count could result in the impression that there are more tests than actually exist, if the error happens after the last test.
Also, in custom reporters it might be helpful to know that there were errors outside the tests, so the test count is unreliable (because it wouldn't get around to running the remaining tests that follow the code that triggered the error.
[2017-04-04 15:53:00 UTC] ashinn wrote:
a separate count might be better, but we only use the current count for test-exit right now, and I'm busy :P