chicken.base#implicit-exit-handler reporting segmentation violation with non-default heap size

[vjorlikowski]

I was experimenting with the "Man-or-boy" test
(​http://rosettacode.org/wiki/Man_or_boy_test) using Chicken Scheme;
my variant looks like this:

vjo@glaucus/pts/11:chicken/man_or_boy % cat test.scm
(import (chicken process-context)
        (chicken fixnum))

(define (A k x1 x2 x3 x4 x5)
  (let
      ([+ fx+]
       [- fx-]
       [<= fx<=])
    (define (B)
      (set! k (- k 1))
      (A k B x1 x2 x3 x4))
    (if (<= k 0)
        (+ (x4) (x5))
        (B))))

(define k (let ([args (command-line-arguments)])
            (if (> (length args) 0)
                (string->number (car args))
                10)))
(print (A k
          (lambda () 1)
          (lambda () -1)
          (lambda () -1)
          (lambda () 1)
          (lambda () 0)))

This compiles fine, and runs great with the default heap size up to
a value of k=27:

vjo@glaucus/pts/11:chicken/man_or_boy % csc -O4 test.scm
vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 26
-21051458
./test 26  25.20s user 1.31s system 99% cpu 26.577 total
vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 27
-46750171
./test 27  101.91s user 4.42s system 99% cpu 1:46.66 total

At k=28, however, we run out of heap, with the default settings:

vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 28
[panic] out of memory - heap has reached its maximum size - execution terminated

...more...
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:13: x4
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:13: x4
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A  <--
./test 28  22.39s user 3.79s system 99% cpu 26.282 total

No problem, say I; let me re-compile with an increased heap.
That works well - and the code gives back the correct answer - but,
I now encounter the bug I wish to report:

vjo@glaucus/pts/11:chicken/man_or_boy % csc -O4 -heap-size 4096M test.scm
vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 28
-103821058

Error: segmentation violation

        Call history:

        test.scm:13: x5
        test.scm:11: A
        test.scm:13: x4
        test.scm:13: x5
        test.scm:13: x5
        test.scm:11: A
        test.scm:13: x4
        test.scm:11: A
        test.scm:13: x4
        test.scm:13: x5
        test.scm:13: x5
        test.scm:11: A
        test.scm:13: x4
        test.scm:13: x5
        test.scm:20: chicken.base#print
        chicken.base#implicit-exit-handler              <--
./test 28  90.91s user 1.71s system 99% cpu 1:32.65 total

It *appears* that the implicit-exit-handler triggers a segmentation
violation for any binary compiled with a non-default heap size
specified; as an illustrative example, here is the same code
compiled with two different heap sizes and executed using k=26. The
first shows what happens with an insufficient heap size, the second
shows what happens when the heap size is *just barely* sufficient,
but non-default value:

vjo@glaucus/pts/11:chicken/man_or_boy % csc -O4 -heap-size 928M test.scm
vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 26
[panic] out of memory - heap full - execution terminated

...more...
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:13: x4
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:13: x4
test.scm:11: A
test.scm:14: B
test.scm:11: A
test.scm:14: B
test.scm:11: A  <--
./test 26  5.01s user 0.44s system 99% cpu 5.471 total

vjo@glaucus/pts/11:chicken/man_or_boy % csc -O4 -heap-size 929M test.scm
vjo@glaucus/pts/11:chicken/man_or_boy % time ./test 26
-21051458

Error: segmentation violation

        Call history:

        test.scm:13: x5
        test.scm:13: x5
        test.scm:11: A
        test.scm:13: x4
        test.scm:11: A
        test.scm:13: x4
        test.scm:11: A
        test.scm:13: x4
        test.scm:13: x5
        test.scm:13: x5
        test.scm:13: x5
        test.scm:11: A
        test.scm:13: x4
        test.scm:13: x5
        test.scm:20: chicken.base#print
        chicken.base#implicit-exit-handler              <--
./test 26  25.87s user 0.73s system 99% cpu 26.620 total

The versions of all software involved have been determined below:

vjo@glaucus/pts/11:chicken/man_or_boy % csc -O4 -kv test.scm          
'/usr/bin/chicken' 'test.scm' -output-file 'test.c' -optimize-level 4 -verbose
'x86_64-linux-gnu-gcc' 'test.c' -o 'test.o' -c  -fno-strict-aliasing -fwrapv
-DHAVE_CHICKEN_CONFIG_H -DC_ENABLE_PTABLES -g -O2
-fdebug-prefix-map=/build/chicken-8b5V5S/chicken-5.1.0=. -fstack-protector-strong -Wformat
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include/chicken
'x86_64-linux-gnu-gcc' 'test.o' -o 'test' -L/usr/lib -Wl,-R/usr/lib -lchicken -lm -ldl

vjo@glaucus/pts/11:chicken/man_or_boy % x86_64-linux-gnu-gcc --version
x86_64-linux-gnu-gcc (Ubuntu 9.2.1-9ubuntu2) 9.2.1 20191008
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

vjo@glaucus/pts/11:chicken/man_or_boy % csc -version                  
CHICKEN
(c) 2008-2019, The CHICKEN Team
(c) 2000-2007, Felix L. Winkelmann
Version 5.1.0 (rev 8e62f718)
linux-unix-gnu-x86-64 [ 64bit dload ptables ]

[vjorlikowski]

Man-or-boy test file source

[sjamaan]

NOTE: This only happens when compiling with -maximum-heap-size. When passing -:hm at runtime, I could not reproduce this issue.

[sjamaan]

This looks like some sort of corruption in the GC, as the pointer to the continuation of cleanup-before-exit seems to be broken.

[sjamaan]

Possibly there's a bug in the reintern option of C_set_or_change_heap_size, as that differs between -:h and -heap-size

[megane]

Just compiling an empty program with -heap-size gives the same result.

[sjamaan]

It looks like when C_set_or_change_heap_size is invoked with reinit=1, it only recreates the symbol table, but the termination continuation is still allocated on the old heap which is now in freed memory. This was allocated by CHICKEN_initialize.

I tried to copy that bit into the if(reintern) ... block of C_set_or_change_heap_size so that the termination continuation is pointing into the correct heap, but that didn't seem to fix the situation. I also tried resetting the temporary stack before doing this, but that didn't help either. I guess by the time the toplevel is called, the continuation pointer that's passed to it is still pointing into the old heap.

I think we need to longjump back and re-invoke the toplevel. But then it needs to know that it shouldn't try to resize the heap *again*. Or somehow we must do something with the termination continuation. Maybe not allocate it on the heap?

Possibly instead treat the heap reallocation as a reallocating GC?

[megane]

Patch posted ​https://lists.nongnu.org/archive/html/chicken-hackers/2021-03/msg00009.html