csc -deploy fails with modules

[Norman Gray]

csc -deploy fails when using modules.

Consider the following:

% cat hello.scm                                   
(module hello (hello)
  (import scheme)
  (define (hello)
    (display "Hello, world")
    (newline)))
% csc -version                                    
(c) 2008-2015, The CHICKEN Team
(c) 2000-2007, Felix L. Winkelmann
Version 4.10.0 (rev b259631)
macosx-unix-clang-x86-64 [ 64bit manyargs dload ptables ]
compiled 2015-08-04 on yves.more-magic.net (Linux)

Enter `chicken -help' for information on how to use the compiler,
or try `csc' for a more convenient interface.

Run `csi' to start the interactive interpreter.
% csc -emit-all-import-libraries -shared hello.scm
% cat main.scm
(use hello)
(hello)
% csc -deploy main.scm
% main/main -:d
[debug] application startup...
[debug] heap resized to 1048576 bytes
[debug] stack bottom is 0x7fff5865a820.
[debug] entering toplevel toplevel...
[debug] entering toplevel library_toplevel...
[debug] entering toplevel build_2dversion_toplevel...
[debug] entering toplevel eval_toplevel...
[debug] entering toplevel expand_toplevel...
[debug] entering toplevel modules_toplevel...
[debug] resizing mutation-stack from 8k to 16k ...
[debug] entering toplevel chicken_2dsyntax_toplevel...
; loading ./hello.so ...
[debug] loading compiled module `./hello.so' (handle is 0x00007fc93150daa0)
[panic] nursery is too small - try higher setting using the `-:s' option - execution terminated

% otool -L main/main
main/main:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
	@executable_path/libchicken.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Expected behaviour:

Running main/main would type "Hello, world"

Actual behaviour:

Running main/main produces a panic.

Notes:

This is on OS X 10.10

The file hello.scm is a simplified version of the module in page 'Modules' of the manual; the use of the -deploy option is as suggested on the page ​http://wiki.call-cc.org/man/4/Deployment

Copying hello.so into the directory main/ (worth a try...) produces the same result.

Guessing the following doesn't work either:

% main/main -:s100000000 
[panic] nursery is too small - try higher setting using the `-:s' option - execution terminated

% 

[sjamaan]

Could you please try ​CHICKEN 4.11.0rc2? We've fixed several OS X/linking related things.

We're about to release 4.11, so you could also wait for that.

[Norman Gray]

No change, I'm afraid.

With hello.scm and main.scm as before:

% csc -version
(c) 2008-2016, The CHICKEN Team
(c) 2000-2007, Felix L. Winkelmann
Version 4.11.0rc2 ((detached from 4.11.0rc2)) (rev e910197)
macosx-unix-clang-x86-64 [ 64bit manyargs dload ptables ]
compiled 2016-04-28 on waldrop (Linux)

Enter `chicken -help' for information on how to use the compiler,
or try `csc' for a more convenient interface.

Run `csi' to start the interactive interpreter.
% csc -emit-all-import-libraries -shared hello.scm
% csc -deploy main.scm
% main/main -:d
[debug] application startup...
[debug] heap resized to 1048576 bytes
[debug] stack bottom is 0x7fff5cbdd7f0.
[debug] entering toplevel toplevel...
[debug] entering toplevel library_toplevel...
[debug] entering toplevel build_2dversion_toplevel...
[debug] entering toplevel eval_toplevel...
[debug] entering toplevel expand_toplevel...
[debug] entering toplevel modules_toplevel...
[debug] resizing mutation-stack from 8k to 16k ...
[debug] entering toplevel chicken_2dsyntax_toplevel...
; loading ./hello.so ...
[debug] loading compiled module `./hello.so' (handle is 0x00007ffae8d0d880)
[panic] nursery is too small - try higher setting using the `-:s' option - execution terminated
% otool -L main/main
main/main:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
	@executable_path/libchicken.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
% uname -a
Darwin ptolemy.astro.gla.ac.uk 14.5.0 Darwin Kernel Version 14.5.0: Mon Jan 11 18:48:35 PST 2016; root:xnu-2782.50.2~1/RELEASE_X86_64 x86_64
%

This is OS X 10.10.5

[sjamaan]

Could you run gdb on this, break on panic, and show us the output of backtrace?

[Norman Gray]

Happy to, but ermmmm: could you advise me how best to do that? The various breakpoint specifications I thought would work, in both gdb and lldb, don't seem to have any effect (see below; I get the same odd lack of effect in lldb), and I'm wondering if there's something subtle about breakpoints in shared libraries that I haven't encountered before.

I'm sure I'm being dim here, and my google-fu seems weak today, but any pointers would be appreciated.

[...]
(gdb) break runtime.c:panic
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (runtime.c:panic) pending.
(gdb) run
Starting program: /private/tmp/chicken/main/main 
warning: `/BinaryCache/coreTLS/coreTLS-35.40.1~1/Objects/coretls.build/coretls.build/Objects-normal/x86_64/system_coretls_vers.o': can't open to read symbols: No such file or directory.
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_ciphersuites.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_handshake.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_record.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_stream_parser.a"
[panic] nursery is too small - try higher setting using the `-:s' option - execution terminated

[Inferior 1 (process 73628) exited with code 01]
(gdb) quit

[sjamaan]

Oh, I forgot to mention: you need to build your CHICKEN by passing DEBUGBUILD=1 to make. This adds debugging symbols to libchicken.

[Norman Gray]

I'm afraid that didn't work for me either – see gdb output below. I also tried building it using clang and using lldb, but with no more success. I also tried a bootstrap build, for both gcc/gdb and clang/lldb. I'll attach the build log to this comment. I can see #define DEBUGBUILD 1 in chicken-defaults.h, so I don't think I've made any typos.

quaxo:/tmp/chicken> which csc
/Data/tools/chicken-4.11.0rc2/bin/csc
quaxo:/tmp/chicken> csc -emit-all-import-libraries -shared hello.scm
quaxo:/tmp/chicken> csc -deploy main.scm
quaxo:/tmp/chicken> sudo gdb main/main
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin14.5.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from main/main...
warning: `/private/tmp/chicken/main.o': can't open to read symbols: No such file or directory.
(no debugging symbols found)...done.
(gdb) break runtime.c:panic
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (runtime.c:panic) pending.
(gdb) run
Starting program: /private/tmp/chicken/main/main 
warning: `/BinaryCache/coreTLS/coreTLS-35.40.1~1/Objects/coretls.build/coretls.build/Objects-normal/x86_64/system_coretls_vers.o': can't open to read symbols: No such file or directory.
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_ciphersuites.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_handshake.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_record.a"
warning: Could not open OSO archive file "/BinaryCache/coreTLS/coreTLS-35.40.1~1/Symbols/BuiltProducts/libcoretls_stream_parser.a"
Error in re-setting breakpoint 1: Cannot access memory at address 0x735c00
warning: `/private/tmp/chicken/hello.o': can't open to read symbols: No such file or directory.
Error in re-setting breakpoint 1: Cannot access memory at address 0x735c00
[panic] nursery is too small - try higher setting using the `-:s' option - execution terminated

[Inferior 1 (process 86691) exited with code 01]
(gdb) quit

[Norman Gray]

Build log of bootstrap build of Chicken, using gcc

[Jim Ursetto]

I have no idea how deploy is supposed to work but I can say 2 things:

1) can reproduce the problem on 4.8.0.6 with OS X 10.11.4
2) why does hello.so refer to the absolute path of the libchicken.dylib, while main refers to the @executable_path?

$ otool -L hello.so
hello.so:

/Users/jim/local/chicken/4.8.0.6/lib/libchicken.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

$ otool -L main
main:

/System/Library/Frameworks/CoreFoundation?.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
@executable_path/libchicken.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

[Jim Ursetto]

OK, the problem is you're trying to use an extension (hello.so) that hasn't been deployed. When you built hello.so, it wasn't linked for deployment (as evidenced by the fact that it didn't get put into the main/ directory).

I don't know if it's possible to deploy a shared library directly. You may have to make it into an extension and deploy it into the main/ directory with chicken-install. People with experience in using -deploy are welcome to chime in.

[Norman Gray]

For completeness, I should note that this doesn't work if I simply drop hello.so and hello.import.scm into the main/ directory; so that doesn't count as 'deployment' of the library (not entirely surprisingly).

---

But you mentioned making this into an extension...

quaxo:/tmp/chicken> cat hello.scm 
(module hello (hello)
  (import scheme)
  (define (hello)
    (display "Hello, world")
    (newline)))
quaxo:/tmp/chicken> cat hello.meta
((author "Me")
 (synopsis "A cool hello-world library")
 (license "BSD")
 (files "hello.scm" "hello.setup"))
quaxo:/tmp/chicken> cat hello.setup
(standard-extension "hello")
quaxo:/tmp/chicken> cat main.scm
(use hello)
(hello)
quaxo:/tmp/chicken> chicken-install -deploy -prefix $PWD/main
retrieving ...
checking platform for `hello' ...
checking dependencies for `hello' ...
install order:
("hello")
installing hello: ...
changing current directory to .
  '/Data/tools/chicken-4.11.0rc2-clang/bin/csi' -bnq -e "(require-library setup-api)" -e "(import setup-api)" -e "(setup-error-handling)" -e "(extension-name-and-version '(\"hello\" \"\"))" -e "(destination-prefix \"/tmp/chicken/main\")" -e "(runtime-prefix \"/tmp/chicken/main\")" -e "(deployment-mode #t)" 'hello.setup'
  '/Data/tools/chicken-4.11.0rc2-clang/bin/csc' -feature compiling-extension    -deployed -dynamic -optimize-level 3 -debug-level 1 hello.scm -emit-import-library hello
  '/Data/tools/chicken-4.11.0rc2-clang/bin/csc' -feature compiling-extension    -deployed -dynamic -optimize-level 3 -debug-level 0 hello.import.scm
  mkdir -p '/tmp/chicken/main'
  chmod a+x '/tmp/chicken/main'
  cp -r 'hello.so' '/tmp/chicken/main/hello.so'
  chmod a+r '/tmp/chicken/main/hello.so'
  cp -r 'hello.import.so' '/tmp/chicken/main/hello.import.so'
  chmod a+r '/tmp/chicken/main/hello.import.so'
  chmod a+r '/tmp/chicken/main/hello.setup-info'
quaxo:/tmp/chicken> rm hello.import.so
quaxo:/tmp/chicken> csc -deploy main.scm                            
quaxo:/tmp/chicken> main/main
Hello, world
quaxo:/tmp/chicken> 

I'm slightly surprised that I need to remove hello.import.so (csc -deploy main.scm fails otherwise), but this appears to work.

Perhaps, therefore, this should be recategorised as a documentation bugreport. At the end of ​http://wiki.call-cc.org/man/4/Deployment it might^{W would be useful to note that if one wishes to deploy a program which uses modules, then it is necessary to wrap them up as (local) extensions, as described in ​http://wiki.call-cc.org/man/4/Extensions#a-simple-library}

Also, if there were a way to detect this situation, and have a less panicky error message, that would be most civil.

Hooray!

[sjamaan]

Deployment no longer exists as such