Unable to deploy sxpath library with Chicken 4.6.7 under Mac OS X

[Ivan Raikov]

Using Chicken 4.6.7, I am able to deploy the sxpath library under Linux, but not under Mac OS X (Darwin 10.7). Can anybody test this under different OSes?

~/bin/chicken/bin/chicken-install -deploy -p sxpath sxpath
retrieving ...
resolving alias `kitten-technologies' to: ​http://chicken.kitten-technologies.co.uk/henrietta.cgi
connecting to host "chicken.kitten-technologies.co.uk", port 80 ...
requesting "/henrietta.cgi?name=sxpath&mode=default" ...

[snipped]

installing sxpath: ...
changing current directory to /tmp/temp9fdc/sxpath

/Users/ivan/bin/chicken/bin/csi -bnq -setup-mode -e "(require-library setup-api)" -e "(import setup-api)" -e "(setup-error-handling)" -e "(extension-name-and-version '(\"sxpath\" \"\"))" -e "(destination-prefix \"/Users/ivan/build/neurolucida/sxpath\")" -e "(runtime-prefix \"/Users/ivan/build/neurolucida/sxpath\")" -e "(deployment-mode #t)" /tmp/temp9fdc/sxpath/sxpath.setup
/Users/ivan/bin/chicken/bin/csc -feature compiling-extension -setup-mode -deployed -s -O2 -o sxpath-lolevel.so chicken/sxpath-lolevel.scm -j sxpath-lolevel
/Users/ivan/bin/chicken/bin/csc -feature compiling-extension -setup-mode -deployed -s -O2 sxpath-lolevel.import.scm
/Users/ivan/bin/chicken/bin/csc -feature compiling-extension -setup-mode -deployed -s -O2 -o txpath.so chicken/txpath.scm -j txpath

Error: during expansion of (import ...) - unbound variable: |\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x08\x00\x00\x00|

Call history:

<syntax> (##core#begin (module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:co......
<syntax> (module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:core-local-name ......
<syntax> (##core#module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:core-loca......
<syntax> (import chicken scheme (only data-structures string-split) (only srfi-1 filter) (only srfi-13 string......
<syntax> (##core#undefined)
<syntax> (require-extension sxpath-lolevel)
<syntax> (##core#require-extension (sxpath-lolevel) #t)
<syntax> (##core#begin (##core#begin (##sys#require (quote sxpath-lolevel)) (import sxpath-lolevel)) (##core#......
<syntax> (##core#begin (##sys#require (quote sxpath-lolevel)) (import sxpath-lolevel))
<syntax> (##sys#require (quote sxpath-lolevel))
<syntax> (quote sxpath-lolevel)
<syntax> (##core#quote sxpath-lolevel)
<syntax> (import sxpath-lolevel) <--

Error: shell command terminated with non-zero exit status 17920: /Users/ivan/bin/chicken/bin/chicken chicken/txpath.scm -output-file txpath.c -dynamic -feature chicken-compile-shared -feature compiling-extension -setup-mode -optimize-level 2 -emit-import-library txpath

Error: shell command failed with nonzero exit status 256:

[felix winkelmann]

Do you have an old version of sxpath installed (in the normal repository)? Can you try this, please, and show me the output:

CSC_OPTIONS=-:d chicken-install ...

[Ivan Raikov]

This is a fresh install of Chicken, so no other copies of sxpath exist on the system. Below are the lines relevant to sxpath compilation when compiling with :-d (full log is attached).

installing sxpath: ...
changing current directory to /tmp/tempa375/sxpath
  /Users/igr/bin/chicken/bin/csi -bnq -setup-mode -e "(require-library setup-api)" -e "(import setup-api)" -e "(setup-error-handling)" -e "(extension-name-and-version '(\"sxpath\" \"\"))" -e "(destination-prefix \"/Users/igr/test\")" -e "(runtime-prefix \"/Users/igr/test\")" -e "(deployment-mode #t)" /tmp/tempa375/sxpath/sxpath.setup
  /Users/igr/bin/chicken/bin/csc -feature compiling-extension -setup-mode   -deployed -s -O2 -o sxpath-lolevel.so chicken/sxpath-lolevel.scm -j sxpath-lolevel
  /Users/igr/bin/chicken/bin/csc -feature compiling-extension -setup-mode   -deployed -s -O2 sxpath-lolevel.import.scm
  /Users/igr/bin/chicken/bin/csc -feature compiling-extension -setup-mode   -deployed -s -O2 -o txpath.so chicken/txpath.scm -j txpath

Error: during expansion of (import ...) - unbound variable: |\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x08\x00\x00\x00|

        Call history:

        <syntax>          (##core#begin (module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:co......
        <syntax>          (module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:core-local-name ......
        <syntax>          (##core#module txpath (sxml:core-last sxml:core-position sxml:core-count sxml:core-id sxml:core-loca......
        <syntax>          (import chicken scheme (only data-structures string-split) (only srfi-1 filter) (only srfi-13 string......
        <syntax>          (##core#undefined)
        <syntax>          (require-extension sxpath-lolevel)
        <syntax>          (##core#require-extension (sxpath-lolevel) #t)
        <syntax>          (##core#begin (##core#begin (##sys#require (quote sxpath-lolevel)) (import sxpath-lolevel)) (##core#......
        <syntax>          (##core#begin (##sys#require (quote sxpath-lolevel)) (import sxpath-lolevel))
        <syntax>          (##sys#require (quote sxpath-lolevel))
        <syntax>          (quote sxpath-lolevel)
        <syntax>          (##core#quote sxpath-lolevel)
        <syntax>          (import sxpath-lolevel)       <--

Error: shell command terminated with non-zero exit status 17920: /Users/igr/bin/chicken/bin/chicken chicken/txpath.scm -output-file txpath.c -dynamic -feature chicken-compile-shared -feature compiling-extension -setup-mode -optimize-level 2 -emit-import-library txpath

Replying to felix:

Do you have an old version of sxpath installed (in the normal repository)? Can you try this, please, and show me the output:

CSC_OPTIONS=-:d chicken-install ...

[felix winkelmann]

The -:d was apparently not passed through. I need the debug-mode output of "chicken" when it attempts to load the import library. Can you somehow hack a "-:d" into the failing compilation command (for example, with "CSC_OPTIONS=\"-compiler 'chicken -:d'\"", or run the compilation steps manually and add {{-compiler "chicken -:d"}} to the last invocation of csc.

Are the hex-values the initial bytes of sxpath.import.so (or some other file)? If yes, then probably some attempt to dynamically load a compiled file fails. You can also use DYLD_DEBUG (or something similar) to list some information regarding dynamic loading. Or something strace-y.

[Ivan Raikov]

build debug log

[Ivan Raikov]

Ok, unfortunately CSC_OPTIONS and CHICKEN_OPTIONS don't seem to be properly passed by chicken-install, so I had to run the compilation steps manually. There is not something as convenient as strace/ltrace in Mac OS X, but I managed to get dyld to output a trace of its activities, and this is all included in the new build log I have attached.

The hex values are the initial bytes of sxpath-lolevel.import.so, and the output of dyld confirms that this is what fails to load. Let me know if there is some additional debug information I can obtain.

Replying to felix:

The -:d was apparently not passed through. I need the debug-mode output of "chicken" when it attempts to load the import library. Can you somehow hack a "-:d" into the failing compilation command (for example, with "CSC_OPTIONS=\"-compiler 'chicken -:d'\"", or run the compilation steps manually and add {{-compiler "chicken -:d"}} to the last invocation of csc.

Are the hex-values the initial bytes of sxpath.import.so (or some other file)? If yes, then probably some attempt to dynamically load a compiled file fails. You can also use DYLD_DEBUG (or something similar) to list some information regarding dynamic loading. Or something strace-y.

[Jim Ursetto]

Replying to iraikov:

There is not something as convenient as strace/ltrace in Mac OS X

dtruss(1m)

[felix winkelmann]

Ivan, can you try to execute the build-commands manually? If the problem can be reproduced, install sxpath in your normal repo and try the manual build again without -setup-mode (for csc).

[Ivan Raikov]

Ok, running the build commands manually results in the same problem. Installing sxpath normally and then running the manual build without -setup-mode succeeds.

Replying to felix:

Ivan, can you try to execute the build-commands manually? If the problem can be reproduced, install sxpath in your normal repo and try the manual build again without -setup-mode (for csc).

[felix winkelmann]

Replying to iraikov:

Ok, running the build commands manually results in the same problem. Installing sxpath normally and then running the manual build without -setup-mode succeeds.

Ok. Apparently the build-product sxpath-lolevel can't be loaded because it links to the libchicken.so inside the deployment directory. We can disable -setup-mode on OS X, but that will require deployed egg to be already installed in the normal repo, when these eggs require libraries that are produced during the build.

[Ivan Raikov]

Would it be possible to do this automatically? That is, if chicken-install is deploying an egg on Mac OS X, it first installs the egg normally, and then it does deployment with -setup-mode disabled. I must admit that I don't fully understand why deployment succeeds with some eggs, but not others.

Replying to felix:

Ok. Apparently the build-product sxpath-lolevel can't be loaded because it links to the libchicken.so inside the deployment directory. We can disable -setup-mode on OS X, but that will require deployed egg to be already installed in the normal repo, when these eggs require libraries that are produced during the build.

[felix winkelmann]

Replying to iraikov:

Would it be possible to do this automatically? That is, if chicken-install is deploying an egg on Mac OS X, it first installs the egg normally, and then it does deployment with -setup-mode disabled. I must admit that I don't fully understand why deployment succeeds with some eggs, but not others.

That would be annoying, IMHO. This only happens for those eggs that have code that {{use}}s shared objects built during their compilation. txpath needs sxpath-lolevel, which is built during the "chicken-install" run. Normally, and egg's code doesn't depend on anything built during the installation.

I think we should disable -setup-mode for deployment. We already do this for cross-compilation in the (default) "target" mode. -setup-mode changes (respository-path) to also search for extensions in the current directory.

[Ivan Raikov]

Ok, should -setup-mode be disabled when deploying on any platform, not just Mac OS X? This would lead to less inconsistent behavior on all platforms. Of course, the deployment chapter in the manual would have to updated accordingly.

Replying to felix:

I think we should disable -setup-mode for deployment. We already do this for cross-compilation in the (default) "target" mode. -setup-mode changes (respository-path) to also search for extensions in the current directory.

[felix winkelmann]

Replying to iraikov:

Ok, should -setup-mode be disabled when deploying on any platform, not just Mac OS X? This would lead to less inconsistent behavior on all platforms. Of course, the deployment chapter in the manual would have to updated accordingly.

Yes, I will implement this. Additionally, we can show a warning when we deploy an extensions that is not already installed in the default repository.

[felix winkelmann]

I have disabled -setup-mode when -deploy is used. The manual actually points this already out: an extension should be installed into the normal repo first, then deployed.

Added to "master" and merged into "prerelease".

I added a note to NEWS. Note that the NEWS-entries since 4.6.0 have not been "consolidated" yet. I can do that, if you want. I increased the version to "4.7.0rc2".

[felix winkelmann]

Milestone 4.7.0 deleted

[felix winkelmann]

Milestone 4.8.0 deleted