Summary

module/import information in statically linked executables has massive runtime overhead

Metadata

Description

While experimenting with statically linked eggs, klm noticed that a simple empty program has a considerable startup time (several seconds). So, for example, the following program:

 (import parley)
 (print 'hello)

compiled like this:

 csc -static hello.scm

then `./hello -:d` will show the initialization sequence, where particular units need quite long to initialize. All of the overhead is very likely to be caused by module registration code and import resolution.

If I instrument `chicken-install` (actually `egg-compile.scm`) to pass `-no-module-registration`, then the overhead is gone. But this will prevent the use of import in evaluated code for all but the core modules (those provided by the `eval-modules` unit).

I'm not sure what to do now. I personally think it's acceptable to require for statically linked executables not to use `eval`, but that does of course not apply to all users. An alternative would be to trigger the module registration lazily, on the first use of `import`, but that may be difficult to change in the compiler or may result in registration ordering issues.

Another option would be a build-time option that decides whether static extensions should be compiled with `-no-module-registration`.

Changes and comments

[2018-04-29 22:13:32 UTC] felix changed version from 4.13.0 to 5.0

[2018-04-30 12:58:25 UTC] sjamaan wrote:

This is probably `merge-se`'s quadratic complexity biting us in the ass again. We really need to fix this.

[2018-05-20 21:09:40 UTC] sjamaan wrote:

Yeah, it's due to `merge-se`.

It is triggered by import libraries being present in static binaries. In binaries which are linked with shared libraries you pay most of the cost at compile time, which you don't notice as easily because it is (usually!) dwarfed by the compilation time itself.

Each import library will contain forms like `(eval '(import scheme chicken.base ...))` to ensure the syntactic environment of any macros exported by the module match the environment inside the module (which is conceptually somewhat broken in itself, but that's something for another day).

When a module exports no syntactic forms, this is 100% wasteful, so we can avoid emitting `eval` for all modules that export only regular values. Patch sent to chicken-hackers.

[2018-05-24 10:13:12 UTC] sjamaan changed status from new to closed

[2018-05-24 10:13:12 UTC] sjamaan set resolution to fixed

[2018-05-24 10:13:12 UTC] sjamaan changed milestone from someday to 5.1

[2018-05-24 10:13:12 UTC] sjamaan wrote:

The worst part is fixed by 36be0fc. We should still look into getting rid of the eval altogether, but I made a new ticket for that: #1466.

[2018-05-24 10:13:27 UTC] sjamaan changed milestone from 5.1 to 5.0