module/import information in statically linked executables has massive runtime overhead

[felix winkelmann]

While experimenting with statically linked eggs, klm noticed that a simple empty program has a considerable startup time (several seconds). So, for example, the following program:

(import parley)
(print 'hello)

compiled like this:

csc -static hello.scm

then ./hello -:d will show the initialization sequence, where particular units need quite long to initialize. All of the overhead is very likely to be caused by module registration code and import resolution.

If I instrument chicken-install (actually egg-compile.scm) to pass -no-module-registration, then the overhead is gone. But this will prevent the use of import in evaluated code for all but the core modules (those provided by the eval-modules unit).

I'm not sure what to do now. I personally think it's acceptable to require for statically linked executables not to use eval, but that does of course not apply to all users. An alternative would be to trigger the module registration lazily, on the first use of import, but that may be difficult to change in the compiler or may result in registration ordering issues.

Another option would be a build-time option that decides whether static extensions should be compiled with -no-module-registration.

[sjamaan]

This is probably merge-se's quadratic complexity biting us in the ass again. We really need to fix this.

[sjamaan]

Yeah, it's due to merge-se.

It is triggered by import libraries being present in static binaries. In binaries which are linked with shared libraries you pay most of the cost at compile time, which you don't notice as easily because it is (usually!) dwarfed by the compilation time itself.

Each import library will contain forms like (eval '(import scheme chicken.base ...)) to ensure the syntactic environment of any macros exported by the module match the environment inside the module (which is conceptually somewhat broken in itself, but that's something for another day).

When a module exports no syntactic forms, this is 100% wasteful, so we can avoid emitting eval for all modules that export only regular values. Patch sent to chicken-hackers.

[sjamaan]

The worst part is fixed by 36be0fc. We should still look into getting rid of the eval altogether, but I made a new ticket for that: #1466.