Chicken can't profile itself

[sjamaan]

I tried to run this:

$ gmake PLATFORM=bsd PREFIX=~/chicken-test CHICKEN='chicken -profile' DEBUGBUILD=1

And this gives me:

Warning: reference to possibly unbound identifier `g555'

Error: module unresolved: setup-api

If I then quickly compile this file without profiling info, I get a binary which barfs that ##sys#error-hook is undefined.

I think this indicates a deeper problem with the profiler which might also occur on some other programs.

It could be a problem in the expander, or in the code that generates profiling info. The undefined g555 occurs somewhere in an expansion of lambda, so my hunch is it's a problem with expand-profile-lambda.

[sjamaan]

Simplified breakage:

(module x ()
  (import chicken scheme)
  ((let ((foo "a"))
     (lambda (x) (display (string-append foo x)))) "b")
  (newline))

[sjamaan]

PS, the module name being x (the same as the lambda's argument) is just an irrelevant coincidence. It doesn't cause the error.

[felix winkelmann]

The reason was a bug in the canonicalization of ##core#begin. Should be better now.

[sjamaan]

Out of curiosity, what changeset fixed this? I don't see any reference to ##core#begin or canonicalization in the git log.

[sjamaan]

After fixing this bug, Chicken still can't profile itself. I get the following error after compiling Chicken from experimental, doing "make spotless" and compiling again with "CHICKEN='/path/to/experimental/chicken -profile'":

[panic] `##sys#error-hook' is not defined - the `library' unit was probably not linked with this executable - execution terminated

[sjamaan]

Clarification: I get the message when compiling another program. This happens after I've successfully installed chicken (with profiling) for the second time.

In other words, we can now _build_ a profiled Chicken, but it doesn't work.

[sjamaan]

It turns out this is caused by the fact that C_profiler_toplevel is invoked by the generated (##core#callunit "profiler") which is added to library.scm by the -profile flag.

The profiler toplevel depends on a few identifiers from library.scm being bound (those in the LETs around the lambdas of ##sys#register-profile-info and ##sys#finish-profile). Getting rid of those surrounding LETs doesn't fix the problem, though.

Compiling the whole thing with CHICKEN='chicken -profile', then deleting library.c and profiler.c and running Make again with CHICKEN=chicken results in a properly functioning Chicken. However, this is a manual step and it leaves a huge part of Chicken uninstrumented with profiling info.

My guess is that locking down profiler.scm into using direct calls everywhere instead of through symbol lookups should fix this bootstrapping problem. Is this possible? I tried using usual-integrations, but I think some are "unusual" integrations :)

Another approach that might work is somehow forcing the profiler toplevel to run after the toplevel stuff in library.scm

Finally, the profiler itself should never be compiled with profiling info, I think. If I compile everything except library.scm with profiling info, I get the error

Error: unbound variable: ##sys#register-profile-info

This happens because profiler.scm is instrumented with itself, so before the toplevel had a chance to run the profiling procedures are already invoked. This *might* also be solvable by not looking up profiling procedures by symbol (but that would be in the _emitted_ profiling code, I think)

I hope these ramblings make sense...

[felix winkelmann]

(Sorry, I didn't notice this)

What is it exactly, you want to profile. Profiling the runtime system is a little bit of an icky thing in general. Is there a particular unit that you'd like get profiled?

[sjamaan]

It all started when Moritz was complaining that compiling (I think it was) his new egg for pulseaudio took too long.

So I thought it would be a good idea to try building chicken for profiling itself because I had no idea where it was spending most of its time. I assume it's in the macro expansion (he uses two rather large macros), but profiling just the expander itself is not enough; it will use procedures from library and other core units, which should also be compiled with profiling info so you can see what exactly is the reason for slowness.

More in general, I'd like to have an idea of where the compiler's performance can be improved without a specific idea of what exactly I want to profile.

Up till now I've been able to produce a working profiling chicken by manually interfering with the compilation process and disabling profiling for two units; the profiler itself and another which I can't remember right now. Probably library.scm?

[sjamaan]

I didn't get very far with that, by the way, because the matchable egg compiled with this Chicken doesn't function properly, which made it impossible to compile foreigners. (and compiling matchable without profiling didn't make too much sense either as my gut feeling says that's where the bulk of the time is being spent; either in executing matchable's rules itself or in the expander doing matchable's work)

[felix winkelmann]

I'm not sure how to proceed here. It is possible to selectively use profiling for some units, but it must be clear profiling the core system is something for which some manual intervention must be acceptable.

[sjamaan]

The bugs I saw with matchable should be investigated further.

Aside from that, it's fine if it needs to be done manually as long as it's known how. I hope to write down some steps when I get a round tuit.

[felix winkelmann]

I close this because this will never be completely possible: you can't use the core system to execute profiling code for the core system. It's possible to enable profiling for single units, but this will need manual work.