Opened 10 years ago

Closed 9 years ago

#458 closed defect (wontfix)

Chicken can't profile itself

Reported by: sjamaan Owned by:
Priority: minor Milestone:
Component: compiler Version: 4.6.x
Keywords: expander, profiler, self-compilation Cc:
Estimated difficulty:


I tried to run this:

$ gmake PLATFORM=bsd PREFIX=~/chicken-test CHICKEN='chicken -profile' DEBUGBUILD=1

And this gives me:

Warning: reference to possibly unbound identifier `g555'

Error: module unresolved: setup-api

If I then quickly compile this file without profiling info, I get a binary which barfs that ##sys#error-hook is undefined.

I think this indicates a deeper problem with the profiler which might also occur on some other programs.

It could be a problem in the expander, or in the code that generates profiling info. The undefined g555 occurs somewhere in an expansion of lambda, so my hunch is it's a problem with expand-profile-lambda.

Change History (15)

comment:1 Changed 10 years ago by sjamaan

Simplified breakage:

(module x ()
  (import chicken scheme)
  ((let ((foo "a"))
     (lambda (x) (display (string-append foo x)))) "b")

comment:2 Changed 10 years ago by sjamaan

PS, the module name being x (the same as the lambda's argument) is just an irrelevant coincidence. It doesn't cause the error.

comment:3 Changed 10 years ago by felix winkelmann

The reason was a bug in the canonicalization of ##core#begin. Should be better now.

comment:4 Changed 10 years ago by felix winkelmann

Resolution: fixed
Status: newclosed

comment:5 Changed 10 years ago by sjamaan

Out of curiosity, what changeset fixed this? I don't see any reference to ##core#begin or canonicalization in the git log.

comment:6 Changed 10 years ago by sjamaan

Resolution: fixed
Status: closedreopened

After fixing this bug, Chicken still can't profile itself. I get the following error after compiling Chicken from experimental, doing "make spotless" and compiling again with "CHICKEN='/path/to/experimental/chicken -profile'":

[panic] `##sys#error-hook' is not defined - the `library' unit was probably not linked with this executable - execution terminated

comment:7 Changed 10 years ago by sjamaan

Clarification: I get the message when compiling another program. This happens after I've successfully installed chicken (with profiling) for the second time.

In other words, we can now _build_ a profiled Chicken, but it doesn't work.

comment:8 Changed 10 years ago by sjamaan

It turns out this is caused by the fact that C_profiler_toplevel is invoked by the generated (##core#callunit "profiler") which is added to library.scm by the -profile flag.

The profiler toplevel depends on a few identifiers from library.scm being bound (those in the LETs around the lambdas of ##sys#register-profile-info and ##sys#finish-profile). Getting rid of those surrounding LETs doesn't fix the problem, though.

Compiling the whole thing with CHICKEN='chicken -profile', then deleting library.c and profiler.c and running Make again with CHICKEN=chicken results in a properly functioning Chicken. However, this is a manual step and it leaves a huge part of Chicken uninstrumented with profiling info.

My guess is that locking down profiler.scm into using direct calls everywhere instead of through symbol lookups should fix this bootstrapping problem. Is this possible? I tried using usual-integrations, but I think some are "unusual" integrations :)

Another approach that might work is somehow forcing the profiler toplevel to run after the toplevel stuff in library.scm

Finally, the profiler itself should never be compiled with profiling info, I think. If I compile everything except library.scm with profiling info, I get the error

Error: unbound variable: ##sys#register-profile-info

This happens because profiler.scm is instrumented with itself, so before the toplevel had a chance to run the profiling procedures are already invoked. This *might* also be solvable by not looking up profiling procedures by symbol (but that would be in the _emitted_ profiling code, I think)

I hope these ramblings make sense...

comment:9 Changed 10 years ago by felix winkelmann

(Sorry, I didn't notice this)

What is it exactly, you want to profile. Profiling the runtime system is a little bit of an icky thing in general. Is there a particular unit that you'd like get profiled?

comment:10 Changed 10 years ago by sjamaan

It all started when Moritz was complaining that compiling (I think it was) his new egg for pulseaudio took too long.

So I thought it would be a good idea to try building chicken for profiling itself because I had no idea where it was spending most of its time. I assume it's in the macro expansion (he uses two rather large macros), but profiling just the expander itself is not enough; it will use procedures from library and other core units, which should also be compiled with profiling info so you can see what exactly is the reason for slowness.

More in general, I'd like to have an idea of where the compiler's performance can be improved without a specific idea of what exactly I want to profile.

Up till now I've been able to produce a working profiling chicken by manually interfering with the compilation process and disabling profiling for two units; the profiler itself and another which I can't remember right now. Probably library.scm?

comment:11 Changed 10 years ago by sjamaan

I didn't get very far with that, by the way, because the matchable egg compiled with this Chicken doesn't function properly, which made it impossible to compile foreigners. (and compiling matchable without profiling didn't make too much sense either as my gut feeling says that's where the bulk of the time is being spent; either in executing matchable's rules itself or in the expander doing matchable's work)

comment:12 Changed 9 years ago by felix winkelmann

Milestone: 4.7.0

comment:13 Changed 9 years ago by felix winkelmann

Priority: majorminor

I'm not sure how to proceed here. It is possible to selectively use profiling for some units, but it must be clear profiling the core system is something for which some manual intervention must be acceptable.

comment:14 Changed 9 years ago by sjamaan

The bugs I saw with matchable should be investigated further.

Aside from that, it's fine if it needs to be done manually as long as it's known how. I hope to write down some steps when I get a round tuit.

comment:15 Changed 9 years ago by felix winkelmann

Resolution: wontfix
Status: reopenedclosed

I close this because this will never be completely possible: you can't use the core system to execute profiling code for the core system. It's possible to enable profiling for single units, but this will need manual work.

Note: See TracTickets for help on using tickets.