implement new build system

[felix winkelmann]

Finding a solution for #166 will require a number of changes to the build, which are likely to exceed the capabilities of the current make(3) based approach.

As iraikov already suggested, a Scheme based (bootstrapped from .c sources) or other high-level build is needed.

[Ivan Raikov]

A while ago, I ported the Chicken 3 build system to Scheme. The results are ​here. Of course, the non hygienic macros would have to be rewritten and the build rules updated for Chicken 4, but at least this is a starting point.

[felix winkelmann]

Replying to iraikov:

A while ago, I ported the Chicken 3 build system to Scheme. The results are ​here. Of course, the non hygienic macros would have to be rewritten and the build rules updated for Chicken 4, but at least this is a starting point.

Yes, I remember. Replacing the build is quite an undertaking I'm afraid, and shouldn't be underestimated. I actually tried to write a tool from scratch (based on C), but failed miserably. Using chicken-make is an option, but I'm afraid of bootstrapping issues. What if it doesn't compile? I also find the approach (basically just a make(3) replacement with full programming capabilities) somewhat boring. I'll have to think some more about it.

[Mario Domenech Goulart]

I thought the plan was to create a tool to generate makefiles from a DSL (it could even be an egg). The generated makefiles would then be shipped with the core sources.

Ruby has rake. Maybe Chicken could have cake. :-)

[Ivan Raikov]

I agree that eventually it would be nice to have a DSL for generating makefiles, but before we can design such a DSL, we need some design cases. Such design cases can only be obtained by a relatively straightforward ('boring') port of the current Makefiles to Scheme, and then iteratively improving and generalizing the resulting build system. I think this would be better than trying to come up with some grand design that doesn't work.

Replying to felix:

Replying to iraikov:

A while ago, I ported the Chicken 3 build system to Scheme. The results are ​here. Of course, the non hygienic macros would have to be rewritten and the build rules updated for Chicken 4, but at least this is a starting point.

Yes, I remember. Replacing the build is quite an undertaking I'm afraid, and shouldn't be underestimated. I actually tried to write a tool from scratch (based on C), but failed miserably. Using chicken-make is an option, but I'm afraid of bootstrapping issues. What if it doesn't compile? I also find the approach (basically just a make(3) replacement with full programming capabilities) somewhat boring. I'll have to think some more about it.

[felix winkelmann]

Replying to mario:

I thought the plan was to create a tool to generate makefiles from a DSL (it could even be an egg). The generated makefiles would then be shipped with the core sources.

Ruby has rake. Maybe Chicken could have cake. :-)

Yes, that's a possible approach and not that difficult to implement. I think this is a good idea.

[felix winkelmann]

Replying to iraikov:

I agree that eventually it would be nice to have a DSL for generating makefiles, but before we can design such a DSL, we need some design cases. Such design cases can only be obtained by a relatively straightforward ('boring') port of the current Makefiles to Scheme, and then iteratively improving and generalizing the resulting build system. I think this would be better than trying to come up with some grand design that doesn't work.

I see your point. The problem is that getting a build to work is a time-consuming and mind-numbing piece of work that needs endless hours of testing and tweaking (I've done it too often, already). I'd like to avoid doing it twice. I do not insist on some grand design, but at least something less monolithic than a big make form.

[(none)]

Milestone 4.4.0 deleted

[sjamaan]

I've just pushed a new branch called "make-refactoring" which includes a heavily refactored set of Makefiles. I've done this the slow and incremental way, piling together lines that were similar and related, and abstracting those into meta-rules. I've also put some things into variables that were previously repeated all over the file. The only place of needless repetition is in the bootstrap .c files. It might be possible to simplify that too (listing just the dependencies that differ from the default, for example).

I used the dreaded "eval", which is the only way to do generate rules programmatically in GNU Make as far as I can see.

Oh well, at least this still uses GNU Make, which is a pretty simple dependency and available almost everywhere. That also saves us from breaking our heads over the nasty bootstrapping problems we'd get with a Scheme-based build system. Since the rules are all moved around and extracted from the original Makefiles, I think the margin of error is also a lot smaller than a grand approach to create something proper from scratch.

One final note: I have not touched defaults.make. There may be some stuff that could be simplified there, but I didn't dare touch that yet. Maybe later (there's one important simplication and that's the way all the executable programs are built; the rules to build those are still quite redundant now)

[Christian Kellermann]

Replying to sjamaan:

Tested on Mac OS X 10.5.8 / i386, installs fine, uninstalls without
leftovers as far as I can see, chicken-install coops works as
expected. Loading modules works. I cannot tell whether make check
passes all tests, at least I haven't seen any failed tests in the
backlog.

[Christian Kellermann]

Same goes for OpenBSD 4.7/i386

[Jim Ursetto]

tested OS X universal build

---- extract make-refactoring on x86 10.5.8
[ok] universal core build on x86 10.5.8
[ok] make check
[ok] chicken-install chicken-doc-admin
[ok] chicken-doc-admin -e eggref/4
[ok] chicken-doc open/rdonly
---- copy x86 installation to ppc
[ok] chicken-doc open/rdonly

not tested (yet?) because I do not have a compiler on the PPC box:

---- wipe ppc install
---- copy x86 core to ppc && make install
[  ] make check
[  ] chicken-install chicken-doc-admin
[  ] chicken-doc-admin -e eggref/4
[  ] chicken-doc open/rdonly
---- make spotless on ppc
[  ] universal core build on PPC 10.5.8
[  ] make check
---- wipe x86 install
---- copy ppc install to x86
[  ] chicken-doc open/rdonly

[sjamaan]

I've successfully tested it under the following platforms:

• NetBSD/amd64
• NetBSD/macppc
• Linux/i386 (Debian 5.0.5)
• Linux/x86_64 (CentOS 5.2)
• native Linux/i386 (CentOS 3.8), cross-compiled from Linux/x86_64 (CentOS 5.2)
• cross-chicken on Linux/x86_64

[Mario Domenech Goulart]

I could build and install chicken from the make-refactoring
branch on the following platforms:

• Linux/PPC (Ubuntu 9.04)
• Linux/x86 (Ubuntu 9.10)

I've built chicken, run make check and installed/run awful
as a test.

On Linux/x86 everything looks fine. On Linux/PPC, however, I get an error when running make check (see #372).

[Christian Kellermann]

Works on OpenBSD/loongson (64 bit MIPSel)

[sjamaan]

I just tested it on Haiku (latest "R1/Alpha2" release) and it works there as well. I could add it was on x86, but Haiku runs on nothing else :)

[johnwcowan]

make-refactoring works fine on Cygwin, once I got the recipe for building on a branch:

$ git checkout make-refactoring
$ git pull
$ make PLATFORM=cygwin
$ make PLATFORM=cygwin install

[Mario Domenech Goulart]

Just tested the make-refactoring branch on WinXP/Cygwin (latest commit is 03113e3ce7be377e07c55779088bcf8ea08a3c02). Seems to work fine. Here are the build parameters I used:

make PLATFORM=cygwin PREFIX=/home/mario/local/make-refactoring CHICKEN=/home/mario/local/master/bin/chicken

The given CHICKEN is 4.6.0rc1.

After built, chicken features manyargs, dload and ptables.

I could run csi and install eggs with chicken-install.

[sjamaan]

I tried testing OpenSolaris?, but couldn't get it working either because it demands shitloads of memory to run or simply doesn't work on qemu. The box I tried it on has "only" 512MB.

My laptop has more memory but is too damn slow to run qemu without going insane.

[sjamaan]

OK, I found a spare x86 box at work, on which I managed to test OpenSolaris?. I used the SunStudio? compiler, which exposed another pre-existing bug in the build. I fixed it in my branch (commits df072494ee5dd85bbb811bc7857abfb81f38730d and 90098530013e9dadd5e5bd3e5807e4a2e264e0c3)

The C compiler crashed on me once, and hung a few times without doing anything (once many hours overnight while I was away. When I got at work in the morning it was still hanging on the same file). Canceling the make process and restarting it fixed that, so I'll chalk that up to the flakiness of Sun's crappy compiler, not our build.

A "make check" and installing and running eggs worked fine. I will try gcc on this platform momentarily. Hopefully by the end of the day (or else tomorrow) I'll have testresults.

[sjamaan]

gcc worked fine on Solaris. It was much much faster and didn't crash or hang.

How about changing the default C compiler on Solaris to gcc? It works better.

[felix winkelmann]

Works fine on mingw/MSYS but fails on raw mingw. I have to investigate this in more detail. I would like to add that I think that rules.make has not gained any more readability and is IMHO not an improvement, besides the reduced size.

[sjamaan]

Replying to felix:

Works fine on mingw/MSYS but fails on raw mingw. I have to investigate this in more detail. I would like to add that I think that rules.make has not gained any more readability and is IMHO not an improvement, besides the reduced size.

I can understand your complaint about readability, but why is it not an improvement? Since the files are now compiled through a recipe generated by a "procedure" there's a lot less chance for individual mistakes in rules. Either everything breaks, hard, or nothing breaks.

I assume the readability is in the "procedure" bodies, or is it something else? What specifically is bothering you the most? Perhaps I can work on improving that a little.

[felix winkelmann]

Replying to sjamaan:

Replying to felix:

Works fine on mingw/MSYS but fails on raw mingw. I have to investigate this in more detail. I would like to add that I think that rules.make has not gained any more readability and is IMHO not an improvement, besides the reduced size.

I can understand your complaint about readability, but why is it not an improvement? Since the files are now compiled through a recipe generated by a "procedure" there's a lot less chance for individual mistakes in rules. Either everything breaks, hard, or nothing breaks.

I assume the readability is in the "procedure" bodies, or is it something else? What specifically is bothering you the most? Perhaps I can work on improving that a little.

Don't take my remark too seriously. There is just too much make-magic involved, for my taste. Somehow I liked the plain stupidity of the old rules.make. Generating it via Scheme would be an approach I have personally preferred, nevertheless you have done great work, so just ignore me.

[felix winkelmann]

Replying to sjamaan:

gcc worked fine on Solaris. It was much much faster and didn't crash or hang.

How about changing the default C compiler on Solaris to gcc? It works better.

Yes, I'll change this accordingly.

[felix winkelmann]

The make-refactoring branch has recently been merged into experimental. This cleans up rules.make significantly and makes the whole thing much easier to maintain. Since I currently have no idea how to provide a more generic or portable build system that has minimal dependencies and can be cleanly bootstrapped, I close this ticket unless someone has a better idea.