make CHICKEN_REPOSITORY support a search path

[Jim Ursetto]

It's been suggested that CHICKEN_REPOSITORY could be made to support a colon-separated search path, allowing you to (for example) overlay* the primary repository for testing purposes. This is very much a wishlist item and its feasibility is unknown.

• no pun intended

[Jim Ursetto]

Just wanted to mention that I did implement the search path for modules some time ago, but ran into a major roadblock, which is why this didn't go anywhere. In the current implementation, you can specify a module path like lib/chicken/6:.:lib/chicken-mine/6. The problem is with support files, specifically data and binaries, when an egg is installed in two places in the search path.

Data: stuff in share/chicken/ is placed there by certain eggs and would be overwritten by a twice-installed egg. One possibility is to move the data into the lib/ dir instead, although putting dynamic data there would be odd. We would likely need an indirection so that eggs can find their data, instead of assuming where it is, but this would break backward compatibility.

Binaries: this is harder, because binaries are intended to be invoked by the user, and should go into a well-known location, rather than being hidden inside the egg. I can't think of a non-ridiculous way (symlink farm!) to address this.

In an alternate implementation, it should be possible to overlay an entire repository over another, rather than just the shared modules. This addresses some of the issues above nicely, but what I liked about the lib search path was simplicity: you could use "." to load compiled modules from your current (egg) directory when developing, skipping the install step and also getting rid of the gross -setup-mode hack which I use half the time for its side effect of changing the search path order so I don't accidentally load an earlier installed version of the extension I am developing. Breathe, zbigniew.

Some languages address this by having their extensions be self-contained, i.e. all its support files and shared libs under one directory. I think this is too big a change and not worth pursuing.

So I am still thinking about this but it's on the way back burner.

[Alaric Snell-Pym]

It sounds like your existing work would be sufficient to make Chicken work nicely with the Nix package manager.

Under Nix, each installed package (which chicken itself, as well as eggs, would count as) is installed into its own unique prefix. When the user then requests that a certain bunch of packages be made available in their "profile", a symlink farm is created to "merge" the per-package prefixes together.

So Chicken as it stands would run fine - you'd have a single directory tree with bin/ containing csi and chicken and the results of all the egg install-program actions, and lib/ containing core and egg chicken libs, as you'd expect.

However, chicken-install can't work as-is in this environment, as profiles are strictly read/execute-only; new eggs can't be installed into them by the likes of chicken-install.

The hard part, then, is building the egg packages in the first place. When a package is built, it is passed the path it should install to, and the paths of all of its dependencies. If we could set CHICKEN_INSTALL_PREFIX to the output directory, leave CHICKEN_PREFIX pointing at the chicken package (which all eggs would depend upon) so it can find csc and friends, and set CHICKEN_REPOSITORY to a search path of all the eggs the egg being built depends upon - I think it would then correctly compile the egg into the output directory.

The one exception would be if an egg uses an installed program from another egg as part of its build process, in which case we'd need to ensure that was added to the PATH before running chicken-install.

So: Can you share your patch, so I can see if I can use it to properly integrated Chicken with Nix?

If so, the next step would be to write a script that reads the entire egg list, pulls out the dependencies from the meta files, and tries to make Nix package descriptions out of them (needing me to manually declare any external dependencies, of course...) :-)

[sjamaan]

Let's try at putting this in 4.10.0!

[sjamaan]

Doing this for CHICKEN 4 seems a bit ambitious. Maybe CHICKEN 5, if we have to change this import/require/use etc code anyway.

[Jim Ursetto]

Chicken repository multispec implementation

[Jim Ursetto]

repospec musings

[Jim Ursetto]

Dumping my patch and some ramblings, belatedly. I don't recommend using this, as I think it's the wrong approach for Chicken. However, you might find something slightly interesting in it.

[Jim Ursetto]

Also, this is for Chicken 4.7.0 and was originally written in 2012, and hasn't been reviewed. Please examine at your own risk.

[felix winkelmann]

An implementation of this has been added to the chicken-5-new-egg-install repo, as an addition to the new egg installation system. This is mainly intended to drop the various ways of specifying installation prefixes, while still allowing salmonella to work, somehow.

Note that the various issues raised above are not addressed. The egg path only specifies where eggs + inmport libs are installed, not where executables and data files are going. The system is intentionally very simple, and the user is completely on his own with ensuring a working installation.

[sjamaan]

Good enough (for now?)