Ticket #736: 00repospec.txt

other things to check out:
  virtualenv (python)  (see EOF)
  (single-path) alt repo script: http://paste.call-cc.org/?id=fc11a30f00b18360f1957c9ffca8797a811ed8b3

warnings:
   eval.scm needs data-structures, a new dependency.

notes:
 - .setup-info file list contains redundant full path to each file (for whatever reason).  
   Therefore, you cannot simply copy an extension into a repository; chicken-status will
   list the wrong installed files, and chicken-uninstall will uninstall the wrong files.

 - Possibly we should allow chicken-status to sort eggs by repo, preserving all eggs instead
   of discarding later ones in the path.


 - Host/target extensions may not be handled properly.  Examples:
   - chicken-status prints entire repo-path interspersed by ; when cross-chicken.  Target extensions
     appear to be hardcoded to C_TARGET_LIB_HOME/chicken/6 only; repository-pathspec should only
     affect host extensions.  I believe this is handled correctly.

 - Private repository should perhaps be the "dpath" default repository, inserted when there is
   an empty element, instead of overriding the entire pathspec.

 - Changing LOAD to follow repository path may change behavior when repository is set to (say) foo/,
   but the loaded extension is not in foo; currently, it will try . as a last resort.
   - Specifically, ##sys#find-extension automatically adds "." to end of search path, or
     to beginning of search path if -setup-mode is set.
   - Example: may be implicit in -ignore-repository option in batch-driver.scm, which sets
     repository-path to #f -- although not currently working, it may be intended to search "."
     only and would need to be set to (".") instead.
   - If env var is unset, pathspec may need to default to (list dpath ".") to simulate current behavior.
   - If env var is set, it will probably break the CWD last resort behavior, as I don't want to
     default to searching CWD.
   - Setting pathspec to include "." at end is not exactly equivalent to current behavior.
     For example, neither modules.db nor types.db are currently searched for in CWD, but they
     would be if "." was set in pathspec.  Also, extension-information never searches CWD.
     However, resolve-include-filename does search CWD before trying other include paths (which include
        the repo only when when using -extend, and -inline-global).



- modules.db creation requires that modules be imported, since import files can define arbitrary modules
  not necessarily named correctly.   This is quite rare; see prometheus which defines srfi-8, etc.
  We cannot rely on .import.scm name to check module path / shadowing.
  Problem: we can't track which repo a module was imported from.  Possible solution: modify module record
  to contain the repository path from which it was imported (if any).  Once all import files are loaded
  in path order, write out per-path modules.db.
  - Note: modules.db is only (currently) used for module import suggestion when the compiler cannot find a module.
    It is currently impossible to track back a module to its .import.scm (i.e. library name) except by
    assuming its name is equivalent to its library name.  In fact, modules.db contains spurious entries
    like (receive syntax srfi-8) when prometheus is installed, even though srfi-8 is an internal artifact,
    and (use srfi-8) makes no sense outside of prometheus' context.
  - Also, these "internal" modules can already shadow each other if their names conflict and their eggs
    are installed in the same repository.
  - Therefore, it could be argued that only the module named like the import file should be placed in modules.db.
    It's possible the current behavior of traversing the module table exists just because it's easy.
  - Or we could just assume "import name == module name" and use this to determine which main modules are
    shadowed, while still writing the whole table to individual modules.db.  Unfortunately we would have no
    idea where to put the "internal" modules as they have no filename analog; we might even get confused if
    an internal module name conflicts with a main module name.
  "
     02:57:11 < zbigniew> hmm; it seems that the behavior of modules.db may not even be right currently, with respect to import files that register multiple modules
     02:59:41 < zbigniew> given that modules.db is used to make possible import suggestions like "suggesting: (import colorize)"
     03:01:00 < zbigniew> it generally makes little sense to suggest importing a module that is buried inside a differently-named import file -- as this will almost always fail!
     03:02:43 < zbigniew> as an interesting example, if you were to compile the simple file:
     03:02:46 < zbigniew> (module foo () (import scheme) (receive))
     03:02:55 < zbigniew> and prometheus is installed, you will get:
     03:03:10 < zbigniew> Warning: reference to possibly unbound identifier `receive'
     03:03:10 < zbigniew> Warning:    suggesting one of:
     03:03:10 < zbigniew> Warning:    (import srfi-8)
     03:03:10 < zbigniew> Warning:    (import chicken)
     03:03:48 < zbigniew>
     03:05:43 < zbigniew> which is essentially nonsense; so I wonder if slavishly adhering to the current behavior is even necessary, if it entails intrusive changes.
  "



* virtualenv / pip

setuptools (and pip) have multiple paths but are a bit schizophrenic because the functionality was added over a long time
Native python has only sys.path (afaik) module lookup, without any versioning or dependencies.

- multiple installed versions are supported via runtime manipulation of the path
- installs get their own dedicated and versioned directory and old versions are not necessarily deleted after upgrade
  - active install is specified, I *believe*, by .pth file in every site-packages/ dir
- scripts are (always?) stub trampolines to an entry point in the main code, which resides in the code proper
- setuptools creates the stub scripts for you, given instructions in the setup.py file
- scripts are installed apart from main code, unversioned, in a dedicated bin directory.  Bin dir overridable by user in config file.
- you can have multiple script versions installed, but you have to manually `cp script script-ver` after every version install;
  the stub code itself refers to the egg version, so the correct code library version will still be called
- pkg_resources API is used to indirectly locate files included with the egg.  Primarily this allows resources
  to be transparently contained within a zip file.  In Chicken we could use a similar technique with a special resources egg.
- (UNTESTED) pkg_resources always refers to resources in a versioned code dir, I think, so you should have one full copy of
  all resources with each installed version.
- (UNKNOWN) Can you write to a resource directory at runtime, or how else do you find the path to variable files?
  And if you did write to a resource dir, I assume your changes are wiped out (or inaccessible) when you upgrade the egg, due to versioning.