source: project/wiki/eggref/4/salmonella @ 37793

[[tags: egg salmonella]]

== salmonella

[[toc:]]

=== Introduction

Salmonella is a set of tools to test CHICKEN eggs.  Here's a brief
summary of features:

* very light on dependencies (none at the moment)
* parallelizable ({{salmonella-epidemy}})
* can be run with or without Internet access

Note: starting on version 2.0, salmonella is a rewrite of the
salmonella 1.x series, which was available since CHICKEN 2.x.  The
documentation for the 1.x series is still available at
[[/egg/salmonella-1.x|http://wiki.call-cc.org/egg/salmonella-1.x]].


=== Author

[[/users/mario-domenech-goulart|Mario Domenech Goulart]]


=== Repository

salmonella is hosted on [[http://github.com|github]]:
[[https://github.com/mario-goulart/salmonella|https://github.com/mario-goulart/salmonella]]


=== Tools

This egg provides some command line tools.  The next sections describe
each of them.



==== salmonella

This is the core tool for testing eggs.  {{salmonella}} can perform
the following tests:

* installation of eggs
* dependencies verification
* {{.meta}} file consistency check
* egg tests execution
* general egg consistency (i.e., can be used as an ''egg lint'' to
avoid common mistakes when publishing a new egg or a new egg
version)

{{salmonella}} generates a sexpr-based log file as output.  That file
can be used by tools like
[[/egg/salmonella-html-report|salmonella-html-report]] to generate a
pretty output in HTML.

When executing, salmonella prints some basic information on the
standard output.  Detailed information can be viewed with the
aforementioned [[/egg/salmonella-html-report|salmonella-html-report]]
or with {{salmonella-log-viewer}}, a tool which is provided by this
egg. {{salmonella-log-viewer}} parses the log file generated by
salmonella and formats the data on the standard output.

===== Command line options

Here's the output of {{salmonella -h}} with an explanation for each
command line option:

  salmonella [ -h | --help ]
  salmonella --version
  salmonella [ [ <options> ] eggs ]
  
  When called without eggs in the command line, salmonella will try to
  find a .egg file in the current directory and process it (just like
  chicken-install).
  
  <options>:
  --log-file=<logfile>
      The name for the log file to be generated by salmonella
      (default=salmonella.log).
  
  --chicken-installation-prefix=<prefix dir>
      If you want to test eggs using a chicken installed on a certain directory,
      you can use this option (it should point to the same directory as given to
      `PREFIX' when installing CHICKEN). If omitted, salmonella uses CHICKEN
      tools from the current runtime's installation prefix.
  
  --chicken-install-args=<install args>
      This option can be used customize chicken-install's arguments.  You can
      use <repo> to indicate where you want the actual repository directory
      to be replaced by salmonella.
  
  --eggs-source-dir=<eggs dir>
      By default, salmonella fetches eggs from the egg server.  If you have a
      local copy of eggs code, you can use this option to point to the directory
      where they are located.
  
  --eggs-doc-dir=<doc dir>
      By default, salmonella checks if documentation for eggs exist by accessing
      the CHICKEN wiki.  If you have a local copy of the wiki documentation for
      eggs, you can use this option to point to the directory where they can be
      found.
  
  --keep-repo
      For each egg that salmonella tests, it sets the egg installation repository
      empty and removes it at the end of its execution.  This option makes
      salmonella keep the egg installation repository after testing each egg and
      after finishing its execution.  This option can save a lot of time when
      testing several eggs, at the cost of potentially making salmonella unable
      to catch dependencies problems.
  
  --skip-eggs=<comma-separated list of eggs to skip>
      A comma-separated list of eggs to be skipped.
  
  --repo-dir=<path to repo dir to be used>
      Alternative location for the egg installation directory used by salmonella.
      By default, salmonella generates a `salmonella-tmp-xxxxx' directory in the
      current directory.  This option can be useful when used with `--keep-repo'
      to reuse egg installation repositories for several salmonella executions.
  
  --clear-chicken-home
      Remove Scheme files from <chicken-installation-prefix>/share/chicken.
      WARNING: use this option with care.  If you don't understand the
      consequences of this option, DON'T USE IT.  Extra care when you don't
      use --chicken-installation-prefix -- in this case --clear-chicken-home
      will remove Scheme files from your "host" CHICKEN installation.
      This option is only effective when --keep-repo is NOT used.
  
  --verbosity=<number>
      A number to indicate salmonella's verbosity level.  0 means practically
      silent. 1 is mostly silent and 2 (default) prints some useful information
      while salmonella is running.
  

===== Some quick tips by example

Simplest case: testing eggs using the remote egg server and the
chicken tools ({{chicken-install}} and {{csi}}) in the system {{PATH}}

  $ salmonella big-chicken slice


You can test all the available eggs using one of the following
approaches:

1. If you have CHICKEN >= 4.7.2:

  $ salmonella `chicken-install -list`


2. If you don't have CHICKEN >= 4.7.2:

  $ salmonella `wget http://code.call-cc.org/cgi-bin/henrietta.cgi?list=1 -q -O -`


If you don't want to test some specific eggs, you can skip them:

  $ salmonella --skip-eggs=macosx,hfs+ `chicken-install -list`


You can tell salmonella to use a specific CHICKEN version:

  $ salmonella --chicken-installation-prefix=/usr/local/chicken-4.7.3 big-chicken slice


Not cleaning the egg installation repository after installing each egg
may significantly speed up the salmonella execution time (the default
behavior is to set the egg installation repository empty after testing
each egg):

  $ salmonella --keep-repo big-chicken slice

warning: if you use --keep-repo, salmonella will not be able to catch
dependencies problems.


If you want to reuse the same egg installation repository for multiple
salmonella runs, you can provide a specific directory:

  $ salmonella --repo-dir=my-repo --keep-repo big-chicken slice


By default, salmonella generates a log file named {{salmonella.log}} in
the current directory.  You can change that by using the {{--log-file}}
command line option:

  $ salmonella --log-file=my-log-file.log big-chicken slice


====== Using salmonella as an egg lint tool

Suppose you are working on a new egg and you want to check if it is
working ok before releasing it (or a new version).  You can use
salmonella to check it:

  $ cd my-egg    # where your egg code is stored
  $ salmonella



==== salmonella-log-viewer

This tool can be used to turn salmonella log files into something
readable on the standard output.

Just provide a salmonella log file as argument to
{{salmonella-log-viewer}}:

    $ salmonella-log-viewer salmonella.log

Alternatively, you can use
[[/egg/salmonella-html-report|salmonella-html-report]] for a prettier
and more complete format.



==== salmonella-epidemy

Note: this tool is currently not supported on Windows systems.

{{salmonella-epidemy}} can be used to run multiple salmonella
instances in parallel.  It can be handy when you have a multi-core
machine and you want to make use of all cores, for example.  The
command line options are basically the same as for {{salmonella}},
plus a {{--instances=<number>}} to indicate how many {{salmonella}}
instances you want to run in parallel.

This tool can significantly speed up salmonella execution times on
multi-core machines.

Usage example:

  $ salmonella-epidemy --instances=2 big-chicken slice spiffy amb



==== salmonella-log-merger

This tool simply puts more than one salmonella log into a single one,
so it can be used by tools like {{salmonella-log-viewer}} and
[[/egg/salmonella-html-report|salmonella-html-report]].


Usage example:

  $ salmonella-log-merger --log-file=full.log file1.log file2.log



==== salmonella-log-inquirer

{{salmonella-log-inquirer}} can be used to query salmonella log files.
Below you can see its help message and a usage example.

  salmonella-log-inquirer [ -h | -help | --help ]
    Print this message.
  
  salmonella-log-inquirer --version
    Show version and exit.
  
  salmonella-log-inquirer --log-info <log file>
    Show information about the environment where salmonella was executed
    to generate the given <log-file> (e.g., environment variables, C
    compiler, CHICKEN version, salmonella command line etc.)
  
  salmonella-log-inquirer --statistics <log file>
    Show simple statistics on installation, tests and documentation from
    the log file .
  
  salmonella-log-inquirer --list-eggs <log file>
    List eggs covered in <log file>.
  
  salmonella-log-inquirer --action=<action> --egg=<egg> [ --part=<part> ] <log file>
    Query <action> and, optionally, <part> for <egg> in <log file>.
  
  <action>s:
  fetch
  install
  check-version
  test
  meta-data
  
  <part>s (the default part is "message"):
  message
  status
  duration
  

Usage example:

  $ salmonella-log-inquirer --action=fetch --egg=this salmonella.log
  '/home/chicken/salmonella/build/salmonella-run-publish/chicken/bin/chicken-install' -r -v -test this 2>&1
  this not cached
  resolving alias `kitten-technologies' to: http://chicken.kitten-technologies.co.uk/henrietta.cgi
  resolving alias `call-cc' to: http://code.call-cc.org/cgi-bin/henrietta.cgi
  trying server http://chicken.kitten-technologies.co.uk/henrietta.cgi ...
  downloading this: http://chicken.kitten-technologies.co.uk/henrietta.cgi
  connecting to host "chicken.kitten-technologies.co.uk", port 80 ...
  requesting "/henrietta.cgi?name=this&release=5&mode=default&tests=yes" ...
  reading response ...
  HTTP/1.1 200 OK
  Date: Thu, 25 Jul 2019 11:47:58 GMT
  Server: Apache/2.2.31 (Unix) DAV/2 PHP/5.5.36 mod_fastcgi/2.4.6
  Connection: close
  Transfer-Encoding: chunked
  Content-Type: text/plain
  reading chunks .
  reading files ...
    ./README.md
    ./this.egg
    ./this.release-info
    ./this.scm
  cp -r '/tmp/temp303b.18990'/* '/home/chicken/salmonella/build/salmonella-run-publish/salmonella-repo/repo/cache/this'
  this located at /home/chicken/salmonella/build/salmonella-run-publish/salmonella-repo/repo/cache/this
  

==== Log file format

The command line tool writes a log file which contains records in the
following format:

 (<egg> <action> <status> <message> <duration>)


===== <egg>

Can be either a symbol that indicates the egg-name or {{#f}} to
indicate the {{start}} and {{end}} actions (logged when salmonella is
started and when it finishes testing, respectively).

===== <action>

A symbol to indicate the action that was executed.  Can be one of the
following values:

* {{start}}: starting salmonella
* {{fetch}}: fetching egg
* {{install}}: installing egg
* {{skip}}: skipping egg
* {{test}}: testing egg
* {{end}}: salmonella has finished

===== <status>

A numeric value indicating the exit status of the executed action.
When the action is {{test}} and {{status}} is {{-1}}, it means that
the egg has no tests.


===== <message>

The output generated by the commands executed to perform {{<action>}}.
A string.


===== <duration>

The time (in seconds) that was taken to execute {{<action>}}.

For the {{start}} and {{end}} actions, the value is the seconds since
epoch, so the total salmonella execution time can be determined by
subtracting the start time from the end time.


=== Modules

Salmonella provides two modules:

; {{salmonella}}: the core salmonella functionality and basic data strucuture for logs ({{report}} records)

; {{salmonella-log-parser}}: provides procedures to access log files and compute simple statistics regarding log file data.


==== salmonella

===== report
<record>(report egg action status message duration)</record>
<procedure>make-report</procedure>
<procedure>report?</procedure>
<procedure>report-egg</procedure>
<procedure>report-action</procedure>
<procedure>report-status</procedure>
<procedure>report-message</procedure>
<procedure>report-duration</procedure>
<procedure>report-egg-set!</procedure>
<procedure>report-action-set!</procedure>
<procedure>report-status-set!</procedure>
<procedure>report-message-set!</procedure>
<procedure>report-duration-set!</procedure>

{{report}} objects.  Each log file registry is represented by a
{{report}} object (serialized as a list).

===== report->list
<procedure>(report->list report)</procedure>

Convert a {{report}} object to a list.


===== log!
<procedure>(log! report log-file)</procedure>

Print the {{report}} representation to {{log-file}}.


===== make-salmonella
<procedure>(make-salmonella tmp-dir #!key chicken-installation-prefix chicken-install-args eggs-source-dir eggs-doc-dir this-egg?)</procedure>

The salmonella maker.  Returns a procedure that receives symbols
(''methods'') to indicate the actions to be performed.  The available
methods are (example considering a {{salmonella}} object returned by
{{make-salmonella}}):

====== {{(salmonella 'init-repo!)}}

====== {{(salmonella 'clear-repo!)}}

====== {{(salmonella 'fetch <egg>)}}

====== {{(salmonella 'install <egg>)}}

====== {{(salmonella 'test <egg>)}}

====== {{(salmonella 'check-version <egg>)}}

====== {{(salmonella 'env-info)}}

====== {{(salmonella 'meta-data <egg>)}}

====== {{(salmonella 'check-dependencies <egg> <meta data>)}}

====== {{(salmonella 'check-category <egg> <meta data>)}}

====== {{(salmonella 'check-license <egg> <meta data>)}}

====== {{(salmonella 'check-author <egg> <meta data>)}}

====== {{(salmonella 'check-doc <egg>)}}



==== salmonella-log-parser

===== Reading

<procedure>(read-log-file filename)</procedure>

Reads the log file {{filename}} and returns a list of report records.

===== {{fetch}} action

====== fetch-status
<procedure>(fetch-status egg log)</procedure>

====== fetch-message
<procedure>(fetch-message egg log)</procedure>

====== fetch-duration
<procedure>(fetch-duration egg log)</procedure>



===== {{install}} action

====== install-status
<procedure>(install-status egg log)</procedure>

====== install-message
<procedure>(install-message egg log)</procedure>

====== install-duration
<procedure>(install-duration egg log)</procedure>



===== {{check-version}} action

====== check-version-status
<procedure>(check-version-status egg log)</procedure>

====== check-version-message
<procedure>(check-version-message egg log)</procedure>

====== egg-version
<procedure>(egg-version egg log)</procedure>

====== check-version-ok?
<procedure>(check-version-ok? egg log)</procedure>


===== {{test}} action

====== test-status
<procedure>(test-status egg log)</procedure>

====== test-message
<procedure>(test-message egg log)</procedure>

====== test-duration
<procedure>(test-duration egg log)</procedure>

====== has-test?
<procedure>(has-test? egg log)</procedure>


===== {{meta-data}} action

====== meta-data
<procedure>(meta-data egg log)</procedure>

====== egg-dependencies
<procedure>(egg-dependencies egg log)</procedure>

====== egg-license
<procedure>(egg-license egg log #!key with-test-dependencies? with-versions?)</procedure>



===== {{check-doc}} action

====== doc-exists?
<procedure>(doc-exists? egg log)</procedure>



===== {{start}} & {{end}} actions

====== start-time
<procedure>(start-time log)</procedure>

====== end-time
<procedure>(end-time log)</procedure>

====== total-time
<procedure>(total-time log)</procedure>

====== salmonella-info
<procedure>(salmonella-info log)</procedure>



===== Statistics

====== count-install-ok
<procedure>(count-install-ok log)</procedure>

====== count-install-fail
<procedure>(count-install-fail log)</procedure>

====== count-test-ok
<procedure>(count-test-ok log)</procedure>

====== count-test-fail
<procedure>(count-test-fail log)</procedure>

====== count-no-test
<procedure>(count-no-test log)</procedure>

====== count-total-eggs
<procedure>(count-total-eggs log #!key with-skipped?)</procedure>

====== count-documented
<procedure>(count-documented log)</procedure>

====== count-undocumented
<procedure>(count-undocumented log)</procedure>


===== Miscelaneous

====== prettify-time
<procedure>(prettify-time time)</procedure>

====== sort-eggs
<procedure>(sort-eggs eggs)</procedure>

====== log-eggs
<procedure>(log-eggs log)</procedure>

====== log-skipped-eggs
<procedure>(log-skipped-eggs log)</procedure>



=== Environment variables

When running, salmonella sets the {{SALMONELLA_RUNNING}} environment
variable.  If you need to check if your tests code is being run by
salmonella, this variable can be used.


=== Caveats

==== Testing executable files installed by eggs

Salmonella sets a private repository to install and test eggs, so when
extensions install programs, they get installed into
{{<salmonella-repo-dir>/bin}}, not into {{$CHICKEN_PREFIX/bin}}.

So, how to invoke (from tests/run.scm, for example) a binary
executable file your egg installs?  You have the following options:

* you can build up the absolute pathname for the executable, like
  {{(make-pathname (program-path) "the-executable")}}.  That is not
  going to work with salmonella, since the executable file will get
  installed into {{<salmonella-repo-dir>/bin}}, not
  {{$CHICKEN_PREFIX/bin}}.  Since {{program-path}} points to
  {{$CHICKEN_PREFIX/bin}} (or, when {{$CHICKEN_PREFIX}} is not set, it
  falls back to {{C_INSTALL_BIN_HOME}}, which is set during CHICKEN's
  build time), salmonella won't find it, as salmonella didn't install
  it there.  Salmonella ''may'' find it, in case you have that egg
  installed, but in this case you'll be testing a different program.

* you can rely on the executable being found in your system's {{PATH}}
  variable.  Salmonella actually prepends
  {{<salmonella-repo-dir>/bin}} to your system's {{PATH}} variable
  when running, so, if you call the executable file installed by your
  egg with with {{(system "the-executable")}}, for example, salmonella
  will run the right file.  On the other hand, you if run
  {{chicken-install}} with the {{-test}} option, and the CHICKEN tools
  are not in the system's {{PATH}} (e.g.
  {{/some/dir/not/in/PATH/chicken-install -test}}), the executable installed by your egg
  will not be found.

So far the only "portable" solution is a combination of both options,
which happens to be a bit ugly.  Salmonella sets the
{{SALMONELLA_RUNNING}} environment variable when it runs, so your test
script may check that variable and assume your test executable will be
found in {{PATH}} if it is set.  If {{SALMONELLA_RUNNING}} is not set,
just build up the absolute path to your executable using
{{(program-path)}}.  Here's an example:

<enscript highlight=scheme>
(use setup-api)

(define executable
  (make-pathname
   (if (get-environment-variable "SALMONELLA_RUNNING")
       #f ;; salmonella adds its REPO_PREFIX/bin to PATH
       (program-path))
   "executable-filename"))
</enscript>

==== salmonella-epidemy on Windows

{{salmonella-epidemy}} is currently not supported on Windows systems.


=== License

  Copyright (c) 2010-2019, Mario Domenech Goulart
  All rights reserved.
  
  Redistribution and use in source and binary forms, with or without
  modification, are permitted provided that the following conditions
  are met:
  1. Redistributions of source code must retain the above copyright
     notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright
     notice, this list of conditions and the following disclaimer in the
     documentation and/or other materials provided with the distribution.
  3. The name of the authors may not be used to endorse or promote products
     derived from this software without specific prior written permission.
  
  THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS
  OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY
  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
  GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
  IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
  OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
  IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


=== Version history

==== Version 3.0.0

* CHICKEN 5 support.  Now both CHICKEN 4 and CHICKEN 5 are supported. Most of the code is shared.  Where CHICKEN 4 and 5 are not compatible, {{cond-expand}} is used for version-specific implementations.

* New command line tool: {{salmonella-log-inquirer}}.  This tool can be used to query salmonella log files.

* More strict command line parser. Now invalid command line options are detected and reported as errors.

* Exit non-zero in case of errors.  Salmonella now exits non-zero in case of any check failure.

* Add {{--clear-chicken-home}} command line option for {{salmonella}} and {{salmonella-epidemy}}.  Don't use this option unless you know exactly what you are doing.  Check the documentation of this option before using it.

* The {{--eggs-source-dir}} option has been removed.  It is still possible to use salmonella with a local directory containing egg sources by configuring {{chicken-install}} ({{-t local -l <eggs-source-dir>}} in CHICKEN 4 and by configuring the {{location}} form in {{PREFIX/share/chicken/setup.defaults}} in CHICKEN 5).

* Replace --salmonella-prefix command line option by {{--salmonella-tools-dir}}.  {{salmonella-epidemy}} relies on both {{salmonella}} and {{salmonella-log-merger}}.  Since portably determining the directory of {{salmonella-epidemy}} is not trivial, and we don't want to rely on {{$PATH}} to call external tools, {{--salmonella-tools-dir}} has been created.  With this, {{--salmonella-prefix}} became useless.

* Salmonella now copy {{csc}} and {{csi}} from {{<host-prefix>/bin}} to {{<tmp-dir>/bin}}.  With that, salmonella no longer adds <host-prefix>/bin to {{$PATH}}, which would cause salmonella to overlook eggs that rely on executables installed in {{<host-prefix>/bin}} but not declared as dependencies.

* Many bugs fixed


==== Version 2.7

* Command line tools handle {{--version}} (suggested by Michael Stapelberg)
* salmonella: prepend {{<repo dir>/bin}} and {{<chicken-installation-prefix>/bin}} to {{PATH}}
* salmonella: fix scrutinizer warnings
* salmonella: set {{CHICKEN_C_INCLUDE_PATH}}
* salmonella: print/log environment variables set when running
* salmonella-epidemy: delete repo-dir, unless {{--keep-repo}} is provided
* salmonella-epidemy: write per-instance temporary directories in repo-dir
* salmonella-epidemy: show when instances are finished
* salmonella-epidemy: fix verbosity handling
* salmonella-epidemy: properly escape special shell characters in {{--chicken-install-args}}
* salmonella-epidemy: remove {{--eggs-source-dir}} support due to concurrency issues
* salmonella-epidemy: unless {{--salmonella-prefix}} is provided, salmonella-epidemy calls salmonella from the same prefix
* salmonella-epidemy: add {{-h}}|{{--help}} command line options
* salmonella-epidemy: correctly handle {{--skip-eggs}}


==== Version 2.6

* Handle broken .meta files
* Handle version specification in {{test-depends}} (thanks to [[/users/peter-bex|Peter Bex]] for pointing that out)
* Fixed filenames for {{chicken}} and {{csi}} on mingw ({{.exe}} extension).  Thanks to Dan Leslie for pointing that out.
* The {{--this-egg}} command line option has been deprecated


==== Version 2.5

*  Log {{check-version}} actions even when fetching eggs from remote servers (i.e., no {{--eggs-source-dir}}), so version numbers can get logged


==== Version 2.4

* "Fixed" hack to remove {{-test}} from {{chicken-install}} arguments when installing eggs


==== Version 2.3

* Bug fix: reference to unbound variable {{eggs}} (thanks to Shawn Rutledge for noticing that)


==== Version 2.2

* Only create the temporary repo dir when it is necessary (i.e., not for handling {{--help}} or when there's nothing to do)


==== Version 2.1

* Hours, minutes and seconds formatted with exact numbers ({{prettify-time}})


==== Version 2.0

* Rewrite of the 1.x version