source: project/wiki/eggref/4/scsh-process @ 36676

Last change on this file since 36676 was 36676, checked in by sjamaan, 10 months ago

scsh-process: Add 1.5.0 to changelog

File size: 21.4 KB
Line 
1[[tags: egg]]
2
3== SCSH-Process
4
5[[toc:]]
6
7=== Description
8
9A reimplementation for Chicken of SCSH's [[http://www.scsh.net/docu/html/man-Z-H-3.html|process notation]].
10
11=== Requirements
12
13[[llrb-tree]]
14
15=== Documentation
16
17This egg implements all the special forms required to implement SCSH's
18convenient process notation.  This notation is called "EPF", which
19stands for ''extended process form''.
20
21The procedural equivalents for these special forms are also available.
22Where it's required or makes sense, a handful of other procedures are
23also provided.  This egg does '''not''' strive for full SCSH
24compatibility; it exists only to bring the convenience of EPF notation
25to Chicken.
26
27==== Caveats
28
29There are a few important caveats, especially related to threading.
30Please read this section thoroughly before using scsh-process!  If
31none of this makes sense yet, read the rest of the manual first, then
32come back.  The issues are listed in order of decreasing importance.
33
34===== Forking and threads
35
36In versions of Chicken before 4.8.1 (rev 47b5be71), there is no
37support for killing threads upon forking.  This means that other
38threads will always continue running whenever you fork.  Please note
39that all of the EPF forms implicitly fork for each process in a
40pipeline.  If you intend to use {{scsh-process}} in a program which
41uses threads, please use a Chicken which supports this, otherwise you
42''will'' get unpredictable behavior.
43
44If you are using threads at all, it is important to realize that
45thread killing is completely insensitive to your delicate
46synchronization primitives or resource management.  Ensure that all
47locks are released and you don't have any non-GCable memory allocated
48in other threads.  If you have finalizers registered, those will still
49run, though.
50
51===== Signal handling
52
53Beware that if you set a {{signal/chld}} handler, you will need to
54remember the original handler and call it from your handler.  If you
55don't, you must manually wait for all children.  If you've installed a
56handler before scsh-process is loaded, it will be automatically
57chained by the handler installed by scsh-process.
58
59===== Process reaping
60
61Loading scsh-process will transparently cause the posix {{process-wait}}
62procedure to be replaced so it can update the bookkeeping information
63about reaped child processes.  See the description in the SCSH manual
64about how SCSH performs
65[[http://www.scsh.net/docu/html/man-Z-H-4.html#node_sec_3.4.1|process reaping]].
66Currently only the "early" reaping strategy is supported.
67
68==== Macros
69
70The special forms provided by this egg are the most convenient to use,
71but also slightly less powerful than the underlying procedures they
72use.
73
74===== Basic process macros
75
76<macro>(run pf [redirection ...])</macro>
77<macro>(& pf [redirection ...])</macro>
78<macro>(exec-epf pf [redirection ...])</macro>
79
80These forms run a new process pipeline, described by process form
81{{pf}}, with optional redirections of input and output descriptors
82indicated by any number of {{redirection}} patterns.  These processes
83don't interact with the calling process; if you want to capture their
84output there are other forms, like {{run/string}}, {{run/port}} and
85friends.  Please also beware that these forms don't do anything to
86indicate nonzero exit statuses in pipelines.  The {{&&}} and {{||}}
87macros might help if you need to do status checks.
88
89The {{run}} form simply runs the process and waits for the final
90process in the pipeline to exit.  It returns three values relating to
91this final process:
92
93* Either its exit status as an integer if the process terminated normally, or the signal number that terminated/stopped the process.
94* #t if the process exited normally, #f if it terminated abnormally.
95* Its process id (an integer).
96
97You'll note that these are just the same values returned by
98{{process-wait}}, but in a slightly different order ({{wait}} utilises
99this modified order, as well).  This is for compatibility reasons:
100In SCSH this form returns only the exit status, but because Chicken
101accepts multiple values in single value contexts (discarding all
102but the first), we can provide the other ones as extra values, thereby
103avoiding gratuitous incompatibility.
104
105The {{&}} form simply runs the process in the background and returns
106one value: a process object representing the last process in the
107pipeline.
108
109The {{exec-epf}} form never returns; it replaces the current process
110by the last process in the pipeline.  The others are implemented in
111terms of this form:
112
113<enscript highlight="scheme">
114(& . epf) => (process-fork (lambda () (exec-epf . epf)) #t)
115(run . epf) => (process-wait (& . epf))
116</enscript>
117
118A process form followed by a set of redirections is called "extended
119process form" in SCSH terminology.  A process form can be one of
120the following:
121
122<enscript highlight="scheme">
123(<program> <arg> ...)
124(pipe <pf> ...)
125(pipe+ <connect-list> <pf> ...)
126(epf <pf> <redirection> ...)
127(begin <s-expr> ...)
128</enscript>
129
130The arguments to a {{<program>}} rule are implicitly quasiquoted.  The
131basic building blocks are {{<program>}} and {{<begin>}}, which always
132correspond to one process in a pipeline.  The other rules are ways to
133combine these two.
134
135The {{pipe}} rule will hook up standard output and standard error of
136each {{pf}} to the next {{pf}}'s standard input, just like in a
137regular Unix shell pipeline.
138
139The {{pipe+}} rule is like pipe, but it allows you to hook up
140arbitrary file descriptors between two neighbouring processes.  This
141is done through the {{connect-list}}, which is a list of fd-mappings
142describing how ports are connected from one process to the next.  It
143has the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  The
144{{from-fd}}s correspond to outbound file descriptors in one process,
145the {{to-fd}}s correspond to inbound file descriptors in the other
146process.
147
148The {{epf}} rule is to get extended process forms in contexts where
149only process forms are accepted, like the {{pipe}} and {{pipe+}}
150subforms, and in the {{&&}} and {{||}} macros (so you can do file
151redirects here).
152
153The {{begin}} rule allows you to write scheme code which will be run
154in a forked process, having its {{current-input-port}},
155{{current-output-port}} and {{current-error-port}} hooked up to its
156neighbouring processes in the pipeline.
157
158A redirection can be one of the following:
159
160<enscript highlight="scheme">
161(> [<fd>] <file-name>)      ; Write fd (default: 1) to the given filename
162(>> [<fd>] <file-name>)     ; Like >, but append instead of overwriting
163(< [<fd>] <file-name>)      ; Read fd (default: 0) from the filename
164(<< [<fd>] <scheme-object>) ; Like <, but use object's printed representation
165(= <fd> <fd-or-port>)       ; Redirect fd to fd-or-port
166(- <fd-or-port>)            ; Close fd
167stdports                    ; Duplicate fd 0, 1, 2 from standard Scheme ports
168</enscript>
169
170The arguments to redirection rules are also implicitly quasiquoted.
171
172To tie it all together, here are a few examples:
173
174<enscript highlight="scheme">
175(use scsh-process)
176
177;; Writes "1235" to a file called "out" in the current directory.
178;; Shell equivalent:  echo 1234 + 1 | bc > out
179(run (pipe (echo "1234" + 1) ("bc")) (> out))
180
181(define message "hello, world")
182
183;; Writes 13 to stdout, with a forked Scheme process writing the data.
184;; Shell equivalent (sort of): echo 'hello, world' | wc -c
185(run (pipe (begin (display message) (newline)) (wc -c)))
186
187;; A verbose way of doing the same, using pipe+.  It connects the {{begin}}
188;; form's standard output and standard error to standard input of {{wc}}:
189(run (pipe+ ((1 2 0)) (begin (display message) (newline)) (wc -c)))
190
191;; Same as above, using redirection instead of port writing:
192(run (wc -c) (<< ,(string-append message "\n")))
193
194;; Writes nothing because stdout is closed:
195(run (wc -c) (<< ,message) (- 1))
196
197;; A complex toy example using nested pipes, with input/output redirection.
198;; Closest shell equivalent:
199;; ((sh -c "echo foo >&2") 2>&1 | cat) | cat
200(run (pipe+ ((1 0))
201            (pipe+ ((2 0)) (sh -c "echo foo >&2") (cat))
202            (cat)))
203</enscript>
204
205===== Process macros for interfacing with Scheme
206
207<macro>(run/port pf [redirection ...])</macro>
208<macro>(run/file pf [redirection ...])</macro>
209<macro>(run/string pf [redirection ...])</macro>
210<macro>(run/strings pf [redirection ...])</macro>
211<macro>(run/sexp pf [redirection ...])</macro>
212<macro>(run/sexps pf [redirection ...])</macro>
213
214These forms are equivalent to {{run}}, except they wire up the current
215process to the endpoint of the pipeline, allowing you to read the
216''standard output'' from the pipeline as a whole.  If you also need
217standard error or have even more specialized needs, take a look at
218the {{run/collecting}} form.
219
220The difference between these forms is in how this output is returned,
221and when the call returns:
222
223* {{run/port}} immediately returns after forking, and returns a port from which you can read.
224* {{run/file}} returns after the final process exits, resulting in a string which indicates a temporary file containing the process' output.
225* {{run/string}} returns when the process closes its standard output (ie, when EOF is read), collecting the standard output into a string.
226* {{run/strings}} is like {{run/string}}, but returns a list of strings, split at newline characters.
227* {{run/sexp}} reads an s-expression, and returns as soon as a complete s-expression was read.
228* {{run/sexps}} reads all s-expressions until eof, returning a list of s-expressions.  It returns as soon as EOF is read.
229
230===== Macros for conditional process control
231
232<macro>(&& pf ...)</macro>
233<macro>(|| pf ...)</macro>
234
235These macros act like their counterpart shell operators; they run the
236given process forms in sequence and stop either on the first "false"
237value (nonzero exit) or on the first "true" value (zero exit), respectively.
238
239The result value of these is {{#f}} or {{#t}}, so they act a lot like
240regular Scheme {{and}} and {{or}}.
241
242'''Note:''' The name of the {{||}} macro is really the empty symbol
243whereas in SCSH's reader, it reads as a symbol consisting of two pipe
244characters.  The name of these macros may change in the future if it
245turns out to cause too much trouble.
246
247===== Collecting multiple outputs
248
249<macro>(run/collecting fds pf ...)</macro>
250
251This form runs the {{pf}} form, redirecting each the file descriptors
252in the {{fds}} list to a separate tempfile, and waits for the process
253to complete.
254
255The result of this expression is {{(status file ...)}}.  {{status}} is
256the exit status of the process.  Each {{file}} entry is an opened
257input port for the temporary file that belongs to the file descriptor
258at the same offset in the {{fds}} list.  If you close the port, the
259tempfile is removed.
260
261See the [[http://www.scsh.net/docu/html/man-Z-H-3.html#node_sec_2.4.2|SCSH documentation]]
262for an extended rationale of why this works the way it does.
263
264==== Procedural interface
265
266These procedures form the basis for the special forms documented
267above, and can be used to implement your own, more specialized macros.
268
269===== Basic forking and pipeline primitives
270
271<procedure>(fork [thunk [continue-threads?]])</procedure>
272<procedure>(%fork [thunk [continue-threads?]])</procedure>
273
274If {{thunk}} is provided and not {{#f}}, the child process will invoke
275the thunk and exit when it returns.  If {{continue-threads}} is provided
276and {{#t}}, all existing threads will be kept alive in the child process;
277by default only the current thread will be kept alive.
278
279{{fork}} differs from the regular {{process-fork}} in its return value.
280Instead of a pid value, this returns a process object representing the
281child is returned in the parent process.  When {{thunk}} is not
282provided, {{#f}} (not zero!) is returned in the child process.
283
284Currently {{%fork}} is just an alias for {{fork}}.
285
286<procedure>(wait [pid-or-process [nohang]])</procedure>
287
288Like {{process-wait}}, but nonblocking: Suspends the current process
289until the child process described by {{pid-or-process}} (either a
290numerical process ID or a scsh-process object) has terminated using
291the UNIX system call waitpid(). If {{pid-or-process}} is not given,
292then this procedure waits for any child process. If {{nohang}} is
293given and not {{#f}} then the current process is not suspended.
294
295This procedure returns three values, '''in a different order from
296{{process-wait}}''':
297
298* Either the exit status, if the process terminated normally or the signal number that terminated/stopped the process.
299* #t if the process exited normally or #f otherwise.
300* {{pid}} or 0
301
302All values are {{#f}}, if {{nohang}} is true and the child process has
303not terminated yet.
304
305It is allowed to wait multiple times for the same process after it has
306completed, if you pass a process object.  Process IDs can only be
307waited for once after they have completed and will cause an error
308otherwise.
309
310Since version 0.8 of scsh-process, this procedure is nonblocking,
311which means you can {{wait}} for a child in a thread and have the
312other threads continue to run; the process isn't suspended, but a
313signal handler will take care of unblocking the waiting thread.
314
315
316<procedure>(signal-process proc signal)</procedure>
317
318Like {{process-signal}} from the POSIX unit, but accepts a process
319object ({{proc}}) instead of a pid.  Sends {{signal}} (an integer) to
320the given process.
321
322<procedure>(process-sleep sec)</procedure>
323
324Put the entire process to sleep for {{sec}} seconds.  Just an alias
325for {{sleep}} from the POSIX unit.
326
327<procedure>(process? object)</procedure>
328<procedure>(proc? object)</procedure>
329
330Is {{object}} an object representing a process?  The {{process?}}
331predicate is deprecated; {{proc?}} is API-compatible with SCSH.
332
333<procedure>(proc:pid proc)</procedure>
334
335Retrieve the process id (an integer) from the process object {{proc}}.
336
337<procedure>(fork/pipe [thunk [continue-threads?]])</procedure>
338<procedure>(%fork/pipe [thunk [continue-threads?]])</procedure>
339
340These fork the process as per {{fork}} or {{%fork}}, but additionally
341they set up a pipe between parent and child.  The child's standard
342output is set up to write to the pipe, while the parent's standard
343input is set up to read to the pipe.  Standard error is inherited
344from the parent.
345
346The return value is a process object or {{#f}}.
347
348Currently {{fork%/pipe}} is just an alias for {{fork/pipe}}.
349
350'''Important''': These procedures only set up the file descriptors,
351not the Scheme ports.  {{current-input-port}}, {{current-output-port}}
352and {{current-error-port}} still refer to their old file descriptors
353after a fork.  This means that you'll need to reopen the descriptors
354to get a Scheme port that reads from the child or writes to the parent:
355
356<enscript highlight="scheme">
357(use scsh-process posix)
358
359(process-wait
360  (fork/pipe (lambda ()
361               (with-output-to-port (open-output-file* 1)
362                 (lambda () (display "Hello, world.\n"))))))
363
364(read-line (open-input-file* 0)) => "Hello, world"
365</enscript>
366
367<procedure>(fork/pipe+ conns [thunk [continue-threads?]])</procedure>
368<procedure>(fork%/pipe+ conns [thunk [continue-threads?]])</procedure>
369
370These are like {{fork/pipe}} and {{fork%/pipe}}, except they allow you
371to control how the file descriptors are wired.  Conns is a list of
372lists, of the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  See the
373description of {{pipe+}} under the {{run}} special form for more
374information.
375
376Currently {{fork%/pipe+}} is just an alias for {{fork/pipe+}}.
377
378===== Executing programs from the path
379
380<procedure>(exec-path program [args ...])</procedure>
381<procedure>(exec-path* program arg-list)</procedure>
382
383This will simply execute {{program}} with the given arguments in
384{{args}} or {{args-list}}.  The program replaces the currently running
385process.  All arguments (including {{program}}) must be strings,
386symbols or numbers, and they will automatically be converted to
387strings before invoking the program.
388
389The program is looked up in the user's path, which is simply the
390{{$PATH}} environment variable.  There currently is no separately
391maintained path list like in SCSH.  The difference between the two
392procedures is that {{exec-path}} accepts a variable number of
393arguments and {{exec-path*}} requires them pre-collected into a list.
394
395===== Pipeline procedures
396
397<procedure>(run/port* thunk)</procedure>
398<procedure>(run/file* thunk)</procedure>
399<procedure>(run/string* thunk)</procedure>
400<procedure>(run/strings* thunk)</procedure>
401<procedure>(run/sexp* thunk)</procedure>
402<procedure>(run/sexps* thunk)</procedure>
403
404These set up a pipe between the current process and a forked off child
405process which runs the {{thunk}}.
406See the "unstarred" versions {{run/port}}, {{run/file}} ... {{run/sexps}}
407for more information about the semantics of these procedures.
408
409===== Collecting multiple outputs
410
411<procedure>(run/collecting* fds thunk)</procedure>
412
413Like {{run/collecting}}, but use a {{thunk}} instead of a process form.
414
415=== Changelog
416
417* 1.5.0 - Fix a race condition which caused {{SIGCHLD}} to be missed sometimes, and replace srfi-69 with llrb-tree (thanks to Jörg F. Wittenberger).
418* 1.4.0 - Add CHICKEN 5 compatibility.
419* 1.3.0 - Fix {{>}} redirection to truncate existing files (thanks to Jörg F. Wittenberger).
420* 1.2.2 - Fix {{(= 1 2)}} style redirection to be the correct way around, to match documentation and the original SCSH implementation (thanks to Diego "dieggsy").
421* 1.2.1 - Do not connect stdout of subprocess to stdin of parent in {{run/file}}; this is not needed because only the returned file should be written to (thanks to Diego "dieggsy").
422* 1.2.0 - Do not redirect stderr to stdout in {{fork/pipe}}, {{run/file*}} and {{run/file}}; instead, stderr is inherited from the parent (thanks to Jörg F. Wittenberger).  This improves compatibility with scsh.
423* 1.1.0 - Move signal handler into a separate thread to allow signaling the thread that was interrupted by the handler.
424* 1.0.0 - Fix {{fork}} restoration of signal mask to what it was before {{fork}} rather than blindly unmasking it.  Fix {{wait}} test with {{#f}} argument.  Fix (conditional) unmasking of signal/chld in the child thunk after performing a {{fork}}.
425* 0.9.0 - Fix race condition that sometimes caused the loss of the exit statuses of already reaped child processes, which would cause an infinite loop when waiting for the process again.
426* 0.8.3 - Fix {{wait}} with a plain pid or {{#f}} so that it updates any corresponding scsh-process objects that may exist (thanks to Jörg F. Wittenberger).
427* 0.8.2 - Clean up file descriptors in {{run/..*}} procedures (again, thanks to Jörg F. Wittenberger).
428* 0.8.1 - Reinstall deadlock detection workaround thread after forking and killing all threads (thanks to Jörg F. Wittenberger).
429* 0.8 - Add support for waiting for children from threads without blocking the entire process (thanks to Jörg F. Wittenberger).
430* 0.7.1 - Fix version number in {{.setup}} file (thanks to Jörg F. Wittenberger).
431* 0.7 - Actually export {{pid:proc}}.  Clear pending child process table on fork (thanks to Jörg F. Wittenberger).
432* 0.6 - Add {{signal-process}}, {{process-sleep}} and {{pid:proc}}.  Deprecated {{process?}} in favor of {{proc?}}.  Thanks to Jörg F. Wittenberger for the suggestion of adding some of these.
433* 0.5 - Standard error is no longer redirected by default, making it more consistent with UNIX shells and the original SCSH.  Thanks to Haochi Kiang for pointing this out and providing a patch.
434* 0.4.1 - Allow the use of unquote-splicing in run macro forms (thanks to [[/users/moritz-heidkamp|Moritz Heidkamp]] for pointing this out)
435* 0.4 - Support {{continue-threads}} parameter for {{fork}} on newer Chickens which support {{kill-all-threads}} option in {{process-fork}}.  This should make this library safe for use in threaded programs.
436* 0.3.1 - Change {{wait}} result values to all be {{#f}} if {{nohang}} is {{#t}}.
437* 0.3 - Fix segfault properly, revert back to using standard-extension.  Add {{wait}} and do not accept scsh-process objects in POSIX {{process-wait}}.
438* 0.2.1 - Workaround for segmentation fault caused by compiled version of scsh-process in 4.8.0 due to -O3 being the default compilation of a standard-extension.
439* 0.2 - Fix {{<<}}-redirection and increase robustness of test for it.
440* 0.1.2 - Fix setup-file to have non-bogus version identifier.  Don't rely on "bc" being present in the tests.
441* 0.1.1 - Fix order of values returned by {{process-wait}} to be consistent with POSIX unit.
442* 0.1 - Initial release, created at the [[/event/chicken-uk-2012|Chicken UK 2012]] hacking event.
443
444=== Author
445
446[[/users/peter-bex|Peter Bex]]
447
448=== Repository
449
450[[http://code.more-magic.net/scsh-process]]
451
452=== License
453
454  Copyright (c) 2012-2018, Peter Bex
455  All rights reserved.
456 
457  Redistribution and use in source and binary forms, with or without
458  modification, are permitted provided that the following conditions
459  are met:
460  1. Redistributions of source code must retain the above copyright
461     notice, this list of conditions and the following disclaimer.
462  2. Redistributions in binary form must reproduce the above copyright
463     notice, this list of conditions and the following disclaimer in the
464     documentation and/or other materials provided with the distribution.
465 
466  THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR
467  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
468  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
469  IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT,
470  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
471  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
472  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
473  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
474  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
475  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Note: See TracBrowser for help on using the repository browser.