source: project/wiki/eggref/5/scsh-process @ 36301

Last change on this file since 36301 was 36301, checked in by sjamaan, 15 months ago

Add scsh-process to eggref/5

File size: 20.8 KB
Line 
1[[tags: egg]]
2
3== SCSH-Process
4
5[[toc:]]
6
7=== Description
8
9A reimplementation for Chicken of SCSH's [[http://www.scsh.net/docu/html/man-Z-H-3.html|process notation]].
10
11=== Requirements
12
13none
14
15=== Documentation
16
17This egg implements all the special forms required to implement SCSH's
18convenient process notation.  This notation is called "EPF", which
19stands for ''extended process form''.
20
21The procedural equivalents for these special forms are also available.
22Where it's required or makes sense, a handful of other procedures are
23also provided.  This egg does '''not''' strive for full SCSH
24compatibility; it exists only to bring the convenience of EPF notation
25to Chicken.
26
27==== Caveats
28
29There are a few important caveats, especially related to threading.
30Please read this section thoroughly before using scsh-process!  If
31none of this makes sense yet, read the rest of the manual first, then
32come back.  The issues are listed in order of decreasing importance.
33
34===== Forking and threads
35
36If you are using threads, it is important to realize that thread
37killing is completely insensitive to your delicate synchronization
38primitives or resource management.  Ensure that all locks are released
39and you don't have any non-GCable memory allocated in other threads.
40If you have finalizers registered, those will still run, though.
41
42===== Signal handling
43
44Beware that if you set a {{signal/chld}} handler, you will need to
45remember the original handler and call it from your handler.  If you
46don't, you must manually wait for all children.  If you've installed a
47handler before scsh-process is loaded, it will be automatically
48chained by the handler installed by scsh-process.
49
50===== Process reaping
51
52Loading scsh-process will transparently cause the posix {{process-wait}}
53procedure to be replaced so it can update the bookkeeping information
54about reaped child processes.  See the description in the SCSH manual
55about how SCSH performs
56[[http://www.scsh.net/docu/html/man-Z-H-4.html#node_sec_3.4.1|process reaping]].
57Currently only the "early" reaping strategy is supported.
58
59==== Macros
60
61The special forms provided by this egg are the most convenient to use,
62but also slightly less powerful than the underlying procedures they
63use.
64
65===== Basic process macros
66
67<macro>(run pf [redirection ...])</macro>
68<macro>(& pf [redirection ...])</macro>
69<macro>(exec-epf pf [redirection ...])</macro>
70
71These forms run a new process pipeline, described by process form
72{{pf}}, with optional redirections of input and output descriptors
73indicated by any number of {{redirection}} patterns.  These processes
74don't interact with the calling process; if you want to capture their
75output there are other forms, like {{run/string}}, {{run/port}} and
76friends.  Please also beware that these forms don't do anything to
77indicate nonzero exit statuses in pipelines.  The {{&&}} and {{||}}
78macros might help if you need to do status checks.
79
80The {{run}} form simply runs the process and waits for the final
81process in the pipeline to exit.  It returns three values relating to
82this final process:
83
84* Either its exit status as an integer if the process terminated normally, or the signal number that terminated/stopped the process.
85* #t if the process exited normally, #f if it terminated abnormally.
86* Its process id (an integer).
87
88You'll note that these are just the same values returned by
89{{process-wait}}, but in a slightly different order ({{wait}} utilises
90this modified order, as well).  This is for compatibility reasons:
91In SCSH this form returns only the exit status, but because Chicken
92accepts multiple values in single value contexts (discarding all
93but the first), we can provide the other ones as extra values, thereby
94avoiding gratuitous incompatibility.
95
96The {{&}} form simply runs the process in the background and returns
97one value: a process object representing the last process in the
98pipeline.
99
100The {{exec-epf}} form never returns; it replaces the current process
101by the last process in the pipeline.  The others are implemented in
102terms of this form:
103
104<enscript highlight="scheme">
105(& . epf) => (process-fork (lambda () (exec-epf . epf)) #t)
106(run . epf) => (process-wait (& . epf))
107</enscript>
108
109A process form followed by a set of redirections is called "extended
110process form" in SCSH terminology.  A process form can be one of
111the following:
112
113<enscript highlight="scheme">
114(<program> <arg> ...)
115(pipe <pf> ...)
116(pipe+ <connect-list> <pf> ...)
117(epf <pf> <redirection> ...)
118(begin <s-expr> ...)
119</enscript>
120
121The arguments to a {{<program>}} rule are implicitly quasiquoted.  The
122basic building blocks are {{<program>}} and {{<begin>}}, which always
123correspond to one process in a pipeline.  The other rules are ways to
124combine these two.
125
126The {{pipe}} rule will hook up standard output and standard error of
127each {{pf}} to the next {{pf}}'s standard input, just like in a
128regular Unix shell pipeline.
129
130The {{pipe+}} rule is like pipe, but it allows you to hook up
131arbitrary file descriptors between two neighbouring processes.  This
132is done through the {{connect-list}}, which is a list of fd-mappings
133describing how ports are connected from one process to the next.  It
134has the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  The
135{{from-fd}}s correspond to outbound file descriptors in one process,
136the {{to-fd}}s correspond to inbound file descriptors in the other
137process.
138
139The {{epf}} rule is to get extended process forms in contexts where
140only process forms are accepted, like the {{pipe}} and {{pipe+}}
141subforms, and in the {{&&}} and {{||}} macros (so you can do file
142redirects here).
143
144The {{begin}} rule allows you to write scheme code which will be run
145in a forked process, having its {{current-input-port}},
146{{current-output-port}} and {{current-error-port}} hooked up to its
147neighbouring processes in the pipeline.
148
149A redirection can be one of the following:
150
151<enscript highlight="scheme">
152(> [<fd>] <file-name>)      ; Write fd (default: 1) to the given filename
153(>> [<fd>] <file-name>)     ; Like >, but append instead of overwriting
154(< [<fd>] <file-name>)      ; Read fd (default: 0) from the filename
155(<< [<fd>] <scheme-object>) ; Like <, but use object's printed representation
156(= <fd> <fd-or-port>)       ; Redirect fd to fd-or-port
157(- <fd-or-port>)            ; Close fd
158stdports                    ; Duplicate fd 0, 1, 2 from standard Scheme ports
159</enscript>
160
161The arguments to redirection rules are also implicitly quasiquoted.
162
163To tie it all together, here are a few examples:
164
165<enscript highlight="scheme">
166(import scsh-process)
167
168;; Writes "1235" to a file called "out" in the current directory.
169;; Shell equivalent:  echo 1234 + 1 | bc > out
170(run (pipe (echo "1234" + 1) ("bc")) (> out))
171
172(define message "hello, world")
173
174;; Writes 13 to stdout, with a forked Scheme process writing the data.
175;; Shell equivalent (sort of): echo 'hello, world' | wc -c
176(run (pipe (begin (display message) (newline)) (wc -c)))
177
178;; A verbose way of doing the same, using pipe+.  It connects the {{begin}}
179;; form's standard output and standard error to standard input of {{wc}}:
180(run (pipe+ ((1 2 0)) (begin (display message) (newline)) (wc -c)))
181
182;; Same as above, using redirection instead of port writing:
183(run (wc -c) (<< ,(string-append message "\n")))
184
185;; Writes nothing because stdout is closed:
186(run (wc -c) (<< ,message) (- 1))
187
188;; A complex toy example using nested pipes, with input/output redirection.
189;; Closest shell equivalent:
190;; ((sh -c "echo foo >&2") 2>&1 | cat) | cat
191(run (pipe+ ((1 0))
192            (pipe+ ((2 0)) (sh -c "echo foo >&2") (cat))
193            (cat)))
194</enscript>
195
196===== Process macros for interfacing with Scheme
197
198<macro>(run/port pf [redirection ...])</macro>
199<macro>(run/file pf [redirection ...])</macro>
200<macro>(run/string pf [redirection ...])</macro>
201<macro>(run/strings pf [redirection ...])</macro>
202<macro>(run/sexp pf [redirection ...])</macro>
203<macro>(run/sexps pf [redirection ...])</macro>
204
205These forms are equivalent to {{run}}, except they wire up the current
206process to the endpoint of the pipeline, allowing you to read the
207''standard output'' from the pipeline as a whole.  If you also need
208standard error or have even more specialized needs, take a look at
209the {{run/collecting}} form.
210
211The difference between these forms is in how this output is returned,
212and when the call returns:
213
214* {{run/port}} immediately returns after forking, and returns a port from which you can read.
215* {{run/file}} returns after the final process exits, resulting in a string which indicates a temporary file containing the process' output.
216* {{run/string}} returns when the process closes its standard output (ie, when EOF is read), collecting the standard output into a string.
217* {{run/strings}} is like {{run/string}}, but returns a list of strings, split at newline characters.
218* {{run/sexp}} reads an s-expression, and returns as soon as a complete s-expression was read.
219* {{run/sexps}} reads all s-expressions until eof, returning a list of s-expressions.  It returns as soon as EOF is read.
220
221===== Macros for conditional process control
222
223<macro>(&& pf ...)</macro>
224<macro>(|| pf ...)</macro>
225
226These macros act like their counterpart shell operators; they run the
227given process forms in sequence and stop either on the first "false"
228value (nonzero exit) or on the first "true" value (zero exit), respectively.
229
230The result value of these is {{#f}} or {{#t}}, so they act a lot like
231regular Scheme {{and}} and {{or}}.
232
233'''Note:''' The name of the {{||}} macro is really the empty symbol
234whereas in SCSH's reader, it reads as a symbol consisting of two pipe
235characters.  The name of these macros may change in the future if it
236turns out to cause too much trouble.
237
238===== Collecting multiple outputs
239
240<macro>(run/collecting fds pf ...)</macro>
241
242This form runs the {{pf}} form, redirecting each the file descriptors
243in the {{fds}} list to a separate tempfile, and waits for the process
244to complete.
245
246The result of this expression is {{(status file ...)}}.  {{status}} is
247the exit status of the process.  Each {{file}} entry is an opened
248input port for the temporary file that belongs to the file descriptor
249at the same offset in the {{fds}} list.  If you close the port, the
250tempfile is removed.
251
252See the [[http://www.scsh.net/docu/html/man-Z-H-3.html#node_sec_2.4.2|SCSH documentation]]
253for an extended rationale of why this works the way it does.
254
255==== Procedural interface
256
257These procedures form the basis for the special forms documented
258above, and can be used to implement your own, more specialized macros.
259
260===== Basic forking and pipeline primitives
261
262<procedure>(fork [thunk [continue-threads?]])</procedure>
263<procedure>(%fork [thunk [continue-threads?]])</procedure>
264
265If {{thunk}} is provided and not {{#f}}, the child process will invoke
266the thunk and exit when it returns.  If {{continue-threads}} is provided
267and {{#t}}, all existing threads will be kept alive in the child process;
268by default only the current thread will be kept alive.
269
270{{fork}} differs from the regular {{process-fork}} in its return value.
271Instead of a pid value, this returns a process object representing the
272child is returned in the parent process.  When {{thunk}} is not
273provided, {{#f}} (not zero!) is returned in the child process.
274
275Currently {{%fork}} is just an alias for {{fork}}.
276
277<procedure>(wait [pid-or-process [nohang]])</procedure>
278
279Like {{process-wait}}, but nonblocking: Suspends the current process
280until the child process described by {{pid-or-process}} (either a
281numerical process ID or a scsh-process object) has terminated using
282the UNIX system call waitpid(). If {{pid-or-process}} is not given,
283then this procedure waits for any child process. If {{nohang}} is
284given and not {{#f}} then the current process is not suspended.
285
286This procedure returns three values, '''in a different order from
287{{process-wait}}''':
288
289* Either the exit status, if the process terminated normally or the signal number that terminated/stopped the process.
290* #t if the process exited normally or #f otherwise.
291* {{pid}} or 0
292
293All values are {{#f}}, if {{nohang}} is true and the child process has
294not terminated yet.
295
296It is allowed to wait multiple times for the same process after it has
297completed, if you pass a process object.  Process IDs can only be
298waited for once after they have completed and will cause an error
299otherwise.
300
301This procedure is nonblocking, which means you can {{wait}} for a
302child in a thread and have the other threads continue to run; the
303process isn't suspended, but a signal handler will take care of
304unblocking the waiting thread.
305
306
307<procedure>(signal-process proc signal)</procedure>
308
309Like {{process-signal}} from the POSIX unit, but accepts a process
310object ({{proc}}) instead of a pid.  Sends {{signal}} (an integer) to
311the given process.
312
313<procedure>(process-sleep sec)</procedure>
314
315Put the entire process to sleep for {{sec}} seconds.  Just an alias
316for {{sleep}} from the POSIX unit.
317
318<procedure>(process? object)</procedure>
319<procedure>(proc? object)</procedure>
320
321Is {{object}} an object representing a process?  The {{process?}}
322predicate is deprecated; {{proc?}} is API-compatible with SCSH.
323
324<procedure>(proc:pid proc)</procedure>
325
326Retrieve the process id (an integer) from the process object {{proc}}.
327
328<procedure>(fork/pipe [thunk [continue-threads?]])</procedure>
329<procedure>(%fork/pipe [thunk [continue-threads?]])</procedure>
330
331These fork the process as per {{fork}} or {{%fork}}, but additionally
332they set up a pipe between parent and child.  The child's standard
333output is set up to write to the pipe, while the parent's standard
334input is set up to read to the pipe.  Standard error is inherited
335from the parent.
336
337The return value is a process object or {{#f}}.
338
339Currently {{fork%/pipe}} is just an alias for {{fork/pipe}}.
340
341'''Important''': These procedures only set up the file descriptors,
342not the Scheme ports.  {{current-input-port}}, {{current-output-port}}
343and {{current-error-port}} still refer to their old file descriptors
344after a fork.  This means that you'll need to reopen the descriptors
345to get a Scheme port that reads from the child or writes to the parent:
346
347<enscript highlight="scheme">
348(import scsh-process (chicken file posix))
349
350(process-wait
351  (fork/pipe (lambda ()
352               (with-output-to-port (open-output-file* 1)
353                 (lambda () (display "Hello, world.\n"))))))
354
355(read-line (open-input-file* 0)) => "Hello, world"
356</enscript>
357
358<procedure>(fork/pipe+ conns [thunk [continue-threads?]])</procedure>
359<procedure>(fork%/pipe+ conns [thunk [continue-threads?]])</procedure>
360
361These are like {{fork/pipe}} and {{fork%/pipe}}, except they allow you
362to control how the file descriptors are wired.  Conns is a list of
363lists, of the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  See the
364description of {{pipe+}} under the {{run}} special form for more
365information.
366
367Currently {{fork%/pipe+}} is just an alias for {{fork/pipe+}}.
368
369===== Executing programs from the path
370
371<procedure>(exec-path program [args ...])</procedure>
372<procedure>(exec-path* program arg-list)</procedure>
373
374This will simply execute {{program}} with the given arguments in
375{{args}} or {{args-list}}.  The program replaces the currently running
376process.  All arguments (including {{program}}) must be strings,
377symbols or numbers, and they will automatically be converted to
378strings before invoking the program.
379
380The program is looked up in the user's path, which is simply the
381{{$PATH}} environment variable.  There currently is no separately
382maintained path list like in SCSH.  The difference between the two
383procedures is that {{exec-path}} accepts a variable number of
384arguments and {{exec-path*}} requires them pre-collected into a list.
385
386===== Pipeline procedures
387
388<procedure>(run/port* thunk)</procedure>
389<procedure>(run/file* thunk)</procedure>
390<procedure>(run/string* thunk)</procedure>
391<procedure>(run/strings* thunk)</procedure>
392<procedure>(run/sexp* thunk)</procedure>
393<procedure>(run/sexps* thunk)</procedure>
394
395These set up a pipe between the current process and a forked off child
396process which runs the {{thunk}}.
397See the "unstarred" versions {{run/port}}, {{run/file}} ... {{run/sexps}}
398for more information about the semantics of these procedures.
399
400===== Collecting multiple outputs
401
402<procedure>(run/collecting* fds thunk)</procedure>
403
404Like {{run/collecting}}, but use a {{thunk}} instead of a process form.
405
406=== Changelog
407
408* 1.4.0 - Port to CHICKEN 5 (thanks to Vasilij Schneidermann for providing an initial patch).
409* 1.3.0 - Fix {{>}} redirection to truncate existing files (thanks to Jörg F. Wittenberger).
410* 1.2.2 - Fix {{(= 1 2)}} style redirection to be the correct way around, to match documentation and the original SCSH implementation (thanks to Diego "dieggsy").
411* 1.2.1 - Do not connect stdout of subprocess to stdin of parent in {{run/file}}; this is not needed because only the returned file should be written to (thanks to Diego "dieggsy").
412* 1.2.0 - Do not redirect stderr to stdout in {{fork/pipe}}, {{run/file*}} and {{run/file}}; instead, stderr is inherited from the parent (thanks to Jörg F. Wittenberger).  This improves compatibility with scsh.
413* 1.1.0 - Move signal handler into a separate thread to allow signaling the thread that was interrupted by the handler.
414* 1.0.0 - Fix {{fork}} restoration of signal mask to what it was before {{fork}} rather than blindly unmasking it.  Fix {{wait}} test with {{#f}} argument.  Fix (conditional) unmasking of signal/chld in the child thunk after performing a {{fork}}.
415* 0.9.0 - Fix race condition that sometimes caused the loss of the exit statuses of already reaped child processes, which would cause an infinite loop when waiting for the process again.
416* 0.8.3 - Fix {{wait}} with a plain pid or {{#f}} so that it updates any corresponding scsh-process objects that may exist (thanks to Jörg F. Wittenberger).
417* 0.8.2 - Clean up file descriptors in {{run/..*}} procedures (again, thanks to Jörg F. Wittenberger).
418* 0.8.1 - Reinstall deadlock detection workaround thread after forking and killing all threads (thanks to Jörg F. Wittenberger).
419* 0.8 - Add support for waiting for children from threads without blocking the entire process (thanks to Jörg F. Wittenberger).
420* 0.7.1 - Fix version number in {{.setup}} file (thanks to Jörg F. Wittenberger).
421* 0.7 - Actually export {{pid:proc}}.  Clear pending child process table on fork (thanks to Jörg F. Wittenberger).
422* 0.6 - Add {{signal-process}}, {{process-sleep}} and {{pid:proc}}.  Deprecated {{process?}} in favor of {{proc?}}.  Thanks to Jörg F. Wittenberger for the suggestion of adding some of these.
423* 0.5 - Standard error is no longer redirected by default, making it more consistent with UNIX shells and the original SCSH.  Thanks to Haochi Kiang for pointing this out and providing a patch.
424* 0.4.1 - Allow the use of unquote-splicing in run macro forms (thanks to [[/users/moritz-heidkamp|Moritz Heidkamp]] for pointing this out)
425* 0.4 - Support {{continue-threads}} parameter for {{fork}} on newer Chickens which support {{kill-all-threads}} option in {{process-fork}}.  This should make this library safe for use in threaded programs.
426* 0.3.1 - Change {{wait}} result values to all be {{#f}} if {{nohang}} is {{#t}}.
427* 0.3 - Fix segfault properly, revert back to using standard-extension.  Add {{wait}} and do not accept scsh-process objects in POSIX {{process-wait}}.
428* 0.2.1 - Workaround for segmentation fault caused by compiled version of scsh-process in 4.8.0 due to -O3 being the default compilation of a standard-extension.
429* 0.2 - Fix {{<<}}-redirection and increase robustness of test for it.
430* 0.1.2 - Fix setup-file to have non-bogus version identifier.  Don't rely on "bc" being present in the tests.
431* 0.1.1 - Fix order of values returned by {{process-wait}} to be consistent with POSIX unit.
432* 0.1 - Initial release, created at the [[/event/chicken-uk-2012|Chicken UK 2012]] hacking event.
433
434=== Author
435
436[[/users/peter-bex|Peter Bex]]
437
438=== Repository
439
440[[http://code.more-magic.net/scsh-process]]
441
442=== License
443
444  Copyright (c) 2012-2018, Peter Bex
445  All rights reserved.
446 
447  Redistribution and use in source and binary forms, with or without
448  modification, are permitted provided that the following conditions
449  are met:
450  1. Redistributions of source code must retain the above copyright
451     notice, this list of conditions and the following disclaimer.
452  2. Redistributions in binary form must reproduce the above copyright
453     notice, this list of conditions and the following disclaimer in the
454     documentation and/or other materials provided with the distribution.
455 
456  THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR
457  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
458  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
459  IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT,
460  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
461  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
462  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
463  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
464  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
465  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Note: See TracBrowser for help on using the repository browser.