source: project/wiki/eggref/4/scsh-process @ 33252

Last change on this file since 33252 was 33252, checked in by sjamaan, 5 years ago

Update scsh-process repo URL

File size: 18.3 KB
Line 
1[[tags: egg]]
2
3== SCSH-Process
4
5[[toc:]]
6
7=== Description
8
9A reimplementation for Chicken of SCSH's [[http://www.scsh.net/docu/html/man-Z-H-3.html|process notation]].
10
11=== Requirements
12
13none
14
15=== Documentation
16
17This egg implements all the special forms required to implement SCSH's
18convenient process notation.  This notation is called "EPF", which
19stands for ''extended process form''.
20
21The procedural equivalents for these special forms are also available.
22Where it's required or makes sense, a handful of other procedures are
23also provided.  This egg does '''not''' strive for full SCSH
24compatibility; it exists only to bring the convenience of EPF notation
25to Chicken.
26
27==== Caveats
28
29There are a few important caveats, especially related to threading.
30Please read this section thoroughly before using scsh-process!  If
31none of this makes sense yet, read the rest of the manual first, then
32come back.  The issues are listed in order of decreasing importance.
33
34===== Forking and threads
35
36In versions of Chicken before 4.8.1 (rev 47b5be71), there is no
37support for killing threads upon forking.  This means that other
38threads will always continue running whenever you fork.  Please note
39that all of the EPF forms implicitly fork for each process in a
40pipeline.  If you intend to use {{scsh-process}} in a program which
41uses threads, please use a Chicken which supports this, otherwise you
42''will'' get unpredictable behavior.
43
44If you are using threads at all, it is important to realize that
45thread killing is completely insensitive to your delicate
46synchronization primitives or resource management.  Ensure that all
47locks are released and you don't have any non-GCable memory allocated
48in other threads.  If you have finalizers registered, those will still
49run, though.
50
51===== Signal handling
52
53Beware that if you set a {{signal/chld}} handler, you will need to
54remember the original handler and call it from your handler.  If you
55don't, you must manually wait for all children.  If you've installed a
56handler before scsh-process is loaded, it will be automatically
57chained by the handler installed by scsh-process.
58
59===== Process reaping
60
61Loading scsh-process will transparently cause the posix {{process-wait}}
62procedure to be replaced so it can update the bookkeeping information
63about reaped child processes.  See the description in the SCSH manual
64about how SCSH performs
65[[http://www.scsh.net/docu/html/man-Z-H-4.html#node_sec_3.4.1|process reaping]].
66Currently only the "early" reaping strategy is supported.
67
68==== Macros
69
70The special forms provided by this egg are the most convenient to use,
71but also slightly less powerful than the underlying procedures they
72use.
73
74===== Basic process macros
75
76<macro>(run pf [redirection ...])</macro>
77<macro>(& pf [redirection ...])</macro>
78<macro>(exec-epf pf [redirection ...])</macro>
79
80These forms run a new process pipeline, described by process form
81{{pf}}, with optional redirections of input and output descriptors
82indicated by any number of {{redirection}} patterns.  These processes
83don't interact with the calling process; if you want to capture their
84output there are other forms, like {{run/string}}, {{run/port}} and
85friends.  Please also beware that these forms don't do anything to
86indicate nonzero exit statuses in pipelines.  The {{&&}} and {{||}}
87macros might help if you need to do status checks.
88
89The {{run}} form simply runs the process and waits for the final
90process in the pipeline to exit.  It returns three values relating to
91this final process:
92
93* Either its exit status as an integer if the process terminated normally, or the signal number that terminated/stopped the process.
94* #t if the process exited normally, #f if it terminated abnormally.
95* Its process id (an integer).
96
97You'll note that these are just the same values returned by
98{{process-wait}}, but in a slightly different order ({{wait}} utilises
99this modified order, as well).  This is for compatibility reasons:
100In SCSH this form returns only the exit status, but because Chicken
101accepts multiple values in single value contexts (discarding all
102but the first), we can provide the other ones as extra values, thereby
103avoiding gratuitous incompatibility.
104
105The {{&}} form simply runs the process in the background and returns
106one value: a process object representing the last process in the
107pipeline.
108
109The {{exec-epf}} form never returns; it replaces the current process
110by the last process in the pipeline.  The others are implemented in
111terms of this form:
112
113<enscript highlight="scheme">
114(& . epf) => (process-fork (lambda () (exec-epf . epf)) #t)
115(run . epf) => (process-wait (& . epf))
116</enscript>
117
118A process form followed by a set of redirections is called "extended
119process form" in SCSH terminology.  A process form can be one of
120the following:
121
122<enscript highlight="scheme">
123(<program> <arg> ...)
124(pipe <pf> ...)
125(pipe+ <connect-list> <pf> ...)
126(epf <pf> <redirection> ...)
127(begin <s-expr> ...)
128</enscript>
129
130The arguments to a {{<program>}} rule are implicitly quasiquoted.  The
131basic building blocks are {{<program>}} and {{<begin>}}, which always
132correspond to one process in a pipeline.  The other rules are ways to
133combine these two.
134
135The {{pipe}} rule will hook up standard output and standard error of
136each {{pf}} to the next {{pf}}'s standard input, just like in a
137regular Unix shell pipeline.
138
139The {{pipe+}} rule is like pipe, but it allows you to hook up
140arbitrary file descriptors between two neighbouring processes.  This
141is done through the {{connect-list}}, which is a list of fd-mappings
142describing how ports are connected from one process to the next.  It
143has the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  The
144{{from-fd}}s correspond to outbound file descriptors in one process,
145the {{to-fd}}s correspond to inbound file descriptors in the other
146process.
147
148The {{epf}} rule is to get extended process forms in contexts where
149only process forms are accepted, like the {{pipe}} and {{pipe+}}
150subforms, and in the {{&&}} and {{||}} macros (so you can do file
151redirects here).
152
153The {{begin}} rule allows you to write scheme code which will be run
154in a forked process, having its {{current-input-port}},
155{{current-output-port}} and {{current-error-port}} hooked up to its
156neighbouring processes in the pipeline.
157
158A redirection can be one of the following:
159
160<enscript highlight="scheme">
161(> [<fd>] <file-name>)      ; Write fd (default: 1) to the given filename
162(>> [<fd>] <file-name>)     ; Like >, but append instead of overwriting
163(< [<fd>] <file-name>)      ; Read fd (default: 0) from the filename
164(<< [<fd>] <scheme-object>) ; Like <, but use object's printed representation
165(= <fd> <fd-or-port>)       ; Redirect fd to fd-or-port
166(- <fd-or-port>)            ; Close fd
167stdports                    ; Duplicate fd 0, 1, 2 from standard Scheme ports
168</enscript>
169
170The arguments to redirection rules are also implicitly quasiquoted.
171
172To tie it all together, here are a few examples:
173
174<enscript highlight="scheme">
175(use scsh-process)
176
177;; Writes "1235" to a file called "out" in the current directory.
178;; Shell equivalent:  echo 1234 + 1 | bc > out
179(run (pipe (echo "1234" + 1) ("bc")) (> out))
180
181(define message "hello, world")
182
183;; Writes 13 to stdout, with a forked Scheme process writing the data.
184;; Shell equivalent (sort of): echo 'hello, world' | wc -c
185(run (pipe (begin (display message) (newline)) (wc -c)))
186
187;; A verbose way of doing the same, using pipe+.  It connects the {{begin}}
188;; form's standard output and standard error to standard input of {{wc}}:
189(run (pipe+ ((1 2 0)) (begin (display message) (newline)) (wc -c)))
190
191;; Same as above, using redirection instead of port writing:
192(run (wc -c) (<< ,(string-append message "\n")))
193
194;; Writes nothing because stdout is closed:
195(run (wc -c) (<< ,message) (- 1))
196
197;; A complex toy example using nested pipes, with input/output redirection.
198;; Closest shell equivalent:
199;; ((sh -c "echo foo >&2") 2>&1 | cat) | cat
200(run (pipe+ ((1 0))
201            (pipe+ ((2 0)) (sh -c "echo foo >&2") (cat))
202            (cat)))
203</enscript>
204
205===== Process macros for interfacing with Scheme
206
207<macro>(run/port pf [redirection ...])</macro>
208<macro>(run/file pf [redirection ...])</macro>
209<macro>(run/string pf [redirection ...])</macro>
210<macro>(run/strings pf [redirection ...])</macro>
211<macro>(run/sexp pf [redirection ...])</macro>
212<macro>(run/sexps pf [redirection ...])</macro>
213
214These forms are equivalent to {{run}}, except they wire up the current
215process to the endpoint of the pipeline, allowing you to read the
216''standard output'' from the pipeline as a whole.  If you also need
217standard error or have even more specialized needs, take a look at
218the {{run/collecting}} form.
219
220The difference between these forms is in how this output is returned,
221and when the call returns:
222
223* {{run/port}} immediately returns after forking, and returns a port from which you can read.
224* {{run/file}} returns after the final process exits, resulting in a string which indicates a temporary file containing the process' output.
225* {{run/string}} returns when the process closes its standard output (ie, when EOF is read), collecting the standard output into a string.
226* {{run/strings}} is like {{run/string}}, but returns a list of strings, split at newline characters.
227* {{run/sexp}} reads an s-expression, and returns as soon as a complete s-expression was read.
228* {{run/sexp}} reads all s-expressions until eof, returning a list of s-expressions.  It returns as soon as EOF is read.
229
230===== Macros for conditional process control
231
232<macro>(&& pf ...)</macro>
233<macro>(|| pf ...)</macro>
234
235These macros act like their counterpart shell operators; they run the
236given process forms in sequence and stop either on the first "false"
237value (nonzero exit) or on the first "true" value (zero exit), respectively.
238
239The result value of these is {{#f}} or {{#t}}, so they act a lot like
240regular Scheme {{and}} and {{or}}.
241
242'''Note:''' The name of the {{||}} macro is really the empty symbol
243whereas in SCSH's reader, it reads as a symbol consisting of two pipe
244characters.  The name of these macros may change in the future if it
245turns out to cause too much trouble.
246
247===== Collecting multiple outputs
248
249<macro>(run/collecting fds pf ...)</macro>
250
251This form runs the {{pf}} form, redirecting each the file descriptors
252in the {{fds}} list to a separate tempfile, and waits for the process
253to complete.
254
255The result of this expression is {{(status file ...)}}.  {{status}} is
256the exit status of the process.  Each {{file}} entry is an opened
257input port for the temporary file that belongs to the file descriptor
258at the same offset in the {{fds}} list.  If you close the port, the
259tempfile is removed.
260
261See the [[http://www.scsh.net/docu/html/man-Z-H-3.html#node_sec_2.4.2|SCSH documentation]]
262for an extended rationale of why this works the way it does.
263
264==== Procedural interface
265
266These procedures form the basis for the special forms documented
267above, and can be used to implement your own, more specialized macros.
268
269===== Basic forking and pipeline primitives
270
271<procedure>(fork [thunk [continue-threads?]])</procedure>
272<procedure>(%fork [thunk [continue-threads?]])</procedure>
273
274If {{thunk}} is provided and not {{#f}}, the child process will invoke
275the thunk and exit when it returns.  If {{continue-threads}} is provided
276and {{#t}}, all existing threads will be kept alive in the child process;
277by default only the current thread will be kept alive.
278
279{{fork}} differs from the regular {{process-fork}} in its return value.
280Instead of a pid value, this returns a process object representing the
281child is returned in the parent process.  When {{thunk}} is not
282provided, {{#f}} (not zero!) is returned in the child process.
283
284Currently {{%fork/pipe}} is just an alias for {{fork/pipe}}.
285
286<procedure>(wait [pid-or-process [nohang]])</procedure>
287
288Like {{process-wait}}: Suspends the current process until the child
289process described by {{pid-or-process}} (either a numerical process ID
290or a scsh-process object) has terminated using the UNIX system call
291waitpid(). If {{pid-or-process}} is not given, then this procedure
292waits for any child process. If {{nohang}} is given and not {{#f}}
293then the current process is not suspended.
294
295This procedure returns three values, '''in a different order from
296{{process-wait}}''':
297
298* Either the exit status, if the process terminated normally or the signal number that terminated/stopped the process.
299* #t if the process exited normally or #f otherwise.
300* {{pid}} or 0
301
302All values are {{#f}}, if {{nohang}} is true and the child process has
303not terminated yet.
304
305It is allowed to wait multiple times for the same process after it has
306completed, if you pass a process object.  Process IDs can only be
307waited for once after they have completed and will cause an error
308otherwise.
309
310<procedure>(process? object)</procedure>
311
312Is {{object}} an object representing a process?
313
314<procedure>(fork/pipe [thunk [continue-threads?]])</procedure>
315<procedure>(%fork/pipe [thunk [continue-threads?]])</procedure>
316
317These fork the process as per {{fork}} or {{%fork}}, but additionally
318they set up a pipe between parent and child.  The child's standard
319output and stdandard error are set up to write to the pipe, while the
320parent's standard input is set up to read to the pipe.
321
322The return value is a process object or {{#f}}.
323
324Currently {{fork%/pipe}} is just an alias for {{fork/pipe}}.
325
326'''Important''': These procedures only set up the file descriptors,
327not the Scheme ports.  {{current-input-port}}, {{current-output-port}}
328and {{current-error-port}} still refer to their old file descriptors
329after a fork.  This means that you'll need to reopen the descriptors
330to get a Scheme port that reads from the child or writes to the parent:
331
332<enscript highlight="scheme">
333(use scsh-process posix)
334
335(process-wait
336  (fork/pipe (lambda ()
337               (with-output-to-port (open-output-file* 1)
338                 (lambda () (display "Hello, world.\n"))))))
339
340(read-line (open-input-file* 0)) => "Hello, world"
341</enscript>
342
343<procedure>(fork/pipe+ conns [thunk [continue-threads?]])</procedure>
344<procedure>(fork%/pipe+ conns [thunk [continue-threads?]])</procedure>
345
346These are like {{fork/pipe}} and {{fork%/pipe}}, except they allow you
347to control how the file descriptors are wired.  Conns is a list of
348lists, of the form {{((from-fd1 from-fd2 ... to-fd) ...)}}.  See the
349description of {{pipe+}} under the {{run}} special form for more
350information.
351
352Currently {{fork%/pipe+}} is just an alias for {{fork/pipe+}}.
353
354===== Executing programs from the path
355
356<procedure>(exec-path program [args ...])</procedure>
357<procedure>(exec-path* program arg-list)</procedure>
358
359This will simply execute {{program}} with the given arguments in
360{{args}} or {{args-list}}.  The program replaces the currently running
361process.  All arguments (including {{program}}) must be strings,
362symbols or numbers, and they will automatically be converted to
363strings before invoking the program.
364
365The program is looked up in the user's path, which is simply the
366{{$PATH}} environment variable.  There currently is no separately
367maintained path list like in SCSH.  The difference between the two
368procedures is that {{exec-path}} accepts a variable number of
369arguments and {{exec-path*}} requires them pre-collected into a list.
370
371===== Pipeline procedures
372
373<procedure>(run/port* thunk)</procedure>
374<procedure>(run/file* thunk)</procedure>
375<procedure>(run/string* thunk)</procedure>
376<procedure>(run/strings* thunk)</procedure>
377<procedure>(run/sexp* thunk)</procedure>
378<procedure>(run/sexps* thunk)</procedure>
379
380These set up a pipe between the current process and a forked off child
381process which runs the {{thunk}}.
382See the "unstarred" versions {{run/port}}, {{run/file}} ... {{run/sexps}}
383for more information about the semantics of these procedures.
384
385===== Collecting multiple outputs
386
387<procedure>(run/collecting* fds thunk)</procedure>
388
389Like {{run/collecting}}, but use a {{thunk}} instead of a process form.
390
391=== Changelog
392
393* 0.5 - Standard error is no longer redirected by default, making it more consistent with UNIX shells and the original SCSH.  Thanks to Haochi Kiang for pointing this out and providing a patch.
394* 0.4.1 - Allow the use of unquote-splicing in run macro forms (thanks to [[/users/moritz-heidkamp|Moritz Heidkamp]] for pointing this out)
395* 0.4 - Support {{continue-threads}} parameter for {{fork}} on newer Chickens which support {{kill-all-threads}} option in {{process-fork}}.  This should make this library safe for use in threaded programs.
396* 0.3.1 - Change {{wait}} result values to all be {{#f}} if {{nohang}} is {{#t}}.
397* 0.3 - Fix segfault properly, revert back to using standard-extension.  Add {{wait}} and do not accept scsh-process objects in POSIX {{process-wait}}.
398* 0.2.1 - Workaround for segmentation fault caused by compiled version of scsh-process in 4.8.0 due to -O3 being the default compilation of a standard-extension.
399* 0.2 - Fix {{<<}}-redirection and increase robustness of test for it.
400* 0.1.2 - Fix setup-file to have non-bogus version identifier.  Don't rely on "bc" being present in the tests.
401* 0.1.1 - Fix order of values returned by {{process-wait}} to be consistent with POSIX unit.
402* 0.1 - Initial release, created at the [[/event/chicken-uk-2012|Chicken UK 2012]] hacking event.
403
404=== Author
405
406[[/users/peter-bex|Peter Bex]]
407
408=== Repository
409
410[[http://code.more-magic.net/scsh-process]]
411
412=== License
413
414  Copyright (c) 2012-2015, Peter Bex
415  All rights reserved.
416 
417  Redistribution and use in source and binary forms, with or without
418  modification, are permitted provided that the following conditions
419  are met:
420  1. Redistributions of source code must retain the above copyright
421     notice, this list of conditions and the following disclaimer.
422  2. Redistributions in binary form must reproduce the above copyright
423     notice, this list of conditions and the following disclaimer in the
424     documentation and/or other materials provided with the distribution.
425 
426  THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR
427  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
428  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
429  IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT,
430  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
431  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
432  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
433  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
434  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
435  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Note: See TracBrowser for help on using the repository browser.