source: project/gazette/src/issues/21.wiki @ 25546

Last change on this file since 25546 was 25546, checked in by andyjpb, 9 years ago

Copyediting for gazette #21

File size: 43.0 KB
Line 
1((title . "Issue 21")
2 (authors "Andy Bennett")
3 (date . 1321465864))
4
5== 0. Introduction
6
7Welcome to the 21st issue of the Chicken Gazette. While the gazette
8was on hiatus the chicken community was definitely not.
9
10With the stabilisation of Chicken 4.7 well underway since the last
11gazette...
12
13Some people have also started to set up salmonella farms and C-Keen
14has completed his "Chicken Zoo".
15
16Enjoy!
17
18
19== 1. Chickens Out & About
20
21=== Nuremberg (December 2011)
22[[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] wrote
23([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00114.html|2011/10/22 17:38Z]]) to
24confirm the planned Chicken Hacking Sprint scheduled for the beginning of December.
25
26Interested and available Chickenauts will meet in Nuremberg on the weekend of 9th, 10th and 11th
27of December. If you haven't done so already, you can let us know that you will be joining us by
28placing a note at the foot of the [[https://wiki.call-cc.org/chicken-sprint-nuremberg|wiki page]].
29If you can't attend in person then we'll be keeping an eye on IRC so feel free to book yourself
30a slot in your cave at home and join us throughout the weekend and into the evenings, Central European Time.
31
32For those of you keeping score, this will bring the total number of Chicken meets this year to
335 (FOSDEM, Cologne, FrOSCon, T-DOSE and Nuremberg).
34
35
36=== Cologne (March 2011)
37
38In March, Chickens met up at the Chaos Computer Club in Cologne, Germany for a Chicken Hacking Weekend. Here's
39[[http://wiki.call-cc.org/users/moritz-heidkamp|Moritz Heidkamp]]'s report on what they got up to.
40
41TODO: INCLUDE MORIT'Z REPORT
42
43
44=== FrOSCon (August 2011)
45At FrOSCon in August we sported not only a project booth but a presentation room as well so we took the
46opportunity to lay on a couple of talks for the delegates.
47
48[[http://wiki.call-cc.org/users/moritz-heidkamp|Moritz Heidkamp]] presented
49"[[http://programm.froscon.org/2011/events/869.html|An introduction to Lisp]]:
50Why to talk to computers in parentheses" whilst
51[[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] offered
52"[[http://programm.froscon.org/2011/events/867.html|A guided tour through the republic of CHICKEN]]:
53get up to speed with the practical scheme implementation".
54[[http://wiki.call-cc.org/users/christian-kellermann|Christian]] has been keeping his talk up-to-date and
55you can find the latest version [[http://wiki.call-cc.org/chickenista-guide|on the wiki]].
56
57Naturally, there are [[http://www.call-cc.org/pictures/froscon2011/|pictures]].
58
59
60=== T-DOSE 2011 (November 2011)
61European Chickens met up at T-DOSE in Eindhoven, The Netherlands on the weekend of the 5th, 6th and 7th of
62November. [[http://wiki.call-cc.org/users/peter-bex|Peter Bex]] arranged for us to contribute a "Chicken
63Scheme Project Booth" and we subsequently lured a few Dutch hackers into #chicken on IRC.
64
65[[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] knocked up some impressive demos
66of [[http://en.wikipedia.org/wiki/Pong|PONG]] using Chicken's [[http://wiki.call-cc.org/eggref/4/cairo|Cairo Egg]].
67
68[[http://wiki.call-cc.org/users/alaric-blagrave-snellpym|Alaric Snell-Pym]] finished off the tests for his
69[[http://wiki.call-cc.org/eggref/4/ugarit|Ugarit]] backup egg which allowed him to release version 1.0.
70
71...and [[http://wiki.call-cc.org/users/moritz-heidkamp|Moritz Heidkamp]] started on a replacement for the
72[[http://wiki.call-cc.org/eggref/4/environments|Environments Egg]] as it has unfortunately has been overtaken
73by developments in the Chicken development branch.
74
75Pictures are up in the [[http://www.call-cc.org/pictures/t-dose2011/|usual place]] on the Chicken website.
76
77
78
79== 2. Chicken Talk
80
81=== R7RS to be dedicated to the memory of John McCarthy
82
83Matt Welland wrote in with the sad news that the "Father of Lisp", John McCarthy had sadly passed away ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00115.html|2011/10/25 15:59Z]]). [[http://wiki.call-cc.org/users/john-cowan|John Cowan]], member of the R7RS working group, told us that R7RS,
84the upcoming version of the Scheme standard, would be dedicated to his memory ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00116.html|16:22Z]])
85
86
87=== Parallel build patch for Chicken Makefile
88
89A lively discussion broke out on the [[https://wiki.call-cc.org/mailing-lists|Chicken Users mailing list]] regarding
90[[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00025.html|parallelisation of the Chicken build process]].
91
92Vitaly Magerya supplied a [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00025.html|patch]] (2011/10/06 12:37Z) to fix some problems
93he'd been having with builds failing when specifying the {{-j}} to {{gmake}}.
94[[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario Domenech Goulart]] noted that this functionality was already being
95addressed by [[http://bugs.call-cc.org/ticket/526|ticket 526]]
96([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00027.html|2011/10/06 12:39Z]]) and Vitaly was keen to see it committed
97soonly as he maintains the FreeBSD port and people had been asking him for parallel builds ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00028.html|2011/10/06 14:24Z]]). Whilst still keen, [[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00029.html|advised caution]] as he wanted to see it proven ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00031.html|2011/10/06 14:40Z]]).
98
99Vitaly was keen to push things forward ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00033.html|2011/10/06 15:45Z]]) and
100offered continued testing under FreeBSD 8.2-RELEASE on x86_64. He noted that, as far as he could tell, {{gmake check}} passed when using his
101patch, modified with [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00026.html|Moritz Wilhelmy's suggestions]]. Moreover, he
102offered to try the patch from [[http://bugs.call-cc.org/ticket/526|ticket 526]] as well.
103
104[[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00034.html|suggested]]
105that using the [[http://bugs.call-cc.org/ticket/526|ticket 526]] patch was a good idea as it had, as far as he knew, received testing under Linux and MacOSX.
106
107Toby Thain spoke in favour of the patch, saying that a working parallel build gives confidence that the Makefile is, in fact, correct.
108He offered to test on three platforms, namely Solaris 10 SPARC SMP and both PowerPC and Intel version of MacOSX SMP.
109
110Vitaly updated the [[http://bugs.call-cc.org/ticket/526|ticket 526]] patch to the then current sources and confirmed that the build worked at {{-j8}}
111and the tests passed in his FreeBSD 8.2-RELEASE, x86_64 environment. He did, however, note that parallelisation does not work for the install phase
112([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00048.html|2011/10/08 12:29Z]]).
113
114Toby Thain reported successful builds on the 64-bit version MacOSX-10.6 as well as MacOSX-10.5 on a dual-processor G5. He noted "{{make -j8}}
115reduces the make step from 3:39 to 0:47 on my 8-core system". Unfortunately {{make check}} did not pass on either system, with or without the patch
116([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00071.html|2011/10/12 01:59Z]] &
117[[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00100.html|2011/10/15 19:53Z]])
118
119[[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] noted that it might necessary to install the compiled binaries
120([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00101.html|2011/10/15 20:07Z]]) however, Toby did not report back as to whether this
121had any effect on the results of the tests.
122
123
124=== 64-bit SPARC Build
125
126Nicolas Pelletier came to the [[https://wiki.call-cc.org/mailing-lists|mailing list]]
127unable to build for Solaris 10 running on SPARC64 due to a missing {{apply-hack.S}}
128([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00012.html|2011/11/09 11:48Z]]).
129
130It turned out that there was no SPARC64 support for Chicken at all and [[http://wiki.call-cc.org/users/john-cowan|John Cowan]]
131came to the rescue ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00013.html|15:36Z]])
132with instructions for how to add it to the supported architectures list and disable the
133apply hack. [[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] chimed in with affirmations
134around the utility of using Chicken without an apply hack ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00014.html|15:54Z]])
135
136Nicolas struggled with disabling the apply hack ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00015.html|2011/11/10 11:10Z]])
137due to the Two Lies Of The README. Over the course of the next few minutes [[http://wiki.call-cc.org/users/christian-kellermann|Christian]]
138and [[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario Domenech Goulart]]]
139confirmed that the lies, which related to the way bootstrap compilers are distributed and invoked, had
140already been fixed in the development versions.
141
142Meanwhile, Toby Thain tried the build on his SPARC64 ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00020.html|12:40Z]])
143and discovered that GNU make 3.82 is required as well as the addition of {{-lrt}} in {{Makefile.solaris}}. Toby's
144build eventually succeeded and appeared to be good, albeit having only used 1 of his 4 CPUs and with a couple of {{make check}} errors.
145
146Later on the Thursday, [[http://wiki.call-cc.org/users/jim-ursetto|Jim Ursetto]] noted ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00021.html|16:47Z]])
147that the [[http://bugs.call-cc.org/raw-attachment/ticket/526/parallel-build-2.diff.txt|parallel build patch]] in
148[[http://bugs.call-cc.org/ticket/526|ticket 526]] still works on Chicken 4.7.0.3-st. Toby [[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00022.html|confirmed]]
149that the patch did indeed work on Solaris and build time was reduced to 10 just of his Earth minutes from the original 40.
150Of course, user time across the four CPUs was still in the region of 40 minutes in total. :-)
151
152To round everything off, Toby offered [[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00028.html|binary SYSV packages for Solaris 10 on SPARC64]].
153Contact him via the [[https://wiki.call-cc.org/mailing-lists|mailing list]] if you're interested.
154
155
156=== Static Linking & Standalone Binaries.
157
158Serg Kozhemyakin [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00105.html|asked]] whether it is possible to build static
159binaries after having trouble using the {{-static}} flag to {{csc}}. He also wondered about building standalone binaries for Windows.
160[[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00106.html|advised]]
161against static linking as many of the eggs cannot cope with it. He did, however, suggest that the {{-deploy}} option may well solve both
162issues as it ensures that all the DLLs (including the Chicken runtime and other shared objects) are bundled together into a single directory.
163
164
165=== Tracing with {{,tr}} moved to an egg
166
167[[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario Domenech Goulart]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00007.html|helped]]
168Curtis Cooley in his [[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00006.html|search for the {{,tr}} command]].
169The functionality had been moved from the core into an egg in Chicken 4.3.0.
170
171
172=== {{gmake check}} sometimes fails.
173
174Pekka Niiranen wrote to tell us about
175[[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00009.html|problems with {{gmake check}} in Chicken 4.7.0-st]]
176on OpenBSD 5.0.
177
178[[http://wiki.call-cc.org/users/jim-ursetto|Jim Ursetto]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00010.html|said]]
179that one of the failures had been fixed in the stability branch after the Chicken 4.7.0-st release and he filed
180[[https://bugs.call-cc.org/ticket/724|ticket 724]] for the segmentation fault issue.
181
182[[http://wiki.call-cc.org/users/peter-bex|Peter Bex]] mentioned that the [[https://bugs.call-cc.org/ticket/724|ticket 724]] segmentation
183fault had also been observed under NetBSD and Arch Linux. He said that "It's known to be broken" but that it is not properly reproducible
184as it only happens some of the time ([[http://lists.nongnu.org/archive/html/chicken-users/2011-11/msg00011.html|2011/11/06 09:51|]]).
185
186
187== 3. Salmonella and other tests
188Salmonella is Chicken's egg testing framework.
189
190For some time we have had daily salmonella runs of the master development
191version of Chicken under Linux on x86. Throughout November, MacOSX-x86_64 has also
192been regularly experiencing the runs. [[http://wiki.call-cc.org/users/mario-domenech-goulart|Mario Domenech Goulart]] who maintains the testing
193effort at [[http://tests.call-cc.org/]], as well as the [[https://github.com/mario-goulart/salmonella|Salmonella codebase]]
194itself, is always on the lookout for more feeds. We are aware that testing on
195other platforms is being organised but if you've got a private Salmonella
196running regularly on an interesting operating system or version of Chicken then
197give us a shout and we'll see if we can get it included with the other reports
198on [[http://tests.call-cc.org/]]
199
200There is also [[https://github.com/mario-goulart/salmonella|an entirely new version of Salmonella]] over at github so test that too. The new version is more modular -- some features that were previously built in salmonella are now independent eggs (e.g., [[https://github.com/mario-goulart/salmonella-html-report|salmonella-html-report]] and [[https://github.com/mario-goulart/salmonella-feeds|salmonella-feeds]].  There's also a new egg that provides a new feature: [[https://github.com/mario-goulart/salmonella-diff|salmonella-diff]] which generates HTML output to render differences between two salmonella logs.
201
202Here's short list of new salmonella features:
203
204* egg lint mode: salmonella can now easily check for common egg mistakes before the code hits the egg repository.  Just run {{salmonella --this-egg}} from the egg directory
205* a new execution mode that can significantly reduce salmonella execution times.  In this mode, salmonella won't set the temporary egg installation directory empty before installing new eggs.  While this mode won't spot egg dependencies problems, it can drastically effect the execution time
206* a new tool to take advantage of multi-core systems ({{salmonella-epidemy}})
207* a simple text mode log file viewer ({{salmonella-log-viewer}})
208* [[https://github.com/mario-goulart/salmonella-html-report|salmonella-html-report]] generates reverse dependencies graphs.  That can be useful, for
209example, to ilustrate how many eggs depend on a given egg.
210
211
212[[http://wiki.call-cc.org/users/alaric-blagrave-snellpym|Alaric Snell-Pym]] is one of the people who has been setting up a Chicken testing
213environment. After [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00068.html|some discussion]], he chose Chicken 4.5.0 under
214NetBSD ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00035.html|2011/10/06 21:18Z]]). He's been keeping track of all the
215Egg dependencies and plans to publish them as a list of NetBSD pkgsrc packages.
216
217Thanks to [[http://www.bevuta.com/|bevuta IT GmbH]], the CHICKEN test
218infrastructure will soon have more machines available to test the
219CHICKEN compiler, tools and eggs.  A new salmonella environment for
220Linux/x86-64 is currently being set up.
221
222In addition to salmonella, work on the
223[[http://wiki.call-cc.org/playground|Chicken Playground]] project has
224been resumed.  The Chicken Playground project provides environments
225that can be used by tools like {{chroot}} to test CHICKEN tools and
226eggs.  Those environments contain tools and libraries required to
227build and test most eggs.  Currently a Linux/x86 environment is
228available. A Linux/x86-64 environment is under development and should
229be available soon.
230
231The new test infrastructure, based on new versions of salmonella and
232Chicken Playground and running on new platforms will probably go into
233production during the Nuremberg hackaton.
234
235
236== 4. Core Development
237
238TODO: INCLUDE CHRISTIAN'S REPORT.
239
240I need some blurb on this from people Who Know.
241I'd like something on the scrutinizer and stability work that's been going on as well as the proposal to bug fix until r7rs
242and then do Chicken 5.0.
243
244http://wiki.call-cc.org/development-process
245
246sparc64
247mario, felix, ckeen
248lots of bug fixes, sparc64 support, scrutinizer fixes
249
250 * On origin/master:
251   * (e1a5437): when conditional branches differ in the number of results, do not merge the results (felix, 2 days ago)
252   * (bae7d92): clustering optimization added (felix, 5 weeks ago)
253   * (b1907c5): use different default stack-sizes on 32/64 bit platforms (felix, 6 days ago)
254   * (986bd3f): disable apply-hack on sparc64 (felix, 6 days ago)
255   * (28a36bf): Remove scripts/make-egg-rss-feed.scm (imported into the chicken-infrastructure repo) (Mario Domenech Goulart, 5 days ago)
256   * (fbeffd1): scripts/tools.scm: needs regex for `string-match' (Mario Domenech Goulart, 5 days ago)
257   * (1fca9c6): use equal? in case MSYSTEM is not defined (makedist.scm) (felix, 6 days ago)
258   * (42a8263): don't remove build-info files on make confclean or it is not possible to build from a tarball without preinstalled chicken (felix, 9 days ago)
259   * (3bb9645): write debugging output for optimizations into generated source file (felix, 3 weeks ago)
260   * (e7fe2fb): allow procedure in operator-position of evaluated list (#705, suggested by John Cowan) (felix, 2 weeks ago)
261   * (e2fb31c): fix subtype check for fixed-size list/vector types in scrutinizer (felix, 10 days ago)
262   * (fd018df): strip std-envs of non-std identifiers, ignore ##sys#macro-environment when evaluating with an explicit environment (felix, 13 days ago)
263   * (fe8f63c): minor change in port-tests.scm to handle case when HOME is not defined (Windows) (felix, 2 weeks ago)
264   * (723a469): cut of union types above a certain length (felix, 2 weeks ago)
265   * (919a7c7): fixes in build + Windows test script, added note to README (thanks to mario) (felix, 2 weeks ago)
266   * (ddb96f9): Also added runtests.bat to manifest. (felix, 3 weeks ago)
267   * (0df9bb8): another trivial fix for making boot-chicken work on cmd.exe (felix, 3 weeks ago)
268   * (cea668f): trivial Windows-related corrections in rules.make, found by Mario - not yet working, though (felix, 3 weeks ago)
269   * (1bc311d): undocumented scrutinize declaration (obsolete), reported by ckeen (felix, 3 weeks ago)
270   * (10494f6): fixed stupid paren-matching error (felix, 3 weeks ago)
271   * (f9a6702): disable debug output in scrutinizer (felix, 5 weeks ago)
272   * (2483f8d): check not necessary for lambda and  warn if using keyword as variable in binding form (felix, 3 weeks ago)
273   * (aa6b2bb): cleaning up uninitialized variables in C_reclaim, add debug output to C_mutate on m-stack resize, grow m-stack at higher rate (felix, 3 weeks ago)
274   * (1e04156): use write-string port-class method (felix, 3 weeks ago)
275   * (59cdaaf): remove obsolete C_h_... allocation functions (pointed out by Joerg Wittenberger) (felix, 3 weeks ago)
276   * (159611d): handle macro-aliases when resolving declared identifier, making sure things are done right inside modules. (felix, 4 weeks ago)
277   * (0cc88a4): removed obsolete file (felix, 4 weeks ago)
278   * (1f56f3a): ##core#type makes subtype-check optional; quit compile when type-mismatches in strict mode (felix, 5 weeks ago)
279
280
281== 5. Stability is for Chickens
282
283[[http://wiki.call-cc.org/users/jim-ursetto|Jim Ursetto]] has set up a Chicken Stability Branch
284effort over at [[http://wiki.call-cc.org/stability]]. Jim describes it as "a fork of the Chicken
285Scheme core which backports certain useful bugfixes and minor features from the future to 'stable'
286maintenance versions". Releases for stability/4.7.0 have version numbers like {{4.7.0.x-st}} (for
287example, {{4.7.0.1-st}}). If you're interested in running a stable Chicken in production, we recommend
288that you check out Jim's work.
289
290
291
292== 6. Hatching Farm
293
294There has been a lot of egg activity over the course of the past few months. I've selected a few,
295focussing on ones that were announced or discussed on the [[https://wiki.call-cc.org/mailing-lists|mailing list]].
296
297TODO: Write about the distributed egg repository
298
299By far the most notable thing to have happened is the announcment and implementation of the new
300Egg management system.
301
302* [[http://wiki.call-cc.org/users/ivan-raikov|Ivan Raikov]]'s [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00058.html|proposal]]
303  to recategorise eggs filed under "Uncategorized or invalid" and "Miscellaneous" was met with enthusiasm with a number of people pitching in to
304  help as well as [[http://wiki.call-cc.org/users/peter-bex|Peter Bex]] suggesting that the category list be revisited
305  ([[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00064.html|2011/10/11 08:12Z]]). [[http://wiki.call-cc.org/users/peter-bex|Peter]]
306  specifically suggested the addition of a "Communication"s category.
307
308* [[http://wiki.call-cc.org/users/thomas-chust|Thomas Chust]] announced some new eggs: [[http://www.chust.org/fossils/webkit|bindings to WebKitGTK+ and JavaScriptCore]], [[http://www.chust.org/fossils/webgate|webgate: a tiny (S)CGI application framework]] and a [[http://www.chust.org/fossils/berkeley-db|small binding for the BerkelyDB library]]
309
310* [[http://wiki.call-cc.org/users/christian-kellermann|Christian Kellermann]]
311  [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00103.html|fixed some bugs]] in the
312  [[http://wiki.call-cc.org/eggref/4/cairo|Cairo Egg]] and tagged version 0.1.11.
313
314* [[http://wiki.call-cc.org/users/alyn.post|Alan Post]] [[http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00001.html|patched]] the
315  [[http://wiki.call-cc.org/eggref/4/sandbox|Sandbox Egg]] to bring it up-to-date with API changes in the Chicken core. His patch was duly accepted.
316
317
318
319== 7. Donations
320
321It's now possible to donate to your favourite scheme (as long as it's Chicken)!
322
323[[http://wiki.call-cc.org/donate]]
324
325We've already had two donations since the account opened a month ago so get donating before it fills up! :-)
326
327Thank you [[http://wiki.call-cc.org/users/sven-hartrumpf|Sven Hartrumpf]] and Ross Lonstein!
328
329
330== 8. Omelette Recipes
331
332This issue's omelette is the second installment of
333[[http://wiki.call-cc.org/users/alaric-blagrave-snellpym|Alaric Snell-Pym]]'s
334"financial book-keeping in scheme" series. If you missed the first part,
335check it out [[http://gazette.call-cc.org/issues/20.html#omelette-recipes|here]].
336
337The dream of many Schemers is to write Scheme for a living. Due to the
338regrettable tendency of employers to frown upon good ideas, the best
339way to achieve this is to start your own company, so you can be your
340own boss, and frown upon the good ideas of others instead.
341
342However, running your own company comes at a price; and a big part of
343that price is the horror of "book keeping"; the requirement to track
344all flows of money and other, more abstract, forms of value in and out
345of your company.
346
347Some people will tell you book-keeping is simple. "Just keep all your
348receipts and bank statements and bills and invoices", they say. "Then
349send them to your accountant at the end of the year."
350
351"But what about value-added tax?", you ask (or "sales tax" in some
352countries). "And what about income tax paid at source for my employees
353(including myself)?". "And why do I need to pay an accountant so much
354to do a job that a computer can easily do in milliseconds?" And then
355the smug smile slowly drips from the face of the "Oh it's easy" crowd.
356
357Clearly, book-keeping is complicated. And yet also simple, in that it
358is determined by sets of rules.
359
360We know what to do, don't we? Let's get coding!
361
362The problem is that book-keeping involves several different kinds of
363inputs - bills (that people send you), invoices (that you send
364people), transfers of money (bills and invoices being paid), loans (to
365and from the company), employees being paid, interest payments from
366the bank, dividend payments, and so on; while it also involves several
367different outputs - tax reports for the various taxes involved (in the
368UK, I had to deal with VAT every three months, income tax and national
369insurance when paying myself and my wife as employees every month, an
370annual filing fee, and annual corporation tax and dividend payments),
371statutory filing of certain financial summaries (generally annually),
372and internal reporting: How much was I spending? How much did each
373client owe? How much should be in the bank by when? Plus, it's nice to
374be able to generate nice invoices to send to folks who owe you
375money. That's a form of specialised report, too, just reporting on a
376single invoice.
377
378Each of the output reports depend in complex ways on different
379information from the inputs. The VAT reports mainly have to add up how
380much VAT I've paid when being billed by others, and how much VAT I've
381charged when invoicing - meaning that VAT needs to be tracked on all
382bills and invoices so it can be extracted. They also want to know
383totals of actual money in and out of the company in the period (even
384stuff where VAT isn't an issue), presumably to check up on
385me. Meanwhile, end of year reports tend to need to know how much I've
386invoiced for various different kinds of work, and what I've spent on
387what kinds of things: buying equipment that will last me for several
388years is handled differently to expenses like travel, or buying stuff
389that I eventually resell to clients (so in our invoices, we need to
390keep track of money charged for services separately to money charged
391for things).
392
393Some reports care about virtual money moving hands. As soon as I
394invoice somebody, then the company now has a virtual asset - some
395money owed to it. That's worth as much as cash in the bank from some
396perspectives (generally, I have to pay tax on it as soon as it's
397invoiced, even if I've not been paid). And yet some care only about
398actual cash changing hands (working out my bank balance, for instance).
399
400Sometimes our clients invite us to incur expenses in doing work for
401them (such as extra travel) and then invoice them on for those
402expenses, so they pay us back - in which case, expenses need to be
403able to be tied to invoices, as well. Sometimes we decide to cancel an
404invoice, which can't be done by just pretending it never existed, for
405audit-trail reasons; we need to issue a "negative" invoice called a
406credit note.
407
408Just to complicate matters more, the actual movement of money isn't
409atomic. If I invoice somebody on date A, they might post me a cheque
410which arrives on date B, which I pay into the bank on date C, which
411actually clears into the account (and thereby appears on my bank
412statement, when I get it) on date D. So at date A the company now has
413a "we are owed" pretend-money asset, which goes through various stages
414until it finally turns into money in the bank on date D.
415
416I handled my book-keeping with some hacky scripts written in Chicken
417Scheme. What I'm going to document here is partly what I've done, and
418partly what I should have done - it was a very iterative process,
419refining the best way to handle stuff, and there's lots of
420improvements I've wanted to make but not had time to. So I'm going to
421describe the ideal case, not the hacky half-way house I actually have
422right now!
423
424The approach I took was to have a file called a "ledger" that I enter
425all my invoices and so on into. This is parsed to build up a bunch of
426data structures in memory, from which the various reports can easily
427be obtained. Firstly, for each kind of input object (invoices, bills,
428etc) there's a list of them, or more often a hashmap to make them easy
429to find by some identifier (I can give my invoices unique symbolic
430names, for instance). That contains the raw data as parsed from the
431ledger file. But then we also create summary structures, which are
432used by the more general reports to generate their output without
433having to special-case each and every different input object type, and
434to enable sharing of common functionality between reports.
435
436The main summary structure is the double-entry transaction list, which
437models the entire financial activity of the company as transfers
438between accounts.
439
440Imagine I invoice Widget Corp for setting up and installing a router:
441
442  INVOICE INV005: Issued 2011-04-25
443  Router setup and installation: GBP 800
444  1 router from my stock: GBP 350
445  1 train ticket for me to go to their site: GBP 35 (no VAT due)
446  Subtotal: GBP 1,185
447  VAT on the above: GBP 230
448  Total due: GBP 1,415
449
450As part of the work, I lose a router (worth GBP 350) from my stock,
451and have to spend GBP 35 on a train fare.
452
453This might expand into the following transactions:
454
455 * 2011-03-02: "Expense for Widget Corp (INV005)"
456   * expenses.travel +35 "Travel to site"
457   * cash -35
458
459 * 2011-04-25 "Invoice Widget Corp (INV005)"
460   * income.work -800 (Router set up and installation)
461   * stock.balances -350 (1 of LX300 router, serial number 0343248)
462   * clients.widgetcorp.expenses -35 (Travel 2011-03-02)
463   * taxes.vat -230
464   * clients.widgetcorp.balance +1415
465
466And, eventually, they might pay me, which hits my bank account some
467time later:
468
469 * 2011-05-07 "Payment from Widget Corp (INV005)"
470   * clients.widgetcorp.balance -1415
471   * bank.balance +1415
472
473And then one day I'll pay my VAT bill, which will look something like:
474
475 * 2011-06-01 "VAT payment for period from 2011-03-01 to 2011-06-01"
476   * taxes.vat 230
477   * bank.balance -230
478
479Note a few tricky things. Each transactions "splits", as the lines
480within them are known, have to sum to zero for everything to balance
481correctly, which tells us that nothing has gone missing. So when we
482start being owed GBP 1,415 by Widget Corp, we need to account for
483where that asset has come from. Special accounts with names such as
484"income.work" (for value generated by me working) and
485"clients.widgetcorp.expenses" (for previously-paid expenses that, as
486of this invoice, I can charge the client for) pop into
487existence. "taxes.vat" looks as if VAT is a form of income for me, as
488money comes "from" it in the transaction - which is sort of true; I'm
489charging Widget Corp for some VAT alongside for the actual work
490done. Figuring out what signs to put on all the items in the invoice
491is mind-bending and painful, but if you just concentrate on making it
492all add up to zero in the end and starting from things that are
493obvious (is money going into or out of the bank account, or the "owed
494to me by this customer" account?), you can figure it out.
495
496From the above, we can start to flesh out some data structures:
497
498<enscript>
499(define-record txn
500               date customer code description splits)
501
502(define-record txn-split
503               account amount notes)
504
505
506(define *txns* (make-hash-table))
507(define (register-txn! txn)
508  (if (hash-table-exists? *txns* (txn-date txn))
509      (begin
510        (set! (hash-table-ref *txns* (txn-date txn))
511              (cons txn
512                    (hash-table-ref *txns* (txn-date txn)))))
513      (begin
514        (set! (hash-table-ref *txns* (txn-date txn))
515              (list txn)))))
516</enscript>
517
518What is an "account"? There's a few kinds, and what kind of account it
519is matters in reporting. Accounts might be assets within the company -
520such as "clients.widgetcorp.balance" or "bank.balance" or
521"stock.balance". Or they may be places where money (be it real or
522virtual) is created from or destroyed by (from the perspective of the
523company), such as "income.work" and "expenses.travel". The important
524difference is that balance-type accounts have a balance that is
525increased when money is sent to them and decreased when it's taken
526out, and that balance is part of the value of the company, while the
527income/expense type accounts don't. In my terminology, these are
528"balance" accounts and "delta" accounts. Each account also begins to a
529group, used to aggregate them in reports: there's income accounts,
530bank accounts, client accounts, and so on. And accounts may be tied to
531a third party - I've given an example of a client above, but also, the
532organisations that send me bills have balances (the money I owe
533them). In general, every third party (be they ones that bill me, or
534ones that I invoice, or both - I've interacted with other freelancers,
535sometimes working for them, sometimes vice versa) has a set of
536accounts attached to them for their balance, expenses I can claim from
537them, and so on. That implies another set of record types:
538
539<enscript>
540(define-record third-party
541               name address balance-account expenses-account)
542
543(define-record account
544               name type group third-party)
545
546
547(define *third-parties* (make-hash-table))
548(define *accounts* (make-hash-table))
549
550(define (find-account acct-name)
551  (hash-table-ref *accounts* acct-name))
552
553(define (register-account! acct)
554  (set! (hash-table-ref *accounts* (account-name acct)) acct))
555</enscript>
556
557An account's {{third-party}} slot may be {{#f}} if it's not part of a
558third party.
559
560Now, to make things easy, I parse the ledger by just defining a heap
561of procedures and macros, and then diving into the ledger with
562eval. It's just Scheme code that, as it is executed, builds up the
563data structures. We'll need some helpers to set up third parties
564properly:
565
566<enscript>
567(define (define-third-party name address group)
568        (let* ((balance-account
569                (make-account
570                 (string-append name ".balance")
571                 'balance
572                 group
573                 #f))
574               (expenses-account
575                (make-account
576                 (string-append name ".expenses")
577                 'delta
578                 'expenses-reclaimed
579                 #f))
580               (third-party
581                (make-third-party
582                 name address balance-account expenses-account)))
583          (account-third-party-set! balance-account third-party)
584          (account-third-party-set! expenses-account third-party)
585          (register-account! balance-account)
586          (register-account! expenses-account)
587          (set! (hash-table-ref *third-parties* name) third-party)))
588</enscript>
589
590Now we can work on a way of representing bills and invoices. A nice
591input syntax would be:
592
593<enscript>
594(define-third-party "clients.widgetcorp" "123 Any Street" 'clients)
595(register-account! (make-account "income.work" 'delta 'income #f))
596(register-account! (make-account "expenses.travel" 'delta 'travel #f))
597(register-account! (make-account "stock.balance" 'balance 'stock #f))
598(register-account! (make-account "cash" 'balance 'cash #f))
599(register-account! (make-account "taxes.vat" 'balance 'vat #f))
600
601(invoice "INV005" "clients.widgetcorp" (ymd 2011 04 25)
602         (service "income.work" 800 (vat20) "Router setup and installation")
603         (sale "stock.balance" 350 (vat20) "1 of LX300 router, serial number 0343248")
604         (expense (ymd 2011 03 02) "expenses.travel" "cash" 35 () "Travel to site"))
605</enscript>
606
607The idea is that the sales taxes incurred on a line are specified as a
608list after the amount. If there's no taxes due, then we use an empty
609list. Otherwise we have a list of taxes, which are either plain tax
610names (to have the system compute the tax due itself) or two-element
611lists joining a tax name to a precomputed amount (often, when we pass
612on an expense, we know the tax we paid as it's on the receipt, so we
613should use that (even if they made a mistake working it out) rather
614than calculating our own).
615
616A nice way to implement that might be to make "invoice" a macro that
617absorbs its first three arguments as an invoice code, the name of the
618third party to invoice, and the date; then treats the rest as a body
619to be wrapped in a dynamic environment in which a parameter allows
620{{sale}}, {{expense}}, and {{service}} to add lines to the
621invoice. This is easily arranged:
622
623<enscript>
624(define-record date year month day)
625
626(define-record invoice
627  name date third-party lines)
628
629(define (register-line! invoice line)
630  (invoice-lines-set! invoice
631                      (cons line (invoice-lines invoice))))
632
633;; Compute sales taxes
634(define (compute-tax tax amount)
635  (case tax
636    ((vat20) (* 0.20 amount)) ;; Current UK rate
637    ((vat15) (* 0.15 amount)) ;; Previous UK rate
638    ((vat175) (* 0.175 amount)))) ;; Previous UK rate
639
640;; Expand a list of taxes, some of which might be bare symbols
641;; naming taxes to work out, or (<tax> <amount>) lists for
642;; ready-computed taxes, into an alist of tax names to tax amounts
643(define (resolve-taxes amount taxes)
644  (map (lambda (tax-desc)
645         (if (list? tax-desc)
646             (cons (car tax-desc) (cadr tax-desc))
647             (cons tax-desc (compute-tax tax-desc amount))))
648       taxes))
649
650(define-record invoice-service-line
651  income-account amount taxes description)
652
653(define-syntax service
654  (syntax-rules ()
655    ((service income-account amount taxes description)
656     (register-service! (*current-invoice*) income-account amount 'taxes description))))
657
658(define (register-service! invoice income-account amount taxes description)
659  (let ((service
660         (make-invoice-service-line
661          (find-account income-account)
662          amount
663          (resolve-taxes amount taxes)
664          description)))
665    (register-line! invoice service)))
666
667(define-record invoice-sale-line
668  stock-account amount taxes description)
669
670(define (register-sale! invoice stock-account amount taxes description)
671  (let ((sale
672         (make-invoice-sale-line
673          (find-account stock-account)
674          amount
675          (resolve-taxes amount taxes)
676          description)))
677    (register-line! invoice sale)))
678
679(define-syntax sale
680  (syntax-rules ()
681    ((sale stock-account amount taxes description)
682     (register-sale! (*current-invoice*) stock-account amount 'taxes description))))
683
684(define-record invoice-expense-line
685  expense-account payment-account amount taxes description)
686
687(define (register-expense! invoice date expense-account payment-account amount taxes description)
688  (let ((expense
689         (make-invoice-expense-line
690          (find-account expense-account)
691          (find-account payment-account)
692          amount
693          (resolve-taxes amount taxes)
694          description)))
695    (register-line! invoice expense)))
696
697(define-syntax expense
698  (syntax-rules ()
699    ((expense (ymd year month day) expense-account payment-account amount taxes description)
700     (register-expense!
701      (*current-invoice*)
702      (make-date year month day)
703      expense-account
704      payment-account
705      amount
706      'taxes
707      description))))
708
709(define *current-invoice* (make-parameter #f))
710(define *invoices* (make-hash-table))
711
712(define-syntax invoice
713  (syntax-rules (service sale expense)
714    ((invoice name third-party (ymd year month day) body ...)
715     (let ((inv
716            (make-invoice name
717                          (make-date year month day)
718                          (hash-table-ref *third-parties* third-party)
719                          '())))
720       (parameterize
721        ((*current-invoice* inv))
722        (begin body ...))
723       (set! (hash-table-ref *invoices* name) inv)
724       (generate-invoice-transactions! inv)))))
725</enscript>
726
727We end the expansion of the {{invoice}} macro with a call to
728{{generate-invoice-transactions!}}, which will do the task of creating
729the double-entry transactions for the invoice. Other types of summary
730structure can be added by calling additional generator procedures at
731this point. This is largely a matter of going through the invoice
732lines, handling them on a case-by-case basis to generate lists of
733transaction splits that we can append together to generate the invoice
734transaction. The case of expense lines is interesting, in that an
735extra transaction has to be generated for each expense, to record its
736initial spending, as well as a split to record the expense being
737claimed in the invoice.
738
739For now, let's just handle one case:
740
741<enscript>
742(define (generate-invoice-transactions! inv)
743  (register-txn! (make-txn
744    (invoice-date inv)
745    (string-append "Invoice " (invoice-name inv) " for "
746                   (third-party-full-name (invoice-third-party inv)))
747    (let ((txn-balance-account
748           (third-party-balance-account
749            (invoice-third-party inv))))
750      (flatten
751       (map
752        (lambda (line)
753          (cond
754           ((invoice-expense-line? line)
755            (list)) ;; FIXME: Not implemented
756           ((invoice-sale-line? line)
757            (list)) ;; FIXME: Not implemented
758           ((invoice-service-line? line)
759            (list
760             (make-txn-split
761              (invoice-service-line-income-account line)
762              (- (invoice-service-line-amount line))
763              (invoice-service-line-description line))
764             (make-tax-splits
765              (invoice-service-line-taxes line)
766              txn-balance-account)
767             (make-txn-split
768              txn-balance-account
769              (invoice-service-line-amount line)
770              #f)))))
771        (invoice-lines inv)))))))
772
773(define (make-tax-splits taxes txn-balance-account)
774  (map (lambda (tax)
775         (let ((tax-type (car tax))
776               (tax-amount (cdr tax)))
777           (case tax-type
778             ((vat20 vat15 vat175)
779              (list
780               (make-txn-split
781                (find-account "taxes.vat")
782                (- tax-amount)
783                #f)
784               (make-txn-split
785                txn-balance-account
786                tax-amount
787                #f))))))
788       taxes))
789</enscript>
790
791Feeding in the above example invoice, then checking out the resulting
792double-entry transaction list, shows that it worked:
793
794<enscript>
795(hash-table-for-each *txns*
796                     (lambda (date txns)
797                       (for-each
798                        (lambda (txn)
799                          (printf "Date: ~A Desc: ~A\n"
800                                  (txn-date txn)
801                                  (txn-description txn))
802                          (for-each (lambda (split)
803                                      (printf "Acct: ~A Delta: ~A Notes: ~A\n"
804                                              (account-name (txn-split-account split))
805                                              (txn-split-amount split)
806                                              (txn-split-notes split))) (txn-splits txn)))
807                        txns)))
808</enscript>
809
810   date: #<date> Desc: Invoice INV005 for Widget Corp
811   Acct: income.work Delta: -800 Notes: Router setup and installation
812   Acct: taxes.vat Delta: -160.0 Notes: #f
813   Acct: clients.widgetcorp.balance Delta: 160.0 Notes: #f
814   Acct: clients.widgetcorp.balance Delta: 800 Notes: #f
815
816We've ended up with multiple splits for the same account, as we record
817that both VAT and the money due for the service are to come from the
818client's balance account - and other splits will add plenty more. To
819fix this, we need to write a procedure that canonicalises a list of
820splits, and call that on the splits before calling
821{{make-txn}}. Canonicalisation consists of finding all the splits that
822refer to the same account and have the same notes (be it {{#f}} or a
823string) and merging them into one with the total of the amounts. But
824I'll leave that (along with implementing bills, payments, and some
825actual reports) as an exercise to the reader... It's easy to imagine
826how to generate a VAT report from the list of transactions, by
827filtering them for membership of the required date range and looking
828for splits involving "taxes.vat", or to generate a nicely formatted
829invoice by extracting a single invoice record, or to work out the
830balance of an account at any point in time by adding up all the
831transaction splits that involve it up to that point in time. Also, the
832core engine needs to be wrapped up in a module that only exposes the
833required bindings, and hides internals.
834
835Having automated one's book-keeping and financial reporting, many
836operations (such as the VAT returns) can be done without involving an
837accountant; in my case, the accountant is only needed to help with the
838annual corporation tax computation and filing of official accounts,
839which requires deep understanding of the UK tax system to do
840everything properly. Having said that, if I studied the system
841properly (and tracked the changes each year), I'm sure I could
842automate that, too...
843
844== 9. About the Chicken Gazette
845
846The Gazette is produced occasionally by a volunteer from the Chicken
847community. The latest issue can be found at
848[[http://gazette.call-cc.org]] or you can follow it in your feed
849reader at [[http://gazette.call-cc.org/feed.atom]]. If you'd like to
850write an issue, [[http://wiki.call-cc.org/gazette|consult the wiki]]
851for the schedule and instructions!
Note: See TracBrowser for help on using the repository browser.