Summary

utf8 egg: Missing char sets and outdated tables

Metadata

Attachments

Description

The unicode-char-sets module of the utf8 egg is missing several character sets. In particular, there is no set for characters with the Numeric property (making it impossible to implement a Unicode-aware 'char-numeric?' in CHICKEN) or for any of the punctuation properties. The utf8-srfi-14 module includes char-set:digit and char-set:punctuation, but these are throwaway ASCII-only implementations (in a file that begins with "Unicode capable char-sets", no less!). These sets should be added.

Furthermore, the sets that unicode-char-sets does provide seem to be built on data that is extremely out-of-date. The header comment in unicode-char-sets.scm claims the tables were generated in 2007.

Changes and comments

[2025-06-27 19:53:47 UTC] zaifir wrote:

I'm working on an updated version of the utf8 egg which fetches & generates the Unicode character sets from the official tables. In the process, I've learned a lot more about the complexities of the Unicode property architecture.

While I think char-set:numeric is still absolutely necessary & should be added, I now believe that the egg authors did the right thing in not extending beyond ASCII char-set:digit, char-set:punctuation, & the rest of the old SRFI 14 sets. This is a bit confusing, however, since the names of some utf8-srfi-14 sets are very similar to those in unicode-char-sets, e.g. char-set:whitespace & char-set:white-space.

Since some Unicode char sets are quite large, I propose splitting (unicode-char-sets) into submodules, each containing one set. For example, char-set:arabic should be provided by (unicode-char-sets arabic). I'm implementing this in my new version of the egg.

More soon.

[2025-07-02 18:33:29 UTC] zaifir attached generate-sets.sh (description=Trivial driver script (run by custom-build).)

[2025-07-02 18:34:56 UTC] zaifir wrote:

I've attached my set-module generation script & a patch for utf8.egg.

[2025-07-02 18:53:12 UTC] zaifir attached generate-sets.scm (description=Script to generate Unicode char set modules from current UCD data.)

[2025-07-02 18:55:57 UTC] zaifir attached unicode-char-sets.scm (description=Catch-all unicode-char-sets module.)

[2025-07-04 17:49:33 UTC] zaifir attached utf8_egg.diff (description=Changes to utf8 egg file.)

[2025-07-04 17:53:52 UTC] zaifir wrote:

Per Pietro Cerutti's comments on chicken-users, I've changed the egg file to build the set extensions normally, rather than have it call `generate-sets` as a custom build script. I expect the egg maintainer to run `generate-sets` periodically to regenerate the extension files.

The egg also defines an extension for each character set. Redirect complaints about this to Felix; I can't get `chicken-install` to build the egg correctly with any other configuration.

[2026-07-04 22:28:18 UTC] felix wrote:

Proper UTF-8 support has been implemented in CHICKEN 6.

[2026-07-04 22:28:19 UTC] felix changed status from new to closed

[2026-07-04 22:28:19 UTC] felix set resolution to wontfix