Major garbage collector runs all the time

[megane]

The major garbage collector runs too frequently if you happen to get unfortunate
"correct" combination of

• heap size,
• total size of "permanent objects", and
• you're generating lots of garbage objects.

(By permanent objects I mean the set of objects that survives a major garbage
collection.)

In this situation you can effectively get a major GC for every N allocated
bytes that survive the 1st generation.

This N is basically just heap_size - permanent_objects, i.e. the free heap
space just after a major collection.

In practice, the N may get quite small. In this case the program is mostly
just doing garbage collection.

One incarnation of this is a program that gradually gets slower and slower, and
then suddenly gets fast again. In this case the sum of permanent objects gets
bigger and bigger until the objects no longer fit in the heap. At this point the
heap is grown and everything gets smooth again.

One possible workaround is to manually specify a big heap with -:h or -:hi.
This has the side-effect of

1. Making finalizers run less often (they are checked during each major collection)
2. Causing memory fragmentation. (Objects are compacted during the major GC.)
3. You have to choose a size. With -:hi you can still run into the problem if you choose too small size.

One example of affected program:
​https://stackoverflow.com/questions/62826083/chicken-scheme-read-line-taking-too-long/63215403

(define *corpus*
  (call-with-input-file "largeish_file.txt"
    (lambda (input-file)
      (let loop ([line (read-line input-file)]
                 [tokens '()])
        (if (eof-object? line)
            tokens
            (loop (read-line input-file)
              (append tokens (string-split line))))))))

This program generates lots of garbage by creating lots of temporary lists.
Using big heap (using -:h) makes things a lot faster.

I've posted a patch here:
​https://lists.nongnu.org/archive/html/chicken-hackers/2020-01/msg00002.html
This works by growing the heap if it's too full after a major collection.

[Mario Domenech Goulart]

Patches have been merged.

Follow-up thread: ​https://lists.nongnu.org/archive/html/chicken-hackers/2021-01/msg00003.html