Ticket #874 (closed defect: fixed)

Opened 3 years ago

Last modified 2 years ago

Compiling some files may cause infinite optimization loops, eating all available memory

Reported by: mario Owned by: felix
Priority: major Milestone: 4.9.0
Component: compiler Version: 4.7.x
Keywords: Cc:

Description

chicken (master -- e5f341a81cb2d5b8c4f7d8eedabcd35c1ecb7281) dies with a segmentation fault when attempting to compile the attached file (srfi-14-test.scm).

While compiling, it uses more than 2GB of RAM. It crashed after running for more than 9 minutes on a 2.4GHz processor. The system is Linux x86.

I tried building chicken with DEBUGBUILD=1 and running it under gdb, but it didn't crash in more than 10 hours, when I stopped it (it was still using 2GB of RAM).

Attachments

srfi-14-test.scm (10.1 kB) - added by mario 3 years ago.
srfi-14-test.scm

Change History

Changed 3 years ago by mario

srfi-14-test.scm

  Changed 2 years ago by sjamaan

  • summary changed from High memory usage for compiling to Compiling some files may cause infinite optimization loops, eating all available memory
  • milestone set to 4.8.0

csc -debug o srfi-14.scm shows that it gets stuck contracting the same procedures over and over.

This is potentially a severe bug affecting other cases too so I'm promoting this to milestone 4.8.0.

  Changed 2 years ago by sjamaan

It's not an infinite loop, but it keeps contracting the same procedure more than once. If you rip out all the tests under "shiver's tests" the problem goes away. Put some of them back, it takes longer. Put more back, it takes even longer etc.

It's not the macro either. Looks like the cond is causing the problem. Looks like an O(n2) behavior with n being the number of branches in the cond.

Here's a simplified case that clearly shows what's going on:

(cond ((not (= 1 2)) (print 2))
      ((not (= 5 6)) (print 4))
      (else (print 'nothing)))
csc -debug o test.scm
Removed `not' forms: 2 
contracted procedure: k31 
folded constant expression: (= (quote 5) (quote 6)) 
contracted procedure: k22 
contracted procedure: k31 
folded constant expression: (= (quote 5) (quote 6)) 
folded constant expression: (= (quote 1) (quote 2)) 
replaced variables: 21 
contracted procedure: k22 
removed binding forms: 2 
substituted constant variable: r23 
contracted procedure: k31 
removed binding forms: 4 
removed conditional forms: 1 
removed binding forms: 1 

Note the double occurrance of the folded expression, which is probably a result of the fact that the procedure k31 that's generated by cond's expansion is contracted twice. Add a line and you get three entries for the newly added line:

(cond ((not (= 1 2)) (print 2))
      ((not (= 5 6)) (print 4))
      ((not (= 7 8)) (print 5))
      (else (print 'nothing)))
csc -debug o test.scm
Removed `not' forms: 3 
contracted procedure: k40 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k31 
contracted procedure: k40 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
contracted procedure: k22 
contracted procedure: k40 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k31 
contracted procedure: k40 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
folded constant expression: (= (quote 1) (quote 2)) 
replaced variables: 53 
contracted procedure: k22 
removed binding forms: 2 
substituted constant variable: r23 
contracted procedure: k31 
removed binding forms: 4 
removed conditional forms: 1 
removed binding forms: 1 

And another line:

(cond ((not (= 1 2)) (print 2))
      ((not (= 5 6)) (print 4))
      ((not (= 7 8)) (print 5))
      ((not (= 9 10)) (print 6))
      (else (print 'nothing)))
csc -debug o test.scm
Removed `not' forms: 4 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k40 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k31 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k40 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
contracted procedure: k22 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k40 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k31 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k40 
contracted procedure: k49 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
folded constant expression: (= (quote 1) (quote 2)) 
replaced variables: 117 
contracted procedure: k22 
removed binding forms: 2 
substituted constant variable: r23 
contracted procedure: k31 
removed binding forms: 4 
removed conditional forms: 1 
removed binding forms: 1 

Interestingly, it does not happen when you remove the "not".

  Changed 2 years ago by sjamaan

Further simplification: if you convert it to a nested if the problem still occurs:

(if (not (= 1 2))
    (print 2)
    (if (not (= 5 6))
        (print 4)
        (if (not (= 7 8))
            (print 5)
            (if (not (= 9 10))
                (print 6)
                (print 'nothing)))))
$ csc -debug o test.scm
Removed `not' forms: 4 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k37 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k28 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k37 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
contracted procedure: k19 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k37 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
contracted procedure: k28 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
contracted procedure: k37 
contracted procedure: k46 
folded constant expression: (= (quote 9) (quote 10)) 
folded constant expression: (= (quote 7) (quote 8)) 
folded constant expression: (= (quote 5) (quote 6)) 
folded constant expression: (= (quote 1) (quote 2)) 
replaced variables: 117 
contracted procedure: k19 
removed binding forms: 2 
substituted constant variable: r20 
contracted procedure: k28 
removed binding forms: 4 
removed conditional forms: 1 
removed binding forms: 1 

  Changed 2 years ago by sjamaan

Could this be due to the fact that perform-pre-optimization! mutates the nodes? Perhaps this creates an inconsistency between the analysis database and the node tree. Felix?

  Changed 2 years ago by felix

  • owner set to felix
  • status changed from new to assigned

Multiple contractions of the same function is already incorrect ("contraction" means inlining of a function called once). I think the real problem are nested contractions. In the last example above, k46 is nested in k37. The former gets contracted, then the latter (in the same optimization pass). In this case it seems to happens repeatedly. For inlining this case (an inlined procedure being the target of additional inlining) is taken care of by having the inline-target db entry - we probably need this (or something similar) for contraction, too.

  Changed 2 years ago by sjamaan

Do multiple contractions of the same form even happen for programs without perform-pre-optimization!?

follow-up: ↓ 8   Changed 2 years ago by sjamaan

In other words, do you think the bug is in the parts of the optimizer that handle contractions rather than perform-pre-optimization!?

in reply to: ↑ 7   Changed 2 years ago by felix

Replying to sjamaan:

In other words, do you think the bug is in the parts of the optimizer that handle contractions rather than perform-pre-optimization!?

I do, yes. There is a check for the inline-target db entry, which is done for procedures where inlining takes place. I think this must also be done for contractions, but I have to try it out first (real soon).

  Changed 2 years ago by felix

This is actually not related to nested contractions or inlining conflicts. The problem appears to be that copy-propagation inserts references to contracted procedures.

  Changed 2 years ago by mario

  • status changed from assigned to closed
  • resolution set to fixed

Fixed by 285f53dbca729cffb4c4d9ee84e4ba893c882546

  Changed 2 years ago by felix

  • milestone changed from 4.8.0 to 4.9.0

Milestone 4.8.0 deleted

Note: See TracTickets for help on using tickets.