#!key args not assigned when |eval| called on compiler output

[Alan Post]

Set Up

This ticket is a follow-up to the ​e-mail thread I posted about this issue.

I'm running chicken from the experimental branch, version 04af68b541adab8dcaa24dd0915c7e68de6eb7ed. [fixed bug in ##sys#halt (thanks to Jeronimo Pellegrini)]

I'm running OpenBSD 4.8 on i386 in a VMWare virtual machine.

This bug refers to the ​genturfa'i egg.

For this bug, please check out r21579 from the egg repository and build the egg:

svn co -r21579 https://code.call-cc.org/svn/chicken-eggs/release/4/genturfahi/trunk
cd trunk
chicken-install -s -test

This revision of the egg should pass all of its tests.

Expected vs Actual Result

The problem that I'm having is that the routine test-samselpla-re​ does not get it's #!key argument (|gerna|) assigned in all cases in which it is called, even though the #!key name is passed as a parameter to the function each time.

This routine is called in three ways:

1. By directly including the parser​ in the test.
2. By parsing re.peg​ using the bootstrap compiler​, writing the output of the compiler to a port​, and reading the port back and calling eval on the result.
3. By parsing re.peg​ using the bootstrap compiler​, and calling eval directly on the results​. Note that this is the same compiler (bootstrap.scm) compiling the same input file (tests/re.peg) as option #2.

option #1 and #2 work as expected. Option #3 does not assign #!key arguments, through the same argument list is passed to test-samselpla-re in each of the three cases.

option #3 is currently commented out, and will need to be uncommented to see this bug. Each option is grouped at the bottom of the tests/re.scm file​.

I would expect option #3 to work like option #1 and option #2, passing the test suite for this PEG grammar.

Detailed Debugging Information

Because the volume of code for this bug is so large, I provide here pointers to the relevant sections.

The runtime system for the compiler is responsible for calling test-samselpla-re. That call is performed by the function generated by nunjavni-samselpla​.

That routine generates the parse tree by calling the variable nunjavni, filtering out any result without an associated name (called cmene or cme), and creating the argument list by appending the key/value pairs, which are then applied to the supplied routine.

Looking at the inlined compiler output in option #1, you can see that nunjavni-samselpla is called with test-samselpla-re as one of it's arguments. This is done again in option #2 and option #3.

You can see the output produced by the parser by running the following command from the svn checkout:

genturfahi-peg tests/re.peg

It compiles tests/re.peg:

gerna <- #gerna: (digit / alpha)* FAhO {test-samselpla-re}
digit <- [:digit:]+
alpha <- [:alpha:]+
FAhO  <- !.

Into the following scheme code, which is the code being evaluated in option #2 and option #3:

(letrec ((gerna (nunjavni-morji
                  (nunjavni-samselpla
                    test-samselpla-re
                    (nunjavni-je
                      (nunjavni-*
                        (nunjavni-jonai
                          (nunjavni-cmene
                            (nunjavni-naselci digit)
                            cmene:
                            'digit:)
                          (nunjavni-cmene
                            (nunjavni-naselci alpha)
                            cmene:
                            'alpha:))
                        cmene:
                        'gerna:)
                      (nunjavni-cmene
                        (nunjavni-naselci FAhO)
                        cmene:
                        'FAhO:)))))
         (digit (nunjavni-morji (nunjavni-re "[[:digit:]]+")))
         (alpha (nunjavni-morji (nunjavni-re "[[:alpha:]]+")))
         (FAhO (nunjavni-morji (nunjavni-fanmo))))
  gerna)

The test-samselpla-re routine is attached to the grammar rule |gerna| in order to return only the grammar, but not the end of input token.

Conclusion

I'm sorry that I have not been able to make this test case simpler. I know that there is a lot of code involved in recreating this problem, and I'm not certain I'm even seeing a bug.

If I have forgotten a useful piece of information, please ask me for clarification, as I obviously understand this code better than anyone else and can likely save time in determining what is going on.

[felix winkelmann]

Alan, do you convert strings read in manually to symbols? string->symbol will not automatically convert into keywords, regardless of a trailing colon. This must be done with string->keyword.

[felix winkelmann]

Replying to felix:

Alan, do you convert strings read in manually to symbols? string->symbol will not automatically convert into keywords, regardless of a trailing colon. This must be done with string->keyword.

(without the colon)

[Alan Post]

I have committed r21593, which uses string->keyword rather than string->symbol to construct #!key arguments.

This fixes the problem with option #3. I have uncommented the PEG parser regression tests for the re and mex unit, as they now pass.

Thank you for such a quick response on this, Felix and Alex.

[Alan Post]

For completeness, I want to record additional information on this ticket. This comment is not actionable.

I reworked the handling of string->keyword conversion to happen as late as possible, and to be done whether I was hand-writing a parser or generating one with the parser generator. This was a slightly larger commit than r21593, and includes a syntax change to the PEG grammar. I like this solution better, as it applies regardless of how the parser is written, and it localizes the keyword handling with the code that calls routines with keyword arguments.

Previously I'd been constructing symbols that looked like keywords, but only sometimes acted like keywords. Now I always construct strings that are converted to keywords.

commit r21604.

[felix winkelmann]

Milestone 4.7.0 deleted

[felix winkelmann]

Milestone 4.8.0 deleted