#!rest and #!key are not internally consistent

[Alan Post]

Run the following program through csi with the test egg installed.

I've tried to group these tests such that if one in a group succeeds, all of them should. It appears that #!rest and #!key are not self-consistent, following separate rules in conceptually identical situations.

the separation of (foo ...) and (apply foo ...) shows that these tests aren't affected by (apply ...), so the (foo ...) vs (apply foo ...) difference is not essential to this issue.

The main observations:

1) #!rest affects where #!key args must be placed.
2) sometimes #!key args in the middle of an argument list work, and sometimes they don't.

I'm not trying to point out a specific test as being wrong, but that the set of these tests should be consistent and aren't.

(use test)

;; "initial" fails and "initial+rest" succeeds.  I would expect that,
;; given the "initial" test failure, that "initial+rest" would also fail.
;;
;; "end" succeeds and "end+rest" fails.  I would expect that, since
;; "end" succeeds "end+rest" would also succeed.
;;
;; Even if either of these expectations is false:
;;
;; #!key args must be in the initial position if you're calling
;; a function that also uses #!rest, but must be in the terminal
;; position if the function does not use rest.  I would expect one
;; or the other of these (or both) to succeed, but I least expected
;; the failure condition to be the opposite of the "initial" and
;; "initial+rest" cases.
;;
;; I would expect, if only one position (front or back) works, that
;; the same  position works regardless of whether the function also
;; accepts #!rest arguments.
;;
;; All of the "middle(+rest)*" tests fail *except* "middle+rest (2)".
;; I consider it strange that this test succeeds where all of the
;; remaining tests fail.
;;
;; It seems as if they should all fail or all succeed.
;;

(define (fn x y z #!key k)
  `((,x ,y ,z) ,k))

(define (fn-rest #!rest r #!key k)
  `(,r ,k))

;;; initial position
;;;

;; Here we *do not* match #!key k, unless we have a #!rest.
;;
(test-group "initial"
  (test '((0 1 2) 3) (fn k: 3 0 1 2))
  (test '((0 1 2) 3) (apply fn '(k: 3 0 1 2))))

(test-group "initial+rest"
  (test '((k: 3 0 1 2) 3) (fn-rest k: 3 0 1 2))
  (test '((k: 3 0 1 2) 3) (apply fn-rest '(k: 3 0 1 2))))


;;; middle position
;;;

;; both of these fail
;;
(test-group "middle"
  (test '((0 1 2) 3) (fn 0 k: 3 1 2))
  (test '((0 1 2) 3) (apply fn '(0 k: 3 1 2))))

(test-group "middle+rest"
  (test '((0 k: 3 1 2) 3) (fn-rest 0 k: 3 1 2))
  (test '((0 k: 3 1 2) 3) (apply fn-rest '(0 k: 3 1 2))))


;; The "middle+rest (2)" succeeds, where I would expect it to fail
;; with everything else (or better, I expect all of the middle
;; groups to succeed.)
;;
(test-group "middle (2)"
  (test '((0 1 2) 3) (fn 0 1 k: 3 2))
  (test '((0 1 2) 3) (apply fn '(0 1 k: 3 2))))

(test-group "middle+rest (2)"
  (test '((0 1 k: 3 2) 3) (fn-rest 0 1 k: 3 2))
  (test '((0 1 k: 3 2) 3) (apply fn-rest '(0 1 k: 3 2))))


;;; end position
;;;

;; here we *do* match #!key k, unless we have a #!rest.  Opposite of
;; the "initial" tests.  I would expect the success/failure pattern
;; to match the "initial" tests, rather than being opposite of them.
;;
(test-group "end"
  (test '((0 1 2) 3) (fn 0 1 2 k: 3))
  (test '((0 1 2) 3) (apply fn '(0 1 2 k: 3))))

(test-group "end+rest"
  (test '((0 1 2 k: 3) 3) (fn-rest 0 1 2 k: 3))
  (test '((0 1 2 k: 3) 3) (apply fn-rest '(0 1 2 k: 3))))

(test-exit)

[Christian Kellermann]

The section in DSSSL ​http://wiki.call-cc.org/man/4/Extensions%20to%20the%20standard mentions the order in which evaluation takes place. Where is the inconsistency you see?

By comparing this description with your very nice test cases (we should include them in the test suite really) I cannot see something amiss. Can you point me to it again please?

[felix winkelmann]

Alan, the required parameters are never considered for keyword lookup - I think this is a misunderstanding of the DSSSL semantics. Keyword-arguments are only looked up in everything that follows #!key.

[Alan Post]

Replying to felix:

Alan, the required parameters are never considered for keyword lookup - I think this is a misunderstanding of the DSSSL semantics. Keyword-arguments are only looked up in everything that follows #!key.

That helps me to understand, thank you. In this case, why does the "middle+rest" test fail, where the "middle+rest (2)" test succeeds? Given your description, I would expect both of those to fail or succeed.

[Alan Post]

Here is the use case I have in mind that makes this behavior frustrating.

Imagine I have a library that is given a callback. I am going to call the callback procedure with a set of #!key arguments and a set of positional arguments. It would be very nice if I could call the procedure with a finite set of #!key arguments and a variable number of positional arguments. I *can* actually do that, as show in the "initial+rest" test. Good!

Now imagine instead the same callback, but the caller *knows* how many arguments they're getting, they want to do away with variable number of positional arguments and explicitly name them. They change the signature from "initial+rest" to "initial", and now suddenly they are not able to bind #!key arguments.

Instead, *I* have to change the way I call the routine from the calling site. I've become the "end" test case, except now callers that were "initial+rest" are now "end+rest," and suddenly they don't work!

If I want my calling site to accommodate both kinds of callback routines, and I can't. I have to know whether the client is using #!rest or not, and alter the placement of my #!key arguments when I call the function.

Essentially, it is impossible to have both #!key arguments and a variable number of positional arguments.

I understand that this follows from the *implementation* of #!rest and #!key, but from the perspective of someone using these features, it looks arbitrary, particularly when "middle+rest (2)" is the only "middle"* test that does so.

[felix winkelmann]

Replying to alanpost:

That helps me to understand, thank you. In this case, why does the "middle+rest" test fail, where the "middle+rest (2)" test succeeds? Given your description, I would expect both of those to fail or succeed.

Because fn-rest has no required/optional parameters - these are simply taken out of the list of possible keyword arguments.

[felix winkelmann]

Replying to alanpost:

Essentially, it is impossible to have both #!key arguments and a variable number of positional arguments.

It is possible if you deconstruct the argument list completely inside the callee (just using #!rest/#!key and no positional arguments). I'm not sure I understand completely, but what do you mean with "variable number of positional arguments"? For the same procedure? Varying over what?

I understand that this follows from the *implementation* of #!rest and #!key, but from the perspective of someone using these features, it looks arbitrary, particularly when "middle+rest (2)" is the only "middle"* test that does so.

It's not arbitrary at all, it's perfectly well defined in the DSSSL language and implemented like this in several Scheme systems.

[felix winkelmann]

Milestone 4.7.0 deleted

[felix winkelmann]

Milestone 4.8.0 deleted