source: project/wiki/eggref/4/intarweb @ 27253

[[tags: egg]]

== Intarweb

[[toc:]]

=== Description

Intarweb is an advanced http library.  It parses all headers into more
useful Scheme values.  It serves as a low-level basis for http servers
or clients.  For a more high-level server library that is based on
intarweb, see [[spiffy]].  For a high-level client library based on
intarweb, see [[http-client]].

You would rarely need to build an application directly on top of raw
Intarweb, but when using one of the above-mentioned libraries you
often need to interact with intarweb's API.

=== Author

[[/users/peter-bex|Peter Bex]]

=== Requirements

Requires the [[defstruct]], [[base64]] and [[uri-common]] extensions.

=== Documentation

The intarweb egg is designed to be used from a variety of
situations. For this reason, it does not try to be a full HTTP client
or server. If you need that kind of functionality, see eggs like
[[spiffy]] or [[http-client]].

=== Requests

<procedure>(make-request #!key uri port (method 'GET) (major 1) (minor 1) (headers (make-headers '())))</procedure>

Create a request object (a [[defstruct]]-type record). The client will
generally write requests, while the server will read them.

The URI defines the entity to retrieve on the server, which should be
a [[uri-common]]-type URI object. The PORT is the scheme I/O port
where the request is written to or read from.  The METHOD is a symbol
that defines the HTTP method to use (case sensitive). MAJOR and MINOR
identify the major and minor version of HTTP to use. Currently, 0.9,
1.0 and 1.1 are supported (but be careful with 0.9, it has some weird
consequences and is not widely supported). HEADERS must be a headers
object.

<procedure>(update-request old-request #!key uri port method major minor headers)</procedure>

Like {{make-request}}, except this takes an {{old-request}} object as
a template for values which are missing from the parameter list, thereby
providing a way to do a purely functional update of that object.


<procedure>(request-uri REQUEST) => URI</procedure><br>
<procedure>(request-port REQUEST) => PORT</procedure><br>
<procedure>(request-method REQUEST) => SYMBOL</procedure><br>
<procedure>(request-major REQUEST) => NUMBER</procedure><br>
<procedure>(request-minor REQUEST) => NUMBER</procedure><br>
<procedure>(request-headers REQUEST) => HEADERS</procedure><br>

An existing request can be picked apart with these accessors.

<procedure>(write-request REQUEST) => REQUEST</procedure>

Write a request line with headers to the server.  In case it
is a request type that has any body data, this should be written to
the the request's port. Beware that this port can be modified by
write-request, so be sure to write to the port as it is returned by
the write-request procedure!

<procedure>(read-request PORT) => REQUEST</procedure>

Reads a request object from the given input-port.  An optional request
body can be read from the request-port after calling this procedure.

<parameter>(request-parsers [LIST])</parameter>

Requests are parsed using parse procedures, which can be customized
by overriding this parameter.

LIST is a list of procedures which accept a request line
string and produce a request object, or {{#f}} if the
request is not of the type handled by that procedure.

The predefined request parsers are:

* {{http-0.9-request-parser}}
* {{http-1.x-request-parser}}

<procedure>(http-0.9-request-parser STRING) => REQUEST</procedure><br>
<procedure>(http-1.x-request-parser STRING) => REQUEST</procedure><br>

Predefined request parsers for use with {{request-parsers}}.

<parameter>(request-unparsers [LIST])</parameter>

Requests are written using unparse procedures, which can be
customized by overriding this parameter.

LIST is list of procedures which accept a request object and write
to the request's output port and return the new, possibly updated
request object. If the request object is not unparsed by this
handler, it returns {{#f}}.

The predefined request unparsers are:

* {{http-0.9-request-unparser}}
* {{http-1.x-request-unparser}}

<procedure>(http-0.9-request-unparser REQUEST) => REQUEST</procedure><br>
<procedure>(http-1.x-request-unparser REQUEST) => REQUEST</procedure><br>

Predefined request unparsers for use with {{request-unparsers}}.
They return the request, and as a side effect they write the
request to the request object's port.

=== Responses

<procedure>(make-response #!key port status (code 200) (reason "OK") (major 1) (minor 1) (headers (headers '())))</procedure>

Create a response, a [[defstruct]]-type record.  A server will usually
write a response with {{write-response}}; a client will read it
with {{read-response}}.

You can either supply a status symbol or a code and/or reason to set
the response status.  If you do, the code will be set to match that
symbol and the reason is set to the default reason belonging to that
code, as provided by the HTTP standard(s).  The allowed symbols are
generally just Schemified versions of the default reason.

Only the code and reason are actual fields in the object; the status
is a virtual field.

See {{http-status-codes}} for a list of all known default statuses.

<procedure>(update-response old-response #!key port status code reason major minor headers)</procedure>

Like {{make-response}}, except this takes an {{old-response}} object as a
template for values which are missing from the parameter list, thereby
providing a way to do a purely functional update of that object.

<procedure>(response-port RESPONSE) => PORT</procedure><br>
<procedure>(response-code RESPONSE) => NUMBER</procedure><br>
<procedure>(response-reason RESPONSE) => STRING</procedure><br>
<procedure>(response-class RESPONSE-OR-CODE) => NUMBER</procedure><br>
<procedure>(response-major RESPONSE) => NUMBER</procedure><br>
<procedure>(response-minor RESPONSE) => NUMBER</procedure><br>
<procedure>(response-headers RESPONSE) => HEADERS</procedure><br>
<procedure>(response-status RESPONSE-OR-CODE) => SYMBOL</procedure><br>

An existing response can be picked apart using these accessors.

The PORT, MAJOR, MINOR and HEADERS are the same as for requests. CODE
and REASON are an integer status code and the short message that
belongs to it, as defined in the spec (examples include: 200 OK, 301
Moved Permanently, etc).  CLASS is the major class of the response
code (100, 200, 300, 400 or 500).  {{response-class}} can be called
either on a response object or directly on a response code number.

{{response-status}} attempts to fetch the symbolic status from the
response object based on its response code.  If no matching symbol can
be found in the {{http-status-codes}} parameter, an exception is thrown.
{{response-status}} can also be called on a response object or directly
on a response code number.

<procedure>(response-port-set! RESPONSE PORT)</procedure><br>
<procedure>(response-code-set! RESPONSE NUMBER)</procedure><br>
<procedure>(response-reason-set! RESPONSE STRING)</procedure><br>
<procedure>(response-major-set! RESPONSE NUMBER)</procedure><br>
<procedure>(response-minor-set! RESPONSE NUMBER)</procedure><br>
<procedure>(response-headers-set! RESPONSE HEADERS)</procedure><br>
<procedure>(response-status-set! RESPONSE SYMBOL)</procedure><br>

These procedures mutate an existing response object and set the
corresponding slot. {{response-status-set!}} will attempt to look up
the code and reason in {{http-status-codes}} and set both slots.
If the symbolic status is unknown, an exception is thrown.

<parameter>(http-status-codes [ALIST])</parameter>

This is an alist mapping symbolic status indicators to HTTP codes and
reason strings.

These can be used to make your code a bit more expressive and to
reduce duplication of hardcoded strings; instead of using a numeric
"magic number" HTTP code plus the same human-readable string
everywhere the same code occurs, you can instead use a descriptive
symbol.

The default value of this mapping is as follows:

<enscript highlight=scheme>
'((continue . (100 . "Continue"))
  (switching-protocols . (101 . "Switching Protocols"))
  (ok . (200 . "OK"))
  (created . (201 . "Created"))
  (accepted . (202 . "Accepted"))
  (non-authoritative-information . (203 . "Non-Authoritative Information"))
  (no-content . (204 . "No Content"))
  (reset-content . (205 . "Reset Content"))
  (partial-content . (206 . "Partial Content"))
  (multiple-choices . (300 . "Multiple Choices"))
  (moved-permanently . (301 . "Moved Permanently"))
  (found . (302 . "Found"))
  (see-other . (303 . "See Other"))
  (not-modified . (304 . "Not Modified"))
  (use-proxy . (305 . "Use Proxy"))
  (temporary-redirect . (307 . "Temporary Redirect"))
  (bad-request . (400 . "Bad Request"))
  (unauthorized . (401 . "Unauthorized"))
  (payment-required . (402 . "Payment Required"))
  (forbidden . (403 . "Forbidden"))
  (not-found . (404 . "Not Found"))
  (method-not-allowed . (405 . "Method Not Allowed"))
  (not-acceptable . (406 . "Not Acceptable"))
  (proxy-authentication-required . (407 . "Proxy Authentication Required"))
  (request-time-out . (408 . "Request Time-out"))
  (conflict . (409 . "Conflict"))
  (gone . (410 . "Gone"))
  (length-required . (411 . "Length Required"))
  (precondition-failed . (412 . "Precondition Failed"))
  (request-entity-too-large . (413 . "Request Entity Too Large"))
  (request-uri-too-large . (414 . "Request-URI Too Large"))
  (unsupported-media-type . (415 . "Unsupported Media Type"))
  (request-range-not-satisfiable . (416 . "Requested range not satisfiable"))
  (expectation-failed . (417 . "Expectation Failed"))
  (internal-server-error . (500 . "Internal Server Error"))
  (not-implemented . (501 . "Not Implemented"))
  (bad-gateway . (502 . "Bad Gateway"))
  (service-unavailable . (503 . "Service Unavailable"))
  (gateway-time-out . (504 . "Gateway Time-out"))
  (http-version-not-supported . (505 . "HTTP Version not supported")))
</enscript>

<procedure>(write-response RESPONSE) => RESPONSE</procedure>

Write the response object RESPONSE to the {{response-port}}.

If there is a response body, this must be written to the response-port
after sending the response headers.

<procedure>(read-response PORT) => RESPONSE</procedure>

Reads a response object from the port. An optional response body can
be read from the response-port after calling this procedure.

<parameter>(response-parsers [LIST])</parameter>

Responses are parsed using parse procedures, which can be customized
by overriding this parameter.

LIST is a list one of procedures which accept a response line string
and produce a response object, or {{#f}} if the response is not of the
type handled by that procedure.

The predefined response parsers are:

* {{http-0.9-response-parser}}
* {{http-1.x-response-parser}}

<procedure>(http-0.9-response-parser REQUEST) => REQUEST</procedure><br>
<procedure>(http-1.x-response-parser REQUEST) => REQUEST</procedure><br>

Predefined response parsers for use with {{response-parser}}.

<parameter>(response-unparsers [LIST])</parameter>

Responses are written using unparse procedures, which can be
customized by overriding this parameter.

LIST is a list of procedures which accept a response object and write
to the response's output port and return the new, possibly updated
response object. If the response object is not unparsed by this
handler, it returns {{#f}}.

The predefined response unparsers are the following:

* {{http-0.9-response-unparser}}
* {{http-1.x-response-unparser}}

<procedure>(http-0.9-response-unparser REQUEST) => REQUEST</procedure><br>
<procedure>(http-1.x-response-unparser REQUEST) => REQUEST</procedure><br>

Predefined response unparsers for use with {{response-unparser}}.

=== Headers

<procedure>(headers ALIST [HEADERS]) => HEADERS</procedure>

This creates a header object based on an input list.

Requests and responses contain HTTP headers wrapped in a special
header-object to ensure they are properly normalized.

The input list has header names (symbols) as keys, and
lists of values as values:

<enscript highlight="scheme">
(headers `((host ("example.com" . 8080))
           (accept #(text/html ((q . 0.5)))
                   #(text/xml ((q . 0.1)))))
          old-headers)
</enscript>

This adds the named headers to the existing headers in
{{old-headers}}. The host header is a pair of hostname/port.
The accept header is a list of allowed mime-type symbols. 

As can be seen here, optional parameters or "attributes" can be added
to a header value by wrapping the value in a vector of length 2. The
first entry in the vector is the header value, the second is an alist
of attribute name/value pairs.

<procedure>(headers->list HEADERS) => ALIST</procedure>

This converts a header object back to a list.  See {{headers}}
for details.

<procedure>(header-values NAME HEADERS) => LIST</procedure>

Obtain the value of header NAME in the HEADERS object.

The NAME of the header is a symbol; this procedure will return all the values
of the header (for example, the Accept header will have several values
that indicate the set of acceptable mime-types).

<procedure>(header-value NAME HEADERS [DEFAULT]) => value</procedure>

If you know in advance that a header has only one value, you can use
{{header-value}} instead of {{header-values}}.  This will return the
first value in the list, or the provided default if there is no value
for that header.

<procedure>(header-params NAME HEADERS) => ALIST</procedure>

This will return all the params for a given header, assuming there is
only one header.  An empty list is returned if the header does not exist.

<procedure>(header-param PARAM NAME HEADERS [DEFAULT]) => value</procedure>

This will return a specific parameter for the header, or DEFAULT
if the parameter isn't present or the header does not exist.  This
also assumes there's only one header.

<procedure>(header-contents NAME HEADERS) => LIST</procedure><br>
<procedure>(get-value VECTOR) => value</procedure><br>
<procedure>(get-params VECTOR) => ALIST</procedure><br>
<procedure>(get-param PARAM VECTOR [DEFAULT]) => value</procedure><br>

Procedures such as {{header-values}} are just shortcuts; these are
the underlying procedures to query the raw contents of a header.

Header contents are lists of 2-element vectors; the first value
containing the value for the header and the second value containing
an alist with "parameters" for that header value. Parameters are
attribute/value pairs that define further specialization of a header's
value. For example, the {{accept}} header consists of a list of
mime-types, which optionally can have a quality parameter that
defines the preference for that mime-type.  All parameter names
are downcased symbols, just like header names.

Here's a few examples on how to retrieve info from headers:

<enscript highight="scheme">
;; This would be returned by a server and retrieved via (response-headers r):
(define example-headers
  (headers '((accept #(text/html ((q . 0.1)))
                     #(text/xml ((q . 0.5)))
                     text/plain)
             (allow HEAD GET)
             (content-type #(text/html ((charset . utf-8))))
             (max-forwards 2))))

;;; Basic procedures
(define c (header-contents 'accept example-headers))
c ; => (#(text/html ((q . 0.5))) #(text/xml ((q . 0.1))) #(text/plain ()))

(get-value (car c))
; => text/html
(get-params (car c))
; => ((q . 0.5))
(get-param 'q (car c))
; => 0.5

;;; Simplified helpers
(header-values 'accept example-headers)
; => (text/html text/xml text/plain)
(header-values 'max-forwards example-headers)
; => (2)
(header-values 'nonexistent-header example-headers)
; => ()

;; This assumes there's only one value (returns the first)
(header-value 'max-forwards example-headers)
; => 2
(header-value 'nonexistent-header example-headers)
; => #f
(header-value 'nonexistent-header example-headers 'not-here)
; => not-here
;; Tricky:
(header-value 'accept example-headers)
; => text/html

;; This is tricky: this just returns the first, which is not the preferred
(header-params 'accept example-headers)
; => ((q . 0.1))
;; Quick access
(header-param 'charset 'content-type example-headers)
; => utf-8
</enscript>

==== Header types

The headers all have their own different types.  Here follows a list
of headers with their value types:

<table>
<tr><th>Header name</th><th>Value type</th><th>Example value</th></tr>
<tr>
<td>{{accept}}</td>
<td>List of mime-types (symbols), with optional {{q}} attribute
indicating "quality" (preference level)</td>
<td>{{(text/html #(text/xml ((q . 0.1))))}}</td>
</tr>
<tr>
<td>{{accept-charset}}</td>
<td>List of charset-names (symbols), with optional {{q}} attribute</td>
<td>{{(utf-8 #(iso-8859-5 ((q . 0.1))))}}</td>
</tr>
<tr>
<td>{{accept-encoding}}</td>
<td>List of encoding-names (symbols), with optional {{q}} attribute</td>
<td>{{(gzip #(identity ((q . 0))))}}</td>
</tr>
<tr>
<td>{{accept-language}}</td>
<td>List of language-names (symbols), with optional {{q}} attribute</td>
<td>{{(en-gb #(nl ((q . 0.5))))}}</td>
</tr>
<tr>
<td>{{accept-ranges}}</td>
<td>List of range types acceptable (symbols). The spec only defines
{{bytes}} and {{none}}.</td>
<td>{{(bytes)}}</td>
</tr>
<tr>
<td>{{age}}</td>
<td>Age in seconds (number)</td>
<td>{{(3600)}}</td>
</tr>
<tr>
<td>{{allow}}</td>
<td>List of methods that are allowed (symbols).</td>
<td>{{(GET POST PUT DELETE)}}</td>
</tr>
<tr>
<td>{{authorization}}</td>
<td>Authorization information. This consists of a symbol identifying the
authentication scheme, with scheme-specific attributes.
{{basic}} is handled specially, as if it were a regular symbol with two
attributes; {{username}} and {{password}}.</td>
<td>{{(#(basic ((username . "foo") (password . "bar"))) #(digest ((qop . auth) (username . "Mufasa") (nc . 1))))}}</td>
</tr>
<tr>
<td>{{cache-control}}</td>
<td>An alist of key/value pairs. If no value is applicable, it is {{#t}}</td>
<td>{{((public . #t) (max-stale . 10) (no-cache . (age set-cookie)))}}</td>
</tr>
<tr>
<td>{{connection}}</td>
<td>A list of connection options (symbols)</td>
<td>{{(close)}}</td>
</tr>
<tr>
<td>{{content-disposition}}</td>
<td>A symbol indicating the disposition</td>
<td>{{(#(inline ((filename . "test.pdf"))))}}</td>
</tr>
<tr>
<td>{{content-encoding}}</td>
<td>A list of encodings (symbols) applied to the entity-body.</td>
<td>{{(deflate gzip)}}</td>
</tr>
<tr>
<td>{{content-language}}</td>
<td>The natural language(s) of the "intended audience" (symbols)</td>
<td>{{(de nl en-gb)}}</td>
</tr>
<tr>
<td>{{content-length}}</td>
<td>The number of bytes (an exact number) in the entity-body</td>
<td>{{(10)}}</td>
</tr>
<tr>
<td>{{content-location}}</td>
<td>A location that the content can be retrieved from (a uri-common object)</td>
<td>{{(<#uri-common# ...>)}}</td>
</tr>
<tr>
<td>{{content-md5}}</td>
<td>The MD5 checksum (a string) of the entity-body</td>
<td>{{("12345ABCDEF")}}</td>
</tr>
<tr>
<td>{{content-range}}</td>
<td>Content range (pair with start- and endpoint) of the entity-body, if partially sent</td>
<td>{{((25 . 120))}}</td>
</tr>
<tr>
<td>{{content-type}}</td>
<td>The mime type of the entity-body (a symbol)</td>
<td>{{(#(text/html ((charset . iso-8859-1))))}}</td>
</tr>
<tr>
<td>{{date}}</td>
<td>A timestamp (10-element vector, see {{string->time}}) at which the message originated. ''Important'': Note that you will always need to supply (an empty list of) attributes, because otherwise it is ambiguous whether it's a vector with attribs or a bare timestamp.</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{etag}}</td>
<td>An entity-tag (pair, car being either the symbol weak or strong, cdr being a string) that uniquely identifies the resource contents.</td>
<td>{{((strong . "foo123"))}}</td>
</tr>
<tr>
<td>{{expect}}</td>
<td>Expectations of the server's behaviour (alist of symbol-string pairs), possibly with parameters.</td>
<td>{{(#(((100-continue . #t)) ()))}}</td>
</tr>
<tr>
<td>{{expires}}</td>
<td>Expiry timestamp (10-element vector, see {{string->time}}) for the entity. Also see the note for {{date}}</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{from}}</td>
<td>The e-mail address (a string) of the human user who controls the client</td>
<td>{{("info@example.com")}}</td>
</tr>
<tr>
<td>{{host}}</td>
<td>The host to use (for virtual hosting). This is a pair of hostname and port. The port will be {{#f}} if the port should be the default one for the requested service.</td>
<td>{{(("example.com" . 8080))}}</td>
</tr>
<tr>
<td>{{if-match}}</td>
<td>Either {{'*}} (a wildcard symbol) or a list of entity-tags (pair, weak/strong symbol and unique entity identifier string).</td>
<td>{{((strong . "foo123") (strong . "bar123"))}}</td>
</tr>
<tr>
<td>{{if-modified-since}}</td>
<td>Timestamp (10-element vector, see {{string->time}}) which indicates since when the entity must have been modified.</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{if-none-match}}</td>
<td>Either {{'*}} (a wildcard symbol) or a list of entity-tags (pair, weak/strong symbol and unique entity identifier symbol).</td>
<td>{{((strong . foo123) (strong . bar123))}}</td>
</tr>
<tr>
<td>{{if-range}}</td>
<td>The range to request, if the entity was unchanged</td>
<td>TODO</td>
</tr>
<tr>
<td>{{if-unmodified-since}}</td>
<td>A timestamp (10-element vector, see {{string->time}}) since which the entity must not have been modified</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{last-modified}}</td>
<td>A timestamp (10-element vector, see {{string->time}}) when the entity was last modified</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{location}}</td>
<td>A location (an URI object) to which to redirect</td>
<td>{{(<#uri-object ...>)}}</td>
</tr>
<tr>
<td>{{max-forwards}}</td>
<td>The maximum number of proxies that can forward a request</td>
<td>{{(2)}}</td>
</tr>
<tr>
<td>{{pragma}}</td>
<td>An alist of symbols containing implementation-specific directives.</td>
<td>{{((no-cache . #t) (my-extension . my-value))}}</td>
</tr>
<tr>
<td>{{proxy-authenticate}}</td>
<td>Proxy authentication request.  Equivalent to {{www-authenticate}}, for proxies.</td>
<td>{{(#(basic ((realm . "foo")) ) #(digest ((realm . "foo") (domain . (<#uri object> <#uri object>)) (qop . (auth auth-int)) (nonce . "012345abc"))))}}</td>
</tr>
<tr>
<td>{{proxy-authorization}}</td>
<td>The answer to a {{proxy-authentication}} request. Equivalent to {{authorization}}, for proxies.</td>
<td>{{(#(basic ((username . "foo") (password . "bar"))) #(digest ((qop . auth) (username . "Mufasa") (nc . 1))))}}</td>
</tr>
<tr>
<td>{{range}}</td>
<td>The range of bytes (a pair of start and end) to request from the server.</td>
<td>{{((25 . 120))}}</td>
</tr>
<tr>
<td>{{referer}}</td>
<td>The referring URL (uri-common object) that linked to this one.</td>
<td>{{(<#uri-object ...>)}}</td>
</tr>
<tr>
<td>{{retry-after}}</td>
<td>Timestamp (10-element vector, see {{string->time}}) after which to retry the request if unavailable now.</td>
<td>{{(#(#(42 23 15 20 6 108 0 309 #f 0) ()))}}</td>
</tr>
<tr>
<td>{{server}}</td>
<td>List of products the server uses (list of 3-tuple lists of strings; product name, product version, comment. Version and/or comment may be {{#f}}). Note that this is a single header, with a list inside it!</td>
<td>{{((("Apache" "2.2.9" "Unix") ("mod_ssl" "2.2.9" #f) ("OpenSSL" "0.9.8e" #f) ("DAV" "2" #f) ("mod_fastcgi" "2.4.2" #f) ("mod_apreq2-20051231" "2.6.0" #f)))}}</td>
</tr>
<tr>
<td>{{te}}</td>
<td>Allowed transfer-encodings (symbols, with optional q attribute) for the response</td>
<td>{{(deflate #(gzip ((q . 0.2))))}}</td>
</tr>
<tr>
<td>{{trailer}}</td>
<td>Names of header fields (symbols) available in the trailer/after body</td>
<td>{{(range etag)}}</td>
</tr>
<tr>
<td>{{transfer-encoding}}</td>
<td>The encodings (symbols) used in the body</td>
<td>{{(chunked)}}</td>
</tr>
<tr>
<td>{{upgrade}}</td>
<td>Product names to which must be upgraded (strings)</td>
<td>TODO</td>
</tr>
<tr>
<td>{{user-agent}}</td>
<td>List of products the user agent uses (list of 3-tuple lists of strings; product name, product version, comment. Version and/or comment may be {{#f}}). Note that this is a single header, with a list inside it!</td>
<td>{{((("Mozilla" "5.0" "X11; U; NetBSD amd64; en-US; rv:1.9.0.3") ("Gecko" "2008110501" #f) ("Minefield" "3.0.3" #f)))}}</td>
</tr>
<tr>
<td>{{vary}}</td>
<td>The names of headers that define variation in the resource body, to determine cachability (symbols)</td>
<td>{{(range etag)}}</td>
</tr>
<tr>
<td>{{via}}</td>
<td>The intermediate hops through which the message is forwarded (strings)</td>
<td>TODO</td>
</tr>
<tr>
<td>{{warning}}</td>
<td>Warning code for special status</td>
<td>TODO</td>
</tr>
<tr>
<td>{{www-authenticate}}</td>
<td>If unauthorized, a challenge to authenticate (symbol, with attributes)</td>
<td>{{(#(basic ((realm . "foo"))) #(digest ((realm . "foo") (domain . (<#uri object> <#uri object>)) (qop . (auth auth-int)) (nonce . "012345abc"))))}}</td>
</tr>
<tr>
<td>{{set-cookie}}</td>
<td>Cookies to set (name/value pair (both strings), with attributes)</td>
<td>{{(#(("foo" . "bar") ((max-age . 10) (port . '(80 8080))))}}</td>
</tr>
<tr>
<td>{{cookie}}</td>
<td>Cookies that were set (name/value string pair, with attributes)</td>
<td>{{(#(("foo" . "bar") ((version . 1) (path . #(uri path: (/ ""))) (domain . "foo.com"))))}}</td>
</tr>
<tr>
<td>{{x-forwarded-for}}</td>
<td>The chain of IP addresses which each intermediate proxy will add to.  Plain strings representing the IP-address or "unknown" when the proxy couldn't determine the client address or the option is disabled. ''Never accept this value without question; it can easily be spoofed!''</td>
<td>{{("192.168.1.2" "unknown" "123.456.789.012" "some-made-up-value-by-an-attacker")}}</td>
</tr>

</table>

Any unrecognised headers are assumed to be multi-headers, and the
entire header lines are put unparsed into a list, one entry per line.


==== Header parsers and unparsers

<parameter>(header-parsers [ALIST])</parameter><br>
<parameter>(header-unparsers [ALIST])</parameter><br>

The parsers and unparsers used to read and write header values can be
customized with these parameters.

These (un)parsers are indexed with as key the header name (a symbol)
and the value being a procedure.

A header parser accepts the contents of the header (a string, without
the leading header name and colon) and returns a ''list of vectors''
which represents the values of the header.  For headers that are
supposed to only have a single value, the last value in the list will
be stored as the value (as determined by {{single-headers}}).

A header unparser accepts one argument: the header's contents (a
vector).  It should return a list of strings, each of which represents
one line's worth of header contents (without the header name).  For
each entry, a header line will automatically be printed with the
header name preceding it.

The parser driver will call {{update-header-contents!}} with the
parser's result.

<parameter>(header-parse-error-handler [HANDLER])</parameter>

When there is an error parsing a given header, this
parameter's procedure will be invoked.

{{HANDLER}} is a procedure accepting four values: the header name, the
header contents, the current headers and the exception object.  The
procedure must return the new headers.  Defaults to a procedure that
simply returns the current headers.  When an error occurs while
parsing the header line itself (for example when a colon is missing
between the header name and contents), the error will not be caught.

In such a case, Servers should return a 400 Bad Request error and
clients should error out.  The reason that malformed error lines are
ignored is that there are several servers and clients that send
headers content values that are slightly off, even though the rest of
the request is OK.  In the interest of the "robustness principle",
it's best to simply ignore these headers with "bad" content values.

<procedure>(replace-header-contents NAME CONTENTS HEADERS) => HEADERS</procedure><br>
<procedure>(replace-header-contents! NAME CONTENTS HEADERS) => HEADERS</procedure><br>
<procedure>(update-header-contents NAME CONTENTS HEADERS) => HEADERS</procedure><br>
<procedure>(update-header-contents! NAME CONTENTS HEADERS) => HEADERS</procedure><br>

The {{replace}} procedures replace any existing contents of the named
header with new ones, the {{update}} procedures add these contents to
the existing header. The procedures with a name ending in bang are
linear update variants of the ones without the bang. The header
contents have to be normalized to be a 2-element vector, with the
first element being the actual value and the second element being an
alist (possibly empty) of parameters/attributes for that value.

The update procedures append the value to the existing header if it is
a multi-header, and act as a simple replace in the case of a
single-header.

<parameter>(single-headers [LIST])</parameter>

Whether a header is allowed once or multiple times in a request or
response is determined by this parameter.

The value is a list of symbols that define header-names which are
allowed to occur only once in a request/response.

<procedure>(http-name->symbol STRING) => SYMBOL</procedure><br>
<procedure>(symbol->http-name SYMBOL) => STRING</procedure><br>

These procedures convert strings containing the name of a header or
attribute (parameter name) to symbols representing the same. The
symbols are completely downcased.  When converting this symbol back to
a string, the initial letters of all the words in the header name or
attribute are capitalized.

<procedure>(remove-header name headers) => headers</procedure><br>
<procedure>(remove-header! name headers) => headers</procedure><br>

These two procedures remove all headers with the given name.

===== Header subparsers and subunparsers

Some headers are modular ''themselves''.  This means they need some
way to extend them.  This is done through subparsers and subunparsers.

<parameter>(authorization-param-subparsers [ALIST])</parameter>

This is an alist of subtypes for the {{authorization}} header parser.
A subparser of this kind accepts the string containing the header and
an integer position in the string.  It should parse from that position
onwards, and return the parsed contents as an alist of header
parameters.  Usually, these are actually pseudo-parameters; they don't
necessarily have to appear in parameter syntax in the header.  The
unparser should be configured to expect the same parameters and combine
them back into a string, though.

This parameter defaults to:

<enscript highligh=scheme>
`((basic . ,basic-auth-subparser)
  (digest . ,digest-auth-subparser))
</enscript>

<procedure>(basic-auth-param-subparser STR POS)</procedure>

Parses {{STR}} at {{POS}} by extracting the username and password
components from a base64-encoded string.  These are returned in its
first value as an alist with keys {{username}} and {{password}}.  Its
second return value is the position after which the next header value
may begin.

<procedure>(digest-auth-param-subparser STR POS)</procedure>

Parses {{STR}} at {{POS}} by reading the various components from a
parameter list.  These are returned in its first return value as an
alist with keys {{nc}}, {{uri}}, {{qop}} and {{algorithm}}.  Its
second return value is the position after which the next header value
may begin.

<parameter>(authorization-param-subunparsers [ALIST])</parameter>

This is an alist of subtypes for the {{authorization}} header
unparser.  An unparser of this kind accepts an alist containing the
parameters that it needs to unparse and should return a string
containing the raw unparsed parameters only.

This parameter defaults to:

<enscript highligh=scheme>
`((basic . ,basic-auth-subunparser)
  (digest . ,digest-auth-subunparser))
</enscript>

<procedure>(basic-auth-param-subunparser PARAMS)</procedure>

This unparses the {{PARAMS}} alist into a base64-encoded string for
basic authentication.  It expects {{username}} and {{password}}
parameters.

<procedure>(digest-auth-param-subunparser PARAMS)</procedure>

This unparses the {{PARAMS}} alist into a string for digest
authentication. It expects {{username}}, {{uri}}, {{realm}},
{{nonce}}, {{cnonce}}, {{qop}}, {{nc}}, {{response}}, {{opaque}} and
{{algorithm}} parameters.  The {{response}} parameter should be
pre-encoded in the way digest auth expects (this is not done here
because the MD5 sum of the contents may be required, which is not
available to the parsers).

TODO: This will probably change in the future; the md5 can be passed
and all the hard stuff can be done in intarweb.

=== Other procedures

<parameter>(http-line-limit [length])</parameter>

The maximum length of any line that's read by intarweb as part of the
request/response cycle.  This includes the request and response lines
as well as the headers.  If this is exceeded, an exception of type
{{(exn http line-limit-exceeded)}} is raised.

You can set this to {{#f}} to disable this check. However, this will
open up a resource consumption vulnerability (attackers can cause
your application to blow up by letting it use all available memory).

Defaults to {{1024}}.

<parameter>(http-header-limit [count])</parameter>

The maximum number of headers that are allowed to be sent, as part
of a request or response. If this is exceeded, an exception of type
{{(exn http header-limit-exceeded)}} is raised.

You can set this to {{#f}} to disable this check. However, this will
open up a resource consumption vulnerability (attackers can cause
your application to blow up by letting it use all available memory).

Defaults to {{256}}.

<procedure>(keep-alive? request-or-response)</procedure>

Returns {{#t}} when the given request or response object belongs to a
connection that should be kept alive, {{#f}} if not.  Remember that
both parties must agree on whether the connection is to be kept alive
or not; HTTP/1.1 defaults to keep alive unless a {{Connection: close}}
header is sent, HTTP/1.0 defaults to closing the connection, unless a
{{Connection: Keep-Alive}} header is sent.

<parameter>(request-has-message-body? [predicate])</parameter>

This parameter holds a predicate which accepts a request object and
returns {{#t}} when the request will have a message body.  By default
in HTTP/1.1, this is the case for all requests that have a {{content-length}}
or {{transfer-coding}} header.

The parameter is useful for servers to determine whether to read a
request body or not.

<procedure>(read-urlencoded-request-data request [max-length])</procedure>

Convenience procedure to read URLencoded request data (regular POST
data; ''not'' multipart data!) from the given {{request}} object.  It
will return an alist, as would be returned by {{form-urldecode}} from
the [[uri-common]] egg.

You have to take care of checking the request type whether there
really will be request data yourself (it can optionally use
{{request-has-message-body?}} for this, but it's probably advisable to
check the request type anyway).

This will read at most {{max-length}} bytes.  If not specified,
{{max-length}} defaults to the current value of
{{http-urlencoded-request-data-limit}}.  If this maximum is exceeded,
an exception of type {{(exn http urlencoded-request-data-limit-exceeded)}}
is raised.

<parameter>(http-urlencoded-request-data-limit [length])</parameter>

Set the default limit for request body data.  Defaults to 4194304 (4MB).

<parameter>(response-has-message-body-for-request? [predicate])</parameter>

This parameter holds a predicate which accepts two arguments: a
response object and a request object.  It returns {{#t}} when the
response will have a message body for the given request.  By default
in HTTP/1.1, this is '''not''' the case for responses with a
response-code of 204 and 304 or in the 1xx class, nor for {{HEAD}}
requests.  All other responses will have a message body.

The parameter is useful for deciding in clients whether a message body
will follow (otherwise, trying to read will probably result in an error
or in case of HTTP pipelining in a synchronisation problem)

<procedure>(safe? request-or-method)</procedure>

Returns {{#t}} when the given request object or symbol (method) is a
''safe'' method.  A method is defined to be safe when a request of
this method will have no side-effects on the server.  In practice this
means that you can send this request from anywhere at any time and
cause no damage.

'''Important''': Quite a lot of software does not abide by these
rules!  This is not necessarily a reason to treat all methods as
unsafe, however.  In the words of the standard "the user did not
request the side-effects, so therefore cannot be held accountable for
them".  If a safe method produces side-effects, that's the server-side
script developer's fault and he should fix his code.

<parameter>(safe-methods [symbols])</parameter>

A list of methods which are to be considered safe.  Defaults to
{{'(GET HEAD OPTIONS TRACE)}}.

<procedure>(idempotent? request-or-method)</procedure>

Returns {{#t}} when the given request object or symbol (method) is a
''idempotent'' method.  A method is defined to be idempotent when a
series of identical requests of this method in succession causes the
exact same side-effect as just one such request.  In practice this
means that you can safely retry such a request when an error occurs,
for example.

'''Important''': Just as with the ''safe'' methods, there is no
guarantee that methods that ''should be'' idempotent really are
idempotent in any given web application.  Furthermore, a sequence of
requests which each are individually idempotent is not necessarily
idempotent as a whole.  This means that you cannot replay requests
starting anywhere in the chain.  To be on the safe side, only retry
the last request in the chain.

<parameter>(idempotent-methods [symbols])</parameter>

A list of methods which are to be considered idempotent.  Defaults to
{{'(GET HEAD PUT DELETE OPTIONS TRACE)}}.

<procedure>(etag=? a b)</procedure>

Do the etag values {{a}} and {{b}} strongly match?  That is, their
{{car}} and {{cdr}} must be equal, and neither can have a {{car}} of
{{weak}} (both must be {{strong}}).

<procedure>(etag=-weakly? a b)</procedure>

Do the etag values {{a}} and {{b}} weakly match?  That is, their
{{car}} and {{cdr}} must be equal.  A {{car}} of {{weak}} is allowed.

<procedure>(etag-matches? etag matches)</procedure>

Does the {{etag}} strongly match any of the etags in the list
{{matches}}?  {{matches}} is a plain list of etag values, but it can
also contain the special symbol {{*}}, which matches any etag.

<procedure>(etag-matches-weakly? etag matches)</procedure>

Does the {{etag}} weakly match any of the etags in the list
{{matches}}?  {{matches}} is a plain list of etag values, but it can
also contain the special symbol {{*}}, which matches any etag.

=== Changelog

* 0.8 Treat the charset attribute for Content-Type header as case-insensitive token for consistency with Accept-Charset header.  Remove dependency on the [[/eggref/4/regex|regex egg]] and improve correctness of a few parsers.  Add {{request-has-message-body?}} and {{response-has-message-body-for-request?}} procedures.  Add parser for Content-Disposition header and improve unparser by adding date support (Thanks to Evan Hanson).  Implement line length and header count limit checking.  Add {{read-urlencoded-request-data}} with built-in limit check.
* 0.7 Add trivial {{x-forwarded-for}} "parser".  Add easier overriding of {{authorization}} headers through parameter instead of having to rewrite the entire parser.  Add {{content-disposition}} unparser to accommodate the fact that filenames ''must'' always be quoted.  Add {{http-status-codes}} parameter and {{status:}} key to {{update-response}} and {{make-response}} procedures, as well as {{response-status}} and {{response-status-set!}} procedures.
* 0.6 Change path parameters on cookies to be uri-common objects
* 0.5 Add regex requirement to make it work with Chicken 4.6.2+
* 0.4 Don't unparse "attributes" (aka "params") by titlecasing their names. Don't default Host header's port to 80, but use #f
* 0.3 Add rfc1123 to default unparser list for the 'date' header. Add etag procedures and if-match unparser. Change procedure signature for header unparsers
* 0.2 Make cookie header parsers/unparsers preserve cookie name case. Change header unparse procedure semantics slightly.
* 0.1 Initial version

=== License

  Copyright (c) 2008-2012, Peter Bex
  All rights reserved.
  
  Redistribution and use in source and binary forms, with or without
  modification, are permitted provided that the following conditions are
  met:
  
  Redistributions of source code must retain the above copyright
  notice, this list of conditions and the following disclaimer.
  
  Redistributions in binary form must reproduce the above copyright
  notice, this list of conditions and the following disclaimer in the
  documentation and/or other materials provided with the distribution.
  
  Neither the name of the author nor the names of its contributors may
  be used to endorse or promote products derived from this software
  without specific prior written permission.
  
  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
  FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
  COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
  INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
  (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
  HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
  OF THE POSSIBILITY OF SUCH DAMAGE.