Low-level web programming in Racket + a wiki in 500 lines

[article index] [] [@mattmight] [+mattmight] [rss]

Racket provides a simple yet flexible web framework for serving dynamic web content.

This post covers the basics for using the low-level layer of that framework, including:

  • using low-level responses
  • using X-Expression responses
  • enabling SSL
  • enabling basic authentication
  • handling form content
  • serving static files

I’ve combined all of these concepts together to create a “minimum viable” academic wiki for a small research lab.

It’s about 500 lines, and it supports:

  • LaTeX formatting for math
  • markdown syntax
  • version control for each page
  • remote/offline page editing
  • syntax-highlighting for code
  • user authentication
  • SSL-encrypted connections

Read below for the intro and the wiki.

Update: Many thanks to Jay McCarthy for pointing out several simplifications for the the code in the article and in the wiki.

Web programming in Racket

Racket has a rich, well-abstracted web programming framework, and the documention includes an excellent tutorial in which it constructs a blogging framework.

Unlike that tutorial, the code in this article avoids using the more advanced features – including continuations – opting instead to explore the low-level features which should be more familiar (and perhaps more comfortable) to programmers that have used other platforms, like node.

To get started, let’s create a simple “hello world” server using the lower-level HTTPD interface in Racket:

#lang racket 

(require web-server/servlet
         web-server/servlet-env)

(define (hello-servlet req)
  (response
   200                 ; response code
   #"OK"               ; response message
   (current-seconds)   ; timestamp
   TEXT/HTML-MIME-TYPE ; MIME type for content
   '()                 ; additional headers

   ; the following parameter accepts the output port
   ; to which the page should be rendered.
   (λ (client-out)
     (write-string "Hello, world!" client-out))))
  
; start serving:
(serve/servlet hello-servlet)

This program instantiates a servlet that runs at

http://localhost:8000/servlets/standalone.rkt

The procedure serve/servlet expects to receive a “servlet” – a function that maps incoming requests from clients into responses.

By altering the parameters to serve/servlet, we can give the servlet control over which paths it controls.

For instance, the paramater #:servlet-regexp accepts a regular expression to determine if it is responsible for a particular path.

By providing it with the blank regex #rx"", it will serve any path:

(serve/servlet hello-servelet
               #:servlet-regexp #rx"")

so that it will also serve at:

http://localhost:8000/ 

The keyword #:port provides control over the port, which should be set to 80 for the default web port or to 443 for https.

If you noticed the web server opening each time you run the program, you can turn that off with #:launch-browser #f.

A higher-level approach with X-Expressions

The Racket web server provides a simplified mechanism for crafting HTML-based responses with X-Expressions.

The form (response/xexpr <xexpr>) returns the HTML corresponding to the provided X-Expression:

#lang racket

(require web-server/servlet
         web-server/servlet-env)


(define (hello-servlet req)
  (response/xexpr
   `(html
     (head)
     (body
      (p "Hello, world!")))))
     
     
  
; start serving:
(serve/servlet hello-servlet)

An X-Expression (xexpr) is a transcription of XML into S-Expressions.

The relevant portion of the grammar for X-Expressions is small:

 <xexpr> ::= <string>
          |  (<symbol> (<attr> ...) <expr> ...)
          |  (<symbol> <expr> ...)
          |  <symbol>
          |  <char>

 <attr> ::= [<symbol> <string>]

(Full X-Expressions also support CDATA elements and comments.)

To use HTML5 or (or include any kind of DOCTYPE header to the response), the #:preamble keyword accepts a byte string:

#lang racket

(require web-server/servlet
         web-server/servlet-env)

(define (hello-servlet req)
  (response/xexpr
   #:preamble #"<!DOCTYPE html>"
   `(html
     (head)
     (body
      (p "Hello, world!")))))

(serve/servlet hello-servlet)

Serving static files

To serve static files requires the URI from the request and the path from the URI.

Serving static files also requires knowing the MIME type of the file’s content, which is easily handled as a table that maps file paths to extensions.

Putting this into practice, the following Racket program begins serving files from the current directory:

#lang racket

(require web-server/servlet
         web-server/servlet-env)

; serve files from this directory:
(define document-root
  (path->string (current-directory)))

; a table to map file extensions to MIME types:
(define ext=>mime-type
  #hash((#""     . #"text/html; charset=utf-8")
        (#"html" . #"text/html; charset=utf-8")
        (#"png"  . #"image/png")
        (#"rkt"  . #"text/x-racket; charset=utf-8")))

; a handler for static files:
(define (handle-file-request req)
  
  ; extract the URI from the request:
  (define uri (request-uri req))
  
  ; extract the resource from the URI:
  (define resource 
    (map path/param-path (url-path uri)))
  
  ; find the file location:
  (define file (string-append
                document-root
                "/" 
                (string-join resource "/")))
  
  (cond
    
    ; serve the file if it exists:
    [(file-exists? file)
     ; =>
     
     ; find the MIME type:
     (define extension (filename-extension file))
     (define mime-type
       (hash-ref ext=>mime-type extension 
                 (λ () TEXT/HTML-MIME-TYPE)))
     
     ; read the file contents:
     (define data (file->bytes file))
     
     ; construct the response:
     (response
      200 #"OK"
      (current-seconds)
      mime-type
      '()
      (λ (client-out)
        (write-bytes data client-out)))]
    
    ; send an error page otherwise:
    [else
     ; =>
     (response/xexpr
      #:code     404
      #:message  #"Not found"
      `(html
        (body
         (p "Not found"))))]))

(serve/servlet handle-file-request
               #:servlet-regexp #rx"")

Using the built-in static file handler

You can also delegate static-file handling to the Racket web server simply by specifying directories for static files with the keywoard #:extra-files-paths.

In the following code, it will also serve files directly from the current directory:

#lang racket

(require web-server/servlet
         web-server/servlet-env)


(define root (current-directory))

(define (hello-servlet req)
  (response
   200                 ; response code
   #"OK"               ; response message
   (current-seconds)   ; timestamp
   TEXT/HTML-MIME-TYPE ; MIME type for content
   '()                 ; additional headers

   (λ (client-out)
     (write-string "Hello, dynamic!" client-out))))
  
; start serving:
(serve/servlet hello-servlet
               #:servlet-path "/hello"
               #:extra-files-paths (list (build-path root)))

Handling forms

Forms are a fixture of web-based programming.

Fortunately, Racket has library routines that make extracting and processing form data straightforward.

The net/uri-codec library can parse form data from either the query string or posted data with the function form-urlencoded->alist, which maps a string to an associative list for the fields in the form.

This simple web app demonstrates this by presenting a form and printing back the data posted to it:

#lang racket

(require web-server/servlet
         web-server/servlet-env)

(require net/uri-codec)

(define (form-servlet req)
  
  (define uri (request-uri req))
  (define path (map path/param-path (url-path uri)))    
  (define page (car path))
  
  (cond 
    
    ; /form
    [(equal? page "form")
     (response/xexpr
      `(html
        (body
         (form ([method "POST"] [action "/print-form-data"])
               "user name: " (input ([type "text"] [name "user"]))
               (br)
               "comment: " (input ([type "text"] [name "comment"]))
               (br)
               (input ([type "submit"]))))))]
    
    ; /print-form-data
    [(equal? page "print-form-data")
  
     ; extract the form data:
     (define post-data (bytes->string/utf-8 (request-post-data/raw req)))
     
     ; convert to an alist:
     (define form-data (form-urlencoded->alist post-data))

     ; pull out the user and comment:
     (define name    (cdr (assq 'user form-data)))
     (define comment (cdr (assq 'comment form-data)))
     
     ; send back the extracted data:
     (response/xexpr
      `(html
        (body
         (p "Your name: " ,name)
         (p "You comment: " ,comment))))]
    
    ; another page?
    [else
     (response/xexpr
      `(html
        (body
         (p "Page not found!"))))]))
         
(serve/servlet form-servlet
               #:servlet-regexp #rx""
               #:servlet-path "/form")

Enabling HTTPS

Racket’s web server makes it delightfully easy to enable secure web serving with HTTPS.

To serve run a server with HTTPS, serve/servlet takes three additional parameters, #:ssl?, #:ssl-cert and #:ssl-key.

The following code assumes the certificate server-cert.pem and the private key private-key.pem are in the current directory:

#lang racket

(require web-server/servlet
         web-server/servlet-env)

(define root (path->string (current-directory)))

; path to the server certificate:
(define cert-path (string-append root "/server-cert.pem"))

; path to the private key:
(define key-path (string-append root "/private-key.pem"))

(define (hello-servlet req)
  (response/xexpr
   #:preamble #"<!DOCTYPE html>"
   `(html
     (head)
     (body
      (p "Hello, world!")))))

(serve/servlet hello-servlet
               #:servlet-regexp #rx""
               #:port 443
               #:ssl? #t
               #:ssl-cert cert-path
               #:ssl-key  key-path)

(The Continue tutorial for Racket provides a short example of how to generate a self-signed key and cert with OpenSSL.)

Basic authentication

Basic authentication has the client send a username and password in the clear back to the server. Do not use basic authentication unless also using HTTPS!

The program htpasswd, which is provided by Apache and available on most Unix systems, is commonly used to generate flat, hashed password files for use with web authentication.

For example, you can create a password file with one user with:

htpasswd -cs <passwd-file> <username>

The -s specifies SHA1 hashing of passwords, which is what this tutorial will support.

The library function request->basic-credentials can extract credentials from a request, and requesting credentials requires sending back a 401 Unauthorized response if the credentials aren’t provided or don’t match.

A very simple server that validates a user against a local file called passwd is straightforward:

#lang racket

(require web-server/servlet
         web-server/servlet-env)

(require file/sha1)
(require net/base64)

(define root (path->string (current-directory)))

; path to the passwd file:
(define passwd-file (string-append root "/passwd"))

(define (any? pred list)
  ; returns true iff any element matches pred:
  (match list
    ['()  #f]
     [(cons hd tl)
     (or (pred hd) (any? pred (cdr list)))]))

; a password checking routine:
(define (htpasswd-credentials-valid?
         passwd-file
         username
         password)
  ; checks if the given credentials match those in the database
  ; it assumes all entries as SHA1-encoded as in `htpasswd -s`.

  ; read in the lines from the password file:
  (define lines (call-with-input-file passwd-file 
                  (λ (port) (port->lines port))))
  
  ; convert the password to sha1:
  (define sha1-pass (sha1-bytes (open-input-bytes password)))
  
  ; then to base64 encoding:
  (define sha1-pass-b64 
    (bytes->string/utf-8 (base64-encode sha1-pass #"")))
  
  ; check if both the username and the password match:
  (define (password-matches? line)

      (define user:hash (string-split line ":"))
      
      (define user (car user:hash))
      (define hash (cadr user:hash))
      
      (match (string->list hash)
        ; check for SHA1 prefix
        [`(#\{ #\S #\H #\A #\} . ,hashpass-chars)
         (define hashpass (list->string hashpass-chars))
         (and (equal? username (string->bytes/utf-8 user)) 
              (equal? hashpass sha1-pass-b64))]))
  
  ; check to see if any line validates:
  (any? password-matches? lines))


(define (req->user req)
  ; extracts the user for this request
  (match (request->basic-credentials req)
    [(cons user pass)   user]
    [else               #f]))

(define (authenticated? req)
  ; checks if a request has valid credentials:
  (match (request->basic-credentials req)
    [(cons user pass)
     (htpasswd-credentials-valid? passwd-file user pass)]
    
    [else     #f]))

(define (hello-servlet req)
  
  (cond
    ; check for authentication:
    [(not (authenticated? req))
     (response
      401 #"Unauthorized" 
      (current-seconds) 
      TEXT/HTML-MIME-TYPE
      (list
       (make-basic-auth-header
        "Authentication required"
        ))
      void)]
    
    ; if authenticated, serve the page:
    [else
     (response/xexpr
      #:preamble #"<!DOCTYPE html>"
      `(html
        (head)
        (body
         (p "Hello, " ,(bytes->string/utf-8 (req->user req)) "!"))))]))


(serve/servlet hello-servlet
               #:servlet-regexp #rx"")

Putting it all together: A wiki

All of the above concepts are tied together in the implementation of a minimalist wiki.

The source code for the “uiki” is available on github.

Related reading

  • Realm of Racket is an introduction to programming in Racket with an emphasis on game programming.

'eof