Hopefully some people coming from a non-Haskell background will get something out of this, though the Haskell syntax is likely to be very WTF-inducing for those who haven’t seen it before.

I should begin with a few things that this guide is not about:

• Categories. The etymology of the word “monad” is a red herring. Trust me. Knowing a lot about category theory will make you a better programmer in the same way that playing a lot of checkers will make you better at chess: there’s probably some benefit, but it’s not a good way to get up and running with the basics.

  We will be treating monads as a design pattern instead of monoids in the category of endofunctors. The latter perspective is interesting to the people who like to design languages; the former perspective is interesting to people who like to write code.

• The full generality of monads in programming. Lots of things are monads, and I will be ignoring most of them. I’m going to focus on how monads can be used to translate a particular Java class into Haskell, and what it looks like as we add functionality to both versions of the code.
• Haskell evangelism. I’ve already written about the things in Haskell that I love. I love Haskell, and I use it to make my dreams come true. On the other hand, I’m not at all invested in whether or not you like Haskell.

  Though if you want to learn more about this language, I’d like to help you along your journey.

What I am going to talk about is how to use monads to do something in Haskell that is easy to do in Java.

Some idiomatic Java code and some non-idiomatic Haskell code

Let’s start by taking a look at a class in Java:

public class IntWrapper {
  private int m_i;

  public IntWrapper(int i) { m_i = i; }

  public void print() { System.out.println(m_i); }

  public void inc() { m_i++; }

  public void nextPrime() {
    while (!isPrime()) inc();
  }

  public boolean isPrime() {
    for (int t = 2; t < m_i; t++)
      if (m_i % t == 0)
        return false;
    return true;
  }
}

We’re going to re-implement this in Haskell using monads. To do this, we’re going to need to look at some facets of this code that are so obvious we usually wouldn’t even think to mention them:

• (Valid by construction) This class has a constructor. The constructor is there to make sure that the class methods have valid data to work with.
• (Data hiding) Once we’ve created a new IntWrapper object there is no way to get its private member m_i out: the only code that can access this member variable are the member methods.
• (Implicit scoping) Member methods can refer to m_i without specifying an object instance. This is because the variable m_i is implicitly this.m_i.
• (Encapsulation) Member methods can refer to m_i belonging to other instances, but methods in other classes (that is, methods not in the IntWrapper class) cannot do this.

How might a translation of this class look in Haskell? Here’s a “day one” approach:

module IntWrapper (intWrapper, printiw, inc, isPrime, nextPrime) where

data IntWrapper = IntWrapper Integer

-- We don't export the IntWrapper constructor because we want to
-- preserve encapsulation.  Because of this, we need to export a
-- function that wraps the constructor.  That's what intWrapper does.
intWrapper i = IntWrapper i

printiw (IntWrapper i) = putStrLn (show i)

inc (IntWrapper i) = IntWrapper (i+1)

isPrime (IntWrapper i) = isPrime' 2 i
  where isPrime' t i | t >= i         = True
                     | i `mod` t == 0 = False
                     | otherwise      = isPrime' (t+1) i

nextPrime iw = until isPrime inc iw

What makes this a “day one” approach? It’s all about the interface. Let’s do a compare and contrast between using the IntWrapper class in Java versus the IntWrapper data type in Haskell.

Java:

public class UseIntWrapper {
  public static void main(String[] argv) {
    IntWrapper myInt = new IntWrapper(20);
    myInt.print();
    for (int i = 0; i < 4; i++)
      myInt.inc();
    myInt.print();
    myInt.nextPrime();
    myInt.print();
  }
}

Here’s what this might look like in Haskell (again, a “day one” translation):

main =
     do let iw1 = intWrapper 20
        printiw iw1
        let iw2 = (iterate inc iw1) !! 4
        printiw iw2
        let iw3 = nextPrime iw2
        printiw iw3

This is utterly terrible. Because we aren’t mutating any of our variables, we’ve been forced to define three variables (iw1, iw2, iw3) to try and describe a succession of modified values. It’s obviously error-prone. In fact, when I did this translation, I typo’d my way into print iw2 for the last line instead of print iw3 (seriously). If you like this style, that’s cool, but if you want to tell me that I should like it as well, be prepared to get punched in the face.

The essential weakness of this style is that it requires us to manually thread the IntWrapper data through each of the functions. The Java code also does this (we need to refer to myInt throughout) but clearly Java’s support for mutable state means it isn’t as ugly as our Haskell code.

The difference between the two

Isn’t this an example of why Java has better syntax than Haskell? Well, no: our Java code looks better than our Haskell code because we have used good Java practices and bad Haskell practices.

It’s quite easy to write equally crappy code in Java. In fact, let’s do just that. When we’re done, we’ll compare the bad Java to the good Java, make a few observations, and apply those lessons to our Haskell code.

Here is the IntWrapper class, written so that

1. The m_i variable is final, thereby simulating the way in which Haskell prevents us from mutating variables, and
2. each of the methods will be static, thereby simulating the way in which our Haskell functions led us to have iw1, iw2, iw3 in our code.

Here is the resulting mess:

// IntWrapper2.java
public class IntWrapper2 {
  private final int m_i;

  public IntWrapper2(int i) { m_i = i; }

  public static void print(IntWrapper2 iw) { System.out.println(iw.m_i); }

  public static IntWrapper2 inc(IntWrapper2 iw) {
    return new IntWrapper2( iw.m_i + 1 );
  }

  public static IntWrapper2 nextPrime(IntWrapper2 iw) {
    int i = iw.m_i;
    while (!isPrimeHelper(i))
      i++;
    return new IntWrapper2(i);
  }

  public static boolean isPrime(IntWrapper2 iw) {
    return isPrimeHelper(iw.m_i);
  }

  private static boolean isPrimeHelper(int i) {
    for (int t = 2; t < i; t++)
      if (i % t == 0)
        return false;
    return true;
  }
}

// UseIntWrapper2.java
public class UseIntWrapper2 {
  public static void main(String[] argv) {
    IntWrapper2 iw1 = new IntWrapper2(20);
    IntWrapper2.print( iw1 );
    IntWrapper2 iw2 = iw1;
    for (int i = 0; i < 4; i++)
      iw2 = IntWrapper2.inc(iw2);
    IntWrapper2.print( iw2 );
    IntWrapper2 iw3 = IntWrapper2.nextPrime(iw2);
    IntWrapper2.print(iw3);
  }
}

Where we had originally made use of objects to pass state between function, in this crap code we are not doing any such thing. It looks strikingly similar to our Haskell code, but with extra verbosity (obvious fact: this isn’t an argument against Java, because in practice no one codes like this). Let’s analyze the differences between this crappy implementation and our original Java code.

An analogy and a lesson

In our original Java code, the IntWrapper class has a mutable member variable m_i. Our client code could then create a new IntWrapper, call it myInt, and repeatedly use the same variable as it did its calculations.

I want you to think of the line IntWrapper myInt = new IntWrapper(20) as a declaration of a context. It is saying “hey, make me a new context for carrying around an integer. Call it myInt.” Then I want you to read lines like myInt.print() as saying “go into the myInt context and print.” This is precisely what this code means. In other words, to some approximation, classes in Java provide us a way to define a stateful context.

This mode of thinking is important enough in what follows that I want to expand on it a bit. Let’s leave the world of programming for a moment and enter into an analogy. You and Sue are both talking about a mutual friend named Bob. You’ve both known Bob for years. Your conversation might look something like this:

Sue: Did you see the postcard Bob sent me from Alaska?
You: Yeah. I couldn’t believe what he wrote on the back!
Sue: Oh, the thing about the reindeer? That’s an old inside joke we had from when we were dating.
You: I didn’t know you guys had dated.
Sue: It was before we met. We broke up because he wanted kids and I didn’t.
You: Really? I never figured him to be the fatherhood type.

In this exchange there are a few things to observe:

• After Sue mentions Bob in the first line, for the rest of the conversation he is only mentioned by pronoun. The conversation implicitly carries the information that you’re talking about Bob, and indeed, it appears to carry the information about which Bob you’re talking about.
• Sue and Bob have their own contextual information (the inside joke) that wasn’t known to you until Sue alluded to it.
• For that matter, the information that the two of them had dated wasn’t known to you until Sue mentioned it.
• Before the end of the conversation, your mental model of Bob has shifted: you now know that he wants to be a father, whereas before you did not.

Structurally, when the conversation started, it was established that you and Sue are working in a context where “Bob” is a known entity. This is analogous to the Java statement IntWrapper myInt = new IntWrapper(20); which establishes myInt as a known entity. As Sue reveals information about Bob, changing your mental model of who he is (his state), she is speaking statements similar to myInt.nextPrime().

In our crappy Java code, we’ve basically abandoned this whole “context” idea.

So to fix our Haskell code, we’re going to do what we already were trained to do in Java: introduce a notion of context. To some approximation, this is what monads are about. A subtle point is that there are different notions of context that arise in programming, and different monads for modeling each. Following our conversation example from above, this is already something you are familiar with: some conversations are friendly, some are professional, some are adversarial, etc, and each carries with it a unique set of rules for what’s appropriate and what is not.

This is what different monads do: each comes with its own set of operations that are legal within the context that the monad is modeling. In some cases — as with some conversations — it becomes necessary to nest contexts. This is what monad transformers do: they give us a way to define some nested contexts and to move information back and forth through the nesting. Again, an example conversation to illustrate the point:

You: Say, have you heard any news about Bob’s mother?
Sue: Yeah, I heard she’s doing alright.
You: Has Bob gotten over the fact that she’s dating so soon?
Sue: Last time I brought it up he changed the subject.
You: Did she mention anything about whether or not she can make it to Bob’s surprise party?
Sue: She said she could if we schedule it for Saturday, but not if we are shooting for Sunday.

Here the conversation enters a sub-context dealing with Bob’s mother. Information about Bob’s relationship with his mother is moved around, based on the fact that there are two related contexts now in play in the conversation. Monad transformers provide a tool for moving information between nested contexts as well.

In Java, the analogous situation is where you have an object which references members that are also objects. This is obviously an extremely common situation. Later, when we add logging to our examples, we’ll see this explicitly.

Enough analogies, time to be concrete

We will add some context to our Haskell code via the StateT monad transformer. You can think of StateT as a design pattern for when you think your code would be simplified if you had some mutable state. (Unsurprisingly, this is a common thing.) I frequently see people claim that Haskell doesn’t support mutable state; we’re about to see that this is false.

The StateT monad gives us a single mutable variable. It gives us three functions for working with this: put, get, and modify (the latter is a convenience function built as a combination of put and get). Refactoring our Haskell code to use StateT will complicate it a bit, but ultimately boils down to two types of changes:

• Whereas our earlier Haskell code didn’t have any type annotations, once we start using StateT it will be better if we include some. The reason is that, while Haskell is clever enough to infer that we are using some monad, it isn’t always clever enough to infer which monad we’re using. (This is often related to the monomorphism restriction, but that’s a topic for another day.) The type annotations that we’ll be putting on will be pretty bland.
• We’ll use some do notation, which will make our code look more imperative. We’ll also use get and modify.

There’s one other change that we’ll make: we’re going to replace our intWrapper constructor with a different style of constructor that will wind up exposing a better interface to our client code.

When you start using StateT, you can think of the functions you write as being very similar to methods in Java: the function implicitly has access to some data (in Java it was m_i, here it is the data accessed by get and put). The type signatures on these functions documents this fact. It’s important to know, however, that StateT isn’t the same thing as a Java class, due to some differences I’ll mention at the end.

Here’s what the conversion looks like:

module IntWrapper2
        (runIntWrapper, printiw, inc, isPrime, nextPrime) where

import Control.Monad.State

data IntWrapper = IntWrapper Integer

runIntWrapper i f =
     do (a, i) <- runStateT f (IntWrapper i)
        return a

printiw :: MonadIO m => StateT IntWrapper m ()
printiw =
     do (IntWrapper i) <- get
        liftIO $ putStrLn (show i)

inc :: Monad m => StateT IntWrapper m ()
inc = modify (\(IntWrapper i) -> IntWrapper (i+1))

isPrime :: Monad m => StateT IntWrapper m Bool
isPrime =
     do (IntWrapper i) <- get
        return (isPrime' 2 i)
  where isPrime' t i | t >= i         = True
                     | i `mod` t == 0 = False
                     | otherwise      = isPrime' (t+1) i

nextPrime :: Monad m => StateT IntWrapper m ()
nextPrime =
     do isItPrime <- isPrime
        if isItPrime then return ()
                     else do inc
                             nextPrime

This code has definitely gotten more complex in appearance, but if you pick any given function, the differences between this version and the original are small.

Before we analyze this code, let’s take a look at what effect these changes have had on our client code:

import IntWrapper2

main :: IO ()
main = runIntWrapper 20 $
             do printiw
                sequence (replicate 4 inc)
                printiw
                nextPrime
                printiw

This is substantially better than our “day one” implementation, and (in my opinion) is even better than our original Java implementation as well. The runIntWrapper function takes two arguments: the initial integer (20) and a body of code to execute. The body of code (the stuff following the do keyword) is just a sequence of instructions that we want to perform on our IntWrapper. Behind the scenes, runIntWrapper is implicitly threading our IntWrapper to each of these instructions. This really obviates the whole “think of this as declaring a context” idea.

In our conversation analogy, this might be something like sending Bob an email:

To: Bob
From: You
Subject: Directions to my house

From I-90, take exit 18 up to the Highlands. From there, take the first right turn, passing by Cafe Ladro. Drive until the road ends. My house is at the very end; park anywhere on the street. The doorbell is broken, so you’ll need to call Sue when you arrive.

The directions don’t say Bob’s name anywhere except for the To field of the email, which establishes that the instructions are implicitly for him.

Some analysis of the transformation

I already mentioned that StateT is a monad transformer. Let’s dig into what this means.

All along I’ve been trying to push the idea that monads provide us with a way of describing “contexts” for our code. When you sit down and try to get some work done with this idea, you’ll quickly run into the following question: how can I nest contexts?

I’ve already mentioned that this is what monad transformers do. Let’s dig into that a bit.

Most monads come in two flavors: a standard flavor and a transformer flavor. The former provides a notion of context; the latter provides a notion of context and is compatible with nesting.

Our printiw function actually demonstrates this idea rather nicely. As a refresher, here’s the code for printiw, exactly as it appears above in our StateT version:

printiw :: MonadIO m => StateT IntWrapper m ()
printiw =
     do (IntWrapper i) <- get
        liftIO $ putStrLn (show i)

Notice that this function does exactly two things:

1. It uses the get accessor to grab a copy of our integer.
2. It uses putStrLn to print this integer.

What’s so significant about this? In Haskell, all things related to the environment (writing to files, opening sockets, printing to the console, etc) are modeled in the IO monad (read: context). So printiw must be able to work with two contexts: our StateT and IO. The type signature

printiw :: MonadIO m => StateT IntWrapper m ()

documents this: it says that printiw foremost resides in a StateT context, but it must also have access to an IO-capable context m (that’s what the MonadIO m assumptions means). The line liftIO $ putStrLn (show i) does two things:

1. It indicates that we want to print to the console. Trouble is, printing to the console is an action in the IO context, and our code resides within the StateT context. So to fix this…
2. it then takes this action and “lifts” it to the broader (StateT) context (via liftIO).

Note to the reader. If this seems like boilerplate, it’s because it is. Lifting is a facet of the monad design pattern that sometimes introduces boilerplate, but is sometimes done automatically (the distinction depends on which monad transformers you’re using, and sadly, it seems to take some experience and trial-and-error to get a sense of when you need to lift explicitly and when it’ll be done for you). In a moment, when we add logging to our example, lifting will be done automatically for us, and our code won’t have any additional lifts in it.

You’ll notice that most of our functions have a signature like Monad m => StateT IntWrapper m () instead of using MonadIO. That’s because printiw is the only function that needs to do an IO action; the others have no such need, so we are able to give them a more general-purpose signature.

Isn’t this a giant hassle?

Experienced programmers know that there are many philosophies when it comes to writing code. Usually these philosophies come in opposing pairs: static versus dynamic typing, functional versus imperative, compiled versus interpreted, etc.

What we’ve just seen is an example of a currently not-so-mainstream philosophy: that types should be used to annotate the ways in which code interacts with contexts. For instance, the type signature for printiw explicitly documents that this function requires access to the IO monad, while the type signature for nextPrime is sufficiently general that we can conclude that this function does not use the IO monad.

There is no related idea in Java or .Net (though the latter is introducing the pure keyword, at present time support is extremely limited). In Java, a function may or may not write to the console; the only way to know is to analyze the code, as the type system isn’t strong enough to make claims one way or the other.

Haskell has a popular reputation for making IO difficult. My experience — and I’d suspect that other experienced Haskell programmers would agree — is that working with IO in Haskell is no hassle at all; the difficulty is in shifting one’s thinking away from Java’s fast-and-loose approach to Haskell’s explicit-by-default approach.

Whether or not this is a good thing is decidedly a matter of one’s programming philosophy.

Lessons in this code that can be reused elsewhere

When I’m programming, I usually start without monads, and add them as it becomes clear that they will simplify my design. Over time, intuition develops that will allow you to decide from the beginning if you want to be using a monad (and which monad to use).

If you want to have some notion of mutated state, StateT is a good way to do it. To take code that is not written for StateT and refactor it so that it is, the basic steps are:

1. Write a “constructor” function. In our example, that’s runIntWrapper. The constructor function takes the initial value of the internal value (we supplied 20 in our example) and some code to execute within this new context. In terms of implementation, it’s just a wrapper for runStateT, which returns two things: the value of your internal state variable after the given code has executed (in our example we discarded this value) as well as any data returned by the given code.
2. Use get, put, and modify in your functions.
3. Provide some type signatures to help Haskell along.

Obviously this is just a quick and dirty to-do list, and while the steps sometimes vary based on the application, the complexity involved is usually at about this level.

Nesting contexts

Let’s beef up this example a bit, both in Java and in Haskell. We’ll add a logging feature to our wrapper: every time a function is called to act on our wrapper, it’ll make an entry in the log.

In Java the easiest way to introduce logging is to just have a list of strings. Here’s what the code looks like:

public class IntWrapper3 {
  private int m_i;
  private List<String> m_log;

  public IntWrapper3(int i) {
    m_i = i;
    m_log = new ArrayList<String>();
  }

  public void print() {
    m_log.add("print");
    System.out.println(m_i);
  }

  public void inc() {
    m_log.add("inc");
    m_i++;
  }

  public void nextPrime() {
    m_log.add("nextPrime start");
    while (!isPrime()) inc();
    m_log.add("nextPrime done");
  }

  public boolean isPrime() {
    m_log.add("isPrime");
    for (int t = 2; t < m_i; t++)
      if (m_i % t == 0)
        return false;
    return true;
  }

  public void printLog() {
    for (Iterator it = m_log.iterator(); it.hasNext(); )
      System.out.println( it.next() );
  }
}

Pretty simple. Here’s how I’d do the same job in Haskell. While I could use my current StateT to carry log data, it would be a bit of a mess; after all, “logging” and “working with IntWrapper” are orthogonal concerns. Let’s reflect that fact in our code by using another monad transformer in addition to StateT. Folks who have read other monad tutorials will not be surprised when I say that we’re going to use WriterT.

Here’s what it looks like:

module IntWrapper3
        (runIntWrapper, printiw, inc, isPrime, nextPrime) where

import Control.Monad.State
import Control.Monad.Writer

data IntWrapper = IntWrapper Integer

runIntWrapper i f =
     do ((a, i), l) <- runWriterT (runStateT f (IntWrapper i))
        return (a, l)

printiw :: MonadIO m => StateT IntWrapper (WriterT [String] m) ()
printiw =
     do tell ["printiw"]
        (IntWrapper i) <- get
        liftIO $ putStrLn (show i)

inc :: Monad m => StateT IntWrapper (WriterT [String] m) ()
inc =
     do tell ["inc"]
        modify (\(IntWrapper i) -> IntWrapper (i+1))

isPrime :: Monad m => StateT IntWrapper (WriterT [String] m) Bool
isPrime =
     do tell ["isPrime"]
        (IntWrapper i) <- get
        return (isPrime' 2 i)
  where isPrime' t i | t >= i         = True
                     | i `mod` t == 0 = False
                     | otherwise      = isPrime' (t+1) i

nextPrime :: Monad m => StateT IntWrapper (WriterT [String] m) ()
nextPrime =
     do tell ["nextPrime start"]
        nextPrime'
        tell ["nextPrime done"]
  where nextPrime' =
             do isItPrime <- isPrime
                if isItPrime then return ()
                             else do inc
                                     nextPrime'

There’s a few basic things to notice. First, signatures like Monad m => StateT IntWrapper m () have become Monad m => StateT IntWrapper (WriterT ["String"] m) (). This indicates that we have a nested context, or in the Haskell terminology, a monad stack. The inner monad (read: context) (WriterT) provides us with our logging interface (granted by the tell function). The ["String"] specifies the implementation of our log: we’re just using a list of strings.

Second, we’ve changed runIntWrapper so that it is able to accommodate the logging. In addition to using runStateT, it now also uses runWriterT to get the contents of the log. It is now returning these contents to its caller.

Let’s look at our client code:

main :: IO ()
main =
     do (_, l) <- runIntWrapper 20 $
                     do printiw
                        sequence (replicate 4 inc)
                        printiw
                        nextPrime
                        printiw
        mapM_ putStrLn l

Not surprisingly, only a little has changed. We are now capturing the data returned by runIntWrapper. The runIntWrapper function yields both the output of our given code (which we ignore with a little _, as the code we’re giving doesn’t have anything to return) as well as the contents of the log at the end of execution (which we capture as l). We then print the contents of the log using mapM_ putStrLn l.

A qualitative analysis

There are a lot of important differences between our Haskell and Java code, some we’ve already discussed, some we have not.

One difference is the way in which we separated concerns. In our Haskell code, we wound up using two separate monads (StateT and WriterT) to implement our wrapped integer with logging. In our Java code, we used a single class with a member variable (a List<String>).

In Haskell, using a stack of monad transformers is an idiomatic way to separate concerns. The WriterT transformer is an expert in providing a write-only record; the StateT transformer is an expect in providing a mutable variable. If we were being more serious with our coding, we would have defined a new type like

type IntWrapperT m a = StateT IntWrapper (WriterT [String] m) a

to ease up our type signatures. (This practice also makes it easier to later come along and factor in additional monad transformers. Indeed, defining this type synonym is definitely the preferred way to do things: by leaving it out of the samples I’ve unfortunately demonstrated bad behavior, but I didn’t want to have a sidebar discussion of type synonyms in the middle of the example.)

If we had been more serious on the Java side of things, we could have created a class that was purpose-built for logging. We may have then simply had IntWrapper inherit from this class.

In my view, adding logging in Java and Haskell had about the same amount of overhead (in terms of the work it took as a programmer). Then again, I’ve been working in both languages for years, so adding a member variable or another monad transformer is a pretty routine thing for me. I’d presume that someone who is new to Haskell (or Java for that matter) would have a different view on the issue.

There’s an important topic I’ve thus far ignored: should all Java classes be translated into monads? My experience has been that this is usually a fine way to implement the class’ functionality in Haskell, in situations where the class is used to carry and mutate state.

It’s important to realize, however, that these two approaches to design are not interchangeable. In Java it makes sense to have several instances of a class interacting with one another; in Haskell one usually would not have several blocks of monadic code interacting, unless they were all running within the same monad (read: context).

So while we could implement a method in Java for adding two IntWrapper instances together, a similar function in Haskell would probably require us to refactor a bit (we’d need to look at the MonadPlus typeclass). This would be simple to do, but would have taken us down a different path. This difference does illustrate that Java and Haskell are sufficiently different that a direct translation is not always a natural thing to do.

Of course, there are examples that work the other way as well (that is, monad idioms that are commonplace in Haskell but hard to translate naturally into Java). I wouldn’t say that this is a good way to go about deciding which language is “better,” but it is definitely an example of why Haskell monads and Java classes aren’t really solving the same problems (though clearly their problem domains overlap).

Is the approach I demonstrated in Haskell easy or hard? I believe this to be a matter of one’s experience. I can attest that it took me some time to really grok this approach, but then again, I can also attest that it took me some time before I was able to write good object-oriented code as well.

I think that part of the problem is that, while it’s pretty simple to motivate how one should think about object oriented programming, similar models for how to think about monads have taken some time to develop. I suspect this is a pedagogical problem more than an intrinsic difficulty with the monad design pattern, but ultimately only time will tell.

Using monads in your own code

We’ve just looked at an example using the StateT and WriterT monad transformers. Moving forward, here some important questions:

1. Will I always use these two monad transformers? Yes and no. When you are writing your own applications and libraries, you’ll probably wind up using these transformers (perhaps as well as ReaderT, which provides a read-only variable) as the foundation for your work. As suggested above, you’ll probably define a type synonym to hide this fact. You’ll frequently have the IO monad at the base (so that you can have IO), but will sometimes use the Identity moand at the base (when you are writing functions that don’t need IO). (The Identity monad literally does nothing. If you have a stack of monad transformers, and need a monad at the base that won’t change the semantics of your program, you can use the Identity monad.)

   When you use libraries written by others (such as parsec or Template Haskell) you will find that these libraries provide you with their own monads that they want you to use. It will be convenient and natural. You’ll use these monads in a manner similar the client code we were writing earlier.

   (Parsec, a library for writing parsers, is so natural and intuitive that it’s completely changed my outlook on writing parsers.)

2. Should I always use the transformer versions of the monads, or should I use the “regular” versions? Why use StateT instead of State? I tend to always use the transformer versions, simply because I find it makes refactoring easier, and it makes the code slightly more general purpose in some situations. If I think I want to use the regular State monad, I often use StateT transformer on top of the Identity monad, yielding the same behavior.

   Of course, that’s just how I do things. It’s certainly not the law. I’d presume that someone out there will have a very good list of reasons why I’m wrong to do what I just described, and in the interest of being a better programmer, I’d appreciate hearing from them.

3. When should I write my own monad? Aside from creating a type synonym, I rarely find myself writing my own monads (and with a type synonym, you don’t need to define a monad instance anyway). Monads are most frequently used to carry state around, and the StateT, WriterT, and ReaderT monad transformers do that handsomely.

   Sometimes a need for extra high performance can drive the need to write your own monad. Sometimes you need to write your own monad to interact with some weird Haskell extension (like the rank-2 types example I mention here).

   Usually, though, you can go pretty far just using the standard monads and transformers. I’ve talked with many multi-year Haskell programmers who have said that they’ve never written their own monad. Your mileage may vary.

4. What are the most important monads for a beginner to know about? This would be StateT, WriterT, ReaderT, and ErrorT. Here’s what they are used for:
   • StateT is used to pass around mutable state, like our IntWrapper example.
   • WriterT provides an interface for writing to a collection. It is very often used for introducing simple logging to an application.
   • ReaderT provides a read-only variable. It is most often used to pass around application configuration data in such a way that consumers of this data can’t modify it (which is a useful design invariant in many situations).
   • ErrorT provides a way to perform operations which might fail, and to manage failure gracefully. You can think of it as a way to perform computations which might throw exceptions.

From here, if you want to know more about the practice of using monads to get work done in Haskell, I’d suggest checking out Real World Haskell, which has some good discussion about using monads to engineer solutions to real problems, and the Typeclassopedia.

Haskell is my go-to language, both for scripting, and for getting work done. This is not because of any particular allegiance to the language. Haskell and I have an open relationship, and the moment I find a language that out-Haskells Haskell, you can be sure I’ll move on.

Here I want to describe my favorite things about Haskell. You’ll note that they are all about the type-system. I don’t feel too strongly one way or the other about laziness, or about monads (though I won’t give them up without first finding something to take their place). I don’t even particularly care that it’s a functional language, in as much as I can have these features in a non-functional environment.

Some of these features are already available elsewhere. This is wonderful! If you know of any examples of this, please tell me in the comments.

This is a list of my favorite things:

Separation of class and data definitions.

Haskell’s notion of classes is more like Java’s notion of interfaces. A class is a list of function prototypes, and any data type for which such functions can be defined is an instance of that class. One does not inheret a parent class, but rather, one implements a class. It’s a weird distinction if you haven’t seen it before, but after I learned how to use it, I must say I prefer it.

The first example most people see is the Show class. Here is how it’s defined (to get this listing, I just asked ghci — the interactive GHC prompt — to give me the definition):

Prelude> :info Show
class Show a where
  showsPrec :: Int -> a -> ShowS
  show :: a -> String
  showList :: [a] -> ShowS
  	-- Defined in GHC.Show

This says that any data type a which is an instance of Show provides functions with these signatures. (Edited: The first of these functions are used for implementing a Haskell idiom for fast string construction, while the last is related to a restriction in the unmodified Haskell 98 standard.)

When I define a new datatype, I can either ask Haskell to derive a Show instance for me automatically, or I can specify one myself:

data Car = Person { make :: String, year :: Int } deriving Show
data Pet = Pet { name :: String, animal :: String, age :: Int }
instance Show Pet where
  show p = "My pet is named " ++ name p ++
           " and he is a " ++ animal p ++
           " and he is " ++ show (age p) ++ " years old."

I understand that Google’s Go language has this notion of class (they call it interface), and that Scala provides this as well (they call it trait).

Typed side-effects.

In Haskell, if a function wants to communicate with the environment, then the function’s type signature will document this fact. Want to print to the console? Open a socket? Read a file? Any of these actions will put your function into the IO monad, which is a red-flag to other programmers that the function communicates with the environment. When your application works with library code (and whose doesn’t?) this is a handy feature.

Haskell uses the monad design pattern as the underpinning of how it types side-effects. I don’t particularly care that it’s monads per se, I just like that there is something which statically documents which functions communicate with the environment.

Why is this useful? One huge answer is concurrency. If your function has side-effects, it is not obviously thread-safe. If it has no side-effects, it is thread-safe. The monad design pattern provides a way to define application-specific notions of side-effects, which allows you to dial in the granularity on this as much as is appropriate for your application.

With respect to typed side-effects, a common Haskell idiom is to break up your program into different layers of state. For instance, in a web framework, you might have a “user-input” layer which is read-only, and on top of that a “logging” layer, and on top of that your application-specific stuff. (Each of these layers is a monad, or more precisely, a monad transformer.) Haskell allows you to statically track which functions rely on which layers, which is a useful thing if you want to call a function and be certain that it won’t modify some data out from under you.

If you’re new to Haskell and monads, in my humble opinion this idiom is the real reason to give a damn about monads. But that’s just my perspective.

(And it’s certainly not obvious from the beginning, but a lot of bugs can be eliminated this way.)

Type safe macros.

No language is completely free from the occasional boilerplate. One way around this is to use macros.

In C, macros can be very tricky. The preprocessor takes all instances of a macro, replaces it with the corresponding text, then passes off to the compiler. If it turns out that you used the macro incorrectly, the compiler isn’t really there to help you out: after all C macros are all about find-and-replace.

Haskell’s macro system is called Template Haskell. Macros written in Template Haskell are actually written in Haskell syntax. The compiler then takes this code, compiles it like it would any other Haskell, and then uses it to expand your usage of the macros. Everything is typed the whole way through, and if there are errors, the compiler can tell you where they are and why (with its usual level of precision, for better or worse).

When I recently ran into a scenario where (for some very long-winded reason) I had to define 20 essentially-identical datatypes, then give them all essentially the same class instances, I was able to quickly whip up some Template Haskell to do all the lifting for me. When I realized I needed to modify those class instances, it was as simple as modifying the Template Haskell that was generating them.

This is how macro-ing should be. Instead of a deal with the devil, it should be safe enough to be accepted practice.

Quasi-quoting.

This is one of the many fine ways to embed a language in Haskell. Here’s a typical use case: you’re writing a library and the most natural way for a developer to specify some options is in a simple configuration language. You could implement a function String -> MyLibOptions, but if they have any typos in their configuration string, you won’t be able to catch them until run-time. If the configuration isn’t known until run-time that’s fine, but if the configuration is known at compile-time, you’d like the error to be caught at compile-time. (I need to mention that quasi-quoting is able to mix run-time and compile-time data — I’m just simplifying things to describe this use case.)

Quasi-quoting to the rescue. I recently gave an example of Haskell’s quasi-quoting abilities in a post about how it can be used to provide an injection-proof form of string interpolation (via Interpolique). One of my favorite applications is Michael Snoyman‘s Hamlet, a type-safe HTML generation library.

(If you’d like to see what it looks like to implement a quasi-quoter in Haskell, I’ve got some code up on github that demonstrates this in the case of string interpolation, as mentioned above.)

Quasi-quoting is basically syntactic sugar for Template Haskell. Consequently your quasi-quoters are able to reach into the environment and interact with the rest of the code (all in a type-safe, purely functional way, of course). In the string interpolation example above, for instance, the code

author = "broker"
content = "' or 1=1;"

query = [$interpolique| insert into posts values(^^author , ^^content ); |]

set query equal to the following

*Test> query
InterpoliquedString
    " insert into posts values(b64d(\"YnJva2Vy\"), b64d(\"JyBvciAxPTE7\")); "

which was generated by inspecting the values of author and content at run-time, encoding them in base64, and then interpolating them into the result you see here. The fact that author and content were strings was determined at compile-time, so there wasn’t any chance of any shenanigans when the code actually executed.

For instance, if I instead had the code

author = 2 :: Int
content = "' or 1=1;"

query = [$interpolique| insert into posts values(^^author , ^^content ); |]

I’d get a compile-time error:

Test.hs:9:23:
    Couldn't match expected type `String' against inferred type `Int'
    In the first argument of `InterpoliqueQQ.b64enc', namely `author'

which I think is pretty cool.

Type families and associated types.

I must admit that I only use the “associated types” half of this, although the feature is slightly more general. Anyway, I’ll describe the part that I use.

Type families give you a way to compute which type you want to use. Yes, sounds weird, but it’s amazing.

A typical first example of this is the associated list. Every modern language has these: it is just an array where the lookup doesn’t need to be an Int (think HashMap and the like).

In Haskell this can be described like so:

class GenericMap a where
  type Key a
  type Value a
  get :: a -> Key a -> Value a
  set :: a -> Key a -> Value a -> a

The first two parts of this class definition are the so-called “associated types.” The easiest way to see this in use is with an example of what an instance might look like. Here I’ll do something crazy and define the function type String -> Int as an instance of this class (the Haskell Wiki article on type families has other examples, some of which you might find more conventional):

instance GenericMap (String -> Int) where
  type Key (String -> Int) = String
  type Value (String -> Int) = Int
  get f k = f k
  set f k v' = \k' -> if k == k' then v' else f k'

This instance works:

sampleMap :: String -> Int
sampleMap s = length s

sampleMap' = set sampleMap "foo" 4

...

*Main> get sampleMap "monkey"
6
*Main> get sampleMap "foo"
3
*Main> get sampleMap' "foo"
4
*Main> get sampleMap' "bar"
3
*Main> get sampleMap' "monkey"
6

which is all well and good.

Now, I haven’t yet given any reasons why this type families business is any good. The answer has to do with polymorphism: sometimes you want to write a function whose type signature is so damned flexible you just can’t figure out how to write it. You try a few examples, but each is too restrictive. But there’s a pattern to it. If you’re in this boat, type families can help.

I’d give an example of this, except I already did in polymorphic first class labels. (Which, by the way, is another feature I’d like to see in other languages.)

Another application of type families is type-level programming (functional dependencies can also be used for this, but as type families get better, my interest in seeing functional dependencies in other languages will dwindle). Type-level programming is an insane idea where you do computation in the type system at compile-time.

This actually can be helpful in situations where you have really complicated properties you want to express about your program statically. For instance, I had a situation where certain types had a “size” associated to them. I had functions that were polymorphic over arguments of a given size. Some of these functions would yield a new type that was twice the size as the input.

How do you express that statically, if the goal is to still be polymorphic? Type families can do it. I basically wrote a class whose sole job was to use type families to do arithmetic in the freaking type system. This basically looked like

class HasSize a where
  type Size a

class Doubler a where
  type Double a

...

-- The ~ operator asserts type equality, so this next
-- line basically reads "the size of b is `Double' the
-- size of a."
someFunction :: ( Size b ~ Double (Size a) ) => a -> b
someFunction = ...

I would not describe it as pretty, but it solved my problem, and it gave me a compile-time guarantee that an important design invariant was being met. The syntax is easy to read as well. And if it looks like I’m applying functions to types, it’s because I am.

Rank-2 types.

You don’t often see this on the list of great things about Haskell, but I love them. To say that a type is “rank-2″ is basically a statement about just how polymorphic it is. I use this feature in two different ways: the first is to solve a polymorphism problem, the second is to prevent tainted data from leaking into places it doesn’t belong (I’m in love with this second application and I have no clue how to statically do it in any other language — tell me in the comments if you do!).

Here is an example of how I use it to get some extra polymorphism:

useFoo :: (forall a . a -> [a]) -> b -> ([b], [String])
useFoo f b = ( f b, f "bar" )

This function takes two arguments (another function and some other type) and uses them to build a tuple (by applying that function twice). The forall asserts that the function we give must work for any type a, hence why we can apply it to the mystical input of type b or to an ordinary String.

If I were to rewrite this function without the forall I'd get a type error (two type errors, actually):

useFoo1 :: (a -> [a]) -> b -> ([b], [String])
useFoo1 f a = ( f a, f "bar" )

gives me

temp.hs:14:18:
    Couldn't match expected type `[Char]' against inferred type `b'
      `b' is a rigid type variable bound by
          the type signature for `useFoo1' at temp.hs:13:25
    In the first argument of `f', namely `a'
    In the expression: f a
    In the expression: (f a, f "bar")

temp.hs:14:23:
    Couldn't match expected type `a' against inferred type `[Char]'
      `a' is a rigid type variable bound by
          the type signature for `useFoo1' at temp.hs:13:12
    In the first argument of `f', namely `"bar"'
    In the expression: f "bar"
    In the expression: (f a, f "bar")

Absent the forall, the type checker assumes that the function I'm providing works for some type a, and attempts to determine just which type that happens to be. That is, the compiler is allowing me to be a little ambiguous with my type signature, figuring that there is a particular type I have in mind and that it will use type inference to determine what that would be. But then I try to use the function on two different types -- b and String -- and therefore is quite upset. (In fact, it is already upset because, the way I've written the signature for useFoo1, Haskell assumes that a and b must be distinct, and in fact this is what those errors above are telling me: a is not the same as b, nor is it the same as String.)

While this application is nice, as I alluded above, in my mind the killer application is tracking tainted data. Here are two common scenarios where this is something you want to do:

• You have some function which accepts untrusted user input, and you want to be certain that whatever value it returns has been scrubbed clean. This is handy for a function like, say, useUserInputToBuildSQLQuery. (There are many other ways to solve this problem, of course.)
• You have a function which allocates some resources, uses them, then frees them, and you want to make sure it doesn't return a dangling handle. (I'm not aware of another way of solving this problem, and again would appreciate any comments with other ideas.)

The best example of that second scenario is Haskell's ST monad. Code that executes with the ST monad is able to create mutable variables. If you have a function that is written in the ST monad, you can execute it using the runST function, whose signature is

Prelude> :m +Control.Monad.ST
Prelude Control.Monad.ST> :t runST
runST :: (forall s. ST s a) -> a

The key to how this works is the forall in the signature of runST. In essence, it is preventing code in the ST monad from returning one of these mutable variables. So the following code works:

{-# LANGUAGE Rank2Types #-}

import Control.Monad.ST
import Data.STRef

exampleST :: ST s Int
exampleST =
     do myMutableVar <- newSTRef 0
        modifySTRef myMutableVar (\n -> n+1)
        n <- readSTRef myMutableVar
        return n

...

*Main> runST exampleST
1

but the following code does not:

{-# LANGUAGE Rank2Types #-}

import Control.Monad.ST
import Data.STRef

exampleST1 :: ST s (STRef s Int)
exampleST1 =
     do myMutableVar <- newSTRef 0
        modifySTRef myMutableVar (\n -> n+1)
        return myMutableVar

...

*Main> runST exampleST1

:1:0:
    Inferred type is less polymorphic than expected
      Quantified type variable `s' escapes
    In the first argument of `runST', namely `exampleST1'
    In the expression: runST exampleST1
    In the definition of `it': it = runST exampleST1

People who read my blog will not be surprised when I mention that Oleg Kiselyov and Chung-chieh Shan have shown that this approach can be used to implement region based resource management with good granularity. (This is a paper I've been bringing up a lot recently, as it is the underpinning of memory management in Potential.)

Conclusion.

Haskell has a reputation for being hard to learn, though I feel this reputation is a bit dated now that we have good resources like Learn You a Haskell for Great Good and Real World Haskell. Certainly one of the hardest parts about learning Haskell is that so many of the examples of "good Haskell" that we hold up rely on many of the features I mentioned above, and most of them seem foreign to new Haskellers. That's hard to avoid: Haskell is, after all, a research testbed.

I don't know if you'll feel the same way as I do, but after gaining some experience with using these tools in my own code, it is frustrating to leave them behind when working in other languages. Every language has its seed of grace, elegance, and brilliance that, if it gets into you and grows, will make you into a zealot. I feel that these are Haskell's seeds.

Experienced teachers know that claims of “lost” work are frequent. If we want to be objective about this (and we do), the claims need to be taken seriously, since lost things rarely leave a trail. All we have when analyzing such claims is the following:

• The missing work never seems to turn up. Not after a week, a month, a semester, a year, or ever.
• If a person rarely finds that they misplace his own belongings, it’s hard to accept that he is misplacing student work (assuming they treat student work with a reasonable amount of care, as we typically do, given how terrible it would be to lose it!)
• These claims never seem to come from students who are doing well on exams; they tend to come from students who are backed into a corner, grade-wise.

Of course, it is entirely conceivable that these claims are occasionally correct, and it would be terrible to allow such mistakes — our mistakes — to adversely effect our students.

Last Spring was the only time a student has accused me of losing their work. It was a lousy situation that I have no intention of ever repeating. So when I was assigned to teach a half-term class this summer, I decided it was time to try something new. I’ve recently finished teaching that class; here’s what I did.

The idea

A digital signature is a cryptographic technique to verify that a document was authored by a particular person. They are frequently used in situations where someone might later claim to have not authored a document in order to weasel out of a precarious situation. It is usually the recipient of the document who insists that a digital signature be used.

Digital signatures can be used in other situations where authenticity is important. For instance, concert tickets can be printed with a digital signature that validates they were printed by the ticketing agent. The signature on each ticket will need to be different, but this isn’t hard to arrange. If the cryptography is sound, no one will be able to forge the signature, hence no one should be able to print a phony ticket.

This summer I experimented with using digital signatures to provide students with a way to prove that they’ve turned an assignment in. Every time I made an assignment I produced a corresponding batch of digitally-signed receipts. One receipt per student per assignment. I wrote a program to automate this: it takes a list of students and the name of the assignment and produces a PDF file of receipts. I’d then print the PDF, take it to the cutting board, and produce an alphabetized stack of receipts for each assignment (the program was clever enough to sort the receipts on the printed page in such a way that the alphabetizing didn’t require me to sort through the cut up pages).

I then instituted a policy: I won’t take your homework unless you take your receipt. Because the receipts were sorted this didn’t take long to do (though to save class time, I only accept homework before and after class, or during problem sessions).

I’d then periodically create printed grade reports for each individual (again I had a program that automated this, taking data from my spreadsheet and turning it into a PDF with one page per student). This allowed students to check whether or not I was giving them credit for their work. At the end of the term, I gave them one last report that showed the grade I was going to submit for them. No one can claim that I was withholding information.

The wonderful thing about this system is that it gives the students proper recourse: if it looks like I lost an assignment, the receipt proves they are right, and I’ll give them full credit. Because the receipts are digitally-signed, they cannot be forged, so there’s no funny business to be had by any bad apples.

Reception

Naturally I described my plan to the bosses in the math department. My department chair conjectured that no one on earth had tried this before. The associate chair laughed at the lengths I was going through. My peers suspected that the students would find this system confusing, annoying, and unnecessary.

When I implemented this system the results were fantastic. I don’t believe anyone in class understood how the digital signatures worked, but they didn’t need to: all they had to know is that the receipts give them recourse if I lose their work. And they loved it. Why wouldn’t they? It was apparent that I was doing extra work to expose myself for their sake (nevermind that I was originally motivated to try this to avoid getting in hot water myself).

Contrary to initial predictions that I would be mired down in extra paperwork, this process did not take much time on my end. Once I had the programs written to automate the work (which took an afternoon) the time investment was nearly zero. The most time consuming part was passing out the receipts as students turned in their work, but since this was taking place during problem sessions, it didn’t actually increase my working time.

For me the most interesting thing is that I did apparently lose someone’s assignment! I have no clue how, and part of me still hasn’t accepted this, but the evidence suggests that it happened. I gave someone a grade report, it said they were missing an assignment, but sure enough they had a receipt. They called me on it in front of the class, and when I reaffirmed my promise to give them credit, a couple people in the class praised me. The receipt system actually took my mistake and turned it into an asset. Incredible.

Edit: this is what a sample of my receipt-PDF looks like (here shown with just a small number of hypothetical students). Notice how it’s sorted — makes it very easy to just go to the chopping board, stack the pages, make 4 cuts, and concatenate the little stacks.

How I did it

There’s a considerable about of machinery that goes into such a scheme. If you’re interested in implementing something like this, and have a good familiarity with computing, you can follow these steps to get rolling. For all of this I’m using openssl, which is installed by default on my Mac, and also on every Linux distribution I’ve ever used. I’ll show how this is done using a test receipt.

Step 1: Create the key files. Create a directory to do your work in. Within this directory, issue the following two commands:


$ openssl genrsa -out private.pem 1024
$ openssl rsa -in private.pem -out public.pem -outform PEM -pubout


This will create a public and a private file, so-named by openssl. The public file can be put on your website, or included in your syllabus, allowing a 3rd party to arbitrate any disputes that may arise. The private file must be kept private, as it is the key to signing the receipts.

Step 2: Create a test receipt to work with. Later you’ll do this step for each student, for each assignment (consider using a script to do this for you!). Here’s my test receipt:


$ cat Test-receipt.txt
FAKE, STUDENT ASSIGNMENT 0001


Step 3: Sign the receipt, putting the output into base64 (this will allow you to print the signed receipt so you can give it to your student in writing)


$ openssl rsautl -sign -inkey private.pem -in Test-receipt.txt |
openssl enc -base64 -out Test-receipt.sig

You can examine the output:


$ cat Test-receipt.sig
hRqaY5LAns3CrzueaMXirehihYCn6TI6K4Luwo9T6F4JVMXiBb10wSN4fDLnM12m
NICQihiAt5prlqDxjwqpr2J4tPMmQZpXr8dpFKdyQgxn6IesLiEm9HIVjYUELRMW
kzxv86+8oVl6qQny+kMVWo3w7pI/JTTnHP3yLl1NJJw=

When you go to print your student’s receipt, include both the contents of the receipt (in my case, Test-receipt.txt) as well as this gibberish.

Step 4: Make sure you know how to verify a receipt! This isn’t so bad. If someone gives you their receipt, you enter the gibberish into a file (in this case, Test-receipt.sig) and execute the following:


$ cat Test-receipt.sig | openssl enc -base64 -d | openssl rsautl -verify
-pubin -inkey public.pem
FAKE, STUDENT ASSIGNMENT 0001


If the signature is correct (that is, not entered wrongly, nor an invalid forgery) you should see the receipt in plain text, as demonstrated here.

As an example, if I modify even a single letter in Test-receipt.sig, I’ll wind up with something that makes no sense, or more likely, I’ll get an error. For instance, if I replace the first letter (a lower-case h) with an upper case H, I get the following:


$ cat Test-receipt.sig-error | openssl enc -base64 -d | openssl rsautl -verify -pubin -inkey public.pem
RSA operation error
286:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:/SourceCache/OpenSSL098/OpenSSL098-27/src/crypto/rsa/rsa_pk1.c:100:
286:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:/SourceCache/OpenSSL098/OpenSSL098-27/src/crypto/rsa/rsa_eay.c:697:


This will allow you to detect cheap forgeries.

Conclusions

This summer I ran this experiment with a class of 55 students, and in this setting, homework collection took about 15 minutes. It worked fantastically. In the Fall I’m teaching a group of 180. The turn-in time might make this plan infeasible for such a large group unless I can have my TA’s come to class to help collect work. I’ll definitely post my experience with this group in the comments to this post.

Today we’ll look at how Interpolique can be implemented in Haskell in such a way that we force the developer to use Interpolique when creating a SQL query, precluding the possibility of the $/^^ mixup bug. In doing so we’ll see that we don’t need anything like PHP’s eval to get the job done.

All of the code for this post is on github: InterpoliqueQQ.

Since version 6.10 of the Glorious Haskell Compiler, we have had the ability to essentially define new language syntax. This functionality — called quasi-quotation — is useful for embedding mini-languages into Haskell in a type-safe way. Since Interpolique is basically a mini-language (it’s only operator is the ^^ interpolator), it is natural to use quasi-quotation when implementing Interpolique in Haskell.

Let’s look at an example of what this looks like when it’s being used. On the Interpolique site, an example of an attempted SQL injection is given. The crux of the example is the following code:

$conn->query(eval(b('insert into posts values(^^_POST[author] , ^^_POST[content] );')));

In InterpoliqueQQ (see Test.hs in the InterpoliqueQQ code), the same code can be written as

query = [$interpolique| insert into posts values(^^author , ^^content ); |]

If we hop into interactive-mode with GHC, we can see the value of query:

*Test> query
InterpoliquedString " insert into posts values(b64d(\"Zm9v\"), b64d(\"JyBvciAxPTE7\")); "

Thus the run-time value of query is, in fact, an Interpolique’d SQL query.

Why this is interesting

The important feature of InterpoliqueQQ is that this syntax offers protection in the form of static typing. If we inspect the type of query we get

*Test> :t query
query :: InterpoliqueQQ.InterpoliquedString

That is, this syntax creates a query whose type is InterpoliquedString. In this implementation of Interpolique, the only way to obtain an instance of InterpoliquedString is via this syntax. In other words, if a function is given a query of type InterpoliquedString, it can be completely cetain that the query was generated using Interpolique. Since this syntax does not allow PHP-style string interpolation (that is, there is no analogue of 'insert into posts values($author, $content)), there is no way for a developer to introduce a SQL injection bug due to a misused interpolation operator.

(We can also note that InterpoliqueQQ does not use anything similar to PHP’s eval, thereby rendering any objection to the presence of eval moot.)

The implementation

InterpoliqueQQ is implemented in Haskell in less than 75 lines of code, relying on the powerful parsec parser combinator library, GHC’s quasi-quotation support, and the base64 encoder of the dataenc library.

This implementation is entirely proof-of-concept. In particular, it’s missing two things:

1. Field-testing. Interpolique hasn’t (yet) been out long enough for peer review to have run its course, so certainly nothing can be said about whether or not this particular implementation is secure.
2. Library support. This implementation is built on the InterpoliquedString type. In order for this to be useful, there needs to be a SQL library which is ready to act on this type. For the time being, I’ve included the runQuery function in InterpoliqueQQ which just takes an InterpoliquedString and prints (to stdout) the corresponding SQL code, as in
   

   *Test> runQuery query
    insert into posts values(b64d("Zm9v"), b64d("JyBvciAxPTE7"));
   

At present time Haskell is a decidedly non-standard language to use for web development. Examples like this, however, suggest that it could be a powerful tool in this domain in the future.

There’s been a lot of buzz about the Snap framework, so I thought I’d give it a look. My personal website doesn’t have anything dynamic going on, so arguably Snap is “overkill,” but then again so is Apache, so what the heck. Snap is entirely experimental at this time: in their own words, “it is early-stage software,” so every single critique given here should be read with an implied expiration date.

So the question is: how does one host a static site on Snap? At present time there’s no tutorial for this, so I fumbled around until I got something working. Here’s my code:

main = do
    putStrLn "ninj4net online"
    quickServer config site

site :: Snap ()
site =
    route [ ("kinetic", static "kinetic")
          , ("math1010", static "math1010")
          , ("math1030", static "math1030")
          , ("math1100", static "math1100")
          , ("math1210", static "math1210")
          , ("", static "")
          ]
    (writeBS "general error")

static d = do
    let html_file = "static/" ++ d ++ "/index.html"
        xml_file  = "static/" ++ d ++ "/index.xml"
    html <- liftIO $ doesFileExist html_file
    xml  <- liftIO $ doesFileExist xml_file
    ( (ifTop (fileServeSingle $ if html then html_file else xml_file))
      (fileServe $ "static/" ++ d) )

Discussion

Some of my directories use an index.html file, while others use an index.xml file. I need to use System.Directory.doesFileExist function to determine whether or not these files exist — trying ifTop (fileServeSingle "something that doesn't exist") will not switch to the alternative using <|>, so an explicit check is needed (it will whine about an exception being thrown, completely undermining the choice operator!). This is presumably a bug (I submitted it to their github).

I’m sure there is a much more elegant approach, but this was the best I could muster during lunch.

A thing Snap is missing

I came across a design decision that makes me nervous. As with any web framework, there are facilities for getting strings from the user. Unfortunately, Snap does not use types to distinguish between user-provided strings (dirty) and programmer-provided strings (clean).

Why does this matter? Segregating user input into its own type is a formidable defense against (say) SQL injection, since it obviates that "select * from myData where foo='" ++ userInput ++ "'" isn’t well-typed (presumably SQL code should be its own type, say, SQLString, and the function UserString -> SQLString should be some kind of escape routine). It would be nice to see framework support for this types-based defense.

The most obvious example of this is in getParam, which simply returns a Maybe ByteString.

Another example is provided by getSafePath and fileServeSingle. The former returns a FilePath provided by the user (a “safe” path, which — looking at the source — means that the “/../”‘s get removed), and the latter takes a FilePath and opens the corresponding local file. I suppose the idea is that the code

do p <- getSafePath
   fileServeSingle p

shouldn’t escape the implied sandbox of the file system. Of course, if the application has tighter requirements than this, the type system isn’t there to help out. (For instance, perhaps a path is considered “safe” if it excludes certain keywords in addition to the constraints imposed by getSafePath).

A natural work-around is to build an application-specific wrapper around Snap, and perhaps this is the better approach; I’m not yet sure.

Conclusions

I’m glad that Snap has been announced, as it has proven interesting to look at. Of course, Haskell already has (at least) two other web frameworks (yesod and happstack) and it’s not clear which will win-out in mindshare, nor is it obvious (to me, anyway) which one would be the best choice for someone wanting to sit down and make a site. Of course, it’s possible that some mix-and-match might be the best approach: the web server of one project, the HTML generation of another, and the persistent storage of the third. (This possibility deserves some consideration, especially as projects like BlazeHTML really take off.) Hopefully in the coming months we’ll see more high-powered applications of these frameworks, giving us a fountain of lessons we can capitalize on, and providing some compelling show cases of Haskell’s power as a web development language.

$conn->query('insert into posts values($_POST[author] , $_POST[content] );');

we write

$conn->query(eval(b('insert into posts values(^^_POST[author] , ^^_POST[content] );')));

The b function is provided by interpolique. It essentially translates the input string into some PHP code (which is then reified using eval) that base64 encodes the user-input and wraps that encoding up in a call to the MySQL function for base64 decoding.

The idea is that the resulting query is given to MySQL in a format where the user input is base64 encoded. As Dan points out, there aren’t any known injection techniques that can escape the MySQL base64 decoder, and the decoder won’t try to evaluate the resulting string as a SQL expression, so no injection is possible.

I have mixed feelings about this approach. On the one hand, it’s really just another form of escaping (instead of inserting a bunch of \‘s into the string, we’re base64 encoding it), and escaping is an error-prone thing. After all, there’s nothing preventing a tired developer from accidentally mixing some $ in with their ^^, nor could there be — if the developer writes $ instead of ^^, PHP will interpolate the string before passing it off to the b function, so no run-time check will be able to save the day.

(I’m not elated about the use of eval, but (a) I see no way around it if the plan is to use a syntactic approach, as is currently the case, and (b) the only vector I can see for attacking it requires a programmer to leave out the call to b, which is something they’d likely catch during development unless they also used $ instead of ^^, and that double-accident seems unlikely, barring a stupid refactoring snafu.)

On the other hand, if this technique is applied correctly, it seems likely to be robust (peer review should weigh in on this pretty quickly).

When I look at interpolique, I see the next generation of escaping: if you forget to do it you’re screwed, but if you do it correctly you’re safe. interpolique’s contribution is that its style of escaping is much simpler than trying to scan strings for dangerous characters, hence less likely to contain silly errors and edge cases, and that it is cross-language ready, in that base64 encoding isn’t target-language-specific (unlike escaping, which certainly is).

interpolique does not improve upon escaping biggest failure, though: if you’ve got 50,000 lines of PHP, the only way to know that interpolique (or escaping) is being used throughout is to look through the code. This is a PHP shortcoming, of course. We could certainly produce some static analysis tool to check for this design pattern, but then again, if writing tools that understand strings in PHP were easy we wouldn’t have the code injection mess in the first place.

Future direction

interpolique does provide a novel improvement for how to move data across the language barrier. This makes the core idea useful even in situations where programmers aren’t using PHP (or other half-brained-but-common-anyway languages).

In the long term, however, we still need to address the fact that we’re abusing the String type. User-input should be its own distinct type. The fact that this isn’t the case in .Net and Java completely explains why those type-safe languages don’t fare any better than PHP in terms of code injection.

Following the interpolique idea, the only function from UserString to String could be a base64 encoder. Languages could provide syntactic sugar to allow things like

$conn->query('insert into posts values($_POST[author] , $_POST[content] );');

to implicitly denote the interpolique style, thereby preserving type-safety (in this case, separation of user-input from SQL code) without compromising string interpolation style.

(Of course, both of these ideas are already possible in Haskell using algebraic data types and Template Haskell, but this is of little comfort to the vast majority of programmers since (a) most haven’t heard of Haskell and (b) Haskell is still in its web-development-language infancy.)

Moving forward I am interested in seeing whether interpolique passes peer review (probably will), becomes a common practice and reduces the incidence of code injection. Regardless of how these questions fare, the core idea is elegant, doesn’t seem to have a performance penalty, and can likely be carried forward fruitfully in future technologies.

• Cyclone, perhaps the most-cited example of using types to protect memory in low-level settings.
• Habit, a proposed Haskell dialect which uses a viable form of dependent types to model low-level data structures and their memory management which I recently learned about on reddit, coincidentally in a comment on an earlier post of mine.
• Some work due to Oleg Kiselyov and Chung-chieh Shan showing that Haskell is a viable setting for embedding a low-level language.

(If you know of others, put them in the comments!)

In December of 2009 I became interested in typed assembly languages and began working on my own, quickly deciding to embed the language into Haskell. At present there are many facets of the language which work well, but there is still a good deal of work to be done.

When I started this, I was unaware of much of the work that had already been done in this direction (I especially wish I had been aware of the Kiselyov-Shan proof of concept). In the course of the project, I’ve learned a bunch of great tricks, and also have learned of a good deal of excellent work done by others that I hadn’t seen promoted elsewhere. Over the next few posts, my goals are threefold:

• To describe my own project: where it is, what the challenges have been, where I hope for it to go. And to release some source (on github), such as it is.
• To describe some of the general lessons I’ve learned working on an EDSL in Haskell, to evangelize this approach to problem solving, and to describe some caveats to conventional Haskell wisdom in this setting.
• To promote some of the work others have done in this area, in the aim of showing just how far along this idea is.

The Potential programming language

Potential is an adaptation of x86-64 assembly language as a domain specific language, embedded in Haskell. It’s ultimate purpose is to provide a setting in which to implement a practical operating system kernel (the next iteration of the Kinetic project). The basic idea is pretty simple:

• Functions are written in Haskell which more or less correspond to many of the mnemonics in assembly language.
• The functions carry type signatures which encode the effect that the mnemonic has on machine state, in the form of an indexed (or parametrized) monad.
• These functions can be composed to form blocks of code. Haskell type inference can be used to determine the assumptions the code blocks make, as well as the condition they leave the machine in upon return.
• When these functions are executed, they output abstract syntax for x86-64 assembly, which can then be pretty-printed to a .asm file, which is then passed onto a toolchain for actually compiling the code.

Potential provides the following:

• An ability to safely handle bit-fields. Bit fields can be defined using Template Haskell, which will automatically generate code for getting and setting elements of the bit field. Potential uses type-level integers to keep track of the sizes of fields, thereby providing a static guarantee that data isn’t getting truncated along the way. (This idea has also been proposed independently in Habit). Potential is able to use this static data to compute the shifts and bit masks necessary for updating and using bit fields.
• Higher data structures (those built out of bit-fields) are commonplace in the x86-64 standard, and Potential provides a syntax for assembling these as well. Syntax is also provided for defining arrays of such structures, as is needed (for instance) in defining the interrupt descriptor table.
• A form of linear types, inspired directly by Kiselyov and Shan’s Lightweight monadic regions, for managing pointers.
• Overloaded >>, >>=, and return operators for use with do notation, allowing programmers to write their assembler in Haskell’s imperative syntax.
• Predefined versions of the important x86-64 data structures, as well as instructions for making type-level assertions about the state of the hardware. (For instance: are interrupts disabled? Is paging enabled? Is the current privilege level Kernel mode?)
• Type-level, static tracking of many assembly language subtleties. For instance, the assembly comparison mnemonic (cmp) releases a variable to the programmer which is then passed to the conditional mnemonics (such as je) as a way of demonstrating that the correct comparison is being examined. As an example, the code sequence
  

  compare1 <- cmp rax rbx
  compare2 <- cmp r09 r10
  sje <some function> compare1
  

  fails to type check because Potential knows that the result of the comparison labeled by compare1 is no longer stored in the CPU’s flags register. As more of assembly gets introduced into Potential, there will be more examples of this type of subtlety being brought directly to the programmer’s eyes, all modeled by types.

• Every good assembly language needs a good macro language to scrap boiler plate. In the case of Potential, all of Haskell is available as the macro language.
• Since the language is embedded in Haskell, the result is an assembly language which is essentially as polymorphic as Haskell is, including support for Haskell type class machinery.

All of the above is (at least partially) implemented today. The next major step (according to the plan) is to expand the the memory system so that it can deal with the complexities of paged memory.

Here are just a few of the simple things that are (in some cases, should be) possible in Potential:

• Code which must execute with interrupts disabled can assert this requirement on the level of types; Haskell will verify that no function invokes such code without first disabling interrupts.
• The processor flag which enables paged memory cannot be enabled unless the current interrupt descriptor table has a present vector for handling page faults.
• The task-switching interrupt handler can assert that processor state is preserved between task changes.
• If two registers carry pointers to the same address, and one of the pointers is used to modify a data structure in a way which modifies its type, the other pointer is implicitly invalidated, and subsequent attempts to use it will result in an error detected by the Haskell type system.

The language is very much still a work in progress with large amounts of rework taking place every few days (owing to the complexity of getting the types to line up with expectations and getting them to be inferred with the appropriate balance of generality and specificity).

Some examples of code written in Potential

Let’s look at some code which is used to modify an interrupt gate. Interrupt gates are used in x86-64 to describe interrupt handlers to the CPU. Structurally, they consist of a small number of bitfields which are used to indicate things like the privilege level that the interrupt handler should execute as (called the “descriptor privilege level,” or DPL). We’ll look at this particular field in these examples.

In Potential, the interrupt gate data structure is defined like so:

intGate = mkStruct "InterruptGate"
                   [ (field 16 "offset_lo")
                   , (field 16 "segsel")
                   , (field 3 "ist")
                   , ($(constField 2) CB0 CB0)
                   , ($(constField 3) CB0 CB0 CB0)
                   , ($(constField 4) CB1 CB1 CB1 CB0)  -- defines Interrupt Gate type
                   , ($(constField 1) CB0)
                   , (field 2  "dpl")
                   , (field 1  "p")
                   , (field 16 "offset_mid")
                   , (field 32 "offset_hi")
                   , ($(reservedField 32))
                   ]

We see the use of some Template Haskell as a matter of convenience (to allow for the constField variadic function). This defines various named fields that can be modified at run-time, as well as some constant fields, and some areas which are reserved for future changes to the x86-64 platform. As is the rule with Template Haskell, this code resides in a module (IntGateStruct) which is, in turn, loaded by another module (IntGate) that reifies it.

Once the IntGate module reifies the structure, Template Haskell is used to define a sequence of new types and top level definitions that can be used to modify InterruptGate structures. It introduces a new InterruptGate type:

> :info InterruptGate
newtype InterruptGate offset_lo segsel ist dpl p offset_mid offset_hi
  = InterruptGate (offset_lo,
                   segsel,
                   ist,
                   dpl,
                   p,
                   offset_mid,
                   offset_hi)
  	-- Defined at Potential/Machine/IntGate.hs:26:0-19

and provides us with some tools we can use to modify such types.

The setField instruction, for instance, is an example of a Haskell-built macro in Potential that can be used to update fields within such structures. Using it, we’re able to write a function that can be used to update the DPL of an interrupt gate:

-- rax contains Ptr64 to interrupt gate
-- rbx contains the new DPL
setDPL = asCode "setDPL" $
     do push rcx
        push rbx
        setField dpl rax rbx rcx
        pop rbx
        pop rcx
        ret

Here some (Haskell) code comments document that rax is used to pass in pointer to the interrupt gate we want to update, and rbx contains the new value for the DPL. The setField macro also needs a register to temporarily use: in this case, we’ve chosen rcx, whose value we are preserving by using the stack (the macro will also modify rbx, so we’re using the stack to preserve that value as well).

We can then ask Potential to translate this code into AT&T-syntax assembler (of course, we could modify the pretty-printing to use a different syntax). Here’s the result of that output (presuming I haven’t many any silly errors in the printer, this should be the right sequence of instructions for performing this operation — if you notice anything to the contrary, let me know!)

> renderFn setDPL
setDPL:
    push %rcx
    push %rbx
    // updating field "dpl" at (%rax) using value in %rbx, with %rcx as a temp register
    mov 0(%rax) %rcx
    shl 45 %rbx
    and 0xffff9fffffffffff %rcx
    or %rbx %rcx
    mov %rcx 0(%rax)
    // update complete
    pop %rbx
    pop %rcx
    ret

But what about types? Even though we haven’t given a type signature for setDPL, Haskell is (of course) able to infer one for us. We can inspect the type using the Potential function getType.

We can inspect the type of setDPL, as shown below. To make things manageable, I’ve elided much of the class-related output. You can see how type-level integers are used to encode assumptions about the side of arguments, in this case verifying that the arguments are (1) appropriately sized to be pushed onto the stack, and (2) appropriately sized to fill in the DPL field in the interrupt gate:

> :t getType setDPL
getType setDPL
  :: (SZ rcx
      :<= S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S Z))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))),
      SZ rbx
      :<= S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S (S Z))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))),
      SZ rbx :== S (S Z)) =>
     Function
       ConstraintsOn
       (MS
          (Ptr64
             h1
             (InterruptGate offset_lo segsel ist base p offset_mid offset_hi))
          rbx
          rcx
          rdx
          rsi
          rdi
          rbp
          (Ptr64 h (Stack (Ptr64 h11 b11) (Stack a' b')))
          rflags
          rip
          r08
          r09
          r10
          r11
          r12
          r13
          r14
          r15
          (Allocator hn hs cs)
          cmp)
       (MS
          (Ptr64
             (HS (HS hn))
             (InterruptGate offset_lo segsel ist rbx p offset_mid offset_hi))
          rbx
          rcx
          rdx
          rsi
          rdi
          rbp
          (Ptr64 b3 (Stack a' b'))
          rflags
          rip
          r08
          r09
          r10
          r11
          r12
          r13
          r14
          r15
          (Allocator
             (HS (HS (HS (HS (HS (HS hn))))))
             (C (HS (HS (HS (HS (HS hn))))) hs'5)
             (C b2 (C b1 (C h2 (C h1 (C b (C h cs)))))))
          cmp)

Notice that the Ptr64 type carries two things: the first is a handle, in the style of Kiselyov and Shan’s Lightweight monadic regions, and the second is the type of the data being pointed to.

As pointers are manipulated, the handles are closed and new ones issued, in order to statically prevent pointer aliasing from undermining type safety. As we see in this example, the handle for our interrupt gate pointer has changed from h1 to (HS (HS hn)). We can also see that, in the process, the h1 handle is removed from the list of open handles (the third field in the Allocator type records handles which have been closed).

Here is a typical example of why this is necessary: rax points to an interrupt gate whose DPL is set to User. After mov rax rbx, rbx now aliases this pointer. If we then update the DPL field via the rax pointer, we run into a situation where rax and rbx carry the same address but give it different types!

To get around this, whenever a structure manipulated in a way that changes its type, the pointer is modified to keep up: its handle (just prior to update) is closed and replaced with a new handle, reflecting the change. Any aliases of the pointer (prior to update) are now invalid, as tracked by the Haskell type system.

In fact, such trouble is indeed caught and detected by getType. Here we see the code for a function which does exactly what I described in this example: it aliases a pointer to an interrupt gate, then attempts to modify both pointers to wind up with an inconsistent DPL in the type. The code for this function:

testSetDPL2 = asCode "testSetDPL2" $
     do assumeType rbx (undefined :: PrivLevelUser)
        mov rax r10
        scall setDPL
        comment "Now rax has PrivLevelUser, r10 has unknown dpl"
        swap rax r10
        comment "Now rax has unknown dpl, r10 has PrivLevelUser"
        pop rbx
        assumeType rbx (undefined :: PrivLevelKernel)
        scall setDPL
        comment "Now rax has PrivLevelKernel, r10 has PrivLevelUser, but it's the same ptr"
        ret

The code will render (at present, the render function doesn’t verify types — only getType does this), and if we render it, we get the following (AT&T) assembly:

> renderFn testSetDPL2
testSetDPL2:
    mov %rax %r10
    call setDPL
    // Now rax has PrivLevelUser, r10 has unknown dpl
    // swapping %rax with %r10
    push %rax
    mov %r10 %rax
    pop %r10
    // swap complete
    // Now rax has unknown dpl, r10 has PrivLevelUser
    pop %rbx
    call setDPL
    // Now rax has PrivLevelKernel, r10 has PrivLevelUser, but it's the same ptr
    ret

However, once we try to inspect the type, we get the following:

> :t getType testSetDPL2

Top level:
    Couldn't match expected type `Potential.Handles.False'
           against inferred type `Potential.Handles.True'
    When using functional dependencies to combine
      Potential.Handles.LOr
        n1 Potential.Handles.True Potential.Handles.True,
        arising from the dependency `n1 n2 -> t'
        in the instance declaration at Potential/Handles.hs:101:9
      Potential.Handles.LOr
        t Potential.Handles.True Potential.Handles.False,
        arising from a use of `testSetDPL2' at :1:8-18

(Of course, ghci isn’t giving us very useful information about why the failure happened; more work is definitely needed in this area. I expect that type families could be used to replace my functional dependencies, and would give much more meaningful errors, but I believe this will require equality constraints in superclasses, which is why I haven’t yet gone down that road.)

Although it is not currently implemented, it should be possible (once a memory allocator is written) to extend this system to also ensure that memory is freed at appropriate times. For instance, when writing the task manager for an operating system kernel, one will need to allocate memory as tasks are spawned. It should be possible to extend the handle system to track when memory is allocated for a particular task, so that when that task exits, it can be ensured that all associated memory is freed. I’m unaware of any trick in C or C++ that would allow such a guarantee to be made.

Here is another example. In this example, test2 is just a simple function whose behavior is immaterial — it just provides a place that we can conditionally jump to.

Consider the code

test11 = asCode "test11" $
     do pop rax
        pop rbx
        pop rcx
        rabxCmp <- cmp rax rbx
        racxCmp <- cmp rax rcx
        sje test2 racxCmp
        ret

We see two comparisons being performed. The latter comparison is used to conditionally jump to test2. If we inspect the type of test11, we get what we’d expect (again we elide much of the class constraint output for the sake of brevity):

> :t getType test11
getType test11
  :: Function
       ConstraintsOn
       (MS
          rax
          rbx
          rcx
          rdx
          rsi
          rdi
          rbp
          (Ptr64
             h
             (Stack
                Int64
                (Stack
                   Int64
                   (Stack
                      Int64
                      (Stack (Ptr64 h11 b11) (Stack (Ptr64 h12 b12) (Stack a' b')))))))
          (FlagsRegister cf pf af zf sf tf if df of' iopl rf ac vif vip id)
          rip
          r08
          r09
          r10
          r11
          r12
          r13
          r14
          r15
          (Allocator hn hs cs)
          cmp)
       (MS
          Int64
          Int64
          Int64
          rdx
          rsi
          rdi
          rbp
          (Ptr64 b5 (Stack a' b'))
          (FlagsRegister
             (CF (CS (CS cmp)))
             (PF (CS (CS cmp)))
             (AF (CS (CS cmp)))
             (ZF (CS (CS cmp)))
             (SF (CS (CS cmp)))
             tf
             if
             df
             (OF (CS (CS cmp)))
             iopl
             rf
             ac
             vif
             vip
             id)
          rip
          r08
          r09
          r10
          r11
          r12
          r13
          r14
          r15
          (Allocator
             (HS (HS (HS (HS (HS (HS (HS (HS hn))))))))
             (C (HS (HS (HS (HS (HS (HS (HS hn))))))) hs'6)
             (C b4 (C b3 (C b2 (C h1 (C b1 (C b (C h cs))))))))
          (CS (CS cmp)))

The very last line indicates that two comparisons have been performed in this block.

Now suppose we modify the function to introduce a bug:

test11 = asCode "test11" $
     do pop rax
        pop rbx
        pop rcx
        rabxCmp <- cmp rax rbx
        racxCmp <- cmp rax rcx
        sje test2 rabxCmp
        ret

The change is subtle — here we are now attempting to conditionally jump on rabxCmp, which is a comparison whose results should no longer be held in the flags register. Indeed, Potential catches this bug when we attempt to load the code in ghci:

TestCode.hs:54:1:
    Occurs check: cannot construct the infinite type: cmp1 = CS cmp1
      Expected type: PState
                       Instr
                       c
                       (Potential.MachineState.Set
                          Potential.MachineState.RRflags
                          (FlagsRegister
                             (CF (CS (CS cmp1)))
                             (PF (CS (CS cmp1)))
                             (AF (CS (CS cmp1)))
                             (ZF (CS (CS cmp1)))
                             (SF (CS (CS cmp1)))
                             tf
                             if'
                             df
                             (OF (CS (CS cmp1)))
                             iopl
                             rf
                             ac
                             vif
                             vip
                             id)
                          rax
                          rbx
                          rcx
                          rdx
                          rsi
                          rdi
                          rbp
                          (Ptr64 h (Stack (Ptr64 h1 b1) (Stack a' b')))
                          (FlagsRegister
                             (CF (CS cmp1))
                             (PF (CS cmp1))
                             (AF (CS cmp1))
                             (ZF (CS cmp1))
                             (SF (CS cmp1))
                             tf
                             if'
                             df
                             (OF (CS cmp1))
                             iopl
                             rf
                             ac
                             vif
                             vip
                             id)
                          rip1
                          r081
                          r091
                          r101
                          r111
                          r121
                          r131
                          r141
                          r151
                          alloc
                          (CS (CS cmp1)))
                       y
                       a
      Inferred type: PState
                       Instr
                       c
                       (MS
                          rax
                          rbx
                          rcx
                          rdx
                          rsi
                          rdi
                          rbp
                          (Ptr64 h (Stack (Ptr64 h1 b1) (Stack a' b')))
                          (FlagsRegister
                             (CF (CS (CS cmp1)))
                             (PF (CS (CS cmp1)))
                             (AF (CS (CS cmp1)))
                             (ZF (CS cmp1))
                             sf
                             tf1
                             if
                             df1
                             of
                             iopl1
                             rf1
                             ac1
                             vif1
                             vip1
                             id1)
                          rip
                          r08
                          r09
                          r10
                          r11
                          r12
                          r13
                          r14
                          r15
                          (Allocator hn hs cs)
                          cmp)
                       (MS
                          rax
                          rbx
                          rcx
                          rdx
                          rsi
                          rdi
                          rbp
                          (Ptr64 b (Stack a' b'))
                          (FlagsRegister
                             (CF (CS (CS cmp1)))
                             (PF (CS (CS cmp1)))
                             (AF (CS (CS cmp1)))
                             (ZF (CS cmp1))
                             sf
                             tf1
                             if
                             df1
                             of
                             iopl1
                             rf1
                             ac1
                             vif1
                             vip1
                             id1)
                          rip
                          r08
                          r09
                          r10
                          r11
                          r12
                          r13
                          r14
                          r15
                          (Allocator (HS (HS (HS (HS hn)))) hs'1 cs')
                          cmp)
                       ()
    In a stmt of a 'do' expression: sje test2 rabxCmp
    In the second argument of `($)', namely
        `do { pop rax;
              pop rbx;
              pop rcx;
              rabxCmp <- cmp rax rbx;
              .... }'

There’s a lot of output, but towards the bottom we see that ghci correctly points to sje test2 rabxCmp as the source of the trouble.

Source preview

At present time, nothing in Potential is being taken as “stable,” so the code is not available on Hackage. However, the project is on github at Potential. The code as of this blog post is in the initial-blog-announce branch.

Basic usage is pretty simple. To play around a bit, get started like so:

$ ghci -fcontext-stack=160 TestCode.hs
*TestCode> renderFn test1
test1:
    pop %rax
    pop %rbx
    pop %rbx
    // swapping %rax with %rbx
    push %rax
    mov %rbx %rax
    pop %rbx
    // swap complete
    ret

(The large context stack is there for the type-level integers.)

Since Potential makes heavy use of optional class constraints, the best way to inspect the type of a function is with :t getType <function name goes here>.

What’s next (Potential-wise)

The system that tracks pointers is still being adjusted. It seems to be doing its job, but it can undoubtedly be improved in terms of producing good human-readable output via getType. Ultimately I’d like to move the system away from functional dependencies, but without equality constraints in superclasses, the illegality of incoherent instances for associated types seems to make this infeasible.

There’s also a good deal of basic assembly that isn’t present in the language right now. jmp can be invoked on top-level defined functions (it’s name in Potential is sjmp for “static jump”), but is not currently implemented for non-literals (that is, jmp rax isn’t implemented).

The only conditional jump instruction in the current source is sje, which permits conditional jumps on equality to a top-level-defined function. At present it is just a hack — type inference with this instruction isn’t quite correct. (This problem relates to verifying that the instructions following the sje mnemonic are typed the same as the code which might be jumped to. This condition, currently missing, is essentially the same as the condition in Haskell that the then and else blocks in an if statement must have the same type.)

For me, the biggest question is one of scale: while Potential seems capable at managing my small examples, will it be practical to write an operating system in this language? Will Potential’s promises of static checking via types deliver? I believe it might, but this issue is far from settled.

What’s next (blog-wise)

In the days to come, I will give some more examples of code written in Potential, as well as some downloads that can be played with. I’ll also dig into some of the issues that have come up during the design and implementation of the language, much in the style of my posts on optional class constraints and polymorphic first-class labels, which both arose during this project. I’ll also describe some related work that has really come in to save my skin design-wise.

My EDSL is essentially an attempt at grafting the Haskell static type system onto a dynamically typed language. The idea is that a programmer writes their code in my EDSL — that is, he’s really written his code in an indexed monad I’ve set up in Haskell — and when he executes his EDSL code, the output is a bunch of code written in the target language, which can then be compiled using whatever toolchain he likes. We’ve basically embedded the target language into Haskell, which means that the Haskell type system can now be used to model concepts in the target language. The target language has a very weak dynamic type system which, if I’ve done my modeling correctly, allows a proper superset of the programs that my EDSL will allow.

Anyway, with this design, there are two operations that the programmer needs to be able to perform with their code:

• Make sure that the code type-checks. Obviously the Haskell compiler will do this on its own.
• If the code type-checks, render the code from the EDSL into the target language.

As alluded to above, the EDSL resides within an indexed monad, say PState assumes returns a, which encodes the Hoare types of code blocks. Consequently, there is a render function, appropriately typed render :: PState assumes returns () -> TargetAST, where TargetAST is some algebraic data type that I can pretty print into the target language’s syntax.

Obviously I’m leaving out a good deal of detail about the EDSL, which I’ll talk about in a later post; today I want to talk about a problem that arises with this style.

It is a common theme that the true zen of Haskell code is modeling your program’s behavior in the type system. As a basic example, a program that takes user input and puts it into a syntax-sensitive structure can use types to ensure that non-escaped strings never make it from the user to the structure; as a more complex example, configuration flags can be managed at the type level, if one is sufficiently clever. These are all ideas that would be wonderful to have in other languages, but which are typically absent due to the limitations of the type systems in question.

Of course, when one wishes to encode program behavior into types, one often works into a box where type classes need to be introduced. I recently found myself: my target language has a notion of “operand size,” and some functions in the language only work with certain sizes of data; it was thus necessary to have a representation for this in the EDSL. Since operand size is a property of a type, rather than a type itself, this introduced type classes.

All of this is fine, of course, when it comes to type checking: I was able to employ a common trick to encode sizes in types, then have a two-parameter class

class HasSZ d size

whose only job is to describe a size predicate on a type. But now I have EDSL code with a type signature predicated on a class constraint, as in

someEDSLFunction :: HasSZ d (S (S Z))
        => PState (some complex expression using d) returns ()

Since the implementation of render is essentially (using ScopedTypeVariables) just

render :: PState assumes returns () -> TargetAST
render f = getTargetAST $ runPState f (undefined :: assumes)

we now have a problem: if f has a class constraint like that found in someEDSLFunction, we get a type error:

> render someEDSLFunction 

:1:7:
    No instance for (HasSZ d (S (S Z)))
      arising from a use of `someEDSLFunction' at :1:7-22
    Possible fix: add an instance declaration for (HasSZ d (S (S Z)))
    In the first argument of `render', namely `someEDSLFunction'
    In the expression: render someEDSLFunction
    In the definition of `it': it = render someEDSLFunction

This is particularly frustrating because (though I haven’t yet explained this) we’re using undefined for the sole reason that we don’t want the code which is generated to depend on the input — that is, the class constraints on the input are only a formality, there to make sure that function calls within the EDSL are legal in the context of the target language!

So that’s the problem. Here’s the fix: optional class constraints. The central issue is that, during the type-checking phase, we want the class constraints to be there — but during the rendering phase, we wish they weren’t. We can achieve this by introducing a new type class, along with some suspicious looking instances.

In addition to the HasSZ class, we’ll introduce the following class (and instances):

data ClassConstraintsOn
data ClassConstraintsOff

class MaybeHasSZ d size c
instance (HasSZ d size) => MaybeHasSZ d size ConstraintsOn
instance MaybeHasSZ d size ConstraintsOff

We then take our indexed monad, PState, and add an extra type parameter to track whether or not we want class constraints enabled or disabled. Functions in our EDSL now have type PState c assumes returns a.

We now give the function someEDSLFunction a signature like

someEDSLFunction :: MaybeHasSZ d (S (S Z)) c =>
        PState c (some complex expression using d) returns ()

By coupling the class constraint with the c variable in our PState, we’ve given ourselves a way to “flip a switch” to turn the HasSZ constraint on and off: during type checking, the EDSL binds c to ConstraintsOn, so that the constraint MaybeHasSZ d (S (S Z)) c reduces to the constraint HasSZ d (S (S Z)). To fix rendering, we just give render the signature

render :: PState ConstraintsOff assumes returns () -> TargetAST

so that the expression render someEDSLFunction collapses the constraint MaybeHasSZ d (S (S Z)) c down to nothing, allowing the render function to do its job.

The expression “first class labels” refers to the idea that, for record data types, one should be able to pass around the labels just as they would any other type. For instance, if I have a record like

data Foo a b = { biz :: a, baz :: b }

the value biz shouldn’t just denote the function biz :: Foo a b -> a, but should also be usable as a way of updating records, that is, a function like biz' :: Foo a b -> a' -> Foo a' b.

The Mythical Haskell’ includes some proposals for updating the records system with features aimed at supporting this idea, but for the time being, many people prefer to use fclabels, which achieves much of this magic using Template Haskell.

Recently, while working on an EDSL, I found myself wishing I had first class labels. I ran into a problem, though, which (along with solution) I’ll now describe.

Consider the following code:

module Label where

data Foo a b = Foo a b deriving Show

updatea :: a' -> Foo a b -> Foo a' b
updatea a (Foo _ b) = Foo a b

updateb :: b' -> Foo a b -> Foo a b'
updateb b (Foo a _) = Foo a b

worksFine foo0 = let foo1 = updatea 'a' foo0
                     foo2 = updatea "a" foo1
                 in foo2

Here I’ve defined a data structure Foo with two fields, along with a pair of functions for updating these fields. Then I defined a function worksFine which uses updatea to modify a Foo.

Obviously I could also write the following function:

worksFine' foo0 = let foo1 = updateb 'a' foo0
                      foo2 = updateb "a" foo1
                  in foo2

which is exactly the same, except that it uses updateb instead, thereby modifying the other field in Foo.

So now we have an obvious place to generalize: instead of having both worksFine and worksFine', why not have a single function which takes the updater as a parameter?

If we try it out, the first attempt looks like this:

trouble u foo0 = let foo1 = u 'a' foo0
                     foo2 = u "a" foo1
                 in foo2

Only trouble is that this fails to type check:

    Couldn't match expected type `[Char]' against inferred type `Char'
      Expected type: [Char] -> t1 -> t
      Inferred type: Char -> t2 -> t1
    In the expression: u "a" foo1
    In the definition of `foo2': foo2 = u "a" foo1

The problem is that trouble doesn’t believe that the argument u is polymorphic enough. This is a typical rank-2 issue: we don’t want trouble to bind the type variables in the signature for u.

Using rank-2 types, we can get very close to a solution. We can write the functions

alsoWorksFinea :: (forall a a' . a' -> Foo a b -> Foo a' b)
               -> Foo a b -> Foo [Char] b
alsoWorksFinea u foo0 = let foo1 = u 'a' foo0
                            foo2 = u "a" foo1
                        in foo2

alsoWorksFineb :: (forall b b' . b' -> Foo a b -> Foo a b')
               -> Foo a b -> Foo a [Char]
alsoWorksFineb u foo0 = let foo1 = u 'a' foo0
                            foo2 = u "a" foo1
                        in foo2

but neither is able to accept both of our update functions, even though each function has exactly the same body. Worse, I wasn’t able to find a sufficiently general type signature that would allow me to have one function which would be able to accept both update functions.

Luckily, where rank-2 types have failed me, type families have saved me. Any time you need more flexibility in your type signatures than the syntax will allow, you might be in a box where type families are the way to go. Here’s what it looked like in my case.

First I defined some typed to represent the two fields of my Foo structure:

data A = A
data B = B

Obviously I can pass around these values in a first-class manner, no trouble at all.

Then I defined a class for describing updating and getting:

class Field f x y where
  type Updated f x y
  update :: f -> x -> y -> Updated f x y
  type Gotten f y
  get :: f -> y -> Gotten f y

Now there are two instances to give, one for each field in Foo:

instance Field A a' (Foo a b) where
  type Updated A a' (Foo a b) = Foo a' b
  update A a' (Foo a b) = Foo a' b
  type Gotten A (Foo a b) = a
  get A (Foo a b) = a

instance Field B b' (Foo a b) where
  type Updated B b' (Foo a b) = Foo a b'
  update B b' (Foo a b) = Foo a b'
  type Gotten B (Foo a b) = b
  get B (Foo a b) = b

And we’re basically done. We can now write the function that started this whole mess:

shouldWorkFine f foo0 = let foo1 = update f 'a' foo0
                            foo2 = update f "a" foo1
                        in foo2

GHCi is able to give us a very promising type signature for it:

> :t shouldWorkFine
shouldWorkFine
  :: (Field f [Char] (Updated f Char y), Field f Char y) =>
     f -> y -> Updated f [Char] (Updated f Char y)

While this technique introduces type classes and type families into our program (something which can make typing troublesome in other areas), it delivers something I don’t know how to otherwise get: polymorphic first class labels.

Clearly the next step is to implement a library like fclabels which uses Template Haskell to define instances of the Field class.

I’ll start with the code. As this example uses Template Haskell, we need to have the source broken up into two files:

Testa.hs:

{-# LANGUAGE
        TemplateHaskell #-}
module Testa where

import Language.Haskell.TH

someth = [| () |]

unit = ()

mrIssue :: (Monad m) => b -> m b
mrIssue = return

Testb.hs:

{-# LANGUAGE
        NoMonomorphismRestriction,
        TemplateHaskell #-}
module Testb where

import Testa

g = $(someth)
foo1 = mrIssue $(someth)

f = ()
foo2 = mrIssue ()

h = unit
foo3 = mrIssue unit

Now, if we fire up ghci Testb.hs, we get the following:

$ ghci Testb.hs
GHCi, version 6.10.4: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer ... linking ... done.
Loading package base ... linking ... done.
[1 of 2] Compiling Testa            ( Testa.hs, interpreted )
[2 of 2] Compiling Testb            ( Testb.hs, interpreted )
Loading package syb ... linking ... done.
Loading package array-0.2.0.0 ... linking ... done.
Loading package packedstring-0.1.0.1 ... linking ... done.
Loading package containers-0.2.0.1 ... linking ... done.
Loading package pretty-1.0.1.0 ... linking ... done.
Loading package template-haskell ... linking ... done.
Ok, modules loaded: Testb, Testa.
*Testb> :t foo1
foo1 :: (Monad m) => m ()
*Testb> :t g

:1:0:
    Ambiguous type variable `m' in the constraint:
      `Monad m' arising from a use of `g' at :1:0
    Probable fix: add a type signature that fixes these type variable(s)
*Testb>

Notice that, while we’re able to get the type for foo1, for some reason g is ill-typed with an ambiguous type variable in the constraint Monad m. The only problem, of course, is that when we look at our source we don’t see any way in which the type for g should have this constraint!

So maybe the problem is that g needs a type signature. But if we go in and modify Testb.hs, giving it

g :: ()
g = $(someth)

then we get the following from ghci:

Testb.hs:10:7:
    Could not deduce (Monad m) from the context ()
      arising from a use of `mrIssue' at Testb.hs:10:7-23
    Possible fix:
      add (Monad m) to the context of the type signature for `g'
    In the expression: mrIssue ($someth)
    In the definition of `foo1': foo1 = mrIssue ($someth)
Failed, modules loaded: Testa.
*Testa>

So now it's upset about the type for foo1. Fine. Let's give it a signature as well:

g :: ()
g = $(someth)
foo1 :: (Monad m) => m ()
foo1 = mrIssue $(someth)

Now we get a new error from ghci:

Testb.hs:11:0:
    Contexts differ in length
      (Use -XRelaxedPolyRec to allow this)
    When matching the contexts of the signatures for
      g :: ()
      foo1 :: forall (m :: * -> *). (Monad m) => m ()
    The signature contexts in a mutually recursive group should all be identical
    When generalising the type(s) for g, foo1
Failed, modules loaded: Testa.

If we add RelaxedPolyRec to our list of LANGUAGE extensions, the problem does, indeed, go away. In this case, we can even remove our type signature for g or for foo1, but not both -- we need to have at least one of them present.

Lastly, if we go back to our original source given above, but replace the signature for mrIssue with

mrIssue :: b -> IO b
mrIssue = return

then we can remove the NoMonomorphismRestriction, and everything works just fine (we can :t g and :t foo1 without any problems).

I'm entirely unsure of what's going on here. Any theories?