Go I/O is Fun*!

* For certain definitions of fun.

One of my favorite parts of Go is the interface type. There has been sufficient hate recently about the missing generics but, to be honest, I haven’t missed them one bit. One of the reasons I haven’t missed them is the use of interfaces in go, especially in the standard library. The io package is a great example.

Whether you open a file, read from a network, or request a web page, they all implement the io.Reader interface. This means that your code can be generic in the sense that the source of the data is irrelevant.

The notion of interfaces allow for very interesting design patterns. One pattern that I’ve found useful lately is to wrap an interface with some logic, but provide the same interface so that existing code doesn’t need to change. Let me give you an example of what I mean.

Suppose that you already have an application that POSTs data to a web server for permanent storage. Right now your code accepts an io.Reader and uses that as the data being POSTed. Now lets say you want encrypt that data beforehand. One option would be to encrypt the data to another file and then use that file’s io.Reader to send the data.

It’s not a bad solution, but a more elegant solution would be to encrypt the data on the fly before it’s sent to the server. There would be little overhead and no additional disk space or I/O would be required. We can do this because http.Request accepts any io.Reader. Our encryption interface need only implement the io.Reader interface and then the http.Request will happily use it.

This has become common enough for me that I’ve written a library called wrapio. It wraps a given io.Reader or io.Writer with some mutator or inspector. As data is passed through, it can be manipulated or checked without making huge copies in memory or disk.

A working example of this can be found at this gist. Let’s walk through some of the code.

block = wrapio.NewBlockReader(bme.BlockSize(), in)
last = wrapio.NewLastFuncReader(pad, block)
crypt = wrapio.NewFuncReader(func(p []byte) {
	bme.CryptBlocks(p, p)
}, last)

First, since we are using AES to encrypt the data, we need to do some things before we do the actual encryption. Block mode ciphers can only encrypt blocks whose length is a multiple of the block size. We use the NewBlockReader to wrap the input around another io.Reader that only sends data in full blocks, except possibly the last block. To accommodate that last block being smaller than the block size, we use the NewLastFuncReader to call the pad function on the last block. It basically looks at the last block and adds some padding to the end of the block if necessary. Finally, we wrap that io.Reader in another io.Reader that does the encryption before passing it along.

In just a few lines of code we’ve implemented an on-the-fly encryption mechanism that can send the data anywhere an io.Reader is used. Existing parts of your application that use an io.Reader can generally accept it without any changes.

What if we are getting the data back from the server and want to decrypt it? No problem, we just do the same with the io.Reader we get back from the response body.

block = wrapio.NewBlockReader(bmd.BlockSize(), in)
crypt = wrapio.NewFuncReader(func(p []byte) {
	bmd.CryptBlocks(p, p)
}, block)
last = wrapio.NewLastFuncReader(unpad, crypt)

The order is slightly different because we want to decrypt the block before we remove the padding from the end of the data.

I can’t vouch for the effectiveness of the encryption being done here, but I think it gives you a good idea of how powerful interfaces are. By wrapping the I/O interfaces you can do just about anything with your data as it’s being passed through. Functions like ioutil.ReadAll and io.Copy become extremely useful and in some of my cases have become the only function call in the main loops of my programs.

As always, feedback, pull request, comments, etc. are welcome. Enjoy your I/O!