Reasonable Performance

Insights and understanding — AI won't take your job

Sun, 07 Apr 2024 00:00:00 +0000

When I last visited my parents, my mom and dad asked me whether I was worried about ChatGPT or other AI systems taking away programmers’ jobs and whether I was worried about my own future. My answer to them—and this might be a naïve take—is that I’m not very worried and that I think that programmers are going to be around for a while yet.

The reason I think this is because the current breed of AIs that make headlines are generative AIs, i.e., AIs that are able to produce text, images, songs, videos, etc. Importantly, those AIs do not understand what they output. Generative AIs are really good at stringing together words and symbols in an order that makes them look intelligent, but it’s just a very impressive illusion; they don’t have the capacity to do real analysis.

Programmers, contrary to the common perception, don’t just spend all day writing code at a keyboard without thought. The important part of our job is the time we spend understanding a problem. We use our ability to recognize patterns, to notice oddities, to question the status quo, to wonder “what if” to augment our understanding of a problem and of its solution space. This ability to take fragments of information—code, data, timings, test programs, colleagues, documentation, dashboards, etc.—, to establish links between them, and to derive new knowledge, that to me is real intelligence, and as far as I know, it’s still only a human thing.

The recent xz backdoor provides a good example of this. Andres Freund noticed that his OpenSSH daemon was taking more CPU than usual and that SSH connections were taking longer than usual. His initial observation and subsequent digging unveiled an extremely sophisticated attack that could’ve created a huge cybersecurity nightmare. GitHub CoPilot might be good at auto-completing the stanza for opening a file in Java, but unless it got an impressive upgrade in recent days, it cannot notice a slow SSH connection, use debugging tools to find the the source of the CPU usage, notice that the slowdown comes from an suspicious unnamed symbol in a compression library, decipher the complex dance for creating the backdoor, and reporting its findings to the community.

Obviously, the day-to-day of most programmers doesn’t involve finding hidden backdoors, nevertheless, this ability to understand a problem, to investigate, to derive insights is primordial for doing our jobs. Generative AIs are not going to replace programmers; they might become a useful addition to our toolbox, but when the dust has settled, their usage will certainly be more modest than the current hype predicts.

Here are some ways to gain understanding and insights:

Inspect your data regularly: programs transform data, so make sure you know how to look at your data and how to “ask it questions”; whether it’s with SQL queries, awk scripts, or jq filters, being able to slice-and-dice data is a good way to find places where improvements can be made. Make sure you are given access to servers, dashboards, cloud consoles, etc. that you’ll need.
Revisit your code: how you think your program performs a task and how the program actually does the task might drift as people maintain the code. Revisiting sub-systems once in a while is a good way to see if they’ve changed and also, it might be only the third read that a spark of insight strikes you.
Talk to your colleagues: if something looks weird in the code and you can’t figure out why, asking a question like “does anybody remember why we do x in module foo?” is a good way to start a conversation, understand contraints or concerns that aren’t evident from the code (aside: there’s no such thing as “self-documenting code”), and might cause the team to revisit previous assumptions.
Trust, but verify: it’s common for a piece of incorrect knowledge to spread among a team and for everyone to accept it and not verify it. When someone says “we can’t do x because y and z”, it’s useful to verify. It might have been true in the past, but it might no longer be; it might have never been true in the past, but someone misread the documentation and came away with the wrong conclusion. Verify.

Simple Precedence

Sun, 04 Feb 2024 00:00:00 +0000

A discussion between Jonathan Blow and Casey Muratori on the handling of precedence in Jon’s compiler recently popped in my YouTube feed. The discussion is three hours long and focuses on implementing operator precedence more easily and more simply in Jai using Pratt parsing. Jon and Casey also talk about the previous implementation of operator precedence in Jai which used tree-rewriting and the classic approach which organizes the different precedence levels into different production rules.

My favorite approach to operator precedence is the one used in APL, J, and K: get rid of it. Array languages are known for their symbol-heavy programs, but something rarely mentioned is that in languages with so many esoteric operators, the rules for reading those operators are the simplest and most straight-forward of all:

All operators have the same precedence, no exception.
All operators associate to the right, no exception.

With these two rules, it’s easy to understand a complex expression: we read right-to-left. Take the following snippet of code (written completely randomly):

5 | 1 << 3 & 3 | 1 << 4

In a language like Python or C, do you know what this expression evalutes to without running it or referring to the documentation? In a language with only the two precedence rules above, it’s very easy:

1 << 4 = 16
3 | 16 = 19
3 & 19 = 3
1 << 3 = 8
5 | 8 = 13

Having only two simple rules makes it easy for readers of programs to understand complex expressions. The simple rules also make the life of the language implementors easier: instead of deciding whether to use Pratt Parsing or multi-leveled productions, and deciding on whether << should have a higher or lower precedence than &, the parsing of operators becomes something that doesn’t require a three hour discussion.

The one thing that is lost by going for this radically simpler operator precedence is familiarity. We’ve had PEDMAS drilled into our skulls since junior high; remembering PEDMAS is extremely important for those quizzes on Twitter claming that “only 1% of the population gets this question right”. But if we are willing to give up familiarity for a little while—I say for a little while, because eventually the no-precedence approach becomes second nature—we can have a language that’s simpler to understand and simpler to implement.

Unfortunately, this kind of radical simplicity is rarely seen because programmers don’t value simplicity very much and don’t mind the extra complexity of things like precedence rules—they even like extra, unnecessary complexity. But that’ll be the subject of another post…

My Impressions of Hare

Wed, 24 Jan 2024 00:00:00 +0000

When Hare was announced back in 2022, I saw the post on Hacker News, went to the website, looked at the documentation, and my reaction was a lukewarm “meh”. Hare was a new entry in what I call the “Modern Cs”: languages that use C as their base inspiration and try to improve upon C by adding new features, removing some footguns, and polishing some sharp corners. These languages also try to avoid the complexity and cognitive load of C++, Ada, or Rust by omitting certain features (e.g., exceptions, RAII). The Modern Cs include:

Zig: Probably the best-known Modern C, its main claim to fame is comptime, the ability to run Zig code at compile-time. This one feature is used pervasively in the language and it subsumes many features of other language: for example, a generic data structure can be implemented by calling a comptime function which accepts a type and returns a struct where the input type has been filled in.
Odin: Another well-known Modern C, Odin is a simpler language than Zig and it draws its inspiration from C, but also Pascal and Go. One cool feature of Odin is its implicit context argument which allows custom loggers or memory allocators to be passed around without being explicitly in a function’s parameters’ list.
C3: A language that attempts to stay closer to C than the others, it still manages to improve upon C with better error handling, modules, generics, and cleaner semantics.
Jai: Jonathan Blow’s language which is currently in private beta. Like Zig, the big idea of Jai is to allow any part of a program to be run at compile-time by prefixing a statement with #run. The language is currently used for the creation of a AAA game, but no one knows when it’ll be available to the general public.

In my first quick look at the documentation of Hare, I saw a language that clearly improved upon C: it had better error handling, a stronger type system, bound-checked slices, defer for resource cleanup, UTF-8 support, and fewer syntactic gotchas than C (e.g., no fallthrough in switches, more intuitive declaration syntax). Hare doesn’t have generics, macros, or any means of doing compile-time programming, which was what cooled my interest for the language. Also, unlike the other languages on this list, Hare uses QBE for its backend instead of LLVM which can raise questions about how fast the generated programs are. At the time, I thought Hare was cute, but not a language I was interested in or a language that I thought I could be interested in.

Fast-forward a couple years, and I hear Drew DeVault, Hare’s creator, on the podcast Developer Voices talking about Hare. I liked what Drew had to say, many of his sensibilities about programming languages matched my own, and since I had a week off for Christmas and New Year, I figured that I should give the language a try. When I tried Hare, something unexpected happened: I liked it more than I expected! The language is more basic than the other Modern Cs and boasts fewer of the features I look for in a language, yet I found that it was the one that most quickly and most comfortably “fit my hand”. I learned the basics of the language in a couple of hours and the built-in haredoc command let me explore the standard library from the comfort of my terminal. Before long, I could write small programs in Hare without consulting the documentation at every line. The more I used Hare, the more I realized that I had previously judged it as just a list of features; now that I was actually using it, the holistic experience was better than a simple reading of the features led me to believe.

Some of my readers may know that I used to be quite a fan of very high-level programming languages, of functional programming, of advanced type system features, etc. However, over the past few years, age has made me become more conservative in my programming language preferences—I now value simplicity more than I did before and I find myself scowling at some of the advanced modern programming features, grumbling that they exist to solve uncommon problems or, worse, to provide an exciting toy to otherwise bored programmers. Were I 5–10 years younger, I would totally blow off Hare, but now I value different things in the software that I use and build, Hare appeals to me a lot more.

One aspect of Hare that I will follow with interest is its evolving story toward safety. Hare is already a lot safer than C: the compiler inserts bound check for array and slice accesses, the compiler refuses to compile code which doesn’t handle error conditions, strings don’t have the ambiguity of C where you don’t know if length includes the terminating NUL byte or not, etc. A lot of the common C footguns are nullified. However, the language does not track ownership and lifetimes—the programmer must have that discipline—which opens Hare programs to use-after-free and double-free bugs. There are early discussions of using linear types to prevent those kinds of errors; time will tell whether the Hare developers are able to integrate them in the language without sacrificing the simplicity and “feel” of the language.

I’ll finish by repeating how pleasantly surprised I was by Hare. If you value simplicity, directness, and performance, I recommend you give it a go for few hours; I think you’ll also be pleasantly surprised.

The Tyranny of Existing Code

Sun, 12 Mar 2023 00:00:00 +0000

Most programmers have experienced that feeling of worthlessness and powerlessness when we are trying to implement what should be a simple change, but the existing code base resists our attempts. Worse, often we can see what is causing us to be blocked, yet we cannot bring ourselves to bring down those barriers. I call this the tyranny of existing code.

In a young software project, one with little infrastructure in place, it’s usually straightforward to make changes. However in a project that’s older, one where other programmers have put in place structures to make the codebase “clean”, “maintainable”, or “extensible”, it’s often much harder to make changes. Often, the structures that have been put there “to make changes easier” or “to make it harder to shoot yourself in the foot” are the very ones that block us.

And it is the tyranny of existing code that will prevent us—at least initially—from taking the steps needed to unblock ourselves. We reason that if there are restrictions in place, “they must be there for a reason”. We think about the time that the the original author must have spent to design and implement those structures and we don’t want to insult them by deleting their code. We certainly don’t want to put ourselves in a position where we think we know more than them, only to be shown to be wrong. We show deference to the existing structure by trying to work within its constraints and limitations, even as it impedes our ability to solve the problem. We day dream of ripping out all that shitty code and replacing it with our own and how much better it would all be; but we quickly return to reality, continue grinding at the problem, and continue finding new ways in which the current code blocks us. And so we waste hours, days, even weeks, letting the existing code make our lives miserable.

For example, suppose that the web framework of our application hides the HTTP headers behind a dictionary-like abstraction and only provides the ability to get a specific header.

impl HttpRequest {
    fn get_header(header_name: &str) -> Option<&str>;
}

If we were tasked with logging all the headers of incoming requests, how would we do it? The current interface is clearly insufficient—we need access to all the headers—but what can we do? We might think about replacing the framework for one that does not hide the headers, but that suggestion sounds ludicrous and risky and will make us sound naïve and unprofessional. So we start looking for ways to work within the framework. Maybe there’s a new version that exposes the headers? Maybe there’s a middleware system that let us access the raw request? There’s probably, definitely a reason why the original author didn’t want us to have access to all the headers—we don’t know what kind of problems they have saved us from by not giving us access to all the headers and we should be thankful; really, it’s our task that’s the problem!

In time, we find a solution, make our change, and we feel good about ourselves; we overcame the difficulties, our manager congratulated us at the stand-up, and we didn’t need to throw a bunch of existing code in the garbage. We might feel a bit bad about the time and effort it took us and about the extra complexity we introduced though; but it’s better to have made the change than to have re-written the whole thing, right?

But what will happen next time? Are we going to leave the structures in even as they become more and more inappropriate for the kinds of changes we need to make? Are we going to live with them even as they make further changes more difficult and the complexity of the project continues to increase?

The tyranny of existing code suggests that yes, we will. Code does not have mass, but it has a gravitational pull. The larger and more complex the code, the more energy is required to escape it: more code to understand, more code to replace, more managers to convince, more programmers to get on board, etc. Eventually, the code that only needed to handle simple HTTP requests has become so complex because of the web framework and the extra layers of complexity that we added that simple changes require weeks of work and no one dares suggest that the problem is the web framework and that it needs to be replaced.

A skill programmers lack: practice

Sat, 10 Dec 2022 00:00:00 +0000

In the talk Solving the Right Problems for Engine Programmers, Mike Acton spends a section of his talk on what he considers the three fundamental areas in which most programmers are incompetent: practice, reasonable defaults, problem solving skills. The first one in particular is quite interesting, because it enables the two others: someone who practices the craft of programming regularly will develop reasonable defaults and problem solving skills. Acton says that practicing is an activity that we do individually and where we try to fill in gaps in our knowledge and that as professional, it’s our responsibility and obligation to practice.

But practicing is not a thing that most programmers do. Mostly, we get thrown into a problem, we read just enough documentation to allow us to complete our work before we’re shuttled to the next project with a looming deadline. It’s hard to imagine that we can acquire and build a solid foundation that way. And it’s not the way it happens in other professions: professional sport athletes are expected to practice multiple times a week to maintain their skill levels and improve their areas of weakness; musicians practice their pieces for many days/weeks before their public concerts; astronauts spend years preparing for their missions, they don’t just try and wing it.

Since I saw this video, I’ve been trying to make time every week to practice the skills of my profession. For example, over the last 2–3 weeks, I’ve blocked 30 minutes every other day to implement and re-implement sorting algorithms (bubble, insertion, selection, quick, merge, and radix) in multiple languages. The goals are to solidify my knowledge of the general strategy of those algorithms, understand the different invariants that need to hold (it’s very easy to get into an infinite loop because we accidentally use <= instead of <), and correctly implementat the algorithms in different languages. When I started, I had forgotten most of what I knew about sorting algorithms and I spent most of the 30 minutes fumbling about; but with regular practice, I’m now able to write all 6 algorithms and their tests in about 20 minutes. More importantly, I can explain how they work and what invariants need to hold. Next, I will try and include shell sort and heap sort to the list of sorting algorithms that I understand and can implement.

My suggestion for people who want to improve as programmers is to try to have a practice routine. Block time in your schedule a few times a week—not a lot, 30 minutes is plenty—and focus on an area that you want to improve. It can be data structures and algorithms, creating and running Docker containers, making SQL queries, etc.—whatever you feel that you need to improve. Then open a blank file and try to get as far as you can. After the 30 minutes are done, stop, wherever you are. Let your brain sleep on what you did for a day or two before going back. The next time you practice, do not open the previous day’s file: create another new one and start from scratch again; this is not a project, this is practice. You are trying to become good enough at a skill that it just flows out of your fingers; saying “I accomplished the thing” is not enough, you need to aim for “I can accomplish the thing quickly and consistently.” After a few sessions, you’ll see that you are better at finishing your task in the allotted time. Then start spacing it out (you can have a daily practice for another topic): practice the same thing a week later, then a month later, three months later, etc. This is to make sure that the knowledge that you struggled to acquire remains in your mind.

Tech companies making parental controls useless

Sun, 13 Nov 2022 00:00:00 +0000

How hard is it to get Minecraft for a game console? If you’re an adult you drive to your favorite video game store, buy Minecraft, pop the cartridge in the console, and play. If you’re a kid, your experience is similar, it has one extra step: you nag your parents for a while to get the game, they drive to the store, buy it, you pop the cartridge in the console, and play. Easy.

What about getting Minecraft for the PC (Java Edition), how hard is that? If you are an adult, the process is pretty straight-forward: login with your Microsoft account, click the “Buy now” button, pay for the game, download and install. However, if you are a parent trying to buy the PC version of Minecraft for a kid, the process is much more frustrating. It looks okay from the outside, but thanks to bugs in the implementation, you have basically no choice but to lie and pretend that your kid is 18 years old and opt-out of all the parental controls in order to work around a product that doesn’t work.

The problem stems from this: the PC version of Minecraft needs to be downloaded while connected with the Microsoft account that will play the game. There are many problems with that.

First, why should I even need a Microsoft account to download the game? Why not just allow me to pay for the game and download the installer anonymously? There’s a single player mode after all, why is internet connectivity even required?

Second problem: to create an account for Microsoft, you need to have your own email address. Kids typically don’t have email addresses, and their parents are reluctant to give them one. But if parent and kid want to play Minecraft together, they need to have two different emails: you can’t have multiple profiles under the same email. And let me tell you, email providers don’t make it easy to get an email for kids. When I tried to get a Gmail for my son, all the parental consent stuff was convoluted and buggy and in the end I threw in the towel and just lied about his age so that we’d avoid those difficulties.

Same deal with Microsoft: if you want to get a Microsoft Account for a child under 13, you need to link it to a parent’s account. That worked fine for me since I already had one, but that would be a lot of extra friction for parents who don’t. Then, the purchase process actually looked okay: my son clicks “Buy now” in his account and that tells him “we’re sending an email to your parents for their consent”. In my inbox, I had an email with a big, blue button saying “Allow”. That button took me to a page to make the purchase on behalf of your child: that looks sensible and promising. But alas, there’s a bug in the form: I had to enter my address (why? My address in PayPal isn’t good enough?) and the form would not allow me to change the country. For some reason, it was locked under U.S. for me. Both my profile and my son’s clearly say “Canada” in country of residence.

And so after an hour of extremely frustrating bullshit, I gave up. I went in the parental settings, said that my son is allowed to buy whatever he wants without my consent and linked my PayPal account to his profile.

The process for getting the PC version should be exactly the same as getting a console cartridge. Kid nags their parents, the parents comes to the PC, click the “Buy now” link (no logins required), download the installer, and start the game. Instead, Microsoft opted to have a process that is more complicated and we can see that it’s more complicated, because it doesn’t work.

I’ll close with this quote from gaben:

Piracy is almost always a service problem and not a pricing problem

You already know how to parse by hand

Mon, 31 Oct 2022 00:00:00 +0000

An article was published in /r/rust a couple weeks ago, Parsing With Nom, a blog post explaining how to use nom to create a parser for a data format in the BitTorrent protocol called Bencode. Nom is a Rust library that provides parser combinators, small functions that provide simple parsing capabilities and which can be combined together to form functions that can parse more complex input.

The article praises nom for being an easy way to write parsers for people who don’t have a CS degree and who haven’t dedicated the time and effort needed to read the 200 pages of parsing material in the Dragon Book. That characterization of parsing saddened me; parsers are super useful, they should be part of every programmer’s toolbox, and it’s a shame that their reputation as being hard discourages programmers from learning to write them. It’s especially sad, since writing a parser requires only basic programming concepts: functions, loops, ifs, arrays, structs, and recursion.

Parsing appears difficult because there are indeed 200 pages of dense theoretical concepts in the Dragon Book. Academics don’t make it appear any easier: to sound authoritative and learned, they use technical terms, like LR(1), LL(k), ε-closures, start sets, shift-reduce conflicts, etc. further reinforcing the impression that parsing is a complex matter better left to people who have the proper training.

Fortunately, most of what a working programmer who wants to convert text to an in-memory data structure is learned in Programming for Dummies. To support this claim, I will demonstrate writing a parser for the Bencode format—the same one as in the nom article—using only programming constructs that you are already familiar with. By the end of this article, you will find that there is nothing especially hard, technical, or scary about writing parsers and you’ll know how to create one yourself.

The data types

A parser needs two main pieces of information: the data to be parsed (bytes or tokens) and the position of the current piece of data to be analyzed. For Bencode, I start with a struct containing two fields: a vector of bytes and an index in that vector. If more data needed to be tracked, such as line numbers, column numbers, list of errors, etc., the Parser struct would be a good place to store them, but for this simple parser, we won’t bother with other information.

pub struct Parser {
    buf: Vec<u8>,
    pos: usize,
}

impl Parser {
    pub fn new(buf: Vec<u8>) -> Parser {
        Parser { buf, pos: 0 }
    }
}

We also need a data structure that will be returned by the parsing methods. In the literature, this is called an abstract syntax tree—an AST. In Rust, it’s typical to use an enum to represent an AST.

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum BValue {
    Int(i64),
    Str(String),
    List(Vec<BValue>),
    Dict(Vec<(String, BValue)>),
}

Notice how our AST has cases for each of the Bencode value types: integers, strings, lists, and dictionaries. That’s typical: we want the AST to closely mirror specification.

Finally, we need a type for reporting errors. Since this is a simple parser, we’ll use String as our error type. A more involved parser would likely use an enum to explicitly list the different kinds of parsing errors that can occur.

type Error = String;

Utility methods

Next, we add a few utility methods to our parser. There are typically two kinds of methods in a parser: parsing methods which know how to parse a small part of the language—we’ll start writing those in the next section—and utility methods which are used by the parsing methods. Here are the four utility methods we’ll use.

peek(): this returns the byte at buf[pos]; if pos is past the end of the buffer, we return the byte 0. For some kinds of parsers, especially binary parsers, 0 could be a valid byte, and so another approach should be used to represent EOF—Option<u8> would be one such choice.
advance(): this is like peek(), and it also moves pos forward by one.
expect(b): this method calls advance() and verifies that the byte b was consumed; if any other byte was seen, it returns an error.
eof(): this returns true if all the input buffer has been consumed and false otherwise. It’s not completely necessary since we can get the functionality with peek() == 0, but it makes the source code easier to read.

(A parser for a more complex language would have more utility methods, such as skip_spaces() or report_error() to avoid some tedium, but I’d expect to find our four methods in all parsers.)

impl Parser {
    fn peek(&self) -> u8 {
        if self.pos >= self.buf.len() {
            return 0;
        } else {
            return self.buf[self.pos];
        }
    }

    fn advance(&mut self) -> u8 {
        let b = self.peek();
        if b != 0 {
            self.pos += 1;
        }
        return b;
    }

    fn expect(&mut self, expected: u8) -> Result<(), Error> {
        let actual = self.advance();
        if actual != expected {
            return Err(format!(
                "syntax error: expected {:?}, found {:?}",
                expected as char, actual as char
            ));
        }
        return Ok(());
    }

    fn eof(&self) -> bool {
        return self.peek() == 0;
    }
}

And that’s it, we have all the building blocks needed to write our parser. The rest of the parser will be built on top of these utility methods and will use the programming constructs that you’ve been familiar with since you started programming.

The parsing functions

Predicting the kind of data to read

Our first parsing method guides the parsing process: it peeks the current byte in the input to decide how to proceed. In parser jargon, this is called a predictive parser. This is a simple and easy-to-understand strategy; it doesn’t always work (what if we can’t tell just from the next byte what to do?), but when it does we should use it.

impl Parser {
    fn parse_value(&mut self) -> Result<BValue, Error> {
        match self.peek() {
            b'i' => return self.parse_int(),
            b'l' => return self.parse_list(),
            b'd' => return self.parse_dict(),
            b'0'..=b'9' => return self.parse_str(),
            c => {
                return Err(format!("parse value: invalid initial byte {:?}", c as char));
            }
        }
    }
}

If the parser is currently looking at the letter 'i', we parse an integer and return its AST (or its error); if the parser is looking at a 'l' or 'd' we parse a list or a dictionary, respectively; if the next character is a digit, we parse a string; finally, if we see anything else, we report an error.

Integers

Now that we have parse_value to guide the parsing, we need to fill in the other parsing methods. The first one we’ll tackle is parse_int. In Bencode, integers look like this:

# General form
i<digits>e

# Examples
i0e    (0)
i-3e   (-3)
i999e  (999)

Our strategy for parsing will be:

Read the leading i
Read an optional -
Read the digits, accumulating them into a value
Read the trailing e
Return the accumulated value

impl Parser {
    fn parse_int(&mut self) -> Result<BValue, Error> {
        self.expect(b'i')?;

        let mult: i64 = if self.peek() == b'-' {
            self.advance();
            -1
        } else {
            1
        };

        let start_pos = self.pos;
        let mut value: i64 = 0;
        while !self.eof() && self.peek().is_ascii_digit() {
            let b = self.advance();
            value = value * 10 + (b - b'0') as i64;
        }

        if self.pos == start_pos {
            return Err(String::from("invalid integers: no digits"));
        }

        self.expect(b'e')?;
        return Ok(BValue::Int(mult * value));
    }
}

I think it’s pretty straight-forward; nothing that anyone would claim requires a CS degree or having read the Dragon Book.

We expect to see the byte 'i'; if we find anything else, we report an error;
We peek for a minus sign; if it’s there, we advance pos and set the multiplier to -1; otherwise we leave pos where it is and set the multiplier to 1;
We advance over the digits until we reach EOF or peek a byte that’s not an ascii digit; we accumulate the digits in a i64 value;
After we exit the loop, we check that we actually read digits; if not, we report an error;
We expect to see the byte 'e' ; if we find anything else, we report an error;
We multiply the accumulated value with the multiplier to give it the right signedness and return a BValue::Int.

Not very hard and it will correctly parse Bencode integers, just like the nom version.

Strings

Next up, string. They have the following format:

# General form
<integer>:<bytes>

# Examples
0:        ("")
3:foo     ("foo")
3:foobar  ("foo")

As with integers, we’ll only need our small utility methods glued with some loops and if statements.

impl Parser {
    fn parse_str(&mut self) -> Result<BValue, Error> {
        let mut len: usize = 0;
        while !self.eof() && self.peek().is_ascii_digit() {
            let b = self.advance();
            len = len * 10 + (b - b'0') as usize;
        }

        self.expect(b':')?;

        let mut buf = String::new();
        for _ in 0..len {
            let b = self.advance();
            if b == 0 {
                return Err(String::from("invalid string: reached EOF"));
            }
            buf.push(b as char);
        }
        return Ok(BValue::Str(buf));
    }
}

We start by reading the digits of the length of the string in exactly the same way that we did for integers. One difference though is that we don’t check to make sure that there actually were digits, because we know there is at least one: parse_value called us because it had peeked a leading digit.
We consume a colon or report an error.
We consume len bytes, pushing them into a String; if we encounter EOF before we’ve read all the characters, we report an error.

Two parsing functions down, two more to go.

Lists and dictionaries

Let’s tackle lists and dictionaries at the same time. Their definitions will be recursive: they will read values by calling parse_value. Recursion can be scary sometimes, but as we’ll see, it makes the parsing methods very short.

impl Parser {
    fn parse_list(&mut self) -> Result<BValue, Error> {
        self.expect(b'l')?;
        let mut values: Vec<BValue> = Vec::new();
        while !self.eof() && self.peek() != b'e' {
            values.push(self.parse_value()?);
        }
        self.expect(b'e')?;
        return Ok(BValue::List(values));
    }

    fn parse_dict(&mut self) -> Result<BValue, Error> {
        self.expect(b'd')?;
        let mut values: Vec<(String, BValue)> = Vec::new();
        while !self.eof() && self.peek() != b'e' {
            let key = self.parse_str()?;
            let value = self.parse_value()?;
            if let BValue::Str(key) = key {
                values.push((key, value));
            } else {
                return Err(String::from("invalid dict: keys must be strings"));
            }
        }
        self.expect(b'e')?;
        return Ok(BValue::Dict(values));
    }
}

Let’s focus on parse_list, since it’s the simpler of the two. We use expect to consume the opening 'l'. After we see the opening 'l' (technically, since parse_value already saw the 'l', it would be correct to use advance(); I just like that the code for parsing a list shows the full structure of what a list looks like), we loop until EOF or until we peek an 'e', and each time we call parse_value() to get the next element from the list and store it in a vector. This is the power of recursion: we could have lists within lists and this code would work correctly. (We’d only get into trouble if the input had very deeply nested lists and blew the stack.)

For dictionaries, it’s the same thing: we use parse_str() to read the key and parse_value to read the value.

Putting a bow on it

At this point, we got a fully functional parser. We can now add one method to wrap it nicely. The parse() method will be our entry point; it will call parse_value() to get a Bencode value and then ensure that there’s no extra garbage in the input.

impl Parser {
    pub fn parse(&mut self) -> Result<BValue, Error> {
        let value = self.parse_value()?;
        if self.eof() {
            return Ok(value);
        } else {
            return Err(String::from("garbage at the end of the input string"));
        }
    }
}

And there we go, a complete parser in about 150 lines of simple, beginner-level code! No need for an extra dependency, no need to learn yet another API, no need to remember the intricacies of closures in Rust. Just plain old, boring code.

Pros and cons

Now, let’s talk about the pros and cons of writing your own parser by hand. This is computer science after all, which means that there’s no One Solution To Rule Them All, only trade-offs to be weighed. Let’s start with the cons.

It’s easier to mess up. For example, we can make a mistake with pos and be pointing not exactly where we should be. Without good test coverage, a bug could lurk in a seldom-taken path and only appear in production.
No shared “parsing language” between projects. Everyone on the project needs to learn how this particular parser works; they don’t benefit (as much) from past experience.
It takes more code. If we want to whip out a small parser in a couple minutes, the hand-written approach will require more code, more typing, and will likely take more time than using a parsing library.

And the pros of the hand-written approach.

Simpler code. As we saw, we only needed simple programming language features to build our Bencode parser. It’s not rare for parsing libraries to lean into more complex features, like closures, macros, traits, etc. This makes it easier for people to jump into the code (i.e., they don’t have to learn an external library) and it plays nicer with a debugger.
Better error messages. One reason most production-level compilers have hand-written parsers is to give better error messages. When we can easily look at the state of the parser and have specific things for our language, it’s easier to give the user a clear message; in parsers generated by libraries, the error messages are often more generic and give less context.
Performance. We didn’t spend one iota of energy in making our Bencode parser fast, but in my own tests, it runs about 3x faster than the nom-based parser.
Fewer dependencies. Modern development relies very heavily on third-party libraries, with all the problems that can entail (e.g., left-pad). In Rust, extra dependencies can make the already-slow compile times even slower. Avoiding a third-party library avoids those extra liabilities.

Whether a hand-written parser or a library is preferable will depend on the project, but also on what you value.

Conclusion

You can see the whole parser in the Rust playground.

If you started reading this post thinking that parsing was difficult and needed specialized tools, I hope that this wall of text (and code) has helped you see that it’s actually much simpler and easier than it might appear. Of course, there are languages that are harder to parse than others and which will require more effort. Bencode was relatively easy, but other languages might require more involved techniques, e.g., if we can’t predict the next parsing method to call from just looking at the current byte, we may need to use a backtracking algorithm, which is harder, but in general, you’ll probably find that predictive, recursive-descent parser are more often applicable than they are insufficient.

Just: the most reductive word in software development

Sun, 01 May 2022 22:14:06 -0500

(Intentionally provocative title.)

Do you use the word “just”? Have you ever said something like “oh, we just need to implement that interface for this new type” or “it’s just a small bash script” or something to that effect? I have, many times, and still catch myself saying “just” even though I’m trying to exorcise that word out of my vocabulary.

I don’t like the word “just”. “Just” is extremely reductive. “Just” diminishes the complexity of the problems we work on and it diminishes the effort that people put into making solutions.

Here’s an example scenario. Suppose we have two services, Alpha, which writes data in S3, and Beta which reads data from disk. Alice has been tasked with making Beta process the data generated by Alpha. Since the output format of Alpha isn’t the input format of Beta, Alice will have to do something to make them work together. Bob, always helpful (/s), says to “just write a bash script to download the files to Beta’s server.” Right Bob, just a bash script, how hard can that be?

Actually, there can be a lot of difficulties that lay ahead for Alice. Here are some from the top of my head:

Bash is notoriously hard to get right: spaces can change the meaning of statements, incorrect quoting can cause bugs, etc.
How will credentials to the S3 bucket(s) be made available to the bash script?
How will the bash script be invoked?
Is it sufficient to download the data or must extra transformations be performed?
Are there race conditions between the download and Beta’s processing that need to be considered?

Though the script’s job is conceptually simple—bring data from S3 to disk—Alice actually has quite a bit of work on her plate. It’s not just a bash script, Bob, it’s a whole project! Would you like it if someone made an off-hand comment about the project that you work on as just fill in the blank?

Another problem with Bob saying to just write a bash script is that PMs might take his word that it’s just a bash script and under-estimate the time and effort required for the feature, possibly derailing the schedule of other teams.

Let’s be careful about using the word “just”: computer science and programming are interesting and complex endeavours, what we may think is just may actually have a lot more nuance and complexity than we imagine at first.

Thank you to Mathieu Lapointe for reviewing a draft of this article and suggesting improvements.

On 'Clean Code'

Fri, 12 Nov 2021 00:00:00 +0000

Many programming books give advice on how we can make our code clean. It seems to be quite a lucrative industry, because these keeping coming out year after year and programmers keep buying them.

Unfortunately, these books mostly concern themselves with what the text of the code ought to look like, and not with what the code itself ought to be. They give advice such as “use meaningful names”, “keep methods short”, “prefer dynamic dispatch to switch”, “depend upon abstractions, not concretions”, etc., but programs that abide by that advice are not magically readable or clean. (In fact, it seems that the more of this advice that we apply to a program, the more we create a “clean mess”.) But if we got the core of the program wrong—solving the wrong problem, using the wrong tool, having piles of unnecessary abstractions—no amount of advice about using private fields instead of public ones is going to meaningfully improve our program.

These books should be understood as our industry’s Strunk and White or The Chicago Manual of Style: the advice they give on what text ought to look like can be useful, but if an article is uninteresting, readers won’t care our about our usage of oxford commas.

So let’s make sure when we review our peers’ code that we don’t miss the forest for the trees and give them endless suggestions about how they could make their code “cleaner” by changing superficial details and instead focus on whether their general approach to the problem is sound or whether new requirements justify a different overall architecture.

Rust 2022

Wed, 20 Oct 2021 00:00:00 +0000

In the last few years, at the end of the year, the Rust Development Team asks people to write blog entries about their wishes for the next year. The new year is still a bit off and no call for such posts has yet been made in 2021, but there are a few things that I have in mind and that I want to write about right now. So without further ados, here is my wish list for Rust in 2022

Compile times

I wrote about compile times at the end of 2018 and in 2021 I want to point at that blog post and say: “that”. It’s depressing how little things have improved — from the perspective of a user — when it comes to compile times. I work with a wonderful team on a product in Rust, but though our repository is not even 40,000 lines of code, it takes such a long time to build. It’s roughly 4 minutes for a debug build from scratch and 10 minutes for a release build from scratch; incremental builds in debug take about 90 seconds and it’s about 3 and a half minutes in release mode. This is an absolute killer of productivity. Every time that I make a small change — even just adding a dbg!() statement in a function — I have to sit and wait for Rust to finish re-building and re-linking the project.

Some may argue that a lot of progress has been made since 2018 and will point to charts that show the compile times of various Rust projects going down month after month, year after year. And although they are technically right, it doesn’t matter. I’m still sitting in front of my computer, waiting minutes for a build, getting bored, going on lobste.rs, and losing my flow and concentration. The compile times of Rust are improving, but they’re not even within the ballpark of being acceptable.

Another thing that some people say is to control dependencies. I agree whole-heartedly with this point: bringing in large dependencies in order to use only a tiny fraction of their functionality is not usually a good trade. (I recently stopped using a library to do ISO-8859-1 to UTF-8 conversion when I discovered that it took only two lines of code to do it by hand. Five dependencies gone, just like that!) Unfortunately, some of the most common and useful crates in the Rust ecosystem are big and slow to build. For instance, we use tokio, serde, and hyper in some of our projects; they add minutes to the build times, but what’s the alternative? We have our own expertise and async runtimes and web servers are not it.

I hope that Rust build times improve in 2022. We need them to improve drastically. Not a by a few percent, but by an order of magnitude.

Debugging

Since compiling takes so long, doing exploration of a code base by sprinkling in dbg!() statements is not a great experience. I tried using rust-gdb to follow code paths and inspect data, and although it works well enough for simple programs, it’s not without its issues.

The first problem is probably not super complex to tackle (famous last words): Strings within structs are not pretty-printed. Compare below the output of printing a variable of type String and a variable which is a struct containing a String:

(gdb) p a
$1 = "Hello, world!"

(gdb) p b
$2 = dbg_demo::MyStruct {inner_string: alloc::string::String {vec: alloc::vec::Vec<u8, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<u8, alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: 0x55555559baa0, _marker: core::marker::PhantomData<u8>}, cap: 13, alloc: alloc::alloc::Global}, len: 13}}}

The same happens with other common types: vectors, sets, maps, etc. It would be very nice if printing and navigating a struct in the debugger did not expose the internals of those objects (unless explicitly required I suppose).

The other thing that should be worked on is async support. With rust-gdb, if you set a breakpoint on an async function, it’ll immediately return. In a sense, it’s understandable: the function doesn’t do any work per se; it creates and returns a Future that will do the work, but from a user-experience perspective, not being able to debug async functions the same way we debug regular functions is very problematic.

Normally, I wouldn’t care too much about the state of debug support (I’ve never been quite good at using debuggers), but given that compile times are such as they are, we should be able to extract as much value from one compile as possible. Improved debug support would go a long way toward that.

More restraint in the implementation of features

This one is more of a “get off my lawn you darn kids” rant from a programmer who’s gaining more grey hair each month, so take with a grain of salt. Programming languages cannot be everything for everyone and the job of language designers is to be judicious in the choice of features they include and even more judicious in the choice of features they omit. This could be a reddit-tinted view of the situation, but it appears to me that a number of features are coming down the pipe that will make the language more complex, harder to learn, harder to master (and worst of all, harder to convince other people not to use those features) that address very niche cases.

If I were 5–10 years younger, I’d probably be very excited at those new advanced language features; as an increasingly grumpy and crusty coder, those new features scare me, because I’m afraid (1) that I won’t be able to properly learn them (I still haven’t fully grokked Pin, GATs are probably going to be something I want to avoid, and I’m really hoping the push to make Rust Haskell with type families and GADTs will not come to be); (2) that other programmers will start using those features just because they can and not because they are necessary to the solution to the problems they are solving.

Making Reasonable Use of Computer Resources: Part 2

Sat, 28 Aug 2021 00:00:00 +0000

In my last post I made the case that programmers should try and write their software in a way that makes reasonable use of the resources of the computer. By reasonable use, I didn’t mean to try and perform every crazy micro-optimization possible to extract every last speck of performance from the machine: instead, I meant to have mechanical sympathy, to write software in a way that works with the hardware rather than against it, and to take the specific problem into consideration when creating a solution.

That article had an example of how being mindful of the CPU cache can make one algorithm nearly 16 times faster than another, even though they have the same algorithmic complexity. I’ve been wanting to find a more real-world example, and this week, one feel in my lap.

A year ago, Mark Litwintschik wrote an article where he extracted the domain names from a 12 GiB gzip file containing 1.27 billion lines of JSON and created an ip,domain CSV file. That program was written in Python and took more than a day to run!

Last week, Mark wrote a follow-up article detailing a new program to do the same job, one written in Rust. Rust is a statically-typed language, it’s compiled to native code, and the compiler uses LLVM to apply many optimizations, so we’d expect this version to be faster. And it is: the Rust version processed the data in 48 minutes. After he shared his article on lobste.rs, Mark received suggestions to make the program faster and has since brought the processing down to 30 minutes. His new program is 50 times faster than his original — a much better use of the machine’s resources! And yet, when I initially read the article, I thought that 48 minutes sounded slow.

Thankfully, Mark’s article gave links to the dataset and to the third-party library he was using, and he shared his Rust code and the bash commands he used to execute his program. It made it very easy to replicate his experiment.

Mark’s program works in four phases:

Read and decompress
Parse JSON
Call out to tldextract-rs to extract the domain name
Write a line of output

The first thing I did was very simple: I timed how long it took to zcat the input. This would help me gauge how long steps 1 and 4 took together. I wanted to know if I was right to think that 48 minutes was slow or if my expectations were wrong.

$ time zcat rdns_xaa.json.gz >/dev/null

real    1m34.420s
user    1m34.047s
sys     0m0.364s

So roughly a minute and a half to decompress, read, and write. I didn’t think that parsing JSON would be much slower than decompressing a gzip file, so most of the time has to be spent in phase 3, extracting the domain name. Since the operation sounds quite simple and straight-forward (find the longest top-level domain suffix, take the substring to the left until a period or the beginning of the string is encountered), I suspected that something was probably being wasteful there.

Next, I ran Mark’s original program to know how long it took to run on my machine.

$ perf stat -dd ./target/release/tldextract rdns_xaa.json.gz >/dev/null

 Performance counter stats for './target/release/tldextract rdns_xaa.json.gz':

      1,644,491.19 msec task-clock                #    1.000 CPUs utilized
             8,526      context-switches          #    0.005 K/sec
                72      cpu-migrations            #    0.000 K/sec
             1,103      page-faults               #    0.001 K/sec
 6,678,553,292,544      cycles                    #    4.061 GHz                      (38.46%)
18,119,341,562,549      instructions              #    2.71  insn per cycle           (46.15%)
 3,952,769,540,665      branches                  # 2403.643 M/sec                    (46.15%)
     9,984,392,711      branch-misses             #    0.25% of all branches          (46.15%)
 3,738,006,667,501      L1-dcache-loads           # 2273.048 M/sec                    (46.15%)
    39,713,107,241      L1-dcache-load-misses     #    1.06% of all L1-dcache accesses  (46.15%)
     1,216,763,228      LLC-loads                 #    0.740 M/sec                    (30.77%)
        26,340,299      LLC-load-misses           #    2.16% of all LL-cache accesses  (30.77%)
   <not supported>      L1-icache-loads
   211,529,207,164      L1-icache-load-misses                                         (30.77%)
 3,738,968,599,010      dTLB-loads                # 2273.632 M/sec                    (30.77%)
        18,483,670      dTLB-load-misses          #    0.00% of all dTLB cache accesses  (30.77%)
        38,106,464      iTLB-loads                #    0.023 M/sec                    (30.77%)
       382,508,684      iTLB-load-misses          # 1003.79% of all iTLB cache accesses  (30.77%)

    1644.802173800 seconds time elapsed

    1618.366720000 seconds user
      26.137782000 seconds sys

I like to use `perf stat -dd` instead of `time`; it gives a more complete picture

The program finished in 27 minutes on my machine. While the program was running, I also ran perf top to look at which functions were taking the most time.

When I skimmed the original program and the third-party library, I thought that they allocated a lot. For example, the std::io::Lines iterator allocates a new string for every line of input and when the line goes out of scope, it’s freed. Given the size of the input, that’s 1.27 billion allocations and deallocations. perf top shows malloc/realloc/free taking nearly 10% of the total execution time, so they’re a problem.

But the real big issue is in the third-party library to manipulate domains. Since domains are allowed to have Unicode in them, a lot of time is spent in IDNA code. But is that really necessary? Does the data that we manipulate require that we impose the extra cost of Unicode domain names upon us?

What I found by inspecting the data is that all the JSON objects contain only ASCII characters and only 20 objects out of 1.27 billion contain the substrings \u or \U (JSON unicode escape sequences). That means that the original program is using a library that makes processing every line slower for the benefit of only 0.00000157% of the lines. Instead of solving the generic case, let’s solve the common case.

$ parallel "zcat {} | grep -P '[^\000-\177]' | wc -l" ::: rdns_xa*.gz
0
0
0
0

$ parallel "zcat {} | grep -iF '\u' | wc -l" ::: rdns_xa*.gz
14
1
5
0

Bash commands to search for Unicode in the data

So given what we know, here’s the plan for a (hopefully) faster program. Like the original program we’ll use flate2’s GzDecoder to decompress the data. We are not going to use the lines iterator, instead we’re going to use the method BufReader::read_line which allows us to re-use the same buffer, so we only need to allocate once for all the lines. When we have a line, we’re going to do a quick linear search for the substring \u; if it’s found, we write that line to a file called rejected.json that we can process separately and move on to the next line. We’re going to use serde_json instead of json, because it’s a bit faster. To find the top-level domain, we’ll create a HashSet of all the domain suffixes (obtained from here) and we’ll find the longest suffix we can; once we have that suffix, we can extract the string to its left until we find a period or the beginning of the string. Finally, we write out to stdout, but not with println!, but with a pre-locked Stdout object.

$ perf stat -dd ./target/release/vfb-tldextract public_suffix_list.dat rdns_xaa.json.gz rejected_xaa.json >/dev/null
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627471863\",\"name\":\"152.149.215.39\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627447667\",\"name\":\"152.149.215.40\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627444989\",\"name\":\"152.149.215.43\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627486727\",\"name\":\"152.149.215.45\",\"type\":\"ptr\"}\n"
vfb-tldextract: processed 310673926 lines (14 rejected) in 155.272128427s

 Performance counter stats for './target/release/vfb-tldextract ../tldextract/public_suffix_list.dat ../tldextract/rdns_xaa.json.gz rejected_xaa.json':

        155,270.78 msec task-clock                #    1.000 CPUs utilized
               451      context-switches          #    0.003 K/sec
                 1      cpu-migrations            #    0.000 K/sec
               345      page-faults               #    0.002 K/sec
   631,384,933,925      cycles                    #    4.066 GHz                      (38.46%)
 1,803,019,603,887      instructions              #    2.86  insn per cycle           (46.15%)
   365,196,577,639      branches                  # 2351.998 M/sec                    (46.15%)
     2,314,559,123      branch-misses             #    0.63% of all branches          (46.15%)
   379,148,826,483      L1-dcache-loads           # 2441.856 M/sec                    (46.15%)
     2,166,436,251      L1-dcache-load-misses     #    0.57% of all L1-dcache accesses  (46.15%)
        75,583,470      LLC-loads                 #    0.487 M/sec                    (30.77%)
        25,064,718      LLC-load-misses           #   33.16% of all LL-cache accesses  (30.77%)
   <not supported>      L1-icache-loads
       300,858,674      L1-icache-load-misses                                         (30.77%)
   379,351,678,372      dTLB-loads                # 2443.162 M/sec                    (30.77%)
           105,344      dTLB-load-misses          #    0.00% of all dTLB cache accesses  (30.77%)
           506,821      iTLB-loads                #    0.003 M/sec                    (30.77%)
         2,868,354      iTLB-load-misses          #  565.95% of all iTLB cache accesses  (30.77%)

     155.275422832 seconds time elapsed

     154.847572000 seconds user
       0.423987000 seconds sys

2 minutes and 35 seconds; 10 times faster than the original program and less that twice the time to just `zcat`.

Now the final test, how long does it take to transform the four shards in parallel?

$ time /usr/bin/ls rdns*.json.gz | xargs -P4 -n1 -I{} sh -c './vfb-tldextract public_suffix_list.dat {} {}.rejected.json >{}.csv'
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627442231\",\"name\":\"41.185.37.10\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627473254\",\"name\":\"41.185.37.11\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627458666\",\"name\":\"41.185.37.12\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627479199\",\"name\":\"41.185.37.13\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627439294\",\"name\":\"41.185.37.14\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627471863\",\"name\":\"152.149.215.39\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627447667\",\"name\":\"152.149.215.40\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627444989\",\"name\":\"152.149.215.43\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627486727\",\"name\":\"152.149.215.45\",\"type\":\"ptr\"}\n"
vfb-tldextract: cannot deserialize this line: "{\"timestamp\":\"1627490888\",\"name\":\"210.35.32.3\",\"type\":\"ptr\"}\n"
vfb-tldextract: processed 310673940 lines (0 rejected) in 184.360843703s
vfb-tldextract: processed 310673926 lines (14 rejected) in 184.376276827s
vfb-tldextract: processed 310673939 lines (1 rejected) in 184.705608281s
vfb-tldextract: processed 310673935 lines (5 rejected) in 188.207680141s

real    3m8.222s
user    11m53.263s
sys     0m16.673s

It’s slower than just one shard at a time (I haven’t yet investigated why that is; my guess is that we’re maxing out the write speed of the disk) but still nearly 10x faster than the original program. And I think we did things that most programmers would be able to do:

Get a base line for how fast we can zcat a file
Use perf top to find some bottlenecks that can be addressed
Look at the data to understand what the common case is

And I think that the final program is something that most people reading this would probably have been able to do themselves.

If you’d like to look at my code, it’s available on Github.

Making Reasonable Use of Computer Resources

Sat, 31 Jul 2021 00:00:00 +0000

The computers sitting on our desks are incomprehensibly fast. They can perform more operations in one second than a human could in one hundred years. We live in an era of CPUs that can perform billions of instructions per second, tens of billions if we take multi-cores into account, of memory that can transfer data to the CPU at hundreds of gigabytes per second, of disks that support streaming reads of gigabytes per second. This era of incredibly fast hardware is also the era of programs that take tens of seconds to start from an SSD or NVMe disk; of bloated web applications that take many seconds to show a simple list, even on a broadband connection; of programs that process data at a thousandth of the speed we should expect. Software is laggy and sluggish — and the situation shows little signs of improvement. Why is that?

I believe that the main reason is that most of us who started programming after, say the year 2000, have never learned how to make reasonable use of the computational resources at our disposal. In fact, most of our training has taught us to ignore the computer!

Although our job is ostensibly to create programs that let users do stuff with their computers, we place a greater emphasis on the development process and dev-oriented concerns than on the final user product. SICP contains a quote that I find to be a good summarization of the problem: “programs must be written for people to read, and only incidentally for machines to execute.” Many programmers find that quote wise and inspiring, but users are not interested in reading programs, they’re interested in executing them, fast. We can’t make programs that run fast if we write them in a way that is only “incidentally” executable. The computer is not an implementation detail that can be abstracted away and ignored — it’s an integral part of the solution. A program that makes no room for the target machine in its design will inevitably run slower than one that does.

A common argument against taking the computer into account during the design phase of a program is “premature optimzation is the root of all evil.” Topics such as cache-friendliness, branch prediction, and parallelism are labeled as “optimization”, but really they should be labeled as “reasonable use”. Our users’ computers have resources and we should design our programs in a way that they use those resources.

Take the CPU cache for example. A simple program that shows what a massive impact good cache usage can have on performance is to sum the elements of a matrix row-by-row vs. column-by-column (here’s a Rust Playground link that demonstrates this). In this example, the row-by-row is about 12–16 times faster than the column-by-column, even though they have same Big-O complexity. The difference in performance is not due to an abstract, computer science factor, but due to a very concrete one: better use of the CPU cache. The row-by-row approach brings 16 floats in cache and uses them all before going back to memory; by contrast the column-by-column approach also brings 16 floats, but only uses one of them before going back to RAM, leaving 15 values unused — that’s 94% waste.

There are a few things we should note about this example. First, the compiler did not optimize the code for us. We often hear that programmers can’t beat an optimizing compiler, but in this case we did. We only needed to change the order of the loops, but the Rust compiler which uses the LLVM backend, did not do it. Maybe the famed sufficiently smart compiler would, but for the foreseeable future, arranging data in a way that it can be processed efficiently will not be the compiler’s job — it’ll be ours. Second, we can’t get that speed-up without taking the actual hardware into account in our solution. We do not write code for abstract, idealized, or fictitious machines: we write code that runs on the CPUs of our users — Intel, AMD, Apple, etc. These CPUs have caches and it’s perfectly reasonable to write code that works well with the hardware rather than against it. Third, a 16x speed-up may only be a constant factor in complexity theory, but to a user it can be the difference between a program being pleasant to use and being extremely frustrating. Or the difference between a small and large AWS bill. And finally, we should not consider this to be an optimization: we’re making use of a resource that’s there. The users paid the full price of their CPU, let’s try and give them their money’s worth.

I expect that many will think of the old aphorism “make it work, make it right, make it fast” and argue that taking the machine into account when first writing the program is focusing on “make it fast” before we even got to “make it work”. I get that, but if we do things in order and make our program work and then make it right, when it’s time to make it fast, we often realize that our design works against us and we have to undo/redo a lot of the work we did to make the program work and to make it right.

For example, we may have arranged our data in a row-oriented fashion aka array of structs and we realize that in order to speed up our program we would need column-oriented storage, aka struct of arrays. Changing a program from one form to the other is a major re-architecture project, and one that may not be feasible if there’s a deadline approaching and that’s more slow software on the market and the aphorism becomes “make it work, make it right, dream of making it fast”.

So how do we write code that runs reasonably quickly on modern machines? We start by getting into the habit of not just thinking about the model of what we write, but also the mechanics: how many bytes are needed to represent a data structure? Are there many pointers that will cause cache misses? Is the data organized in a way that the branch prediction will be right often? Would it be easy to slice the work among multiple threads?

Another way is to start thinking in terms of batches and systems. Suppose we are writing a tokenizer for a programming language; if we think about tokens as completely independent pieces of data, each one needs to carry the string associated with the token (e.g., struct Token { tag: Tag, text: String, ... }). If instead we think of tokenization as a system and of tokens as dependents of this system, we can create a string pool in the tokenizer and tokens can have an index into the pool for their associated text. This saves memory by not having to duplicate strings and makes the tokens smaller, which means more can fit in one cache line.

Data-oriented design is an approach to programming that concerns itself with such questions. At the moment, most data-oriented design practitioners work in the game industry, but the insights are valuable in every field. (For example, Andy Kelly spoke highly of data-oriented design in the Zig 0.8.0 release notes.) If you’ve never been exposed to data-oriented design, it can be a bit of a shock — a lot of the common wisdom of programming (e.g., Uncle Bob’s SOLID principles) is eschewed and replaced with cold, hard engineering. It looks scary and different, but it’s what we need to do if we are to deliver software that is not orders of magnitude slower than it can be.

My thanks to Mathieu Lapointe for reading early drafts of this article.

Lesser Known Coding Fonts

Thu, 17 Oct 2019 00:00:00 +0000

In the past few months, I’ve become interested—fascinated even—by programming fonts. As programmers we look at text all day: code, logs, command outputs, monitoring tools, etc. If we’re going to be looking at text all day, we might as well make it easy and pleasant for our eyes to read that text.

One great place to see the fonts used by other people is at /r/unixporn. I have not conducted a thorough survey, but from my observations it seems that, currently, the most popular font for the ricers of /r/unixporn is Hack. As we might expect, many use the built-in system fonts: DejaVu Sans Mono and Ubuntu Mono for Linux; Monaco, Menlo, and SF Mono on macOS. High-quality open source fonts are also very well represented: Fira Code, Inconsolata, Iosevka, Noto, Roboto Mono, and Source Code Pro.

In this post, I want to show five fonts that I don’t see as often on /r/unixporn, but which I believe deserve to be better known. (Edit: ironically enough, this post cannot be posted on /r/unixporn, because they don’t accept content that isn’t screenshots, photos, or videos.)

APL385 Unicode

The first font I want to suggest is APL385 Unicode. APL is a programming language where operations are denoted with operators. In order to make APL code more readable, APL385 has generous spacing between lines and between characters. The terminators (the end of strokes) are rounded which gives the font a fun, whimsical look.

(Click on the image to zoom)

As a programming font, the majority of characters are easy to distinguish, however the lack of slashed or dotted zero can cause confusion with the uppercase Oh. APL385 does not support Powerline glyphs, and there isn’t a patched version like there exists for other fonts.

One thing to note, at least on Linux: at smaller font sizes, the hinting is very bad and the font looks very jaggy. Not recommended if you like very small font sizes.

Homepage	http://apl385.com/fonts/
License	Public domain
Glyph count	769
Weights	Regular
Italics?	No
Ligatures?	No
Customizable?	No

Go Mono

Go Mono is one of the typeface in the Go type family. These fonts were created by the professional firm Bigelow & Holmes, and released under the BSD license. Go Mono is the only serif font in this list. (Serifs are the little “legs” at the end of some strokes.)

(Click on the image to zoom)

Like most programming fonts, all the characters are easy to distinguish, though I will point out the lack of spacing between consecutive underscores; this can be annoying to people who code in languages like Python where the double-underscore is used quite often. Go Mono is easy to read at very small sizes: even with my aging eyes, it’s not difficult for me to make out words at 8 points (although I recommend using a larger font size!) Like the majority of coding fonts, Go Mono comes in two weights, regular and bold, and it has an italic version.

As the name suggests, Go Mono is monospaced; the proportional typeface of the family, Go, is actually serviceable as a coding font. Typically, proportional fonts such as Times or Helvetica suck as coding fonts, but Go makes all characters distinguishable. If you want to try something different, you may want to try coding with a proportional font.

Homepage	https://blog.golang.org/go-fonts
License	BSD
Glyph count	661
Weights	Regular, Bold
Italics?	Yes
Ligatures?	No
Customizable?	No

Sudo

Sudo is the work of one programmer, Jens Kutílek, who decided to create his own coding font in 2009 and is still actively developing it.

(Click on the image to zoom)

Sudo is a narrow font, so if you like having two or three side-by-side splits in your text editor, Sudo is a good choice. All the characters in Sudo are easy to distinguish, and there is a gap between consecutive underscores. One characteristic of Sudo is the height of digits: they are a bit shorter than capital letters; this helps to further disambiguate, say, the digit one and the letter el.

Sudo has a variable font download option, i.e., a single font file that contains the regular, bold, and italic glyphs. This reduces network transfers if you want to use Sudo on your website.

One thing you’ll note about Sudo is that some characters (e.g., the superscript 3 or the lowercase theta) are very pixelated. This is due to the way the font designer works: he creates a pixel version first, then he draws the curves. The pixelated characters are the characters he hasn’t yet gotten around to drawing.

Homepage	https://www.kutilek.de/sudo-font/
License	OFL
Glyph count	1071
Weights	Regular, Bold
Italics?	Yes
Ligatures?	No
Customizable?	No

Monoid

Like Sudo, Monoid is another coding font that was developed by a programmer, Andreas Larsen. Monoid’s number one goal is extreme readability, especially at smaller font sizes and its look reminds of bit-mapped fonts.

(Click on the image to zoom. Note: this is my own customized version of Monoid. The characters 01l$ might look different for you.)

All the characters in Monoid are easy to distinguish. While researching Monoid, I learned that they made an effort to distinguish the Cyrillic letter Ze (З) and the number three (3) by giving three a flat-top. The characters 01l$ can be customized before download; you can also adjust the line spacing and the tracking—the spacing between characters. Monoid is the only font in this list to feature programming ligatures, and these can be turned off if you don’t like them.

The creator of Monoid gave a talk about the language; the audio quality is not great, but you can learn a lot from this presentation.

Homepage	https://larsenwork.com/monoid/
License	MIT + OFL
Glyph count	618
Weights	Regular, Bold
Italics?	Yes
Ligatures?	Yes (optional)
Customizable?	Yes

Input Mono

The last language in our list is Input Mono, a font for programmers by professional font designer Jonathan David Ross. Unlike the other fonts in this list, Input Mono is not open source: it is free for private use, but if you wish to use it on your website or in a print publication, you must purchase a license.

(Click on the image to zoom. Note: this is my own customized version of Input; the characters 0agil{*} might look different for you.)

All the characters in Input are easy to distinguish and it comes with Powerline support out of the box. At the download screen, you can customize Input: you can choose your shape of choice for the characters 0agil{*} and you can increase or decrease the line spacing.

Input Mono is part of a massive font family. In addition to the monospaced variant, there is a sans-serif proportional variant and a serif proportional variant. Though they aren’t monospaced, they are designed with programmers in mind. Then, each of the three variants comes in three widths: regular, narrow, and condensed. If you’d like for Input to be as slim as Sudo, it can. Finally, each font has seven weights and an associated italic. Put it all together, and Input has 126 different fonts for you to choose from.

Input Mono has been my daily driver for many months, and I only have good things to say about it. If you want to hear more about Input, you can watch Jonathan David Ross’s presentation, Cracking the Code.

Homepage	https://input.fontbureau.com/
License	Free for personal use
Glyph count	921
Weights	Thin, Extra Light, Light, Regular, Medium, Bold, Black
Italics?	Yes
Ligatures?	No
Customizable?	Yes

My quick review of Cascadia Code

Thu, 19 Sep 2019 00:00:00 +0000

On September 18th, 2019, Microsoft released a new typeface for coders, Cascadia Code. Cascadia was designed for usage inside Microsoft’s new Terminal application and inside coding editors like VSCode or Emacs. Although it is still just one day old, I thought I’d give my thoughts on Cascadia.

The good

Microsoft released Cascadia Code under the terms of the SIL Open Font License, a copyleft license for typefaces: the author gives up all profit from the sale of the font, but all derivatives must abide by the same terms. This is great as it ensures that the development of Cascadia Code can continue even if Microsoft no longer has the resources to dedicate to the project. The OFL also allows anyone to freely use the font on their website, in their screencasts, in their Power Point presentations, in their published articles, in their books, etc.

I think that Cascadia Code looks really good! When I first used Ubuntu Mono and Fantasque Sans Mono, I was really taken by how whimsical they looked compared to other coding typefaces. The common coding typefaces – DejaVu Sans Mono, Menlo, Consolas, Courier – are all very straight and, dare I say, soulless. Cascadia Code has the “fun” quality that Ubuntu Mono and Fantasque Sans Mono have. Maybe we’ll look back on Cascadia in a few years and say, “oh, that typeface is so 2020!”, but for now I think it brings a needed freshness to coding fonts. Some fonts are easily identifiable by some of their characters; I believe that the f, the k, and the 4 are going to be how most people will recognize Cascadia Code.

Cascadia is easy to read at small sizes; even at 10 points, I have an easy time reading. (Although, for the sake of your eyes, please don’t use tiny font sizes!)

If ligatures are your kind of thing, has a number of them. I don’t use ligatures myself, so I can’t comment on their quality versus other contenders like Fira Code or Iosevka.

In Cascadia Code, the characters that can look similar in regular typefaces are nicely distinct. For example, here’s what the characters that look like the letter “ell” look like:

iIl1|

This is just my opinion, but I don’t think there is any ambiguity here. Same with the “oh’s”:

oO0

The dot inside the zero makes it clear that it’s a numeral and not a letter.

The bad

Cascadia Code’s “bad” is mostly a product of its age. First, it does not support characters with accents, which is necessary for people who write in a foreign language. It also misses many glyphs such as dashes, mathematical symbols, and some symbols used in roguelikes. I’m sure that as Cascadia Code ages, more glyphs will be added.

Another bad aspect of Cascadia that’s certainly a product of its young age: it has no bold face nor italics. It also doesn’t have multiple weights like Source Code Pro or Input. Again, I’m sure we just have to be a bit patient.

It appears to me that in order to contribute to the development of the font I need to use Windows- and MacOS-only tools. Not an atypical situation, but it chagrins me when an open font project uses tools that lock out people who would like to contribute. (I think this shows that there is a need for a better open sourced font development ecosystem.)

Conclusions

I used Cascadia Code today at work and for writing this post. I think it’s a nice font, it’s easy to read, and I don’t mind some of the quirky characters (e.g., f and k) as much as I thought I would.

I definitely recommend that you give it a try, and I wish the team developing Cascadia Code all the best. I can’t wait for the release that has support for accented characters!

Thank you to Stefan Knudsen for reporting a typo.

Objective font criterias

Wed, 24 Jul 2019 00:01:20 -0400

When you choose a font, the most important criteria is “do you like it, and could you stand to look at it for 8 hours per day?” There’s no accounting for taste, so if the short leg of the the letter m in Ubuntu Mono really bugs you, then you should pick a different font and don’t let anyone tell you that you’re wrong. But fonts have characteristics that aren’t subjective, and when you’re choosing a new font, it makes sense to consider these as well.

License

Some fonts require that you purchase a license to use them and they might impose limits on how many people can use them. PragmataPro and Triplicate are such fonts. Others can be used free of charge for private use—Input for example—but a license must be purchased if you want to use them on your website or in a print publication. In the open source community, many fonts use the SIL Open Font License. You can use, modify, and redistribute those fonts freely; derivative work must be licensed under the same terms—the OFL is a copyleft license—and the fonts may not be sold. Fonts licensed under the terms of the OFL include Fantasque Sans Mono, Iosevka, and Inconsolata. You can use these on your blog or in your papers or package them with your applications without worry.

Unicode coverage

Some fonts like DejaVu have extensive Unicode support, and are appropriate for people who write in foreign languages, who write mathematics, or who just want to use emojis. When fonts don’t have good Unicode coverage, some of the unsupported glyphs will be replaced with a blank square, or sometimes a fallback font can be used and that can break the homogeneity of the text.

If you play roguelikes, some (e.g., Dungeon Crawl: Stone Soup) use Unicode characters to display dungeon features. I’ve seen glyphs overlap when using some fonts, typically when a symbol is left of a common character. If this happens to you, try using a different font when playing.

Weights and slants

Anyone who’s used a word processor knows how to make fonts bold and/or italic. But did you know that not all fonts have bold or italic variants? Did you know that some fonts have multiple weights, not just bold?

Monaco and Droid Sans Mono are example of fonts that just have the regular variant—they don’t have italics or bold. Fira Code comes in three weights—light, regular, and bold—but it has no italics. Many programmer fonts offer regular, bold, italic, and bold-italic faces: Fantasque Sans Mono, Meslo, Consolas, Inconsolata, and many more. Finally, some fonts offer many different weights going from very thin to very thick. Input and Iosevka are such fonts.

Customization

Most fonts are given to you as-is—what you see is what you get. However, some fonts let users customize them. Iosevka offers 12 stylistic sets out of the box, and if none are entirely satisfactory, you can modify a TOML file and specify exactly the glyphs that you want. Before downloading Input you can customize the glyph for some characters (g, i, l, 0, *, and the braces). Hack, a font based on DejaVu Sans Mono, has a repository of alternative glyphs; if you don’t like some of their glyphs, for example their zero, you can change them to something more classic.

Active development

Some fonts are “done”; they are no longer being developed. This can be frustrating if you find problems with the font and no one can fix it. At the time of writing this post, many font projects were in active development: Iosevka, Fira Code, IBM Plex, and many more.

Simple usage of GNU parallel

Sat, 13 Jul 2019 09:48:50 -0400

GNU Parallel¹ is a great tool if you want to reduce the running time of a command-line job. Last week, I had a job that took 30+ minutes to run sequentially, but finished in less than two minutes when I ran it with Parallel on a 40-core machine.

Unfortunately, GNU Parallel does not have a very intuitive interface and that’s a significant hurdle for people who want to use it. I’ve been frustrated in the past, and I’m sure many of you reading have too. In this post, I’ll show you how to use GNU Parallel to do two basic, but very useful tasks:

Run one command on multiple inputs in parallel;
Run multiple commands in parallel.

Setup

To help us in our examples, I have created a directory that contains the files we’ll work with. Their names and their contents is listed below. You can download this directory as a zip file.

a.txt:
Avocado

b.txt:
Broccoli

c.txt:
Cauliflower

cmdlist:
du -h a.txt
sha1sum b.txt
tr a-z A-Z <c.txt

filelist:
a.txt
b.txt
c.txt

Running one command on multiple inputs

There are two ways to pass inputs to a command in GNU Parallel: via the command-line arguments or via a file (or stdin).

You tell GNU parallel to take its inputs from the command line with the ::: operator.

$ parallel 'sha1sum {}' ::: *.txt
d8bcd7a949a2fb7a3e1740e4c61e52f46b781aea  a.txt
3db0058f7068968d90679ceb2184441c729782b9  b.txt
a9db4a2565b96696b594965ef52a98ebadfcc2b8  c.txt

$ parallel 'sha1sum {}' ::: a.txt cmdlist
d8bcd7a949a2fb7a3e1740e4c61e52f46b781aea  a.txt
814ca442838f9aa6ec2c01f5ed90de4620cfa990  cmdlist

In this example, the command we want to run is sha1sum and the {} is a placeholder that is be replaced by an input when invoking the command.

(Note: Readers already familiar with GNU Parallel will see that I am being overly explicit in these examples. In the example above, I didn’t need to quote the command; in this particular case, I didn’t even need to use a placeholder. But to be as clear as possible, I will avoid the shortcuts in this post.)

You tell GNU Parallel to take its inputs from a file with the :::: operator. If the filename is -, GNU Parallel reads from stdin.

$ parallel 'sha1sum {}' :::: filelist
d8bcd7a949a2fb7a3e1740e4c61e52f46b781aea  a.txt
3db0058f7068968d90679ceb2184441c729782b9  b.txt
a9db4a2565b96696b594965ef52a98ebadfcc2b8  c.txt

$ grep [ac] filelist | parallel 'tr a-z A-Z <{}' :::: -
AVOCADO
CAULIFLOWER

I think you’ll find this to be the most useful form of GNU Parallel. You can build a Unix pipeline using familiar tools—sed, awk, grep—to construct a list of inputs and then pipe them to parallel.

Running multiple commands in parallel

To run multiple commands in parallel, use the placeholder as the command name. Like in the previous section, you obtain the command name from the command-line arguments with the ::: operator and from a file or stdin with the :::: operator.

$ parallel '{}' ::: 'du -h a.txt' 'rot13 <b.txt'
4.0K        a.txt
Oebppbyv

$ parallel '{} a.txt' ::: sha1sum 'du -h' 'xxd'
d8bcd7a949a2fb7a3e1740e4c61e52f46b781aea  a.txt
4.0K        a.txt
00000000: 4176 6f63 6164 6f0a                      Avocado.

$ parallel '{}' :::: cmdlist
4.0K	a.txt
3db0058f7068968d90679ceb2184441c729782b9  b.txt
CAULIFLOWER

$ grep [ac].txt$ cmdlist | parallel '{}' :::: -
4.0K	a.txt
CAULIFLOWER

Conclusion

This was a super simple introduction to GNU Parallel; there are a lot of features that we didn’t cover:

composing and permuting the inputs from multiple files
controlling the number of concurrent jobs
spreading jobs across multiple machines
using advanced placeholder functionality

But now that you have a more firm grasp of the basic concepts, you can work your way up to this advanced functionality.

Thank you to Simon Symeonidis for proof-reading this article.

¹ O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.

Rust and the values of simplicity and ergonomics

Sat, 11 May 2019 07:06:47 -0400

If you’ve not seen Bryan Cantrill talk about software as a reflection of values, let me give you the gist. He argues that there are many desirable values that we can demand in our software (e.g., performance, reliability, approachability). These values, though they are all desirable, are in tension: we can’t have them all at the same time. He also argues that for some people, some of these values are more important than others and the software they write is a reflection of those core values; these core values are pursued at the expense of other values.

In the video above, Cantrill gives C as an example: the core values of C are performance, simplicity, interoperability, and portability. He gives another example, OpenBSD, and how it has just one core value: security. Now, it’s not that the authors of C did not value, say, readability, or that the OpenBSD developers do not value performance – there are no “bad” values. But when values are in tension and a decision needs to be made, the core values will always win (if they didn’t they wouldn’t be core values). As an example, he points out that OpenBSD disabled hyper-threading, because it was insecure: their core value of security won over the value of performance.

What I’ve come to really appreciate from Cantrill’s thesis is that it offers a way to understand why we are sometimes bothered by some technical decisions, yet cannot offer a strong technical counter-point. When that happens, it’s typically because our own core values are in conflict with the values of the software we use.

If you head over to the Rust website, you’ll see in big bold letters the words Performance, Reliability, and Productivity. A number of new Rust features in recent years have put those three core values above simplicity, for example:

match ergonomics
the ? operator
non-lexical lifetimes
async IO

In all these cases, the language was made less simple, but in none of these degraded performance, reliability, or productivity. Many people, including some of my colleagues, are unhappy with these features and try to find technical reasons for why they are bad. But the technical arguments don’t hold very well. So the argument against them is typically one of values (even if the person doing the arguing does not realize it). They go against the value of simplicity, and that ticks off the people for whom simplicity is more important than productivity. (Or even, for those who argue that simplicity is necessary for productivity.)

Feeling more confident with property-testing

Tue, 19 Feb 2019 21:42:32 -0500

For one of my projects at work, I thought I could reduce memory usage by storing vectors of sorted integers as vectors of ranges. For example, the vector [1,3,4,5,10,14,15,16,17,18] would be represented by [1, 3..5, 10, 14..18].

(Spoiler: the memory usage was higher because there weren’t enough runs of consecutive numbers to offset the cost of using 8 bytes rather than 4 bytes for single numbers.)

I quickly wrote a Rust function to convert a slice of u32s to a vector of ranges.

type Range = (u32, u32);

/// Returns the shortest vector of ranges that covers `xs`.
///
/// The argument `xs` should (1) be sorted, (2) contain no dupes.
/// If `xs` respects (1) and (2), then the following invariant holds:
///     expand(&to_ranges(&xs)) == xs
pub fn to_ranges(xs: &[u32]) -> Vec<Range> {
    let mut v: Vec<Range> = Vec::new();

    if xs.is_empty() {
        return v;
    }

    let mut start_idx = 0;
    for idx in 1 .. xs.len() {
        if xs[idx] != xs[idx-1] + 1 {
            v.push( (xs[start_idx], xs[idx-1]) );
            start_idx = idx;
        }
    }
    v.push( (xs[start_idx], xs[xs.len() - 1]) );
    v.shrink_to_fit();
    return v;
}

Array indices can be hard to get right, and anything more complex than a simple for loop and I start doubting myself and I wonder if there’s a subtlety I’ve gotten wrong. My code seems to work, but I’m afraid that it could blow up at the worst possible time. I’m sure it’s a feeling shared by many programmers.

To give us more confidence, we write unit tests. We prepare carefully crafted inputs and we write down their expected outputs; we pass the inputs to our function, and we ensure that our predicted outputs match the actual outputs. Here’s a unit test for to_ranges:

#[test]
fn manual_tests() {
    assert!(to_ranges(&[]).is_empty());
    assert_eq!(vec![(1,1), (3,3), (5,5)], to_ranges(&[1,3,5]));
    assert_eq!(vec![(1,4)], to_ranges(&[1,2,3,4]));
    assert_eq!(
        vec![(1,1), (3,5), (10,10), (14,18)],
        to_ranges(&[1,3,4,5,10,14,15,16,17,18]));
}

Is this sufficient? I don’t know. All the tests pass, but there are only 4, that’s not very much. Maybe there’s a condition that’s not tested here? Maybe I subconsciously avoided an edge case? We wanted to increase our confidence in our code, but we’ve only increased our self-doubt!

Property testing is just another tool in the testing toolbox, but it’s one that can increase our confidence in our code. Rather than test a function in a case-by-case fashion, we instead write properties: we express relationships that exist between all inputs and their outputs. We then let a generator feed hundreds, even thousands of test cases to our function and verify that our properties hold.

The property for to_ranges is given in its doc-comment: if an acceptable input—a slice of sorted u32s with no duplicates—is given, then we should be able to “expand” the output and obtain the input again. This is a common pattern in property testing.

Using the proptest crate, I create a simple property and I ask the proptest runner to generate 5000 inputs and ensure that the property hold.

use proptest::prelude::*;
use proptest::test_runner::Config;

pub fn expand(ranges: &[Range]) -> Vec<u32> {
    let mut v: Vec<u32> = Vec::new();
    for (start, end) in ranges {
        v.extend(*start ..= *end);
    }
    return v;
}

proptest! {
    #![proptest_config(Config::with_cases(5000))]
    #[test]
    fn prop_to_ranges(bset in prop::collection::btree_set(any::<u32>(), 0 .. 1000)) {
        let input: Vec<u32> = bset.into_iter().collect();
        let output: Vec<u32> = expand(&to_ranges(&input));
        prop_assert_eq!(&input[..], &output[..]);
    }
}

(I use the generator for BTreeSet<u32> to obtain a sorted set of u32s and then convert to a Vec<u32>. I think it’s simpler than generating a vector, sorting it, and de-duplicating it.)

With not very many lines of code, I’m able to greatly increase my confidence that I didn’t mess up the indexes in to_ranges. It’s still possible that there’s something wrong that my property test missed, but it would be much less likely that a bug was present in code that was tested 5000 times than in code that was tested 4 times.

Property tests are a great addition to your testing toolbox. A recent book by Fred Hébert, Property-Based Testing with PropEr, Erlang, and Elixir, offers a great and approachable introduction to property testing using tools for Erlang and Elixir. Although the book is mostly targeted at programmers of these two communities, there is enough language-agnostic advice to make it a good purchase for anyone interested in using property testing to make their programs more robust.

purp, an Emacs theme of few colors

Wed, 12 Dec 2018 20:02:15 -0500

A few years ago, I asked on Reddit whether anyone had recommendations for a theme that used only a few colors. I found that too many colors—what many refer to as a Christmas tree—was not helping me understand code better, but in fact was distracting me. I also thought that turning off syntax highlighting entirely was too radical a solution. I wanted a theme that would just highlight a few, well-chosen elements. Unfortunately, most of the minimal themes that I found were monochrome, which is not what I was looking for.

So I began working on a theme of few colors that I’ve been using and tweaking for a while now. I’ve finally decided to make it available on Github and publish it to MELPA. You can install purp by invoking the command M-x package-install RET purp-theme. A light version called purp-light is also included in the package.

I have not gone and modified the color of every face in every mode, just the ones that I encountered daily in my work. In programming modes, you’ll see the following colors:

Purple for function definitions
Green for comments
Orange for string literals
Yellow on red for dangerous stuff (e.g., the unsafe keyword in Rust)
White (or black) for everything else

The colors for function definitions and comments makes them easy to find and the color for strings helps to spot an un-escaped double quote.

I hope you like purp, there’s a lot of work left to be done, including how to not have it activate as soon as it’s installed, which is a really bad first impression.

Rust 2019

Sun, 09 Dec 2018 10:45:57 -0500

(This is my response to the call for Rust 2019 Roadmap blog posts.)

In 2019, there are three areas where I would like to see the Rust community focus its efforts:

Improved compile times
A community effort to review crates
More “80% solutions”

Improved compile times

I already wrote about compile times in Rust (see Rust at Work—Two Paint Points and How to alleviate the pain of Rust compile times), and I don’t have much more to add. I just want to re-iterate that this issue is very important and that it should be given the attention it deserves.

The Rust compiler is much slower than the compilers of some “competing languages” such as Go, D, or Jai. As Jonathan Blow put it when he first announced that he was working on a programming language, long compile times are a constant source of friction for developers: they lengthen the write-test loop and they disrupt the programmer’s flow. The success of Go can partly be attributed to its quick compile times; it compiles fast enough that it feels like an interpreted language. (Of course, other factors contributed to Go’s success—an extensive standard library, a vibrant ecosystem, a friendly community, great tooling, good performance, etc.) I want new Rust programmers to not be turned off by Rust’s long compile times and I want veteran Rust programmers to iterate more quickly and wait less on rustc.

The situation is improving. Incremental recompilation has been merged and tools like sccache can speed up compilation in CI. Nicholas Nethercote detailed his experiments in making rustc faster. A new, upcoming backend, cranelift, appears to be faster than the current LLVM backend.

I think most Rust programmers realize that improved compile times would be beneficial to everyone, and I look forward to seeing improvements on that front in 2019.

A community effort to review crates

Rust would not be used nearly as much if it weren’t for Cargo and crates.io. The rich ecosystem of Rust is what made it go from an interesting programming language project to a tool that many now use in production. In 2019, I would like to see a code review effort from the Rust community. The goal of these reviews is to provide answers to two questions for anyone looking for a crate.

What are the quality crates? What are the best crates for making HTTP requests, for consuming from Kafka, or for manipulating dates? Are those crates reliable enough for business-critical services? Should I have concerns about their performance or security? Is the code easy to understand and modify? In a nutshell, how does one find quality crates?

At the moment, we try and decide if a crate is worth checking out based on word-of-mouth, number of downloads, number of GitHub stars, the name of the contributors, etc. Code reviews would be stronger indicators of quality, because someone actually read the code. On Reddit, Dawid Ciężarkiewicz announced crev, a language-agnostic tool for publishing and signing code reviews and forming webs of trust. A tool like that would be ideal for the crates ecosystem. At my place of work, we’ve reviewed a number of the crates that we use in production, to convince ourselves that they were worth using. We’d love to have a place to share that work with the reset of the community.

Is this crate compromised? In 2018, we heard a few stories (1, 2) about malicious packages in NPM, the Node.js package repository. I don’t think I’ve heard about any backdoors in a crate yet, but I see nothing that would make such attacks impossible. (We did have an issue back in October with squatting that actually led to a degradation of service.)

These attacks will undoubtedly become more common in the coming years. Some aspects of Cargo make these attacks more difficult: the immutability of crates and the ability to pin dependencies at a particular version (ideally, a versio that has been reviewed). Active code reviews would provide an extra line of defense against these attacks. They would also, hopefully, increase the overall quality of the most popular crates.

More “80% solutions”

Improved compile times and code reviews require time and effort, and are not likely to occur overnight. While we wait, I’d like to see more “80% solutions”—projects that perform a task well enough, but without being perfect. We are lucky to have some incredibly high-quality projects in the Rust ecosystem: serde, hyper, reqwest come to mind. They are also what I’d call 100% solutions: they offer all the features that users could possibly want and they have extremely good performance. They are also complex beasts, and that makes them more difficult to review. Some of these projects (e.g., hyper) also have many dependencies, which makes the review process even more difficult, and also increases compile times.

An example of an 80% project is miniserde: unlike serde, JSON is the only serialization format that miniserde supports, and some Rust datatypes like enums cannot be encoded. Miniserde has also 10x less code than serde (tokei reports 2379 lines of Rust code for miniserde vs. 25,205 for serde) so it’s much easier for a single person to do a full review of the code.

I’d love to see more such projects: a simple HTTP server for services that just need a /status endpoint; an HTTP client that can send simple requests and write the responses in a Write object.

vi, my favorite config-less editor

Fri, 30 Nov 2018 22:14:06 -0500

Before I became an Emacs user, I was a vi user, and though Emacs is my primary editor, I still use vi daily. I use vi for quickly skimming the content of a file and quick edits to configuration files; I use vi when I’m on a remote server or when a command-line utility invokes $EDITOR. The main reason why I still use vi is because I’m able to use it very efficiently even without a configuration file.

I learned vi circa 2001 from a magazine and from vimtutor. It was my primary text editor until 2007 or 2008 when I switched to Emacs. Back then, I had a rather long .vimrc, complete with configurations for Linux and Windows, GTK vim and curses vim, and I used a few plugins such as bufexplorer.vim.

Recently, I deleted my old .vimrc and started a new one from scratch. I no longer use plugins and I don’t use a GUI version of vim. For these reasons, I now consider myself more a vi user than a vim user. (I am however a big fan of text objects, and I want unlimited undo, so using nvi is not something I would consider.) My .vimrc is now only 12 lines and most of these changes are conservative and uncontroversial: backspace works “normally” in insert mode all the time, I enable incremental search, I use 4 spaces for indentations rather than hard tabs, I activate syntax highlighting. Even though vim offers a myriad of settings, a vanilla setup is extremely capable and usable.

I find that being able to use an editor without custom configuration is extremely useful. Here are a couple of examples: I learned how to exploit Linux binaries from the book Art of Exploitation. The book comes with a companion live CD that you can use to follow examples from the book and try to perform the hacks yourself. Emacs is not installed in the live CD, but vim is, and I was perfectly happy and comfortable using it to read the source code of the examples and modify them. Another example: recently at work, I wanted to write a Python script on a server that didn’t have Python installed, but had Docker. I launched a Python container, installed vim-tiny, and proceeded to write, debug, and improve my script without ever feeling that I was editing with a hand tied behind my back.

It is oddly satisfying to know that no matter the kind of machine I find myself on or the kind of restrictions it has (no network to download Emacs and/or my custom configuration, corporate policy against fetching packages from MELPA, etc.), that there will always be an editor that I can happily use without needing to make it “my own”.

Tim O’Reilly and Paul Graham have also cited usability without configuration a reason for preferring vi to Emacs.

Thanks to Richard Kallos for proof-reading an early version of this article.

The evolution of performance in ppbert

Wed, 19 Sep 2018 15:58:49 -0400

Today I released ppbert 0.8.4 after I read a comment on Hacker News that made me leap out of my chair: buffering StdoutLock improves the performance of IO-bound programs! Previously, I thought that doing stdout.lock() was sufficient. Oops!

This release also marks the first time that one of my original test files can be pretty printed in less than a second. I’ll use this occasion to look back on ppbert and how I was able to improve its performance, little by little.

As I mentioned in a previous post, I created ppbert because I needed a tool to inspect the content of the .bert files we use in production, and Erlang was just too slow for that purpose. There were two files that were of particular interest to me; in this post, I’ll just call them FileA and FileB.

FileA is 11 MiB in size, and is the most important .bert file in our production system. If something seems off, it is the first place where I want to go look. On my laptop, Erlang takes 9 secs to parse and pretty print FileA.
FileB is 96 MiB in size and it was the largest file .bert file we had at the time. On my laptop, Erlang takes 50 secs to parse and pretty print FileB.

Pre-historic times: `nom`

Every quarter at work, we reserve one Friday to work on open source projects. I started ppbert during one of those Open Source Fridays, and I used nom to get moving quickly. By the end of the day, I had my first working prototype, and I was very excited: my new prototype was faster than Erlang! I don’t have time measurements from back then; my current laptop processes FileA in 2.63 secs and FileB in 15.60 secs.

Over time, though, I grew disenchanted with nom. I had difficulty using its macros, and it was difficult to have my own custom errors, so I decided to write my own parser. (N.B.: this was with nom 2; I have not tried nom 3.)

The first tagged releases

After a bit of work on my own hand-written parser — not very hard and surprisingly fun for me — I was able to parse and pretty-print again. This is when I started turning my attention to performance. I was happy that I was faster than Erlang, and I was already using ppbert daily at work. Still, I thought it should be possible with Rust to improve the performance of ppbert further. I thought that might make the tool more useful to me and others. (I strongly believe that performance is an enabler: if something is fast enough, people use it more, and they can often use it in new and surprising ways.)

The first tagged release of ppbert, 0.1.2 in April of 2017, took 1.26 secs to process FileA and 5.42 secs for FileB. Before, I was printing directly to stdout; in 0.1.2, I decided to implement the Display trait for my AST, BertTerm. I was really happy that writing to an in-memory buffer had such a noticeable impact on performance!

The next performance improvement came 3 weeks later, ahead of the 0.2.3 release. I noticed that I was computing the indentation string (i.e., a string made of white spaces) for every element of a list or a tuple. I moved this computation outside of the loop, along with a few other minor changes (e.g., using .write_str() and .write_char() instead of write!("{}", expr) when approriate). The time to process FileA and FileB were now 0.7 secs and 3.0 secs respectively.

In June and July, I toyed around with a few micro-optimizations. If an atom contained only ASCII characters, I skipped the latin-1 validation and called the unsafe String::from_utf8_unchecked() function. I also reserved the right amount of memory for the buffers for Strings and Binaries. Together, these small optimizations brought the times down to 0.69 secs and 2.88 secs.

At that point, I was happy with performance, and concentrated mostly on features: adding support for JSON output, IMO one of the best feature of ppbert when combined with the amazing jq; supporting our new .bert2 format at work; a new command-line tool to convert between .bert and .bert2; improving the user interface; cleaning up the internals; etc.

Eight months pass

I kept using ppbert daily, and I was really happy with it. Eventually, I got back to trying to find places where I could improve performance. In ppbert 0.5.2, released in April 2018, I found that it was faster to write my String and Binary literals into a String buffer and send that entire buffer to .write_str(). The time to process FileA stayed pretty much the same at 0.65 secs, an improvement of 40 milli-seconds. However, for FileB, which is larger, the optimization reduced the time needed to pretty print it to 1.77 seconds, a one second improvement!

The 0.6.1 release was interesting: I managed to remove a bunch of unsafe blocks that I had put in for “performance reasons”, but without affecting the overall performance.

And that’s where performance had been, until today, when I added this simple line:

let mut stdout = BufWriter::new(stdout);

Now, ppbert can process FileA in 0.25 secs and FileB in 0.75 secs. I’m really happy with the progress that I made and with the way I made progress: by finding little places where I could improve performance, and slowly making ppbert faster and faster.

Here is a small chart showing the evolution of ppbert’s performance on FileA and FileB:

Thanks to Richard Kallos for proof-reading this article.

How to alleviate the pain of Rust compile times

Sat, 11 Aug 2018 07:10:12 -0400

A few days ago, I wrote about two Rust pain points when using Rust at work. One of these points were the long compile times. In this post, I want to share a few tips that can help alleviate that pain.

Use `cargo check`

We typically use the compiler for two reasons: to verify if the syntax and/or types are correct and to generate a runnable program. When compiling a program, especially a release build, the majority of the time is spent generating LLVM bytecode and optimizing that bytecode. If you only want to know whether your 3-line change typechecks, you don’t want to wait for the optimizer.

For this reason, Cargo has a subcommand called check that only invokes the front-end. It completes faster than a debug build and much faster than a release build.

As an example, here are the timing results of cargo check and cargo build for my personal project, ppbert. (The benchmarks are performed by hyperfine.)

Benchmark #1: cargo clean && cargo check

  Time (mean ± σ):      8.131 s ±  0.424 s    [User: 24.234 s, System: 1.245 s]

  Range (min … max):    7.529 s …  8.845 s

Benchmark #2: cargo clean && cargo build

  Time (mean ± σ):     16.904 s ±  0.794 s    [User: 52.009 s, System: 2.240 s]

  Range (min … max):   15.728 s … 18.098 s

Benchmark #3: cargo clean && cargo build --release

  Time (mean ± σ):     48.454 s ±  2.540 s    [User: 145.644 s, System: 3.205 s]

  Range (min … max):   46.229 s … 54.107 s

Summary

  'cargo clean && cargo check' ran
    2.08x faster than 'cargo clean && cargo build'
    5.96x faster than 'cargo clean && cargo build --release'

From a clean slate, cargo check is 2x faster than cargo build and 6x faster than cargo build --release. During development, when you just want to check that what you wrote is correct, definitely reach for cargo check.

Use `sccache`

A co-worker introduced me to sccache, a compilation caching service by Mozilla that is compatible with Rust. It will cache the build artifacts that Cargo generates, so if you cargo clean your project or go to work on a second project that shares dependencies with the first one, you won’t have to rebuild everything.

You can install sccache with Cargo: cargo install sccache. To activate sccache, add the following line to your .bashrc:

export RUSTC_WRAPPER=sccache

And here are the time differences.

Benchmark #1: cargo clean && cargo build

  Time (mean ± σ):     15.726 s ±  0.330 s    [User: 50.183 s, System: 2.210 s]

  Range (min … max):   15.148 s … 16.210 s

Benchmark #2: cargo clean && RUSTC_WRAPPER=sccache cargo build

  Time (mean ± σ):      6.877 s ±  4.665 s    [User: 6.964 s, System: 0.916 s]

  Range (min … max):    5.135 s … 20.136 s

  Warning: The first benchmarking run for this command was
  significantly slower than the rest (20.136 s). This could be caused
  by (filesystem) caches that were not filled until after the first
  run. You should consider using the '--warmup' option to fill those
  caches before the actual benchmark. Alternatively, use the
  '--prepare' option to clear the caches before each timing run.

Summary

  'cargo clean && RUSTC_WRAPPER=sccache cargo build' ran
    2.29x faster than 'cargo clean && cargo build'

Given the ease of installing and configuring sccache, I see no reason not to use it and get that 2x speed boost. One downside of sccache: it won’t make your continuous integration builds faster. (You could configure sccache to use an S3 bucket, but I prefer to have completely clean builds from my CI.)

Avoid LTO

LTO is the acronym for link-time optimization. Compilers build and optimize compilation units individually. LTO is a mechanism for taking those individually-optimized object files and finding more opportunities for optimization when they are considered as a group. This is great if you want to have the fastest possible program, but the price you pay are higher compilation times. In Rust, the compilation unit is the crate; if the heavy computations are all done in your crate, then maybe LTO isn’t necessary.

Here is the difference in execution time of ppbert with and without LTO on a 100 MiB .bert2 file.

Benchmark #1: /tmp/ppbert-without-lto -2 /tmp/100m.bert2

  Time (mean ± σ):      1.926 s ±  0.005 s    [User: 1.308 s, System: 0.618 s]

  Range (min … max):    1.921 s …  1.937 s

Benchmark #2: /tmp/ppbert-with-lto -2 /tmp/100m.bert2

  Time (mean ± σ):      1.887 s ±  0.012 s    [User: 1.273 s, System: 0.613 s]

  Range (min … max):    1.867 s …  1.905 s

Summary

  '/tmp/ppbert-with-lto -2 /tmp/100m.bert2' ran
    1.02x faster than '/tmp/ppbert-without-lto -2 /tmp/100m.bert2'

There is a slight difference (which is why I enable LTO), but let’s look at what LTO does to compile times (with sccache enabled):

Benchmark #1: sed -i "s/lto.*/lto=false/" Cargo.toml && cargo clean && cargo build --release

  Time (mean ± σ):      5.793 s ±  0.075 s    [User: 11.390 s, System: 0.767 s]

  Range (min … max):    5.693 s …  5.901 s

Benchmark #2: sed -i "s/lto.*/lto=true/" Cargo.toml && cargo clean && cargo build --release

  Time (mean ± σ):     18.483 s ±  0.321 s    [User: 34.288 s, System: 1.182 s]

  Range (min … max):   17.971 s … 18.893 s

Summary

  'sed -i "s/lto.*/lto=false/" Cargo.toml && cargo clean && cargo build --release' ran
    3.19x faster than 'sed -i "s/lto.*/lto=true/" Cargo.toml && cargo clean && cargo build --release'

LTO makes the compilation of ppbert 300% longer for a 2% speed gain at run-time. If you have a project that you find too long to build, I encourage you to measure the compile times with and without LTO, and the execution speed with and without LTO. Maybe you’ll find (like I did in a project at work) that you don’t want to enable LTO.

Control your dependencies

Rust has a rich ecosystem, and every day we have access to more quality crates. This is mostly a blessing, but more crates mean more compilation, and this has a negative effect on your compile times. Now, don’t shun all dependencies and write everything yourself! Rather, make sure that the crates your bring in pay for themselves.

One thing that you can do to control dependencies is to examine your depdendencies and see if they have features that you can disable. For instance, in ppbert I use clap to parse command-line arguments; by default, clap colors its output and offers suggestions when the user makes a spelling mistake. Some people like these features, but I’m not a fan myself. Fortuately, the authors of clap have made such features optional. So I am able to deactivate what I don’t need, and not pay the compilation price for it.

Here’s the dependency tree of clap with default-features enabled, and without.

$ rg default-features Cargo.toml && cargo tree -p clap
33:default-features=true
clap v2.31.2
├── ansi_term v0.11.0
├── atty v0.2.11
│   └── libc v0.2.42
├── bitflags v1.0.3
├── strsim v0.7.0
├── textwrap v0.9.0
│   └── unicode-width v0.1.5
├── unicode-width v0.1.5 (*)
└── vec_map v0.8.1

$ vi Cargo.toml

$ rg default-features Cargo.toml && cargo tree -p clap
33:default-features=false
clap v2.31.2
├── bitflags v1.0.3
├── textwrap v0.9.0
│   └── unicode-width v0.1.5
└── unicode-width v0.1.5 (*)

So I only need to compile 4 external crates rather than 9. Many crates make some of their dependencies optional; make sure you only bring in what you really need.

Conclusion

Rust doesn’t have the fastest compiler in the world, and that creates friction when writing code. I’m looking forward to improvements on that front more than any other Rust feature. Fortunately, in the mean time, there are ways to make Rust compile your projects a little faster.

Rust at Work—Two Paint Points

Thu, 09 Aug 2018 13:47:12 -0400

The 2018 State of Rust Survey asks the following question:

How could we help make Rust more accepted at your company?

We use Rust in three projects at work, and the two primary concerns for people are (1) the crates ecosystem, and (2) compile times. I gave some details in my answer to the survey, and I want to expand on those two points here.

The first pain point we have with Rust at work is the crates ecosystem. We like Cargo and we like that it offers a first-class solution to managing dependencies and building a project. However, it’s difficult—especially for people who are new to Rust—to know which crates are good and which crates should be avoided. We mostly have to rely on our past experience with different crates—I recommend clap, regex, and rayon, because I’ve had a good experience with them, but was I just lucky to avoid their bugs? The number of downloads or the number of GitHub stars can give an indication of the popularity of a project, but it does not tell if the project has had a complete code review by a third party, if it’s been fuzzed, if the authors are using it in production, etc. It would be a great help to navigating the crates ecosystem if there were badges that projects could obtain when their versions have met levels of quality assurance that go beyond a unit test suite.

The other pain point of Rust that makes its adoption less attractive are the long compile times. It’s well known that compiling Rust code takes a long time, especially when using the --release flag for cargo build. There are ways to make compilation faster (this will be the subject of an upcoming post) but the bottom line is that iterating in Rust is slower than in many other languages, including compiled languages like C or Java.

When he announced that he was working on a new programming language, Jonathan Blow said that he wanted to minimize friction in his language, i.e., the difficulty/effort going from an idea to an implementation. I think Rust’s compile times are one of its greatest source of friction. The memory model of Rust—with lifetimes, shared borrows, mutable borrows, moves, etc.—is a steep curve when learning Rust, but eventually people form their own mental models, and can then write code quickly and safely. Waiting on the compiler doesn’t get shorter because you are more experienced with Rust. I’m following with interest the efforts to make rustc faster, and I hope that 2018 or 2019 will bring improvements on that front.

There is one point that no one at work disputes though: the Rust language is incredibly well-designed. The fans of C and low-level imperative programming and the fans of Scala and high-level functional programming find that the Rust core team has made judicious choices in creating a safe, modern, and fast language.

Why you should invest in an editor

Thu, 31 May 2018 21:18:50 -0400

Emacs 26.1 was released this week and I thought I’d use the occasion to write a quick post about the main reason why I stick to Emacs: consistency. Happily, I realized that this reason applies to other text editors as well.

I use Emacs for programming; whether it’s Rust, C, Erlang, Python, or bash scripts, Emacs is the editor I choose. For any language I use, I have syntax highlighting and automatic indentation, and depending on the major mode, I can get other cool features such as jump-to-def, auto-completion, or REPL integration.

I use Emacs for writing; for my blog posts in markdown, for my work journal in org-mode, and for my LaTeX documents, Emacs is the tool I reach for. The usual actions of spell checking a word, formatting a paragraph, jumping from one sentence to the next all work the same no matter the kind of content.

I also use Emacs for other tasks: I use magit to manage my git repos, I use elfeed to follow RSS feeds, I use twittering-mode to read Twitter. I can use usual text tools such as searching by regex or M-x occur if I’m looking for something in particular. The tool doesn’t have to provide the functionality, I get it for free from my editor.

When I’m asked why I don’t pick specialized tools, my answer is that I enjoy having a consistent experience, no matter the kind of work I’m doing. Emacs gives me that. My fingers can perform tasks before I’ve had time to think about them. When I used to switch between Emacs and Eclipse (which I used for Java programming), it would take me a few minutes to readjust when I switched from one tool to the other. Instead, I prefer to use a single tool and not deal with the difficulties of constant context switching.

I use Emacs, but if you use Vim, Sublime, Visual Studio Code, or any other editor, you can probably relate. How fun is it to use Vim text objects in code and in config files, how enjoyable is it to have access to Sublime’s multi-cursors in your master’s thesis as well as in the code of your personal web page? There is great value in taking the time to learn and master your text editor and using it for as much work as is reasonable. (Don’t be a zealot though: if another tool is clearly better, use it. Emacs is a terrible web browser, use Firefox or Chrome instead.)

What's next for ppbert?

Sun, 27 May 2018 12:06:13 -0400

ppbert has improved since the last time I wrote about it:

I’ve done more work to improve the speed of pretty-printing; the large .bert file that took 3 seconds to process in Februrary now takes 1.5 seconds.
I’ve added binary artifacts to releases — binaries for ppbert and bert-convert that are statically-linked against musl. You should be able to download these and run them on any 64-bit Linux system. (I inspired myself heavily from the work of my former co-worker, Sevag.)
I’ve added support for a new type of file, Erlang’s internal disk_log format.

The next feature of ppbert that I have in mind might be the most ambitious yet: support for pretty-printing Erlang records. In Erlang, a record is stored as a tuple: the first element is the name of the record, and the remaining elements are the values of the fields. Consider the person record below.

-record(person, {
    name,
    gender,
    age
}).

#person{name="Obama", gender=male, age=56} = {person, "Obama", male, 56}.

The record expression (#person{ ... }) and the tuple expression ({person, ...}) are equivalent. When term_to_binary/1 converts a record, it returns a binary for a tuple.

ppbert can parse and pretty-print tuples, so it can handle records without any problem. However, it would be useful for users to see their records with the field names, rather than the tuple representation. To support this, I will add a new command-line option to ppbert to read the record declarations in a .erl or .hrl file.

ppbert --record-definitions my_module.erl --record-definitions my_include.hrl ...

One guiding principle for this feature is to never require the user to re-create their record declarations. I want users to point to their existing Erlang files and have ppbert do the hard work. To support this feature, I will need to write a scanner and parser for Erlang. The fields of a record declaration have three parts:

The name of the field (required);
The default value of the field (optional);
The type of the field (optional).

Here is a more complex definition of the person record from above:

-record(person, {
    name                                 :: string(),
    gender = undefined                   :: undefined | male | female,
    age    = floor(100 * rand:uniform()) :: pos_integer()
}).

The default value of a field can be an arbitrary expression (though there are a few limitations), and that’s why I think I need a complete parser. I can’t think of a way to get the name of a record’s fields with just a scanner.

This feature will require a lot of time and effort, but I think it’ll be worth it and should make ppbert an even more useful tool for Erlang programmers.

Reflecting on ppbert

Fri, 09 Feb 2018 19:06:08 -0500

I had an itch: I was pretty-printing the BERT-encoded terms that we use in a production system at work and it was very slow. The Erlang shell took more than two minutes to dump the largest file. (It took about 0.1 second to read and parse the file; the rest was spent in io:format.) I decided to scratch that itch: I wrote ppbert, a command-line utility that reads BERT-encoded values and pretty-prints them. I’ve worked sporadically on ppbert for almost a year now, I use it daily at work, I’m happy with it, and I want to write about some of the things I learned during that journey.

I wanted ppbert to be a good Unix tool. It was essential that it can read from stdin, write to stdout, and thus can be inserted in a Unix pipeline. (I’m astounded—and annoyed—when programs can’t be used in a pipe.) The interface of ppbert should be familiar to Unix users: it reads from stdin, from files, or from a mix of both with - acting as the filename for stdin; normal output goes to stdout, diagnostics and errors go to stderr; the format of warnings and errors follows the typical Unix style (ppbert: reason for error); the return code is 0 on success and 1 on failure; ppbert has a manpage. The main “failure” of ppbert as a Unix tool is that it consumes a whole input file before it starts pretty-printing it (a bit like sort(1)).

I wrote ppbert in Rust because I like the language and because I wanted to use it. But if you’d like to hear technical reasons why choosing Rust made sense, here are a few. First, I wanted something that would be faster than Erlang and it would’ve been counter-productive to reach for Python or Ruby; Rust compiles to native code, uses LLVM to apply thousands of optimizations, and many benchmarks shows that the performance of Rust programs is right up there with C and C++. Rust’s type system and ownership/borrowing systems are also right up my alley: I suck at programming, I make mistakes with strings and memory in C all the time, and the Rust bondage helps me write better software. Rust has a very good story for third-party libraries: cargo is an absolute joy to use, and the selection of quality libraries on crates.io is growing quickly (more about good libraries later). I tried to limit the number of dependencies in ppbert—for example, I don’t use serde to pretty-print JSON—, but I like that complex functionality is a single line in Cargo.toml away. Finally, the Rust compiler produces statically-linked executable, which makes it easier to use ppbert on remote servers that don’t have development tools installed: I just transfer the executable with scp.

I’m happy that I chose Rust: I had a faster pretty-printer the first time that I compiled an executable and with some further tweaks, I was able to improve the speed to a level that I’m happy with. Today, I can pretty-print the large BERT file in 3 seconds rather than 2 minutes. I got some of the extra performance gains by applying lessons that one can learn from any language: buffering your output—the File struct in Rust doesn’t buffer by default; finding ways to do less work in the common cases—in ppbert I got a satisfying speed boost by avoiding UTF-8 validation when all the bytes of an atom’s name where in the ASCII range. I also got some speed boosts by reaching for some of Rust’s unsafe functions, for example by using the .get_unchecked() Vec method when the bound checks had already been performed. I found that using perf with Rust is underwhelming; it could tell me in which functions ppbert spent most of its time, but I had to divine the reasons.

I’m quite happy with the third-party crates that I use. I started using nom, a parser combinator library, but I found that using its macros was unnatural and injecting my own errors was awkward, so I stopped using it and wrote my own parser instead. (This was with nom 2; I have not tried the newer nom 3.) For command-line arguments parsing, I like clap: it’s easy to add new options, and easy to consume them. Rayon blew me away! I used it in ppbert’s sibling program, bert-convert, and simply by using .par_iter() rather than .iter(), I made the conversion of all our production .bert files to .bert2 twice as fast. The crates num and encoding, which I use for bigints and latin-1 support respectively, were easy to use and because they do their jobs without problems, I haven’t needed to look at that code in a long time.

Some people lament the state of Rust’s tooling, particularly IDE support. I’m a simple man: I like some syntax highlighting, automatic indentation, jump-to-definition, and auto-completion. Emacs, rust-mode, and racer-mode gave me all that.

One issue I ran into with Rust itself was how the println!() macro behaves in a pipeline: if a pipe is closed early while printing a long string, println panics. The code below exhibits the behaviour in a simple program.

/tmp$ cat sample.rs
fn main() {
    let mut buf = String::new();
    for _ in 0 .. 65536 {
        buf.push_str("foo\n")
    }
    println!("{}", buf);
}

/tmp$ rustc -O sample.rs

/tmp$ ./sample | head -n 1
foo
thread 'main' panicked at 'failed printing to stdout: Broken pipe (os error 32)', /checkout/src/libstd/io/stdio.rs:690:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.

There is an easy fix, and that’s to use the writeln!() macro and to ignore errors.

/tmp$ cat sample.rs
use std::io::{self, Write};

fn main() {
    let mut buf = String::new();
    for _ in 0 .. 65536 {
        buf.push_str("foo\n")
    }
    let _ = writeln!(io::stdout(), "{}", buf);
}

/tmp$ rustc -O sample.rs

/tmp$ ./sample | head -n 1
foo

It’s something to be wary of if you write Unix utilities in Rust.

Ppbert is not a popular program—I’m probably its only user—because BERT is not a very popular format. Still, I’m happy with the work I did, I’m happy that I scratched that itch, and I’m happy that ppbert has been a good vehicle for learning about Rust, improving performance, and creating good Unix programs.

Quick reflection on first good Dwarf Fortress

Tue, 02 Jan 2018 20:44:17 -0500

I just retired my first good, semi-successful fort in Dwarf Fortress. I decided jot down a few comments while the memories are still fresh. My fort, Commonbread, lasted 6+ years, survived 4 goblin sieges, had 150 dwarfs at its peak, and generated more than ☼1.6M in value.

Main mistakes

My two principal mistakes are related: I picked a cold embark location, and I opened my gates to fight a goblin siege during late-autumn. My 30-dwarfs military had been training for more than a year and was pretty decent: the siege was pushed back with one casualty. A number of dwarfs were injured during the siege and went to the hospital to get patched up. Unfortunately, because the brook was frozen, they did not receive any water and 20 of them died of thirst.

It would have been possible to leave my gates closed and let the siege last for however long. I could have continued playing, but because of some limitations I imposed on myself (explained later), I decided that their deaths would signal the end of the fort. After engraving memorial slabs to the fallen dwarfs, I retired Commonbread.

Small mistakes

My fort layout was not optimal: dwarfs had to slowly carry stones across long distances, the stock piles were too general and too far away from related workshops. It was to be expected because to keep things simple, I challenged myself to keep the entire fort on one Z-level until I had used all the available space.

Another mistake was distribution of labor. I should have created a few classes, e.g., “farmer”, “miner”, “stone worker”, “wood worker”, “hauler”, “military”. This would’ve made it easier and faster to temporarily switch dwarfs. I also learned that some professions, such as fish dissector, small animal dissector, and animal caretaker are practically useless.

I was also a bit too eager to make wooden spikes in a map that had not many trees; when I had to build beds or barrels, I was often out of wood.

Lessons learned

More than anything, this fort was a great vehicle for learning more about Dwarf Fortress. I learned how to use burrows to get my dwarfs to safety during a siege. I learned how to create military squads, how to make them train, how to move them around, and how to attack enemies. Making wooden spikes and wooden spike balls is a great way to raise money quickly for when the trade caravans come. A wall and a raised bridge is a great way to keep an army away. Some industries that seem unimportant at first, like a clothier, seem to become more important as time passes; I had a number of dwarfs unhappy with the crappy old clothes they were wearing.

Most important lesson (re-learned): Dwarf Fortress is incredibly addictive and immensly fun.

Dwarf Fortress and ivy-mode

Wed, 29 Nov 2017 19:14:21 -0500

A new version of Dwarf Fortress was released last week, and it was all the incentive I needed to start playing again. I suck at DF, but I love that game: it savantly combines depth and complete wackiness and the old-school look-and-feel certainly appeals to me. Yet, I think that the game has (at least) one huge flaw: its menu system makes ed(1) look positively user-friendly by comparison.

There are a lot of tasks that one can perform in Dwarf Fortress: select regions to be dug, build all kinds of shops and accessories, manage a militia, etc. All these tasks are accessed with the keyboard by navigating through menus. Consider for instance the Build main menu:

A few of the idiosyncracies of this menu:

The items are not ordered alphabetically nor grouped into related tasks, so if you are new (or haven’t played in a while), you need to read the entire list to find what you want.
A menu item and its hotkeys don’t necessarily match. For instance, seat has the shortcut c (for chair presumably) and the C of Wall/Floor/Stairs/Track is for Construct.
You can’t tell just by looking at it, but some of the items in that menu open sub-menus, e.g., Workshops and Wall/Floor/Stairs/Track. When hunting for a task to perform, you could read all the items in a menu without finding what you’re looking for and not having a clue that you were on the right track.

Looking into those two sub-menus, we see the same idiosyncracies from before: unordered items and unrelated keyboard shortcuts. Also, why is floor in a sub-menu, but floor hatch is in the primary menu?

Tarn Adams, the main developer of Dwarf Fortress, has said that he has very little interest in making the menus better as he constantly adds and removes items. Still I wonder, would it be possible to make the Dwarf Fortress interface better with little development overhead? I think that ivy, a GNU Emacs package, provides an attractive solution to the problem.

Ivy is a completion and narrowing framework; it replaces Emacs’ default auto-completion with a system that is much more powerful. When the user needs to enter an information that can be completed, ivy will show all the possibilites, and the user can navigate this list or type to narrow down the list of possibilities. For example, here is ivy showing the files managed by git in a project of mine:

I can use the up/down and PageUp/PageDown keys to move the selected element. I can also type in a fragment to narrow down the list:

In the screenshot above, I typed term and I now have only one result left. Pressing Enter will open the file.

Thanks to ivy’s integration with Emacs and with external tools (git, grep, etc.), a user can quickly find and jump to anything they want: a file, a function, a man page, a search string, etc. This kind of design would work quite well in Dwarf Fortress: rather than searching haphazardly in a menu, the player could open up a small window, enter a substring, and quickly select the action he wants to perform.

I used ivy to mock what such a system would look like:

And you know who else thinks this type of interface is a good idea? Tarn Adams himself. Indeed, there are some screens in Dwarf Fortress with this exact interface! The management screen allows the player to efficiently issue orders (e.g., Construct 10 wooden barrels). The management screen contains hundreds of tasks—the screenshot below shows what the screen looks like when it is first accessed—and so a filtering interface is ideal.

The management screen after typing wood bar:

Tarn Adams probably wants to focus on the magic system (yes!!) and the code of Dwarf Fortress is closed, however the community is dedicated enough that people have created tools to interact with Dwarf Fortress (e.g., Dwarf Therapist) and also improve Dwarf Fortress itself (DFHack). Perhaps a motivated hacker will attempt to improve the UI by leveraging the search+narrow framework already inside Dwarf Fortress.

What is a Type Error?

Wed, 13 Sep 2017 17:09:48 -0400

The top comment by /u/staticassert in a Reddit thread on type systems makes an interesting point, one that I made in the past myself: what is a type error?

I read those types (pun intended) of threads often enough, and one recurring argument often made by people who doubt the benefits of type systems goes like this: “I rarely make errors that the type checker would catch; my bugs are logic in nature.” The problem with this statement—and this is staticassert’s point—is that it’s not clear what constitutes a type error.

Consider the following scenario: reading one line from a text file when the file is opened in write-only mode. Is this a type error or not? Let’s look at what Python thinks.

with open("/etc/passwd", "w") as f:
    line = f.readline()
    print("[{0}]".format(line))

When we run this program, we obtain the following error at run-time:

Traceback (most recent call last):
  File "ty.py", line 1, in <module>
    with open("/etc/passwd", "w") as f:
IOError: [Errno 13] Permission denied: '/etc/passwd'

Python has a TypeError exception, but this is not the one raised here, so clearly this is a logic error, not a type error—or is it? Here is the equivalent code in OCaml:

let () =
  let f = open_out "/etc/passwd" in
  let line = input_line f in
  Printf.printf "[%s]\n" line

When we try to compile this code, we get the following type error at compile-time:

File "ty.ml", line 3, characters 24-25:
Error: This expression has type out_channel
       but an expression was expected of type in_channel

So which is it, is reading from a write-only file descriptor a type error or not? I think the best answer to this question is “reading from a write-only descriptor can be made into a type error.” At the level of the OS, reading from a file descriptor requires the number of a descriptor, a buffer, and a length. If something goes wrong, such as reading from a write-only descriptor, an error will be generated. Though that’s the reality of the OS, it doesn’t have to be the interface that a language presents to its users.

Here is another example: it’s a common error to add numbers of different unit types, e.g., adding feet and meters without a proper conversion. In F# the float type can be parameterized by a unit of measure (e.g., float<meter> vs. float<foot>) and an attempt to add two values of different type will cause a compilation error. If we ask the question “is adding values of different units a type error or not?”, the answer, like in the previous paragraph, is “adding values of different units can be made into a type error.”

Discussions on type systems would be more interesting if rather than discussing subjective topics without hard data (e.g., productivity, ease-of-use) we instead looked at how an error of logic can be made into a type error and whether the cost of doing this is low enough that we’d want to do it.

A Simple Tip to Improve Rust Program Speed

Wed, 06 Sep 2017 18:05:36 -0400

I am on vacation this week, so I decided to dig out a backtracking Sudoku solver I wrote last summer and see what I could do to improve its performance. A couple of changes to the data structures and the algorithm yieled an important speed-up. Previously, I had used a BTreeSet to store the list of candidates for a cell; I now use a u32 to represent a set of 9 elements. I used to compute the list of neighbors of a cell everytime it was needed; I now pre-compute the matrix of neighbors and avoid the unnecessary re-computations. With these changes, solving 1000 normal-difficulty problems went down from 33 seconds to 1.5 second; the time to solve 1000 very hard problems went down from 15 minutes to 40 seconds.

I wondered if there was a way that I could improve the performance of my code further. I looked for places in the code where I could avoid vector indexing—and in doing so, avoid Rust’s bounds checking—but I didn’t really feel like changing the way the whole program worked. Somewhat randomly, I stumbled on a forum thread about the implementation of count_ones() (popcount), and buried in the detailed answer of sorear is a command-line option to have rustc select the CPU that LLVM should target. By compiling my solver for my particular CPU platform (Intel Skylake), the time to solve the normal problems went down to 0.8 second and the time to solve the very hard problems went down to 20 seconds. Nearly a 2x speed-up by adding an extra flag, not a bad deal!

You can pass flags to rustc from Cargo by using the rustc sub-command and giving your extra arguments after a double-dash.

$ cargo build --release
   Compiling sudoku v0.1.0 (file:///home/vfoley/prog/rust/sudoku)
    Finished release [optimized + debuginfo] target(s) in 1.66 secs

$ time ./target/release/sudoku <very_hard.txt >/dev/null
real    0m36.542s
user    0m36.530s
sys     0m0.008s

$ cargo rustc --release -- -C target-cpu=skylake
   Compiling sudoku v0.1.0 (file:///home/vfoley/prog/rust/sudoku)
    Finished release [optimized + debuginfo] target(s) in 1.65 secs

$ time ./target/release/sudoku <very_hard.txt >/dev/null
real    0m17.473s
user    0m17.466s
sys     0m0.004s

Update: est31 on Reddit mentions that we can replace skylake with native to let the backend figure out the appropriate value.

Update 2: Manisheart on Reddit notes that it is better to use the RUSTFLAGS="-C target-cpu=native" environment variable instead of the rustc subcommand as it will apply the native compilation to all dependencies and not just the top crate.

My Review of The Talos Principle

Sun, 03 Sep 2017 14:02:42 -0400

Now that I have a slightly more powerful computer, I was finally able to play through The Talos Principle and its DLC, The Road to Gehenna. [Spoilers below]

The gameplay is mostly about solving problems by using some items. You have jammers that can open doors, disable bombs and guns; boxes to hold down buttons and to increase the height of an item; connectors that relay a signal from an emitter to receivers and can open doors; fans to elevate yourself or an object; etc. The game is divided in three worlds (A, B, C), each world is divided into 8 levels, and each level contains 3 to 5 puzzle zones. You must use the items in each zone to gain access to a sigil, a tetromino shape. When you have enough of those shapes, you can unlock new items, new zones, etc.

I really enjoyed how problem solving was introduced—there is no hand-holding, rather the game gives you simple puzzles to learn how to use an item and gradually increases the difficulty. Compared to games that give tutorials on every aspect of the gameplay, the process of discovery is satisfying and you feel less overwhelmed.

The plot of the game was very interesting as well. We learn through short audio clips, messages, and chat sessions that humanity is doomed. Global warming was thawed the polar cap and released a virus that affects apes (including humans). Faced with its extinction, some scientists join at the IAN institute to create the EL project, Eternal Life. They created an A.I. that would act and think like a human, thus ensuring that even though humans are gone, humanity remains. The whole game plays in a simulation where a God-like entity, Elohim, tells you to retreive the sigils, but not to escalate the great tower—and of course, we are definitely going up that tower! The goal of the simulation is to “evolve” a machine that can act and think like a human, even with our flaws and inconsistencies.

The philosophical questions addressed in the game are quite interesting. What is a person? What is conciousness? What is the qualitative difference between a tree, a frog, and a human? Can a machine be a person? Can a machine have a conciousness? Coincidentally, I am currently reading Douglas Hofstaeder’s Gödel, Escher, Bach which addresses such questions.

I enjoyed The Talos Principle greatly (I spent 60+ hours playing the game) and I look forward to The Talos Principle 2. If you play, I would advise that you refrain from looking up solutions online. I cheated this way a few times myself, and each time I regretted it.

Speedrunning a Compiler

Mon, 07 Aug 2017 20:00:00 -0400

At AdGear, the developers typically end their week at “Friday Not Friday”, a team activity from 4pm to 5pm. One person gives a presentation on any technical subject while the others listen and sip a beer to wind down before the week-end.

Last week, I gave a demonstration that I called “Speedrunning a Compiler”. I wanted to show my colleagues the basics of writing a compiler, and I thought it would be more instructive if they saw code rather than an abstract flowchart. Thus, I wrote a compiler for a very small language that I call “Minilang”. We went over the classic phases of a compiler: scanner, parser, typechecker, and code generator. I used Python as my implementation language; the code I wrote was not pretty and abused Python’s flexibility (mutation galore, dynamic typing, using strings and dicts instead of objects) and I generated MIPS assembly.

The resulting program is available on Github: https://github.com/gnuvince/fnf-vfoley/blob/master/compiler/compiler.py

I wrote the scanner and parser by hand partly because Minilang is simple enough, but mostly because I wanted the process of transforming text into an AST to not be mysterious and hidden behind tools like flex and bison (or an equivalent in Python, like PLY). Similarly, rather than generating C or JavaScript code, I used assembly because it seems more “real”: had I generated C, the question of how the code becomes executable would’ve been hidden by gcc or clang. Generating assembly code makes the mapping between high-level concepts and low-level constructs more explicit. Also, by generating very naïve assembly—I basically implemented a stack machine—I could show patterns of code that a simple peephole optimizer should recognize and replace.

Thanks to a couple practice runs during the week and some copy-pasting from my completed example, I was able to finish in about 50 minutes and hopefully to enlighten my colleagues a little bit.

Less intrusive color theme

Mon, 17 Apr 2017 12:51:03 -0400

The purpose of syntax highlighting is to draw attention to certain parts of the code: it helps us to identify when a comment or a string literal is not properly terminated, when a keyword is misspelled, or when an escape sequence is invalid. Lately, I find that editors colorize too many elements and use too many colors. (This is a recently-acquired annoyance; maybe age and worse eyesight have something to do with it?) When every syntactic construct is highlighted, none of them jumps out, and doesn’t that defeat the purpose?

Take a look at the screenshot of Sublime Text below; I find it difficult to focus on anything, too much is highlighted.

Some programmers have written about turning off syntax highlighting entirely, but this is too extreme for me. In the TeXBook (p.13), Don Knuth gives this advice to the reader about special typefaces:

Special fonts are effective for emphasis, but not for sustained reading; your eyes would tire if long portions of this manual were entirely set in a bold or slanted or italic face. Therefore roman type accounts for the bulk of most typeset material.

Too many colors—as is common in programming text editors—is not ideal, but neither is a competely barren buffer. A book that exclusively used Times New Roman 10pt would be extremely difficult to read. Moderation is key.

I tried many Emacs themes over the years, and we are lucky to have a large collection of well-designed themes. If you don’t mind a lot of colors, the material themes are quite nice.

However, since I couldn’t find one that used only a few colors, I decided to build my own, Purp. It’s not too hard to do in Emacs, and there are plenty of open-source themes that one can use for inspiration and education. In Purp, I tried to minimize the number of colors and the number of elements that are highlighted. You can see the result in the screenshots below.

I consider that the following elements are important enough to have their own color:

String and character literals: I want to know if they are properly terminated;
Comments: I want to find them quickly, and in languages that don’t have nested comments, syntax highlighting will show when you made an error;
Function names: a theme called Dracula gives function names a bright color and I find that it helps me identify where a function begins better than the indentation does, so I stole the idea;
Pre-processor commands: these are often not subject to the same rules as the base language and I wanted to see them easily;
Warnings: some language modes (e.g., Rust) use this face to identify dangerous constructs, so it’s not a bad idea to make sure they are visible.

Everything else uses the default typeface. I think the code looks much “calmer” than it did in the Sublime screenshot.

Purp isn’t publicly available yet, because there’s still a lot of work left to do. My installation of Emacs defines 418 faces (org mode has 75), but I’ve only defined colors for 25 of those.

Functional Queues

Thu, 13 Apr 2017 20:18:25 -0400

First-in first-out queues are a data structure used in many algorithms (including my favorite one—breadth-first search) and as a building block for complex functionality, the operations on queues should be as efficient as possible. This can be a bit of a challenge in a functional language: the bread-and-butter data structure of functional programming, the list, supports fast insertion and deletion at the head—ideal for enqueuing or dequeuing respectively—but operations at the other end of the list take O(n) time in the length of the list. Must we give up on our aspirations of purity and use mutable cells?

Thankfully, a simple and elegant solution to this problem exists. By using two lists—one for enqueuing items and one for dequeuing them—we can use the fast cons operation to push elements into one list and use pattern matching to pop them out of the other. If we try to dequeue an element when the pop list is empty and the push list isn’t, we reverse the push list, move the elements into the pop list, and continue.

Let’s see how we can implement this data type. We will use the OCaml language and we’ll start with a signature definition, i.e., the interface that our data type will have to support.

module type QUEUE = sig
  type 'a t
  val empty    : 'a t
  val is_empty : 'a t -> bool
  val enqueue  : 'a -> 'a t -> 'a t
  val dequeue  : 'a t -> ('a option * 'a t)
end

If you aren’t familiar with OCaml’s module system, this declaration says that QUEUE is a module signature. A module implements this signature if it defines a type 'a t (t is the name, 'a is a type parameter; in Java it would be written as T<A>) and the functions empty, is_empty, enqueue, and dequeue. A user will manipulate our queue module through this interface and will never be aware of its implementation.

We can then define a module called Queue (it’s common in OCaml, but not required, to use all caps to write the signature name and use Pascal-case for the implementation name).

module Queue : QUEUE = struct
  type 'a t = ('a list * 'a list)

  let empty = ([], [])

  let is_empty = function
    | ([], []) -> true
    | (_, _)   -> false

  let enqueue x (pop, push) =
    (pop, x :: push)

  let rec dequeue = function
    | ([], [])      -> (None, empty)
    | (x::xs, push) -> (Some x, (xs, push))
    | ([], push)    -> dequeue (List.rev push, [])
end

The type 'a t is a tuple of two lists of 'a. We create an empty queue by returning a tuple containing two empty lists and we verify that a queue is empty by pattern matching on the tuple and asserting that the two elements are the empty list. Both operations take constant time.

To enqueue an element, we use the cons operator (::) to add it to the front of the push list. This is also a constant time operation.

The dequeue function is the most interesting, and where the “magic” happens. If the pop list and the push list are empty, there are no elements to dequeue and we return None. If the pop list has an element in its pop list, we remove it from the list and return it. When only the push queue has elements, we reverse them (to make sure the least recent is at the head), make that the new pop queue and invoke dequeue again.

Now, this last operation appears to be O(n), but suppose we have n calls to enqueue followed by n calls to dequeue; the first dequeue takes time proportional to n (the list reversal), but all subsequent dequeues take constant time. This makes the operation O(1) amortised which is why it is acceptable in many applications.

And there you have it, a purely functional FIFO queue! For more purely functional data structures, please check out Purely Functional Data Structures by Chris Okasaki.

Being a T.A.

Fri, 07 Apr 2017 16:01:49 -0400

Being a teaching assistant is a job that most graduate students are asked—pressured even—to do. During my time at McGill, I was a T.A. three times: twice for the introduction to compiler design class (taught by my adviser), and once for a functional programming class. It was difficult, time-consuming, and not well paid, yet being a T.A. was the best job I’ve ever had.

I’ve had many bad T.A.’s when I was an undergrad; absent from the message boards, with a poor understanding of the subject, and confusing in their explanations, they make hard classes more difficult and bad classes unbearable. I was determined to be better and so I spent way more time on my duties than the 5 hours per week that the university paid me. I reviewed the material carefully, I practiced explaining tricky topics, and I attended lectures.

For the compiler class, my advisor allowed me to design a new term project. I selected a subset of the Go programming language (which we call Golite), I implemented a reference compiler, I wrote detailed specifications of the syntax and type system, and I prepared slides for class. This made me very familiar and sympathetic of the difficulties that students encountered. My fellow lab mates commented that I took the job too seriously, but I believe the results were worth the effort. Before, the compiler class had 15-16 students per semester; with the new Golite project, enrollment rose to 50! On a personal level, all the preparation work made me a better programmer, writer, and pedagogue. The real prize, though, came from my interactions with the students. Everyone in the class worked really hard on their project, and I was very pleased with the results. Astute students reported bugs in my reference implementation or inaccuracies in the specification; I learned Haskell tricks and C tools from kids who, at 20, were smarter than I will ever be; I received a huge hug from a student who broke in tears when she and I finally fixed a difficult bug in her project; I became friends with some of the students who frequently came to my office hours; and I found that nurturing students into capable craftsmen was an important responsability and it made me feel good about myself.

At the end of the term, students are asked to evaluate the class. I was so proud that in spite of the class’s difficulty and heavy workload they universally thought that the subjet was interesting and well-taught and that the project was fun, challenging, and practical.

I will leave you with some comments written about me in the students’ class evaluations. It’s a little “braggy”, but I was very moved by their kind words—they made the time, effort, and sweat that I invested worth it.

He is the best teaching assistant I have ever had. If there is an award for being a teaching assistant, give him two of those.

One of the best TA’s of McGill! Always available and willing to help during his office hours.

Best T.A. I’ve ever interacted with at McGill (and I’ve been here a while…)! Extremely helpful, quick to answer questions, very knowledgable, and most importantly cares about the students!

Vincent is the most dedicated TA I’ve ever seen. Knowledgeable, competent, ready and actually attends lectures. An excellent TA amongst the cesspool of horribles TAs in Mcgill CS.

Journalistic Writing Exercises 1

Mon, 27 Mar 2017 19:47:35 -0400

I recently purchased Journalistic Writing by Robert Knight. I’m not a journalist, but I’m sure some of his lessons transfer to other areas of non-fiction writing. At the end of the first chapter, the author presents the reader with a number of sentences that contain unneeded thats, there is’s, there are’s, etc.; the goal of the exercise is to rewrite the sentences without them. Here is my attempt.

There were half a dozen vintage airplanes standing in the foreground of the museum.

Half a dozen vintage airplanes stood in the foreground of the museum.

He said that there was a 13 percent decrease in drunk-driving citations on the state’s highways, roads and streets last year, but there was a substantial increase in the number of accidents blamed on drivers’ cell phone usage.

The number of drunk-driving citations on the state’s highways, roads and streets decreased by 13 percent last year, but the number of accidents blamed on drivers’ cell phone usage increased substantially.
He said the number of drunk-driving citations on the state’s highways, roads and streets decreased by 13 percent last year, while the number of accidents due to cell phone usage increased substantially.

There was a surgeon working intently on a prone figure.

A surgeon was working intently on a prone figure.

The president said that making the country greener is part of his plan to increase jobs.

Making the country greener is part of the president’s plan to increase jobs.
The president’s plan to increase jobs includes making the country greener.

Around the world there a millions of children that are starving.

Millions of children around the world are starving.

There are three things that can happen when you drive through a yellow light, and two of them are bad.

Three things can happen when you drive through a yellow light, two of them are bad.
Two of the three things that can happen when you drive through a yellow light are bad.

The police officiers said that they had a warrant and they wanted to inspect the house for weapons and drugs, but there was a pit bull preventing them from entering the front door.

The police officers had a warrant and wanted to inspect the house for weapons and drugs, but a pit bull prevented them from entering the front door.

There was Yogi Berra who said that “it ain’t over ’til it’s over”

Yogi Berra said: “it ain’t over ’til it’s over.”

She quoted her grandfather, who said that “many a damned fool went to college.”

“Many a damned fool went to college,” her grandfather used to say.

The ambassador said that she wanted to know why there were not enough limousines waiting for members of the delegation.

The ambassador wanted to know why there weren’t enough limousines for the members of the delegation.
Too few limousines were waiting for members of the delegation and the ambassador wanted to know why.

Libre de Penser

Sat, 18 Mar 2017 12:18:58 -0400

En août dernier, j’ai utilisé mon premier chèque de paie pour m’offrir un abonnement au journal Le Devoir. Ça fait maintenant 6 mois que je le lis plus-ou-moins quotidiennement (ces jours-ci, je passe plus de temps à lire sur la situation politique aux États-Unis) et c’est un 17$ par mois très bien investi selon moi: je suis mieux informé qu’avant et je peux lire sur mon téléphone, dans mon navigateur web, et même avec un lecteur PDF.

J’avais jamais auparavant lu un journal à tous les jours. En tant qu’enfant de l’Internet, je me tenais informé principalement via Slashdot ou Reddit et en cliquant sur les histoires qui m’intéressaient—donc j’étais pas si informé que ça, car je laissais de côté les trucs qui m’intéressaient moins. J’essaie maintenant de changer mon attitude et de lire même les articles qui traitent de sujets en dehors de mes intérêts. Il y a beaucoup d’histoires mineures qui prennent que 2 ou 3 paragraphes; elles sont rapides à lire, des fois sont très intéressantes, et sont quasiment jamais sur /r/quebec. Je m’efforce de les lire plus souvent.

Le fait de lire le journal à tous les jours permet aussi de suivre des histoires qui sont en cours. Par exemple, en septembre l’ONE avait prévu tenir des journées d’audience au Palais des Congrès de Montréal. Ces audiences ont été annulées suite à l’intervention de manifestants. Quelques jours plus tard, on apprenait que deux des commissaires de l’ONE avaient rencontré Jean Charest alors qu’il lobbyait pour TransCanada, le promoteur d’Énergie Est. Le conflit d’intérêt était évident, et chaque jour on en apprenait plus sur l’histoire, jusqu’à ce que les deux commissaires fautifs se récusent. Quand on peut suivre une histoire de son début jusqu’à son dénouement, ça nous engage plus.

Le Devoir offre plusieurs types d’abonnement; j’ai choisit l’abonnement électronique, car c’était le moins dispendieux et j’avais pas envie de recevoir un journal papier tous les matins. Avec mon abonnement j’ai accès à une version complète du journal en format PDF. C’est un fac-similé de la version papier et à ma connaissance, Le Devoir est le seul journal qui offre ce produit. Même des gros noms tels que La Presse, le New York Times, ou le Washington Post n’offrent pas un PDF de leur quotidien. C’est dommage, car c’est une façon formidable de lire les nouvelles. D’une part, on voit le journal tel qu’il est supposé être vu et le contenu ne change pas d’heure en heure comme sur le site web ou l’application mobile. D’une autre part, ça évite d’avoir des tas de papiers journaux à la maison. Avec un bon lecteur PDF (je recommende zathura), on peut facilement suivre des liens, faire des recherches dans le texte, etc.

L’application mobile—j’utilise celle sur Android—est aussi très bonne. On trouve toujours en haut de la page d’accueil “Quoi surveiller aujourd’hui”, un résumé en quelques mots de 4 ou 5 événements qui risquent de faire la manchette durant la journée. On peut “swiper” pour passer de nouvelle en nouvelle. On peut bien sûr configurer la grosseur des polices, activer des alertes courriel une nouvelle d’importance est publiée, sélectionner les sujets qui nous intéressent, etc. L’application s’intègre bien avec le reste de l’environnement: je peux facilement partager un article sur Facebook ou dans Signal, si j’ouvre un lien on me demande l’application que je veux utiliser (e.g., Le Devoir pour un lien interne, ou Brave pour un lien externe). L’application garde aussi une cache de quelques articles—pratique quand on lit dans le métro. Petit inconvénient: l’application me déconnecte après deux semaines.

Finalement, l’importance du journalisme n’a jamais été plus importante qu’elle l’est maintenant. Une population informée est absolument nécessaire une démocratie saine. Les révélations sur les liens étroits entre l’administration Trump et la Russie aux États-Unis et les problèmes de collusion au Québec montrent que les gens au pouvoir testent et dépassent les limites des règles établies. Le rôle de chien de garde des média n’a jamais été aussi primordial. Pour l’équivalent d’un trop gros repas chez McDo, je peux supporter nos journalistes et faire ma part pour que les politiciens et les dirigeants d’entreprises répondent de leurs actions.

Understanding Recursion

Sat, 11 Mar 2017 22:46:44 -0500

Recursion is my absolute favorite programming technique. It’s powerful, it’s elegant, and for many problems it yields simple and beautiful solutions. Unfortunately, recursion has a terrible reputation: new programmers find it hard and unnatural and some programmers with years of experience avoid it because of the trauma it incurred when they learned it. There are valid, practical reasons to not use recursion; fear and disdain are not good reasons.

Don’t go down the rabbit hole

To understand a recursive function, many programmers try to visualize the chain of calls and returns. The intent is laudable, but this approach inevitably leads to failure. Humans can only hold 7±2 things in their heads (Miller’s Law); to properly work out the mechanics of a recursive function, we’d have to remember the values of the local variables in every invocation, where recursive calls are made, etc. Very quickly, our brains would get overloaded, we wouldn’t know where we are or where we came from, and we’d be no closer to understanding the function.

The right way to write and understand recursive code is to assume that the function invocations return the right results, and to forget about the mechanics of recursion. It’s easier, faster, and we already do it all the time. When we invoke f in main, do we think about the stack frame of f? No! We rely on the name and/or documentation of the function to know what it does, we use its result, and we move on to the next thing. Why complicate our lives when we see a call to f inside f? The function call mechanism is the same, we can just use the result of the function and be on our way.

The following Python function computes the depth of a tree, i.e., the maximum length between the root node and a leaf node.

def depth(tree):
    if tree is None:
        return 0
    else:
        d_left = depth(tree.left)
        d_right = depth(tree.right)
        return 1 + max(d_left, d_right)

This is a well-formed recursive function: it has a base case, an inductive case, and the recursive calls are made on “smaller” inputs, i.e., they are getting closer to the base case. When the tree is empty (base case), we return zero. When the tree is not empty (inductive case), we compute the depths of the left and right sub-trees, we take the greatest of those two values, and we add one. That’s it! We understand this function and we haven’t given one iota of thought to stack frames.

The recipe

Base case: To write a recursive function, we first need to find the base cases. Most often there is only one, but there may be more. Typical base cases are: zero, the empty string, the empty list, the empty tree, null, etc. If the input to the function is the base case, we return the answer and we’re done.

Inductive case: After we’ve handled the base cases, it’s time to tackle all the other cases. This is where we do recursive calls. The argument of a recursive call needs to be “smaller”—“closer” to the base case than the current argument. If the function accepts an integer and the base case is zero, the typical argument to the recursive call is n-1; if the function accepts a tree, the typical argument is a sub-tree. The idea is that eventually a recursive call will hit the base case. Once you have the result for a smaller input, find how to use it to compute the result of the current input, return it, and you’re done.

How 'Learning Red Hat Linux' Changed My Life

Sun, 05 Mar 2017 18:00:00 -0500

Can you point to a book that changed your outlook on life and had a profound impact on the kind of person you are? For me, that book would be Learning Red Hat Linux.

In 1997, I was 14 and I decided that I wanted to be hacker. In order to look the part, I wanted to install Linux on my computer. This was early in Linux’s life and the distributions of the time didn’t have easy-to-use installers—we had to create root and boot floppies with a tool called rawrite.exe. And since I couldn’t read English very well back then and knew little about computers, I understood almost nothing that the Slackware 3.4 installation program asked. I failed to install Linux. Not a great start for a would-be hacker.

Fast-forward a couple of years, my English had improved and the installation program of Mandrake 6.1 was easy—no questions about the size of inodes—and I managed to get my first Linux desktop up and running! But now what? The CD sleeve gave instructions for the installation, but not for going online or manipulating files.

Somehow, I got my hands on a copy of O’Reilly’s Learning Red Hat Linux and the most amazing thing happened to me: as I read, I learned! During a visit at my grand-parents’, I read the chapter that explained how to use ls, cp, mv, and others to manipulate files and directories. When I got home, I ran to my PC and I was able to use the terminal and the commands I had learned to do meaningful work! After that, I was hooked: I learned about dozens of classic Unix commands, about the directory layout of Linux, and how to install new software with rpm. I even routinely reconfigured and recompiled my own kernel. At 16 years old, Learning Red Hat Linux made me realize that reading was the path to understanding, that I could learn any subject by myself if I just took the time to read about it.

I am 33 years old now, and I still read a lot. Whenever I want to learn something new, I head over to Amazon. I learned to play chess, poker, and backgammon from books; the original Programming Ruby AKA Pickaxe by Thomas and Hunt got me started in programming; these days I read books that give me tips to improve my writing (hopefully they make these articles less painful for you to read); when I got my first girlfriend, I even read A Complete Idiot’s Guide to Amazing Sex!

Learning Red Hat Linux is not a classic like Catcher in the Rye or How to Make Friends and Influence People, but its impact on my life has been immediate and lasting and I will be forever grateful for it.

Le Processus de Sélection d'un Jury

Sun, 26 Feb 2017 08:33:21 -0500

En septembre dernier, j’ai reçu une lettre qui me convoquait à la cours de Montréal pour servir comme candidat-juré. Je ne savais pas du tout comment le processus fonctionnait et quand je suis arrivé au Palais de Justice, j’ai réalisé que la majorité des gens présents étaient également ignorants du processus. J’ai pris des notes durant mes deux journées—que deux, car je n’ai pas été retenu comme juré—que je résume ici et qui, j’espère, vous aideront à mieux comprendre cet aspect de notre système judiciaire.

(Cet article décrit le processus tel qu’il s’est passé à la Cour Supérieure du Québec; il est possible que les choses se déroulent différement ailleur.)

Le déroulement de la journée

La chose la plus importante à savoir si vous êtes convoqué est que votre présence est obligatoire. Si vous ne vous présentez pas, un agent de la cours vous contactera pour connaître la raison de votre absence et vous risquez des représailles. Soyez au rendez-vous!

Voici une liste des étapes de la journées.

Confirmation de votre présence: à votre arrivée au Palais de Justice, vous (ainsi que des centaines d’autres candidats-jurés) présentez votre lettre ainsi qu’une pièce d’identité pour confirmer votre présence. On vous donne en retour une attestation de présence et un formulaire qui vous permet d’être remboursé pour votre déplacement (6.00$) et votre lunch (14.30$).
Accueil: le juge entre dans la salle d’audience, explique l’importance du rôle d’un jury dans notre système de justice, et réitère les informations qui se trouvent dans la convocation. Notamment, il explique les motifs d’exemptions. Certains motifs requièrent qu’une personne se récuse: une avocate ou un policier ne sont pas abilité à être jurés, et doivent demander une exemption. D’autres motifs peuvent être utilisés pour demander une exemption: les gens de 65 ans et plus ou les étudiants à temps plein peuvent, s’ils le désirent, demander une exemption, mais ce n’est pas obligatoire. “Je travaille!” n’est pas un motif valable d’exemption; d’ailleurs, il est interdit à votre employeur de vous congédier parce que vous agissez comme juré.
Présentation de la cause: le juge présente la cause, résume les chefs d’accusation, et lit la liste des avocats et témoins. Si vous connaissez personnellement une de ces personnes, vous devez obligatoirement demander une exemption.
Exemptions: le juge se retire dans une autre salle ou il recevera et écoutera les candidats qui croient avoir une raison valable pour obtenir une exemption. Le processus est long, car des centaines de gens vont demander une exemption. Pendant ces quelques heures, les autres candidats attendent.
Sélection du jury: une fois que les demandes d’exemptions ont été traité, une vingtaine de numéros sont pigés au hasard. Si le numéro qui figure sur votre convocation sort, vous passez devant le juge et les avocats qui décident s’ils vous acceptent dans le jury. Contrairement à ce que vous avez pu voir à la télé, on vous demande seulement votre nom et votre emploi; vous ne pouvez pas tenter d’être refusé en proférant des paroles grossières et racistes. Si la cause a été médiatisé, on peut vous demander si vous vous souvenez de ses détails. Si un jury (12 jurés, 2 suppléants) n’est pas complet après le premier groupe, un second groupe de 20 est pigé et passe devant le juge et les avocats, et ainsi de suite. Encore une fois, les candidats qui n’ont pas été pigé doivent attendre.
Retour le lendemain?: une fois qu’un jury est formé, le juge revient dans la salle d’audience (dans mon cas, il était rendu 16h30) pour en informer les candidats. Si une autre cause est prête à aller devant jury, les candidats doivent se représenter le lendemain afin de former un autre jury. Les procédures se déroulent plus rapidement, car il y a beaucoup moins de gens qui peuvent obtenir une exemption.
Libération: s’il n’y a pas d’autre cause prête à aller devant jury, le juge vous libère et vous pouvez retourner à vos activités quotidiennes. Par contre, vous restez à la disposition de la cours jusqu’à la fin de la session des assises.

Quoi faire quand on attend?

La majeure partie de la journée d’un candidat-juré est passée à attendre. Il y a du wi-fi dans le Palais de Justice, mais plutôt que d’aller sur Reddit ou Facebook, je vous suggère d’utiliser ce temps pour jaser avec d’autres candidats-jurés. Des petits groupes de conversation se forment naturellement, et vous allez certainement rencontrer des gens qui ont des histoires intéressantes à raconter.

J’ai eu l’occasion de parler avec un employé de l’Hôpital Ste-Justine qui argumentait avec passion pour un Québec indépendant. J’ai rencontré une employée de cafétéria dont la fille voulait devenir sexologue; la famille l’agaçait en changeant le poste de la télévision à des chaînes de porno lorsque elle venait au salon. Une dame en était à sa troisième convocation, mais malgré son désir de siéger comme jurée, elle n’avait jamais été retenue.

Et les gens qui ont déjà servi comme jurés ont tous le même commentaire: malgré les désagréments que cela peut causer, c’est une expérience unique, intéressante, et enrichissante.

A month of literate programming

Wed, 22 Feb 2017 14:51:42 -0500

During the month of December, I completed all the Advent of Code problems using a programming technique called Literate Programming (LP). My programs look like LaTeX documents, they mostly contain text that explains the task at hand and details my solution, and they are interspersed with Rust code that gradually builds the solution. The build.sh script transforms the LaTeX-like files into PDFs and executables.

The Teacher Mindset

We don’t typically write programs with readers in mind. We may use meaningful identifiers and split code into self-contained logical units, but how many of us leave detailed notes to help the next programmer understand our code? Beyond clean code considerations, a reader needs to understand the business needs that drive our design, the algorithms that we use, the data structures that underpin those algorithms, the invariants in our code, the system limitations that we must contend with, etc. The code alone cannot be expected to convey all that information.

A good literate programmer acts like a teacher. He presents the problem and the solution in a logical order; he writes his code and his prose carefully; he creates visual aids when they are appropriate; he refers to the literature for more discussion on topics of importance; he fosters understanding.

A great example of a literate program is JonesForth, a Forth implementation by Richard WM Jones. X86 assembler is not an easy language, and the implementation of a programming language can be scary, but the explanations in the comments give the reader all the information he needs to read and understand how the core of a Forth implementation works. Read the sections on threaded code—probably the most important sections of the program—and notice how the author gives us the “why” (saving memory) and uses clear diagrams to explain the “how”.

The Tools of LP

The mindset described in previous section is the most important ingredient for successful literate programming; if that mindset is achieved, some tools can enhance the end result further. Some popular LP systems are Don Knuth’s WEB and CWEB, and Norman Ramsey’s noweb (the tool I used for AoC). These systems mix a typesetting language (TeX or LaTeX) and a programming language (Pascal and C for WEB and CWEB, any language for noweb). The tools in these systems can give the author more liberty in the organization of his program and can improve the document that will be read.

The tangler is the tool that extracts the snippets of code in the document and reorders them in an order that the compiler can digest. The author of a literate program can use this capability to organize his code in an order that makes pedagogical sense. It is not limited to top-level declarations either; the tangler works with text, so even individual statements can be reordered if necessary. For example, the implementation of an algorithm can be presented and discussed before the associated data structures have been defined; or manipulate the content of a file before the file has been opened.

The weaver is the tool that creates the document that a programmer should read. A good weaver adds an index, inserts cross-references between the different snippets and functions, and adds syntax coloring to the code. I recommend reading a literate program in a hammock during the summer for a great time! Ulix, an LP implementation of Unix, shows how beautiful a literate program can look.

My Impressions of LP

Let’s talk about the the aspect of LP that I found the most lacking: the tools. Doing LP with noweb and Rust is spartan, and I would not recommend writing a large program this way. Emacs doesn’t offer syntax highlighting or auto-indenting for noweb programs, the Racer tool for Rust does not work outside a Cargo projects, and the line numbers given by the compiler do not match the lines in the original source file.

In spite of the lackluster tools, I enjoyed writing LP solutions for AoC very much. The overall process was far more difficult than just writing the program itself; I often had difficulty finding the right words to describe the details of my solution, especially when those details were unclear in my head. The act of writing down my thoughts was often helpful in clarifying my ideas. It’s also a great practice for would-be writers.

I should also note that if you feel tired, you should probably not write a literate program. I solved a problem at 11:30 pm and wrote “I’m really pissed off, so fuck explanations”. It’s not a great way to explain how the program works, but it sure is a great reminder that it’s always better to go to sleep than to write bad code and bad text.

I look forward to next November when I re-read the programs and see if I the text I wrote allows me to still understand them.

My Impressions of Final Fantasy IV Advance

Sat, 18 Feb 2017 21:57:13 -0500

Final Fantasy IV is a fantastic game, and Final Fantasy IV Advance is a fine port plagued by an unforgivable flaw.

Final Fantasy IV was released in North America as Final Fantasy II; the US port was modified to be an easier game. Enemies had fewer hit points and the characters had fewer abilities, like Cecil who didn’t have his Darkness attack.

Final Fantasy IV Advance is a port to the GameBoy Advance of Final Fantasy IV as it was in Japan. It features an improved translation (fans will be happy to know that “you spoony bard!” was retained), the simple 16-bit graphics are kept as-is, the music is slightly altered because of the different audio hardware, and a number of game-play bugs are fixed (note: play the European version, the US/Australia version has ATB-related bugs). The game has common quality-of-life features, such as dashing and quick saving. This version retains the role of the fat Chocobo as an item store; the GBA doesn’t lack the memory to store many items that the SNES did, but it’s cool that they kept a staple of the original.

On the surface, FF4A should have been the ultimate way to experience this franchise entry. Unfortunately, one issue makes playing this game a miserable experience: it has a ton of input lag.

When navigating the menus or entering actions during battles, players will notice a considerable delay between pressing a button and that command registering. For example, when healing my party with potions, there was a delay of a whole second between pressing the A button and the actual healing! But if this lag is annoying in the menu, it can be enraging during battle. Often, an input will be entirely discarded, and when that happens, a careless player may do an incorrect action (e.g., targetting a single enemy with magic rather than the whole formation, or selecting the wrong spell to cast.) The lag happens in both the menu and in battles (which account for the majority of game-play time) and this will certainly test the player’s nerves. I don’t understand how Square Enix’s QA could have missed such an obvious defect? And if they didn’t miss it, why was the game allowed to be released?

If you haven’t played Final Fantasy IV, try this version for 10-15 minutes; if you aren’t too annoyed with the input lag, then keep going, otherwise I recommend trying a fan translation ROM for the SNES.

Impressions on Final Fantasy I: Dawn of Souls

Sat, 11 Feb 2017 07:40:32 -0500

Following my playthrough of the original Final Fantasy for the NES, I played the remake for the Gameboy Advance, Final Fantasy I: Dawn of Souls. The game wasn’t very hard, and since I knew exactly what I had to do, I didn’t wander around aimless and was able to finish it in roughly nine hours.

The port to the GBA includes modernized graphics, sound, and music; modernized game play; quality of life additions; improved translation, longer dialogues with NPCs, and modernized names for magic spells; bug fixes; new side quests. But surely the greatest quality of Dawn of Souls is that, even with all the improvements, the developers and artists have managed to make it look and feel like the original Final Fantasy.

(Note: at the moment, I have not played Final Fantasy II: Dawn of Souls, so my comments are all based on my experience with Final Fantasy I: Dawn of Souls.)

Improved graphics and music

To my untrained eye, the graphics of Dawn of Souls look most like Final Fantasy V on the SNES; larger and more detailed than FFIV, smaller than FFVI. They capture the look-and-feel of the 16-bit era Final Fantasies (which, I must confess, is my favorite generation of games). Although there was nothing wrong with the graphics of the original NES and they had their unique charm, I appreciate the more detailed graphics: I find that they contribute to making the world richer and fuller. Look at the difference between the new Lich and the original Lich: she now looks bigger, badder, and more menacing.

Dawn of Souls animates magic spells more richly than the original game could; the visual effects for the high-level spells are beautiful, recognizable, and convey the great power of black and white magic. The spells used by enemies are also animated, and even when two enemies cast the same spells, the animations can differ. Take for instance the ice storm cast by both Winter Wolves and White Dragons: to convey that the Dragons do more damage, the whole screen becomes white with snow and ice, while the wolves’ attack only send a volley of hail toward your party.

The music composed by Nobuo Uematsu for the Final Fantasy series is legendary, and the music of the original game was no exception, giving us the original motifs for the Prelude, the Final Fantasy Theme, and the battle and victory themes. There is a detail I noticed in the soundtrack of the first game on the NES: none of the tracks seem to use the noise channel. Listen to the original battle theme and then compare with the Dawn of Souls battle theme; the drums in the latter make the track more energetic and better capture the excitement of battle. I do not know if the absence of the noise channel on the original version was due to a technical limitation (e.g., using it for attack sound effects) or if Uematsu had yet to figure out how to best use this extra channel. Regardless, the soundtrack of the new game (which was supervised by Uematsu himself) stays true to the original and adds a few track of its own: rather than fighting fiends with the standard battle theme, new tracks were composed for those important showdowns.

Modernized game play and quality of life

I mentioned in my impressions of FFI that I enjoyed the simpler and slower battle system; I’m happy the developers decided to keep the turn-based fighting system rather than replace it with a faster-moving ATB-like system. Like the NES version, the player selects an action for each character, and then those actions are performed, interspersed with enemy actions. Contrary to the NES version, however, the concept of ineffective attacks has been eliminated: if an enemy dies before it was attacked by one of your party member, a new enemy will automatically be re-targeted. This change certainly pleases many gamers, and makes the game more accessible to newer players. I mentioned before that ineffective attacks opened an extra tactical dimension, so it would have been cool if they could be re-enabled through an option. I believe that the PSX or PSP version offer such an option.

The magic system has been modernized and will feel more familiar to anyone who has played a RPG in the last 20 years. Rather than having a fixed amount of charges per spell level, casters have MP that they can use to cast any spell. Tents and cottages restore MP and Ether have been added to the game; caster characters can now use their magic repertoire more freely. A welcome improvement over the NES version involves multi-target spells: rather than acting on one character at a time, they act on all targets at once. I’m sure anyone how has casted FIR3 on 9 undead or had to wait for 6 WhWolf to cast IceStorm in the original NES version will appreciate the quicker fights. The spell names have also been changed: the nomenclature that is now pervasive in Final Fantasy has been back-ported. FIR3 is now Firaga, LIT2 is Thundara, etc.

Many enemies are now weaker, but the bosses are tougher. None of the fiends were push-overs. Chaos, who used to have 2000 HP in the original, now has a whooping 20,000 HP! This buff made for an epic final showdown: my fighting characters kept hitting while my white mage did everything in her power to keep them alive. It took a solid 8-9 minutes before my Light Warriors finally prevailed. That’s the way boss battles ought to be!

A number of quality of life improvements are also included in the game. A surprising feature for a Final Fantasy game is the ability to save anywhere, and not just when using tents or staying at an inn; maybe this was to allow players who play on the go (e.g., train or bus) to enjoy a few minutes of the game and then quickly save their progress. The game also has dashing, making trips into town quicker and descending into dungeons less of a drudgery. Buying items in shops can be done in bulk rather than buying one heal potion at a time (yes!) Similarly, when using an item or a spell on a character in the menu, pressing A multiple times uses that item/casts that spell multiple times; healing your party with items after a difficult battle is thus faster and less of a chore.

There’s also a point I didn’t think I’d ever need to mention in a Final Fantasy game: the controls are responsive. I am currently playing Final Fantasy IV Advance on the GBA and there is a lot of input lag! For instance, there is a 1 second delay between the time I press A to use a potion and the time that it actually applies! Fortunately, the controls in Final Fantasy Dawn of Souls are very responsive and you don’t lose an action because the game was busy at that time.

Fixed bugs

It’s no secret that the Final Fantasy games have been plagued with bugs: status ailments that didn’t work, stats that had no impact on your character, and even game crashing bugs (e.g., the draw ability in FFVI on the SNES). The original Final Fantasy was no exception: many spells didn’t work, the class ability of the thief was incorrectly programmed, etc. Final Fantasy Dawn of Souls fixes most of those bugs. The most disappointing aspect of gameplay bugs is when they take away tactical depth. I’m happy that I can rely on all my spells when making a decision: using the Temper spell is now a great way to boost the attacking abilities of your fighters.

Conclusion

I thought Final Fantasy I: Dawn of Souls was a terrific game, and I would not hesitate to recommend it. People completely new to the series will appreciate a straight-forward game with a simple system while veterans of the series will enjoy a solid port that brings back all the nostalgia and leaves many of the pain points in the past.

My impressions of Final Fantasy for the NES

Sat, 04 Feb 2017 14:09:13 -0500

A week ago, I finished my first playthrough of the original Final Fantasy for the NES. In the past, I had not been super interested in playing this episode since it was said to be (understandably) rough around the edges, being the first entry in the franchise. A few factors changed my mind: first, I heard great comments on the FF1 run at AGDQ 2017 by Feasel and Gyre and I was looking forward to watching a solid speedrun; second, I discovered that a couple of my coworkers were big fans of this game; lastly, I’ve been thinking that there is a void in my video game culture due to not having played (or even watched) the early episodes of Final Fantasy. For these reasons, I decided to go ahead and play the game.

What I liked

I was pleasantly surprised by the game, very much so! Despite its age, the limitations of the platform on which it ran, and that there were no other Final Fantasy games to turn to for inspiration, I thought that it was a much more solid entry than I’d given it credit for. Here are some of the features of the game, in no particular order, that I found genuinely interesting.

The combat system

By far my favorite feature of Final Fantasy is its combat system. Final Fantasy does not have the ATB combat system that became well known with the entries on the SNES and Playstation; before a turn, you select the actions for your characters, and then they execute those actions, with monster attacks interspersed in between. I’m not sure that I know how the turns are selected: purely random maybe?

I like this system, because it is slow, and it gives me time to think (roguelikers will appreciate, I’m sure). I can take my time to figure out what enemy formation I’m fighting, how much I think each of my attacks might do, and formulate an attack plan. If I make a mistake during action selection, I can simply press B and correct my error. This very forgiving mechanic can allow you to realize that there is a better plan of attack while you are selecting your commands.

Most people have heard of ineffective attacks: if an enemy dies, every command that targets this enemy afterwards will whiff rather than re-targeting another enemy. I had thought initially that this was going to be extremely tedious, but it turned out that this opens up an extra tactical dimension: should all your party members target one enemy to ensure it dies in this turn, or should you spread your attacks to end the fight more quickly? It also pushes you to estimate an enemy’s hit points (I got really good at keeping an upper-bound on max HP) and guess-timate if you need one, two, or more attacks to defeat it. It also makes fights more tense, because you can’t hold down A.

The party selection

As far as I know, Final Fantasy is unique within the franchise in allowing you to select the members of your party, and not allowing you to change (e.g., you can change that nature of an FF5 party by switching jobs). I finished my own playthrough with two fighters, one red mage, and one black mage. I had my share of difficulties since I did not have a white mage; for instance, in the Ice Cave, seeing mages was extremely scary, because I did not have a party member that could use the LIFE spell (and the game doesn’t have Phoenix Downs). The speed-run I mentioned earlier used two fighters and two red mages; my colleagues’ parties were four fighters and fighter, white mage, two black belts respectively. Some crazy gamers have won the game using a party of only white mages or black mages, etc. This gives the game a lot of replayability, and many ways to increase the difficulty of the game.

The plot and the game world

Final Fantasy is not a complicated game, it’s even simplistic at times. Yet, I thought it was charming, and because of the NES’s technical limitations, there was only one side quest (the Castle of Ordeals), and everything else was very straight-forward: go fetch a KEY to get TNT so that you can have access to later dungeons. The NPCs have limited dialogue, but they can still gives you some hints about where you need to go and what you need to do to progress; for instance, you need to bring SLAB to Dr. Unne to understand the Lufenian language, and an NPC gives you a hint that this is the person you need to go see, but they forgot his name. The game also introduces many elements that would become staples of the series: black and white magic, different races, Bahamut, crystals, an airship, etc.

The beastiary

Many of the infamous enemies of Final Fantasy were in the very first entry: cockatrices, mind flayers, huge worms, etc. And many of those enemies are scary: I don’t think I ever actually fought mind flayers, I just ran away and hoped they didn’t kill any of my party members. Similarly, a group of 9 cockatrices ambushing you is sure to make even the atheist players pray to some random deity to avoid getting stoned.

The dungeons

Although the NES’s technical capabilities did not allow a lot of variety and flexibility, each dungeon really felt distinct from the others. The colors, the enemies, and the music all contributed to create a unique ambiance for each place you visited.

The music

Nobuo Uematsu is now a legendary figure in the video game world: his work was instrumental to the series’ popularity, and today millions of gamers remember fondly the tunes he composed. Final Fantasy on the NES had a very basic sound system (see this video for a nice primer), yet Uematsu found a way to make that simple system sing. I especially enjoyed Matyoa’s theme, the battle song, and Gurgu Volcano.

What I liked less

Final Fantasy is a great game, but it is certainly not without its flaws. None of the flaws are fatal, but they can detract a little bit from the game.

Bugs galore

One of the most obvious problem with the game is the amount of bugs that you’ll see, even in a normal playthrough. There are many magic spells that never work (e.g. TMPR), the INT stat is bugged and doesn’t do anything, the thief’s special aptitude for running away doesn’t work, etc. There are no showstopper bugs as far as I know, but if a magic spell doesn’t seem to do anything, it’s very difficult to know if it’s because the enemy is resistant/immune to it, or because there is a programming problem. One often has to resort to FAQs to know if it’s worth buying a spell or not.

The magic system

I didn’t care too much for the magic system where you have a limited number of charges for a spell level, and when those are gone, you need to go to an inn or use a house to recharge them. When it’s your first playthrough, it’s difficult to know if you should keep those charges, or if it’s fine to use them, so I ended up using magic only when facing bosses, or when grinding. Speaking of grinding…

Some grinding required

I had to grind quite a few times during the game to reach a level that would allow me to complete a dungeon. Today, such a requirement is considered tedious, but it’s a bit more understandable for a game like Final Fantasy that was exploring the genre for the first time. I still did not enjoy spending 30 minutes here and there just fighting enemies to have a sufficiently high enough level to go through the next phase of the game.

Buying items

The game has very few consumables: for example, the only healing item is a HEAL potion which gives you back 30 HP. As we mentioned, spell charges are limited, and cannot be recharged in a dungeon, so HEAL potions are essential to keep your party alive. The problem is that you buy consumables one at a time. It takes quite a lot of time to get 99 of them. Fortunately, an emulator can lessen the burden by accelerating the speed of the game, and allowing you to keep the Turbo A button pressed.

Limited saving

Since you can only save you game in inns or on the overworld using a tent/cottage/house, going through a dungeon can be really nerve-racking. Although it adds to the tension of the game (which I liked), it’s really tempting to use save states in an emulator to ensure that you don’t lose all your progress should you have an unfortunate encounter.

Parting words

If you are a fan of the Final Fantasy series and you would like to discover the humble beginnings of one of the most popular and beloved series, I can heartily recommend that you give Final Fantasy a try. Desprite being primitive in comparison to recent entries, I’m sure you’ll appreciate how this little seed of a game sprouted a large and strong tree of a series.

In my next entry, I’ll discuss Final Fantasy: Dawn of Souls, the port of Final Fantasy and Final Fantasy II to the Gameboy Advance.

rebirth

Thu, 26 Jan 2017 22:03:24 -0500

I used to have a blog, a long time ago, where I’d sometimes write incoherent ramblings about coding, politics, and any other subject that struck my fancy. After a long hiatus, let’s get going again!

Reasonable Performance

Insights and understanding — AI won't take your job

Simple Precedence

My Impressions of Hare

The Tyranny of Existing Code

A skill programmers lack: practice

Tech companies making parental controls useless

You already know how to parse by hand

The data types

Utility methods

The parsing functions

Predicting the kind of data to read

Integers

Strings

Lists and dictionaries

Putting a bow on it

Pros and cons

Conclusion

Just: the most reductive word in software development

On 'Clean Code'

Rust 2022

Compile times

Debugging

More restraint in the implementation of features

Making Reasonable Use of Computer Resources: Part 2

Making Reasonable Use of Computer Resources

Lesser Known Coding Fonts

APL385 Unicode

Go Mono

Sudo

Monoid

Input Mono

My quick review of Cascadia Code

The good

The bad

Conclusions

Objective font criterias

License

Unicode coverage

Weights and slants

Customization

Active development

Simple usage of GNU parallel

Setup

Running one command on multiple inputs

Running multiple commands in parallel

Conclusion

Rust and the values of simplicity and ergonomics

Feeling more confident with property-testing

purp, an Emacs theme of few colors

Rust 2019

Improved compile times

A community effort to review crates

More “80% solutions”

vi, my favorite config-less editor

The evolution of performance in ppbert

Pre-historic times: nom

The first tagged releases

Eight months pass

How to alleviate the pain of Rust compile times

Use cargo check

Use sccache

Avoid LTO

Control your dependencies

Conclusion

Rust at Work—Two Paint Points

Why you should invest in an editor

What's next for ppbert?

Reflecting on ppbert

Quick reflection on first good Dwarf Fortress

Main mistakes

Small mistakes

Lessons learned

Dwarf Fortress and ivy-mode

What is a Type Error?

A Simple Tip to Improve Rust Program Speed

My Review of The Talos Principle

Speedrunning a Compiler

Less intrusive color theme

Functional Queues

Pre-historic times: `nom`

Use `cargo check`

Use `sccache`