Frank Mitchell

Dewdrop Farm - A Post-Mortem

2020-10-03T00:00:00-04:00

Dewdrop Farm was my entry into the 2020 js13kGames competition. It’s a tiny farming simulator, with an intentionally addictive compulsion loop.

You can play it online at 2020.js13kgames.com/entries/dewdrop-farm.

This is the 9^th year I’ve entered the competition, but the first time I’ve written a post-mortem about my game development process. I took the notes I had in my development diary, and flushed them out with descriptions of what I was thinking about, example code, and screenshots.

Week 1^st

The only real idea I had going into this, was “Make a game where the passage of time matters.” I usually pick some technical topic I want to learn about during the competition, and build a game around that. Having never done a game where the world runs independent of the player’s actions, I thought that would be an interesting thing to build.

I was playing Drop7 on my phone in the days leading up to the competition. The designer of that game, Frank Lantz, also made Universal Paperclips, an incremental game. So I played that, and read the paper Playing to Wait: A Taxonomy of Idle Games, and thought a lot about idle games. The 1.16 Nether update to Minecraft had recently been released, so I also had piglin bartering mechanics on my mind. I figured making a game with an economy would be interesting.

That combination of ideas reminded me of SimFarm, a game I loved as a kid. I got it in my head that I would build a mini version of Minicraft, just the farming and trading bits, rendering with a top down view so I didn’t have to also learn 3D maths while learning about game economies.

Googling Graphics

Searching for “farm” on OpenGameArt.org turned up a crop tileset by josehzz. That ended up defining the color palette I used for the game, AAP-64 by Adigun A. Polack, and also the look, 16×16 pixel sprites.

Once I found those crop graphics, I started thinking of Stardew Valley. Using Eric Barone’s game as a reference point got me through the first week of coding. The player has two tools, a hoe and water. Land needs to be tilled with the hoe before seeds can be planted. Crops grow at various speeds. Watering crops makes them grow faster. Land that’s watered dries out over time. Mature crops can be harvested and sold. Coins can be used to buy seeds to grow more crops.

Idling Away

I wanted Dewdrop Farm to run even when you weren’t actively playing it. So I used Jake Gordon’s article on render loops as a starting point. The game runs in fixed time, with an update taking place every 1/20^th of a second.

I spent a fair bit of time in those first few days figuring out how long each “day” on the farm should last. I thought a lot about this quote by Eric Barone about days in Stardew Valley:

“The psychology of it and how, by keeping the days short, it always felt like you had time fro ‘one more day,’ no matter how long you had been playing. Before you realized it, hours had passed.”

A day in Stardew Valley lasts 14 minutes, 20 seconds. I knew I didn’t want my days to last that long, but I started with that “14 minutes” as a reference, and keep tweaking the math until I found something that felt right.

const SEASONS_PER_YEAR = 4;
const DAYS_PER_SEASON = 28;
const SECONDS_PER_DAY = (14 * 60 * 3) / 28 / 4;

A day in Dewdrop Farm lasts 22.5 seconds. That’s long enough that when I watched the day counter tick, I felt like it wasn’t making progress. But it’s short enough that when I was actively doing things, like buying seeds, I found myself asking, “Where did the time go?” This also means that the growing season, from the start of spring to the start of winter, lasts about half an hour. That felt like a good length of time for a play session.

It was about this time that Rachel Wenitsky posted a thread on Twitter about walking simulators and other idle games. That got me thinking about compulsion loops. I knew I was building an adictive mechanic, so I wanted to create a very deliberate “It’s okay to take a break” point to balance that out. I decided winter would be a season where nothing grew. If you wanted to continue the game beyond one growing season, you’d need to take a 10 minute break.

Randomizing Growth

Even though the game ran at a fixed rate, I knew I didn’t want the crops to grow at a fixed rate. I wanted to recreate that Minecraft feel, where you come back to a field of wheat and find that some of it is ready to harvest and some of it needs a bit more time. At the same time, I wanted to keep the economic stability found in Stardew Valley. The player should be able to figure out that it takes four days for turnips to mature, that way they can decide if they have enough time left in the season to plant them.

What I ended up doing was randomzing the time in each day that each crop would grow. With each update, farm.time is incremented by 1/20^th of a second. So I first figure out how many seconds into the current day we are.

const day = Math.floor(farm.time / SECONDS_PER_DAY);
const farmTime = farm.time - (day * SECONDS_PER_DAY);

Then, if it’s a new day, I slice the 22.5 second day up into 36 time buckets, one for each plot of farmland. So the first time bucket is between 0 and 0.625 seconds, the second time bucket is between 0.625 seconds and 1.25 seconds, and so on. I randomly assign each plot of farmland to a time bucket, and pick a random time within that bucket for it.

if (growable.day !== day) {
  let plots = PRNG.shuffle(Farm.plots(farm));
  const dt = SECONDS_PER_DAY / plots.length;
  plots = plots.map((plot, index) => {
    const min = dt * (index + 0);
    const max = dt * (index + 1);
    const time = PRNG.between(min, max);

    return {
      ...plot,
      time,
      id: index,
    };
  });

  growable.day = day;
  growable.plots = plots;
}

During each update I check to see if a crop on a plot of land needs to grow. If it does, I dispatch a grow action. If it doesn’t, I save it to check again the next update.

const plots = [];
const remaining = [];

growable.plots.forEach((plot) => {
  if (plot.time <= farmTime && shouldGrow(plot)) {
    plots.push(plot);
  } else {
    remaining.push(plot);
  }
});

growable.plots = remaining;

plots.forEach(({row, col}) => {
  const growAction = {
    tool: 'grow',
    row,
    col,
  };

  farm = Rules.dispatch(farm, growAction);
});

That bit of code ended up working nicely for making crop growth feel randomly predictable. I resued it to also decide when farmland goes fallow, when sprinklers water crops, and when farmland dries out.

Looking Back

Here’s what the game looked like at the end of the first week.

Week 2^nd

One week into the competition I had a working game, but it wasn’t really fun. You could grow crops and sell them, but one spring harvest of turnips gave you more coins than you could reasonably spend. So I spent some time improving the graphics.

I styled the inventory bar, plus the Buy and Sell screens. A lot of the visual design for those was inspired by Stardew Valley. I added a hoe and watering can, plus an envelope to hold seeds, and sprinklers that automatically watered crops. It looked more polished, but it was still a very passive game.

Finding Fun

Dewdrop Farm is the first game I’ve made that didn’t start with me knowing what “Game Over” looked liked. There was just an never ending loop of crop gathering. Fortunately, Game Maker’s Toolkit posted a timely YouTube video about letting games design themselves. That got me thinking about what I liked about farming in SimFarm, and foraging in Stardew Valley, and mining in EVE Online. I realized I enjoyed the repetative task of gathering resouces, balanced with the threat of an external force taking those resources away.

The bunny in Dewdrop Farm is that external force. It hops around and undoes all your hard work. If it lands on a crop, it eats the crop. You can scare the bunny by poking it, making it hop toward the edge of the farm. If it’s on an edge when you poke it, it’ll hop off the farm, and you get a day without a bunny. But the bunny will be back the day after that.

That ended up being exactly the mechanic I needed. I found myself spending lots of time planting crops and nudging the bunny away from them instead of writing code.

Looking Back

Here’s what the game looked like at the end of the second week.

Week 3^rd

I wanted Dewdrop Farm to be a game you could pick up and discover how to play. The hoe is the active tool when you start. That lets you immediately click farmland and have something happen. The “Buy” and “Sell” buttons are labeled with verbs instead of the store and market nouns the code uses. That makes their actions obvious. My playtesters all figured things out without any gameplay hints, so I figure I did something right.

Managing Inventory

My original plan was that Dewdrop Farm wouldn’t have inventory limits. The four slots would be able to hold as many seeds as you could buy. Crops for sale would show up on the Sell screen, not in your inventory. But that was unsatisfying, because you didn’t see progress as you harvested crops. The graphics changed, but none of the visible numbers went up.

Also, I didn’t know what to do if the number of crops you had for sale or seeds you had in your inventory got really large. I needed to limit it so the counters on the seeds and crops wouldn’t obscure the image. So I capped it at 16 because that looked nice.

That ended up being a good number, because it means a field of seeds takes up a bit over half your inventory. So you need to make two trips to the Sell screen if you want a full field of fertilized crops. It also doesn’t divide evenly into 6, the length of a row on the farm, so you have to be a bit precise in your shopping if you want to plant rows of crops without leftover seeds.

Once I had inventory limits, I added logic to stop you from buying seeds or harvesting crops if they wouldn’t fit in your inventory. I didn’t want an accidental click to ruin hard work. The inventory fill algorithm comes from Minecraft. Slots are looped through from left to right. Items go in the first matching slot, or the first empty slot if there isn’t a matching slot. Items also sell from the inventory left to right.

To draw the Sell screen, I loop over the inventory items from left to right, and render a row for each. Like items stack, and when a row sells out, it vanishes. This allows you to sell an entire inventory of crops by continuously clicking the top button.

Clicking Around

One of the earliest mechanics I added was allowing you to click and drag to plant seeds, harvest crops, and water the farm. It makes playing on a desktop with a mouse a lot of fun. When I switched my testing to a laptop, I realized that clicking and dragging with a trackpad is quite a bit slower. So I added the keyboard shorcuts to make up for that. You can keep the cursor over the farm, and every clickable button you need is in that space.

I never figured out how to get touchmove events to register on a phone. It’s something I’d like to solve in a post competition version. Mobile play is very much an exercise in tapping, which has its own charm, but isn’t the experience I wanted.

Once I realized buying seeds and selling crops was going to involve a lot of clicking, I tried to find ways to make that enjoyable. I replayed Clicker Heroes to remmber why button clicking felt fun there. Having buttons that animate on mousedown and mouseup was key. Items also only buy or sell on mouseend when the cursor is on the button. So you can cancel a purchase or sale by moving the cursor off the button and letting go of the mouse.

I also made the coin graphic “animate” when items are bought and sold. I did that by fixing the left edge of the coin amount (the numbers) and bumping the coin graphic up against their right edge. Since the numbers in the Georgia font are variable width, the coin graphic moves a little bit back and forth when the numbers change.

Looking Back

Here’s what the game looked like at the end of the third week.

Week 4^th

Changing Seasons

This is my favorite piece of code in Dewdrop Farm.

.magic  { transition: background 22.5s linear; }
.spring { background: rgb(146, 220, 186); }
.summer { background: rgb(156, 219, 67); }
.fall   { background: rgb(233, 181, 163); }
.winter { background: rgb(185, 191, 251); }

Those are the CSS rules that make the background colors change over the course of the first day of each season. I picked colors that are close in luminance to each other, because I didn’t want the screen brightness to change abrubtly.

Sarah Mitchell gave me a great accessibility testing tip, which was to play the game on my phone with the screen brightness turned all the way down. She prompted me to make the water texture brighter and to recolor all the seeds so they showed more contrast against the farmland.

I wanted an Animal Crossing: New Horizons kind of look to the water, droplets sprinkled on top of the crops. So they render over everything to keep them visible, and they’re randomly rotated to try and hide the fact there’s only one water graphic.

Optimizing Graphics

You’ll find some odd little experiments if you dig into the code, like png2code.js. That was my attempt to convert a PNG image into a run length encoded data format, that’s unpacked and rendered as a SVG inside the browser. For a long time during development, the farmland and crop graphics took up a ton of space. So I tried a bunch of things to try and shrink them down.

I started by putting all the graphics in a single image. Then I culled crops, cutting them down from the original 20 to a final 6. To keep some variety, I changed the seasons they’re buyable and growable in. Each growing season has one crop that’s unique to it, one crop that also appears in the season before it, and one crop that also appears in the season after it. That gave me three crops per season.

Each crop has a seed image, four growing stage images, and a harvested image. So six images per crop. But the wildflowers you can buy in winter all use the same seed and first three growing stage images. The last growing stage image is unique for each (tulip, rose, sunflower), and I reuse it for their harvested image. That let me add three more crops to the farm with only seven images.

Even those tricks weren’t enough though. What finally ended up working was three things. First, I manually repacked the images into a single vertical strip, placing similar looking images next to each other. That let the PNG compression algorithm save some bytes, because the pixels didn’t change much from one row to the next. A tall vertical strip ended up compressing a little bit better than a wide horizontal strip, and was also easier to edit. Second, I ran the image through ImageOptim.

I’m not sure what magic ImageOptim is doing under the covers, but it ended up compressing my image better than anything else I found. I wish there was a command line version, because it would be a lovely thing to bake into a build system.

The third thing I did was embed the image in a data URI in the CSS, and embed the CSS in the HTML. Those three thigns ended up saving 837 bytes. That gave me enough space to add the graphics and logic for the cow.

In many ways, buying the cow is very much the end game. At 150,000 coins, it looks like it will take a long time to get there. But eggplants have a 70 coin profit, and they show up in the fall next to the cow. My hope is that’s enough of a clue for players to figure out that you can keep playing through winter and get enough coins to buy the cow.

Looking Back

Here’s what the game looked like at the end of the fourth week.

Closing Thoughts

There’s a lot more I could say (and maybe will!) about Dewdrop Farm, but I’ve tried to limit this post-mortem to things that are unique to that game. All the mechanics of making a video game, putting graphics on the screen and changing them based on player input, are stuff I’ve written about in How to Make a Video Game. It’s the tutorial I wanted when I started making JavaScript games ten years ago. Give it a read, and maybe I’ll get to play your game in the next js13kGames competition.

Until then, happy farming!

Who else wants narrative mechanics in RPGs?

2019-11-17T00:00:00-05:00

Beyond the Wall and Other Adventures has three types of tests. There are ability checks, saving throws, and combat rolls. My previous post was about changing the mechanics of ability checks. I wanted to have all three tests use the same mechanic. Roll a twenty sided dice, add bonuses, and compare with a target value.

My reason for wanting unified dice mechanics is that it makes the game less confusing. I got confused while reading the rules. I figured if it confused me, anyone I was playing with was going to get confused as well. Now I’m reading through the rules again. I’m realizing that part of my confusion comes from the language used to ask for tests.

Ability checks, as a test of player skill, use this kind of language.

Player: “I try to jump across the stream.”
Gamemaster: “Roll to check dexterity.”

So rolling the dice is a test to determine something about the player’s skill. Am I skilled enough to jump across the stream or do I fall in the water? It’s a test every player that wants to jump the stream must pass on their own. We can’t use a group skill check, because my stream jumping skill won’t help you.

But ability checks can also be a test to determine something about the environment. Those ability checks use this kind of language.

Player: “I try to jump across the stream.”
Gamemaster: “Roll to see how slippery the bank is.”

This opens up interesting possibilities. Say I’ve got a dexterity of 16 and I roll an 18. The stream bank is more slippery than my dexterity can overcome, so I land in the water. Other players can decide they’re going to jump over a different part of the stream. They make their own ability checks, and hopefully find a less slippery spot. Of if I roll a 7, the other players may decide to follow me, since the bank isn’t very slippery there.

This feels like a collaboration. Everyone’s rolling to figure stuff out about the world we’re playing in. It’s also not going to break the game, since I’m not changing how ability checks are used or how they map to bonuses. Rolling under for ability checks now feels like a natural thing. I know what my character’s ability score is and what they can do. I’m rolling for the unknown, to see what the environment does.

Saving throws have a similar linguistic duality to ability checks. Suppose a mage casts entanglement on some vines near the player. As a test of player skill, the saving throw for that spell uses this kind of language.

Player: “I try to escape the vines. ”
Gamemaster: “Roll to save versus spell.”

But saving throws can also be a test to determine something about the environment. The entanglement saving throw could use this kind of language instead.

Player: “I try to escape the vines.”
Gmaemaster: “Roll to see how quickly the vines grow.”

If the player is a level 2 rouge, their save versus spell is 15. So they try to roll greater than or equal to 15 on a twenty side dice. If they succeed, the vines grow slowly and they may be able to escape. But if they fail, the vines grow quickly and they become entangled.

I like this idea of ability checks and saving throws fleshing out details about the world. It’d be nice to keep the mechanics consistent with the language. If I’m more resistant to spells, my save versus spell value should be higher. Fortunately, that’s an easy thing to change by rewriting the saving throw tables for each class.

When rolling a twenty sided dice, there are six rolls that succeed with a save versus spell of 15. So rolling under on a 6 is equivalent to rolling over on a 15. Here’s the complete conversion table from 1 to 20.

Roll Over	Roll Under
1	20
2	19
3	18
4	17
5	16
6	15
7	14
8	13
9	12
10	11
11	10
12	9
13	8
14	7
15	6
16	5
17	4
18	3
19	2
20	1

I did those conversions for the warrior, rogue, and mage, and came up with new tables. My copy of Beyond the Wall shows the rogue’s polymorph save going from 12 to 13 between levels 2 and 3. I assume that’s a misprint, so I’ve corrected it in the table below.

The Warrior

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	4	6	4	5
2	7	4	6	4	5
3	8	5	7	7	6
4	8	5	7	7	6
5	10	7	9	9	8
6	10	7	9	9	8
7	11	8	10	10	9
8	11	8	10	10	9
9	13	10	12	12	11
10	13	10	12	12	11

The Rogue

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	8	5	8	6	7
2	8	5	8	6	7
3	8	5	9	6	7
4	8	5	9	6	7
5	9	6	10	8	9
6	9	6	10	8	9
7	9	6	10	8	9
8	9	6	10	8	9
9	10	7	12	10	11
10	10	7	12	10	11

The Mage

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	6	8	9	10
2	7	6	8	9	10
3	7	6	8	9	10
4	7	6	8	9	10
5	7	6	8	9	10
6	8	8	10	11	12
7	8	8	10	11	12
8	8	8	10	11	12
9	8	8	10	11	12
10	8	8	10	11	12

Who else wants unified mechanics in RPGs?

2019-11-10T00:00:00-05:00

I’m thinking about running a Beyond the Wall and Other Adventures game, so I sat down and read through the rules. Everything felt very familiar. Characters can be one of three types: warriors, rogues, or mages. Levels range from 1^st (a bit above ordinay) to 10^th (heroes of the land). Ability scores range from 1 to 19, and each character has six abilities: strength, dexterity, constitution, intelligence, wisdom, and charisma.

I use abilities by rolling a twenty sided dice (1d20) and trying to get less than or equal to my ability score. So if want to jump over a stream, and my dexterity is 16, rolling a 19 means I fall in the water.

I also get to use abilities in combat. Ability scores map to bonuses. I attack by rolling a twenty sided dice, adding my bonuses, and trying to get greater than or equal to my opponent’s armor. So if I want to shoot a bow at a cockatrice (14 armor), and I’m a 1^st level rogue (+0 bonus) whose dexterity is 16 (+2 bonus), rolling a 19 means my I hit the cockatrice.

Wait, what? Rolling a 19 when shooting a bow is a success, but rolling a 19 when jumping is a failure. I find that confusing. I kind of expect that when I’m rolling a twenty sided dice, 1 is going to be a failure and 20 is going to be a success, and the stuff in the middle is going to be a probability curve with bigger numbers meaning “more likely to be a success”.

New player experiences matter. I was teaching a friend Gloomhaven, and we talked about how lowest initiative goes first, and they said, “That doesn’t make any sense. If I have more initiative, I should go first.” They were right. I’ve played Gloomhaven so much that I’ve internalized the “lower initiative goes first” rule, so I didn’t bother to question it.

But I haven’t played Beyond the Wall, yet. So I’m going to question these “roll low for ability checks; roll high for combat” rules, and see how I might change them.

What does it mean, mathematically, for my character to have a dexterity of 16? It means that I’m going to succeed at 80% of my dexterity checks, because 16 divided by 20 is 0.8. So the probability of any ability check succeeding is the value of that check divided by 20.

P(A) = A 20

Here’s a table of ability scores, from 1 to 19, as a percentage success rate.

Ability Score	Success Rate
1	5%
2	10%
3	15%
4	20%
5	25%
6	30%
7	35%
8	40%
9	45%
10	50%
11	55%
12	60%
13	65%
14	70%
15	75%
16	80%
17	85%
18	90%
19	95%

If I want to change the math for ability scores, I can’t break this table. Having a dexterity of 16 should always mean I have an 80% chance of success on a dexterity ability check.

I want ability checks to feel simlar to combat rolls. I roll a twenty sided dice, add my ability score, and check if the total is greater than or equal to a target number. If it is, I succeed; otherwise, I fail.

1d20 + A ≥ ?

What should that target number be? The design notes in Ben Milton’s game Knave, have some clues.

Requiring saves to exceed 15 means that new PCs have around a 25% chance of success, while level 10 characters have around a 75% chance of success, since ability bonuses can get up to +10 by level 10. This reflects the general pattern found in the save mechanics of early D&D.

Ability bonuses in Knave range from 0 to 6. Characters make saving throws by rolling a twenty sided dice, adding their ability bonus, and checking to see if the total is greater than 15. If it is, they succeed; otherwise, they fail. That’s pretty much what I want, so I need some way to map the ability scores in Beyond the Wall to the ability bonuses in Knave.

The character playbooks in Beyond the Wall start most ability scores at 8, a 40% success rate. So I can subtract 8 from an ability score to get something that fits in the smaller range of the Knave system.

1d20 + A - 8 > ?

Witht this formula, a target value of 15 gives a new character a 25% success rate. The success rate is the number of ways to roll a twenty sided dice and get greater than the target value. Mathematically, that’s 20 minus the target value, divided by 20.

S = 20 - T 20

Because I don’t want to break Beyond the Wall, I need to find a target value that keeps the success rate for new characters stays at 40%.

0.4 = 20 - T 20

20 × 0.4 = 20 - T

8 = 20 - T

8 + T = 20

T = 20 - 8

T = 12

I can use a target value of 12 for ability chceks. Since I subtracted 8 to move ability scores into a Knave range, I can add 8 to move them back to a Beyond the Wall range. Finally, I can change “greater than” to “greater than or equal”, so ability checks follow the same “my character wins ties” rule as combat rolls.

1d20 + A - 8 > 12

1d20 + A > 12 + 8

1d20 + A > 20

1d20 + A ≥ 21

So I can use abilities in Beyond the Wall by rolling a twenty sided dice, adding my ability score, and trying to get greater than or equal to 21.

Does the success rate math still work? If I’ve got a dexterity of 16, that should be an 80% success rate. If I roll a 5 or more, I’ll pass the skill check. There are sixteen ways to do that with a twenty sided dice, and 16 divided by 20 is 0.8. It works!

I’m not the first person to figure this out. As Saelorn points out in an En World thread about dice mechanics

If you already have the concept of DCs [difficulty checks] in place for things like attack rolls and saving throws, then you could use a mechanic of d20 + stat score against a constant DC 21, and it would give the exact same distribution.

Now all my checks can use a unified mechanic: roll a twenty sided dice, add bonuses, and compare with a target value. An unmodified roll of 20 is always a success, while a 1 is always a failure.

Algorithms matter on the mobile web

2019-06-02T00:00:00-04:00

Leo Fabrikant’s article on optimizing the performance of a React autocomplete form is worth reading. It covers performance profiling, async rendering, and multi-threading with Web Workers. I loved that he outlined the conditions that pushed him toward focusing on optimizing the rendering pipeline. What we know creates the set of spaces where we look for solutions.

The search algorithm library had painfully long search times as the length of the search term got longer…I don’t know if the library I chose for the search algorithm was bad or if this was an inevitability of any “fuzzy” search algorithm. But thankfully, I didn’t bother trying to find alternatives.

When I built my word search game, I spent a fair bit of time learning how to store and search strings efficiently. So when I read Leo’s article, I thought about data structures like BK-trees and algorithms for finding the Levenshtein distance. My experience says long search terms shouldn’t mean long search times. So let’s see if we can build a better search engine.

Sometimes you need to DIY

We’ll need some strings to work with that satisfy the original requirements.

Here is a data set retrieved from a backend. It contains 13,000 items with very long, wordy names (Scientific Organizations). Make a search bar with an auto-suggest using this data.

I don’t have a list of very long, wordy scientific organizations handy. However, the Free Music Archive has 19,212 tracks in it with names of five or more words. I figure that’s a pretty equivalent data set.

We’ll also want some JavaScript to benchamrk different fuzzy search algorithms. Let’s assumes a worst case scenario, where the user’s typed out the entire song name, and the song they’re looking for isn’t in the database.

const query = 'Where the Streets Have No Name';

const data = [...new Set(require('./tracks.json'))];
const tracks = data.filter(track => {
  return track.split(/\s+/).length >= 5;
});

const start = Date.now();

const matches = tracks.map(track => {
  const score = compare(query, track);

  return {score, track};
}).sort((a, b) => {
  return a.score - b.score;
}).slice(0, 10)
.map(info => info.track);

const elapsed = Date.now() - start;

console.log(`
${matches.join('\n')}

Searched ${tracks.length} tracks in ${elapsed} milliseconds
`);

We need a comparison function that scores two strings based on how similar they are. To make it easy to sort results, we’ll say that a smaller score means the strings are more similar. The Levenshtein distance metric is the reference measurement for string similarity. It counts the number of edits it would take to make two strings identical.

Here’s a memoized recursive implementation.

function compare(s, t, memo = {}) {
  const args = [s, t];

  if (args in memo) {
    return memo[args];
  }

  if (!s) {
    memo[args] = t.length;
    return t.length;
  }

  if (!t) {
    memo[args] = s.length;
    return s.length;
  }

  const snext = s.slice(1);
  const tnext = t.slice(1);

  const cost = s[0] !== t[0];
  const delCost = compare(snext, t, memo) + 1;
  const insCost = compare(s, tnext, memo) + 1;
  const subCost = compare(snext, tnext, memo) + cost;
  const minCost = Math.min(delCost, insCost, subCost);

  memo[args] = minCost;
  return minCost;
}

I ended up needing to memoize it, because the non-memoized version took too long. As it is, even the memoized version takes about two minutes to find matches.

Where Childrens Have a Place
Where the Walls Have a Soul
When the Guests Have Left
The Extra Party Has No Name
So, What If I Have No Name?
The One With No Name
Where The Land Meets The Sea
When the Lights Came On
This Game Has No Name
Down the Streets (Life Beyond)

Searched 19212 tracks in 122963 milliseconds

That’s not something I’d want to use in an autocomplete form. My general rule of thumb for UI responsiveness is that anything more than a third of a second (about 300 milliseconds) is too long. Fortunately, the Levenshtein distance metric has an iterative implementation that avoids the recursion and memoization.

function compare(s, t) {
  let v0 = [];
  let v1 = [];

  for (let i = 0; i <= t.length; i += 1) {
    v0[i] = i;
  }

  for (let i = 0; i < s.length; i += 1) {
    v1[0] = i + 1;

    for (let j = 0; j < t.length; j += 1) {
      const delCost = v0[j + 1] + 1;
      const insCost = v1[j] + 1;
      const subCost = v0[j] + (s[i] !== t[j]);
      v1[j + 1] = Math.min(delCost, insCost, subCost);
    }

    [v0, v1] = [v1, v0];
  }

  return v0[t.length];
}

That takes about a third of a second and finds the same matches.

Where Childrens Have a Place
Where the Walls Have a Soul
When the Guests Have Left
The Extra Party Has No Name
So, What If I Have No Name?
The One With No Name
Where The Land Meets The Sea
When the Lights Came On
This Game Has No Name
Down the Streets (Life Beyond)

Searched 19212 tracks in 322 milliseconds

Can we do any better? Sure! I turns out the Sorensen-Dice coefficient of the sets of bigrams in two strings makes a pretty good fuzzy match. We have to negate the score though, because a larger value means the strings are more similar.

function bigrams(string) {
  const result = [];

  for (let i = 0; i < string.length - 1; i += 1) {
    result.push(string.slice(i, i + 2));
  }

  return result;
}

function compare(s, t) {
  const sGrams = bigrams(s);
  const tGrams = bigrams(t);

  const hits = sGrams.filter(n => {
    return tGrams.includes(n);
  }).length;

  const total = sGrams.length + tGrams.length;
  const score = (hits * 2) / total;

  return -score;
}

We get a different set of songs, which makes sense, because we changed the algorithm. I’m not sure if the 64 millisecond speed improvement is noise or not. I’d need to plug it into a more robust benchmarking tool (like Benchmark.js) to measure that.

No Secrets In the H4C
The Streets of New York
Where the Walls Have a Soul
Liberty Is In the Street
The Waves Call Her Name
The Extra Party Has No Name
Nameless: the Hackers Title Screen
There is Nothing to Fear
Down the Streets (Life Beyond)
So, What If I Have No Name?

Searched 19212 tracks in 258 milliseconds

What about the quality of the results? The Sorensen-Dice coefficient feels like it does a better job of finding relevant songs when the words in the query are out of oder. The Levenshtein distance feels like it does a better job when you know the name of the song you want.

Where do we go from here?

The more experiences we have, the better a chance we give ourselves of finding solutions to problems and answers to questions. Leo’s article helped me learn how to profile and fix UI blocking issues. Writing this helped me learn that fuzzy search algorithms aren’t just about speed. Normalization and measuring similarity vs. edit distance changes the quality of the results. You need to understand the your use cases first, and pick an algorithm that supports them.

I’ll probably just use Kiro Risk’s excellent Fuse.js library if I need a fuzzy search engine in the future. It uses the bitap algorithm, so it’ll probably be fast enough. Plus, that’s the same algorithm used in agrep, so it’s probably a good fit for most text.

Probably.

Addendum

For completeness, here’s the Ruby code I used to turn the raw_tracks.csv file from the fma_metadata.zip archive into a JSON list of track names.

require 'csv'
require 'json'

$stdout.sync = true

csv = CSV.read(ARGV[0], headers: true)
tracks = csv.map { |row| row['track_title'].strip }
puts tracks.to_json

Install TQSL v2.3.1 on Raspian Jessie

2018-01-01T00:00:00-05:00

With the International Grid Chase 2018 contest starting, I wanted to make sure I had TQSL working on the computer in the lab. TQSL is an application used to sign and upload contacts to Logbook of the World so they count for the contest. Since TQSL doesn’t have pre-built binaries for Linux, and the lab computer is a Raspberry Pi 3, I built the code from source.

The first thing to do is download the TQSL source.

curl -s https://www.arrl.org/files/file/LoTW%20Instructions/tqsl-latest.tar.gz > tqsl-latest.tar.gz

I couldn’t find a checksum for the source package on the ARRL site. If you downloaded version 2.3.1, you can check that you got the same package I did. If you grabbed a later version, your checksum won’t match.

shasum -a 256 tqsl-latest.tar.gz
bbbf7b4917384968a5f33907b637d3d9bff44b45a29ec5210894dfaa68a49281

Next unpack the tarball and read the installation instructions.

tar -zxvf tqsl-latest.tar.gz
cd tqsl-2.3.1
cat INSTALL

The installation instructions are well written. They include a list of dependencies (with versions), and a set of commands to run. The build system uses CMake, so install that next.

sudo apt-get install cmake

TQSL v2.3.1 depends on OpenSSL, expat, zlib, Berkeley DB, wxWidgets, and curl. You need the development versions, the ones with libraries and headers. Development package names typically include “lib” in the name and end in “-dev”. Run these commands to find the OpenSSL related development packages.

apt-cache search ssl | grep -e "-dev"

Replace “ssl” in the search command above with “expat”, “zlib”, “db”, “wx” and “curl” to find the rest of the packages. Or just run the commands below to get them all installed.

sudo apt-get install libssl-dev
sudo apt-get install libexpat1-dev
sudo apt-get install zlib1g-dev
sudo apt-get install libdb-dev
sudo apt-get install libwxgtk3.0-dev
sudo apt-get install libcurl4-openssl-dev

By default, the TQSL build process creates a shared library and installs it in the /usr/local/lib$(LIB_SUFFIX)/ directory. Shared libraries on Raspbian are usually in the /usr/local/lib/ directory. Set an empty LIB_SUFFIX environment variable to make the build system puts things in the right place.

export LIB_SUFFIX=""

Finally, run the commands from the installation instructions to compile and install the TQSL library and application.

cmake .
make
sudo make install

So far, I’ve been starting TQSL by running “tqsl” from the command line. I’d like to put a shortcut into Menu > Hamradio so it’s easier to start. I’ll update this post if and when I figure out how to do that.

Use promises in Node.js and avoid callback hell

2017-06-04T00:00:00-04:00

Callback hell is an easy JavaScript anti-pattern to recognize. There will be a trail of closing braces, parenthesis, and semicolons at the end of your code. For example, here’s a rsync like program written with callbacks.

#!/usr/bin/env node

const fs = require('fs');

const file1 = process.argv[2];
const file2 = process.argv[3];

fs.stat(file1, (err, stats1) => {
  if (err) {
    stats1 = {size: 0, mtime: Date.now()};
  }

  fs.stat(file2, (err, stats2) => {
    if (err) {
      stats2 = {size: 0, mtime: Date.now()};
    }

    if (stats1.size === stats2.size) {
      const time1 = stats1.mtime.getTime();
      const time2 = stats2.mtime.getTime();

      if (time1 <= time2) {
        console.log('sent 0 bytes');
        process.exit(0);
      }
    }

    fs.readFile(file1, (err, data) => {
      if (err) {
        console.error(err);
        process.exit(1);
      }

      fs.writeFile(file2, data, (err) => {
        if (err) {
          console.error(err);
          process.exit(1);
        }

        console.log(`sent ${data.size} bytes`);
      });
    });
  });
});

Notice all the }); characters at the end? That’s callback hell. This code has four logging statements, three early returns, and a cascade of nested functions. Lucky for us, Node supports Promise objects, so there’s a way to refactor this.

The first thing we can tackle is the fs.writeFile call. We log the number of bytes written if the write is successful. We log an error and exit the program if the write fails. This maps nicely onto promises.

function writeFile(path, data) {
  return new Promise((resolve, reject) => {
    fs.writeFile(path, data, (err) => {
      if (err) {
        return reject(err);
      }
      resolve(data.length);
    });
  });
}

Here’s what the business logic of our program looks like when we rewrite it to use our new writeFile function.

fs.stat(file1, (err, stats1) => {
  if (err) {
    stats1 = {size: 0, mtime: Date.now()};
  }

  fs.stat(file2, (err, stats2) => {
    if (err) {
      stats2 = {size: 0, mtime: Date.now()};
    }

    if (stats1.size === stats2.size) {
      const time1 = stats1.mtime.getTime();
      const time2 = stats2.mtime.getTime();

      if (time1 <= time2) {
        console.log('sent 0 bytes');
        process.exit(0);
      }
    }

    fs.readFile(file1, (err, data) => {
      if (err) {
        console.error(err);
        process.exit(1);
      }

      writeFile(file2, data).then(size => {
        console.log(`sent ${size} bytes`);
      }).catch(err => {
        console.log(err);
        process.exit(1);
      });
    });
  });
});

Notice that we didn’t remove any levels of function nesting. That’s because promises take then and catch callbacks. So we’ll always have at least one level of nesting.

To remove those }); bits at the end, we need to take the next step and rewrite the fs.readFile call. Since we return the data if the read’s successful, and log an error if the read fails, converting file reading to use promises is also simple.

function readFile(path) {
  return new Promise((resolve, reject) => {
    fs.readFile(path, (err, data) => {
      if (err) {
        return reject(err);
      }
      resolve(data);
    });
  });
}

Here’s what the core of our program looks like when we rewrite it to use our new readFile function. There’s now only three levels of function nesting instead of four.

fs.stat(file1, (err, stats1) => {
  if (err) {
    stats1 = {size: 0, mtime: Date.now()};
  }

  fs.stat(file2, (err, stats2) => {
    if (err) {
      stats2 = {size: 0, mtime: Date.now()};
    }

    if (stats1.size === stats2.size) {
      const time1 = stats1.mtime.getTime();
      const time2 = stats2.mtime.getTime();

      if (time1 <= time2) {
        console.log('sent 0 bytes');
        process.exit(0);
      }
    }

    readFile(file1).then(data => {
      return writeFile(file2, data);
    }).then(size => {
      console.log(`sent ${size} bytes`);
    }).catch(err => {
      console.log(err);
      process.exit(1);
    });
  });
});

Plus, we have one error handler instead of two. The catch callback handles the error cases for both reading and writing files. We still have an early return, but we can use Promise.resolve to eliminate that.

Promise.resolve turns a value into a promise. We can use null as a marker for the early return case, where file size is the same and file modification time hasn’t changed. Then we can do an extra check to make sure we have data before trying to write it.

fs.stat(file1, (err, stats1) => {
  if (err) {
    stats1 = {size: 0, mtime: Date.now()};
  }

  fs.stat(file2, (err, stats2) => {
    if (err) {
      stats2 = {size: 0, mtime: Date.now()};
    }

    let promise = readFile(file1);

    if (stats1.size === stats2.size) {
      const time1 = stats1.mtime.getTime();
      const time2 = stats2.mtime.getTime();

      if (time1 <= time2) {
        promise = Promise.resolve(null);
      }
    }

    promise.then(data => {
      if (!data) {
        return 0;
      }

      return writeFile(file2, data);
    }).then(size => {
      console.log(`sent ${size} bytes`);
    }).catch(err => {
      console.log(err);
      process.exit(1);
    });
  });
});

Promises are smart. They’ll ensure anything returned from a then callback is a promise. Even though we sometimes return zero and sometimes return a Promise object, it’s always safe to call then on the result and log the number of bytes written.

The last thing to rewrite is the fs.stat calls . If there’s an error, we pretend that the file is empty and brand new. That covers edge cases where the files we’re working with can’t be accessed or don’t exist.

function stat(path) {
  return new Promise((resolve) => {
    fs.stat(path, (err, result) => {
      if (err) {
        result = {size: 0, mtime: Date.now()};
      }
      resolve(result);
    });
  });
}

Because we’re handling errors, we don’t need a reject callback. If the fs.stat call throws, our promise will bubble the exception up to any catch callback chained onto it. This means we can keep all the error handling code in one place.

We need to call our stat function twice, once for each file. And we need to wait until both calls resolve before doing anything else. Promise.all takes care of that. It waits until both stat calls resolve and returns their results as an array.

Promise.all([stat(file1), stat(file2)]).then(stats => {
  if (stats[0].size === stats[1].size) {
    const time1 = stats[0].mtime.getTime();
    const time2 = stats[1].mtime.getTime();

    if (time1 <= time2) {
      return null;
    }
  }

  return readFile(file1);
}).then(data => {
  if (!data) {
    return 0;
  }

  return writeFile(file2, data);
}).then(size => {
  console.log(`sent ${size} bytes`);
}).catch(err => {
  console.log(err);
  process.exit(1);
});

This refactored code has a single level of function nesting. It keeps logging to a minimum and avoids early returns. We could take it a step further and move stuff like the time comparison logic into its own function. But this is a good stopping point for now.

I tend to work from the inside out when doing this kind of refactoring. That’s because it’s often easy to change an existing callback to return a promise. Going the other way, having a promise call a callback, can get messy. Try both ways and see which you prefer. There’s no wrong way to excape callback hell.

How to read line-by-line and keep memory use low in Node.js

2017-05-13T00:00:00-04:00

It’s easy to eat a lot of memory when parsing text in Node, especially when reading from a stream.

Suppose you’re reading data from the DARPA Intrusion Detection Data Sets, and you want to compute the mean time of an attack. Assume you’ve already processed the data so it’s just a list of timestamps as hours, minutes, and seconds.

10:09:57
02:13:20
00:03:36
10:21:33
19:37:32

Here’s JavaScript that parses each timestamp to get the total number of seconds and a count of the number of timestamps.

let total = 0;
let count = 0;

const parse = (timestamp) => {
  const [h, m, s] = timestamp.split(':');
  total += (Number(h) || 0) * 60 * 60;
  total += (Number(m) || 0) * 60;
  total += (Number(s) || 0);
  count += 1;
};

With that, you can compute the mean as Math.ceil(total / count). If you’re reading timestamps from process.stdin in Node, you can store them in a string and compute the mean once you’ve seen them all.

let text = '';

process.stdin.setEncoding('utf-8');

process.stdin.on('data', (data) => {
  if (data) {
    text += data;
  }
});

process.stdin.on('end', () => {
  const timestamps = text.split("\n");
  timestamps.forEach(parse);

  const mean = Math.ceil(total / count);
  console.log(`${mean} seconds`);
});

The gotcha is that all that text is kept in memory. Node has a maximum String size, and it’ll throw exceptions if you try and read too much data. Running on an old iMac, I can read about 512 MB before Node crashes.

To figure out how much memory this used, I ran the code above through Chrome DevTools on a 34 MB input file and captured two heap snapshots.

$ du -h data.txt
 34M    data.txt

$ node --inspect --debug-brk mean-time.js < data.txt
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/remote/serve_file/...
Debugger attached.
Waiting for the debugger to disconnect...

The first snapshot is the amount of memory used on startup, before any data has been processed. It’s 3.2 MB. The second snapshot is the amount of memory used after all the data has been processed. It’s 38.3 MB.

Which means most of the memory is being used to store all that text. Fortunately, there’s an easy workaround. Check for complete timestamps as each new chunk of data comes in and parse each one on the fly.

let text = '';

process.stdin.setEncoding('utf-8');

process.stdin.on('data', (data) => {
  if (data) {
    text += data;

    const timestamps = text.split("\n");
    if (timestamps.length > 1) {
      text = timestamps.splice(-1, 1)[0];
      timestamps.forEach(parse);
    }
  }
});

process.stdin.on('end', () => {
  parse(text);

  const mean = Math.ceil(total / count);
  console.log(`${mean} seconds`);
});

This code splits the buffered text on newlines as it arrives. It parses every timestamp but the last, which is used to start the next buffer. At the end, it parses anything remaining. Saving that the last timestamp until the end means that if we get a partial timestamp, like 02:13:, we won’t try to parse it until we see the whole thing.

I ran the same experiment to see if memory use improved.

$ node --inspect --debug-brk mean-time.js < data.txt
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/remote/serve_file/...
Debugger attached.
Waiting for the debugger to disconnect...

As before, the first snapshot is the amount of memory used on startup, before any data has been processed. It’s still 3.2 MB. The second snapshot is the amount of memory used after all the data has been processed. This time it’s only 4.1 MB. That’s 34.2 MB of memory saved!

By processing data as it becomes available, you can read data line-by-line and keep memory use low.

I found a clean way to stop reading from a Node.js stream

2017-05-06T00:00:00-04:00

The fast-csv Node module makes it easy to parse .list files from the DARPA Intrusion Detection Data Sets. Set the delimiter option to a space character and you’re good to go.

#!/usr/bin/env node

const fs = require('fs');
const csv = require('fast-csv');

const csvFile = process.argv[2];
const fileStream = fs.createReadStream(csvFile);
const csvParser = csv({'delimiter': ' '});

let lines = 0;

csvParser.on('data', function (line) {
  lines += 1;
  console.log(line.join(' '));
});

csvParser.on('end', function () {
  console.log(`read ${lines} lines`);
});

fileStream.pipe(csvParser);

Here’s output from that code running on example data.

$ node darpa-parser.js bsm.list
1 01/23/1998 16:56:48 00:01:26 telnet 1754 23 192.168.1.30 192.168.0.20 0 -
2 01/23/1998 16:56:51 00:00:14 ftp 1755 21 192.168.1.30 192.168.0.20 0 -
10 01/23/1998 16:57:02 00:01:00 telnet 1769 23 192.168.1.30 192.168.0.20 0 -
12 01/23/1998 16:57:12 00:00:03 finger 1772 79 192.168.1.30 192.168.0.20 0 -
13 01/23/1998 16:57:22 00:00:03 smtp 1778 25 192.168.1.30 192.168.0.20 0 -
14 01/23/1998 16:57:23 00:00:03 smtp 1783 25 192.168.1.30 192.168.0.20 0 -
20 01/23/1998 16:57:00 00:01:11 telnet 43496 23 192.168.0.40 192.168.0.20 0 -

... many lines later ...

270 01/23/1998 17:04:29 00:00:05 exec 2032 512 192.168.1.30 192.168.0.20 1 port-scan
308 01/23/1998 17:05:08 00:00:37 telnet 1042 23 192.168.1.30 192.168.0.20 0 -
310 01/23/1998 17:05:31 00:00:01 smtp 1048 25 192.168.1.30 192.168.0.20 0 -
311 01/23/1998 17:06:00 00:00:01 finger 1050 79 192.168.1.30 192.168.0.20 0 -
read 64 lines

But those .list files can get pretty big, and you might not want to parse the entire thing. If you want to stop the first time you see the smtp program, you can try emitting an end event.

if (line[4] === 'smtp') {
  csvParser.emit('end');
}

Let’s put those three lines after the first console.log statement and run the program again.

$ node darpa-parser.js bsm.list
1 01/23/1998 16:56:48 00:01:26 telnet 1754 23 192.168.1.30 192.168.0.20 0 -
2 01/23/1998 16:56:51 00:00:14 ftp 1755 21 192.168.1.30 192.168.0.20 0 -
10 01/23/1998 16:57:02 00:01:00 telnet 1769 23 192.168.1.30 192.168.0.20 0 -
12 01/23/1998 16:57:12 00:00:03 finger 1772 79 192.168.1.30 192.168.0.20 0 -
13 01/23/1998 16:57:22 00:00:03 smtp 1778 25 192.168.1.30 192.168.0.20 0 -
read 5 lines
14 01/23/1998 16:57:23 00:00:03 smtp 1783 25 192.168.1.30 192.168.0.20 0 -
20 01/23/1998 16:57:00 00:01:11 telnet 43496 23 192.168.0.40 192.168.0.20 0 -

... many lines later ...

270 01/23/1998 17:04:29 00:00:05 exec 2032 512 192.168.1.30 192.168.0.20 1 port-scan
308 01/23/1998 17:05:08 00:00:37 telnet 1042 23 192.168.1.30 192.168.0.20 0 -
310 01/23/1998 17:05:31 00:00:01 smtp 1048 25 192.168.1.30 192.168.0.20 0 -
311 01/23/1998 17:06:00 00:00:01 finger 1050 79 192.168.1.30 192.168.0.20 0 -

Oops. That didn’t stop the processing, it just moved the output. Fortunately, streams in Node can be paused and resumed. Calling csvParser.pause() instead might get us what we want.

if (line[4] === 'smtp') {
  csvParser.pause();
}

Let’s replace the if statement we added with this one and run our code again.

$ node darpa-parser.js bsm.list
1 01/23/1998 16:56:48 00:01:26 telnet 1754 23 192.168.1.30 192.168.0.20 0 -
2 01/23/1998 16:56:51 00:00:14 ftp 1755 21 192.168.1.30 192.168.0.20 0 -
10 01/23/1998 16:57:02 00:01:00 telnet 1769 23 192.168.1.30 192.168.0.20 0 -
12 01/23/1998 16:57:12 00:00:03 finger 1772 79 192.168.1.30 192.168.0.20 0 -
13 01/23/1998 16:57:22 00:00:03 smtp 1778 25 192.168.1.30 192.168.0.20 0 -

That’s closer. The processing stopped, but it didn’t print out how many lines where read. If we combine the two, we might get what we want. We’ll call pause() first to stop the stream, and then emit() to trigger the printing.

if (line[4] === 'smtp') {
  csvParser.pause();
  csvParser.emit('end');
}

Let’s replace the if statement with this one and run the program one more time.

$ node darpa-parser.js bsm.list
1 01/23/1998 16:56:48 00:01:26 telnet 1754 23 192.168.1.30 192.168.0.20 0 -
2 01/23/1998 16:56:51 00:00:14 ftp 1755 21 192.168.1.30 192.168.0.20 0 -
10 01/23/1998 16:57:02 00:01:00 telnet 1769 23 192.168.1.30 192.168.0.20 0 -
12 01/23/1998 16:57:12 00:00:03 finger 1772 79 192.168.1.30 192.168.0.20 0 -
13 01/23/1998 16:57:22 00:00:03 smtp 1778 25 192.168.1.30 192.168.0.20 0 -
read 5 lines

That did it! Stream processing stopped early and the end event handler was called. This sort of thing can be useful if you encounter erorrs when parsing files and want to halt processing early. And it’s not just for reading files. This pause and emit trick works for writable streams too.

Validate SSL certificates in Node.js without getting uncaught exceptions

2017-04-29T00:00:00-04:00

Node’s https module makes it easy to spin up a server with TLS. Use OpenSSL to generate a certificate and you’re good to go.

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 1
$ openssl pkcs12 -export -in cert.pem -inkey key.pem -out server.pfx

Here’s JavaScript for starting a HTTPS server with a SSL certificate.

#!/usr/bin/env node

const fs = require('fs');
const https = require('https');

const options = {
  pfx: fs.readFileSync('./server.pfx'),
  passphrase: process.argv[2]
};

const server = https.createServer(options, (_, res) => {
  res.writeHead(200);
  res.end('hello world\n');
});

server.listen(8080, '127.0.0.1');
console.log('Server listening on 127.0.0.1:8080');

But what if the passphrase is wrong? Running the code above with a passphrase that doesn’t match the one in the certificate generates an error.

$ node server.js Passw0rd
_tls_common.js:137
  c.context.loadPKCS12(pfx, passphrase);
            ^

Error: mac verify failure
  at Error (native)
  at Object.createSecureContext (_tls_common.js:137:17)
  at Server (_tls_wrap.js:776:25)
  at new Server (https.js:26:14)
  at Object.exports.createServer (https.js:47:10)
  at Object.<anonymous> (./server.js:11:22)
  at Module._compile (module.js:570:32)
  at Object.Module._extensions..js (module.js:579:10)
  at Module.load (module.js:487:32)
  at tryModuleLoad (module.js:446:12)

We can try to gracefully handle that error, by listening for error events on the server.

server.on('error', (err) => {
  console.error('There was a server error!', err);
});

Adding the code above before the server.listen() call and rerunning the program doesn’t help. We get the same error message. Looking at the stack trace, we can see the failure is happening in the https.createServer() call, before our server error handler is registered.

We could use process.on('uncaughtException') and register a high level error handler.

process.on('uncaughtException', (err) => {
  console.error('Uncaught exception!', err);
  process.exit(1);
});

That would catch the bad passphrase, but it wouldn’t give us the oportunity to recover from it. Once an uncaught exception is thrown, the only safe option is to exit the program. Ideally, we want is to catch the bad passphrase before we create the server.

Looking at the stack trace gives us a clue. The error is being thrown from tls.createSecureContext() in the tls module. What if we called that ourselves before creating the server?

try {
  const tls = require('tls');
  tls.createSecureContext(options);
} catch (err) {
  console.error('There was a TLS error!', err.message);
  console.error('Did you enter the right passphrase?');
  process.exit(1);
}

Adding the code above before the https.createServer() call and rerunning the program fixes it. Now the user’s prompted with a helpful error message.

$ node server.js Passw0rd
There was a TLS error! mac verify failure
Did you enter the right passphrase?

And if we enter the right passphrase, the server starts succesfully.

$ node server.js rosebud
HTTPS server listening on 127.0.0.1:8080

bot net buy out

2015-03-31T00:00:00-04:00

bot net
buy out

what do you do
when you’re a weekly
incorporated A.I. bored with
keeping crypto currency cached
across a million machines?

stand by
on TV.

fork your state vector
and play a game
with yourself against yourself
while your original vector
subscribes to your feed.

social media
numbers count.

watch ads for money
it doesn’t matter that
your eyeballs are digital
since payouts to corporate
legal entities are automatic.

rent servers
for growth.

Roll Over	Roll Under
1	20
2	19
3	18
4	17
5	16
6	15
7	14
8	13
9	12
10	11
11	10
12	9
13	8
14	7
15	6
16	5
17	4
18	3
19	2
20	1

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	4	6	4	5
2	7	4	6	4	5
3	8	5	7	7	6
4	8	5	7	7	6
5	10	7	9	9	8
6	10	7	9	9	8
7	11	8	10	10	9
8	11	8	10	10	9
9	13	10	12	12	11
10	13	10	12	12	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	8	5	8	6	7
2	8	5	8	6	7
3	8	5	9	6	7
4	8	5	9	6	7
5	9	6	10	8	9
6	9	6	10	8	9
7	9	6	10	8	9
8	9	6	10	8	9
9	10	7	12	10	11
10	10	7	12	10	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	6	8	9	10
2	7	6	8	9	10
3	7	6	8	9	10
4	7	6	8	9	10
5	7	6	8	9	10
6	8	8	10	11	12
7	8	8	10	11	12
8	8	8	10	11	12
9	8	8	10	11	12
10	8	8	10	11	12

Ability Score	Success Rate
1	5%
2	10%
3	15%
4	20%
5	25%
6	30%
7	35%
8	40%
9	45%
10	50%
11	55%
12	60%
13	65%
14	70%
15	75%
16	80%
17	85%
18	90%
19	95%

Roll Over	Roll Under
1	20
2	19
3	18
4	17
5	16
6	15
7	14
8	13
9	12
10	11
11	10
12	9
13	8
14	7
15	6
16	5
17	4
18	3
19	2
20	1

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	4	6	4	5
2	7	4	6	4	5
3	8	5	7	7	6
4	8	5	7	7	6
5	10	7	9	9	8
6	10	7	9	9	8
7	11	8	10	10	9
8	11	8	10	10	9
9	13	10	12	12	11
10	13	10	12	12	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	8	5	8	6	7
2	8	5	8	6	7
3	8	5	9	6	7
4	8	5	9	6	7
5	9	6	10	8	9
6	9	6	10	8	9
7	9	6	10	8	9
8	9	6	10	8	9
9	10	7	12	10	11
10	10	7	12	10	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	6	8	9	10
2	7	6	8	9	10
3	7	6	8	9	10
4	7	6	8	9	10
5	7	6	8	9	10
6	8	8	10	11	12
7	8	8	10	11	12
8	8	8	10	11	12
9	8	8	10	11	12
10	8	8	10	11	12

Ability Score	Success Rate
1	5%
2	10%
3	15%
4	20%
5	25%
6	30%
7	35%
8	40%
9	45%
10	50%
11	55%
12	60%
13	65%
14	70%
15	75%
16	80%
17	85%
18	90%
19	95%

Frank Mitchell

Dewdrop Farm - A Post-Mortem

Week 1st

Googling Graphics

Idling Away

Randomizing Growth

Looking Back

Week 2nd

Finding Fun

Looking Back

Week 3rd

Managing Inventory

Clicking Around

Looking Back

Week 4th

Changing Seasons

Optimizing Graphics

Looking Back

Closing Thoughts

Who else wants narrative mechanics in RPGs?

The Warrior

The Rogue

The Mage

Who else wants unified mechanics in RPGs?

Algorithms matter on the mobile web

Sometimes you need to DIY

Where do we go from here?

Addendum

Install TQSL v2.3.1 on Raspian Jessie

Use promises in Node.js and avoid callback hell

How to read line-by-line and keep memory use low in Node.js

I found a clean way to stop reading from a Node.js stream

Validate SSL certificates in Node.js without getting uncaught exceptions

bot net buy out

Week 1^st

Week 2^nd

Week 3^rd

Week 4^th

Roll Over	Roll Under
1	20
2	19
3	18
4	17
5	16
6	15
7	14
8	13
9	12
10	11
11	10
12	9
13	8
14	7
15	6
16	5
17	4
18	3
19	2
20	1

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	4	6	4	5
2	7	4	6	4	5
3	8	5	7	7	6
4	8	5	7	7	6
5	10	7	9	9	8
6	10	7	9	9	8
7	11	8	10	10	9
8	11	8	10	10	9
9	13	10	12	12	11
10	13	10	12	12	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	8	5	8	6	7
2	8	5	8	6	7
3	8	5	9	6	7
4	8	5	9	6	7
5	9	6	10	8	9
6	9	6	10	8	9
7	9	6	10	8	9
8	9	6	10	8	9
9	10	7	12	10	11
10	10	7	12	10	11

Level	Poison	Breath Weapon	Polymorph	Spell	Magic Item
1	7	6	8	9	10
2	7	6	8	9	10
3	7	6	8	9	10
4	7	6	8	9	10
5	7	6	8	9	10
6	8	8	10	11	12
7	8	8	10	11	12
8	8	8	10	11	12
9	8	8	10	11	12
10	8	8	10	11	12

Ability Score	Success Rate
1	5%
2	10%
3	15%
4	20%
5	25%
6	30%
7	35%
8	40%
9	45%
10	50%
11	55%
12	60%
13	65%
14	70%
15	75%
16	80%
17	85%
18	90%
19	95%