Josh Smith

Making Development Environments Consistent with Hall & Oates (& JavaScript)

A common theme in software development is “it works on my machine”.

It works on my machine https://t.co/0PepWfzXW2

— Marcel Pociot 🧪 (@marcelpociot) April 24, 2018

Having inconsistencies between each developer’s environment and between development and production application environments can lead to a ton of unexpected behaviors and bugs. JavaScript is a core part of our tech stack and processes at Niche. Outside of our CSS and its different flavors — most of which originates from our Design team — the Front End and Quality Assurance teams predominantly write JavaScript to build and maintain products that serve millions.

Shake it up is all that we know

There are two dependencies in the modern JavaScript ecosystem that need to be used at the global environment level: the server-side runtime Node.js, and a JavaScript package manager (we use npm).

Since these dependencies are handled at the global level, versioning these dependencies across our teams provided more challenges than versioning our local dependencies. Every month, someone would hop onto Slack and ask “What version of Node/npm are we running?”

Fun game: search for “what version of” in your organization’s Slack message history and skim through the results.

We made some efforts to document the versions of Node and npm each project depended on in internal wikis, but that requires developers to know about the existence and location of that documentation. Additionally, those docs quickly became outdated if we didn’t manually keep it in sync with our environments.

Ideally, the required versions of Node and npm for a given project would be specified within that project. In doing so, we could:

On that last point, we would save ourselves a lot of headaches by having a tool that enforces that developers, QA analysts, and any other team member running the application are using those specified versions.

What my head overlooks, the senses will show to my heart

The first step to solving this problem was first seeing what solutions may already exist. JavaScript developers are often derided for relying on too many dependencies, reaching for npm install before understanding the problem.

Legendary Apollo project programmer Margaret Hamilton, next to a printout of the node_modules directory listing for her first Hello World react app pic.twitter.com/tOsBifPtmI

— Thomas “🐈🔭🕹” Fuchs (@thomasfuchs) March 7, 2019

I agree that the less you depend on third-party resources, the more secure and maintainable your code will be. However, there’s definitely merit in leveraging the Node open-source ecosystem in order to move faster and avoid working on solved problems. So it was worth asking the question: Has someone solved it?

Nothing is new, I’ve seen her here before

When exploring different options, I decided to look at what tools were already a part of our process. Much of our team was already using nvm, the “Node Version Manager”, to install multiple versions of Node and switch between them easily.

I discovered that the docs for nvm include a section on an .nvmrc dotfile that developers can set up on a per-project basis. In this file, you can specify which version of Node the project uses. When running an nvm command such as nvm use or nvm install without providing an argument, nvm will use the version specified in .nvmrc.

This gave us a lot of what we were looking for!

However, there were some gaps in what an .nvmrc file could provide us with and the extent of our needs.

nvm doesn’t prevent someone from running a different version of Node from the version specified in the .nvmrc file. The file acts as a guideline, and a default if no version is specified, but it was still easy for someone who is hopping between projects to not switch their Node version before working on said project.

While many people on the team use nvm, we didn’t have full team adoption. Requiring everyone working in the codebase to use it would mean that we have yet another global dependency (and thus another opportunity for development environments to get out of sync).

Even if everyone was on board with nvm and wanted to use it, we’re a cross-platform team, and nvm does not support Windows. While we’ve tried various Windows-compatible version managers, each has had their own hiccups when attempting to add them into our workflow.

Finally, the most glaring issue with relying solely on an .nvmrc file is that it leaves out half of the equation: npm. The required format for the .nvmrc file is very specific:

The contents of a .nvmrc file must be the <version> (as described by nvm --help) followed by a newline. No trailing spaces are allowed, and the trailing newline is required.

Were it to allow arbitrary content on subsequent lines, we might be able to get away with adding the required npm version there.

Times that are broken can often be one again

With .nvmrc not solving the issue, it was back to scouring the internet and library docs.

After some reading, another possible option stood out. The package.json file that is present in practically every Node project that uses packages from npm has an optional engines field.

According to the docs, this property can be used to specify the versions of Node and npm (among other things). It would look like:


“engines”: {
  “node”: “10.15.3”,
  “npm”: “6.9.0”
}

Once again, we had something promising here.

This also had two big advantages over using an .nvmrc as well.

But unfortunately, there were still a few gotchas with using the engines field exclusively.

The npm docs specify that engines is merely advisory, so while there may be warnings to inform you if you’re running the incorrect versions of Node and npm, the docs do not guarantee that it will actually stopping you from doing so.

While enforcing versions might not be built into how npm behaves, I wondered if it was a feature provided by nvm. From the looks of it, it’s been heavily considered, and it’s actively being worked on, but isn’t fully worked out yet by the nvm maintainers.

You make my dreams come true

Given the considerations of different tooling out there, I settled on a home-grown solution to meet our needs.

If engines is Daryl Hall and .nvmrc is John Oates, then our solution, node-can-do, is their fateful meeting in West Philadelphia’s Adelphi Ballroom in 1967.

node-can-do is a script that prevents developers from getting too far when using the incorrect versions of Node and npm. We use it in conjunction with npm scripts in package.json, specifically the pre script hook that npm provides to run scripts before other scripts.

For example, let’s imagine we have a Node app with an important script, and we want to ensure we’re using the correct versions of Node and npm before running that script. In the app’s package.json, I would add the following scripts:


“scripts”: {
  “important-script”: “echo ‘Hello world!’”,
  “preimportant-script”: “node-can-do”
}
Running npm run important-script will first run npm run preimportant-script. At this point, node-can-do will perform multiple checks to ensure that environments are the same across developer and QA machines.

First, node-can-do checks to make sure that the required version of npm is specified in package.json, and that the required version of Node is specified in either package.json or an .nvmrc file.

node-can-do preventing a script from running when Node and/or npm versions aren’t specified.

Next, node-can-do checks if the running version of Node and the globally installed version of npm are the same as the versions specified in engines.

node-can-do preventing a script from running when the running versions of Node and/or npm don’t matched what’s specified in the code.

If the first or second check fails, then an error and instructions to remedy it are logged and the script exits with exit code 1, preventing npm run important-script from being run.

If those checks pass, then the script exits with exit code 0 and then runs npm run important-script.

This allows us to put this script in front of commonly used npm scripts, and if node-can-do fails, then the developer or QA analyst can’t continue without correcting their Node/npm version.

You can get along if you try to be strong

We started out with node-can-do being copy and pasted into every repo that we needed it, which works completely fine since we don’t need to update it regularly. But in an effort to reduce duplication, and also to allow us to share it more freely with others, we’ve moved the code to a new repo as a standalone package, open sourced it on GitHub, and published it to the npm registry!

If you would like to use it in your Node project, feel free to install it with


npm install --save-dev node-can-do

And if you notice any bugs or problems with it, feel free to file an issue. 🙂

I ain’t the way you found me

Despite the fix for this inconsistency issue being a simple script, it’s helped us be more confident in our environment. Developing in our core projects is now more consistent, and rarely-if-ever are two developers running the most updated version of a project while using different versions of Node or npm.

In the time since writing node-can-do, we’ve further expanded how we’re making our project environments more consistent, not only in development, but in production as well. Principal software engineer Nathan Cochran has pushed for implementing Docker at Niche. This has reduced the development friction and inconsistency across machines in a holistic way that’s agnostic of the language that various projects are written in, while node-can-do is only applicable to our Node.js applications. We hope to talk about our experience with Docker in a future technical blog article.

On a personal level, I also learned a lot in developing a solution for this. node-can-do, in its simplicity, helped us solve a challenge we were running into time and time again. I learned a lot about the Node.js API and npm packages.

It was also a lesson in avoiding overengineering. As you can see for yourself, node-can-do has none of the modern tooling of TypeScript’s static type checking, Babel’s transpiling and minification, ESLint’s code quality assurance, or even 😱 unit tests 😱. All of those are great things, but sometimes it’s better to hack something together, make sure it solves the problem, and then — if there’s enough of a need — iterate on it.

Finally, node-can-do was just fun to write. Having an excuse to combine my love of puns, Hall & Oates, and JavaScript took some of the dryness out of an otherwise tedious issue. Finding opportunities to let developers, analysts, and technical folks take initiative on problems and do weird things to solve them is one part of a recipe for creating a good work environment where people are excited by challenges.