Writing applications as a sequence of tiny services that all talk to each other over the network has many upsides, but it can be annoyingly tedious to get all the subsystems up and running.

Running a seaport can help with getting all the services to talk to each other, but running the processes is another matter, especially when you have new code to push into production.

fleet aims to make it really easy for anyone on your team to push new code from git to an armada of servers and manage all the processes in your stack.

To start using fleet, just install the fleet command with npm:

Then on one of your servers, start a fleet hub. From a fresh directory, give it a passphrase and a port to listen on:

Now fleet is listening on :7000 for commands and has started a git server on :7001 over http. There's no ssh keys or post commit hooks to configure, just run that command and you're ready to go!

Next set up some worker drones to run your processes. You can have as many workers as you like on a single server but each worker should be run from a separate directory. Just do:

where x.x.x.x is the address where the fleet hub is running. Spin up a few of these drones.

Now navigate to the directory of the app you want to deploy. First set a remote so you don't need to type --hub and --secret all the time.

Fleet just created a fleet.json file for you to save your settings.

From the same app directory, to deploy your code just do:

The deploy command does a git push to the fleet hub's git http server and then the hub instructs all the drones to pull from it. Your code gets checked out into a new directory on all the fleet drones every time you deploy.

Because fleet is designed specifically for managing applications with lots of tiny services, the deploy command isn't tied to running any processes. Starting processes is up to the programmer but it's super simple. Just use the fleet spawn command:

By default fleet picks a drone at random to run the process on. You can specify which drone you want to run a particular process on with the --drone switch if it matters.

Start a few processes across all your worker drones and then show what is running with the fleet ps command:

Now suppose that you have new code to push out into production. By default, fleet lets you spin up new services without disturbing your existing services. If you fleet deploy again after checking in some new changes to git, the next time you fleet spawn a new process, that process will be spun up in a completely new directory based on the git commit hash. To stop a process, just use fleet stop.

This approach lets you verify that the new services work before bringing down the old services. You can even start experimenting with heterogeneous and incremental deployment by hooking into a custom http proxy!

Even better, if you use a service registry like seaport for managing the host/port tables, you can spin up new ad-hoc staging clusters all the time without disrupting the normal operation of your site before rolling out new code to users.

Fleet has many more commands that you can learn about with its git-style manpage-based help system! Just do fleet help to get a list of all the commands you can run.

npm install -g fleet and check out the code on github!

Introducing difflet, a handy node.js module for computing pretty object diffs!

Just plug in some initial options and the objects you want to compare!

and you'll get a colored and annotated object expressing the differences between the 2 objects you passed in:

Deleted elements between the first and second object are shown in red and commented if comments are turned on. New object show up in green and updated objects show in blue. The comments show what the previous value was, if any.

You can set a bunch of options to adjust the formatting including comma-first output. You can even generate HTML output:

which generates some HTML output that you can stuff into a browser:

Plus, pkrumins and I just rolled out difflet output for testling to make debugging t.deepEqual() statements on big objects easier:

node-tap is now using difflet too in 0.2.1!

Check out the code on github or with npm do:

Pushing out incremental changes to a service-oriented cluster can be tricky, especially when changes span multiple services. Introducing seaport, a service registry written in node.js based on semvers.

With seaport, services are brought up with a name@version string and other processes can connect to services that match a name@semver pattern.

First spin up a new seaport server on a port:

Here's an example seaport service that registers a new service web@1.2.3:

and here's some code that connects to the web@1.2.3 service:

Running the last script prints out beep boop from the web@1.2.3 http server.

You can create seaport servers programatically. This is useful for integrating seaport with an http host router. Here's an example using bouncy:

We can then spin up http services for the semvers 0.1.x and 0.0.x:

and the other server...

Now we can dynamically route to semvers based on the http host header!

These http servers could themselves have dependencies on other services in the cluster. By using seaport, new code can be pushed out that spans multiple servers with very explicit backwards compatability that tolerates and encourages multiple versions of services to satisfy dependencies. The easier it is to push out crazy new experiments, the more often it will happen!

Seaport is a component for these hypothetical cluster commands that would make continuous deployment for clusters super simple.

Check out the seaport source on github or npm install seaport!

Testling is a browserling product that pkrumins and I put together to make running automated browser tests super easy.

Tests usually look something like this:

Then you can run the tests on all the browsers we run using a curl one-liner like this one:

and when the code blows up (in IE7 in this case because it doesn't have Object.keys), you get a full stack trace!

Wow super great! Except perhaps you don't like how the standard test API looks and want something more jasmine-esque and bdd-ish.

Just write a little wrapper like this:

And now you can write your tests like so, require()ing the bdd wrapper node.js-style:

Now you have 2 files but you can still use a one-liner to stitch everything together:

...and strangely enough, IE9 only sleeps for 83 milliseconds when you tell it to sleep for 100! TYPICAL.

Check out the testling documentation, create a browserling account to use with testling, and hack up some crazy browser tests and test runners!

I would like to document an emerging set of programming conventions, philosophies, and values that I see evolving in the node.js community. I call this the node aesthetic.

callback austerity

The very first example of node you are likely to see is on the node.js home page. This snippet is exemplary of the the radical simplicity pioneered by projects like sinatra.

Instead of the http server being an external service that we configure to run our code, it becomes just another tool in our arsenal. Want to spin up a second http server on a different port? Super easy: just call http.createServer() once more. Want to print out how many requests per second you're averaging every minute? Just plop down a counter and a setInterval(). Node makes what was previously merely configurable into something more programmable, orthogonal, and more powerful.

A big part of what empowers node to make these kinds of interfaces possible is its asynchronous nature. No longer do you have statelessness imposed from on high by the likes of apache or rails. You can keep intermediate state around in memory just like in any ordinary program. You don't need to reason about multiple instruction pointers or mutexes or re-entrant, interruptible execution because all the javascript you write just lives in a single thread.

Depending on the event system to already just be there means that the end-users of modules don't have to think about what hoops they need to jump through in order to get the event reactor up, running, and plugged into the component they're trying to use. The event machinery is all running quietly under the hood without getting in the way or making a fuss.

limited surface area

Note also that the http snippet from earlier isn't inheriting from an http.Server base class or anything of the sort. The primary focus of most node modules is on using, not extending. In classical object-oriented design, it's customary to define specific custom functionality by extending more abstract classes. This idea flows pretty naturally from thinking about how you might write a compiler to efficiently pack collections of related properties into memory blocks, but is not such a great approach for writing usable and perhaps more importantly, re-usable interfaces.

Simple function calls with callback arguments have the superficial but important benefit of having less exposed surface area and more encapsulation than most classical designs. But moreover, when you extend a base class, your class is somewhat stuck with the interface and internals that the base class uses. If you want to make things prettier you'll need to shave some yaks to write wrapper classes just to get around the restrictions of the class system. Sometimes this classical style of reusability is appropriate, but in node it is the exception, not the rule, because the implicit consensus is that the interface is paramount, so usability should trump extensibility.

A big part of what makes node modules so great is how they tend to have really obvious entry points as a consequence of focusing on usability and limited surface area. Many modules on npm export only a single function by assigning onto module.exports such that require('modulename') returns just that single function of interest. It also helps that module names have a direct correspondence to the string that you put in your call to require() and that require() just returns the module instead of modifying the environment to include the module. This is known in some circles as doing a qualified import. In node you can't even do any other kind of import without getting really hackish about it and this is a very good thing. This import style means that when you fire up the REPL to play with a new module when you do require('modulename') you can see exactly what functionality 'modulename' provides. Even more importantly, it means that when you read through some new foreign piece of code that uses many modules, you can tell exactly where all the module exports came from.

batteries not included

If software ecosystems are like economies then core libraries are at best government bureaucracies and at worst nationalized state corporations. That is, modules in the core distribution get an unfair advantage over the libraries in userspace by virtue of availability and prominance. Worse still, because these modules are all maintained in the core project, contributing, experimenting, and iterating is much harder than with a typical library on npm. This is especially compounded by the fact that core libraries are pegged to the core project version and don't often have independent versioning.

Core distributions with too many modules result in neglected code that can't make meaningful changes without breaking everything. If a neglected library lives in userspace then at least people can ignore it more easily and the independent versioning makes necessary breaking changes easier to mitigate for legacy code.

The "batteries included" approach may have struck a worthwhile tradeoff back when package managers were primitive and modules were global, but that advantage evaporates in the face of baked-in concurrent library versioning and sophisticated package management.

radical reusability

While the limited surface area approach can hurt extensibility, you can win a great deal of extensibility back by breaking up problems into lots of tiny modules. When you write a module for general enough consumption to put up on npm, you can't contaminate your own implementation with too many implementation-specific details. It's also much easier to test and iterate on code in complete isolation from your application business logic. Node and npm go to great lengths to help you do this.

In node the only modules that can be said to be "global" are compiled into the binary itself. There is not really such a thing as a global, system-level module anymore. Instead, when you run a script that performs a require(), node will search for the module in a "node_modules" directory from the directory that the script is in all the way down to "/", stopping at the nearest "node_modules" directory. Similarly, when you do an npm install, node will place the modules into the nearest "node_modules" directory or "./node_modules".

All of this preferred locality in the module loader has the very important practical upshot in that you can depend on different versions of modules in different places in your program's dependency graph and node will pick the most local dependency for each module. So if a module "foo" was tested against and depends on "baz@0.1.x" and a module "bar" depends on "baz@0.2.x", then when you go to use both "foo" and "bar" in your own program, "foo" and "bar" will both use the version of "baz" that they were tested and depend against!

This approach to versioning means that nearly all modules should be interoperable with each other. It also means that module authors can be more aggressive with iterating and pushing out backwards-incompatible breaking changes to their APIs, so long as the authors are careful to increment the module version appropriately. Concurrent versioning and locality preference drastically accelerate the rate that we can iterate on new experiments and the levels of reuse that we can attain both in a single large project and among projects.

the future

I am wildly optimistic about where this emerging aesthetic of radical reusability and module-driven development will take node and programming in general.

older