commit d8282f8067e8ace850cd57a56bff80d3951cc0b8
Author: James Halliday
Date: Thu Nov 27 18:54:26 2014 -0800

offline decentralized single sign-on in the browser

Recently, browsers have just begun to implement web cryptography. This means that browsers are now capable of the same kind of passwordless decentralized authentication schemes we've had server-side with ssh and tls asymmetric keys for decades.

Imagine if you could just generate a key and sign messages with that key, proving your identity to other users and backend services without the need for a password or even creating an account with yet another web server! We can have the ease of use of signing in with twitter or facebook without any centralized servers and very minimal setup from the end user's perspective.

Even better, this authentication system can work offline. In fact, the system must work offline to be fully secure.

appcache manifesto

By default, web pages can decide to send you whatever javascript and html payload it wants. This payload is probably what you want, at least at first, but consider what might happen if those asymmetric keys are used for commerce or for private communications. Suddenly, the webmaster could easily decide to serve a different payload, perhaps in a targetted manner, that copies private keys to third parties or performs malicious operations. Even if the webmaster is an upstanding person, a government agent could show up at any time with a court order forcing the webmaster to serve up a different payload for some users.

Imagine if whenever you ran the ssh command, your computer fetched the latest version of the ssh binary from openssh.org and then executed it. This would be completely unacceptable for server programs, and browser apps that handle confidental keys and data should be no different!

Luckily, there is another relatively new feature in the browser that can protect against rogue server updates: the appcache manifest. A page can set a manifest file with:

<html manifest="page.appcache">

and then the browser will load a cache policy from page.appcache. The appcache file can be used to make some documents available offline, but can also be used to prevent the browser from fetching updates to documents. If the max-age header on the appcache file itself is set far enough in the future, the appcache file itself can be made permanent so that the server operator can't update this file either. In the future, the service worker API will provide enough hooks to do the same thing, but browser support is not widespread yet.

hyperboot

Upgrading an application should be possible too without going into the bowels of the browser to clear the appcache. This is where hyperboot comes in to give us opt-in application upgrades for end-users. More security-minded users might even want to check with external auditing systems before upgrading.

keyboot

With a versioning system in place, we can now start implementing an offline single sign-on system that exposes the web crypto methods securely without exposing private keys to random websites.

There are another few nifty tricks with the service worker API that can give us realtime communication between tabs and iframes that works completely offline.

To give this new system a try, first open https://keyboot.org in a modern browser and generate a key.

Next open up http://keyboot-example-app.hyperboot.org/ in a new window or tab and paste https://keyboot.org/ into the text box. In the https://keyboot.org/ window, approve the request. Now from http://keyboot-example-app.hyperboot.org/, you can sign messages with your private key!

There is still plenty to do and some unanswered questions about different threat models and how best to prevent replay attacks and domain isolation, but this proof of concept should be good enough to at least start people thinking about decentralized approaches to single sign-on and the changing role of servers and webapps as browser APIs become more capable.

links

commit 5e004e4de5c1da6888c302607e6556b05b354320
Author: James Halliday
Date: Sat May 17 23:15:29 2014 +0200

One of the most common objections I've heard about embracing modularity and favoring libraries that do a single thing well is that it can be difficult and time-consuming to find packages for each piece of functionality you might need for a given task.

This is certainly true at first, but over time and with practice, it is less and less of a problem as you train up your own heuristics and develop a broad working memory of useful packages and authors who tend to produce useful code that suits your own aesthetic preference.

With a bit of training and practice, you will be skimming npm search results at great speed in no time!

my heuristic

Here's my own internal heuristic for evaluating npm packages:

  • I can install it with npm

  • code snippet on the readme using require() - from a quick glance I should see how to integrate the library into what I'm presently working on

  • has a very clear, narrow idea about scope and purpose

  • knows when to delegate to other libraries - doesn't try to do too many things itself

  • written or maintained by authors whose opinions about software scope, modularity, and interfaces I generally agree with (often a faster shortcut than reading the code/docs very closely)

  • inspecting which modules depend on the library I'm evaluating - this is baked into the package page for modules published to npm

When a project tries to do too many things, parts of it will invariably become neglected as the maintenance burden is unsustainable. The more things a project tries to do, the easier it is to be completely wrong about some assumption and this can also lead to abandonment because it's very difficult to revisit assumptions later.

The best, longest-lasting libraries are small pieces of code that are very tricky to write, but can be easily verified. Highly mathematical tend to be very well represented in this category, like the gamma function or an ecosystem of highly decoupled matrix manipulation modules such as ndarray.

When a library is embedded in an ecosystem of other libraries in a thoroughly decoupled way, a mutual dynamic results where the main library doesn't need to inflate its scope but gets enough attention to find subtle bugs while the dependent libraries can offer excellent interoperability and fit into a larger informal organizational structure.

not too important

Here are some things that aren't very important:

  • number of stars/forks - often this is a reverse signal because projects with overly-broad scope tend to get much more attention, but also tend to flame out and become abandoned later because they take too much effort to maintain over a long period of time. However! Some libraries are genuinely mistakes but it took writing the library to figure that out.

  • activity - at a certain point, some libraries are finished and will work as long as the ecosystem around them continues to function. Other libraries do require constant upkeep because they attack a moving problem but it's important to recognize which category of module you're dealing with when judging staleness.

  • a slick web page - this is very often (but not always) a sign of a library that put all of its time into slick marketing but has overly-broad scope. It is sometimes the case that solid modules also have good web pages but don't be tricked by a fancy web page where a solid readme on github would do just as good for a job.

The main crux of this blog post first appeared as a reddit comment.

commit d95a2849d28593758c03c0cde74175cb807db857
Author: James Halliday
Date: Sun Dec 8 16:14:47 2013 -0800

In node I use simple test libraries like tap or tape that let you run the test files directly. For code that needs to run in both the browser and node I use tape because tap doesn't run in the browser very well and the APIs are mostly interchangeable.

The simplest kind of test I might write in test/ looks like:

var test = require('tape');
var someModule = require('../');

test('fibwibblers and xyrscawlers', function (t) {
    t.plan(2);

    var x = someModule();
    t.equal(x.foo(2), 22);

    x.beep(function (err, res) {
        t.equal(res, 'boop');
    });
})

To run a single test file in node I just do:

node test/fibwibbler.js

And if I have multiple tests I want to run I do:

tape test/*.js

or I can just use the tap command even if I'm just using tape because tap only looks at stdout for tap output:

tap test/*.js

The best part is that since tape just uses console.log() to print its tap-formatted assertions, all I need to do is browserify my test files.

To compile a single test in the browser I can just do:

browserify test/fibwibbler.js > bundle.js

or to compile a directory full of tests I just do:

browserify test/*.js > bundle.js

Now to run the tests in a browser I can just write an index.html:

<script src="bundle.js"></script>

and xdg-open that index.html in a local browser. To shortcut that process, I can use the testling command (npm install -g testling):

browserify test/*.js | testling

which launches a browser locally and prints the console.log() statements that executed browser-side to my terminal directly. It even sets the process exit code based on whether the TAP output had any errors:

substack : defined $ browserify test/*.js | testling

TAP version 13
# defined-or
ok 1 empty arguments
ok 2 1 undefined
ok 3 2 undefined
ok 4 4 undefineds
ok 5 false[0]
ok 6 false[1]
ok 7 zero[0]
ok 8 zero[1]
ok 9 first arg
ok 10 second arg
ok 11 third arg
not ok 12 (unnamed assert)
  ---
    operator: ok
    expected: true
    actual:   false
    at: Test.ok.Test.true.Test.assert (http://localhost:47079/__testling?show=true:7772:10)
  ...
# (anonymous)
ok 13 should be equal

1..13
# tests 13
# pass  12
# fail  1
substack : defined $ echo $?
1
substack : defined $

coverage

bonus content: if I want code coverage, I can just sneak that into the pipeline using coverify. This is still experimental but here's how it looks:

$ browserify -t coverify test.js | testling | coverify

TAP version 13
# beep boop
ok 1 should be equal

1..1
# tests 1
# pass  1

# ok

# /tmp/example/test.js: line 7, column 16-28

  if (err) deadCode();
           ^^^^^^^^^^^

# /tmp/example/foo.js: line 3, column 35-48

  if (i++ === 10 || (false && neverFires())) {
                              ^^^^^^^^^^^^

or to run the tests in node, just swap testling for node:

$ browserify -t coverify test.js | node | coverify
TAP version 13
# beep boop
ok 1 should be equal

1..1
# tests 1
# pass  1

# ok

# /tmp/example/test.js: line 7, column 16-28

  if (err) deadCode();
           ^^^^^^^^^^^

# /tmp/example/foo.js: line 3, column 35-48

  if (i++ === 10 || (false && neverFires())) {
                              ^^^^^^^^^^^^

Update (2013-12-21): check out the covert package on npm, which gives you a covert command that runs browserify and coverify for you.

why write tests this way?

The node-tap API is pretty great because it feels asynchronous by default. Since you plan out the number of assertions ahead of time, it's much easier to catch false positives where asynchronous handlers with assertions inside didn't fire at all.

By using simple text-based interfaces like stdout and console.log() it's easy to get tests to run in node and the browser and you can just pipe the output around to simple command-line tools. If you stick to tools that just do one thing but expose their functionality in a hackable way, it's easy to recombine the pieces however you want and swap out components to better suit your specific needs.

commit 742794366c89622b0a7ce2ee848d1edd92d94651
Author: James Halliday
Date: Sat Dec 7 23:39:34 2013 -0800

A new browserify version is upon us, just in time for the FESTIVE SEASON during which we in the northern hemisphere at mid to high latitudes huddle for warmth around oxidizing hydrocarbons!

There are 2 big changes in v3 but most code should be relatively unaffected.

shiny new Buffer

feross forked the buffer-browserify package to create native-buffer-browserify, a Buffer implementation that uses Uint8Array to get buf[i] notation and parity with the node core Buffer api without the performance hit of the previous implementation and a much smaller file size. The downside is that Buffer now only works in browsers with Uint8Array and DataView support. All the other modules should maintain existing browser support.

Update: a shim was added to in 3.1 for Uint8Array and DataView support. Now you can use Buffer in every browser.

direct builtin dependencies

In v3, browserify no longer depends on browser-builtins, in favor of depending on packages directly. Instead of having some separate packages and some files in a builtin/ directory like browser-builtins, browserify now uses only external packages for the shims it uses. By only using external packages we can keep browserify core focused purely on the static analysis and bundling machinery while letting the individual packages worry about things like browser compatibility and parity with the node core API as it evolves.

Individual, tiny packages should also be much easier for newcomers to contribute packages toward because they won't need to get up to speed with all the other pieces going on and the packages can have their own tests and documentation. Additionally, each package may find uses in other projects beside browserify more easily and if people want variations on the versions of shims that ship with browserify core this is easier to do when everything is separate.

Back when we were using browser-builtins there was a large latency between pushing out fixes to the individual packages and getting them into browserify core because we had to wait on browser-builtins to upgrade the semvers in its package.json. With direct dependencies we get much lower latency for package upgrades and much more granular control over upgrading packages.

Here is the list of packages we now directly depend on in v3:

That's it! If you're bold enough to give v3 a spin, just do:

npm install -g browserify
commit c13668f2b5f4f515e97723fa3322aa009181629c
Author: James Halliday
Date: Mon Nov 18 01:52:10 2013 +0800

There are some fancy tools for doing build automation on javascript projects that I've never felt the appeal of because the lesser-known npm run command has been perfectly adequate for everything I've needed to do while maintaining a very tiny configuration footprint.

Here are some tricks I use to get the most out of npm run and the package.json "scripts" field.

the scripts field

If you haven't seen it before, npm looks at a field called scripts in the package.json of a project in order to make things like npm test from the scripts.test field and npm start from the scripts.start field work.

npm test and npm start are just shortcuts for npm run test and npm run start and you can use npm run to run whichever entries in the scripts field you want!

Another thing that makes npm run really great is that npm will automatically set up $PATH to look in node_modules/.bin, so you can just run commands supplied by dependencies and devDependencies directly without doing a global install. Packages from npm that you might want to incorporate into your task workflow only need to expose a simple command-line interface and you can always write a simple little program yourself!

building javascript

I write my browser code with node-style commonjs module.exports and require() to organize my code and to use packages published to npm. browserify can resolve all the require() calls statically as a build step to create a single concatenated bundle file you can load with a single script tag. To use browserify I can just have a scripts['build-js'] entry in package.json that looks like:

"build-js": "browserify browser/main.js > static/bundle.js"

If I want my javascript build step for production to also do minification, I can just add uglify-js as a devDependency and insert it straight into the pipeline:

"build-js": "browserify browser/main.js | uglifyjs -mc > static/bundle.js"

watching javascript

To recompile my browser javascript automatically whenever I change a file, I can just substitude the browserify command for watchify and add -d and -v for debugging and more verbose output:

"watch-js": "watchify browser/main.js -o static/bundle.js -dv"

building css

I find that cat is usually adequate so I just have a script that looks something like:

"build-css": "cat static/pages/*.css tabs/*/*.css > static/bundle.css"

watching css

Similarly to my watchify build, I can recompile css as it changes by substituting cat with catw:

"watch-css": "catw static/pages/*.css tabs/*/*.css -o static/bundle.css -v"

sequential sub-tasks

If you have 2 tasks you want to run in series, you can just npm run each task separated by a &&:

"build": "npm run build-js && npm run build-css"

parallel sub-tasks

If you want to run some tasks in parallel, just use & as the separator!

"watch": "npm run watch-js & npm run watch-css"

the complete package.json

Altogether, the package.json I've just described might look like:

{
  "name": "my-silly-app",
  "version": "1.2.3",
  "private": true,
  "dependencies": {
    "browserify": "~2.35.2",
    "uglifyjs": "~2.3.6"
  },
  "devDependencies": {
    "watchify": "~0.1.0",
    "catw": "~0.0.1",
    "tap": "~0.4.4"
  },
  "scripts": {
    "build-js": "browserify browser/main.js | uglifyjs -mc > static/bundle.js",
    "build-css": "cat static/pages/*.css tabs/*/*.css",
    "build": "npm run build-js && npm run build-css",
    "watch-js": "watchify browser/main.js -o static/bundle.js -dv",
    "watch-css": "catw static/pages/*.css tabs/*/*.css -o static/bundle.css -v",
    "watch": "npm run watch-js & npm run watch-css",
    "start": "node server.js",
    "test": "tap test/*.js"
  }
}

If I want to build for production I can just do npm run build and for local development I can just do npm run watch!

You can extend this basic approach however you like! For instance you might want to run the build step before running start, so you could just do:

"start": "npm run build && node server.js"

or perhaps you want an npm run start-dev command that also starts the watchers:

"start-dev": "npm run watch & npm start"

If you need to add a stage that executes after the watchify bundle is regenerated, you can use a tool like onchange to wire that up:

"watch": "npm run watch-js & npm run watch-css & npm run post-js",
"post-js": "onchange static/bundle.js -- npm test"

onchange can even accept globs as arguments to watch a whole directory tree of files.

You can reorganize the pieces however you want! The dream of unix, alive and well!

when things get really complicated...

If you find yourself stuffing a lot of commands into a single scripts field entry, consider factoring some of those commands out into someplace like bin/.

You can write those scripts in bash or node or perl or whatever. Just put the proper #! line at the top of the file, chmod +x, and you're good to go:

#!/bin/bash
(cd site/main; browserify browser/main.js | uglifyjs -mc > static/bundle.js)
(cd site/xyz; browserify browser.js > static/bundle.js)
"build-js": "bin/build.sh"

windows

A surprising amount of bash-isms work on windows but we still need to get ; and & working to get to "good enough".

I have some experiments in the works for windows compatibility that should fold in very well with this npm-centric approach but in the meantime, win-bash is a super handy little bash implementation for windows.

conclusion

I hope that this npm run approach I've documented here will appeal to some of you who may be unimpressed with the current state of frontend task automation tooling, particularly those of you like me who just don't "get" the appeal of some of these things. I tend to prefer tools that are more steeped in the unix heritage like git or here with npm just providing a rather minimal interface on top of bash. Some things really don't require a lot of ceremony or coordination and you can often get a lot of mileage out of very simple tools that do very ordinary things.

If you don't like the npm run style I've elaborated upon here you might also consider Makefiles as a solid and simple alternative to some of the more baroque approaches to task automation making the rounds these days.

more
git clone http://substack.net/blog.git