DNode: Asynchronous Remote Method Invocation for Node.js and the Browser [ 2010-07-11 06:51:59 UTC ]

Introducing DNode

Introducing DNode, a node.js library for asynchronous bidirectional remote method invocation. Network socket and websocket-style socket.io transports are presently available so system processes can communicate with each other and with processes running in the browser using the same interface.

Remote method invocation (RMI) is the object-oriented cousin of remote procedure calls. In RMI, each side of the connection hosts an object that the other side can call the methods of. A popular example of RMI worth checking out is Ruby's DRb.

DNode is different from DRb in that all remote method calls are asynchronous. Instead of returning explicitly, hosted methods pass return values to the other side of the connection by invoking callbacks that were passed in as arguments. These callbacks execute on the side of the connection where they were defined, and a representation of them gets passed to the remote, so there's no eval().

Here's a simple example.

// server: var DNode = require('dnode'); var server = DNode({ timesTen : function (n,f) { f(n * 10) }, }).listen(6060);
// client: var DNode = require('dnode'); var sys = require('sys'); DNode.connect(6060, function (remote) { remote.timesTen(5, function (result) { sys.puts(result); // 5 * 10 == 50 }); });

Unlike many asynchronous RPC systems, DNode allows the programmer to pass functions as arguments to remote methods, which when called on the remote end signal the near side to execute the function with the arguments that the remote provides. Callbacks are automatically scrubbed and collected from a recursive walk of the arguments list, so they can be arbitrarily nested.

Additionally, each side of the connection can call the other side's methods or the callbacks the other side provides with callbacks of its own or any arguments that can be serialized to JSON.

Bidirectional Example

Here's an example where the client calls a method on the server that calls a method on the client.

// server: var DNode = require('dnode'); DNode(function (client) { this.timesX = function (n,f) { client.x(function (x) { f(n * x); }); }; }).listen(6060);
// client: var DNode = require('dnode'); DNode({ x : function (f) { f(20) } }).connect(6060, function (remote) { remote.timesX(3, function (res) { sys.puts(res); // 20 * 3 == 60 }); });

DNode Browser Example

As I mentioned in the first paragraph, websocket-style connections between the browser and node.js DNode servers are available thanks to socket.io. The following example should work in Chrome, Firefox, Opera, and IE 5.5+. See the browser compatability page for more info.

In this example, the client code runs on the browser and requests the server's timesTen method. The result of the timesTen call is put into the first span element. The client also calls the server's whoAmI method. The server calls the client's name method and performs some regular expression substitions on the result it gets before sending the result back to the client through a callback. The client's name with the substitions applied shows up in the second span element on the page.

Here's an example of what the browser-side code looks like:

<script type="text/javascript" src="/dnode.js"></script> <script type="text/javascript"> DNode({ name : function (f) { f('Mr. Spock') } }).connect(function (remote) { remote.timesTen(10, function (n) { document.getElementById("result").innerHTML = String(n); }); remote.whoAmI(function (name) { document.getElementById("name").innerHTML = name; }); }); </script> <p>timesTen(10) == <span id="result">?</span></p> <p>My name is <span id="name">?</span>.</p>

Here's the server-side compliment to the browser-side code above:

#!/usr/bin/env node var DNode = require('dnode'); var sys = require('sys'); var fs = require('fs'); var http = require('http'); var html = fs.readFileSync(__dirname + '/web.html'); var js = require('dnode/web').source(); var httpServer = http.createServer(function (req,res) { if (req.url == '/dnode.js') { res.writeHead(200, { 'Content-Type' : 'text/javascript' }); res.end(js); } else { res.writeHead(200, { 'Content-Type' : 'text/html' }); res.end(html); } }); httpServer.listen(6061); DNode(function (client) { this.timesTen = function (n,f) { f(n * 10) }; this.whoAmI = function (reply) { client.name(function (name) { reply(name .replace(/Mr\.?/,'Mister') .replace(/Ms\.?/,'Miss') .replace(/Mrs\.?/,'Misses') ); }) }; }).listen({ protocol : 'socket.io', server : httpServer, transports : 'websocket xhr-multipart xhr-polling htmlfile'.split(/\s+/), }).listen(6060);

Installation

DNode is available on npm, a package manager for node.js libraries. You can install dnode by typing:

npm install dnode

Or you can check out the repository and link your development copy with npm:

git clone http://github.com/substack/dnode.git cd dnode npm link .

DNode depends on Socket.IO-node, bufferlist and traverse, all of which are available on npm and are automatically installed when you type `npm install dnode`.

You can fork and follow DNode on github.

And another thing

I've documented the DNode protocol on the README to make it easier to implement the DNode protocol in other languages. Stay tuned.

Special thanks goes out to pkrumins for getting the DNode browser code to work in Internet Explorer.

Thanks for reading. Enjoy this complimentary robot free of charge:

message-passing robot
Tree Traversal and Transformation in Javascript [ 2010-07-09 07:26:02 UTC ]

Lately I've been hacking together a crazy remote method invocation system on top of node.js. Expectedly, JSON.stringify doesn't do functions. In arrays functions get turned into null and objects with keys that point at functions are ignored.

> JSON.stringify([ 7, 8, function () {}, 9, { b : 4, c : function () {} } ]) '[7,8,null,9,{"b":4}]'

I needed a way to pull the functions out of arbitrarily complicated objects so that I can send data about them in a seperate JSON field from the primary data structure.

I couldn't find any modules on the web that traverse and transform arbitrarily nested objects in javascript, possibly owing to all the noise around DOM tree traversal, so tonight I wrote a small library: js-traverse.

Here's a small example that uses Traverse to collect all the leaf nodes from a complicated data structure.

#!/usr/bin/env node var sys = require('sys'); var Traverse = require('traverse').Traverse; var acc = []; Traverse({ a : [1,2,3], b : 4, c : [5,6], d : { e : [7,8], f : 9 } }).forEach(function (x) { if (this.isLeaf) acc.push(x); }); sys.puts(acc.join(' ')); /* Output: 1 2 3 4 5 6 7 8 9 */

Here's another simple example that modifies a tree, adding 128 to all negative numbers.

#!/usr/bin/env node var sys = require('sys'); var Traverse = require('traverse').Traverse; var fixed = Traverse([ 5, 6, -3, [ 7, 8, -2, 1 ], { f : 10, g : -13 } ]).modify(function (x) { if (x < 0) this.update(x + 128); }).get() sys.puts(sys.inspect(fixed)); /* Output: [ 5, 6, 125, [ 7, 8, 126, 1 ], { f: 10, g: 115 } ] */

And finally, this code solves the problem from the introduction. It pulls out the functions and puts them into a secondary data structure. This example also replaces the functions with the string "[Function]" so I won't forget that I'm supposed to replace those on the remote end with actual functions.

#!/usr/bin/env node var sys = require('sys'); var Traverse = require('traverse').Traverse; var id = 54; var callbacks = {}; var obj = { moo : function () {}, foo : [2,3,4, function () {}] }; var scrubbed = Traverse(obj).modify(function (x) { if (x instanceof Function) { callbacks[id] = { id : id, f : x, path : this.path }; this.update('[Function]'); id++; } }).get(); sys.puts(JSON.stringify(scrubbed)); sys.puts(sys.inspect(callbacks)); /* Output: {"moo":"[Function]","foo":[2,3,4,"[Function]"]} { '54': { id: 54, f: [Function], path: [ 'moo' ] } , '55': { id: 55, f: [Function], path: [ 'foo', '3' ] } } */

The traversal library is available on github. You can also install this library with npm, a nifty package manager for node.js.

npm install traverse

Happy hacking!


Binary Stream Parsing in Node.js [ 2010-06-01 11:18:15 UTC ]

binary stream

Recently, I've been collaborating on a web-based virtual machine interface to qemu over its VNC interface. The latest prototype uses node.js for the socket.io library, which provides an abstract, portable, and efficient interface for passing messages between a browser and web server.

In order to bring down latency, I needed a way to decode the RFB protocol, but node.js is asynchronous, so I couldn't just do:

var width = sock.read(2); var height = sock.read(2);

I had to collect up lots of tiny buffers emitted asyncronously and treat them all as one for the purposes of parsing, which went something like this:

var buffers = []; var bytes = 0, offset = 0; sock.addListener('data', function (buf) { buffers.push(buf); bytes += buf.length; if (offset == 0 && bytes >= 2) { var width = read(2); } else if (offset == 2 && bytes >= 4) { var height = read(2); } function read (n) { var buffer = new Buffer(n); var current = buffers[offset]; var current_i = 0; for (var i = 0; i < n; i++) { if (current_i >= current.length) { buffers.shift(); current = buffers[0]; current_i = 0; } buffer[i] = current[current_i++]; } offset += n; return buffer; } });

Yikes, what a mess!

To preventatively cull down the impending complexity, I hacked together node-bufferlist to build linked lists of buffers from the network stream, which made the code simpler:

var BufferList = require('bufferlist').BufferList; var bufferList = new BufferList; var state = 0; sock.addListener('data', function (buf) { bufferList.push(buf); if (state == 0 && bufferList.length >= 2) { var width = read(2); state ++; } else if (state == 1 && bufferList.length >= 2) { var height = read(2); state ++; } function read (n) { var s = bufferList.take(n); bufferList.advance(2); return s; } });

But even with this tool, keeping track of the parser state by hand was very error-prone and ugly. Plus, the real RFB parser would need branches, loops, and nested parsing.

method chain

Borrowing some ideas from haskell's binary Get monad and ruby's methodchain gem, I built a nifty fluent interface for monadic, asynchronous binary bufferlist parsing. Said more simply in code:

var BufferList = require('bufferlist').BufferList; var Binary = require('bufferlist/binary').Binary; var bufferList = new BufferList; Binary(bufferlist) .getWord16be('width') .getWord16be('height') .tap(function (vars) { var width = vars.width; var height = vars.height; // ... // You can even start a new chain inside this block! }) .end() ; sock.addListener('data', function (buf) { bufferList.push(buf); });

Sooooooo much better. Nicer still, with this approach I was able to add goodies like when, unless, repeat, forever, and into, along with some nifty EventEmitter hooks. These abstractions made the resulting node-rfb far prettier and maintainable than it would have been otherwise.

For the code and more examples, you can checkout node-bufferlist on github.


The Russian Doll Pattern in Haskell [ 2010-05-12 05:56:18 UTC ]

The Russian Doll Pattern occurs when functions replace themselves. The example in that link uses javascript, but this technique should be applicable to any language with first-class functions and mutable containers.

nesting robots

Even haskell, with its static types and referential transparency, is capable of emulating this pattern. Here's an example:

-- RussianDoll.hs -- Russian Doll principle in haskell module Main where import Control.Concurrent.MVar import Control.Monad (join) main :: IO () main = do fn <- newEmptyMVar putMVar fn $ do putStrLn "First!" swapMVar fn (putStrLn "Again!") >> return () join $ readMVar fn join $ readMVar fn join $ readMVar fn

Which when executed prints:

First! Again! Again!

An empty MVar is created and bound to fn. MVars are especially convenient for this example since they have an empty state built-in. The first time fn is executed, "First!" is printed, but then the fn MVar is swapped for a new action that prints "Again!".

The inferred type for fn is MVar (IO ()), so join can be used to execute the IO () action inside the MVar.

ghci> :t join join :: (Monad m) => m (m a) -> m a

which means

:t join (readMVar (undefined :: (MVar (IO ())))) join (readMVar (undefined :: (MVar (IO ())))) :: IO ()

Neat!

It's usually a better idea in these cases to use the shared mutable state to delegate to functions instead of storing and swapping the functions themselves. I am however optimistic that this approach could be used in some code to build a more beautiful abstraction.


Maxwell Equations [ 2010-04-28 05:07:52 UTC ]

Maxwell Equations

James Clerk Maxwell was a pretty scruffy-looking guy. I imagine him panhandling in a dingy subway station deep underground.