Introducing DNode
Introducing
DNode,
a node.js library for asynchronous bidirectional remote method invocation.
Network socket and websocket-style
socket.io
transports are presently available so system processes can communicate with each
other and with processes running in the browser using the same interface.
Remote method invocation (RMI) is the object-oriented cousin of remote procedure
calls. In RMI, each side of the connection hosts an object that the other side
can call the methods of. A popular example of RMI worth checking out is
Ruby's DRb.
DNode is different from DRb in that all remote method calls are asynchronous.
Instead of returning explicitly, hosted methods pass return values to the other
side of the connection by invoking callbacks that were passed in as arguments.
These callbacks execute on the side of the connection where they were defined,
and a representation of them gets passed to the remote, so there's no eval().
Here's a simple example.
// server:
var DNode = require('dnode');
var server = DNode({
timesTen : function (n,f) { f(n * 10) },
}).listen(6060);
// client:
var DNode = require('dnode');
var sys = require('sys');
DNode.connect(6060, function (remote) {
remote.timesTen(5, function (result) {
sys.puts(result); // 5 * 10 == 50
});
});
Unlike many asynchronous RPC systems, DNode allows the programmer to pass
functions as arguments to remote methods, which when called on the remote end
signal the near side to execute the function with the arguments that the remote
provides.
Callbacks are automatically scrubbed and collected from a recursive walk of the
arguments list, so they can be arbitrarily nested.
Additionally, each side of the connection can call the other side's methods or
the callbacks the other side provides with callbacks of its own or any arguments
that can be serialized to JSON.
Bidirectional Example
Here's an example where the client calls a method on the server that calls a
method on the client.
// server:
var DNode = require('dnode');
DNode(function (client) {
this.timesX = function (n,f) {
client.x(function (x) {
f(n * x);
});
};
}).listen(6060);
// client:
var DNode = require('dnode');
DNode({
x : function (f) { f(20) }
}).connect(6060, function (remote) {
remote.timesX(3, function (res) {
sys.puts(res); // 20 * 3 == 60
});
});
DNode Browser Example
As I mentioned in the first paragraph, websocket-style connections between the
browser and node.js DNode servers are available thanks to
socket.io.
The following example should work in Chrome, Firefox, Opera, and IE 5.5+.
See the
browser compatability page for more info.
In this example, the client code runs on the browser and requests the server's
timesTen method. The result of the timesTen call is put into the first span
element. The client also calls the server's whoAmI method. The server calls the
client's name method and performs some regular expression substitions on the
result it gets before sending the result back to the client through a callback.
The client's name with the substitions applied shows up in the second span
element on the page.
Here's an example of what the browser-side code looks like:
<script type="text/javascript" src="/dnode.js"></script>
<script type="text/javascript">
DNode({
name : function (f) { f('Mr. Spock') }
}).connect(function (remote) {
remote.timesTen(10, function (n) {
document.getElementById("result").innerHTML = String(n);
});
remote.whoAmI(function (name) {
document.getElementById("name").innerHTML = name;
});
});
</script>
<p>timesTen(10) == <span id="result">?</span></p>
<p>My name is <span id="name">?</span>.</p>
Here's the server-side compliment to the browser-side code above:
#!/usr/bin/env node
var DNode = require('dnode');
var sys = require('sys');
var fs = require('fs');
var http = require('http');
var html = fs.readFileSync(__dirname + '/web.html');
var js = require('dnode/web').source();
var httpServer = http.createServer(function (req,res) {
if (req.url == '/dnode.js') {
res.writeHead(200, { 'Content-Type' : 'text/javascript' });
res.end(js);
}
else {
res.writeHead(200, { 'Content-Type' : 'text/html' });
res.end(html);
}
});
httpServer.listen(6061);
DNode(function (client) {
this.timesTen = function (n,f) { f(n * 10) };
this.whoAmI = function (reply) {
client.name(function (name) {
reply(name
.replace(/Mr\.?/,'Mister')
.replace(/Ms\.?/,'Miss')
.replace(/Mrs\.?/,'Misses')
);
})
};
}).listen({
protocol : 'socket.io',
server : httpServer,
transports : 'websocket xhr-multipart xhr-polling htmlfile'.split(/\s+/),
}).listen(6060);
Installation
DNode is available on npm, a package manager for
node.js libraries. You can install dnode by typing:
npm install dnode
Or you can check out the repository and link your development copy with npm:
git clone http://github.com/substack/dnode.git
cd dnode
npm link .
DNode depends on
Socket.IO-node,
bufferlist
and traverse,
all of which are available on npm and are automatically installed when you
type `npm install dnode`.
You can fork and follow
DNode on github.
And another thing
I've documented the DNode protocol
on the README to make it easier to implement the DNode protocol in other
languages. Stay tuned.
Special thanks goes out to pkrumins for
getting the DNode browser code to work in Internet Explorer.
Thanks for reading. Enjoy this complimentary robot free of charge:
Lately I've been hacking together a crazy remote method invocation system on top
of node.js. Expectedly,
JSON.stringify doesn't do functions. In arrays functions get
turned into null and objects with keys that point at functions are ignored.
> JSON.stringify([ 7, 8, function () {}, 9, { b : 4, c : function () {} } ])
'[7,8,null,9,{"b":4}]'
I needed a way to pull the functions out of arbitrarily complicated
objects so that I can send data about them in a seperate JSON field
from the primary data structure.
I couldn't find any modules on the web that traverse and transform arbitrarily
nested objects in javascript, possibly owing to all the noise around DOM tree
traversal, so tonight I wrote a small library:
js-traverse.
Here's a small example that uses Traverse to collect all the leaf nodes from a
complicated data structure.
#!/usr/bin/env node
var sys = require('sys');
var Traverse = require('traverse').Traverse;
var acc = [];
Traverse({
a : [1,2,3],
b : 4,
c : [5,6],
d : { e : [7,8], f : 9 }
}).forEach(function (x) {
if (this.isLeaf) acc.push(x);
});
sys.puts(acc.join(' '));
/* Output:
1 2 3 4 5 6 7 8 9
*/
Here's another simple example that modifies a tree, adding 128 to all negative numbers.
#!/usr/bin/env node
var sys = require('sys');
var Traverse = require('traverse').Traverse;
var fixed = Traverse([
5, 6, -3, [ 7, 8, -2, 1 ], { f : 10, g : -13 }
]).modify(function (x) {
if (x < 0) this.update(x + 128);
}).get()
sys.puts(sys.inspect(fixed));
/* Output:
[ 5, 6, 125, [ 7, 8, 126, 1 ], { f: 10, g: 115 } ]
*/
And finally, this code solves the problem from the introduction.
It pulls out the functions and puts them into a secondary data structure.
This example also replaces the functions with the string "[Function]" so I won't
forget that I'm supposed to replace those on the remote end with actual
functions.
#!/usr/bin/env node
var sys = require('sys');
var Traverse = require('traverse').Traverse;
var id = 54;
var callbacks = {};
var obj = { moo : function () {}, foo : [2,3,4, function () {}] };
var scrubbed = Traverse(obj).modify(function (x) {
if (x instanceof Function) {
callbacks[id] = { id : id, f : x, path : this.path };
this.update('[Function]');
id++;
}
}).get();
sys.puts(JSON.stringify(scrubbed));
sys.puts(sys.inspect(callbacks));
/* Output:
{"moo":"[Function]","foo":[2,3,4,"[Function]"]}
{ '54': { id: 54, f: [Function], path: [ 'moo' ] }
, '55': { id: 55, f: [Function], path: [ 'foo', '3' ] }
}
*/
The traversal library is
available on github.
You can also install this library with npm, a
nifty package manager for node.js.
npm install traverse
Happy hacking!
Recently, I've been collaborating on a web-based
virtual machine interface to
qemu over its VNC interface.
The latest prototype uses node.js for the
socket.io library, which provides an
abstract, portable, and efficient interface for passing messages between a
browser and web server.
In order to bring down latency,
I needed a way to decode the RFB protocol, but node.js is asynchronous,
so I couldn't just do:
var width = sock.read(2);
var height = sock.read(2);
I had to collect up lots of tiny
buffers
emitted asyncronously and treat them
all as one for the purposes of parsing, which went something like this:
var buffers = [];
var bytes = 0, offset = 0;
sock.addListener('data', function (buf) {
buffers.push(buf);
bytes += buf.length;
if (offset == 0 && bytes >= 2) {
var width = read(2);
}
else if (offset == 2 && bytes >= 4) {
var height = read(2);
}
function read (n) {
var buffer = new Buffer(n);
var current = buffers[offset];
var current_i = 0;
for (var i = 0; i < n; i++) {
if (current_i >= current.length) {
buffers.shift();
current = buffers[0];
current_i = 0;
}
buffer[i] = current[current_i++];
}
offset += n;
return buffer;
}
});
Yikes, what a mess!
To preventatively cull down the impending complexity,
I hacked together
node-bufferlist
to build linked lists of buffers
from the network stream, which made the code simpler:
var BufferList = require('bufferlist').BufferList;
var bufferList = new BufferList;
var state = 0;
sock.addListener('data', function (buf) {
bufferList.push(buf);
if (state == 0 && bufferList.length >= 2) {
var width = read(2);
state ++;
}
else if (state == 1 && bufferList.length >= 2) {
var height = read(2);
state ++;
}
function read (n) {
var s = bufferList.take(n);
bufferList.advance(2);
return s;
}
});
But even with this tool, keeping track of the parser state by hand was very
error-prone and ugly.
Plus, the real RFB parser would need branches, loops, and nested parsing.
Borrowing some ideas from haskell's
binary Get monad and ruby's
methodchain gem,
I built a nifty
fluent interface
for monadic, asynchronous binary bufferlist parsing.
Said more simply in code:
var BufferList = require('bufferlist').BufferList;
var Binary = require('bufferlist/binary').Binary;
var bufferList = new BufferList;
Binary(bufferlist)
.getWord16be('width')
.getWord16be('height')
.tap(function (vars) {
var width = vars.width;
var height = vars.height;
// ...
// You can even start a new chain inside this block!
})
.end()
;
sock.addListener('data', function (buf) {
bufferList.push(buf);
});
Sooooooo much better.
Nicer still, with this approach I was able to add goodies like when, unless,
repeat, forever, and into, along with some nifty
EventEmitter
hooks.
These abstractions made the resulting
node-rfb
far prettier and maintainable than it would have been otherwise.
For the code and more examples, you can
checkout node-bufferlist on github.
The Russian
Doll Pattern occurs when functions replace themselves.
The example in that link uses javascript, but this technique should be
applicable to any language with first-class functions and mutable containers.
Even haskell, with its static types and referential transparency, is capable of
emulating this pattern. Here's an example:
-- RussianDoll.hs
-- Russian Doll principle in haskell
module Main where
import Control.Concurrent.MVar
import Control.Monad (join)
main :: IO ()
main = do
fn <- newEmptyMVar
putMVar fn $ do
putStrLn "First!"
swapMVar fn (putStrLn "Again!")
>> return ()
join $ readMVar fn
join $ readMVar fn
join $ readMVar fn
Which when executed prints:
First!
Again!
Again!
An empty MVar is created and bound to fn.
MVars are especially convenient for this example since they have an empty state
built-in.
The first time fn is executed, "First!" is printed, but then the fn MVar is
swapped for a new action that prints "Again!".
The inferred type for fn is MVar (IO ()), so join can be used to execute the
IO () action inside the MVar.
ghci> :t join
join :: (Monad m) => m (m a) -> m a
which means
:t join (readMVar (undefined :: (MVar (IO ()))))
join (readMVar (undefined :: (MVar (IO ())))) :: IO ()
Neat!
It's usually a better idea in these cases to use the shared mutable state to delegate to functions instead of storing and swapping the functions themselves.
I am however optimistic that this approach could be used in some code to build a more beautiful abstraction.
James Clerk Maxwell was a pretty scruffy-looking guy.
I imagine him panhandling in a dingy subway station deep underground.
