The Changelog

Open Source moves fast. Keep up.

Would you like some Mongo with your Postgres? #

Speaking of PostgreSQL, ToroDB is a JSON database that runs on top of Postgres.

JSON documents are stored relationally, not as a blob/jsonb. This leads to significant storage and I/O savings. It speaks natively the MongoDB protocol, meaning that it can be used with any mongo-compatible client.

MongoDB client compatibility. Smart. Still early days, though:

ToroDB follows a RERO (Release Early, Release Often) policy. Current version is considered a “developer preview” and hence is not suitable for production use. However, any feedback, contributions, help and/or patches are very welcome.

JSON API now has a MIME type #

If you hadn’t heard, JSON API is a format that myself and Yehuda Katz have been persuing to help standardize APIs. It came out of the work done on ember-data and active_model_serializers.

We’ve been tweaking the format after receiving feedback from many people, and both the exemplar implementations as well as a variety of other implementations have made it pretty easy to use our format.

Here’s the news, though: we’re one step further along the standardization process.

Recently, the IANA approved our application for a media type designation for JSON API. This means you can officially serve your APIs with the type application/vnd.api+json. Neat!

Hyperresource: a hypermedia API client for Ruby #

Any of you who know me aren’t surprised about me posting this story: it’s right up my alley!

Basically, hyperresource is intended to be a general hypermedia client. What’s that mean? It can work with any API that happens to use one of the media types it supports. Right now, that’s just HAL+JSON, but more will be added in the future.

What’s that mean in terms of code? Well, here’s an example from the README:

api = '')
# => #<HyperResource:0xABCD1234 @root="" @href="" @namespace=nil ... >
root = api.get
# => #<HyperResource:0xABCD1234 @root="" @href="" @namespace=nil ... >
# => { 'message' => 'Welcome to the API',
#      'version' => 1,
#      '_links' => {
#        'self' => {'href' => '/'},
#        'users' => {'href' => '/users{?email,last_name}', 'templated' => true},
#        'forums' => {'href' => '/forums{?title}', 'templated' => true}
#      }
#    }
jdoe_user = api.users.where(email: "").first
# => #<HyperResource:0x12312312 ...>

How does it know how to look up those users? Well, there’s a users link relation on in the response, and it has a templated URI as its href attribute. This lets the client reflect and know how to look up users by their email or last name.

The power of this generic tooling is that we no longer have to think about how to parse HAL, how to interpret the response… we have this one library which works with any HAL-powered API.

The biggest drawback of hyperresource is that it’s read-only at the moment, nothing but GET. But it shows a lot of promise, and I’m excited to see this kind of stuff pop up in the wild. Everything I see points to 2013 as the year of example code for hypermedia services.

Check it out on GitHub.

JWalk, a command-line JSON inspector #

Sometimes inspecting JSON files can be a huge pain, which is exactly why Nate Kohari is created JWalk, a tiny NPM package that makes JSON inspection incredibly simple. First we will need a file to work with, if you don’t have one, simply run this at the command line to get some of our tweets @thechangelog in JSON format.

> curl -G -d count=5 -d include_rts=1 > sample.json

You will need node.js and NPM installed to use JWalk, but the setup process is simple:

> npm -g install coffee-script
> npm -g install jwalk

After installing jwalk, start up an interactive session (using our sample JSON file):

> jwalk sample.json

The command set is limited, but it amounts to exactly what you need for inspection:

  • cd: jump into a specific node
  • ls: view the contents of the current node
  • exit: leave the session

For example, if you want to see the text of the first tweet returned from our cURL:

> cd 0
> cd text
> ls

Rails::API – Lightweight Rails stack for headless API apps #

“Rails is just an API,” say Alex (featured on Episode 0.7.1) and others. Rails core team member Santiago Pastorino might not endorse the “just” part, but his Rails::API aims to provide a slimmer stack for those wanting to use Rails as an API backend for their client-side or mobile applications. The result is a fine-tuned subset of Rails, letting the developer opt-in to just the middleware needed. Rails::API requires a small change to your controllers:

# instead of
class ApplicationController < ActionController::Base

# do
class ApplicationController < ActionController::API

Check out the README for installation and advanced usage and Santiago’s blog post for more.

We’ll be covering a similar project, Lightrail, when we chat with Tony Arcieri about Celluloid next week.

MBRequest – Easier API wrappers in iOS and OSX #

A good API wrapper should handle network transport, payload serialization/deserialization, and authentication, abstracting these details away in order to let the developer deal with the business domain of the API. Projects like Faraday, Requests, and others have made creating higher level wrappers much easier.

MBRequest from Mobiata does the same for iOS and OSX:

NSURL *url = [NSURL URLWithString:@""];
NSMutableURLRequest *urlRequest = [NSMutableURLRequest requestWithURL:url];
[urlRequest setValue:@"gzip" forHTTPHeaderField:@"Accept-Encoding"];
MBJSONRequest *jsonRequest = [[[MBJSONRequest alloc] init] autorelease];
[jsonRequest performJSONRequest:urlRequest completionHandler:^(id responseJSON, NSError *error) {
    if (error != nil)
        NSLog(@"Error requesting top-rated videos: %@", error);
        NSArray *videos = [[responseJSON objectForKey:@"feed"] objectForKey:@"entry"];
        for (NSDictionary *videoInfo in videos)
            NSString *title = [[videoInfo objectForKey:@"title"] objectForKey:@"$t"];
            NSString *author = [[[[videoInfo objectForKey:@"author"] objectAtIndex:0] objectForKey:@"name"] objectForKey:@"$t"];
            NSLog(@"'%@' by %@", title, author);

Check out the examples folder in the project source for implementation ideas.

A few tools to craft JSON output in a Ruby web app API #

It seems a solid API that returns JSON is almost a prerequisite for any new web app these days. Often these JSON data structures return nested associated data, composed from several models in the system. There has been some discussion this week around tools and approaches to building JSON data structures in a Ruby web app API.

Perhaps the biggest buzz this week has been around Jbuilder which DHH announced on Twitter. While the world doesn’t need another J_Anything_, it does provide a nice API that can be used stand alone from controllers or as a view template:

Jbuilder.encode do |json|
  json.content format_content(@message.content)
  json.(@message, :created_at, :updated_at) do |json|
    json.email_address @message.creator.email_address_with_name
    json.url url_for(@message.creator, format: :json)

  if current_user.admin?
    json.visitors calculate_visitors(@message)


… which produces:

  "content": "<p>This is <i>serious</i> monkey business",
  "created_at": "2011-10-29T20:45:28-05:00",
  "updated_at": "2011-10-29T20:45:28-05:00",

  "author": {
    "name": "David H.",
    "email_address": "'David Heinemeier Hansson' <>",
    "url": ""

  "visitors": 15


Another promising gem is Boxer from the team at Gowalla. While tied more closely to ActiveRecord and Rails, Boxer lets you define multiple views for an object, allowing you to expose extended attributes, usually based on permissions: do |box, user|
  box.view(:base) do
      :name =>,
      :age  => user.age,

  box.view(:full, :extends => :base) do
      :email      =>,
      :is_private => user.private?,

Finally John Nunemaker shared his thoughts learned building the API on his popular Rails Tips blog. John uses the Presenter pattern to craft the JSON output.

For even more projects in this space, check out the JBuilder README which provides a list of resources at the bottom. The most mature of these looks to be RABL from Nathan Esquenazi.

csonv.js: Fetch and transform CSV data into JSON #

Since it’s typed, human readable, and supported darn near everywhere,
JSON is the new hotness for data transport formats.
Unfortunately, many systems don’t expose a JSON API. Relational data is
often represented in the tried and true CSV format.

Paul Engel has introduced
CSONV.js, a JavaScript library
that can consume remote CSV data and transform it to JSON, a more
client-side developer friendly format.

Consider the following delimited files:

# books.csv
1;To Kill an Angry Bird;1
2;The Rabbit;2
4;The Lord of the Things;2
5;The Michelangelo Code;4

# authors.csv
1;Harper Lee;
2;JRR Tolkien;
3;William Shakespeare;
4;Dan Brown;

CSONV.js can transform these two relational sources into the following JSON:

    "id": 1,
    "name": "To Kill an Angry Bird",
    "author": {
      "id": 1,
      "name": "Harper Lee"
    "id": 2,
    "name": "The Rabbit",
    "author": {
      "id": 2,
      "name": "JRR Tolkien"
    "id": 3,
    "name": "Parslet",
    "author": {
      "id": 3,
      "name": "William Shakespeare"
    "id": 4,
    "name": "The Lord of the Things",
    "author": {
      "id": 2,
      "name": "JRR Tolkien"
    "id": 5,
    "name": "The Michelangelo Code",
    "author": {
      "id": 4,
      "name": "Dan Brown"

Be sure and check out the project on
for complete feature list
and advanced usage info.

[Source on GitHub]

jsonpipe: Convert JSON to a UNIX-friendly line-based format using Python #

As the web seems to be moving to JavaScript Object Notation over XML for data transfer, it’s nice to find tools to help you work with JSON from the command line. Changelog listener Jason Williams pointed us to JSON Pipe, a simple Python command line utility for visualizing JSON document structure.

Installation and usage

Install JSON Pipe via pip

pip install jsonpipe

Now we can pipe data into jsonpipe to see a breakdown of the structure of the document:

$ echo '[{"a": [{"b": {"c": ["foo"]}}]}]' | jsonpipe
/   []
/0  {}
/0/a        []
/0/a/0      {}
/0/a/0/b    {}
/0/a/0/b/c  []
/0/a/0/b/c/0        "foo"

How about inspecting an NPM package?

curl | jsonpipe

/ {}
/name "coffee-script"
/description  "Unfancy JavaScript"
/keywords []
/keywords/0 "javascript"
/keywords/1 "language"
/keywords/2 "coffeescript"
/keywords/3 "compiler"
/author "Jeremy Ashkenas"
/version  "1.1.0-pre"
/licenses []
/licenses/0 {}
/licenses/0/type  "MIT"
/licenses/0/url ""
/engines  {}
/engines/node ">=0.2.5"
/directories  {}
/directories/lib  "./lib"
/main "./lib/coffee-script"
/bin  {}
/bin/coffee "./bin/coffee"
/bin/cake "./bin/cake"
/homepage ""
/repository {}
/repository/type  "git"
/repository/url "git://"

Since the output is regular STDOUT, JSON Pipe is piping hot:

curl | jsonpipe | grep "homepage"

/homepage   ""

JSON Pipe will show you the key/value structure of your JSON documents faster than you can say XPath.

[Source on GitHub]

#54: CouchBase, Erlang, and Mergers with J Chris Anderson

Wynn sat down with J Chris Anderson from CouchBase to talk about CouchDB, the merger with Membase, Erlang, and bringing NoSQL to PHPers. Items mentioned in the show: J Chris Anderson is a Couchbase cofounder, Mobile Architect, CouchDB committer, new dad The CouchDB music video (served up from CouchDB no less) Chris sings the official […]

applitude: Elegant DSL to create iPhone apps in Eclipse #

While language purists might balk at the growing popularity of projects that compile to Objective-C, the verbosity of the Cocoa framework has many developers looking for a faster way to create iPhone apps. Let’s face it, the vast majority of apps in the App Store follow a familiar pattern of making a network request, showing an activity indicator, parsing a JSON response, handling errors, loading a UITableView, and handling navigation to the next bit of data.


Applitude is based on Applause, a nifty Domain Specific Language from Heiko Behrens and Peter Friese based on Xtext for creating iOS apps. Ralf Ebert has pared down Applause to have an iPhone focus to create Applitude. Here’s a sample:

application Demo {

tabview Tabs {
  tab {
    title: "Inventors"
    view: Inventors()

type String mapsTo "NSString"

entity Inventor {
  String name
  String imageUrl
  Invention[] inventions

entity Invention {
  String name

contentprovider AllInventors returns Inventor[] fetches JSON from
  "" selects ""

tableview Inventors {
  Inventor[] inventors = AllInventors()

  title: "Inventors"

  section {
    cell Default for inventor in inventors {
      image: inventor.imageUrl
      action: InventorDetail(inventor)

tableview InventorDetail(Inventor inventor) {
  style: Grouped

  section {
    cell Value2 {
      text: "Name"

  section {
    title: "Inventions"
    cell Default for invention in inventor.inventions {

This example creates a simple demo app based on some JSON:

    "name" : "Thomas Edison",
    "imageUrl" : "",
    "inventions" : [
      { "name" : "Light bulb" },
      { "name" : "Motion picture" },
      { "name" : "Phonograph" }
    "name" : "Alexander Graham Bell",
    "imageUrl" : "",
    "inventions" : [
      { "name" : "Telephone" }
    "name" : "Nikola Tesla",
    "imageUrl" : "",
    "inventions" : [
      { "name" : "Tesla coil" },
      { "name" : "Alternating current" }

Applitude bundles popular open source iOS frameworks including TouchJSON, TouchXML, and ASIHTTPRequest. Visit the project home page for advanced usage, requirements, and installation instructions

[Comment on Hacker News]

[Source on GitHub] [Web site]

jsondns: DNS JSON API #

If you’re building a cool domain name service or DNS-related mashup, be sure and check out JSON DNS from Joël Franusic. JSON DNS aims to create a standard for DNS info in JSON, and the project includes a small Rack app for serving that data over HTTP.

// curl
  "header": {
    "aa": false,
    "ra": true,
    "cd": false,
    "opcode": "Query",
    "qr": true,
    "arcount": 0,
    "rcode": "NOERROR",
    "tc": false,
    "qdcount": 1,
    "nscount": 0,
    "ad": false,
    "ancount": 1,
    "id": 30523,
    "rd": true
  "answer": [
      "type": "A",
      "class": "IN",
      "ttl": 3600,
      "name": "",
      "rdata": ""
  "question": [
      "qname": "",
      "qtype": "A",
      "qclass": "IN"
  "additional": [

  "authority": [


JSON DNS supports lookups for A, NS, and MX records and even supports JSONP to allow cross domain access.

[Source on GitHub] [Demo] [Discuss on Hacker News]

resty: Piping hot REST client for bash or zsh #

I love REST APIs, so I’m always looking for new tools for the toolbox. Resty from Micha Niskin caught my eye because it can be loaded into your bash or zsh. To install, just download the script and source it.

$ curl > resty
$ . resty

Now your shell is loaded up with the Resty commands. First, set an API endpoint, like GitHub:

$ resty*

Now we can make any normal GET, POST, PUT, or DELETE as you would expect.

GET /pengwynn.json

For formatted output, pipe it to pretty print:

GET /pengwynn.json | pp

… or save to a file

GET /pengwynn.json > me.json

REST on the Couch

Resty is perfect for CouchDB. Just set your endpoint


… and you’re ready to create datbases

PUT /test '{}'

… and stash documents.

PUT /test/me < me.json


Since Resty sits on top cURL, any additional options after the required path will be passed along to cURL:

GET /repos/show/pengwynn -u pengwynn:0U812

Be sure to check out Jan-Piet Mens’ introductory screencast or the README for advanced usage.

[Source on GitHub] [Screencast]

iJSON: Parse Streams of JSON in Python #

JSON will forever serve as a great alternative for XML, but it has a weakness: big data. This is due to a lack of support for stream processing.

Luckily for Python, there’s now a solution. Ivan Sagalaev developed iJSON, a library for performing SAX-style parsing of JSON.

iJSON allows you to interact with the incoming datastream as a standard iterator.

Example Simple Data use:

from ijson import items

f = urlopen('http://.../')
objects = items(f, 'earth.europe.item')
cities = (o for o in objects if o['type'] == 'city')
for city in cities:

Example Big Data use:

from ijson import parse

f = urlopen('http://.../')
parser = parse(f)
for prefix, event, value in parser:
    if (prefix, event) == ('earth', 'map_key'):
        stream.write('<%s>' % value)
        continent = value
    elif prefix.endswith('.name'):
        stream.write('<object name="%s"/>' % value)
    elif (prefix, event) == ('earth.%s' % continent, 'end_map'):
        stream.write('</%s>' % continent)

To get started:

$ pip install ijson

[PyPi Listing] [GitHub Mirror]

Episode 0.2.6 – Douglas Crockford on JSON and JavaScript from TXJS

While at TXJS, Adam and Wynn caught up with Douglas Crockford, author of both JavaScript: The Good Parts and the JSON spec, and a global namespace unto himself. Items mentioned in the show: JSON – (JavaScript Object Notation) is a lightweight data-interchange format, easy for humans to read and write. The JSON Saga – how […]

JSCouch – Learn CouchDB queries in your browser #

Part CouchDB interactive query console, part tutorial, JSCouch is a neat way to learn CouchDB’s map/reduce functions right in your browser. You can provide some sample JSON (or use the default demo data), write your map and reduce functions, and execute them to see sample results.

JSCouch also includes some sample functions for those learning CouchDB. Just choose a function from the list and read the explanation provided.

Hat tip: Will Leinweber

[Source on GitHub] [Demo]

jsawk – Powerful JSON querying on SpiderMonkey #

As JSON becomes the lingua franca of web-based APIs, you might find yourself needing a way to query large chunks of data in JSON format.

jsawk from Micha Niskin provides awk-like functionality:

cat /data/athletes | jsawk 'this.sports = this.sports.join(",")'

In this example, we read the contents of a JSON file and pipe it into jsawk and concatenate the array of sports into a delimited string.

jsawk also supports the popular JSONQuery syntax with the -q option.

Be sure to check out full installation instructions and extensive examples in the README.

[Source on GitHub]

#14: Andy Gross and Sean Cribbs on Riak

Adam and Wynn caught up with Andy Gross from Basho and Sean Cribbs, a freelance Ruby developer, to discuss Riak, the new Erlang-based NoSQL store and Ripple, Sean’s new Ruby wrapper for Riak. Items mentioned in the show: Ripple New Ruby wrapper for Riak Link walking in Riak RadiantCMS Ruby-powered Content Management System

riak – high performance document store built on JSON and REST #

Riak, REE-ahk, combines the power of Erlang with the ubiquity of JSON to deliver a highly scalable document store over REST. Hold on CouchDB fans, before you pounce, you haven’t heard this one before.

Master-less replication

Riak is architected for master-less replication from the GET-go. All nodes participate equally in the replication so scaling horizontally means just adding new servers to join the party.

Links for powerful object graphs

Riak supports link-walking which is far more powerful than traditional relational DB joins. As Sean Cribbs explains:

Every datum stored in Riak can have one-way relationships to other data via the Link HTTP header. In the canonical example, you know the key of a band that you have stored in the “artists” bucket (Riak buckets are like database tables or S3 buckets). If that artist is linked to its albums, which are in turn linked to the tracks on the albums, you can find all of the tracks produced in a single request. As I’ll describe in the next section, this is much less painful than a JOIN in SQL because each item is operated on independently, rather than a table at a time. Here’s what that query would look like:

GET /raw/artists/TheBeatles/albums,_,_/tracks,_,1

More powerful Map/Reduce

Map/Reduce works a bit differently in Riak. Riak’s Map function expects a list of keys on which it should run. You can of course pass every key in your bucket, but doing Map in this way allows Map to be run on the node where the data is actually stored.

Riak ships with libraries for Python, Ruby, Java, Erlang, and even JavaScript. Want to know more? Stay tuned. Riak will be the focus of an upcoming episode of The Changelog.

[Source on BitBucket] [Homepage] [Brian Fink’s NYC-NoSQL talk]

neo4jr-social: REST interface for Neo4j #

Need to talk to Neo4j, the popular Java NoSQL database but no habla Java? No worries, @mdeiters has your back with Neo4jr-Social:

Neo4jr-Social is a self contained HTTP REST + JSON interface to the graph database Neo4j. Neo4jr-Social supports simple dynamic node creation, building relationships between nodes and also includes a few common social networking queries out of the box (i.e. linkedin degrees of seperation and facebook friend suggestion) with more to come.

[Source on GitHub] [API docs]

Lawnchair, a lightweight client-side JSON document store #

Sorta like a couch except smaller and outside, also, a client side JSON document store. Perfect for webkit mobile apps that need a lightweight, simple and elegant persistence solution.


  • micro tiny storage without the nasty SQL: pure and delicious JSON
  • clean and simple oo design with one db table per store
  • key/value store.. specifying a key is optional
    *happily and handily will treat your store as an array of objects
  • terse syntax for searching and therefore finding of objects

[Source on GitHub] [Homepage]