#75: Travis CI, Riak, and more with Josh Kalderimis and Mathias Meyer

Wynn caught up with Josh Kalderimis and Mathias Meyer from Travis CI to talk about hosted CI in the sky, scaling apps, and a little Riak. Items mentioned in the show: Travis CI a hosted continuous integration service for the open source community. Josh Kalderimis is a core Travis CI team member, Rails contributor, gem […]

#54: CouchBase, Erlang, and Mergers with J Chris Anderson

Wynn sat down with J Chris Anderson from CouchBase to talk about CouchDB, the merger with Membase, Erlang, and bringing NoSQL to PHPers. Items mentioned in the show: J Chris Anderson is a Couchbase cofounder, Mobile Architect, CouchDB committer, new dad The CouchDB music video (served up from CouchDB no less) Chris sings the official […]

Episode 0.5.1 – MongoDB, NoSQL, and Web Scale with Eliot Horowitz

Steve and Wynn sat down with Eliot Horowitz from 10gen to talk about MongoDB, the NoSQL landscape, and the fun of building at Web Scale. Items mentioned in the show: Eliot Horowitz CTO and Co-Founder of 10gen Dwight Merriman CEO & Co-Founder at 10gen NoSQL is a loose term for Key Value Stores, Graph Databases, […]

alfred: Fast in-process key-value store for node.js #

Node.js developers looking for a NoSQL backend for their apps have a growing list of options. New drivers and ODMs are in active development for popular servers from MongoDB to Redis. Pedro Teixeira (of Fugue fame) says with Node’s blazing speed, why not keep everything in-process? Alfred.js promises to be a super fast key-value store for Node.js.

Installation and usage

Alfred can be installed via npm

npm install alfred

… and require it:

var Alfred = require('alfred');

Alfred uses append-only files to store data, offering safe rsync‘ing. To open a database connection, call open and pass the path to your file, querying in the callback, Node.js style:

Alfred.open('path/to/db', function(err, db) {
  if (err) { throw err; }
  // find and log all users with age > 25 and <= 40
  db.users.find({age: {$gt : 25, $lte: 40}}) (function(err, user) {
    if (err) { throw err; }
    console.log(user);
  });
});

Alfred provides an Object-Document-Mapper style access as well:

Alfred.open('path/to/db', function(err, db) {
  if (err) { throw err; }

  // define User model and its properties
  var User = db.define('User', {
    indexes: [{name: 'age',
              fn-340: function(user) { return user.age; }]
  });
  User.property('name', 'string', {
    maxLength: 100
  });
  User.property('active', 'boolean');

  // get user by id
  User.get(id, function(user) {
    console.log(user.inspect());
  };

  // find users
  User.find({age: {$gt: 18}}).all(function(users) {
    console.log('Found ' + users.length + ' users with more than 18 years') ;
  });
});

Be sure and check out the growing list of api docs for advanced usage.

Speed

Pedro has included an impressive list of benchmarks and the code behind them.

benchmarks

Alfred’s features include master-slave replication, streams, and in-memory indexes. If you want to help out with master-master replication or items on the roadmap, get forkin’.

[Source on GitHub] [Pedro’s Blog post]

toystore: Ruby mapper for key-value data stores, and darn near anything #

NewToy (now Zynga with Friends) the folks that brought you the wildly popular mobile game Words With Friends, also have served up some open source code for your enjoyment. John Nunemaker and Geoffrey Dagley have created Toy Store, an ORM that promises to let you completely change your data store in a couple lines of code.

Words With Friends

Toy Store features include:

  • Attributes – attribute :name, String (or some other type) Can be virtual which works just like attr_accessor but all the power of dirty tracking, serialization, etc. Also, can be abbreviated which means :first_name could be the method you use, but in the data store the attribute is :fn. Save those bytes! Allows for default values and defaults can be procs.
  • Typecasting – Same type system as MongoMapper. One day they will share the exact same type system in its own gem, for now duplicated.
  • Callbacks – all the usual suspects.
  • Dirty Tracking – save, create, update, destroy
  • Mass assignment security – attr_accessible and attr_protected
  • Proper cloning
  • Lists – arrays of ids. If user has many games, user would have list :games which stores in game_ids key on user and works just like an association.
  • Embedded Lists – array of hashes. More consistent than MongoMapper, which will soon reap the benefits of the work on Toy Store embedded lists.
  • References – think belongs_to by a different (better?) name. Post model could reference :creator, User to add creator_id key and relate creator to post.
  • Identity Map – On by default. Should be thread-safe.
  • Read/write through caching – If you specific a cache adapter (say memcached), ToyStore will write to memcached first and read from memcached first, populating the cache if it was not present.
  • Indexing – Need to do lookups by email? index :email and whenever a user is saved the user data is written to one key and the email is written as another key with a value of the user id.
  • Logging
  • Serialization (XML and JSON)
  • Validations
  • Primary key factories

Toy Store sits atop of the newly announced Adapter gem, an Active Model-based adapter that provides a unified interface to most NoSQL stores or darn near anything, in the vein of Moneta.

John has created adapters for Riak, memcached, Redis, and Cassandra. If you need support for another store, you can build your own adapter by implementing just four methods for read, write, delete, and clear:

Adapter.define(:memory) do
  def read(key)
    decode(client[key_for(key)])
  end

  def write(key, value)
    client[key_for(key)] = encode(value)
  end

  def delete(key)
    client.delete(key_for(key))
  end

  def clear
    client.clear
  end
end

[Source on GitHub] [John’s Blog post] [Discuss on Hacker News]

webdis: HTTP + JSON API for Redis #

HTTP is the dial tone of the web. Apps that speak HTTP tend to grow in popularity, such as CouchDB whose built-in HTTP-based RESTful API makes it super easy to store and retrieve JSON data.

Nicolas Favre-Felix gives some web love to Redis fans with Webdis, an HTTP interface for Redis that serves up JSON. Built in C, Webdis aims to be fast like Redis itself. The URL structure takes the pattern of:

http://127.0.0.1:7379/COMMAND/KEY/[VALUE]

Let’s look at some concrete examples:

curl http://127.0.0.1:7379/SET/hello/world
→ {"SET":[true,"OK"]}

curl http://127.0.0.1:7379/GET/hello
→ {"GET":"world"}

curl -d "GET/hello" http://127.0.0.1:7379/
→ {"GET":"world"}

While still early in development, the feature set is impressive. Webdis currently offers:

  • Support for GET and POST
  • JSON and JSONP parameter (?jsonp=myFunction).
  • Raw Redis 2.0 protocol output with .raw suffix
  • HTTP 1.1 pipelining
  • TCP or UNIX socket Redis connections
  • CIDR or HTTP Basic Auth security
  • Pub/Sub using Transfer-Encoding: chunked. Coupled with JSONP, Webdis can be used as a Comet server
  • Built-in support for json, txt, ‘html, xml, xhtml, png, and jpg
  • Custom Content-Type types based on file extension, or ?type=some/thing
  • Cross-origin XHR, if compiled with libevent2 (for OPTIONS support)
  • File upload with PUT, if compiled with libevent2 (for PUT support)

Check the README for the complete list. If you’d like to contribute, Nicolas is thinking about how to add other HTTP verb support like PUT and DELETE, Websockets, and Etags.

Also, if you missed it, be sure and catch Episode 0.4.5 with @antirez, the creator of Redis.

[Source on GitHub] [Comment on Hacker News]

#45: Redis with Salvatore Sanfilippo

Wynn caught up with Salvatore Sanfilippo aka @antirez to talk about Redis, the super hot key value store. Items mentioned in the show: VMWare signs the paychecks for Salvatore and Pieter Noordhuis Redis is an open source, advanced key-value store and data structure server wherein keys can contain strings, hashes, lists, sets and sorted sets […]

Watch Wednesday

Quick hitting list of projects to check out on GitHub: pauldowman / gitmodel Git Model is an ActiveModel-compatible NoSQL store based on git. Not ready for primetime but very interesting. assaf / vanity Vanity is declarative A/B testing for Rails. sidelab / tile5 Tile5 is a nifty multi-provider mobile-optimized mapping framework for building mapping or […]

#40: Riak revisited with Andy Gross, Mark Phillips, and John Nunemaker

Wynn sat down with Andy Gross and Mark Phillips of Basho and John Nunemaker of Ordered List to talk about Riak, Riak Search, and moving an open source community to GitHub. Items mentioned in the show: NoSQL smackdown, live from SXSW 2010. Are you web scale? Drop us a ping@thechangelog.com and let us know who […]

coffeeapp: Have some CoffeeScript on the CouchApp #

One of the cool things about CouchDB is its built-in web server which lets you easily create all-in-one HTML5 and JavaScript applications that J Chris Anderson calls CouchApps.

If CoffeeScript (covered in Episode 0.2.9) is your preferred method to sling JavaScript these days, check out CoffeeApp from Andrzej Sliwa. CoffeeApp is a simple wrapper on top of CouchApp that lets you write your apps in Coffee as well as manage multiple releases of your apps with deployment snapshots.

Assuming you’ve gotten CoffeeScript installed including its Node.js dependencies, you can install CoffeeApp via the excellent npm.

npm install coffeeapp

You can then invoke coffeeapp with the normal couchapp options

coffeeapp [options]

CoffeeApp maintains a .releases folder where your *.coffee files are converted to to *.js on-the-fly before pushing to CouchDB.

**UPDATE

Andrzej has made a flurry of changes yesterday including some nifty generators:

$ coffeeapp cgenerate view myview

CoffeeApp (v0.0.5) - simple coffee-script wrapper for CouchApp (http://couchapp.org)
http://github.com/andrzejsliwa/coffeeapp

Running CoffeeApp 'view' generator...
 * creating myview/map.coffee...
 * creating myview/reduce.coffee...
done.

[Source on GitHub]

ACLatraz: Redis-powered access control for your ruby apps #

Authentication options get a lot of press these days, but there is another Auth that can still be a pain: Authorization. ACLatraz from Kriss Kowalik caught our eye because it’s inspired by *nix Access Control Lists (ACLs), powered by Redis, and has a sense of humor.

Install ACLatraz via Ruby gems

gem install aclatraz

and configure your Redis-based storage

Aclatraz.init :redis, "redis://localhost:6379/0"

Everyone is a Suspect

In keeping with the Alcatraz theme, actors in your authorization system are deemed Suspects:

class Account < ActiveRecord::Base
  include Aclatraz::Suspect
end

ACLatraz supports global, class-related, and object-related roles:

# global admin role
@account.roles.assign(:admin) # or ...
@account.is.admin!

# Page class-related role
@account.roles.assign(:responsible, Page) # or...
@account.is.responsible_for!(Page)

# object-related role for page 15
@account.roles.assign(:author, Page.find(15)) # or...
@account.is.author_of!(Page.find(15))

Once, assigned you can interrogate your suspects a couple of ways using has?

@account.roles.has?(:admin)                # => true
@account.roles.has?(:responsible, Page)     # => true
@account.roles.has?(:author, Page.find(15) # => true

… or the more natural semantic shortcuts:

@account.is_not.admin?                      # => false
@account.is_not.responsible_for?(Page)       # => false

Guarding The Rock

To enable access control on an object, include the Aclatraz::Guard module:

class Page
  include Aclatraz::Guard

  suspects :account do
    deny all # notice that it's a method, not symbol
    allow :admin
  end
end

Check the README for even more features including custom actions, aliases, and class inheritance.

[Source on GitHub]

mongomatic: Minimal Ruby mapper for Mongo #

If you’re a close-to-the-metal sort of developer who eschews conveniences like relationships, indexes, and query APIs, then check out Mongomatic from Ben Myles. Mongomatic aims to do ‘just enough’ by mapping your models to MongoDB collections but leaves the rest to you:

  • No additional query API. You simply drop down to the Ruby driver.
  • No relationships. Simply write your own finder methods.
  • No validations. Unless you write your own.

What’s the upside you may ask? Minimal dependencies and better alignment with MongoDB native conventions.

A sample model

require 'mongomatic'

class User < Mongomatic::Base
  def validate
    self.errors << ["Name", "can't be empty"]  if self["name"].blank?
    self.errors << ["Email", "can't be empty"] if self["email"].blank?
  end
end

# set the db for all models:
Mongomatic.db = Mongo::Connection.new.db("mongomatic_test")
# or you can set it for a specific model:
User.db = Mongo::Connection.new.db("mongomatic_test_user")

Find a single user:

found = User.find_one({"name" => "Ben Myles"})
=> #<User:0x00000101939a48 @doc={"_id"=>BSON::ObjectID('4c32834f0218236321000001'), "name"=>"Ben Myles", "email"=>"me@somewhere.com"}, @removed=false, @is_new=false, @errors=[]>

Iterate over a cursor, the MongoDB way:

cursor = User.find({"name" => "Ben Myles"})
=> #<Mongomatic::Cursor:0x0000010195b4e0 @obj_class=User, @mongo_cursor=<Mongo::Cursor:0x80cadac0 namespace='mongomatic_test.User' @selector={"name"=>"Ben Myles"}>>
found = cursor.next
=> #<User:0x00000101939a48 @doc={"_id"=>BSON::ObjectID('4c32834f0218236321000001'), "name"=>"Ben Myles", "email"=>"me@somewhere.com"}, @removed=false, @is_new=false, @errors=[]>
found.remove
=> 67
User.count
=> 0
User.find({"name" => "Ben Myles"}).next
=> nil

If you need a quick-and-dirty model for your MongoDB Ruby app, give Mongomatic a look. It looks like a lightweight alternative to MongoMapper and Mongoid.

[Source on GitHub] [Homepage]

nStore – easy database written for node.js in node.js #

Ever wanted to fire up a quick node.js app, but didn’t want to mess with finding a database driver that worked with the latest node and then installed and set up that database. Ever wanted the simplicity of an in-process db engine (like sqlite).

Wait no more, nStore is here. It’s written in pure js and is contained in a single js file to include in your node project. It supports full CRUD operations on key based documents.

The data is persisted to disk using a smart append-only format that’s easily recoverable by design. You can update/insert documents by key, read documents by key, remove documents by key… (get the picture, it’s key/value based).

Also you can build a node stream interface from a document collection or query the collection using a filter function and get the results as a single callback.

Removing stale data is easy and efficient too. Just supply a filter function to the automatic data compacter and it will remove the offending documents while compacting the data file. Thus the file is never needlessly bloated with delete lines.

Here is a short example that creates a database, inserts a document, and pulls it back out.

// Load the library
var nStore = require('nstore');
// Create a store
var users = nStore('data/users.db');

// Insert a new document with key "creationix"
users.save("creationix", {name: "Tim Caswell": age: 28}, function (err) {
    if (err) { throw err; }
    // The save is finished and written to disk safely
});

// Then later pull it back out
users.get("creationix", function (err, doc, meta) {
  // do something with the data
})

Try is out as a standalone database, or use it with Connect as a session store. http://github.com/creationix/nstore-session

[Source on GitHub]

Video: NoSQL Smackdown Part 4

The last of our NoSQL Smackdown series features CouchDB contributor J Chris Anderson tagging in for Jan to talk about what makes CouchDB development so cool. [Download] [iPhone version]

NoSQL is About… #

Great thoughts from Jan on finding a consistent message for NoSQL:

For me, NoSQL is about choice: The possibility to choose a storage system that fits your project’s needs instead of sticking to a one-size-fits-all-solution that really isn’t one.

Video: NoSQL Smackdown Part 3

The third installment of our NoSQL Smackdown video series asks if only the world’s largest sites have big data needs. Werner says the amount of social interaction on today’s web means even low volume sites have to deal with a lot of data. [Download] [Part 1] [Part 2] [Complete event audio]

Video: NoSQL Smackdown Part 2

Hot on the heels of Part 1, our next video installment of the NoSQL Smackdown has Werner telling you you’re crazy if you run your own database. Ready FIGHT! [Download] [iPhone/iPod version] [Complete EP 0.1.8 audio]

Video: NoSQL Smackdown Part 1

By popular demand we’re posting the video behind Episode 0.1.8 NoSQL Smackdown from SXSW in a series of bite-sized chunks. In part one we talk about massively large documents, inside massively large databases, consistency models and the impact on application design. Enjoy! [Download] [iPhone/iPod version]

#19: James Edward Gray II on Ruby, TextMate, and Red Dirt Ruby Conf

While in OKC for OpenBeta4, Adam and Wynn sat down with James Edward Gray II and talked about his many Ruby gems, TextMate bundles, and his upcoming Ruby conference Red Dirt Ruby Conf this May. RubyQuiz.com Challenge your Ruby fu, feel dumb, learn something, repeat. OpenBeta4 We were blown away by the startup community in […]

#18: NoSQL Smackdown!

While at SXSW Interactive, Adam and Wynn got to attend the Data Cluster Meetup hosted by Rackspace and Infochimps. Things got a bit rowdy when the panel debated features of Cassandra, CouchDB, MongoDB and Amazon SimpleDB and started throwing dirt at everybody else’s favorite NoSQL databases. The participants: Stu Hood from Cassandra Jan Lehnardt from […]

NoRM – Bringing MongoDB to .NET, LINQ, and Mono #

Just because you’re slinging C# doesn’t mean that Microsoft SQL Server is the only database in town, you know. Want to play with MongoDB? Then give NoRM from Andrew Theken a look. NoRM aims to:

  • Wrap the standard MongoDB operations in a strongly-typed interface
  • Provide ultra fast serialization of BSON to .NET CLR types and back
  • LINQ-to-Mongo support
  • Mono compatability

A quick code example from the Wiki:

//open collection
var coll = (new MongoServer()).GetDatabase("Northwind")
                  .GetCollection<Product>();
//create a new object to be added to the collection
var obj = new Product();
obj._id = BSONOID.NewOID();
obj.Title = "Shoes";
//save the object
coll.Insert(obj);
//find the object
var obj2 = coll.FindOne(new { _id = obj._id}).First();

Glad to see Northwind Traders has upgraded to MongoDB…

[Source on GitHub]

#14: Andy Gross and Sean Cribbs on Riak

Adam and Wynn caught up with Andy Gross from Basho and Sean Cribbs, a freelance Ruby developer, to discuss Riak, the new Erlang-based NoSQL store and Ripple, Sean’s new Ruby wrapper for Riak. Items mentioned in the show: Ripple New Ruby wrapper for Riak Link walking in Riak RadiantCMS Ruby-powered Content Management System