Installing and using LMDB on the Raspberry Pi

In a previous post I wrote about choosing and using LevelDB as the database to store readings from our smart meter. But although I got it up and running, it did not really work out for me. There are two reasons for this:

  1. It is not multi-threaded; only one process can access the database at a time. So I have to take care not to read values from the database at the moment my node.js-client puts a new value on the message queue that I want to be written to the database. It is easy to crash node-red this way or just lose a value. And since I want to save a value every minute this is somewhat of a problem for me. Probably I should write ‘thread-safe code’ but it would be much easier (okay, I’m lazy and not much of a programmer) to have a database that is multi-threaded.
  2. It allows for one series only of key-value pairs. So to save different types of data in the same location the differentiation should be made in the keys used. For example, keys for meter readings for electricity could become something like e~201401091000 while keys for meter readings for gas could take the form of g~201401091000. But to me, that is a bit counter intuitive. I would much rather have a key-value-database that I could create different tables in, or at least create several databases in the same location. Probably the database-engine handles this by creating exactly the kind of keys I detailed above, but it would do so transiently and I wouldn’t have to think about it. Did I already mention that I’m lazy and not much of a programmer?

Searching the internet for alternatives I came across Lightning Memory Mapped Database. The quick overview shows some impressing capabilities and these two caught my attention:

  • Supports multi-thread and multi-process concurrency, environments may be opened by multiple processes on the same host.
  • Multiple sub-databases may be created with transactions covering all sub-databases.

There is some debate whether LMDB is better or faster or not than LevelDB. More about that here and here. But since I want multi-process concurrency and multiple subdatabases I decided I don’t care either way and try it anyway.

Installing LMDB

There are two ways to install LMDB for Node.js that I know off: either as a back-end to LevelUP (here) or as a dedicated binding to node.js (here). Somehow the second seems more ‘in the spirit’ of LMDB so that’s what I installed. To install it, type

npm -g install node-gyp

git clone https://github.com/Venemo/node-lmdb.git

After that go to the freshly created directory

cd node-lmdb

and configure the build by typing

node-gyp configure

Now, normally the next step would be to build the binary. But I found that on the Raspberry Pi the build fails. To get it working I had to open the file “binding.gyp” and in the fragment

"cflags": [
 "-fPIC",
 "-fvisibility-inlines-hidden",
 "-O3",
 "-std=c++11"
 ]

I had to change “-std=c++11″ in “-std=c++0x”. After that I could type

node-gyp build

 and it build without further errors.

Update 1: the file binding.gyp on github is updated to reflect the change mentioned above.

Update 2: Node-lmbd recently became available through npm. Installing node-lmdb can now also be done by typing:

npm install node-lmdb

Using LMDB in Node-Red

To use LMDB in Node-Red I had to move the contents of the directory [..]/node-lmdb/build/Release to the module directory of Node-Red, which on my installation is /home/pi/node-red/node_modules. To use LMDB in a function node in Node-Red it has to be made globally available. To do this open the file settings.js (found in the installation directory of Node_red which is on my installation /home/pi/node-red) and at the end of the file type

functionGlobalContext: {lmdb: require('node-lmdb')}

In code the object can than be accessed by typing

var lmdb = context.global.lmdb;

This object can than be used to create an Environment-object, which encapsulates the location of the database:

//open environent-object
var env =new lmdb.Env();
env.open({
         path: './dbMeter',
         maxDbs: 3  
});

The designated directory should exist, otherwise it will throw an error. With the environment-object in hands, it can be used to open the (sub)database:

//get database-object
var db = env.openDbi({
    name: "dbEAV",
    create: 1
});

Typing another name will open or create another sub-database. The number of possible subdatabase is limited by the number designated in the creation of the Environment-object (maxDbs). Finally start a transaction to put, retrieve or delete any key-value pair you like.

//start transaction
var txn = env.beginTxn();
txn.putNumber(db, key, value);

Using LMDB I have now three (sub)databases: one to store every minute the value of the electricity usage at that moment, one for hourly meter readings for gas and one for hourly meter readings for electricity. One the next steps will be to create charts from the values stored in the (sub)databases. I’m curious to the performance of reading from the database on the Raspberry Pi to create these charts. Will it be in seconds or in minutes?

 

One Thought on “Installing and using LMDB on the Raspberry Pi

  1. Hi,

    It’s very interesting to hear about your story & thanks for sharing your experiences with node-lmdb! :)

Post Navigation