Openleveldb¶

Openleveldb is a small pythonic wrapper around Plyvel

Features¶

Transparent object store¶

It works with python objects:

Automatically encodes objects into bytes when saving to leveldb
Automatically decodes bytes into their original type when retrieving objects from leveldb

Supported types include:

int
str
numpy.ndarray
Anything that is serializable by orjson

>>> db['key'] = {'key': [1, 2, 3]}
>>> db['key']
{'key': [1, 2, 3]}

Python dict-like protocol¶

It offers dict-like interface to LevelDB

>>> db["prefix", "key"] = np.array([1, 2, 3], dtype=np.int8)
>>> db["prefix", "key"]
array([1, 2, 3], dtype=int8)

>>> db = db["prefix", ...]
>>> db["key"]
array([1, 2, 3], dtype=int8)

String-only keys¶

The only possible type for the keys is str. It avoids several problems when working with prefixes.

Multiprocessing support¶

Experimental multiprocessing support using a background flask server, exposing the same API of a direct connection:

db = LevelDB(db_path="path_to_db", server_address="http://127.0.0.1:5000")

Installation¶

It is possible to install openleveldb with poetry:

poetry add openleveldb

or with pip:

pip install openleveldb

Verify installation¶

Verify that the installation has been successful and that plyvel correctly installed leveldb, if it is not already installed on the system:

python -c 'import openleveldb'

Verify that openleveldb using the tests

git clone git@github.com:lucmos/openleveldb.git
cd openleveldb
poetry run pytest .

Getting started¶

Open db connection¶

The connection to the db can be direct or pass through a REST server. The only change required in the code is how the LevelDB object is instantiated

Direct connection¶

The first thing to do is to instantiate a LevelDB object to open a connection to leveldb database:

from openleveldb import LevelDB
db = LevelDB(db_path="path_to_db")

REST connection¶

If it’s required to have multiprocessing support, that is not provided by leveldb, it is possible to start a server and connect to the database through REST API. In order to start the server is enough to do:

cd openleveldb
make server

Then it’s possible to instantiate a LevelDB object specifying the server:

from openleveldb import LevelDB
db = LevelDB(db_path="path_to_db", server_address="http://127.0.0.1:5000")

Basic access¶

Storing, reading and deleting an element follow the dict protocol:

>>> db["prefix", "key"] = np.array([1, 2, 3], dtype=np.int8)
>>> db["prefix", "key"]
array([1, 2, 3], dtype=int8)
>>> del db["prefix", "key"]

It is possible to use an arbitrary number of prefixes:

>>> db["prefix1", "prefix2", "key"] = np.array([1, 2, 3], dtype=np.int8)
>>> db["prefix1", "prefix2", "key"]
array([1, 2, 3], dtype=int8)
>>> del db["prefix1", "prefix2", "key"]

Iteration¶

Iteration over (key, value) pairs behaves accordingly:

>>> list(db)
[('a1', 'value1'), ('b1', 'value2'), ('b2', 'value3'), ('c1', 'value4')]

It’s possible to perform advanced form of iteration using the LevelDB.prefixed_iter function:

>>> list(db)
[('a1', 'value1'), ('b1', 'value2'), ('b2', 'value3'), ('c1', 'value4')]
>>> list(db.prefixed_iter(prefixes=["b"]))
[('1', 'value2'), ('2', 'value3')]
>>> list(db.prefixed_iter(prefixes=["b", "1"]))
[('', 'value2')]
>>> list(db.prefixed_iter(starting_by="b"))
[('b1', 'value2'), ('b2', 'value3')]
>>> list(db.prefixed_iter(starting_by=["b", "1"]))
[('b1', 'value2')]

Fancy indexing¶

When a local connection is available, it is possible to use fancy indexing to obtain a stateful LevelDB that remembers the prefixes:

>>> list(db)
[('a1', 'value1'), ('b1', 'value2'), ('b2', 'value3'), ('c1', 'value4')]
>>> db_b = db['b', ...]
>>> db_b["1"]
'value2'
>>> list(db_b)
[('1', 'value2'), ('2', 'value3')]
>>> list(db["c", ...])
[('1', 'value4')]