IOTA IRI rocksdb data storage structure

0x1 Overview

In IOTA IRI, it uses rocksdb as its database with column family to separate different data stored. Here is the introduction about the IRI rocksdb data storage structure and how to manipulate in Python.

0x2 Column Families

In IOTA rocksdb, just think column families as a sheet to separate data stored. No high technique inside here. Also, need to notice that some of the columns are using merge operation with string append operator in “,”. That is, a value may contain multiple items, and separate with “,”.

Above table is the column families list of IOTA IRI rocksdb:

IOTA IRI rocksdb column families list
Column Description key value merged?
default rocksdb default column family, not used in IRI N/A N/A N/A
transaction Where the basic transaction information saved Transaction Hash Transaction (8019 trits, 2673 bytes) No
transaction-metadata The metadata of a transaction Transaction Hash Transaction Metadata No
milestone Saving correspond milestone transaction hash to the index Milestone Index (int, 4 bytes) Tuple[index, Transaction Hash] No
stateDiff — Something using in milestone to check snapshot, I didn’t get this well — Milestone Hash (Transaction Hash) List[(Transaction Hash, Value Changed)] Yes
address Here is the transactions of address saved. Address List[Transaction Hash] Yes
approvee Approvee is the child transactions which directly reference the transaction Transaction Hash List[Transaction Hash] Yes
bundle Bundle means the transaction grouped when creating the transaction Transaction Hash List[Transaction Hash] Yes
tag Tag is a custom field in transaction that can be as a identifier to search Tag (expand to Hash length) List[Transaction Hash] Yes
SOURCE: Louie Lu, IRI Source Code

0x3 Convert between trytes-string and bytes

A big issue between a trytes-string and the database is, in the database, it stores not the “trytes-string” with bytes, but the “trytes-string” convert to bytes. Sounds strange, ugh? (more exactly, is “trits” convert to bytes)

Without any implement details, IOTA using trytes-string as its represent method, and we need to save trytes-string into database, there is a mapping between trits and bytes that can convert trytes-string into bytes.

Given an example in Python, consider the transaction hash (in trytes-string):

UNUK99RCIWLUQ9WMUT9MPQSZCHTUMGN9IWOCOXWMNPICCCQKLLNIIE9UIFGKZLHRI9QAOEQXQJLL99999

its trits will be:

[0, 1, -1, -1, -1, -1, 0, 1, -1, -1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 1, 0, 0, 0, 1, -1, -1, 0, 0, 1, 1, 0, 1, -1, -1, 0, -1, 0, 0, 0, -1, -1, 0, 1, 1, 1, 0, 1, -1, -1, 1, -1, 0, 0, 0, 1, 1, 1, 1, -1, -1, -1, 0, -1, 1, 0, -1, -1, 0, 0, 0, 1, 0, -1, 0, 1, -1, 1, -1, 0, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1, -1, 0, 0, 0, 0, 0, 1, -1, -1, 0, 0, -1, -1, 0, 1, 0, 0, -1, -1, 0, -1, 0, -1, -1, 0, 1, 1, 1, -1, -1, -1, 1, -1, -1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, -1, 0, -1, -1, 1, 1, 0, 1, 1, 0, 1, 1, -1, -1, -1, 0, 0, 1, 0, 0, 1, -1, -1, 1, 0, 0, 0, 0, 1, -1, 0, 0, 1, 0, -1, 1, 1, -1, 1, -1, 1, 1, -1, 0, 0, 0, 1, 1, -1, 0, 1, 0, 0, -1, 0, 0, 1, 0, 0, 0, -1, 0, -1, 1, 0, 0, 0, -1, -1, -1, -1, 1, -1, 0, -1, 0, -1, 0, -1, 0, -1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

taking a trist to bytes convert, then the result will be:

b'\x8e\x9c\x04\x00\x08\xdf]\xe1\xdc^\xeau\xda\x96\x1b>>C\xf3\xca\x93\xb2\xa5t\xbd\x1a\x1c\xe8p\xf3\xcb\x02\xfadeP\x19N\xe5\x02)\xa53o\x0c\x00\x00\x00'.

0x4 Why convert trytes-string to bytes?

Each byte can contain 5 trits, so there is a 60% space reduce for this conversion.

0x5 Why do you need to know about IOTA IRI rocksdb

I want to create another IOTA node implement without java, thus I need to know about the layout of IRI rocksdb.

0x6 Anyway to manipulate with rocksdb data?

Currently, only java and rust (official implement) have the correct way dealing with database. I’m working on Python and mostly done (mlouielu/iota-python).

If you want to manipulate with Python, you will need to setup python-rocksdb, this is the version that I hack about adding column family with string append operator for IOTA (and, there is a hardcode include path in setup.py). And a converter is here in mlouielu/iota-python.

 

3 comments On IOTA IRI rocksdb data storage structure

Leave a reply:

Your email address will not be published.