IOTA IRI rocksdb data storage structure

0x1 Overview

In IOTA IRI, it uses rocksdb as its database with column family to separate different data stored. Here is the introduction about the IRI rocksdb data storage structure and how to manipulate in Python.

0x2 Column Families

In IOTA rocksdb, just think column families as a sheet to separate data stored. No high technique inside here. Also, need to notice that some of the columns are using merge operation with string append operator in “,”. That is, a value may contain multiple items, and separate with “,”.

Above table is the column families list of IOTA IRI rocksdb:

IOTA IRI rocksdb column families list
ColumnDescriptionkeyvaluemerged?
defaultrocksdb default column family, not used in IRIN/AN/AN/A
transactionWhere the basic transaction information savedTransaction HashTransaction (8019 trits, 2673 bytes)No
transaction-metadataThe metadata of a transactionTransaction HashTransaction MetadataNo
milestoneSaving correspond milestone transaction hash to the indexMilestone Index (int, 4 bytes)Tuple[index, Transaction Hash]No
stateDiff— Something using in milestone to check snapshot, I didn’t get this well —Milestone Hash (Transaction Hash)List[(Transaction Hash, Value Changed)]Yes
addressHere is the transactions of address saved.AddressList[Transaction Hash]Yes
approveeApprovee is the child transactions which directly reference the transactionTransaction HashList[Transaction Hash]Yes
bundleBundle means the transaction grouped when creating the transactionTransaction HashList[Transaction Hash]Yes
tagTag is a custom field in transaction that can be as a identifier to searchTag (expand to Hash length)List[Transaction Hash]Yes
SOURCE: Louie Lu, IRI Source Code

0x3 Convert between trytes-string and bytes

A big issue between a trytes-string and the database is, in the database, it stores not the “trytes-string” with bytes, but the “trytes-string” convert to bytes. Sounds strange, ugh? (more exactly, is “trits” convert to bytes)

Without any implement details, IOTA using trytes-string as its represent method, and we need to save trytes-string into database, there is a mapping between trits and bytes that can convert trytes-string into bytes.

Given an example in Python, consider the transaction hash (in trytes-string):

UNUK99RCIWLUQ9WMUT9MPQSZCHTUMGN9IWOCOXWMNPICCCQKLLNIIE9UIFGKZLHRI9QAOEQXQJLL99999

its trits will be:

[0, 1, -1, -1, -1, -1, 0, 1, -1, -1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 1, 0, 0, 0, 1, -1, -1, 0, 0, 1, 1, 0, 1, -1, -1, 0, -1, 0, 0, 0, -1, -1, 0, 1, 1, 1, 0, 1, -1, -1, 1, -1, 0, 0, 0, 1, 1, 1, 1, -1, -1, -1, 0, -1, 1, 0, -1, -1, 0, 0, 0, 1, 0, -1, 0, 1, -1, 1, -1, 0, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1, -1, 0, 0, 0, 0, 0, 1, -1, -1, 0, 0, -1, -1, 0, 1, 0, 0, -1, -1, 0, -1, 0, -1, -1, 0, 1, 1, 1, -1, -1, -1, 1, -1, -1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, -1, 0, -1, -1, 1, 1, 0, 1, 1, 0, 1, 1, -1, -1, -1, 0, 0, 1, 0, 0, 1, -1, -1, 1, 0, 0, 0, 0, 1, -1, 0, 0, 1, 0, -1, 1, 1, -1, 1, -1, 1, 1, -1, 0, 0, 0, 1, 1, -1, 0, 1, 0, 0, -1, 0, 0, 1, 0, 0, 0, -1, 0, -1, 1, 0, 0, 0, -1, -1, -1, -1, 1, -1, 0, -1, 0, -1, 0, -1, 0, -1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

taking a trist to bytes convert, then the result will be:

b'\x8e\x9c\x04\x00\x08\xdf]\xe1\xdc^\xeau\xda\x96\x1b>>C\xf3\xca\x93\xb2\xa5t\xbd\x1a\x1c\xe8p\xf3\xcb\x02\xfadeP\x19N\xe5\x02)\xa53o\x0c\x00\x00\x00'.

0x4 Why convert trytes-string to bytes?

Each byte can contain 5 trits, so there is a 60% space reduce for this conversion.

0x5 Why do you need to know about IOTA IRI rocksdb

I want to create another IOTA node implement without java, thus I need to know about the layout of IRI rocksdb.

0x6 Anyway to manipulate with rocksdb data?

Currently, only java and rust (official implement) have the correct way dealing with database. I’m working on Python and mostly done (mlouielu/iota-python).

If you want to manipulate with Python, you will need to setup python-rocksdb, this is the version that I hack about adding column family with string append operator for IOTA (and, there is a hardcode include path in setup.py). And a converter is here in mlouielu/iota-python.

 




如果你覺得這篇文章不錯,歡迎打賞

BTH: 35QooNA82isrmQLmpEnqXpJoxeZmaPubPf

ETH:0x4cf61fea5EA842D202B85158d8b5e239C872De46

或是點選下方圖片贊助我一杯咖啡:

3 comments On IOTA IRI rocksdb data storage structure

Leave a reply:

Your email address will not be published.

這個網站採用 Akismet 服務減少垃圾留言。進一步了解 Akismet 如何處理網站訪客的留言資料