0x1 Overview
In IOTA IRI, it uses rocksdb as its database with column family to separate different data stored. Here is the introduction about the IRI rocksdb data storage structure and how to manipulate in Python.
0x2 Column Families
In IOTA rocksdb, just think column families as a sheet to separate data stored. No high technique inside here. Also, need to notice that some of the columns are using merge operation with string append operator in ",". That is, a value may contain multiple items, and separate with ",".
Above table is the column families list of IOTA IRI rocksdb:
| Column | Description | key | value | merged? |
|---|---|---|---|---|
| default | rocksdb default column family, not used in IRI | N/A | N/A | N/A |
| transaction | Where the basic transaction information saved | Transaction Hash | Transaction (8019 trits, 2673 bytes) | No |
| transaction-metadata | The metadata of a transaction | Transaction Hash | Transaction Metadata | No |
| milestone | Saving correspond milestone transaction hash to the index | Milestone Index (int, 4 bytes) | Tuple[index, Transaction Hash] | No |
| stateDiff | -- Something using in milestone to check snapshot, I didn't get this well -- | Milestone Hash (Transaction Hash) | List[(Transaction Hash, Value Changed)] | Yes |
| address | Here is the transactions of address saved. | Address | List[Transaction Hash] | Yes |
| approvee | Approvee is the child transactions which directly reference the transaction | Transaction Hash | List[Transaction Hash] | Yes |
| bundle | Bundle means the transaction grouped when creating the transaction | Transaction Hash | List[Transaction Hash] | Yes |
| tag | Tag is a custom field in transaction that can be as a identifier to search | Tag (expand to Hash length) | List[Transaction Hash] | Yes |
0x3 Convert between trytes-string and bytes
A big issue between a trytes-string and the database is, in the database, it stores not the "trytes-string" with bytes, but the "trytes-string" convert to bytes. Sounds strange, ugh? (more exactly, is "trits" convert to bytes)
Without any implement details, IOTA using trytes-string as its represent method, and we need to save trytes-string into database, there is a mapping between trits and bytes that can convert trytes-string into bytes.
Given an example in Python, consider the transaction hash (in trytes-string):
UNUK99RCIWLUQ9WMUT9MPQSZCHTUMGN9IWOCOXWMNPICCCQKLLNIIE9UIFGKZLHRI9QAOEQXQJLL99999
its trits will be:
[0, 1, -1, -1, -1, -1, 0, 1, -1, -1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 1, 0, 0, 0, 1, -1, -1, 0, 0, 1, 1, 0, 1, -1, -1, 0, -1, 0, 0, 0, -1, -1, 0, 1, 1, 1, 0, 1, -1, -1, 1, -1, 0, 0, 0, 1, 1, 1, 1, -1, -1, -1, 0, -1, 1, 0, -1, -1, 0, 0, 0, 1, 0, -1, 0, 1, -1, 1, -1, 0, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1, -1, 0, 0, 0, 0, 0, 1, -1, -1, 0, 0, -1, -1, 0, 1, 0, 0, -1, -1, 0, -1, 0, -1, -1, 0, 1, 1, 1, -1, -1, -1, 1, -1, -1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, -1, 0, -1, -1, 1, 1, 0, 1, 1, 0, 1, 1, -1, -1, -1, 0, 0, 1, 0, 0, 1, -1, -1, 1, 0, 0, 0, 0, 1, -1, 0, 0, 1, 0, -1, 1, 1, -1, 1, -1, 1, 1, -1, 0, 0, 0, 1, 1, -1, 0, 1, 0, 0, -1, 0, 0, 1, 0, 0, 0, -1, 0, -1, 1, 0, 0, 0, -1, -1, -1, -1, 1, -1, 0, -1, 0, -1, 0, -1, 0, -1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
taking a trist to bytes convert, then the result will be:
b'\x8e\x9c\x04\x00\x08\xdf]\xe1\xdc^\xeau\xda\x96\x1b>>C\xf3\xca\x93\xb2\xa5t\xbd\x1a\x1c\xe8p\xf3\xcb\x02\xfadeP\x19N\xe5\x02)\xa53o\x0c\x00\x00\x00'.
0x4 Why convert trytes-string to bytes?
Each byte can contain 5 trits, so there is a 60% space reduce for this conversion.
0x5 Why do you need to know about IOTA IRI rocksdb
I want to create another IOTA node implement without java, thus I need to know about the layout of IRI rocksdb.
0x6 Anyway to manipulate with rocksdb data?
Currently, only java and rust (official implement) have the correct way dealing with database. I'm working on Python and mostly done (mlouielu/iota-python).
If you want to manipulate with Python, you will need to setup python-rocksdb, this is the version that I hack about adding column family with string append operator for IOTA (and, there is a hardcode include path in setup.py). And a converter is here in mlouielu/iota-python.
|
1 2 3 4 5 6 7 8 9 10 11 |
# Convert trits to bytes >>> import iota >>> import iotapy >>> h = iota.Hash('EXAMPLE') >>> iotapy.storage.converter.from_trits_to_binary(h.as_trits()) b'\xb4T\xa1\xa0\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' # Conver bytes to trits >>> iotapy.storage.converter.from_binary_to_trits(b'\xb4T\xa1\xa0\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 243) [-1, -1, 1, 0, -1, 0, 1, 0, 0, 1, 1, 1, 1, -1, -1, 0, 1, 1, -1, -1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] >>> iota.Hash.from_trits(_) Hash(b'EXAMPLE99999999999999999999999999999999999999999999999999999999999999999999999999') |
Leave a Reply