Store the Blockchain in a flat file with Python

View this thread on: d.buzz | hive.blog | peakd.com | ecency.com
·@furion·
0.000 HBD
Store the Blockchain in a flat file with Python
For anyone doing work with Steem data, having a flat file of the entire Blockchain is very convenient for fast, bulk access.

The only dependency is the Official Python Library, which we can install with:
```
pip install -U steem
```

#### The script
```
import json
import os
from contextlib import suppress
from steem.blockchain import Blockchain


def get_last_line(filename):
    if os.path.isfile(filename):
        with open(filename, 'rb') as f:
            f.seek(-2, 2)
            while f.read(1) != b"\n":
                f.seek(-2, 1)
            return f.readline()


def get_previous_block_num(block):
    if not block:
        return -1

    if type(block) == bytes:
        block = block.decode('utf-8')

    if type(block) == str:
        block = json.loads(block)

    return int(block['previous'][:8], base=16)


def run(filename):
    b = Blockchain()
    # automatically resume from where we left off
    start_block = get_previous_block_num(get_last_line(filename)) + 2  # previous + last + 1
    with open(filename, 'a+') as file:
        for block in b.stream_from(start_block=start_block, full_blocks=True):
            file.write(json.dumps(block, sort_keys=True) + '\n')


if __name__ == '__main__':
    output_file = '/home/user/Downloads/steem.blockchain.json'
    with suppress(KeyboardInterrupt):
        run(output_file)
```

The only thing that needs changing is this line, which indicates a location where the flat file will be stored:
```
 output_file = '/home/user/Downloads/steem.blockchain.json'
```

*You can run this script as many times as you like, and it will continue from the last block it synced.*

#### Useful Utilities
To see how many blocks we currently have, we can simply perform a line count.
```
wc -l steem.blockchain.json
```

To inspect a particular block, and pretty-print it:
```
sed '10000q;d' steem.blockchain.json | python -m json.tool
```
Replace 10000 with desired block_number + 1.

![steem_big.png](https://steemitimages.com/DQmeBtkDw9J3evsQdzL4MNFepWxkdn55QUpvxz5dCGkzgWY/steem_big.png)
👍 , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,