Store the Blockchain in a flat file with Python
steem·@furion·
0.000 HBDStore the Blockchain in a flat file with Python
For anyone doing work with Steem data, having a flat file of the entire Blockchain is very convenient for fast, bulk access. The only dependency is the Official Python Library, which we can install with: ``` pip install -U steem ``` #### The script ``` import json import os from contextlib import suppress from steem.blockchain import Blockchain def get_last_line(filename): if os.path.isfile(filename): with open(filename, 'rb') as f: f.seek(-2, 2) while f.read(1) != b"\n": f.seek(-2, 1) return f.readline() def get_previous_block_num(block): if not block: return -1 if type(block) == bytes: block = block.decode('utf-8') if type(block) == str: block = json.loads(block) return int(block['previous'][:8], base=16) def run(filename): b = Blockchain() # automatically resume from where we left off start_block = get_previous_block_num(get_last_line(filename)) + 2 # previous + last + 1 with open(filename, 'a+') as file: for block in b.stream_from(start_block=start_block, full_blocks=True): file.write(json.dumps(block, sort_keys=True) + '\n') if __name__ == '__main__': output_file = '/home/user/Downloads/steem.blockchain.json' with suppress(KeyboardInterrupt): run(output_file) ``` The only thing that needs changing is this line, which indicates a location where the flat file will be stored: ``` output_file = '/home/user/Downloads/steem.blockchain.json' ``` *You can run this script as many times as you like, and it will continue from the last block it synced.* #### Useful Utilities To see how many blocks we currently have, we can simply perform a line count. ``` wc -l steem.blockchain.json ``` To inspect a particular block, and pretty-print it: ``` sed '10000q;d' steem.blockchain.json | python -m json.tool ``` Replace 10000 with desired block_number + 1. 
👍 furion, thedeplorable1, abdullahilyas, steemcollator, runridefly, yuriks2000, cassidyandfranks, beeskee, tommycoin, cupang, teamsteem, dubi, dongu, jyp, very, goodhunter, steemrollin, kyusho, skt, tarindel, samether, virtualgrowth, lamech-m, thegame, steembets, steemland.com, steemprentice, cgame, ogochukwu, tonylondon, evildeathcore, etcmike, syarrf, pheonike, fiveboringgames, dolov, lichtblick, bestofreddit, pmartynov, grandpere, applecrisp, nigelmarkdias, mysticegoistic, joele, ripplerm, myvoter, good-karma, mysteem, demo, feruz, esteemapp, bounties, steempoll, puffin, malaiandrueth, ullikume, gregm, immarojas, justtryme90, thatgermandude, shaka, gargon, pgarcgo, pollux.one, bellastella, jgcastrillo19, sancho.panza, elyaque, handsolo, wartrapa, uwelang, freiheit50, dulcinea, steemperor, steempire, titin, juliamateo, teo, albagargon, mdcomes, juanmiguelsalas, alfredozofio, fyrstikken, aniestudio, loreennaa, ivan-perez-anies, teacher, heiditravels, nelyp, luisucv34, carlos-cabeza, tincho, moisesmcardona, tamim, twinner, beers, jeff001, kachinhenry, pharesim, xeroc, berkah, rossco99, steem-id, lovejoy, pairmike, blakemiles84, theshell, ratel, michaelx, albertogm, roelandp, raymondspeaks, phenom, bitcoiner, bkkshadow, dirty.hera, gomeravibz, letc, brendio, jessicazheng, cmorton, nulliusinverba, finleyexp, tracemayer, rusteemitblog, ninkhisibir, marialin, ianboil, sstefan, garvofe, v4vapid, tamersameeh, robertneleson, torkot, acidsun, tyler-fletcher, isteemit, skapaneas, on0tole, anasya, vl248, sveokla, marinabogumil, kell234, hyiparena, konti, sharker, zaebars, crypto.owl, smailer, dmilash, litrbooh, nekromarinist, always1success, mama-steem, uuuhha, romancs, riosparada, krishtopa, villainblack, leno4ek, nadin3, xanoxt, victoriart, tatianka, kralizec, elena-singer, renzoarg, burnin, ekaterinka, develcuy, borishaifa, htyfn, orenshani7, therajmahal, marel, steemlift, max-max, gildar, throughtheglass, kostaslou, fbechstein, blockchained, singa, anton333, ipumba, hms818, opheliafu, cryptofunk, mark-waser, digital-wisdom, jwaser, bwaser, herpetologyguy, morgan.waser, bottymcbotface, davidjkelley, ethical-ai, ellepdub, strong-ai, technoprogressiv, eem, thecryptodrive, steemdrive, bravenewcoin, unipsycho, annieb, jaycobbell, steemsports, inchonbitcoin, blueorgy, speda, michaellamden68, kenistyles, transhuman, chessmasters, beervangeer, chitty, profitgenerator, kyriacos, honusurf, michaeldodridge, aismor, headsink, rizkymustafa, bedroomshaman, sergey44, sebastianjago, coininstant, prakashghai, flyasfuq, fooblic, sandra, jason, robrigo, lily-da-vine, inertia, raphaelom, cryptomancer, dailybitcoinnews, will.isles, corvuscoraxx, gavvet, ivet, joseph, nomoreheroes7, cryptoctopus, steve-walschot, dragonslayer109, infovore, jacor, sauravrungta, crazymumzysa, keuudeip, ejhaasteem, kuttmargo, picokernel, cube48, shaheer001, contact95, pqlenator, andilorenzo, sqube, cathynsons, jeanviete, doitvoluntarily, vitality, ardina, rizkiavonna, ade-andrian, cqf, btcxchange, personz, blackvapor, andu, rmach, jtest, manycoolthings, alexrickard86, trafalgar, rdollarsign, artem-sokoloff, alekohalkias, r4fken, greenlantern1971, jamhuery, uvas, dez1337, sirwinchester, mandibil, afrog, bullionstackers, anwarabdullah, christoph3, krystle, steemit-life, sk828man, speckofdust, hien-tran, yakuhi, firesteem, thejohalfiles, altccy, dmitriybtc, ianstrat, bdnoble, aggroed, titusfrost, matt123, oleg888, hiimamazing, lastminuteman, justinashby, intelliguy, bitshares101, saleg25, stray, goldbug1450, ajvest, roadscape, nba05, allyouneedtoknow, anri-avgustino, damarth, greatdabu, florio, kenny-crane, voiceofreason, automaton, ausbitbank, arjane, stephenkendal, abit, animal-shelter, torem-di-torem, jonjon1, tonypedroso, nin0000, vadbars, thomasp, ghasemkiani, madlenfox, jonathanyoung, mammasitta, comfortgenius, nspart, fingersik, tommyhansen, gladbhoy, piiibiii, kalamur, zurvanic, chromiumone, ssekulji, blandermania, cryptodragon313, themonetaryfew, forrestphotos, artpoet, mirspirs, jimy74, cristi, baracuda, bitcoinparadise, crok, haraldsteam, alexandergomez, nanzo-scoop, genericusrr, mummyimperfect, emily-cook, mafeeva, ak2020, cybertiger, breezin, fabriciocosta, gogogadget, liondani, charlesnickel, steemquestions, deez, tornadotom2000, animus, ericvancewalton, bue-witness, bue, mini, boy, healthcare, daniel.pan, bunny, moon, helen.tan, craigslist, cosmo.crator, rdickey, ekitcho, getdfs, gallonofpcp, addicted7abinash, askshane, anilk94, qwesttexas, donchate, abdullar, bledarus, morning, envy112, ciao, sharon, sofa, steemster1, steemo, stroully, roto, igtes, jarvis, lillianjones, crion, johnbyrd, thadm, revelbrooks, buffett, coad, hitherise, widell, bane, curpose, prof, steema, troich, wiss, ailo, thomasaustin, vive, ficholl, confucius, fortuner, yorsens, msjennifer, thermor, eavy, raymonjohnstone, alexcodes, nitinchugh, transisto, cryptovolution, voronoi, malake, greencrypto, steemmaster, khairulfajri, slowwalker, oldstone, yasmins, barrydutton, frostedace, ccmamma, noodles.pan, ace108, prameshtyagi, harrisonmir, pepe.maya, arul0312, taufar, silverdragon, dimitrya123, sajzaib, cannabissativa, gynsi, macneo, bitland, jrcornel, tonyson4078, asher123, ne0phyt3, jesuscirino, contentguy, orenz, sochul, superfreek, tafo21, richq11, sephiroth, videodrome, englishtchrivy, piotrdab, tellermachine, solomon007, bitcoinandcoffee, whitey420, sassal, oluwoleolaide, noagenda, gutzofter, sameermanas, thegreatomski, felixxx, lawrenceho84, tanyabtc, bycz, cryptohazard, adrianmada321, aizensou, lukekelley, roshid, legend112, summon, smooth, judasp, saifulbahri, ylowe39, bbkoopsta, blockgods, eric-boucher, bontonstory, officialfuzzy, opmac, xinta, krasotka, remlaps, mathworksheets, mc57330, entice, berniesanders, engagement, thecyclist, steemservices, yougotflagged, theyeti, thebotkiller, j3dy, andrianna, coinhub, bi5h0p, matiasrodrigo, ackza, lianasution, photon-pl, drakos, fahrullah, marnee, mohdmohsin,