FlureeDB is an [open source](https://github.com/fluree/db) blockchain based graph database with many great features that is suitable, among other things as technology for making distributed side-chains for blockchains like HIVE. As a database that is written in the [clojure language](https://clojure.org/), and as a relatively new piece of database technology, the ecosystem surrounding this database was relatively limited from the perspective of the Python/TypeScript/C++ environment me and my project team are working in.

To overcome this limited ecosystem, we created and open sourced two related projects (available through four repos). The [aioflureedb](https://github.com/pibara/aioflureedb) asynchronous Python library, and the [Fluree Schema Scenario Tool](https://github.com/pibara/fluree-schema-scenario-tool) or fsst, a tool aimed at CICD pipeline usage mostly, but suitable for a wide range of testing and debugging activities when developing with FlureeDB.

On march 30th 2023 FlureePBC released the 2.0 version of their blockchain based graph database. I don't have a complete picture of everything that changed in the 2.0 version compared to the 1.0, but a few important ones:

* A considerable performance increase
* More powerful queries in [FlureeQL](https://developers.flur.ee/docs/guides/querying/query-advanced/5/)
* Multiple bug fixed
* Removal of *depricated* APIs

When the 2.0 version of FlureeDB was released, the *fsst* tool stoped working on the *stable* build of the fluree+fsst docker images. In [this](https://github.com/pibara/fluree-fsst-docker) repo, there are docker files and shell scripts for building the docker images and to push them to docker hub so fsst can use them. When a new version of FlureeDB comes out, we have a small collection with currently failing scenarios we want to test the new release with that are in [this repo](https://github.com/pibara/fsst-demo). 

There was a tiny problem with the release of the 2.0 version in that the [API documentation](https://developers.flur.ee/docs/reference/http/overview/) at that point in time had not yet been updated to include documentation for the new API endpoints that were supposed to replace the now removed deprecated API endpoints.  For the most part it was just a name change from API endpoints and request parameters with **db** in the name to API endpoints and request parameters with **ledger** in the name.

## aioflureedb API changes

Basically the API changes for aioflureedb reflect those for FlureeDB itself. They are mostly transparent.

```python
async with aioflureedb.FlureeClient(masterkey=key,
                                        host=host,
                                        port=port) as flureeclient:
        await flureeclient.health.ready()
        await flureeclient.new_db(ledger_id=dbase)
        fdb = await flureeclient[dbase]
        async with fdb(key) as database:
            await database.ready()
```
The above code works with the 1.0 version of FlureeDB, yet not with the version 2.0 version.
```python
async with aioflureedb.FlureeClient(masterkey=key,
                                        host=host,
                                        port=port) as flureeclient:
        await flureeclient.health.ready()
        await flureeclient.new_ledger(ledger_id=dbase)
        fdb = await flureeclient[dbase]
        async with fdb(key) as database:
            await database.ready()
```
By replacing *new_db* with *new_ledger*, the above code works not just with the new 2.0 version of fluree but also still works with the old 1.0 version. In the same way the method *dbs* that lists all the ledgers on the server is now called *ledgers*, and the method *delete_db* is now named *delete_ledger*.

There are two other changes to aioflureedb when running the above code that happen beneath the service.

The internals of the line *await flureeclient.health.ready()* have been changed. This line makes aioflureedb repeatedly query the /fdb/health API endpoint untill it considers the flureedb instance to be ready. But recent tests done with restarting FlureeDB in a new advanced fsst test with hooks has revealed the old implementation of ready() wasn't quite complete.

When FlureeDB starts up, and it isn't the first time flureeDB starts, ever, and the server is *almost* up, first the field *ready* is set to True. A little bit later the field *status* is updated to have a non-null value. The call to ready() now waits for the second condition because if it doesn't, signed transactions will fail during the window will the second condition is met.

A second under the hood change to aioflureedb involves cryptographical database ownership. In the 1.0 version of FlureeDB *new_db* used to make a database bound to initially generated server key. That behavior is now simmulated by aioflureedb, but this happens under the hood through the use of an extra API parameter that is added to the requests.

## Getting the latest versions

If you used fsst before, update to the newest version first. Like before, you have the choice to do so with or without support for *local* domain-api functionality.

```
python3 -m pip install 'fsst[docker,domainapi]' --force
```

or 

```
python3 -m pip install 'fsst[docker,domainapi]' --force
```

Because aioflureedb is a dependency of fsst, you will now have the latest version of both.

After this, it is suggested you remove any remaining local fsst docker docker images if you have them.

```
docker rmi --force $(docker images -q 'pibara/fsst'|uniq)
```

You can now pull the latest fsst/flureedb docker image for both version 1.0 and version 2.0 of FlureeDB

```
docker pull pibara/fsst:stable
docker pull pibara/fsst:v1
``` 

Note that this step isn't strictly needed. The fsst tool will do a docker pull itself of the image if its not available local, but this will make the first run slower because it is fetching the docker image first.

## New fsst features

Making fsst run with the 2.0 version of fluree took some time and effort, and most of all debugging. To allow for this debugging, a few new features were added to the fsst command line.

### debug

The fsst tool already had the *verbosefluree* flag to make it output the FlureeDB console log inside of docker to the console for the fsst command. Next to the *verbosefluree* flag for debug purposes, fsst now also has a **debug** flag. When using this flag, *aioflureedb* inside of the running docker container, is given an environment variable prompting it to log all API queries and responses to the console.
You can use this flag on its own or combined with the verbosefluree flag.

If we run *dockertest* on [this](https://github.com/pibara/fsst-demo/tree/main/demo2) demo directory without the debug flag,

```
fsst dockertest
```
we get:
```
COMMAND: fsst guesttest --target default --network test91340 --runs 1 --stages ALL
IMAGE: pibara/fsst:stable
NOTICE: stable not found, trying to fetch from docker hubi, this may take a moment.
  - Fetched
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
WARNING: No hooks file found:  hooks/hooks.py
RUN: 0
- Database: test91340/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
 ### trias_politica test_index = 0 ###
 ### Test the trias politica scenarios ###
- dbase = test91340/api-trias-politica
 - processing schema transaction sub-set
 - ok, completed 4 transactions on test91340/api-trias-politica
 - ok, added hard-coded _auth collection
 - ok, added auth roles for TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG : ['root']
 - ok, added auth roles for Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88 : ['root']
 - ok, added test user test_trias_politica TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
 - ok, added test user auth31 Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
 + Running run_test_give_role as TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
['artist', 'judicial']
['farmer', 'executive']
['baker', 'legislator']
     res= True EXPECTED
['judge', 'executive']
    ERROR: res= False NOT EXPECTED Shouldn't be able to give a judge executive powers
Unexpected result from scenario for scenario1 at sub-test 0
``` 

Now fasten your seatbelt, because when we add *debug*, 

```
fsst dockertest --debug
```

we get the very verbose output:
```
COMMAND: fsst guesttest --target default --network test91371 --runs 1 --stages ALL
IMAGE: pibara/fsst:stable
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
WARNING: No hooks file found:  hooks/hooks.py
RUN: 0
- Database: test91371/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
Unsigned GET: url = http://localhost:8090/fdb/health , ssl_verify_disabled = False
Result:
{"ready":true,"status":"leader","utilization":0.5}
 ### trias_politica test_index = 0 ###
 ### Test the trias politica scenarios ###
- dbase = test91371/api-trias-politica
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
Signed POST: url = http://localhost:8090/fdb/new-ledger , headers = {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:53 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1c30440220604670952419f4bd205462d5d2842781269a6734d847bc99a0edad2b7cd2442c02200d1bba64f93f35aa56142d4fda651be6050b9d973552364cb1508caee93e24bd"', 'Digest': 'SHA-256=m78LYfJtTeF+tQAXPjZ2Z4bypT1i+jXWcHgsn+WAOXc='} ,ssl_verify_disabled = False
  body =  {"ledger/id":"test91371/api-trias-politica","owners":["Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS"]}
Result:
"3b02d5c8f21045068ffd806827718f84262a1db629f97f1ec3dc7326b5219448"
Signed POST: url = http://localhost:8090/fdb/ledgers , headers = {'Content-Type': 'application/json'} ,ssl_verify_disabled = False
  body =  {}
Result:
[["test91371","api-trias-politica"]]
Signed POST: url = http://localhost:8090/fdb/ledgers , headers = {'Content-Type': 'application/json'} ,ssl_verify_disabled = False
  body =  {}
Result:
[["test91371","api-trias-politica"]]
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
  headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:53 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3046022100e31737f92e3f2f8a3a52e57f293d1636c351b21ef0f69ff0424aae0044357ccc022100b5626383c8ad99840cd45f9d1375ef26f66a741aa99ccc364656f0a294596329"', 'Digest': 'SHA-256=LKglIDoqt+3BTB9EtXjCP+TIaxz43vdeb47iP95JX8U='}
  body: {"select":["_collection/name"],"from":"_collection"}
400
rval: {"status":400,"message":"Ledger test91371/api-trias-politica is not currently available. Status is: :initialize.","error":"db/unavailable"}
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
  headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:56 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3045022052124a4203ab47e5826edf3160ef01023e887d2cbfba0a878125d8be1dfb4794022100cb707b31b0162232455e64c68c97d10d69b579b9b2b9a5bf755644c4df8f6fb3"', 'Digest': 'SHA-256=LKglIDoqt+3BTB9EtXjCP+TIaxz43vdeb47iP95JX8U='}
  body: {"select":["_collection/name"],"from":"_collection"}
200
rval: [{"_collection/name":"_ctx","_id":17592186044426},{"_collection/name":"_setting","_id":17592186044425},{"_collection/name":"_rule","_id":17592186044424},{"_collection/name":"_role","_id":17592186044423},{"_collection/name":"_auth","_id":17592186044422},{"_collection/name":"_user","_id":17592186044421},{"_collection/name":"_fn","_id":17592186044420},{"_collection/name":"_tag","_id":17592186044419},{"_collection/name":"_shard","_id":17592186044418},{"_collection/name":"_collection","_id":17592186044417},{"_collection/name":"_predicate","_id":17592186044416}]
 - processing schema transaction sub-set
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/command False
  headers: {'content-type': 'application/json'}
  body: {
    "cmd": "{\"type\": \"tx\", \"tx\": [{\"_id\": \"_fn$lastblock\", \"name\": \"lastBlock\", \"doc\": \"Get the number of the last known block on the ledger\", \"code\": \"(query (str  \\\"{\\\\\\\"select\\\\\\\": \\\\\\\"?maxBlock\\\\\\\", \\\\\\\"where\\\\\\\": [[\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?bNum\\\\\\\"], [\\\\\\\"?maxBlock\\\\\\\", \\\\\\\"#(max ?bNum)\\\\\\\"], [\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?maxBlock\\\\\\\"]]}\\\" ) )\"}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS\", \"fuel\": 1000, \"nonce\": 5166419320643064, \"expire\": 1683913856508}",
    "sig": "1b304502200a432ba313cb55b9a4987c0c75bdd1f829551c1b19ccfb3f9d4956aab5cf3a58022100c5617df2a8a2791869b387599d4e651fdb0bde13f932debe13e353a240ed857c"
}
200
rval: {"tempids":{"_fn$lastblock":70368744178665},"block":2,"hash":"efa8b0bab51890a112bce50f922072f99608a34f654b436ac47a698c087a5038","instant":1683913736572,"type":"tx","duration":"83ms","fuel":1216,"auth":"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS","status":200,"id":"25999844945b4bd664616395684037865995216c1fdcd308a3fcd2edccc3f996","bytes":1206,"t":-3,"flakes":[[70368744178665,90,"lastBlock",-3,true,null],[70368744178665,92,"(query (str  \"{\\\"select\\\": \\\"?maxBlock\\\", \\\"where\\\": [[\\\"?s\\\", \\\"_block/number\\\", \\\"?bNum\\\"], [\\\"?maxBlock\\\", \\\"#(max ?bNum)\\\"], [\\\"?s\\\", \\\"_block/number\\\", \\\"?maxBlock\\\"]]}\" ) )",-3,true,null],[70368744178665,93,"Get the number of the last known block on the ledger",-3,true,null],[-3,99,"73bb9c7a9423673dc36c37ec22d7a2244189d390f427e5f9b34d6c3a9cf87be7",-3,true,null],[-3,100,"25999844945b4bd664616395684037865995216c1fdcd308a3fcd2edccc3f996",-3,true,null],[-3,101,105553116266496,-3,true,null],[-3,103,5166419320643064,-3,true,null],[-3,106,"{\"type\": \"tx\", \"tx\": [{\"_id\": \"_fn$lastblock\", \"name\": \"lastBlock\", \"doc\": \"Get the number of the last known block on the ledger\", \"code\": \"(query (str  \\\"{\\\\\\\"select\\\\\\\": \\\\\\\"?maxBlock\\\\\\\", \\\\\\\"where\\\\\\\": [[\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?bNum\\\\\\\"], [\\\\\\\"?maxBlock\\\\\\\", \\\\\\\"#(max ?bNum)\\\\\\\"], [\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?maxBlock\\\\\\\"]]}\\\" ) )\"}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS\", \"fuel\": 1000, \"nonce\": 5166419320643064, \"expire\": 1683913856508}",-3,true,null],[-3,107,"1b304502200a432ba313cb55b9a4987c0c75bdd1f829551c1b19ccfb3f9d4956aab5cf3a58022100c5617df2a8a2791869b387599d4e651fdb0bde13f932debe13e353a240ed857c",-3,true,null],[-3,108,"{\"_fn$lastblock\":70368744178665}",-3,true,null]]}
```

I'm leaving out the middle part to keep this post digestible, so here is the end:

```
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
  headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:58 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1c3045022015954f419348d6225cc61db1a9671300b62ab78695dfd811ad89f0b8767ffa8902210096cdb6e5b230c3b46a079ad48c4ca1df995753cbfc5a6f5888252170f6253d8e"', 'Digest': 'SHA-256=bnIsZEIemFsAtLHZpH6lRKvEiXvUAC6+mBHYKKaI6mg='}
  body: {"select":["*"],"from":["_tx/id","58bfe4610286d84e942ab409a29e13a4c63f27788eec541365803c01a47ce832"]}
200
rval: [{"_id":-25,"_tx/hash":"291e7cb361e0f2e368bbf6766193f9070828414751ca7c46c5bf963d1fc008e2","_tx/id":"58bfe4610286d84e942ab409a29e13a4c63f27788eec541365803c01a47ce832","_tx/auth":{"_id":105553116267528},"_tx/nonce":8140585842157148}]
     res= True EXPECTED
['judge', 'executive']
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/command False
  headers: {'content-type': 'application/json'}
  body: {
    "cmd": "{\"type\": \"tx\", \"tx\": [{\"_id\": [\"_user/username\", \"judge\"], \"roles\": [[\"_role/id\", \"executive\"]]}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88\", \"fuel\": 1000, \"nonce\": 6974785935112664, \"expire\": 1683913858449}",
    "sig": "1b3046022100aa43718ea5962dfd41d94b582986347b916ea5b5729305d9a28e047f713eb331022100c098e5ebe5c01be77accc714a81ba141aed20e49ddfac93781b678b62c2a3dff"
}
200
rval: {"block":14,"hash":"c2049f510231cfb473779afd11055531af96df0d2b8f861f25f702e77879ac06","instant":1683913738474,"type":"tx","duration":"22ms","fuel":553,"auth":"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88","status":200,"id":"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d","bytes":546,"t":-27,"flakes":[[87960930223083,52,123145302311914,-27,true,null],[-27,99,"e2935a9eaf80a9e8673a53b0b2d2d33a377965b8fafe789cc6741ac7c9cc028f",-27,true,null],[-27,100,"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d",-27,true,null],[-27,101,105553116267528,-27,true,null],[-27,103,6974785935112664,-27,true,null],[-27,106,"{\"type\": \"tx\", \"tx\": [{\"_id\": [\"_user/username\", \"judge\"], \"roles\": [[\"_role/id\", \"executive\"]]}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88\", \"fuel\": 1000, \"nonce\": 6974785935112664, \"expire\": 1683913858449}",-27,true,null],[-27,107,"1b3046022100aa43718ea5962dfd41d94b582986347b916ea5b5729305d9a28e047f713eb331022100c098e5ebe5c01be77accc714a81ba141aed20e49ddfac93781b678b62c2a3dff",-27,true,null]]}
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
  headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:58 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3046022100c22c03601eae72ed6f21147d956f6a0a0303050f5c41988e8ebfc5eb8d9247c6022100ea940d0bff54af5969d8b346c420c0f5c296dc2175220baa9b875c5d1ae275c2"', 'Digest': 'SHA-256=FTYMZO1hQmIzpJLoK8YujWNyzsDU3SGv1XvVwPQ45/4='}
  body: {"select":["*"],"from":["_tx/id","082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d"]}
200
rval: [{"_id":-27,"_tx/hash":"e2935a9eaf80a9e8673a53b0b2d2d33a377965b8fafe789cc6741ac7c9cc028f","_tx/id":"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d","_tx/auth":{"_id":105553116267528},"_tx/nonce":6974785935112664}]
    ERROR: res= False NOT EXPECTED Shouldn't be able to give a judge executive powers
Unexpected result from scenario for scenario1 at sub-test 0
```
As you can see, we get the complete FlureeDB requests, queries and transactions and their responses, what can be useful when debugging. 


### runs and hooks

Another new feature that the latesr version of fsst provides are multi-run tests with hooks. Before we look into how these work, lets talk about why we created them for fsst.

We have had FlureeDB running on a k8s system that as it turns out has seen some major crashes. I'm no expert in the details of k8s storage, but the result of the crashes has been that for FlureeDB the raft file ended up corrupted, but corrupted in a special way. Basicly spurious zeros were added to the end of the raft file, and we've been able to restore from this situation bysimply removing the trailing zeroes. In the mean time though, on the 1.0 version of FlureeDB, FlureeDB would crash on startup, so considereing if we should upgrade to 2.0, finding out if 2.0 would crash in a similar way was important. 

So let's look into how we tested and found out that the behaviour for 2.0 for now is the same as for 1.0 with these raft file corruptions.

We made a demo setup [here](https://github.com/pibara/fsst-demo/tree/main/demo3). It is basically demo2 but without any real tests, so lets not focus on that part.

The important part is the *hooks* directory with the *hooks.py* file that looks like this:

```json
import os

class Hooks:
    def before():
        print("HOOK: before")

    def between():
        print("HOOK: between")
        print("Appending raft file with zeros")
        with open("./data/group/0.raft", "ab") as raftfile:
            raftfile.write(b"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")

    def after():
        print("HOOK: after")
```

The file defines three hooks. A *before* hook, a *between* hook and an *after* hook.

If we want to enable these hooks we call fsst dockertest with the *hooks* flag.

```
fsst dockertest --hooks
```
The result is similar to the one before:

```
COMMAND: fsst guesttest --target default --network test91485 --runs 1 --hooks --stages ALL
IMAGE: pibara/fsst:stable
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
HOOK: before
RUN: 0
- Database: test91485/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
 ### trias_politica test_index = 0 ###
 ### Test the trias politica scenarios ###
- dbase = test91485/api-trias-politica
 - processing schema transaction sub-set
 - ok, completed 4 transactions on test91485/api-trias-politica
 - ok, added hard-coded _auth collection
 - ok, added auth roles for TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG : ['root']
 - ok, added auth roles for Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88 : ['root']
 - ok, added test user test_trias_politica TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
 - ok, added test user auth31 Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
     res= True
RUN completed: 0
HOOK: after
WARNING: build target has no tests defined
```

The only difference is that the *before* and the *after* hooks have been invoked, thus the two **HOOK:** lines in the output.  The important hook though, the *between* hook has not been invoked.

To get that hook invoked, we need to use the other new command line parameter as wall: *runs*.

```
fsst dockertest --hooks --runs 2
```

Now fsst runs the tests once, stops FlureeDB, uns the **between** hook, then starts flureedb again.

```
COMMAND: fsst guesttest --target default --network test91508 --runs 2 --hooks --stages ALL
IMAGE: pibara/fsst:stable
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
HOOK: before
RUN: 0
- Database: test91508/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
 ### trias_politica test_index = 0 ###
 ### Test the trias politica scenarios ###
- dbase = test91508/api-trias-politica
 - processing schema transaction sub-set
 - ok, completed 4 transactions on test91508/api-trias-politica
 - ok, added hard-coded _auth collection
 - ok, added auth roles for TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG : ['root']
 - ok, added auth roles for Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88 : ['root']
 - ok, added test user test_trias_politica TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
 - ok, added test user auth31 Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
     res= True
RUN completed: 0
RUN: 1
Waiting fof FlureeDB to terminate
- waiting
-terminated
HOOK: between
Appending raft file with zeros
- Database: test91508/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 1
```

Now this is where our run stops outputing data, but it doesn't close. Something is hanging. We Ctrl-C the process and try again with more info:

```
fsst dockertest --hooks --runs 2 --verbosefluree
```

```
COMMAND: fsst guesttest --target default --network test91508 --runs 2 --hooks --stages ALL
IMAGE: pibara/fsst:stable
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
HOOK: before
RUN: 0
- Database: test91508/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
 ### trias_politica test_index = 0 ###
 ### Test the trias politica scenarios ###
- dbase = test91508/api-trias-politica
 - processing schema transaction sub-set
 - ok, completed 4 transactions on test91508/api-trias-politica
 - ok, added hard-coded _auth collection
 - ok, added auth roles for TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG : ['root']
 - ok, added auth roles for Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88 : ['root']
 - ok, added test user test_trias_politica TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
 - ok, added test user auth31 Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
     res= True
RUN completed: 0
RUN: 1
Waiting fof FlureeDB to terminate
- waiting
-terminated
HOOK: between
Appending raft file with zeros
- Database: test91508/api-trias-politica
 - collecting transactions from build subdirs
Going through first 1 stages
  - trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 1
```

The output is too verbose to add here as text, so lets add a screenshot of the important part:


![image.png](https://files.peakd.com/file/peakd-hive/aioflureedb/23swgK4NyL4ZNoaf1DCMnP2eugrvQtshWM65bnp1jsC3GVK52epJMapMdtYGPRnxM2Zbi.png)

We see in the FlureeDB console logs that the raft file parsing throws an exception. The important part:

```
ERROR fluree.raft - Error initializing raft state from logs in:  ./data/group - java.io.EOFException: 
```

Lets check if this is the same as for the 1.0

```
fsst dockertest --hooks --runs 2 --verbosefluree --tag v1
```

And we get the same error:

```
2023-05-12 18:22:50,355 ERROR fluree.raft - Error initializing raft state from logs in:  ./data/group - java.io.EOFException
```

### conclusions

So to sumarize, both aioflureedb and fsst work with version 2.0 of FlureeDB now. There are minor API changes that users of aioflureedb need to take into account when upgrading mostly regarding ledger creation. A workaround has been added for use of the *health* endpoint without problems during database startup.

The fsst *dockertest* sub command has been extended with three command line arguments, *debug*, *hooks*, and *runs*, and finaly, the collection of flureedb/fsst docker images on docker hub has been updated to contain both a 2.0 and an 1.0 image with the latest fsst included.

If you want to get a feel for testing and developing with FlureeDB using FlureeDB 2.0 and maybe using the new options of fsst, try out the examples from this post yourself. Getting your hands into it is much more educational than just reading it from a blog post.