Recover a chain

1Before you start: save the wallet

The chain data can always be downloaded again. Your wallet cannot. Stop the node, then copy the wallet before you change anything. For the original node on testnet on Linux:

cp ~/.divi/testnet3/wallet.dat ~/wallet-before-repair.dat

For IronDivi, the wallet is a folder:

cp -a ~/.irondivi/testnet/wallet ~/irondivi-wallet-before-repair

On the real network, leave out testnet3/ for the original node and use ~/.irondivi/ for IronDivi. If you set a different data folder, use that.

None of the steps below delete or change the wallet. They only touch the chain data.

2What a broken chain looks like

We stopped each node with kill -9, which ends the program the way an out-of-memory kill does and comes close to a power cut, while it was taking in new blocks.

The original node refused to start and printed:

Error: Error loading block database : Block database corruption detected! Failed to find best block in block index

Its debug.log in the data folder said why it checked:

LoadBlockIndexState: Last shutdown was prepared: false

This happened every time we killed it after new blocks had come in. When we killed it 6 seconds after a start, before any new block, it started again without complaint.

IronDivi started again after kill -9 at the same height, with no error.

3Why "just start it again" is not enough

If you start the original node a second time after the corruption error, it usually comes up. That is not a repair. The failed start shuts down tidily and records that it did, so the next start reads Last shutdown was prepared: true and skips the check that caught the problem. Nothing was fixed in between.

In our tests the second start happened to work, and the chain checked out afterwards. You cannot count on that. Use one of the next two steps instead, while the node still knows its last shutdown was not clean. If you have already started it a second time, check it as in step 7, and use -reindex if anything looks wrong.

A node run as a service restarts by itself, so systemd will do the second start for you. Run your node as a service shows what that looks like in the journal.

4Original node: a quick repair

Make this the first start after the error. If you run the node as a service, stop the service first so it does not start the node for you.

divid -recoverblockdb

Add -datadir=… if you use one. In our test the node had been killed at block 360,908. With -recoverblockdb it was answering after 6 seconds at block 360,906, two blocks back, and it had caught up again 30 seconds later. verifychain said true.

What it does: the node keeps two records of the latest block, one for the block index and one for the coin database. When the block index has lost its record, this option copies the coin database's record across and goes on from there. It is not listed in divid -help, but it is in the source, and it worked here.

There is a second option, -forcestart, which divid -help describes as "Attempt to force blockchain corruption recovery on startup". It does not copy any record. It skips the check and lets the node's normal start-up sort things out. We tested it the same way: killed at 360,911, answering at 360,909, caught up 30 seconds later, verifychain true. On our node the two options ended the same way.

After either one, stop the node and start it normally, or start the service again.

5Original node: rebuild with -reindex

Use this if the quick repair fails, if you are not sure what happened, or if verifychain says false. It throws away the block index and the coin database and rebuilds both from the block files already on your disk, so it downloads almost nothing.

divid -reindex

The node answers commands while it works, and divi-cli getblockcount climbs from zero. On our testnet node it took 487 seconds, about 8 minutes, to reach the tip at block 360,890. A second run, on the copy we had damaged on purpose, took 387 seconds. The real network has more and bigger blocks, so expect it to take longer there. We did not time it.

Run -reindex once. When it has finished, later starts need no option.

6IronDivi: delete the chain and sync again

IronDivi 0.2.5 has no -reindex or repair option: irondivid --help lists none. To rebuild its chain, stop the node, delete the two chain folders, and start it again. The wallet folder and irondivi.conf stay where they are.

kill $(pgrep irondivid)                     # irondivi-cli stop does not stop 0.2.5
cd ~/.irondivi/testnet
rm -rf chainstate txindex                   # NOT wallet, NOT irondivi.conf
irondivid --testnet

If you run it as a service, use sudo systemctl stop irondivid and sudo systemctl start irondivid instead of kill and the last line.

The node then downloads the whole chain from its peers. Ours took 75 minutes to get back to the tip on testnet. Its log said Loading wallet, not Creating new wallet, so the wallet was kept.

7Check the result

For the original node, ask it to re-check the last 1,000 blocks:

divi-cli verifychain 3 1000
true

For both nodes, compare the height with the testnet explorer (or a mainnet explorer on the real network), and check that your balance is what it was:

divi-cli getblockcount
divi-cli getbalance
irondivi-cli --testnet getblockcount
irondivi-cli --testnet getbalance

If the balance is lower than you expect while the node is still catching up, wait until it reaches the tip and check again.

If something goes wrong

Next: run your node as a service so it stops cleanly when the computer shuts down, or upgrade a node.