3.3.1. mijin Catapult(v.2) node resynchronization¶
mijin Catapult(v.2) node has stopped synchronizing for some reason, etc. This section describes the resynchronization procedure.
- creation-day:
Aug. 1, 2022
- update date:
September 13, 2022
3.3.1.1. target¶
Node blocking is not progressing.
No longer able to check in /node/peers.
Some containers are Exit in the docker-compose ps command and cannot be recovered.
3.3.1.2. Step.1¶
Remote login to the node.
If you want to log in to AWS MarketPlace’s mijin Catapult(v.2), you can use mijin Catapult(v.2) EC2 instance login how to.
3.3.1.3. Step.2¶
Switch to the ‘catapult’ user running mijin.
$ sudo su - catapult
catapult@api1:~$
3.3.1.4. Step.3¶
Go to the directory where the mijin startup files are located.
Note that the directories are different for API and PEER nodes.
Node |
Directory |
API/Dual |
mijin-catapult-package/package/api/catapult/ |
PEER |
mijin-catapult-package/package/peer/catapult/ |
For API nodes, move as follows
catapult@api1:~$ cd mijin-catapult-package/package/api/catapult/
catapult@api1:~/mijin-catapult-package/package/api/catapult$
3.3.1.5. Step.4¶
Mijin launches multiple containers by docker.
Check multiple containers.
3.3.1.5.1. For API nodes¶
Four containers are running. Make sure that all States are Up. If there is something wrong, the state will be Exit instead of Up and you will be in a down state.
catapult@api1:~/mijin-catapult-package/package/api/catapult$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------
catapult_api-node-broker_1 bash -c /bin/bash /scripts ... Up
catapult_api-node_1 bash -c perl /scripts/wait ... Up 0.0.0.0:7900->7900/tcp
catapult_db_1 docker-entrypoint.sh bash ... Up 27017/tcp
catapult_rest-gateway_1 docker-entrypoint.sh ash - ... Up 0.0.0.0:3000->3000/tcp
3.3.1.5.2. In the case of PEER node¶
One container is activated.
Make sure that all States are Up.
If there is some abnormality, it will be Exit instead of Up and will be in a down state.
catapult@peer1:~/mijin-catapult-package/package/peer/catapult$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
catapult_peer-node_1 bash -c /bin/bash /scripts ... Up 0.0.0.0:7900->7900/tcp
3.3.1.6. Step.5¶
Stop and re-start all mijin containers using docker-compose.
3.3.1.6.1. All containers stopped¶
catapult@api1:~/mijin-catapult-package/package/api/catapult$ docker-compose down
Stopping catapult_rest-gateway_1 ... done
Stopping catapult_api-node_1 ... done
Stopping catapult_db_1 ... done
Removing catapult_api-node-broker_1 ... done
Removing catapult_rest-gateway_1 ... done
Removing catapult_api-node_1 ... done
Removing catapult_db_1 ... done
Removing network catapult_default
3.3.1.6.2. After stopping, check the lock file and delete it if any.¶
catapult@api1:~/mijin-catapult-package/package/api/catapult$ ls -la /mnt/mijin/blocks/data/*.lock
---------- 1 catapult catapult 0 Jul 14 02:17 /mnt/mijin/blocks/data/broker.lock
---------- 1 catapult catapult 0 Jul 14 02:17 /mnt/mijin/blocks/data/server.lock
If a lock file exists after the stop as shown above, it is considered to have stopped abnormally.
Therefore, delete the lock file.
$ rm -rf /mnt/mijin/blocks/data/broker.lock /mnt/mijin/blocks/data/server.lock
3.3.1.6.3. re-start¶
$ docker-compose up -d
Creating network "catapult_default" with the default driver
Creating catapult_db_1 ... done
Creating catapult_rest-gateway_1 ... done
Creating catapult_api-node-broker_1 ... done
Creating catapult_api-node_1 ... done
After startup, execute the items in Step.4 and make sure all containers are Up.
If you are in a similar situation, go to Step.9.
3.3.1.7. Step.6¶
Reset and recover block data on a node. If in a redundant state, data can be automatically retrieved and recovered from other nodes.
3.3.1.7.1. All containers stopped¶
$ docker-compose down
Stopping catapult_rest-gateway_1 ... done
Stopping catapult_api-node_1 ... done
Stopping catapult_db_1 ... done
Removing catapult_api-node-broker_1 ... done
Removing catapult_rest-gateway_1 ... done
Removing catapult_api-node_1 ... done
Removing catapult_db_1 ... done
Removing network catapult_default
3.3.1.7.2. Delete block data and mongo data¶
The catapult user that mijin is running as does not have sudo privileges and cannot delete directories.
Therefore, grant the catapult user sudo privileges.
This sudo setting is required only for the first time.
# catapultユーザーからログアウト Log out of catapult user
$ logout
# rootユーザーにスイッチ Switch to root user
$ sudo su -
# catapultユーザーにsudo権限を付与する Grant sudo privileges to the catapult user
# echo "catapult ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/catapult
# catapultユーザーにスイッチ Switch to catapult user
# su - catapult
Delete block data with sudo privileges.
$ sudo rm -rf /mnt/mijin/blocks/data
If API nodes are targeted, mongo data deletion is also required
$ sudo rm -rf /mnt/mijin/mongo/db
3.3.1.7.3. Execute recovery commands¶
Go to the file with docker-compose and run the RECOVER script.
When the API node is the target
catapult@api1:~$ cd mijin-catapult-package/package/api/catapult/
catapult@api1:~/mijin-catapult-package/package/api/catapult$
When the PEER node is the target
catapult@peer1:~$ cd mijin-catapult-package/package/peer/catapult/
catapult@peer1:~/mijin-catapult-package/package/peer/catapult$
Execute scripts. (Common)
catapult@api1:~/mijin-catapult-package/package/api/catapult$ bash scripts/recover.sh
2021/07/14 02:52 Start: mijin Recovery
2021/07/14 02:52 Check: /home/catapult/mijin-catapult-package/package/api/catapult
2021/07/14 02:52 Check: /home/catapult/mijin-catapult-package/package/api/catapult OK
2021/07/14 02:52 Check: Started docker?
2021/07/14 02:52 Check: Started docker Stop OK
2021/07/14 02:52 Check: Block Directory
2021/07/14 02:52 Check: Block Directory Empty OK
2021/07/14 02:52 Start: Make Block Directory
2021/07/14 02:52 Check: mongo Directory
2021/07/14 02:52 Check: mongo Directory OK
2021/07/14 02:52 Start: Make mongo Directory
2021/07/14 02:52 Start: Create mongo Init Data
about to fork child process, waiting until server is ready for connections.
forked process: 10
child process started successfully, parent exiting
[+] Preparing db
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:27017/catapult?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("c296d026-dfdf-4ea2-ba03-91d7404e21c9") }
MongoDB server version: 4.2.5
Loading LockHash
Loading LockSecret
Loading Metadata
Loading Mosaic
Loading Multisig
Loading Namespace
Loading RestrictionAccount
Loading RestrictionMosaic
===== accountRestrictions INDEXES =====
{ "_id" : 1 }
{ "accountRestrictions.address" : 1 }
===== accounts INDEXES =====
{ "_id" : 1 }
{ "account.publicKey" : 1 }
{ "account.address" : 1 }
===== addressResolutionStatements INDEXES =====
{ "_id" : 1 }
{ "statement.height" : 1, "statement.unresolved" : 1 }
===== blocks INDEXES =====
{ "_id" : 1 }
{ "block.signerPublicKey" : 1 }
{ "block.timestamp" : -1 }
{ "block.height" : -1 }
{ "block.type" : 1, "block.height" : -1 }
{ "block.signerPublicKey" : 1, "block.height" : -1 }
{ "block.beneficiaryAddress" : 1, "block.height" : -1 }
===== finalizedBlocks INDEXES =====
{ "_id" : 1 }
{ "block.finalizationEpoch" : -1 }
{ "block.height" : -1 }
===== hashLocks INDEXES =====
{ "_id" : 1 }
{ "lock.hash" : 1 }
{ "lock.ownerAddress" : 1 }
===== metadata INDEXES =====
{ "_id" : 1 }
{ "metadataEntry.compositeHash" : 1 }
{
"metadataEntry.sourceAddress" : 1,
"metadataEntry.metadataType" : 1,
"metadataEntry.scopedMetadataKey" : 1
}
{
"metadataEntry.targetAddress" : 1,
"metadataEntry.metadataType" : 1,
"metadataEntry.scopedMetadataKey" : 1
}
===== mosaicResolutionStatements INDEXES =====
{ "_id" : 1 }
{ "statement.height" : 1, "statement.unresolved" : 1 }
===== mosaicRestrictions INDEXES =====
{ "_id" : 1 }
{ "mosaicRestrictionEntry.compositeHash" : 1 }
===== mosaics INDEXES =====
{ "_id" : 1 }
{ "mosaic.id" : 1 }
{ "mosaic.ownerAddress" : 1 }
===== multisigs INDEXES =====
{ "_id" : 1 }
{ "multisig.accountAddress" : 1 }
===== namespaces INDEXES =====
{ "_id" : 1 }
{ "namespace.level0" : 1, "meta.index" : 1, "namespace.depth" : 1 }
{
"meta.latest" : -1,
"meta.index" : 1,
"namespace.level0" : 1,
"namespace.depth" : 1
}
{ "meta.latest" : -1, "namespace.level1" : 1, "namespace.depth" : 1 }
{ "meta.latest" : -1, "namespace.level2" : 1, "namespace.depth" : 1 }
{ "meta.latest" : -1, "namespace.ownerAddress" : 1 }
===== partialTransactions INDEXES =====
{ "_id" : 1 }
{ "transaction.signerPublicKey" : 1, "_id" : -1 }
{ "transaction.recipientAddress" : 1, "_id" : -1 }
{ "meta.hash" : 1 }
{ "meta.addresses" : 1 }
{ "meta.aggregateId" : 1 }
{ "meta.aggregateHash" : 1 }
===== secretLocks INDEXES =====
{ "_id" : 1 }
{ "lock.compositeHash" : 1 }
{ "lock.ownerAddress" : 1 }
===== system.profile INDEXES =====
===== transactionStatements INDEXES =====
{ "_id" : 1 }
{
"statement.height" : 1,
"statement.source.primaryId" : 1,
"statement.source.secondaryId" : 1
}
===== transactionStatuses INDEXES =====
{ "_id" : 1 }
{ "status.hash" : 1 }
{ "status.deadline" : -1 }
===== transactions INDEXES =====
{ "_id" : 1 }
{ "transaction.signerPublicKey" : 1, "_id" : -1 }
{ "transaction.recipientAddress" : 1, "_id" : -1 }
{ "meta.hash" : 1 }
{ "meta.addresses" : 1 }
{ "meta.aggregateId" : 1 }
{ "meta.height" : -1 }
{ "transaction.deadline" : -1 }
{ "transaction.cosignatures.signerPublicKey" : 1 }
{ "transaction.id" : 1, "transaction.type" : 1 }
===== unconfirmedTransactions INDEXES =====
{ "_id" : 1 }
{ "transaction.signerPublicKey" : 1, "_id" : -1 }
{ "transaction.recipientAddress" : 1, "_id" : -1 }
{ "meta.hash" : 1 }
{ "meta.addresses" : 1 }
{ "meta.aggregateId" : 1 }
{ "meta.aggregateHash" : 1 }
bye
[.] (exit code: 0)
/
[+] db prepared, checking account indexes
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:27017/catapult?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("a9d9d654-d0c9-4760-bf52-94138f4c6871") }
MongoDB server version: 4.2.5
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "catapult.accounts"
},
{
"v" : 2,
"key" : {
"account.publicKey" : 1
},
"name" : "account.publicKey_1",
"ns" : "catapult.accounts"
},
{
"v" : 2,
"unique" : true,
"key" : {
"account.address" : 1
},
"name" : "account.address_1",
"ns" : "catapult.accounts"
}
]
2021-07-14T02:52:54.345+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2021-07-14T02:52:54.349+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup
killing process with pid: 10
2021/07/14 02:52 Start: mijin docker-compose
Creating network "catapult_default" with the default driver
Creating catapult_db_1 ... done
Creating catapult_api-node-broker_1 ... done
Creating catapult_rest-gateway_1 ... done
Creating catapult_api-node_1 ... done
2021/07/14 02:52 End: ALL Success
3.3.1.8. Step.7¶
Check the operation.
3.3.1.8.1. Confirmation of container operation¶
Make sure that all States are Up.
$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------
catapult_api-node-broker_1 bash -c /bin/bash /scripts ... Up
catapult_api-node_1 bash -c perl /scripts/wait ... Up 0.0.0.0:7900->7900/tcp
catapult_db_1 docker-entrypoint.sh bash ... Up 27017/tcp
catapult_rest-gateway_1 docker-entrypoint.sh ash - ... Up 0.0.0.0:3000->3000/tcp
3.3.1.8.2. Check to see if the block is progressing.¶
Check to see if the blocks are stacked.
Run it on other nodes to see if they are progressing in the same block.
For API nodes
catapult@api1:~/mijin-catapult-package/package/api/catapult$ docker-compose logs --tail=20 api-node| grep heights
api-node_1 | 2021-07-14 02:59:36.958509 0x00007f95631c8700: <info> (chain::ChainSynchronizer.cpp@217) peer returned 1 blocks (heights 400 - 400)
api-node_1 | 2021-07-14 02:59:36.958645 0x00007f95631c8700: <debug> (disruptor::Disruptor.cpp@43) disruptor queuing element 27 (1 blocks (heights 400 - 400) [00000000] from Remote_Pull with size 424B)
api-node_1 | 2021-07-14 02:59:37.046493 0x00007f953bfff700: <info> (disruptor::ConsumerDispatcher.cpp@44) completing processing of element 27 (1 blocks (heights 400 - 400) [F15ACAA0] from Remote_Pull with size 424B), last consumer is 0 elements behind
In the case of PEER node
$ docker-compose logs --tail=20 peer-node| grep heights
peer-node_1 | 2021-07-14 03:00:26.168343 0x00007fb1c396a700: <debug> (disruptor::Disruptor.cpp@43) disruptor queuing element 942 (1 blocks (heights 403 - 403) [00000000] from Remote_Push with size 376B)
peer-node_1 | 2021-07-14 03:00:26.257224 0x00007fb1a57fa700: <info> (disruptor::ConsumerDispatcher.cpp@44) completing processing of element 942 (1 blocks (heights 403 - 403) [5C675B6B] empty from Remote_Push with size 376B), last consumer is 0 elements behind
peer-node_1 | 2021-07-14 03:00:26.317777 0x00007fb1c3169700: <debug> (disruptor::Disruptor.cpp@43) disruptor queuing element 943 (1 blocks (heights 403 - 403) [00000000] from Remote_Push with size 376B)
peer-node_1 | 2021-07-14 03:00:26.368263 0x00007fb1a57fa700: <info> (disruptor::ConsumerDispatcher.cpp@44) completing processing of element 943 (1 blocks (heights 403 - 403) [5C675B6B] from Remote_Push with size 376B), last consumer is 0 elements behind
3.3.1.8.3. For API nodes, check if rest can connect¶
Check if node information can be obtained.
$ curl -Ss http://localhost:3000/node/info | jq -r
{
"version": 16777216,
"publicKey": "E4BF3706483B4D42243F3DCB2625021C3E3AE7C253CC466154EEDF9775012C20",
"networkGenerationHashSeed": "2DE20B93EBE048A3BA132CC9874BCABBC21C87E18FE9836B8D5D002E57640D4B",
"roles": 70,
"port": 7900,
"networkIdentifier": 96,
"host": "api1.mijin.internal",
"friendlyName": "api1.mijin.internal",
"nodePublicKey": "27E7EEAF5819493D60CA848BAA48145A1A97DF63596ED41394563C791303C778"
}