# Archiving

Miyabi archiving is an administrative offline tool. It can be used to perform archiving operations on Miyabi node's data, for example, backup, purge, and restore. Miyabi archiving functionality is based on physical snapshots of files having key/value pairs.

# Snapshot types

The archiving operation provides the ability to take data snapshots of two different types:

Block snapshot: The snapshot contains data related to blocks and its corresponding transactions.
State snapshot: The snapshot contains data related to world state.

# Block snapshot

The Block snapshot related operations can be used for below purposes:

Creating a backup of block data
Purging old block data
Restoring block data from an existing backup

# State snapshot

The State snapshot related operations can be used for below purposes:

Creating backup of world state data
Re-initializing a node using existing backup

# Archiving Operations

To perform archiving operation on block snapshot data or state snapshot data, below prerequisites are required:

The node on which archiving operations has to be performed, needs to be stopped.
miyabi-admin-cli (shipped with Miyabi binaries) must be available to be used.

# Block snapshot specific operations

# 1. Create block backup

Description: Create backup of blocks (block data and corresponding transactions data) of the storage at a given height or current height.

Usage:

miyabi-admin-cli -n <path of 'config_node.json' file> createblockbackup [Options]

Options:
  --help                 Show help information
  -h|--height <height>   [Optional] Height at which backup to be created.
  -d|--directory <path>  [Optional] Snapshot directory path
  --force                [Optional] Force execution, avoids user confirmation

Remarks

If height h is not specified, all the blocks till current height of node will be included in the snapshot.
If directory d is not specified, snapshot will be saved in path mentioned in config_node.json

 "snapshotConfig": {
    "blockBackupLocation": "<path for block snapshot backup>",
  },

If field blockBackupLocation doesn't exists or is empty, snapshot will be created where Miyabi binaries are located under _snapshots folder.

Use --force option to avoid confirmation prompt before executing the operation. (Not recommended)
Example of snapshot: <d>\BlockBackup\0_100.zip [i.e. snapshot has block data and transactions data from height 0 till given/current height 100]

# 2. Purge block data

Description: Purge all the blocks (block data and corresponding transactions data) to a given height and create an optional backup snapshot.

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> -b <path of 'config_blockchain.json' file> applyblockpurge [Options]

Options:
  --help                 Show help information
  -h|--height <height>   [Mandatory] Height at which purge to be applied.
  --no-backup            [Optional] No backup snapshot to be created
  -d|--directory <path>  [Optional] Snapshot directory path
  --force                [Optional] Force execution, avoids user confirmation

Remarks

For given height h, there can be below possibilities:
- All old blocks can be purged for a given height => Operation will return success status.
- No blocks are eligible to be purged because all blocks have been added recently => Operation will return failed status and can not be processed.
- Some old blocks can be purged but few recently added blocks cannot be purged => Operation will return a failed status with the suggested max eligible height which can be used as a new value for height h
--no-backup will permanently delete the data (Not recommended), use it with caution. If --no-backup is not specified, a backup will be created before the purge.
If directory d is not specified, backup snapshot will be saved in path mentioned in config_node.json

 "snapshotConfig": {
    "blockBackupLocation": "<path for block snapshot backup>",
  },

If field blockBackupLocation doesn't exist or is empty, snapshot will be created in the Miyabi binaries location under _snapshots folder.

Use --force option to avoid confirmation prompt before executing the operation. (Not recommended)

# 3. Restore block backup

Description: Apply the existing block backup snapshot to the current storage to restore blocks (block data and corresponding transactions data).

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> restoreblockbackup [Options]

Options:
  --help                 Show help information
  -d|--directory <path>  [Optional] Snapshot directory path
  --force                [Optional] Force execution, avoids user confirmation

Remarks

If directory d is not specified, the tool looks for the snapshot path automatically from the metadata log file. Metadata log file is a hidden file with name log.metadata and is stored at below path in config_node.json

"dbConfig": {
    "baseLocation": "<db path>",

If the snapshot to restore is not found, the tool looks for the snapshot in path mentioned in config_node.json

 "snapshotConfig": {
    "blockBackupLocation": "<path for block snapshot backup>",
  },

If field blockBackupLocation doesn't exists or is empty, the tool looks for the snapshot where Miyabi binaries are located under _snapshots folder. Make sure snapshot file, which has to be restored, is available at any of these path.

Use --force option to avoid confirmation prompt before executing the operation. (Not recommended)
Node can only restore its own block backup, i.e. block backup of one node can not be restored to another node.

# 4. Restore all block backups

Description: Apply all existing block backup snapshots to the current storage. The single command can restore all existing backup snapshots.

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> restoreallblockbackup [Options]

Options:
  --help   Show help information
  --force  [Optional] Force execution, avoids user confirmation

Remarks

The tool looks for all existing snapshot path automatically from the metadata log file. Metadata log file is a hidden file with name log.metadata and is stored at below path in config_node.json

"dbConfig": {
    "baseLocation": "<db path>",

If a particular snapshot is not found, the tool looks for it in the path mentioned in config_node.json

 "snapshotConfig": {
    "blockBackupLocation": "<path for block snapshot backup>",
  },

If field blockBackupLocation doesn't exists or is empty, the tool looks for the snapshot where Miyabi binaries are located under _snapshots folder. Make sure all snapshot files, which have to be restored, are available at any of these path.

Use --force option to avoid confirmation prompt before executing the operation. (Not recommended)
Node can only restore its own block backup, i.e. block backup of one node can not be restored to another node.

# 5. Get current purged height

Description: API which gets the height at which blocks have been purged and don't exist anymore.

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> getblockprunedheight [Options]

Options:
  --help                Show help information
  --lookup-in-database  [Optional] Check in database if enabled else check in metadata.

Remarks

--lookup-in-database can be used when metadata log file is corrupted, missing, or does not exist. (Not recommended)

# 6. Get archiving history

Description: Get history of all archiving operations from the metadata log file

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> getprunehistory [Options]

Options:
  --help                Show help information
  --show-block-backup   [Optional] Show only block backup history.
  --show-block-purge    [Optional] Show only block purge history.
  --show-block-restore  [Optional] Show only block restore history.

Remarks

If no option is provided, it lists all the archiving actions (including actions on block and state snapshot) performed on the node.

# State snapshot specific operations

# 1. Create state backup

Description: Create world state backup of the storage at the current height.

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> -b <path of 'config_blockchain.json' file> createstatebackup [Options]

Options:
  --help                   Show help information
  -d|--directory <path>    [Optional] Snapshot directory path
  --include-private-state  [Optional] Include private state data
  --force                  [Optional] Force execution, avoids user confirmation

Remarks

If directory d is not specified, snapshot will be saved in path mentioned in config_node.json

 "snapshotConfig": {
    "stateBackupLocation": "<path for state snapshot backup>",
  },

If field stateBackupLocation doesn't exists or is empty, snapshot will be created where Miyabi binaries are located under _snapshots folder.

Use --include-private-state to include private data. This option must be used only when Miyabi's PrivateData module is enabled to create private channel(s).
Use --force option to avoid confirmation prompt before executing the operation. (Not recommended)
Example of snapshot: <d>\WorldStateBackup\0_100.zip [i.e. snapshot has world state data from height 0 till current height 100]
Keep security in mind if you share the state snapshot (with other party to initialize/run the new node).

# 2. Restore state backup

Description: There are mainly 2 use cases of restoring state snapshot:

Node's storage data has been deleted accidentally.
When a new node joins the network and needs to catch up
- The new node can not prepare the world state because all other nodes of the network have already purged some or all of their blocks.
- The process of loading all the blocks from the inventory layer and preparing world state is very time-consuming and requires network bandwidth.
- The new node doesn't need historical blocks because restoring state snapshot does not restore old blocks automatically. Please use restoreblockbackup command to restore blocks explicitly.

Usage: Restoring of state backup has no cli command and must be done at the time of node bootstrapping.

Below are the steps to restore state snapshot for same or newly fresh node.

Set the state snapshot path in config_node.json

"snapshotConfig": {
  "stateBackupLocation": "<path for state snapshot backup>",
},

Copy the physical snapshot to the above path.
Start the node which will automatically restore and initialize the state from the snapshot.

Remarks

Restoring a snapshot can be applied on same node or newly added node only when the node's height is 0. It means, you must first delete (after taking a backup) any existing old storage at below path in config_node.json:

"dbConfig": {
    "baseLocation": "<db path>",

Security must be kept in mind when sharing the state snapshot with other parties.

# 3. Get pruning/archiving history

Description: Get history of all archiving operations performed by the archiving tool

Usage:

Usage: miyabi-admin-cli -n <path of 'config_node.json' file> getprunehistory [Options]

Options:
  --help                Show help information
  --show-state-backup   Show only world (and private) state backup history.

Remarks

If no option is provided, it lists all the archiving actions (including actions on block and state snapshot) performed on the node.

# Scenarios

# Block snapshot

# Scenario : Purge block data up to a specific height `h`

Stop the node
Purge block data at height h. The operation will automatically create a backup (e.g.: for given h=100, backup snapshot file name would be \BlockBackup\0_100.zip)
Copy this block backup file (i.e.: \BlockBackup\0_100.zip) to cold storage (e.g.: S3)
Now the block backup file can be safely removed from the node
If needed in future, restore this or all other backups back to node
Restart the node

# Scenario : Create a block backup up to a specific height

Stop the node
Create block backup (e.g.: for given or current height h=100, backup snapshot file name would be \BlockBackup\0_100.zip)
Copy this block backup file (i.e.: \BlockBackup\0_100.zip) to cold storage (e.g.: S3)
Now the block backup file can be safely removed from the node
Restart the node

# Scenario : Create a block backup and purge block data at height `h`

Stop the node
Create block backup (e.g.: for given or current height h=100, backup snapshot file name would be \BlockBackup\0_100.zip)
Copy this block backup file (i.e.: \BlockBackup\0_100.zip) to cold storage (e.g.: S3)
Purge block data at height h (option no-backup can be used because backup has already been taken)
Now the block backup file can be safely removed from the node
If needed in future, restore this or all other backups back to node
Restart the node

# State snapshot

# Scenario : Create a state backup up to a specific height

Stop the node
Create state backup (e.g.: for current height h=100, backup snapshot file name would be \WorldStateBackup\0_100.zip)
Copy this state backup file (i.e.: \WorldStateBackup\0_100.zip) to cold storage (e.g.: S3)
Now the state backup file can be safely removed from the node
Restart the node

# Scenario : Create a state backup and apply back up on other nodes

Stop the node
Create state backup (e.g.: for current height h=100, backup snapshot file name would be \WorldStateBackup\0_100.zip)
Copy this state backup file (i.e.: \WorldStateBackup\0_100.zip) to cold storage (e.g.: S3)
Now the state backup file can be safely removed from the node
Restart the node
Share the state backup with other new node
New node restores the state data from the shared backup file at the time of bootstrap

← Node Config

# Archiving

# Snapshot types

# Block snapshot

# State snapshot

# Archiving Operations

# Block snapshot specific operations

# 1. Create block backup

# 2. Purge block data

# 3. Restore block backup

# 4. Restore all block backups

# 5. Get current purged height

# 6. Get archiving history

# State snapshot specific operations

# 1. Create state backup

# 2. Restore state backup

# 3. Get pruning/archiving history

# Scenarios

# Block snapshot

# Scenario : Purge block data up to a specific height h

# Scenario : Create a block backup up to a specific height

# Scenario : Create a block backup and purge block data at height h

# State snapshot

# Scenario : Create a state backup up to a specific height

# Scenario : Create a state backup and apply back up on other nodes

# Scenario : Purge block data up to a specific height `h`

# Scenario : Create a block backup and purge block data at height `h`