Validator Incident Response Playbook

Validator downtime can quickly escalate into missed blocks, jail events, and potential slashing depending on chain rules. Fast response and disciplined operational procedures are essential for maintaining validator uptime and delegator trust.

This guide outlines the practical response workflow when validator alerts indicate a potential issue. All commands below are verified against a live Infinite Drive mainnet validator.

Operators commonly rely on monitoring systems to detect validator health issues quickly. FoxxOne Validator Alerts provides Telegram notifications for missed blocks, jailed status, governance proposals, and stake movement events.

Last updated: 2026-06-27

Validator Incident Response Guide

Structured response flow for live operational incidents. Read the workflow phases below first, then jump to the Terminal Runbook for verified commands.

Preventing Validator Incidents

Most validator incidents come from predictable operational faults. Baseline prevention should include:

• Maintaining healthy peer connectivity.
• Monitoring node sync status and block-height drift.
• Tracking missed block counters continuously.
• Ensuring signer process stability and key access health.
• Watching disk space — full disks silently halt the node.
• Keeping the system clock in tight NTP sync — Cosmos consensus rejects out-of-sync signers.
• Maintaining validator infrastructure alerts as a first-line signal.

Runbook Workflow

Detection

Missed blocks or jailed validator alerts are usually the first indicator of an operational issue.

Immediate Response

1. Acknowledge the alert and record a timestamp.
2. Confirm if the issue is local (your stack) or chain-wide. Compare your local block height against a known-good RPC — if both are stuck, the chain itself has halted (very rare); if only yours is stuck, the issue is local.
3. Pause non-essential maintenance until validator health stabilizes.

Diagnosis

1. Check node logs for consensus/signing/runtime errors.
2. Check sync state and block-height movement (catching_up must be false before any recovery action).
3. Check disk space on the data partition — a full disk silently freezes the node.
4. Check peer connectivity and network latency.
5. Check signer process, key access, and service state.
6. Check system clock synchronisation (NTP drift greater than ~1s causes signing rejection).

Recovery

1. Restart services only after cause is identified — restarting without fixing the root cause leads to repeated jail cycles.
2. NEVER start the validator on a new machine without first stopping the old one and verifying the last signed block in priv_validator_state.json. Double-signing results in permanent tombstoning — no recovery possible.
3. If jailed: check tombstone status first (permanent if true), then confirm self-delegation is above the minimum, then execute the chain-specific unjail flow only when fully synced.
4. Monitor block signing recovery and missed-block trend after recovery.

Post-Incident Review

1. Document the incident and timeline in your ops log.
2. Update your runbook with the exact remediation sequence.
3. Adjust monitoring thresholds or alert routing where needed.

Terminal Runbook (Infinite Chain)

Commands verified against a live Infinite Drive mainnet validator. They assume the FoxxOne Drive deployment layout — if your infinited binary lives elsewhere, drop the ./drive.sh exec infinite wrapper and use infinited directly with the same flags.

• Service directory: ~/drive/services/node0-infinite
• Run commands from this directory unless noted.
• Keyring backend: file (prompts for passphrase on tx commands).
• Ignore the proto: duplicate proto type registered noise — those lines fire on every command and are harmless.
• Always pass -o json to queries piped through jq (the binary defaults to YAML output otherwise).
• Replace <your-key-name> with the name from infinited keys list (e.g. validator-key) and <your-valoper-address> / <your-wallet-address> with your validator's bech32 addresses.

Verify Node Sync Status

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq '.sync_info'

Critical check: catching_up must be false. If true, do not attempt unjail.

Quick Sync Check

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq -r '.sync_info.catching_up'

Expected output: false

Chain-vs-Local Halt Check

If your node is stuck, compare its block height to a public Infinite Drive endpoint. If both are stuck at the same height, the chain itself has halted (rare). If only yours is stuck, it's a local issue. Infinite Drive is a Cosmos-EVM chain — block heights are identical across both layers, so an EVM RPC peer works for this comparison.

cd ~/drive/services/node0-infinite

# Local Cosmos block height (decimal)
./drive.sh exec infinite infinited status -o json | jq -r '.sync_info.latest_block_height'

# Public peer block height via Infinite Drive EVM RPC (hex result, converted to decimal)
curl -sS -X POST https://evm-rpc.infinitedrive.xyz \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  | jq -r '.result' | xargs printf "%d\n"

The two values should match within ~1-2 blocks (in-flight delay). If your local is stuck thousands of blocks behind, it's a local sync issue. If the peer is also stuck at the same height as yours, the chain itself has halted.

Disk Space (top cause of silent node halt)

df -h ~/drive/services/node0-infinite

Anything below ~5% free needs urgent action: prune state, restore from snapshot, or expand disk.

System Clock / NTP Sync

timedatectl status | grep -E "synchronized|NTP"

If "System clock synchronized: no", fix NTP before any other recovery action — Cosmos rejects signers more than ~1s out of sync.

Get Your Validator's Consensus Address

Slashing queries take a consensus address (infinitevalcons1…), not your operator or wallet address. Get yours:

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited comet show-address

Save the output — it's used in the slashing queries below.

Alert-Specific Quick Checks

These checks map directly to common alert types and avoid duplicating the sync/unjail steps below.

Missed Blocks Signal — your validator specifically

The first line below captures your validator's consensus address into $VALCONS so the rest of the slashing commands in this guide can reuse it. Run all three lines in one paste.

cd ~/drive/services/node0-infinite
VALCONS=$(./drive.sh exec infinite infinited comet show-address)
./drive.sh exec infinite infinited q slashing signing-info "$VALCONS" -o json | jq

Key fields: missed_blocks_counter (current count), jailed_until (timestamp when an active jail expires), and tombstoned (true = validator permanently removed — see Recovery section). When values are zero/false the SDK omits them from JSON — use // "0" or // false fallbacks in jq if you script around this.

Slashing Parameters (jail thresholds)

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q slashing params -o json | jq

Infinite Drive mainnet values (as of 2026-06-27): signed_blocks_window 10,000, min_signed_per_window 0.05, downtime_jail_duration 10m, slash_fraction_double_sign 5%, slash_fraction_downtime 0.01%. In practice: jail triggers at 9,500 missed blocks within a 10,000-block window — generous downtime tolerance, but double-signing slashes 5% of stake and tombstones the validator. The 10-minute jail period is short — quick recovery is possible once root cause is fixed.

Governance Voting Window

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q gov proposals --status voting_period -o json | jq

Delegation / Stake Movement Baseline

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q staking validator <your-valoper-address> -o json | jq '.tokens,.delegator_shares,.min_self_delegation'

cd ~/drive/services/node0-infinite
VALCONS=$(./drive.sh exec infinite infinited comet show-address)
./drive.sh exec infinite infinited q slashing signing-info "$VALCONS" -o json \
  | jq -r '.val_signing_info.tombstoned // false'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq -r '.sync_info.catching_up'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q staking validator <your-valoper-address> -o json | jq '.jailed'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q staking validator <your-valoper-address> -o json \
  | jq '.min_self_delegation,.tokens,.delegator_shares'
./drive.sh exec infinite infinited q staking delegation <your-wallet-address> <your-valoper-address> -o json \
  | jq '.balance'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx staking delegate <your-valoper-address> 1000000000000000000drop \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.5 \
  --gas-prices 5000000000drop -y'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited q bank balances <your-wallet-address> --home ~/.infinited -o json | jq'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited keys list --keyring-backend file --home ~/.infinited

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx slashing unjail \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.5 \
  --gas-prices 5000000000drop -y'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited q staking validator <your-valoper-address> -o json | jq '.jailed,.status'

cd ~/drive/services/node0-infinite
VALCONS=$(./drive.sh exec infinite infinited comet show-address)
watch -n 6 "./drive.sh exec infinite infinited q slashing signing-info $VALCONS -o json | jq '.val_signing_info'"

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited q distribution rewards <your-wallet-address> --home ~/.infinited -o json | jq'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited q distribution commission <your-valoper-address> --home ~/.infinited -o json | jq'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx distribution withdraw-all-rewards \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.3 --gas-prices 5000000000drop -y'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx distribution withdraw-rewards <your-valoper-address> \
  --commission \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.3 --gas-prices 5000000000drop -y'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx staking delegate \
  <your-valoper-address> \
  400000000000000000000drop \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.3 --gas-prices 5000000000drop -y'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited q bank balances <your-wallet-address> --home ~/.infinited -o json | jq'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited tx gov vote 7 yes \
  --from <your-key-name> \
  --chain-id infinite_421018-1 \
  --keyring-backend file \
  --home ~/.infinited \
  --gas auto --gas-adjustment 1.3 --gas-prices 5000000000drop -y'

./drive.sh exec infinite node-stop
./drive.sh exec infinite node-start

cd ~/drive/services/node0-infinite
./drive.sh node-logs

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq -r '.sync_info.latest_block_height'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq -r '.sync_info.latest_block_time'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq -r '.node_info.network'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite infinited status -o json | jq '.validator_info'

cd ~/drive/services/node0-infinite
./drive.sh exec infinite bash -lc 'infinited keys show <your-key-name> -a --keyring-backend file --home ~/.infinited'

Why Rapid Response Matters

Missing blocks for extended periods can lead to validator jail. Repeated incidents can result in delegator withdrawals and reduced validator reputation. Monitoring and fast operational response help minimise downtime and protect stake participation.

On Infinite Drive specifically: downtime tolerance is generous (~9,500 missed blocks within a 10,000-block window before jail, jail duration only 10 minutes), but double-signing is severe — 5% slash plus permanent tombstoning. The playbook prioritises preventing double-signs over rushing recovery.

Operational Monitoring

Operational monitoring lets validator operators detect issues early and respond before incidents escalate. FoxxOne Validator Alerts (@FoxxWatch_bot) provides Telegram notifications for missed blocks, jailed validator events, governance proposals, and stake movement monitoring.

This enables faster response when validator health changes.