Validator FAQ

Heimdall shows "Failed Sanity Checks"

Addressbook warnings can be ignored without an issue most of the time. If your node is connected to sufficient number of peers, these kind of errors can be ignored. Your pex is just trying to re-establish its connections with peers already present in addrbook.json.

Heimdall and Bor logs are fine and even my bridge is running correctly but my node is not signing any checkpoints

This could happen if you have missed adding the ETH_RPC_URL in the heimdall-config.toml file. Please check if you have added it. If not, ensure that you add the correct URL and then restart your Heimdall service.

Can I start Bor before Heimdall is completely synced?

No, you cannot. If you start your Bor without Heimdall being completely synced, you face issues on your Bor.

Validator Heimdall is unable to connect to peers

This typically means that your sentry Heimdall is running into issues. Check your sentry Heimdall and see if the service is running fine. If the service is stopped, then restarting the service on your sentry should resolve this issue. Similarly, after fixing your sentry, a restart of your Heimdall service should also resolve the problem.

Heimdall shows "pong timeout"

Full error:

E[2021-03-01|13:19:12.252] Connection failed @ sendRoutinemodule=p2ppeer=3[email protected]3.217.49.94:26656 conn=MConn{3.217.49.94:26656} err="pong timeout"

Usually restarting the Heimdall service should resolve the problem for you.

Heimdall shows "Error: Wrong Block.Header.AppHash. Expected xxxx"

This usually occurs due to an incorrect installation of Heimdall.

To fix, run:

heimdalld unsafe-reset-all

Then start Heimdall services again.

See:

Heimdall shows "dpkg: error processing archive"

Full error:

dpkg: error processing archive matic-heimdall_1.0.0_amd64.deb (--install): trying to overwrite '/heimdalld-rest-server.service', which is also in package matic-node 1.0.0

This occurs mainly because of a previous installation of Polygon on your machine. To resolve you can run:

sudo dpkg -r matic-node

I'm not clear on which private Key should I add when I generate validator key

The private key to be used is your wallet's ETH address where your Polygon tokens are stored.

Is there a way to know if Heimdall is synced?

You can run the following command to check it:

curl http://localhost:26657/status

Check the value of catching_up. If it is false, then the node is all synced up.

Whats the difference between ~.heimsdall and /etc/heimsdall?

~/.heimsdall is the Heimdall directory when you use the binary installation method.

/etc/heimdall is for the Linux package installation method.

Where can I find Heimdall account info location?

For binaries: ~/.heimdalld/config.

For Linux package /etc/heimdall/config.

Which file do I add the persistent_peers?

You can add the persistent_peers to ~/.heimdalld/config/config.toml.

Heimdall shows “Did you reset Tendermint without resetting your application's data?”

Reset the Heimdall config data and try running the installation again:

heimdalld unsafe-reset-all
rm -rf $HEIMDALLDIR/bridge

Heimdall shows a Panic error

Full error:

panic: Unknown db_backend leveldb, expected either goleveldb or memdb or fsdb

Change the config to goleveldb in config.toml.

Are the private keys same for Heimdall and Bor keystore?

Yes, the private key used for generating the validator keys and Bor keystore is the same. The private key used in this instance is your wallet's ETH address where your Polygon tokens are stored.

Sentry Bor shows 'Looking for peers' and cannot find peers

This could happen when Bor has lost connectivity with other peers. Generally checking the ~/node/bor/start.sh file should show you your bootnodes. Check if the bootnodes are entered correctly without any formatting issues. If you have made any changes to the file, then please restart your Bor service and check if the issue is resolved.

If the issue persists, contact support team on Discord.

Bor shows "Failed to prepare header mining at block 0"

This happens because of a formatting issue in your ~/.bor/data/bor/static-nodes.json file. Ensure there are no space and no additional characters like < / > . If you have made any changes to the file then please restart your Bor service and you should see logs printing.

Bor shows "30303 or invalid command: /home/ubuntu/.bor/password.txt"

This is because you have not created the Bor keystore and the password file for it. Ensure that you follow all the steps from the guide setup.

Bor shows "Impossible reorg, please file an issue"

Let these logs be. Your node should ideally not suffer because of this and the issue should be automatically resolved.

If your node is suffering because of this, please contact the support team on Discord.

Bor shows "Failed to prepare mining for header"

This message is not an error.

The message indicates that the Bor node is not the one creating blocks right now.

Bor shows "Invalid Merkle root" or "Retrieved hash chain is Invalid"

Typically, this issue occurs because of 2 reasons. One where your Bor has seemingly crashed and has started giving you these errors or it has lost out sync with Heimdall.

To resolve this there are 2 ways to do this:

  • Restart your Bor service and check if the issue is resolved. Usually restarting your Bor service should resolve the issue.
  • Check if your Heimdall is running correctly. If your Heimdall has stopped, restart your Heimdall service and let your Bor starty syncing and it should resolve the issue.

If the issue is not resolved, contact support team on Discord.

Bor shows "Address is required as argument"

This means that you have not added your signer address to the metadata. You can add it using this path /etc/matic/metadata . Once the address is added, you can then restart the Bor service and everything should be fine.

Bor shows "Failed to unlock account (0x...) No key for given address or file"

This error occurs because the path for the password.txt file is incorrect.

To fix:

  1. Copy the Bor keystore file to /etc/bor/dataDir/keystore
  2. Copy the password.txt file to /etc/bor/dataDir/
  3. Make sure you have added the correct address in /etc/bor/metadata

For binaries:

  1. Copy the Bor keystore file to ~/.bor/keystore/
  2. Copy password.txt file to ~/.bor/password.txt

Node is not signing any checkpoints

Your node not signing checkpoints could be for multiple reasons:

  1. Check if your Heimdall service is running correctly on your sentry and validator nodes. If the service has stopped abruptly or you see any errors, try restarting your Heimdall service and see it comes back to normal. If the issue persists, contact support team on Discord.
  2. Check your Bor service and see if it has halted abruptly or there are any errors on the logs. Try restarting your Bor service to resolve this issue. If the issue persists, contact support team on Discord.
  3. Check if your Heimdall Bridge is running or not or if it has any errors in the logs. Try restarting the service and see if the issue resolves. If the issue persists, contact support team on Discord.

If none of this is the issue, contact support team on Discord.

How to set up a validator node on the mainnet?

See Getting Started

How to set up a non-validating node?

See:

Why do I have to keep ETH in my signer account?

ETH is required on your signer account because for submitting checkpoints to Ethereum, all transactions require ETH to be used as gas. Hence ETH is required on your signer account.

Setting up a node with Ansible errors out with "Host not found"

This could be because your inventory.yml file may have some formatting issues. Correct them with proper indentation and then try again.

As a validator do I need to run both a sentry and a validator node?

Yes, you have to run both a sentry and a validator node.

The Polygon ecosystem and architecture demands that you run a sentry + validator setup to ensure that your validator node is not exposed to the public and only your sentry node is.

Your sentry node gleans information / blocks from the network and then relays them to the validator for validation.

What is the minimum disk space required to run a Validator node?

See Validator Node System Requirements.

Bridge shows "Error while fetching mainchain receipt error="

These are normal logs. Do not do anything to your bridge. Let it run as is.

Validator Bor is stuck on a block for a long time

This means that your Bor on your sentry is also stuck because your validator gets information from your sentry.

Please check your Bor logs on your sentry and see if everything is okay.

Restart the Bor service one on your Bor and then simultaneously restart your Bor service on your validator as well.

Upgrading Bor shows "build github.com/ethereum/go-ethereum/cmd/geth cannot load hash/maphash: malformed module path "hash/maphash": missing dot in first path element"

This is because your Go Version is outdated. The recommended Go version is 1.15.x.

Can I run multiple sentries for a validator?

Yes, you can.

Can I run multiple validators using the same signer key?

No. You cannot. Polygon's architecture currently does not allow validators running multiple validator nodes using the same signer key.

Is there a way to run a light Bor node?

There is no light node option as of now.

To run a full node, see:

What is the uptime percentage calculation on the staking dashboard?

It is calculated as per the last 200 checkpoints submitted to the ones you have actually signed.

What ports are to be kept open on the sentry node?

You will need to make sure that you open ports 22, 26656 and 30303 to world (0.0.0.0/0) on sentry node firewall.

What is the command to check the latest block height on Heimdall?

You can run this command curl localhost:26657/status.

What is the command to check the latest block height on Bor?

Run the following command:

curl http://<your ip>:8545 -X POST -H "Content-Type: application/json" -d '
{"jsonrpc":"2.0", "id":1, "method":"bor_getSigners", "params":["0x98b3ea"]}
'

Bor shows "ERROR[03-01|13:22:55.320] Block receipts missing, can't freezenumber=9397329 hash="2c38b0...cb41e7"**

This is generally not an error and should resolve on its own.

Standard upgrade commands for Heimdall

cd ~/heimdall
git pull
git checkout <branch tag>
make install
sudo service heimdalld restart

Standard upgrade commands for Bor

cd ~/bor
git pull
git checkout <branch tag>
sudo service bor stop
make bor-all
sudo service bor start

How do I delete remnants of Heimdall and Bor?

If you want to delete the remnants of Heimdall and Bor, you can run the following commands

Bor:

For Linux package:

sudo dpkg -i matic-bor

And delete Bor directory:

sudo rm -rf /etc/bor

For Binaries:

sudo rm -rf /etc/bor

And:

sudo rm /etc/heimdall

Bridge shows "Object "start" is unknown"

Check which bridge — if it's /usr/sbin/bridge, you are not running the right "bridge" program.

Try ~/go/bin/bridge instead (or $GOBIN/bridge)

Error: Unable to unmarshall config Error 1 error(s) decoding

Full error:

'' has invalid keys: clerk_polling_interval, matic_token, span_polling_interval, stake_manager_contract, stakinginfo_contract

This occurs mostly because when there are typos, or some missing parts or an old config file which is still a remnant. You will need to clear all the remnants and then try setting it up again.

To stop Heimdall and Bor services

For Linux packages:

Stop Heimdall: sudo service heimdalld stop

Stop Bor: sudo service bor stop or

  1. ps -aux | grep bor. Get the PID for Bor and then run the following command.
  2. sudo kill -9 PID

For Binaries:

Stop Heimdall: pkill heimdalld

Stop Bridge: pkill heimdalld-bridge

Stop Bor: Go to CS-2001/bor and then run, bash stop.sh

To remove Heimdall and Bor directories

For Linux packages:

Delete Heimdall: sudo rm -rf /etc/heimdall/*

Delete Bor: sudo rm -rf /etc/bor/*

For Binaries:

Delete Heimdall: sudo rm -rf ~/.heimdalld/

Delete Bor: sudo rm -rf ~/.bor

List of common commands

Where to find Heimdall genesis file

$CONFIGPATH/heimdall/config/genesis.json

Where to find heimdall-config.toml

/etc/heimdall/config/heimdall-config.toml

Where to find config.toml

/etc/heimdall/config/config.toml

Where to find heimdall-seeds.txt

$CONFIGPATH/heimdall/heimdall-seeds.txt

Start Heimdall

$ sudo service heimdalld start

Start Heimdall rest-server

$ sudo service heimdalld-rest-server start

Start Heimdall bridge-server

$ sudo service heimdalld-bridge start

Heimdall logs

/var/log/matic-logs/

Where to find Bor genesis file

$CONFIGPATH/bor/genesis.json

Start Bor

sudo service bor start

Check Heimdall logs

tail -f heimdalld.log

Check Heimdall rest-server

tail -f heimdalld-rest-server.log

Check Heimdall bridge logs

tail -f heimdalld-bridge.log

Check Bor logs

tail -f bor.log

Kill Bor process

For Linux

  1. ps -aux | grep bor. Get the PID for Bor and then run the following command.
  2. sudo kill -9 PID

For binaries

Go to CS-2003/bor and then run, bash stop.sh

Diagnosing what went wrong in a node

You can use this script to check periodially the sync status of your node.

Last updated on