SafeStake: Running an Operator Node (on going)

Updates happen frequently! Our Github always has the latest operator node resources and setup instructions.

Deploy the Operator node

Dependencies

Server Host

  • Public Static Network IP of IPv4 (need to disable IPv6)
  • Hardware
    • (Standalone Mode Recommend)
      • CPU: 16
      • Memory: 32G
      • Disk: 600GB
    • (Light Mode Recommend)
      • CPU: 2
      • Memory: 4G
      • Disk: 200GB
  • OS
    • Unix
  • Software
    • Docker
    • Docker Compose

Running Mode Of Operator Node

Standalone Mode
Standalone mode contains the following list of programs/soft on a single host:
  • Geth/Nethermind/Besu/Erigon Service
  • Lighthouse Service
  • OperatorNode Service
Light Mode
Light mode contains only the OperatorNode service, the following list of programs/soft on a host:
  • OperatorNode Service
Geth/Nethermind/Besu/Erigon service and Lighthouse service can run on other hosts. Users should configure the beacon node endpoint (discussed later) in order to connect to Lighthouse's beacon node instance. The purpose of this is to make the architecture clearer and easier to scale operator nodes. And the cost efficiency ratio of infrastructure will be higher.

Preparation: Get your Infura WS_URL

  • Select "WEBSOCKETS"
  • Select "Görli" network under "Ethereum"
  • Copy your "WS_URL", mkae sure the url begins with wss:// instead of https://

Deployment

1. Set firewall rule

Log in to your host cloud service provider, open the following firewall inbound rules:
Type
IpProtocol
Port
IpRanges
Usage
Inbound/Ingress
TCP & UDP
30303
0.0.0.0/0
Geth/Nethermind/Besu/Erigon p2p
Inbound/Ingress
TCP & UDP
9000
0.0.0.0/0
Lighthouse p2p
Inbound/Ingress
TCP
5052
Internal
Operator - Lighthouse
Inbound/Ingress
TCP
8551
Internal
Lighthouse - Geth/Nethermind/Besu/Erigon
Inbound/Ingress
TCP
26000
0.0.0.0/0
hotstuff consensus
Inbound/Ingress
TCP
26001
0.0.0.0/0
hotstuff consensus
Inbound/Ingress
TCP
26002
0.0.0.0/0
hotstuff consensus
Inbound/Ingress
TCP
26003
0.0.0.0/0
When aggregating signatures, operator nodes use this port to request signature from each other
Inbound/Ingress
UDP
26004
0.0.0.0/0
Node discovery
Inbound/Ingress
TCP
26005
0.0.0.0/0
DKG port, which will listen only when DKG is triggered. By default, the port won't listen.

2. SSH Login to your server (jumpserver recommand)

3. Install Docker and Docker compose

4. Enable docker service and start it immediately.

sudo systemctl enable docker

5. Create local volume directory

sudo mkdir -p /data/geth
# OR, if you use Nethermind/Besu/Erigon:
# sudo mkdir -p /data/nethermind
# sudo mkdir -p /data/besu
# sudo mkdir -p /data/erigon
sudo mkdir -p /data/lighthouse
sudo mkdir -p /data/jwt
sudo mkdir -p /data/operator

6. Generate your jwt secret to jwt dirctory

openssl rand -hex 32 | tr -d "\n" | sudo tee /data/jwt/jwtsecret

7. Clone operator code from Github

git clone --recurse-submodules https://github.com/ParaState/SafeStakeOperator.git dvf

8. Running Geth/Nethermind/Besu/Erigon & Lighthouse Service

NOTE: This step is to provide a quick way to setup and run the execution client and consensus client of goerli testnet. If you already have a node running execution client and consensus client, you can skip this step.
cd dvf
cp .env.example .env
sudo docker compose -f docker-compose-operator.yml up geth -d
# OR, if you use Nethermind/Besu/Erigon:
# sudo docker compose -f docker-compose-operator.yml up nethermind -d
# sudo docker compose -f docker-compose-operator.yml up besu -d
# sudo docker compose -f docker-compose-operator.yml up erigon -d
sudo docker compose -f docker-compose-operator.yml up lighthouse -d
NOTE: Remember to open the 5052 firewall port for this host
Syncing data may take several hours. You can use the command to see the latest logs of lighthouse to check if the data is synced:
sudo docker compose -f docker-compose-operator.yml logs -f --tail 10 lighthouse
Once the data is synced, you will see output like below:
INFO Synced, slot: 3690668, block: 0x1244…cb92, epoch: 115333, finalized_epoch: 115331, finalized_root: 0x0764…2a3d, exec_hash: 0x929c…1ff6 (verified), peers: 78
or you can use this command to check if lighthouse is synced:
curl -X GET "http://localhost:5052/lighthouse/syncing" -H "accept: application/json"
if the output shows {"data":"Synced"}, it means it is already synced.

9. Edit local environment variables

vim .env
Now that we have open the .env file, we will update the values based on our own configuration.
Leave these variables unchanged now:
GETH_NETWORK=goerli
NETHERMIND_NETWORK=goerli
BESU_NETWORK=goerli
ERIGON_NETWORK=goerli
LIGHTHOUSE_NETWORK=prater
OPERATOR_NETWORK=prater
IMAGE_TAG=v1.2-testnet
REGISTRY_CONTRACT_ADDRESS=f31605c163b54C00371b10af21E8eDa32B969F21
NETWORK_CONTRACT_ADDRESS=C1b4AA96afA5D3566A86920e69Fc6C274d54F3B4
API_SERVER=https://api-testnet.safestake.xyz/v1/
# different chain has different ttd
TTD=10790000
# separated by ',' for multiple relays, such as MEV_BOOST_RELAYS=xxx,xxx,xxx
MEV_BOOST_RELAYS=https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de3[email protected]
#gas limit. [default: 30,000,000]
GAS_LIMIT_INTEGER=30000000
OPERATOR_ID=<YOUR_OPERATOR_ID>
Update these variables with yours
WS_URL= #YOUR Infura WSS URL
BEACON_NODE_ENDPOINT= # The beacon node endpoint. Depending on whether you are running single-node mode or multi-node mode, fill in the correct Lighthouse beacon node service url, e.g. http://127.0.0.1:5052 for a local node
For BEACON_NODE_ENDPOINT, if you follow the previous step to run Geth/Nethermind/Besu/Erigon and Lighthouse and you want operator runs on the same machine, then you can use a local IP:
BEACON_NODE_ENDPOINT=http://127.0.0.1:5052
Otherwise, suppose the host where you run the Lighthouse & Geth/Nethermind/Besu/Erigon service has an IP 12.102.103.1, then you can set:
BEACON_NODE_ENDPOINT=http://12.102.103.1:5052

10. Generate a registration public and private key

sudo docker compose -f docker-compose-operator.yml up dvf_key_tool
Output:
...
dvf-dvf_key_tool-1 | INFO: node public key AtzozvDHiWUpO+oJph2ikv+EyBN5pdBXsfgZqLi0+Yqd
dvf-dvf_key_tool-1 exited with code 0
Save the public key, which will be used later. Or you can find the public key in the "name" field of the file /data/operator/v1/prater/node_key.json
  • Click "Join As Operator".
  • Select a wallet where you have enough goerli testnet token to pay minimum fee to sign a transaction.
  • After you connect your wallet, click "Register Operator"
  • Your wallet address is auto filled. You need to enter the "Display Name" for your node and the "Operator Public Key" got from the previous step. Then click "Next".
  • Click "Register Operator"
  • Wallet extension page will pop out. You need to click "Confirm" to sign the transaction.
After we register an Operator on the Safestake website, we will be shown our OPERATOR ID, which is the unique identifier we need to start with. We will need to update the OPERATOR ID to the .env file before running the operator service.

12. Edit local environment variables for OPERATOR_ID

vim .env
OPERATOR_ID= #The Operator ID is the ID you receive after registering the operator on SafeStake website

13. Start operator service

sudo docker compose -f docker-compose-operator.yml up --force-recreate -d operator
Congratulations, now the Operator program has been installed and deployed.

Some final notes about Operator's private/public keys

You can always view your public key in case you forget it with the command:
sudo docker compose -f docker-compose-operator.yml logs -f operator | grep "node public key"
output
dvf-operator-1 | [2022-08-13T16:01:33.814Z INFO dvf::node::node] node public key Al0wMNz3JpkYDH7HVp93dZfLMt1GJHypLfhwOWS0NwC/
It is a good practice to back up your operator private key file
Keep it safe and put it in a safe place!
/data/operator/v1/prater/node_key.json
Your SafeStake Operator Node is now configured
then you may go to SafeStake website to register a validator and then choose your operator.

Backup and Migration

If you are using our default settings, all data other than configration files is stored in the folder /data. It is possible for Geth/Nethermind/Besu/Erigon and lighthouse to resync data in a new machine. For operator, it is important to always backup and copy the folder /data/operator/ to the new machine before you start operator in the new machine.
Some description of the folders and files under /data/operator/v1/prater/:
── prater
├── contract_record.yml # record the current synced block number
├── dvf_node_db # hotstuff consensus files
├── node_key.json # operator's public and private key
├── secrets # secret files for encryption
├── validators # data files of the validators that the operator is serving, inherited from the native folder of lighthouse validator client, including slashing_protection.sqlite, etc.

Common issues troubleshooting

graph TD;
A[Operator is shown as \n inactive/idle in explorer] --> B{any validator chooses\nthe operator?};
B --> |No| C[register a validator\n in our website and \n choose your operator];
B --> |Yes| D[check if the following errors \nshown in the log of first 100 lines];
D --> |?| E[Wrong scheme: https];
E --> |solution| F["WS_URL in .env file should be set\n beginning with wss:// instead of https://"];
F --> G[change the block number in the file\n /data/operator/v1/prater/contract_record.yml to \na block number before the registration of the validator];
G --> H[restart operator];
D --> |?| I["Ssl(Error { code : ErrorCode(1)"];
I --> |solution| J[ws is not supported, use\n wss instead of ws for WS_URL];
J --> G;
D --> |?| K["Failed to connect to {ip}:26000"];
K --> |solution| L[need to open the port 26000 to the internet,\n also carefully check if other firewall rules shown\n in the doc are set correctly in your server];