SafeStake: Running an Operator Node

Updates happen frequently! Our Github always has the latest operator node resources and setup instructions.

Deploy the Operator node

Dependencies

Server Host

  • Public Static Network IP of IPv4 (need to disable IPv6)

  • Hardware

    • (Standalone Mode Recommend)

      • CPU: 16

      • Memory: 32G

      • Disk: 2TB

    • (Light Mode Recommend)

      • CPU: 2

      • Memory: 4G

      • Disk: 200GB

  • OS

    • Unix

  • Software

    • Docker

    • Docker Compose

Running Mode Of Operator Node

Standalone Mode

Standalone mode contains the following list of programs/soft on a single host:

  • Geth/Nethermind/Besu/Erigon Service

  • Lighthouse Service

  • OperatorNode Service

Light Mode

Light mode contains only the OperatorNode service, the following list of programs/soft on a host:

  • OperatorNode Service

Geth/Nethermind/Besu/Erigon service and Lighthouse service can run on other hosts. Users should configure the beacon node endpoint (discussed later) in order to connect to Lighthouse's beacon node instance. The purpose of this is to make the architecture clearer and easier to scale operator nodes. And the cost efficiency ratio of infrastructure will be higher.

Deployment

1. Set firewall rule

Log in to your host cloud service provider, open the following firewall inbound rules:

Type
IpProtocol
Port
IpRanges
Usage

Inbound/Ingress

TCP & UDP

30303

0.0.0.0/0

Geth/Nethermind/Besu/Erigon p2p

Inbound/Ingress

TCP & UDP

9000

0.0.0.0/0

Lighthouse p2p

Inbound/Ingress

TCP

5052

Internal

Operator - Lighthouse

Inbound/Ingress

TCP

8546

Internal

Operator - Geth/Nethermind/Besu websocket (port 8545 for Erigon)

Inbound/Ingress

TCP

8551

Internal

Lighthouse - Geth/Nethermind/Besu/Erigon

Inbound/Ingress

TCP

26000

0.0.0.0/0

hotstuff consensus

Inbound/Ingress

TCP

26001

0.0.0.0/0

hotstuff consensus

Inbound/Ingress

TCP

26002

0.0.0.0/0

hotstuff consensus

Inbound/Ingress

TCP

26003

0.0.0.0/0

When aggregating signatures, operator nodes use this port to request signature from each other

Inbound/Ingress

UDP

26004

0.0.0.0/0

Node discovery

Inbound/Ingress

TCP

26005

0.0.0.0/0

DKG port, which will listen only when DKG is triggered. By default, the port won't listen.

2. SSH Login to your server (recommand jumpserver)

3. Install Docker and Docker compose

4. Enable docker service and start it immediately.

sudo systemctl enable --now docker

5. Create local volume directory

sudo mkdir -p /data/geth
# OR, if you use Nethermind/Besu/Erigon:
# sudo mkdir -p /data/nethermind
# sudo mkdir -p /data/besu
# sudo mkdir -p /data/erigon
sudo mkdir -p /data/lighthouse
sudo mkdir -p /data/jwt
sudo mkdir -p /data/operator

6. Generate your jwt secret to jwt dirctory

openssl rand -hex 32 | tr -d "\n" | sudo tee /data/jwt/jwtsecret

7. Clone operator code from Github

git clone --recurse-submodules https://github.com/ParaState/SafeStakeOperator.git dvf

8. Running Geth/Nethermind/Besu/Erigon & mev-boost & Lighthouse Service

NOTE: This step is to provide a quick way to setup and run the execution client and consensus client. If you already have a node running execution client and consensus client, you can skip this step.

cd dvf
cp .env.example .env
sudo docker compose -f docker-compose-operator-mev.yml up geth -d
# OR, if you use Nethermind/Besu/Erigon:
# sudo docker compose -f docker-compose-operator-mev.yml up nethermind -d
# sudo docker compose -f docker-compose-operator-mev.yml up besu -d
# sudo docker compose -f docker-compose-operator-mev.yml up erigon -d
sudo docker compose -f docker-compose-operator-mev.yml up mev-boost -d
# in .env, set MEV_ENDPOINT to the url of mev-boost, e.g. http://127.0.0.1:18550 then start lighthouse
sudo docker compose -f docker-compose-operator-mev.yml up lighthouse -d

NOTE: Remember to open the 5052 firewall port for this host

Syncing data may take several hours. You can use the command to see the latest logs of lighthouse to check if the data is synced:

sudo docker compose -f docker-compose-operator-mev.yml logs -f --tail 10 lighthouse

Once the data is synced, you will see output like below:

INFO Synced, slot: 3690668, block: 0x1244…cb92, epoch: 115333, finalized_epoch: 115331, finalized_root: 0x0764…2a3d, exec_hash: 0x929c…1ff6 (verified), peers: 78

or you can use this command to check if lighthouse is synced:

curl -X GET "http://localhost:5052/lighthouse/syncing" -H  "accept: application/json"

if the output shows {"data":"Synced"}, it means it is already synced.

9. Edit local environment variables

vim .env

Now that we have open the .env file, we will update the values based on our own configuration.

Update these variables with yours

WS_URL= #YOUR WS URL: ws://<geth/nethermind/besu node ip>:8546 or ws://<erigon node ip>:8545
BEACON_NODE_ENDPOINT= # The beacon node endpoint. Depending on whether you are running single-node mode or multi-node mode, fill in the correct Lighthouse beacon node service url, e.g. http://127.0.0.1:5052 for a local node
# public ipv4 ip of the server running your operator
NODE_IP=<IP_ADDRESS>

For BEACON_NODE_ENDPOINT, if you follow the previous step to run Geth/Nethermind/Besu/Erigon and Lighthouse and you want operator runs on the same machine, then you can use a local IP:

BEACON_NODE_ENDPOINT=http://127.0.0.1:5052

Otherwise, suppose the host where you run the Lighthouse & Geth/Nethermind/Besu/Erigon service has an IP 12.102.103.1, then you can set:

BEACON_NODE_ENDPOINT=http://12.102.103.1:5052

10. Generate a registration public and private key

sudo docker compose -f docker-compose-operator-mev.yml up dvf_key_tool

Output:

...
dvf-dvf_key_tool-1  | INFO: node public key AtzozvDHiWUpO+oJph2ikv+EyBN5pdBXsfgZqLi0+Yqd
dvf-dvf_key_tool-1 exited with code 0

Save the public key, which will be used later. Or you can find the public key in the "name" field of the file /data/operator/v1/mainnet/node_key.json

11. Go to SafeStake website:

  • Click "Join As Operator".

  • Select a wallet where you have enough ETH to pay minimum fee to sign a transaction.

  • After you connect your wallet, click "Register Operator"

  • Your wallet address is auto filled. You need to enter the "Display Name" for your node and the "Operator Public Key" got from the previous step. Then click "Next".

  • Click "Register Operator"

  • Wallet extension page will pop out. You need to click "Confirm" to sign the transaction.

After we register an Operator on the Safestake website, we will be shown our OPERATOR ID, which is the unique identifier we need to start with. We will need to update the OPERATOR ID to the .env file before running the operator service.

12. Edit local environment variables for OPERATOR_ID

vim .env
OPERATOR_ID= #The Operator ID is the ID you receive after registering the operator on SafeStake website

13. (Optional) Customize the base port

You are able to change the ports that will be exposed in case the default ports 26000-26005 conflict with the ports you are using. In the file docker-compose-operator-mev.yml, change --base-port=26000 to the port you want in the operator's start command.

14. Start operator service

sudo docker compose -f docker-compose-operator-mev.yml up --force-recreate -d operator

Congratulations, now the Operator program has been installed and deployed.

Some final notes about Operator's private/public keys

You can always view your public key in case you forget it with the command:

sudo docker compose -f docker-compose-operator-mev.yml logs -f operator | grep "node public key"

output

dvf-operator-1 | [2022-08-13T16:01:33.814Z INFO dvf::node::node] node public key Al0wMNz3JpkYDH7HVp93dZfLMt1GJHypLfhwOWS0NwC/

It is a good practice to back up your operator private key file

Keep it safe and put it in a safe place!

/data/operator/v1/mainnet/node_key.json

Your SafeStake Operator Node is now configured

then you may go to SafeStake website to register a validator and then choose your operator.

Backup and Migration

If you are using our default settings, all data other than configration files is stored in the folder /data. It is possible for Geth/Nethermind/Besu/Erigon and lighthouse to resync data in a new machine. For operator, it is important to always backup and copy the folder /data/operator/ to the new machine before you start operator in the new machine.

Some description of the folders and files under /data/operator/v1/mainnet/:

── mainnet
    ├── contract_record.yml # record the current synced block number
    ├── dvf_node_db # hotstuff consensus files
    ├── node_key.json # operator's public and private key
    ├── secrets # secret files for encryption
    ├── validators # data files of the validators that the operator is serving, inherited from the native folder of lighthouse validator client, including slashing_protection.sqlite, etc.

Monitoring

You can use prometheus to fetch metrics from port 5064 of operator and monitor if the metric vc_signed_attestations_total{status="success"} is increasing to know if your operator is active.

Common issues troubleshooting

graph TD;
    A[Operator is shown as \n inactive/idle in explorer] --> B{any validator chooses\nthe operator?};
    B --> |No| C[register a validator\n in our website and \n choose your operator];
    B --> |Yes| D[check if the following errors \nshown in the log of first 100 lines];
    D --> |?| E[Wrong scheme: https];
    E --> |solution| F["WS_URL in .env file should be set\n beginning with ws:// or wss:// instead of https://"];
    F --> G[change the block number in the file\n /data/operator/v1/mainnet/contract_record.yml to \na block number before the registration of the validator];
    G --> H[restart operator];
    D --> |?| K["Failed to connect to {ip}:26000"];
    K --> |solution| L[need to open the port 26000 to the internet,\n also carefully check if other firewall rules shown\n in the doc are set correctly in your server];

Validator

Registration

Before registering a validator in SafeStake website, users need to deposit a validator first and the wait the validator to become active. Then users need provide the keystore file and select 4 operators which require at least 3 operators to do attestations. Finally, users need to deposit DVT to make operators start attestions for this validator.

Remove

Users can remove their validator from SafeStake by clicking the remove button in the account page of SafeStake website. It will return the remaining DVT but will not withdraw ETH.

Voluntary Exits (Full Withdrawals)

Users can exit their validator by clicking the exit button in the account page of SafeStake website and then upload the keystore file.

Otherwise users need to connect their own beacon node and run commands to do exit, for example, reference the guide of lighthouse https://lighthouse-book.sigmaprime.io/voluntary-exit.html

PreviousSafeStake: Running a Boot NodeNextDistributed Key Generation

Last updated