Edit

Monitoring your validator with Grafana and Prometheus

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Official documentation is available here. Grafana is a dashboard used to visualize the collected data.

6.1 Installation

Install prometheus and prometheus node exporter.

sudo apt-get install -y prometheus prometheus-node-exporter

Install grafana.

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

Enable services so they start automatically.

sudo systemctl enable grafana-server.service prometheus.service prometheus-node-exporter.service

Create the prometheus.yml config file. Choose the tab for your eth client. Simply copy and paste.

cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'nodes'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:5054']
   - job_name: 'validators'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5064']
EOF

Setup prometheus for your execution client. Start by editing prometheus.yml

nano $HOME/prometheus.yml

Append the applicable job snippet for your execution client to the end of prometheus.yml. Save the file.

Spacing matters. Ensure all job_name snippets are in alignment.

   - job_name: 'geth'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
     - targets: ['localhost:6060']

Move it to /etc/prometheus/prometheus.yml

sudo mv $HOME/prometheus.yml /etc/prometheus/prometheus.yml

Update file permissions.

sudo chmod 644 /etc/prometheus/prometheus.yml

Finally, restart the services.

sudo systemctl restart grafana-server.service prometheus.service prometheus-node-exporter.service

Verify that the services are running properly:

sudo systemctl status grafana-server.service prometheus.service prometheus-node-exporter.service
🔥 Grafana Security: SSH Tunnels

Do not expose Grafana (port 3000) to the public internet as this invites a new attack surface! A secure solution would be to access Grafana through a ssh tunnel.

Example of how to create a ssh tunnel in Linux or MacOS:

ssh -N -v <user>@<staking.node.ip.address> -L 3000:localhost:3000

Example of how to create a ssh tunnel in Windows with Putty:

Navigate to Connection > SSH > Tunnels > Enter Source Port 3000 > Enter Destination localhost:3000 > Click Add

Now you can access Grafana on your local machine by pointing a web browser to http://localhost:3000

📶 6.2 Setting up Grafana Dashboards

  1. Open http://localhost:3000 or http://<your validator's ip address>:3000 in your web browser.

  2. Login with admin / admin

  3. Change password

  4. Click the configuration gear icon, then Add data Source

  5. Select Prometheus

  6. Set Name to "Prometheus"

  7. Set URL to http://localhost:9090

  8. Click Save & Test

  9. Download and save your consensus client's json file. More json dashboard options available below. [ Lighthouse | Teku | Nimbus | Prysm | Prysm > 10 Validators | Lodestar ]

  10. Download and save your execution client's json file [ Geth | Besu | Nethermind | Erigon ]

  11. Download and save a node-exporter dashboard for general system monitoring

  12. Click Create + icon > Import

  13. Add the consensus client dashboard via Upload JSON file

  14. If needed, select Prometheus as Data Source.

  15. Click the Import button.

  16. Repeat steps 12-15 for the execution client dashboard.

  17. Repeat steps 12-15 for the node-exporter dashboard.

🔥 Troubleshooting common Grafana issues

Symptom: Your dashboard is missing some data.

Solution: Ensure that the execution or consensus client has enabled the appropriate metrics flag.

  • Geth: geth --http --metrics --pprof

  • Besu: besu --metrics-enabled=true

  • Nethermind: Nethermind.Runner --Metrics.Enabled true

  • Erigon: erigon --metrics

  • Lighthouse beacon-node: lighthouse bn --validator-monitor-auto

  • Nimbus: nimbus_beacon_node --metrics --metrics-port=8008

  • Teku: --metrics-enabled=true --metrics-port=8008

  • Lodestar beacon-node: lodestar beacon --metrics true

Example of Grafana Dashboards for each consensus client.

Beacon Chain dashboard by sigp
Validator Client dashboard by sigp

Beacon Chain JSON Download link: https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/Summary.json

Validator Client JSON download link: https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/ValidatorClient.json

Credits: https://github.com/sigp/lighthouse-metrics/

LH dashboard by Yoldark

JSON Download link: https://raw.githubusercontent.com/Yoldark34/lighthouse-staking-dashboard/main/Yoldark_ETH_staking_dashboard.json

Credits: https://github.com/Yoldark34/lighthouse-staking-dashboard

Example of Grafana Dashboards for each execution client.

Dashboard by karalabe

Credits: https://gist.github.com/karalabe/e7ca79abdec54755ceae09c08bd090cd

Example of Node-Exporter Dashboard

General system monitoring

Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。

Credits: starsliao

⚠️ 6.3 Setup Alert Notifications

Setup alerts to get notified if your validators go offline.

Get notified of problems with your validators. Choose between email, telegram, discord or slack.

  1. Visit https://beaconcha.in/

  2. Sign up for an account.

  3. Verify your email

  4. Search for your validator's public address

  5. Add validators to your watchlist by clicking the bookmark symbol.

PreviousStep 5: Installing consensus clientNextMobile App Node Monitoring by beaconcha.in

Last updated