Monitoring your validator with Grafana and Prometheus

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Official documentation is available here. Grafana is a dashboard used to visualize the collected data.

6.1 Installation

Install prometheus and prometheus node exporter.

sudo apt-get install -y prometheus prometheus-node-exporter

Install grafana.

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

Enable services so they start automatically.

sudo systemctl enable grafana-server.service prometheus.service prometheus-node-exporter.service

Create the prometheus.yml config file. Choose the tab for your eth client. Simply copy and paste.

cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'nodes'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:5054']
   - job_name: 'validators'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5064']
EOF

Setup prometheus for your execution client. Start by editing prometheus.yml

nano $HOME/prometheus.yml

Append the applicable job snippet for your execution client to the end of prometheus.yml. Save the file.

Spacing matters. Ensure all job_name snippets are in alignment.

   - job_name: 'geth'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
     - targets: ['localhost:6060']

Move it to /etc/prometheus/prometheus.yml

sudo mv $HOME/prometheus.yml /etc/prometheus/prometheus.yml

Update file permissions.

sudo chmod 644 /etc/prometheus/prometheus.yml

Finally, restart the services.

sudo systemctl restart grafana-server.service prometheus.service prometheus-node-exporter.service

Verify that the services are running properly:

sudo systemctl status grafana-server.service prometheus.service prometheus-node-exporter.service
πŸ”₯ Grafana Security: SSH Tunnels

Do not expose Grafana (port 3000) to the public internet as this invites a new attack surface! A secure solution would be to access Grafana through a ssh tunnel.

Example of how to create a ssh tunnel in Linux or MacOS:

ssh -N -v <user>@<staking.node.ip.address> -L 3000:localhost:3000

Example of how to create a ssh tunnel in Windows with Putty:

Navigate to Connection > SSH > Tunnels > Enter Source Port 3000 > Enter Destination localhost:3000 > Click Add

Now you can access Grafana on your local machine by pointing a web browser to http://localhost:3000

πŸ“Ά 6.2 Setting up Grafana Dashboards

  1. Open http://localhost:3000 or http://<your validator's ip address>:3000 in your web browser.

  2. Login with admin / admin

  3. Change password

  4. Click the configuration gear icon, then Add data Source

  5. Select Prometheus

  6. Set Name to "Prometheus"

  7. Click Save & Test

  8. Download and save your consensus client's json file. More json dashboard options available below. [ Lighthouse | Teku | Nimbus | Prysm | Prysm > 10 Validators | Lodestar ]

  9. Download and save your execution client's json file [ Geth | Besu | Nethermind | Erigon ]

  10. Download and save a node-exporter dashboard for general system monitoring

  11. Click Create + icon > Import

  12. Add the consensus client dashboard via Upload JSON file

  13. If needed, select Prometheus as Data Source.

  14. Click the Import button.

  15. Repeat steps 12-15 for the execution client dashboard.

  16. Repeat steps 12-15 for the node-exporter dashboard.

πŸ”₯ Troubleshooting common Grafana issues

Symptom: Your dashboard is missing some data.

Solution: Ensure that the execution or consensus client has enabled the appropriate metrics flag.

  • Geth: geth --http --metrics --pprof

  • Besu: besu --metrics-enabled=true

  • Nethermind: Nethermind.Runner --Metrics.Enabled true

  • Erigon: erigon --metrics

  • Lighthouse beacon-node: lighthouse bn --validator-monitor-auto

  • Nimbus: nimbus_beacon_node --metrics --metrics-port=8008

  • Teku: --metrics-enabled=true --metrics-port=8008

  • Lodestar beacon-node: lodestar beacon --metrics true

Example of Grafana Dashboards for each consensus client.

Example of Grafana Dashboards for each execution client.

Example of Node-Exporter Dashboard

General system monitoring

Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。

Credits: starsliao

⚠️ 6.3 Setup Alert Notifications

Setup alerts to get notified if your validators go offline.

Get notified of problems with your validators. Choose between email, telegram, discord or slack.

  1. Sign up for an account.

  2. Verify your email

  3. Search for your validator's public address

  4. Add validators to your watchlist by clicking the bookmark symbol.

Last updated