Monitoring your validator with Grafana and Prometheus

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Official documentation is available here. Grafana is a dashboard used to visualize the collected data.
6.1 Installation
Install prometheus and prometheus node exporter.
sudo apt-get install -y prometheus prometheus-node-exporter
Install grafana.
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana
Enable services so they start automatically.
sudo systemctl enable grafana-server.service prometheus.service prometheus-node-exporter.service
Create the prometheus.yml config file. Choose the tab for your eth client. Simply copy and paste.
Lighthouse
Nimbus
Teku
Prysm
Lodestar
cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
​
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'
​
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'nodes'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:5054']
   - job_name: 'validators'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5064']
EOF
cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
​
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'
​
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'nodes'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
EOF
cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
​
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'
​
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'nodes'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
EOF
cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
​
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'
​
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'validator'
     static_configs:
       - targets: ['localhost:8081']
   - job_name: 'beacon node'
     static_configs:
       - targets: ['localhost:8080']
   - job_name: 'slasher'
     static_configs:
       - targets: ['localhost:8082']
EOF
cat > $HOME/prometheus.yml << EOF   
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Lodestar'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
EOF
Setup prometheus for your execution client. Start by editing prometheus.yml
nano $HOME/prometheus.yml
Append the applicable job snippet for your execution client to the end of prometheus.yml. Save the file.
Spacing matters. Ensure all job_name snippets are in alignment.
Geth
Besu
Nethermind
Erigon
   - job_name: 'geth'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
     - targets: ['localhost:6060']
   - job_name: 'besu'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /metrics
     scheme: http
     static_configs:
     - targets:
       - localhost:9545
   - job_name: 'nethermind'
     static_configs:
       - targets: ['localhost:6060']
   - job_name: 'erigon'
     scrape_interval: 10s
     scrape_timeout: 3s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
       - targets: ['localhost:6060']
Move it to /etc/prometheus/prometheus.yml
sudo mv $HOME/prometheus.yml /etc/prometheus/prometheus.yml
Update file permissions.
sudo chmod 644 /etc/prometheus/prometheus.yml
Finally, restart the services.
sudo systemctl restart grafana-server.service prometheus.service prometheus-node-exporter.service
Verify that the services are running properly:
sudo systemctl status grafana-server.service prometheus.service prometheus-node-exporter.service
​🔥
 Grafana Security: SSH Tunnels
Do not expose Grafana (port 3000) to the public internet as this invites a new attack surface! A secure solution would be to access Grafana through a ssh tunnel.
Example of how to create a ssh tunnel in Linux or MacOS:
ssh -N -v <user>@<staking.node.ip.address> -L 3000:localhost:3000
Example of how to create a ssh tunnel in Windows with Putty:
Navigate to Connection > SSH > Tunnels > Enter Source Port 3000 > Enter Destination localhost:3000 > Click Add
​
​
Now you can access Grafana on your local machine by pointing a web browser to http://localhost:3000​
​📶
 6.2 Setting up Grafana Dashboards
1.Open http://localhost:3000 or http://<your validator's ip address>:3000 in your web browser.
2.Login with admin / admin
3.Change password
4.Click the configuration gear icon, then Add data Source
5.Select Prometheus
6.Set Name to "Prometheus"
7.Set URL to http://localhost:9090​
8.Click Save & Test
9.Download and save your consensus client's json file. More json dashboard options available below. [ Lighthouse | Teku | Nimbus | Prysm | Prysm > 10 Validators | Lodestar ]
10.Download and save your execution client's json file [ Geth | Besu | Nethermind | Erigon ]
11.Download and save a node-exporter dashboard for general system monitoring
12.Click Create + icon > Import
13.Add the consensus client dashboard via Upload JSON file
14.If needed, select Prometheus as Data Source.
15.Click the Import button.
16.Repeat steps 12-15 for the execution client dashboard.
17.Repeat steps 12-15 for the node-exporter dashboard.
​🔥
 Troubleshooting common Grafana issues
Symptom: Your dashboard is missing some data.
Solution: Ensure that the execution or consensus client has enabled the appropriate metrics flag.
Geth: geth --http --metrics --pprof
Besu: besu  --metrics-enabled=true
Nethermind: Nethermind.Runner  --Metrics.Enabled true
Erigon: erigon  --metrics
Lighthouse beacon-node: lighthouse bn  --validator-monitor-auto
Nimbus: nimbus_beacon_node  --metrics  --metrics-port=8008
Teku: --metrics-enabled=true --metrics-port=8008
Lodestar beacon-node: lodestar beacon --metrics true
Example of Grafana Dashboards for each consensus client.
Lighthouse
Nimbus
Teku
Prysm
Lodestar
Beacon Chain dashboard by sigp
Validator Client dashboard by sigp
Beacon Chain JSON Download link: https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/Summary.json​
Validator Client JSON download link: https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/ValidatorClient.json​
Credits: https://github.com/sigp/lighthouse-metrics/​
LH dashboard by Yoldark
JSON Download link: https://raw.githubusercontent.com/Yoldark34/lighthouse-staking-dashboard/main/Yoldark_ETH_staking_dashboard.json​
Credits: https://github.com/Yoldark34/lighthouse-staking-dashboard​
Dashboard by status-im
Credits: https://github.com/status-im/nimbus-eth2/​
Nimbus dashboard by metanull-operator
JSON download link:
Credits: https://github.com/metanull-operator/eth2-grafana/​
Teku by PegaSys Engineering
Credits: https://grafana.com/grafana/dashboards/13457​
Prysm dashboard by GuillaumeMiralles
Credits: https://github.com/GuillaumeMiralles/prysm-grafana-dashboard​
Prysm dashboard by metanull-operator
JSON download link: https://github.com/metanull-operator/eth2-grafana/raw/master/eth2-grafana-dashboard-single-source.json​
Credits: https://github.com/metanull-operator/eth2-grafana/​
Work in progress.
Example of Grafana Dashboards for each execution client.
Geth
Besu
Nethermind
Erigon
Dashboard by karalabe
Credits: https://gist.github.com/karalabe/e7ca79abdec54755ceae09c08bd090cd​
Credits: https://grafana.com/dashboards/10273​
Credits: https://github.com/NethermindEth/metrics-infrastructure​
Credits: https://github.com/ledgerwatch/erigon/tree/devel/cmd/prometheus/dashboards​
Example of Node-Exporter Dashboard
Node-Exporter Dashboard by starsliao
General system monitoring
Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。
Credits: starsliao​
​⚠
 6.3 Setup Alert Notifications
Setup alerts to get notified if your validators go offline.
Get notified of problems with your validators. Choose between email, telegram, discord or slack.
Email Notifications
Telegram Notifications
Discord Notifications
Slack Notifications
1.Visit https://beaconcha.in/​
2.Sign up for an account.
3.Verify your email
4.Search for your validator's public address
5.Add validators to your watchlist by clicking the bookmark symbol.
1.On the menu of Grafana, select Notification channels under the bell icon.
2.Click on Add channel.
3.Give the notification channel a name.
4.Select Telegram from the Type list.
5.To complete the Telegram API settings, a Telegram channel and **bot **are required. For instructions on setting up a bot with @Botfather, see this section of the Telegram documentation. You need to create a BOT API token.
6.Create a new telegram group.
7.Invite the bot to your new group.
8.Type at least 1 message into the group to initialize it.
9.Visit https://api.telegram.org/botXXX:YYY/getUpdates where XXX:YYY is your BOT API Token.
10.In the JSON response, find and copy the Chat ID. Find it between **chat **and title. Example of Chat ID: -1123123123
"chat":{"id":-123123123,"title":
11.Paste the Chat ID into the corresponding field in Grafana.
12.Save and test the notification channel for your alerts.
13.Now you can create custom alerts from your dashboards. Visit here to learn how to create alerts.​
1.On the menu of Grafana, select Notification channels under the bell icon.
2.Click on Add channel.
3.Add a name to the notification channel.
4.Select Discord from the Type list.
5.To complete the set up, a Discord server (and a text channel available) as well as a Webhook URL are required. For instructions on setting up a Discord's Webhooks, see this section of their documentation.
6.Enter the Webhook URL in the Discord notification settings panel.
7.Click Send Test, which will push a confirmation message to the Discord channel.
1.On the menu of Grafana, select Notification channels under the bell icon.
2.Click on Add channel.
3.Add a name to the notification channel.
4.Select Slack from the Type list.
5.For instructions on setting up a Slack's Incoming Webhooks, see this section of their documentation.
6.Enter the Slack Incoming Webhook URL in the URL field.
7.Click Send Test, which will push a confirmation message to the Slack channel.