CoinCashew
Spanish
Spanish
  • Home
  • About Us
  • Donations
  • Contributing
    • Contributor Covenant Code of Conduct
    • Style Guide
  • Disclaimer
  • Wallets
    • Guide: Crypto Wallet Tips 101 - Do's and Don'ts
      • Review: Metal Bitcoin Seed Storage by jlopp
  • Coins
    • Ethereum: ETH
      • 🛡️EthPillar: one-liner setup tool and node management TUI
      • 🥩Guide | How to setup a validator for Ethereum staking on mainnet
        • Overview - Manual Installation
        • PART I - INSTALLATION
          • Step 1: Prerequisites
          • Step 2: Configuring Node
          • Step 3: Installing execution client
            • Nethermind
            • Besu
            • Geth
            • Erigon
            • Reth
          • Step 4: Installing consensus client
            • Lighthouse
            • Lodestar
            • Teku
            • Nimbus
            • Prysm
          • Step 5: Installing Validator
            • Setting up Validator Keys
            • Installing Validator
              • Lighthouse
              • Lodestar
              • Teku
              • Nimbus
              • Prysm
            • Next Steps
          • Monitoring your validator with Grafana and Prometheus
          • Mobile App Node Monitoring by beaconcha.in
          • Monitoring with Uptime Check by Google Cloud
        • PART II - MAINTENANCE
          • Updating Execution Client
          • Updating Consensus Client
          • Backups Checklist: Critical Staking Node Data
          • Uninstalling Staking Node
          • Finding the longest attestation slot gap
          • Checking my eth validator's sync committee duties
          • Checklist | Confirming a healthy functional ETH staking node
        • PART III - TIPS
          • Voluntary Exiting a Validator
          • Verifying Your Mnemonic Phrase
          • Adding a New Validator to an Existing Setup with Existing Seed Words
          • Switching / Migrating Consensus Client
          • 🛡️Switching / Migrating Execution Client
          • ⚡Using Node as RPC URL endpoint
          • Using All Available LVM Disk Space
          • Reducing Network Bandwidth Usage
          • How to re-sync using checkpoint sync
          • Important Directory Locations
          • Improving Validator Attestation Effectiveness
          • EIP2333 Key Generator by iancoleman.io
          • 😁Geth - Enabling path-based state storage
          • Disk Usage by Execution / Consensus Client
          • Dealing with Storage Issues on the Execution Client
        • Join the Community
        • Credits
        • See Also
        • Changelog
      • ⛓️Guide | How to setup a validator for Ethereum staking on testnet HOLESKY
        • Overview - Manual Installation
        • Step 1: Prerequisites
        • Step 2: Configuring Node
        • Step 3: Installing execution client
          • Nethermind
          • Besu
          • Geth
          • Erigon
          • Reth
        • Step 4: Installing consensus client
          • Lighthouse
          • Lodestar
          • Teku
          • Nimbus
          • Prysm
        • Step 5: Installing Validator
          • Setting up Validator Keys
          • Installing Validator
            • Lighthouse
            • Lodestar
            • Teku
            • Nimbus
            • Prysm
          • Next Steps
        • Maintenance
          • Updating Execution Client
          • Updating Consensus Client
          • Backups Checklist: Critical Staking Node Data
          • Uninstalling Staking Node
      • 💰Guide | MEV-boost for Ethereum Staking
        • MEV Relay List
      • 🔎Guide | Recover Ethereum Validator Mnemonic Seed
      • 🦉Update Withdrawal Keys for Ethereum Validator (BLS to Execution Change or 0x00 to 0x01) with ETHDO
      • 📜Archived Guides
        • Guide Version 1 | How to setup a validator for Ethereum staking on MAINNET
          • PART I - INSTALLATION
            • Step 1: Prerequisites
            • Step 2: Configuring Node
            • Step 3: Setting up Validator Keys
            • Step 4: Installing execution client
            • Step 5: Installing consensus client
            • Monitoring your validator with Grafana and Prometheus
            • Mobile App Node Monitoring by beaconcha.in
            • Security Best Practices for your ETH staking validator node
            • Synchronizing time with Chrony
            • Monitoring with Uptime Check by Google Cloud
          • PART II - MAINTENANCE
            • Updating your consensus client
            • Updating your execution client
            • Uninstalling V1 Staking Node
            • Finding the longest attestation slot gap
            • Checking my eth validator's sync committee duties
            • Pruning the execution client to free up disk space
            • Checklist | Confirming a healthy functional ETH staking node
          • PART III - TIPS
            • 🛡️Switching / Migrating Execution Client
            • Voluntary Exiting a Validator
            • Verifying Your Mnemonic Phrase
            • Adding a New Validator to an Existing Setup with Existing Seed Words
            • Switching / Migrating Consensus Client
            • Using All Available LVM Disk Space
            • Reducing Network Bandwidth Usage
            • How to re-sync using checkpoint sync
            • Important Directory Locations
            • Hosting Execution client on a Different Machine
            • Adding or Changing Graffiti flag
            • Improving Validator Attestation Effectiveness
            • EIP2333 Key Generator by iancoleman.io
            • Disk Usage by Execution / Consensus Client
            • Dealing with Storage Issues on the Execution Client
          • Join the Community
          • Credits
          • See Also
          • Changelog
        • Guide Version 1 | How to setup a validator for Ethereum staking on testnet GOERLI
          • Step 1: Prerequisites
          • Step 2: Configuring Node
          • Step 3: Setting up Validator Keys
          • Step 4: Installing execution client
          • Step 5: Installing consensus client
        • Guide Version 2 | How to setup a validator for Ethereum staking on testnet GOERLI
          • Step 1: Prerequisites
          • Step 2: Configuring Node
          • Step 3: Installing execution client
            • Nethermind
            • Besu
            • Geth
            • Erigon
          • Step 4: Installing consensus client
            • Lighthouse
            • Lodestar
            • Teku
            • Nimbus
            • Prysm
          • Step 5: Installing Validator
            • Setting up Validator Keys
            • Installing Validator
              • Lighthouse
              • Lodestar
              • Teku
              • Nimbus
              • Prysm
            • Next Steps
          • Maintenance
            • Updating Execution Client
            • Updating Consensus Client
            • Backups Checklist: Critical Staking Node Data
            • Uninstalling Staking Node
        • Guide | Ethereum Staking on Zhejiang Testnet
        • Guide | Besu + Lodestar | Most Viable Diverse Client | Staking Ethereum on Kiln testnet
        • Guide | How to setup a validator for Ethereum staking on Pithos testnet in 10 minutes or less
        • Ethereum Merge Upgrade Checklist for Home Stakers and Validators
        • Guide | Operation Client Diversity: Migrate Prysm to Teku
      • Guide: How to buy ETH
    • Cardano: ADA
      • Guide: How to Set Up a Cardano Stake Pool
        • PART I - INSTALLATION
          • Prerequisites
          • Hardening an Ubuntu Server
          • Setting Up chrony
          • Installing the Glasgow Haskell Compiler and Cabal
          • Compiling Cardano Node
        • PART II - CONFIGURATION
          • Downloading Configuration Files
          • Configuring Legacy Stake Pool Topology
          • Configuring an Air-gapped, Offline Computer
          • Creating Startup Scripts and Services
        • PART III - OPERATION
          • Starting the Nodes
          • Accessing Built-in Help
          • Generating Keys for the Block-producing Node
          • Setting Up Payment and Stake Keys
          • Registering Your Stake Address
          • Registering Your Stake Pool
          • Verifying Stake Pool Operation
          • Configuring Legacy Network Topology
          • Setting Up Dashboards
          • Configuring Slot Leader Calculations
          • Securing Your Stake Pool Using a Hardware Wallet
        • PART IV - ADMINISTRATION & MAINTENANCE
          • Checking Stake Pool Rewards
          • Claiming Stake Pool Rewards
          • Delegating to a Stake Pool
          • Issuing a New Operational Certificate
          • Updating Stake Pool Information
          • Upgrading a Node
          • Retiring Your Stake Pool
          • Auditing Your nodes configuration
          • KES Key Rotation / Operational Certificate Companion Script
        • PART V - TIPS
          • Submitting a Simple Transaction
          • Transferring Files Using SSH
          • Updating Configuration Files
          • Enabling Peer-to-peer Network Topology
          • Uploading Pool Metadata to GitHub Pages
          • Obtaining a PoolTool API Key
          • Configuring Glasgow Haskell Compiler Runtime System Options
          • Reducing Missed Slot Leader Checks and Improving Cardano Node Performance
          • Increasing Swap File Size
          • Setting Up an External Passive Relay Node
          • Setting Up WireGuard
          • Monitoring Node Security Using OSSEC Server and Slack
          • Resetting an Installation
          • Fixing a Corrupt Blockchain
          • Verifying an ITN Stake Pool
          • Fixing the Mnemonic Staking Balance Bug
        • Appendix A - Best Practices Checklist
        • Appendix B - Cardano Resource Index
        • Telegram Chat Channel
        • See Also
        • Credits
      • Guide: How to buy ADA
      • Guide: How to stake ADA
    • Monero: XMR
      • Guide | How to run your own Monero node
      • Guide: How to mine Monero
      • Create a XMR paper wallet
      • External Reading Material
        • Movie: Monero Means Money
        • Guide: Zero to Monero
        • Book: Mastering Monero
Powered by GitBook
On this page
  • 1. Install Prometheus and Node Exporter
  • 2. Install Grafana
  • 3. Enable services so they start automatically
  • 4. Create the prometheus.yml config file
  • 5. Setup prometheus for your execution client
  • 6. Create a SSH Tunnel to Grafana
  • 7. Setup Grafana Dashboards
  • 8. Setup Alert Notifications
Edit on GitHub
  1. Coins
  2. Ethereum: ETH
  3. Guide | How to setup a validator for Ethereum staking on mainnet
  4. PART I - INSTALLATION

Monitoring your validator with Grafana and Prometheus

PreviousNext StepsNextMobile App Node Monitoring by beaconcha.in

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

Grafana is a dashboard used to visualize the collected data.

1. Install Prometheus and Node Exporter

sudo apt-get install -y prometheus prometheus-node-exporter

2. Install Grafana

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

3. Enable services so they start automatically

sudo systemctl enable grafana-server prometheus prometheus-node-exporter

4. Create the prometheus.yml config file

Remove the default prometheus.yml configuration file and edit a new one.

sudo rm /etc/prometheus/prometheus.yml
sudo nano /etc/prometheus/prometheus.yml

Choose the tab for your consensus client. Paste the following configuration into the file.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'lighthouse'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'lighthouse_validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Nimbus'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'Nimbus_Validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Teku'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'Teku_Validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'validator'
     static_configs:
       - targets: ['localhost:8009']
   - job_name: 'Prysm'
     static_configs:
       - targets: ['localhost:8008']
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'beacon'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'validator'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8009']

5. Setup prometheus for your execution client

Append the applicable job snippet for your execution client to the end of prometheus.yml.

Spacing matters. Ensure all job_name snippets are in alignment.

   - job_name: 'geth'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
       - targets: ['localhost:6060']
   - job_name: 'besu'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /metrics
     scheme: http
     static_configs:
     - targets:
       - localhost:6060
   - job_name: 'nethermind'
     static_configs:
       - targets: ['localhost:6060']
   - job_name: 'erigon'
     scrape_interval: 10s
     scrape_timeout: 3s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
       - targets: ['localhost:6060']
   - job_name: 'reth'
     metrics_path: "/"
     scrape_interval: 10s
     static_configs:
       - targets: ['localhost:6060']

Here's an example of a Lighthouse-Nethermind config:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Lighthouse'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5054']
   - job_name: 'validators'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5064']
   - job_name: 'nethermind'
     static_configs:
       - targets: ['localhost:6060']

To exit and save, press Ctrl + X, then Y, then Enter.

Update file permissions.

sudo chmod 644 /etc/prometheus/prometheus.yml

Restart the services.

sudo systemctl restart grafana-server prometheus prometheus-node-exporter

Verify that the services are running.

sudo systemctl status grafana-server prometheus prometheus-node-exporter

6. Create a SSH Tunnel to Grafana

Each time you want to access Grafana, create a SSH tunnel with port 3000 forwarded.

Example of how to create a SSH tunnel in Linux or MacOS:

ssh -N -v <user>@<staking.node.ip.address> -L 3000:localhost:3000

#Full Example
ssh -N -v ethereum@192.168.1.69 -L 3000:localhost:3000

Navigate to Connection > SSH > Tunnels > Enter Source Port 3000 > Enter Destination localhost:3000 > Click Add

Now save your configuration. Navigate to Session > Enter a session name > Save

Click Open to open a connection

7. Setup Grafana Dashboards

  1. Login with admin / admin

  2. Change password

  3. Click the configuration gear icon, then Add data Source

  4. Select Prometheus

  5. Set Name to "Prometheus"

  6. Click Save & Test

  7. Click Create + icon > Import

  8. Add the consensus client dashboard via Upload JSON file

  9. If needed, select Prometheus as Data Source.

  10. Click the Import button.

  11. Repeat steps 12-15 for the execution client dashboard.

  12. Repeat steps 12-15 for the node-exporter dashboard.

Symptom 1: Your dashboard is missing some data_._

Solution 1: Ensure that the execution or consensus client has enabled the appropriate metrics flag.

cat /etc/systemd/system/execution.service
cat /etc/systemd/system/consensus.service
  • Geth: --http --metrics --pprof

  • Besu: --metrics-enabled=true

  • Nethermind: --Metrics.Enabled true

  • Erigon: --metrics

  • Reth: --metrics 127.0.0.1:9001

  • Lighthouse: --validator-monitor-auto

  • Nimbus: --metrics --metrics-port=8008

  • Teku: --metrics-enabled=true --metrics-port=8008

  • Lodestar: --metrics true

Symptom 2: Don't want to use SSH tunnels and you want to expose port 3000 to access Grafana, but understand the security concerns.

Solution 2: Open port 3000 in your ufw firewall. Access grafana at http://<node ipaddress>:3000

sudo ufw allow 3000

Example of Grafana Dashboards for each consensus client.

JSON download link:

Example of Grafana Dashboards for each execution client.

Example of Node-Exporter Dashboard

General system monitoring

Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。

8. Setup Alert Notifications

Setup alerts to get notified if your validators go offline.

Get notified of problems with your validators. Choose between email, telegram, discord or slack.

Option 1: Email Notifications
  1. Sign up for an account

  2. Verify your email

  3. Search for your validator's public address

  4. Add validators to your watchlist by clicking the bookmark symbol.

Option 2: Telegram Notifications
  1. On the menu of Grafana, select Notification channels under the bell icon.

  2. Click on Add channel.

  3. Give the notification channel a name.

  4. Select Telegram from the Type list.

  5. Create a new telegram group.

  6. Invite the bot to your new group.

  7. Type at least 1 message into the group to initialize it.

  8. In the JSON response, find and copy the Chat ID. Find it between **chat **and title. Example of Chat ID: -1123123123

    "chat":{"id":-123123123,"title":
  9. Paste the Chat ID into the corresponding field in Grafana.

  10. Save and test the notification channel for your alerts.

Option 3: Discord Notifications
  1. On the menu of Grafana, select Notification channels under the bell icon.

  2. Click on Add channel.

  3. Add a name to the notification channel.

  4. Select Discord from the Type list.

  5. Enter the Webhook URL in the Discord notification settings panel.

  6. Click Send Test, which will push a confirmation message to the Discord channel.

Option 4: Slack Notifications
  1. On the menu of Grafana, select Notification channels under the bell icon.

  2. Click on Add channel.

  3. Add a name to the notification channel.

  4. Select Slack from the Type list.

  5. Enter the Slack Incoming Webhook URL in the URL field.

  6. Click Send Test, which will push a confirmation message to the Slack channel.

Example of how to create a SSH tunnel in Windows with :

Now you can access Grafana on your local machine by pointing a web browser to

Open

Set URL to

Download and save your consensus client's json file. More json dashboard options available below. [ | | | | | ]

Download and save your execution client's json file [ | | | | ]

Download and save a for general system monitoring

Troubleshooting common Grafana issues

Beacon Chain JSON Download link:

Validator Client JSON download link:

Credits:

JSON Download link:

Credits:

Credits:

Credits:

Credits:

Credits:

JSON download link:

Credits:

Credits:

Credits:

Blob enabled dashboard:

Credits:

Credits:

Credits:

Credits:

Credits:

Visit

To complete the Telegram API settings, a Telegram channel and **bot **are required. For instructions on setting up a bot with @Botfather, see of the Telegram documentation. You need to create a BOT API token.

Visit where XXX:YYY is your BOT API Token.

Now you can create custom alerts from your dashboards.

To complete the set up, a Discord server (and a text channel available) as well as a Webhook URL are required. For instructions on setting up a Discord's Webhooks, see of their documentation.

For instructions on setting up a Slack's Incoming Webhooks, see of their documentation.

🥩
🔥
📖
Official documentation is available here.
Putty
http://localhost:3000
http://localhost:3000
http://localhost:9090
Lighthouse
Teku
Nimbus
Prysm
Prysm > 10 Validators
Lodestar
Geth
Besu
Nethermind
Erigon
Reth
node-exporter dashboard
https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/Summary.json
https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/ValidatorClient.json
https://github.com/sigp/lighthouse-metrics/
https://raw.githubusercontent.com/Yoldark34/lighthouse-staking-dashboard/main/Yoldark_ETH_staking_dashboard.json
https://github.com/Yoldark34/lighthouse-staking-dashboard
https://github.com/status-im/nimbus-eth2/
https://github.com/metanull-operator/eth2-grafana/
https://grafana.com/grafana/dashboards/13457
https://github.com/GuillaumeMiralles/prysm-grafana-dashboard
https://github.com/metanull-operator/eth2-grafana/raw/master/eth2-grafana-dashboard-single-source.json
https://github.com/metanull-operator/eth2-grafana/
https://raw.githubusercontent.com/ChainSafe/lodestar/unstable/dashboards/lodestar_summary.json
https://gist.github.com/karalabe/e7ca79abdec54755ceae09c08bd090cd
https://github.com/ethereum/go-ethereum/files/14211070/Geth-Cancun-Prometheus.json
https://grafana.com/dashboards/10273
https://github.com/NethermindEth/metrics-infrastructure
https://github.com/ledgerwatch/erigon/tree/devel/cmd/prometheus/dashboards
https://github.com/paradigmxyz/reth/blob/main/etc/grafana/dashboards/overview.json
starsliao
https://beaconcha.in/
this section
https://api.telegram.org/botXXX:YYY/getUpdates
Visit here to learn how to create alerts.
this section
this section
Beacon Chain dashboard by sigp
Validator Client dashboard by sigp
LH dashboard by Yoldark
Dashboard by status-im
Nimbus dashboard by metanull-operator
Teku by PegaSys Engineering
Prysm dashboard by GuillaumeMiralles
Prysm dashboard by metanull-operator
Dashboard by karalabe