English
🔮
16. Setup Prometheus and Grafana Dashboard
  • Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Official documentation is available here.
  • Grafana is a dashboard used to visualize the collected data.

🐣
16.1 Installation

Install prometheus and prometheus node exporter.
relaynode1
block producer node
1
sudo apt-get install -y prometheus prometheus-node-exporter
Copied!
1
sudo apt-get install -y prometheus-node-exporter
Copied!
Install grafana.
relaynode1
1
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
Copied!
relaynode1
1
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
2
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
Copied!
relaynode1
1
sudo apt-get update && sudo apt-get install -y grafana
Copied!
Enable services so they start automatically.
relaynode1
block producer node
1
sudo systemctl enable grafana-server.service
2
sudo systemctl enable prometheus.service
3
sudo systemctl enable prometheus-node-exporter.service
Copied!
1
sudo systemctl enable prometheus-node-exporter.service
Copied!
Update prometheus.yml located in /etc/prometheus/prometheus.yml
Change the <block producer ip address> in the following command.
relaynode1
1
cat > prometheus.yml << EOF
2
global:
3
scrape_interval: 15s # By default, scrape targets every 15 seconds.
4
5
# Attach these labels to any time series or alerts when communicating with
6
# external systems (federation, remote storage, Alertmanager).
7
external_labels:
8
monitor: 'codelab-monitor'
9
10
# A scrape configuration containing exactly one endpoint to scrape:
11
# Here it's Prometheus itself.
12
scrape_configs:
13
# The job name is added as a label job=<job_name> to any timeseries scraped from this config.
14
- job_name: 'prometheus'
15
16
static_configs:
17
- targets: ['localhost:9100']
18
- targets: ['<block producer ip address>:9100']
19
- targets: ['<block producer ip address>:12798']
20
labels:
21
alias: 'block-producer-node'
22
type: 'cardano-node'
23
- targets: ['localhost:12798']
24
labels:
25
alias: 'relaynode1'
26
type: 'cardano-node'
27
EOF
28
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
Copied!
Finally, restart the services.
relaynode1
1
sudo systemctl restart grafana-server.service
2
sudo systemctl restart prometheus.service
3
sudo systemctl restart prometheus-node-exporter.service
Copied!
Verify that the services are running properly:
relaynode1
1
sudo systemctl status grafana-server.service prometheus.service prometheus-node-exporter.service
Copied!
Update ${NODE_CONFIG}-config.json config files with new hasEKG and hasPrometheus ports.
block producer node
relaynodeN
1
cd $NODE_HOME
2
3
sed -i ${NODE_CONFIG}-config.json -e "s/127.0.0.1/0.0.0.0/g"
Copied!
1
cd $NODE_HOME
2
3
sed -i ${NODE_CONFIG}-config.json -e "s/127.0.0.1/0.0.0.0/g"
Copied!
Port forwarding and firewall config:
On block producer node (or relaynodeN), you need to open ports 12798 and 9100
On relaynode1, you will need to open ports 3000 for grafana.
Stop and restart your stake pool.
block producer node
relaynode1
1
sudo systemctl restart cardano-node
Copied!
1
sudo systemctl restart cardano-node
Copied!
Verify the metrics are working by querying the prometheus port.
block producer and relaynodes
1
curl -s 127.0.0.1:12798/metrics
2
3
# Example output:
4
# rts_gc_par_tot_bytes_copied 123
5
# rts_gc_num_gcs 345
6
# rts_gc_max_bytes_slop 4711111
7
# cardano_node_metrics_served_block_count_int 8112
8
# cardano_node_metrics_Stat_threads_int 8
9
# cardano_node_metrics_density_real 4.67
Copied!

📶
16.2 Setting up Grafana Dashboards

  1. 1.
    On relaynode1, open http://localhost:3000 or http://<your relaynode1 ip address>:3000 in your local browser. You may need to open up port 3000 in your router and/or firewall.
  2. 2.
    Login with admin / admin
  3. 3.
    Change password
  4. 4.
    Click the configuration gear icon, then Add data Source
  5. 5.
    Select Prometheus
  6. 6.
    Set Name to "Prometheus"
  7. 7.
    Set URL to http://localhost:9090
  8. 8.
    Click Save & Test
  9. 9.
    Download and save this json file.
  10. 10.
    Click Create + icon > Import
  11. 11.
    Add dashboard by Upload JSON file
  12. 12.
    Click the Import button.
Credits to KAZE stake pool for this dashboard
Community contributer Sansky.de is sharing a very detailed Grafana tutorial
🙏
https://sanskys.de/dashboard/
Congratulations. You're basically done. More great operational and maintenance tips in section 18.
Be sure to review the stake pool operator's best practice checklist to ensure a smooth sailing with your pool.
Last modified 2mo ago