# Monitoring ## Monitoring, Metrics, Grafana dashboards ### Quickstart Here you'll find a quick start guide to run the Prometheus, Grafana, and Node exporter stack. Check out the README [here](https://github.com/Eoracle/Eoracle-operator-setup/blob/main/data-validator/monitoring/README.md) for more details. If you want to manually set this up, follow the steps below. ### Metrics {% stepper %} {% step %} ### Check the data validator metrics Replace the `EO_PROMETHEUS_PORT` with the value of `EO_PROMETHEUS_PORT` from the `data-validator/.env` ```bash curl http://localhost:/metrics ``` You should see something like: ``` # HELP eigen_performance_score The performance metric is a score between 0 and 100 and each developer can define their own way of calculating the score. The score is calculated based on the performance of the Node and the performance of the backing services. # TYPE eigen_performance_score gauge eigen_performance_score{avs_name=""} 100 ... ``` {% endstep %} {% step %} ### Setup the monitoring stack We use [prometheus](https://prometheus.io/download) to scrape the metrics from the EO data validator container. Make sure to edit the prometheus.yml file, located at `operator-setup/data-validator/monitoring`, replacing the placeholders 'EO\_PROMETHEUS\_PORT', `EO_MAIN_ADDRESS`, and `mainnet|testnet` with your specific values ```bash cd operator-setup/data-validator/monitoring ``` The relevant lines are: ```yml - job_name: 'eo-data-validator' static_configs: - targets: ['eo-data-validator:'] ``` {% endstep %} {% step %} ### Setup sending data validator metrics to EO monitoring We allow operators to push the data validator metrics to EO monitoring system for extra monitoring. To do so, make sure to edit the vmagent.yml file, located at `operator-setup/data-validator/monitoring`, replacing the placeholders `EO_PROMETHEUS_PORT`, `EO_MAIN_ADDRESS`, and `mainnet|testnet` with your specific values ``` cd operator-setup/data-validator/monitoring ``` The relevant lines are: ```yaml - job_name: 'eo-data-validator' static_configs: - targets: ['eo-data-validator:'] metric_relabel_configs: - target_label: operator_address replacement: - target_label: eochain replacement: ``` {% endstep %} {% step %} ### Start the monitoring stack You can start all the monitoring stack, Prometheus, Grafana, and Node exporter all at once or only specific component ```bash docker compose up -d ``` {% endstep %} {% step %} ### Connect docker networks Since the EO data validator is running in a different docker network, we will need to have the Prometheus container in the same network of blockchain oracle-data-validator. To do that, run the following command. ```bash docker network connect eo-data-validator prometheus ``` {% endstep %} {% endstepper %} ### Troubleshooting * If you see the following error: ``` permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied ``` Use the same command by prepending `sudo` in front of it. ### Grafana We use Grafana to visualize the metrics from the EO AVS. You can use [OSS Grafana](https://grafana.com/oss/grafana/) for it or any other Dashboard provider. You should be able to navigate to `http://:3000` and log in with `admin`/`admin`. This container of Grafana has a Prometheus data source setup using port 9090. If you change the Prometheus port, you need to add a new data source or update the existing data source. You can do this by navigating to `http://:3000/datasources` **Useful Dashboards** The EO Data Validator dashboard can be used to monitor performance, issues and data source statuses. Explaining each panel on below -

You can import the data validator dashboard by importing data-validators.json file present in the monitoring directory.

**Score** The score panel shows the gauge metric `eigen_performance_score`between 0-100 which is calculated based on the performance of the AVS operator and the performance of the backing services. **RPC Req** The RPC Req panel shows the counter and histogram of the total number of json-rpc ``\ requests from the execution client. **EO Errors &** EO **Errors Avg 5 min** The EO Errors panel shows the counter for the number of errors encountered by the execution client. **Update Rate Duration (s)** The Update Rate duration panel helps visualize the frequency of updates in seconds. For example, in the above chart 91% of the submitted transactions were processed within 0.1 seconds. **EO Chain Performance (s)** The EO chain panel helps visualize the time between blocks on the EO chain. As per the above dashboard, the majority of the blocks are produced within 0.005 seconds. **Data Providers All** The Data providers panel shows the connection status of each data source that the validator pings for price feeds. If one of the sources shows 'FAIL' instead of 'OK', it means the connection to that source is broken. You can find the json file to import the above dashboard [here](https://github.com/Eoracle/Eoracle-operator-setup/tree/main/data-validator/dashboards). Once you have Grafana set up, feel free to import the dashboards. ### Node exporter The EO data validator emits EO specific metrics. However, it's also important to keep track of the node's health. For this, we will use [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) which is a Prometheus exporter for hardware and OS metrics exposed by \*NIX kernels, written in Go with pluggable metric collectors. By default, it is installed and started when you start the entire monitoring stack. If you want to modify the stack, you can install the binary or use docker to [run](https://hub.docker.com/r/prom/node-exporter) it. In Grafana dashboards screen, import the node-exporter to see host metrics. ### VMAgent The vmagent is the docker that submits data validator metrics to EO central monitoring. This allow us to help you in troubleshoot your operator. If you don't want to share with us the metrics, remove the vmagent from the `docker-compose.yml` in the `data-validator/monitoring` folder