Grafana - Exploring Log Data

[0] Preparation

Continue from previous tutorial, https://mohamadrazzimy.hashnode.dev/grafana-news-tutorial

[1] Add Log Data source

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

In the sidebar, hover over the Configuration (gear) icon, and then click Data Sources.

Click Add new data source.

In the list of data sources, search for loki and click the search result for Loki.

In the URL box, enter http://loki:3100.

Click Save & Test to save your changes.

[2] Explore your logs

Grafana Explore not only lets you make ad-hoc queries for metrics, but lets you explore your logs as well.

In the sidebar, click the Explore (compass) icon.

In the data source list at the top, select the Loki data source.

In the Query editor, enter:

{filename="/var/log/tns-app.log"}

Grafana displays all logs within the log file of the sample application. The height of each bar in the graph encodes the number of logs that were generated at that time.

Click and drag across the bars in the graph to filter logs based on time.

Not only does Loki let you filter logs based on labels, but on specific occurrences.

Let’s generate an error, and analyze it with Explore.

In the sample application, post a new link without a URL to generate an error in your browser that says empty url.

Go back to Grafana and enter the following query to filter log lines based on a substring:

{filename="/var/log/tns-app.log"} |= "error"

Click on the log line that says level=error msg="empty url" to see more information about the error.

Note: If you’re in Live mode, clicking logs will not show more information about the error. Instead, stop and exit the live stream, then click the log line there.

Logs are helpful for understanding what went wrong. Later in this tutorial, you’ll see how you can correlate logs with metrics from Prometheus to better understand the context of the error.