How to query Azure Monitor for logs based upon Kubernetes Pod labels

Kubernetes makes it easy to spin up containers at will. In addition, it is relatively easy to query Kubernetes itself (using kubectl) to find out information about specific pods.

However, it is a little more challenging to query Azure Monitor to get similar information.

If we look at the KubePodInventory table in Log Analytics, we can see the data we want is in the PodLabels column. But we need to parse the data to get the actual label values.

We could use the contains keyword in our Kusto query, but that could be expensive and just does string matching. What we really want is to query the JSON representation of the labels attached to the pod in a similar fashion to kubectl.

Luckily, Kusto has 2 features we can combine to make this work.

extend

The extend operator lets us create a “virtual” column with the data from the PodLabels column.

parse_json()

We can use the parse_json() function to interpret the JSON label string that is in the PodLabels column as JSON. This enables us to index into the data and retrieve specific fields (such as the app label).

Kusto Query

We can use the following query to get the pod labels as fields we can filter on. We create a “virtual” column with the PodLabel data, then index into it and filter on the app label.

KubePodInventory
| where Namespace == 'default'
| extend podLabels = parse_json(PodLabel)
| where podLabels[0].app == 'azure-vote-front'

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *