docs: Add topic for editing default OTEL labels (#17958)

Signed-off-by: J Stickler <julie.stickler@grafana.com>
Co-authored-by: Ed Welch <edward.welch@grafana.com>

docs/sources/get-started/labels/_index.md

@@ -99,6 +99,11 @@ For Grafana Cloud Logs, see the [current OpenTelemetry guidance](https://grafana
 
 The default list of resource attributes to store as labels can be configured using `default_resource_attributes_as_index_labels` under the [distributor's otlp_config](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#distributor). You can set global limits using [limits_config.otlp_config](/docs/loki/<LOKI_VERSION>/configure/#limits_config). If you are using Grafana Cloud, contact support to configure this setting.
 
+{{< admonition type="caution" >}}
+Because of the potential for high [cardinality](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/labels/cardinality/), `k8s.pod.name` and `service.instance.id` are no longer recommended as default labels. But because removing these resource attributes from the default labels would be a breaking change for existing users, they have not yet been deprecated as default labels. If you are a new user of Grafana Loki, we recommend that you modify your Alloy or OpenTelemetry Collector configuration to convert `k8s.pod.name` and `service.instance.id` from index labels to structured metadata.
+For sample configurations, refer to [Remove default labels](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/labels/remove-default-labels).
+{{< /admonition >}}
+
 ## Labeling is iterative
 
 You want to start with a small set of labels. While accepting the default labels assigned by Grafana Alloy or the OpenTelemetry Collector or the Kubernetes Monitoring Helm chart may meet your needs, over time you may find that you need to modify your labeling strategy.

docs/sources/get-started/labels/bp-labels.md

@@ -4,7 +4,7 @@ menuTitle:  Label best practices
 description: Describes best practices for using labels in Grafana Loki.
 aliases:
 - ../../best-practices/ # /docs/loki/<LOKI_VERSION>/best-practices/
-weight: 700
+weight: 100
 ---
 # Label best practices
 

docs/sources/get-started/labels/cardinality.md

@@ -1,7 +1,7 @@
 ---
 title: Cardinality
 description:  Describes what cardinality is and how it affects Loki performance.
-weight: 
+weight: 200
 ---
 
 # Cardinality

docs/sources/get-started/labels/remove-default-labels.md

@@ -0,0 +1,165 @@
+---
+title: Remove default OpenTelemetry labels
+menuTitle: Remove default labels
+description: Describes how to modify your Alloy or OpenTelemetry Collector configuration to demote default index labels to structured metadata.
+weight: 400
+---
+
+# Remove default OpenTelemetry labels
+
+When Grafana Loki first started supporting OpenTelemetry, we selected a set of [default resource attributes to promote to labels](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/labels/#default-labels-for-opentelemetry). We included some labels such as k8s.pod.name to be consistent with the scrape configs we have been suggesting for many years for Promtail and Grafana Alloy. However, Loki has evolved a lot over the years and with the introduction of Structured Metadata we no longer recommend indexing a few of the labels we included in the defaults as they often have very high [cardinality](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/labels/cardinality/) and Structured Metadata is a much better place for them. We no longer recommend the following as default labels:
+
+- `k8s.pod.name`
+- `service.instance.id`
+
+Because removing these resource attributes from labels would be a breaking change for existing users, they have not yet been deprecated. If you are a new user of Grafana Loki, we recommend that you modify your Grafana Alloy or OpenTelemetry Collector configuration to convert these resource attributes from index labels to structured metadata.
+
+## Alloy configuration file example
+
+If you are using [Grafana Alloy](https://grafana.com/docs/alloy/latest/) to collect logs, the following example shows how to update your `alloy.config` file to demote `k8s.pod.name` from an index label to structured metadata.
+
+```alloy
+// discovery.kubernetes allows you to find scrape targets from Kubernetes resources.
+// It watches cluster state and ensures targets are continually synced with what is currently running in your cluster.
+discovery.kubernetes "pod" {
+  role = "pod"
+}
+
+// discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules.
+// If no rules are defined, then the input targets are exported as-is.
+discovery.relabel "pod_logs" {
+  targets = discovery.kubernetes.pod.targets
+
+  // Label creation - "namespace" field from "__meta_kubernetes_namespace"
+  rule {
+    source_labels = ["__meta_kubernetes_namespace"]
+    action = "replace"
+    target_label = "namespace"
+  }
+
+  // Label creation - "pod" field from "__meta_kubernetes_pod_name"
+  //rule {
+  //  source_labels = ["__meta_kubernetes_pod_name"]
+  //  action = "replace"
+  //  target_label = "pod"
+  //}
+
+
+  // Label creation - "container" field from "__meta_kubernetes_pod_container_name"
+  rule {
+    source_labels = ["__meta_kubernetes_pod_container_name"]
+    action = "replace"
+    target_label = "container"
+  }
+
+  // Label creation -  "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name"
+  rule {
+    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
+    action = "replace"
+    target_label = "app"
+  }
+
+  // Label creation -  "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name"
+  // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name
+  rule {
+    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
+    action = "replace"
+    target_label = "job"
+    separator = "/"
+    replacement = "$1"
+  }
+
+  // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name"
+  // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log
+  rule {
+    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
+    action = "replace"
+    target_label = "__path__"
+    separator = "/"
+    replacement = "/var/log/pods/*$1/*.log"
+  }
+
+  // Label creation -  "container_runtime" field from "__meta_kubernetes_pod_container_id"
+  rule {
+    source_labels = ["__meta_kubernetes_pod_container_id"]
+    action = "replace"
+    target_label = "container_runtime"
+    regex = "^(\\S+):\\/\\/.+$"
+    replacement = "$1"
+  }
+}
+
+// loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
+loki.source.kubernetes "pod_logs" {
+  targets    = discovery.relabel.pod_logs.output
+  forward_to = [loki.process.pod_logs.receiver]
+}
+
+// loki.process receives log entries from other Loki components, applies one or more processing stages,
+// and forwards the results to the list of receivers in the component's arguments.
+loki.process "pod_logs" {
+  stage.static_labels {
+      values = {
+        cluster = "<CLUSTER_NAME>",
+      }
+  }
+
+  forward_to = [loki.write.<WRITE_COMPONENT_NAME>.receiver]
+}
+```
+
+## Kubernetes Monitoring Helm chart configuration example
+
+If you are using the [Kubernetes Monitoring Helm chart](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/) to collect Kubernetes logs to send to Loki, the following example shows how to update the `values.yaml` file for your Helm chart to demote `k8s.pod.name` and `service.instance.id` from index labels to structured metadata.
+
+```yaml
+# Enable pod log collection for the cluster. Will collect logs from all pods in both the meta and loki namespace. 
+
+podLogs:
+  enabled: true
+  collector: alloy-singleton
+  #List of attributes to use as index labels in oki
+  labelsToKeep:
+    - app
+    - app_kubernetes_io_name
+    - component
+    - container
+    - job
+    - level
+    - namespace
+    - service_name
+    - cluster
+  gatherMethod: kubernetesApi
+  namespaces:
+    - meta
+    - loki
+  #This is the section that specifies to send k8s.pod.name and service.instance.id to structured metadata
+  structuredMetadata:
+    instance_id:
+    pod:
+  ```
+
+## Loki configuration example
+
+If you are running Loki in a Docker container or on-premises, here is an example of how to modify your Loki `values.yaml` file to demote `k8s.pod.name` and `service.instance.id` from index labels to structured metadata.
+
+```yaml
+
+distributor:
+  otlp_config:
+    # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
+    # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
+      default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]
+
+```
+
+Kubernetes example `values.yaml`:
+
+```yaml
+loki:
+  distributor:
+    otlp_config:
+    # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
+    # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
+      default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]
+```

docs/sources/get-started/labels/structured-metadata.md

@@ -2,6 +2,7 @@
 menuTitle: Structured metadata
 title: What is structured metadata
 description: Describes how to enable structure metadata for logs and how to query using structured metadata to filter log lines.
+weight: 300
 ---
 # What is structured metadata