Provide Terraform script for setting up S3 and document role based access. (#7656)

**What this PR does / why we need it**:
Role based access to S3 on an EKS cluster via service account annotation
was not documented well and let to some confusion. This change documents
how a role based accessis achieved with providing an AWS secret access
key and access key id. Thus there is no change of accidentally leaking
credentials.

A Terraform module is provided under `production/terraform/modules/s3`
that provisions and S3 bucket and IAM role and policy.

**Which issue(s) this PR fixes**:
Fixes #7279

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the `CONTRIBUTING.md` guide
- [x] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`

Co-authored-by: Pablo Angulo <pablo.angulo@grafana.com>

docs/sources/installation/helm/configure-storage/index.md

@@ -15,13 +15,13 @@ keywords:
 
 The [scalable](../install-scalable/) installation requires a managed object store such as AWS S3 or Google Cloud Storage or a self-hosted store such as Minio. The [single binary](../install-monolithic/) installation can only use the filesystem for storage.
 
-This guide assumes Loki has been installed in on of the modes above and that a `values.yaml ` has been created.
+This guide assumes Loki will be installed in on of the modes above and that a `values.yaml ` has been created.
 
 **To use a managed object store:**
 
 1. Set the `type` of `storage` in `values.yaml` to `gcs` or `s3`.
 
-2. Configure the storage client under `storage.gcs` or `storage.s3`.
+2. Configure the storage client under `loki.storage.gcs` or `loki.storage.s3`.
 
 
 **To install Minio alongside Loki:**
@@ -34,3 +34,32 @@ This guide assumes Loki has been installed in on of the modes above and that a `
     minio:
       enabled: true
     ```
+
+**To grant access to S3 via an IAM role without providing credentials:**
+
+1. Provision an IAM role, policy and S3 bucket as described in [Storage](../../../storage/#aws-deployment-s3-single-store).
+   - If the Terraform module was used note the annotation emitted by `terraform output -raw annotation`.
+
+2. Add the IAM role annotation to the service account in `values.yaml`:
+
+   ```
+   serviceAccount:
+     annotations:
+       "eks.amazonaws.com/role-arn": "arn:aws:iam::<account id>:role/<role name>:
+   ```
+
+3. Configure the storage:
+
+   ```
+   loki:
+     storage:
+       type: "s3"
+       s3:
+         region: eu-central-1
+       bucketNames:
+         chunks: <bucket name>
+         ruler: <bucket name>
+         admin: <bucket name>
+   ```
+
+   Note that `endpoint`, `secretAccessKey` and `accessKeyId` have been omitted.

docs/sources/operations/storage/_index.md

@@ -58,34 +58,7 @@ When using S3 as object storage, the following permissions are needed:
 
 Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`
 
-The following policy sets these permissions
-
-```json
-{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Sid": "LokiStorage",
-            "Effect": "Allow",
-            "Principal": {
-                "AWS": [
-                    "arn:aws:iam::<account_ID>"
-                ]
-            },
-            "Action": [
-                "s3:ListBucket",
-                "s3:PutObject",
-                "s3:GetObject",
-                "s3:DeleteObject"
-            ],
-            "Resource": [
-                "arn:aws:s3:::<bucket_name>",
-                "arn:aws:s3:::<bucket_name>/*"
-            ]
-        }
-    ]
-}
-```
+See the [AWS deployment section](../../storage/#aws-deployment-s3-single-store) on the storage page for a detailed setup guide.
 
 ### DynamoDB
 

docs/sources/storage/_index.md

@@ -238,6 +238,64 @@ storage_config:
       dynamodb_url: dynamodb://region
 ```
 
+The role should have a policy with the following permissions attached.
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "LokiStorage",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": [
+                    "arn:aws:iam::<account_ID>"
+                ]
+            },
+            "Action": [
+                "s3:ListBucket",
+                "s3:PutObject",
+                "s3:GetObject",
+                "s3:DeleteObject"
+            ],
+            "Resource": [
+                "arn:aws:s3:::<bucket_name>",
+                "arn:aws:s3:::<bucket_name>/*"
+            ]
+        }
+    ]
+}
+```
+
+**To setup an S3 bucket and an IAM role and policy:** 
+
+This guide assumes a provisioned EKS cluster.
+
+1. Checkout the Loki repository and navigate to [production/terraform/modules/s3](https://github.com/grafana/loki/tree/main/production/terraform/modules/s3).
+
+2. Initialize Terraform `terraform init`.
+
+3. Export the AWS profile and region if not done so:
+
+   ```
+   export AWS_PROFILE=<profile in ~/.aws/config>
+   export AWS_REGION=<region of EKS cluster>
+   ```
+
+4. Save the OIDC provider in an enviroment variable:
+
+   ```
+   oidc_provider=$(aws eks describe-cluster --name <EKS cluster> --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
+   ```
+
+   See the [IAM OIDC provider guide](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for a guide for creating a provider.
+
+5. Apply the Terraform module `terraform -var region="$AWS_REGION" -var cluster_name=<EKS cluster> -var oidc_id="$oidc_provider"`
+
+   Note, the bucket name defaults to `loki-data` but can be changed via the
+   `bucket_name` variable.
+
+
 ### On prem deployment (Cassandra+Cassandra)
 
 **Keeping this for posterity, but this is likely not a common config. Cassandra should work and could be faster in some situations but is likely much more expensive.**

production/terraform/modules/s3/main.tf

@@ -0,0 +1,106 @@
+provider "aws" {
+  region = var.region
+}
+
+data "aws_caller_identity" "current" {}
+
+data "aws_eks_cluster" "current" {
+  name = var.cluster_name
+}
+
+locals {
+  oidc_id = replace(data.aws_eks_cluster.current.identity[0].oidc[0].issuer, "https://", "")
+}
+
+data "aws_iam_policy_document" "oidc" {
+  statement {
+    actions = ["sts:AssumeRoleWithWebIdentity"]
+    effect  = "Allow"
+
+    condition {
+      test     = "StringEquals"
+      variable = "${local.oidc_id}:sub"
+      values   = ["system:serviceaccount:${var.namespace}:${var.serviceaccount}"]
+    }
+
+    condition {
+      test     = "StringEquals"
+      variable = "${local.oidc_id}:aud"
+      values   = ["sts.amazonaws.com"]
+    }
+
+    principals {
+      identifiers = [
+        "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_id}"
+      ]
+      type        = "Federated"
+    }
+  }
+}
+
+resource "aws_s3_bucket" "loki-data" {
+  bucket = "${var.bucket_name}"
+}
+
+resource "aws_s3_bucket_policy" "grant-access" {
+  bucket = aws_s3_bucket.loki-data.id
+  policy = jsonencode({
+    Version: "2012-10-17",
+    Statement: [
+        {
+            Sid: "Statement1",
+            Effect: "Allow",
+            Principal: {
+                AWS: aws_iam_role.loki.arn  
+            },
+            Action: [
+              "s3:PutObject",
+              "s3:GetObject",
+              "s3:DeleteObject",
+              "s3:ListBucket"
+            ],
+            Resource: [
+	      aws_s3_bucket.loki-data.arn,
+	      "${aws_s3_bucket.loki-data.arn}/*"
+            ]
+        }
+    ]
+  })
+}
+
+resource "aws_iam_role" "loki" {
+  name               = "LokiStorage-${var.cluster_name}"
+  assume_role_policy = data.aws_iam_policy_document.oidc.json
+
+  inline_policy {}
+}
+
+resource "aws_iam_policy" "loki" {
+  name        = "LokiStorageAccessPolicy-${var.bucket_name}"
+  path        = "/"
+  description = "Allows Loki to access bucket"
+
+  policy = jsonencode({
+    Version: "2012-10-17",
+    Statement: [
+        {
+            Effect: "Allow",
+            Action: [
+		"s3:ListBucket",
+		"s3:PutObject",
+		"s3:GetObject",
+		"s3:DeleteObject"
+	    ],
+            Resource: [
+		    aws_s3_bucket.loki-data.arn,
+		    "${aws_s3_bucket.loki-data.arn}/*"
+	    ]
+        }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "loki-attach" {
+  role       = aws_iam_role.loki.name
+  policy_arn = aws_iam_policy.loki.arn
+}

production/terraform/modules/s3/outputs.tf

@@ -0,0 +1,4 @@
+output "annotation" {
+  description = "Service account annotation"
+  value       = "eks.amazonaws.com/role-arn=${aws_iam_role.loki.arn}" 
+}

production/terraform/modules/s3/variables.tf

@@ -0,0 +1,26 @@
+variable "region" {
+  description = "AWS region"
+  type        = string
+  default     = "us-east-2"
+}
+
+variable "bucket_name" {
+  description = "Bucket name for Loki storage"
+  type    = string
+}
+
+variable "cluster_name" {
+  description = "Name of EKS cluster"
+  type = string
+}
+
+variable "namespace" {
+  description = "Namespace of Loki installation"
+  type        = string
+}
+
+variable "serviceaccount" {
+  description = "Service account of Loki installation"
+  type        = string
+  default     = "loki"
+}

production/terraform/modules/s3/versions.tf

@@ -0,0 +1,15 @@
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 4.15.0"
+    }
+
+    random = {
+      source  = "hashicorp/random"
+      version = "3.1.0"
+    }
+  }
+
+  required_version = "> 1.2.0"
+}