apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	8eac851962	Remove print statements introduced by #8727 . (#8775 )	3 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Trevor Whitney	dac3b84d08	Loki cloud integration instructions (and necessary mixin changes) (#8492 ) What this PR does / why we need it: This PR is a first stab at documentation for setting up the Grafana Cloud Loki Integration to monitor a local Loki cluster, installed using the helm chart, using Grafana Cloud. In addition to instructions on how to collect the necessary kubernetes metrics, there were also some things with the Loki mixin (which provides dashboards, alerts, and rules to the integration) that needed to be fixed. For example, our dashboards rely on there being a cluster label on our recording rules, yet our mixin does not include this? Also, we are still including `cortex-gw` panels even when the mixin is compile with `internal_components: false`. Special notes for your reviewer: The changes to the mixin will not be reflected in the current version of the integration. We need to merge this PR first then update the integration. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Susana Ferreira	b8118b1e7b	Helm enterprise-logs: update kubernetes secrets creation (#8467 ) What this PR does / why we need it: Fix the Kubernetes secrets creation for the tokengen and provisioner jobs for multiple helm installations/uninstallations. The execution flow for the helm installation is the following: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container creates a new Kubernetes secret for the generated admin token. 3) `provisioner` job `provisioner` container creates the new admin resources via the Admin API and writes the tokens to the `bootstrap` mount. 4) `provisioner` job `create-secret` container creates new Kubernetes secrets for each token of the additional tenants. ## Previous implementation Uninstall and re-install the enterprise-logs helm: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. This is because this job uses an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) mount which is deleted once the job completes. When a new job is created, the directory will be again empty, forcing the `tokengen` to generate a new token. 2) `tokengen` job `create-secret` container fails, because an admin token secret already exists: ``` Defaulted container "create-secret" out of: create-secret, loki (init) error: failed to create secret secrets "enterprise-logs-admin-token" already exists ``` 3) The helm installation fails ❌ ## Implementation in this PR 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container updates the previously created Kubernetes secret for the admin token. 3) `provisioner` job `provisioner` container checks that the admin resources already exist and doesn't write the token files to the `bootstrap` mount. 4) `provisioner` job `create-secret` container checks that the token files are not present on the `bootstrap` mount, and does not create new secrets. 5) The helm installation succeeds ✅ Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	3 years ago
Susana Ferreira	00d5142c40	Helm: add sleep argument to helm makefile on local development (#8502 )	3 years ago
Susana Ferreira	27d5b83c24	Helm: add debug option to helm command for verbose output on local development (#8479 )	3 years ago
Trevor Whitney	ef8b382c06	[helm] Single binary object storage (#8284 ) Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Trevor Whitney	9920a27f06	Update GEL plugin in k3d env to latest (#8378 )	3 years ago
Trevor Whitney	cc39334391	[helm] Fix helm dashboards (#8261 ) Fix the dashboards shipped with the helm chart. Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Trevor Whitney	71846b0430	[helm] add local k3d loki cluster (#8218 )	3 years ago
Trevor Whitney	40ed217620	Make secret job fail when not logged in (#8215 ) Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Karsten Jeschkies	44e8422dc4	Enalbe multi tenant queries on Helm test chart. (#8228 ) Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Trevor Whitney	b6004050a6	[helm] Fix self-monitoring tenant secret (#8198 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/8081 introduced a bug that caused the `provisioner` job to create an incorrect secret for the self-monitoring tenant used by the canary. This fixes that bug, as well as adding a readme on how to use the `helm-cluster` k3d env. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Whitney	eb39e2645f	[helm] allow provisioned tokens to be created in different namespaces (#8081 ) What this PR does / why we need it: _Note_: This PR bumps the chart to 4.0, so it includes a few other breaking changes we've been waiting to make. The new helm chart has a few different ways to create tokens when running in enterprise mode: * The tokengen job creates the initial admin token * The provisioner job creates the self-monitoring tenant if enabled * The provisioner job can also create additional tenants if configured The problem with these jobs is they put the generated read/write tokens for these tenants in secrets in the same namespace as the loki deploiyment. This is problematic if you need these tokens for a specific application, for example a Grafana datasource, that is deployed in a separate namespace. This PR introduces two breaking changes, one of which is breaking as it significantly changes how both the self monitoring and additional tenants are configured. First it adds the value `enterprise.adminToken.additionalNamespaces` and moves `enterprise.adminTokenSecret` to `enterprise.adminToken.secret`. If `additionalNamespaces` are included, additional secrets containing the GEL admin token are created in those namespaces as well as the release namespace. Second, it changes the structure of defining a tenant from just a string name, to an object containing a `name` and `secretNamespace` field. The `secretNamespace` is the namespace the secret containing that tenants token(s) will be placed in. In the case of the self-monitoring token, a secret will still be created in the default namespace in addition to the `secretNamespace` specified as that token is required by multiple components.	3 years ago
Trevor Whitney	712bfdd0e0	[helm] add third scalable target `backend` (#7920 ) What this PR does / why we need it: Adds the new 3rd scalable deployment mode target (backend) to the helm chart. This new target was added in #7650	3 years ago
Trevor Whitney	3f9b9cc961	[helm] fix race between minio bucket job and enterprise tokengen job (#8064 ) What this PR does / why we need it: This PR replaces https://github.com/grafana/loki/pull/7985. It solves the same problem, which is the race condition between the minio create bucket job and the enterprise tokengen job, but without removing minio.	3 years ago
Trevor Whitney	98fd01aef8	Add empty k3d cluster for testing helm chart (#7984 ) What this PR does / why we need it: Add a new k3d cluster which is a cluster that has all Loki dependencies for purposes of testing the helm charts using `helm install`. The other environments use the helm charts, but deploy them using `tanka` and therefore can't catch certain race conditions or issues that occur with helm installs.	3 years ago
Trevor Whitney	6c53113be1	[helm] Add a loki canary test to the helm chart (#7229 ) What this PR does / why we need it: Add a helm chart test that utilizes the Loki canary to confirm the Loki cluster is functioning correctly. Which issue(s) this PR fixes: Fixes #7228 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Karsten Jeschkies	9bc6e85604	Helm: Allow users to define remote write for metrics. (#7399 )	3 years ago
Trevor Whitney	afd63c5981	[helm] Add Loki Canary to Helm Chart (#7173 ) What this PR does / why we need it: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. Which issue(s) this PR fixes: Fixes #7019 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Trevor Whitney	4b6df86333	Add local k3d environment for enterprise-logs (#7146 ) What this PR does / why we need it: This adds a local k3d based development cluster for testing out the enterprise logs side of the helm chart, which can be enabled by setting: ``` enterprise: enabled: true ``` It requires both the `gex-plugins` and `enterprise-logs` repos to be checked out locally (though the latter soon will not be needed once the provisioner is published). We need this environment when developing enterprise features for the helm chart.	3 years ago
Karsten Jeschkies	c23e541fe7	Generate Helm chart README. (#7012 ) What this PR does / why we need it: The Helm CI would break because the Helm chart README was not generated. This change * updates the README * introduces a filter that will only run the Helm CI on changes to the Helm chart directory * removes trailing whitespace * corrects the Minio chart repository	3 years ago
Karsten Jeschkies	1c3f5d0985	Helm: Upgrade Minio chart dependency to 4.0.12. (#6974 ) What this PR does / why we need it: Changes to a supported Minio chart. Minio is also enabled by default. Which issue(s) this PR fixes: Fixes #6973 Checklist - [x] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979 ) Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Christian Simon	88e2705034	Update alpine and golang base images (#6897 ) * Upgrade base images to alpine 3.15.6 everywhere https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html * Update golang images to 1.18.5	3 years ago
Karsten Jeschkies	51b09d771f	Open access to Minio console in dev setup. (#6807 )	3 years ago
Kaviraj Kanagaraj	0d99d80acd	upgrade `go:1.17.9` to `go:1.18.4` (#6726 )	3 years ago
Christian Simon	084a0804f1	docker-compose: Disable auth in Grafana (#5998 )	4 years ago
Christian Simon	9d0fd3a7e3	Upgrade golang version to v1.17.9 (#5899 ) ``` git ls-files '[Dd]ockerfile' \| grep -v ^vendor/ \| xargs sed -i "s/golang:[0-9\\.]*/golang:1.17.9/g" \| ag 'golang:' ```	4 years ago
Karsten Jeschkies	7193f540c2	Return bad request for multi-tenant tail. (#5792 ) Closes #5753	4 years ago
Christian Simon	3d941ccccb	Update alpine images to 3.15.4 (#5780 )	4 years ago
Ed Welch	b136d0dab3	Increase flush_op_timeout to 10m, 10s is often much too short for a big WAL replay which can generate a lot of chunks very quickly. (#5768 ) Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620 ) What this PR does / why we need it: The original change injected the user id when the org id was required to fetch all data. Which issue(s) this PR fixes: Fixes grafana/loki-private#230 Special notes for your reviewer: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1\|2" --data-urlencode 'query={compose_service=~"log-gen."}' \| jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. Checklist* - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Salva Corts	651ac093b6	Fix circular dependencies (#5420 )	4 years ago
Trevor Whitney	c477e29c3f	Add tooling for local k3d clusters under tools/dev (#5272 ) * add k3d folder with loki-distributed env * add readme * shellcheck create_cluster.sh * lint jsonnet * Update grafana image to latest	4 years ago
Susana Ferreira	2fdd24038b	Move querier.split-queries-by-interval to a per-tenant configuration (#5023 )	4 years ago
Callum Styan	096cf0cfaf	Update golang and loki-build-image image versions. (#4481 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Karsten Jeschkies	c3aaebe509	Support frontend V2 with query scheduler. (#4071 ) * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Remove unused querier option. * Use new query-scheduler module in Loki. * Remove unused option. * Document `max_queriers_per_tenant`.	4 years ago
Karsten Jeschkies	5a5bcb16e2	Document 0 meaning for retries. (#4096 )	4 years ago
Karsten Jeschkies	17f4a73a24	Filter instant queries and shard them. (#3984 ) * Filter instan queries and shard them. * [WIP] Test instant query sharding * Trace casting error. * WRap LokiInstantRequest for Params interface. * Convert paras to request. * Convert vector. * Use proper query time stamp. * Assert number of calls. * Convert vector to multiple samples. * Format code. * Use type switch. * Format code. * Rename Matrix to proto conversion. * Remvoe comment on caching. * Update pkg/querier/queryrange/roundtrip_test.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove nil middlware. * Handle LokiInstantRequest correctly. * Assert reponse type. * Return LokiPromResponse. * Verify err in tests. * Set shards. * Correct result type. * Use limits middleware. * Use shars parameter in querier. * Return vector. * Unify params. * Change direction to FORWARD. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Lize Cai	8370ed3b04	fix typo in loki docker-compose (#4016 )	5 years ago
Cyril Tovena	86000dd320	Development Docker Compose Setup (#3941 ) * Adds the first version of a docker compose dev stack. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add index-gateway and code source. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better docker ignore. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better bash comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update tools/dev/loki-boltdb-storage-s3/README.md Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * lint bash scripts. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago

44 Commits (Alex3k-patch-1)