apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	de2b0c6e9d	Rename per-request query limits. (#9011 ) What this PR does / why we need it: `maxQueryTime` is more in line with the other per-request limits we have. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Karsten Jeschkies	ffb605fa85	Fix: Ensure query limits tripperware wraps frontend. (#8789 ) What this PR does / why we need it: The query frontend would not enforce the per-request query limits because the injected tripperware was wrapped after the frontend has been created. That is a bug. The order must be changed so that the `queryrange.LimitsMiddleware` runs after the `querylimits.limitsMiddleware`. Which issue(s) this PR fixes: Fixes #8790 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	61311957a3	Integration Tests: allow loki logs to be optionally output in integration tests (#8793 ) What this PR does / why we need it: Debugging integration tests can be difficult without Loki's logs being output. Changing the call from `cluster.New()` to `cluster.New(true, level.WarnValue())` now allows the author the option to temporarily or permanently enable logging for their test, as well as configure a level filter to control the noise. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744 ) What this PR does / why we need it: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. Which issue(s) this PR fixes: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (LID 0002: Remote Rule Evaluation).	2 years ago
Karsten Jeschkies	28ebf8ec12	Impose per-request limit on label queries. (#8777 ) What this PR does / why we need it: #8727 introduced per-request limits. These should be enforce for label queries as well. This change adds the required middleware to all label query endpoints. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	2 years ago
Christian Haudum	9a75c0e05b	Fix flaky `TestMicroServicesDeleteRequest` integration test (#8436 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Michel Hollands	20b13cb317	Fix comparison in a delete request integration test (#8389 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Sandeep Sukhani	b2d0481cbe	deletion: fix log deletion with line filters (#8151 )	2 years ago
Travis Patterson	b704cf986b	Remove flaky helm test (#8122 ) This test is causing tens of timeouts across all PRs in our system. Delete it to improve CI stability	2 years ago
Trevor Whitney	f5fbfabd84	Add 3rd target (`backend`) to SSD/Scalable mode (#7650 ) What this PR does / why we need it: This adds a 3rd target to SSD/Scalable mode, as well as a config flag to run the legacy `read` mode from the original 2 target configuration in order to give people time to migrate before we remove this option in Loki 3.0 (hopefully). The 3rd target has the two major advantages: 1. Allows the read path to be run as a deployment and thus auto-scaled using our existing auto-scaling logic for queriers 2. Creates consistency with Mimir since they went with a 3 target model for their SSD deployment.	2 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814 ) What this PR does / why we need it: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708 ) What this PR does / why we need it: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> Which issue(s) this PR fixes: https://github.com/grafana/loki/issues/7696 Special notes for your reviewer: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	3 years ago
Sandeep Sukhani	97a7ed0b61	fix logs deletion integration test to properly validate deleted data (#7751 ) What this PR does / why we need it: Currently, the integration test for deletion does not validate whether the delete request properly deleted the requested data or not. It just currently checks if the delete request got processed. This was due to the singleton `boltdb-shipper` client causing problems in flushing chunks. I have used `ResetBoltDBIndexClientWithShipper` to reset the boltdb-shipper client to get the chunk flush from the ingester working. The test now does the checks to ensure that the right data got deleted by the compactor. This should help us catch any bugs earlier for any future changes in deletion code. Checklist - [x] Tests updated	3 years ago
Travis Patterson	e89c30ce6b	Fix Flaky Integration Tests (#7360 ) Allow the OS kernel to decide what's an open port instead of trying to guess	3 years ago
Dylan Guedes	2dc5a71a67	Tests: Reduce flakiness of GCPLog and integration tests (#7273 ) What this PR does / why we need it: - Reduce flakiness of GCPLog tests by reusing context and stopping GCP topic - Reduce flakiness of integration tests by: - using port as they're assigned - reusing context - adding timeout to calls (hence, if something get stuck, tests fail earlier)	3 years ago
Mohamed-Amine Bouqsimi	3d4788fb51	Add ability for Ruler to remote write to multiple clients (#6823 )	3 years ago
Christian Simon	d63add09c5	Fix free port allocation flake (#6898 ) Occassionaly we can see test flakes like that: > server: listen tcp :34473: bind: address already in use This is because we allocated ports by checking if they are free and then using them. Altough there is a chance that already allocated ports are used twice. This changes prevents this from happening by keeping track of the ports that are about to be used.	3 years ago
Travis Patterson	983ab80e7c	Add delete api validations (#6860 ) * break out the middleware * Add validation to the API - Time must be in RFC3339 or a 10-digit-unix-seconds timestamp - Start time must always exist * lint * docs * improve tests * cleanup * cleanup * access runtime config via function on validation	3 years ago
Travis Patterson	bd3d80ad80	Add ability to limit the impact of pathological delete requests. (#6854 ) * make per-tenant override the mode rather than binary on/off * Rely on overrides to configure deletion - Remove 'deletion_mode' flag * add ability to delete requests at any time * don't load requests from disabled tenants * review comments * fix tests * loadDeleteRequestsToProcess cleanup * lint * docs * review feedback * lint	3 years ago
Trevor Whitney	5073b48b62	Add integration for SSD (read & write targets) (#6585 ) * Add integration for SSD (read & write targets) * Hardcode randStringRunes to fix linter	3 years ago
Michel Hollands	f80e487dcf	Remove whole stream deletion mode (#6435 ) * Remove whole-stream-deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove whole-stream-deletion from docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Sort changelog entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove link to wrong configuration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set default deletion mode to disabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extra white line in documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value in docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add DeletionEnabled method on mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Michel Hollands	b4e6c59936	Compactor: add per tenant compaction delete enabled flag (#6410 ) * Add per tenant compaction delete enabled flag Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove changes in wrong place Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add compactor deletion enabled field Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use limit in compactor Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use http middleware and add test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix lint issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert to default setting if no override Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add default value command line option Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Enable access to deletion API for integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename flag to allow_deletes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update per review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Michel Hollands	65e3148bc9	Update delete integration test (#6192 ) * Add wait until processed Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add query lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove Cortex reference Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deleted lines metric and check in test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set negative cancel period so default 1 minute delay is compensated for Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Reduce timeout Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add helper function Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Comment out assertions that only work with flush Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused metric type param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Check counter in unit test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename ClientOption to Option as per the linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove redundant check Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * fill in MsgAndArgs field of test assertion Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Travis Patterson	e313d8a873	Remove __name__ from label calls (#6229 )	3 years ago
Michel Hollands	b6ff9d5889	Add integration test for delete request (#6188 ) * Add integration test for delete request Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add line filter Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Christian Simon	213bb86802	Add integration tests which test happy path (#5968 ) * Add integration tests which test happy path This PR adds simple go based integration tests, which can run Loki in two modes: - single-binary - micro-services with index-gateway After standing up it performs the most basic duties like: - ingest logs - run a range query - query label names and label values This is meant to catch more significant problem as early as possible in the PR. We should aim to keep this as simple as possible. * Replace config generation with template Makes configuration much more readable and avoids easy mistakes * Apply suggestions from code review Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago

29 Commits (fd5d78784fbc0200bb2ca9250515b818d4c24c0e)