apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ashwanth	0489d16af8	fix flaky integration TestMicroServicesDeleteRequest test (#9337 ) What this PR does / why we need it: `loki_compactor_deleted_lines` increments each time we find a line to delete from a chunk. If the chunk is indexed in multiple tables, we increment it for each of these tables as they get compacted. Currently the chunk (that holds the line to be deleted) starts @`now - 46h`, so depending on when the test runs it gets indexed in 2 or 3 tables. It has been failing between hours `22:00 - 23:59` as the chunk only overlaps and gets indexed in 2 tables while the test expects it to be indexed in 3 https://drone.grafana.net/grafana/loki/23063/2/1 https://drone.grafana.net/grafana/loki/23067/2/1 Updating the chunk start time to now-48h so that it always gets indexes in 3 tables	3 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447 ) What this PR does / why we need it: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](`a85d01c9cd/docs/sources/upgrading/_index.md`) for more details. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/7276 Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754 ) Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> What this PR does / why we need it: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](`ff7b462973/pkg/storage/factory.go (L188)`) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). Note: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. Which issue(s) this PR fixes: Fixes #7276 Special notes for your reviewer: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Karsten Jeschkies	de2b0c6e9d	Rename per-request query limits. (#9011 ) What this PR does / why we need it: `maxQueryTime` is more in line with the other per-request limits we have. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Karsten Jeschkies	ffb605fa85	Fix: Ensure query limits tripperware wraps frontend. (#8789 ) What this PR does / why we need it: The query frontend would not enforce the per-request query limits because the injected tripperware was wrapped after the frontend has been created. That is a bug. The order must be changed so that the `queryrange.LimitsMiddleware` runs after the `querylimits.limitsMiddleware`. Which issue(s) this PR fixes: Fixes #8790 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Danny Kopping	61311957a3	Integration Tests: allow loki logs to be optionally output in integration tests (#8793 ) What this PR does / why we need it: Debugging integration tests can be difficult without Loki's logs being output. Changing the call from `cluster.New()` to `cluster.New(true, level.WarnValue())` now allows the author the option to temporarily or permanently enable logging for their test, as well as configure a level filter to control the noise. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744 ) What this PR does / why we need it: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. Which issue(s) this PR fixes: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (LID 0002: Remote Rule Evaluation).	3 years ago
Karsten Jeschkies	28ebf8ec12	Impose per-request limit on label queries. (#8777 ) What this PR does / why we need it: #8727 introduced per-request limits. These should be enforce for label queries as well. This change adds the required middleware to all label query endpoints. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	3 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	3 years ago
Christian Haudum	9a75c0e05b	Fix flaky `TestMicroServicesDeleteRequest` integration test (#8436 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Michel Hollands	20b13cb317	Fix comparison in a delete request integration test (#8389 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Sandeep Sukhani	b2d0481cbe	deletion: fix log deletion with line filters (#8151 )	3 years ago
Travis Patterson	b704cf986b	Remove flaky helm test (#8122 ) This test is causing tens of timeouts across all PRs in our system. Delete it to improve CI stability	3 years ago
Trevor Whitney	f5fbfabd84	Add 3rd target (`backend`) to SSD/Scalable mode (#7650 ) What this PR does / why we need it: This adds a 3rd target to SSD/Scalable mode, as well as a config flag to run the legacy `read` mode from the original 2 target configuration in order to give people time to migrate before we remove this option in Loki 3.0 (hopefully). The 3rd target has the two major advantages: 1. Allows the read path to be run as a deployment and thus auto-scaled using our existing auto-scaling logic for queriers 2. Creates consistency with Mimir since they went with a 3 target model for their SSD deployment.	4 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814 ) What this PR does / why we need it: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	4 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708 ) What this PR does / why we need it: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> Which issue(s) this PR fixes: https://github.com/grafana/loki/issues/7696 Special notes for your reviewer: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	4 years ago
Sandeep Sukhani	97a7ed0b61	fix logs deletion integration test to properly validate deleted data (#7751 ) What this PR does / why we need it: Currently, the integration test for deletion does not validate whether the delete request properly deleted the requested data or not. It just currently checks if the delete request got processed. This was due to the singleton `boltdb-shipper` client causing problems in flushing chunks. I have used `ResetBoltDBIndexClientWithShipper` to reset the boltdb-shipper client to get the chunk flush from the ingester working. The test now does the checks to ensure that the right data got deleted by the compactor. This should help us catch any bugs earlier for any future changes in deletion code. Checklist - [x] Tests updated	4 years ago
Travis Patterson	e89c30ce6b	Fix Flaky Integration Tests (#7360 ) Allow the OS kernel to decide what's an open port instead of trying to guess	4 years ago
Dylan Guedes	2dc5a71a67	Tests: Reduce flakiness of GCPLog and integration tests (#7273 ) What this PR does / why we need it: - Reduce flakiness of GCPLog tests by reusing context and stopping GCP topic - Reduce flakiness of integration tests by: - using port as they're assigned - reusing context - adding timeout to calls (hence, if something get stuck, tests fail earlier)	4 years ago
Mohamed-Amine Bouqsimi	3d4788fb51	Add ability for Ruler to remote write to multiple clients (#6823 )	4 years ago
Christian Simon	d63add09c5	Fix free port allocation flake (#6898 ) Occassionaly we can see test flakes like that: > server: listen tcp :34473: bind: address already in use This is because we allocated ports by checking if they are free and then using them. Altough there is a chance that already allocated ports are used twice. This changes prevents this from happening by keeping track of the ports that are about to be used.	4 years ago
Travis Patterson	983ab80e7c	Add delete api validations (#6860 ) * break out the middleware * Add validation to the API - Time must be in RFC3339 or a 10-digit-unix-seconds timestamp - Start time must always exist * lint * docs * improve tests * cleanup * cleanup * access runtime config via function on validation	4 years ago
Travis Patterson	bd3d80ad80	Add ability to limit the impact of pathological delete requests. (#6854 ) * make per-tenant override the mode rather than binary on/off * Rely on overrides to configure deletion - Remove 'deletion_mode' flag * add ability to delete requests at any time * don't load requests from disabled tenants * review comments * fix tests * loadDeleteRequestsToProcess cleanup * lint * docs * review feedback * lint	4 years ago
Trevor Whitney	5073b48b62	Add integration for SSD (read & write targets) (#6585 ) * Add integration for SSD (read & write targets) * Hardcode randStringRunes to fix linter	4 years ago
Michel Hollands	f80e487dcf	Remove whole stream deletion mode (#6435 ) * Remove whole-stream-deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove whole-stream-deletion from docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Sort changelog entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove link to wrong configuration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set default deletion mode to disabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extra white line in documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value in docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add DeletionEnabled method on mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Michel Hollands	b4e6c59936	Compactor: add per tenant compaction delete enabled flag (#6410 ) * Add per tenant compaction delete enabled flag Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove changes in wrong place Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add compactor deletion enabled field Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use limit in compactor Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use http middleware and add test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix lint issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert to default setting if no override Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add default value command line option Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Enable access to deletion API for integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename flag to allow_deletes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update per review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Michel Hollands	65e3148bc9	Update delete integration test (#6192 ) * Add wait until processed Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add query lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove Cortex reference Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deleted lines metric and check in test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set negative cancel period so default 1 minute delay is compensated for Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Reduce timeout Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add helper function Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Comment out assertions that only work with flush Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused metric type param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Check counter in unit test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename ClientOption to Option as per the linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove redundant check Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * fill in MsgAndArgs field of test assertion Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Travis Patterson	e313d8a873	Remove __name__ from label calls (#6229 )	4 years ago
Michel Hollands	b6ff9d5889	Add integration test for delete request (#6188 ) * Add integration test for delete request Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add line filter Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Christian Simon	213bb86802	Add integration tests which test happy path (#5968 ) * Add integration tests which test happy path This PR adds simple go based integration tests, which can run Loki in two modes: - single-binary - micro-services with index-gateway After standing up it performs the most basic duties like: - ingest logs - run a range query - query label names and label values This is meant to catch more significant problem as early as possible in the PR. We should aim to keep this as simple as possible. * Replace config generation with template Makes configuration much more readable and avoids easy mistakes * Apply suggestions from code review Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	4 years ago

32 Commits (a70d092df646ec1fcd5d8cd6b71c85d3692ea8d9)