14 Commits (91cae03fe1a7c302f2075dd928f5fe7033c15d95)

Author	SHA1	Message	Date
Ashwanth	91cae03fe1	chore: restructure compactor package (#10775) ... **What this PR does / why we need it**: This pr moves the compactor pkg from `pkg/storage/stores/shipper/indexshipper/compactor` to `pkg/compactor` - same level as other high level components (querier, distributor...) table compaction code however resides in the respective store pkg closer to other store specific code. `pkg/storage/stores/shipper/indexshipper/boltdb/compactor/table_compactor.go` `pkg/storage/stores/shipper/indexshipper/tsdb/compactor.go` **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729) ... **What this PR does / why we need it**: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │   └── compactor ├── compactor │   ├── client │   │   └── grpc │   ├── deletion │   ├── deletionmode │   ├── generationnumber │   └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │   ├── index │   ├── testdata │   └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... **Special notes for your reviewer**: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Travis Patterson	cb2b3ca9ea	Add limit to the series volume endpoint (#9833) ... To protect index gateways, this PR adds a per-tenant limit to the `series_volume` endpoint so the accumulation of series can't grow without bound.	2 years ago
Travis Patterson	806674fdaa	Add log-volume feature flag (#9762) ... Adds a feature flag for use with the new log-volume endpoints so associated features can be rolled out incrementally.	3 years ago
Danny Kopping	a63ad06509	Querier/Ruler: query blocker (#7785) ... Block malicious or expensive queries using a per-tenant runtime configuration.	3 years ago
Dylan Guedes	b1d4efab12	Loki: Per-tenant stream sharding (#7311) ... **What this PR does / why we need it**: - Move stream sharding configuration to its own package to avoid cyclic imports - Change stream sharding to be a per-tenant configuration - Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding - Change stream sharding flags prefix from `distributor.shard-stream` to `shard-stream`	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835) ... What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Travis Patterson	bd3d80ad80	Add ability to limit the impact of pathological delete requests. (#6854) ... * make per-tenant override the mode rather than binary on/off * Rely on overrides to configure deletion - Remove 'deletion_mode' flag * add ability to delete requests at any time * don't load requests from disabled tenants * review comments * fix tests * loadDeleteRequestsToProcess cleanup * lint * docs * review feedback * lint	3 years ago
Trevor Wood	aed11c25e6	fix(loki): allow global and per tenant sigv4 config (#6358) ... Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>	4 years ago
Danny Kopping	a961612436	Ruler: Refactoring remote-write config overrides (#4429) ... * Refactoring remote-write config overrides Replacing mergo.Merge because it cannot distinguish between a nil and an empty slice, which is needed to overwrite relabel configs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Simplifying condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Merging headers instead of overwriting blindly Adding extra check for header smuggling Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Unexporting OverwriteMarshalingStringMap field, preferring constructor Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	415d4b03fa	Safe per tenant overrides loading (#4421) ... * safe per tenant overrides loading * exports submap for OverwriteMarshalingStringMap	4 years ago
Kaviraj	a4b897482b	Add custom UnmarshalJSON for bytesize type (#4289) ... * Add custom UnmarshalJSON for bytesize type * Add more test case * Fix limits test	4 years ago
Travis Patterson	8a2081a39e	Distributor: Truncate rather than drop log lines (#4051) ... * Truncate rather than drop logs - When log lines are longer than maxLineSize, truncate them to maxLineSize rather than dropping them Fixes: #3747 * Review feedback: - Rename 'max_line_size_should_truncate' to 'max_line_size_truncate' - Add configurable truncated indicator - Add Truncated Line metric - Changed truncated metrics to have the name mutated_{samples,bytes}_total * Linter * Review Feedback - Make metric variable names match metric names - Add benchmarks - reduce truncation allocs by 100x * Linter * Remove truncation indicator	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632) ... Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	5 years ago