apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Dylan Guedes	9b7e48aca9	Ingesters: Revert "Add metrics.go log line when reading from an ingester (#11571 )" (#11818 ) What this PR does / why we need it: This reverts commit `f04d0db434` Although it is useful, the amount of logging it generates is too massive. In the future we might revisit it but with a different approach.	2 years ago
Michel Hollands	f04d0db434	Add metrics.go log line when reading from an ingester (#11571 ) What this PR does / why we need it: Add a metrics.go log line when a querier reads from an ingester. The output is like this: ``` level=info ts=2024-01-03T13:59:06.919965546Z caller=metrics.go:275 component=ingester org_id=fake traceID=0b216e7d014c5f87 latency=fast query_type=ingester_series start=2024-01-03T13:00:00Z end=2024-01-03T13:59:06.834Z start_delta=59m6.91996413s end_delta=85.964255ms length=59m6.834s duration=110.792µs status=200 query=<omitted> query_hash=2166136261 total_entries=9 ``` Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Sandeep Sukhani	0e5da205a6	structured metadata: track usage of structured metadata in queries (#11505 ) What this PR does / why we need it: Adds support for tracking queries referencing structured metadata. It tracks the following kinds of references: * Fields in structured metadata used as label filters, `drop`/`keep` stages etc. in the query. * Fields in structured metadata used in metric queries like `uwrap` and `by` or `without` clauses in aggregations. Checklist - [x] Tests updated	2 years ago
Travis Patterson	f668a1b80b	Add context and tenant to pipeline/extractor wrappers (#11528 ) Adding the context here means the wrapping pipeline can know when the query has stopped or been cancelled.	2 years ago
Karsten Jeschkies	9aae25008d	Optimize series response format by using repeated fileds. (#11498 ) What this PR does / why we need it: The Protobuf map type is encodied as a repeated field of map entries. Decoding them to a slice is much faster than decoding them into a map. Since Loki is not using the fast key check for a map we can use the slice decoding. This change also allows us to decode the JSON directly into the right protobuf struct. This doulbes the JSON decoding speed and reduces the memory pressure by ~40%. ``` › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > before.log › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > after.log › benchstat before.log after.log goos: darwin goarch: arm64 pkg: github.com/grafana/loki/pkg/querier/queryrange │ before.log │ after.log │ │ sec/op │ sec/op vs base │ ResponseMerge/mergeStreams_unlimited-10 32.36m ± 0% 32.63m ± 2% ~ (p=0.393 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_unlimited-10 1.050m ± 1% 1.080m ± 3% +2.84% (p=0.005 n=10) ResponseMerge/mergeStreams_limited-10 33.02m ± 0% 32.60m ± 1% -1.29% (p=0.004 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_limited-10 15.11m ± 0% 15.07m ± 0% ~ (p=0.075 n=10) _CodecDecodeLogs-10 4.395m ± 1% 4.364m ± 0% -0.72% (p=0.005 n=10) _CodecDecodeSamples-10 16.97m ± 0% 16.84m ± 2% -0.77% (p=0.023 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 745.8µ ± 8% 736.8µ ± 12% ~ (p=0.739 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 15.37m ± 1% 10.60m ± 0% -31.03% (p=0.000 n=10) _MergeResponses-10 1186.9m ± 2% 149.8m ± 1% -87.38% (p=0.000 n=10) _UnwrapSeries-10 9.399m ± 1% 4.049m ± 0% -56.92% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 666.0m ± 3% 194.6m ± 0% -70.79% (p=0.000 n=10) geomean 18.87m 12.51m -33.70% │ before.log │ after.log │ │ B/op │ B/op vs base │ _CodecDecodeLogs-10 3.649Mi ± 0% 3.649Mi ± 0% ~ (p=0.364 n=10) _CodecDecodeSamples-10 18.12Mi ± 0% 18.12Mi ± 0% ~ (p=0.926 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 7.647Mi ± 0% 7.647Mi ± 0% ~ (p=0.587 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 27.94Mi ± 0% 16.99Mi ± 0% -39.18% (p=0.000 n=10) _MergeResponses-10 2.362Mi ± 0% 2.408Mi ± 0% +1.98% (p=0.000 n=10) _UnwrapSeries-10 19.495Mi ± 0% 8.595Mi ± 0% -55.91% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 772.3Mi ± 0% 772.3Mi ± 0% ~ (p=0.912 n=10) geomean 17.50Mi 14.54Mi -16.91% │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _CodecDecodeLogs-10 41.10k ± 0% 41.10k ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSamples-10 411.9k ± 0% 411.9k ± 0% ~ (p=1.000 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSeries/application/json;_charset=utf-8-10 304.2k ± 0% 298.1k ± 0% -2.01% (p=0.000 n=10) _MergeResponses-10 100.1k ± 0% 100.1k ± 0% -0.00% (p=0.002 n=10) _UnwrapSeries-10 201.1k ± 0% 198.0k ± 0% -1.54% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 203.1k ± 0% 203.1k ± 0% ~ (p=0.621 n=10) geomean 48.95k 48.70k -0.51% ¹ all samples are equal ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Travis Patterson	be71a80b15	Implement hooks to instrument query pipelines (#11493 ) This PR implements interfaces to inject new pipelines/extractors into the query path of queriers and ingesters.	2 years ago
Karsten Jeschkies	f67fff3eb2	Approximate `quantile_over_time` (#10417 ) What this PR does / why we need it: This change shards `quantile_over_time` queries using t-digest or DDSketch approximations. It can be enabled with `querier.shard_aggregations=quantile_over_time`. Outstanding - [x] Replace generic return type of `StepEvaluator` with interface `StepResult`. - [x] Send mapped query with quantile sketch expression from frontend to querier over the wire. - [x] Serialize sketches. See https://github.com/influxdata/tdigest/issues/34 - [x] Add feature flag. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Kaviraj Kanagaraj	bbfb13c3e4	Pin `chunk` and `index` format to `schema` version. (#10213 ) We pin all three `Chunk`, `HeadBlock` and `TSDB` Version to `schema` version in period config. This is the following mapping (after being discussed with @owen-d and @sandeepsukhani ) `v12` (current existing schema) - ChunkFormatV3 (UnorderedHeadBlock) + TSDBv2 `v13` (introducing new schema) - ChunkFormatV4 (UnorderedWithNonIndexedLabelsHeadBlockFmt) + TSDBv3 Note the new schema `v13` supports the latest chunk and index format. NOTES for Reviewer 1. General approach is we removed the idea of `index.LiveFormat`, `chunkenc.DefaultChunkFormat` and `chunkenc.DefaultHeadBlockFmt` and made following two changes. These variables were used before to tie chunk and tsdb formats specific to Loki versions. This PR remove that coupling and pin these formats to `schema` version instead. 1. These variables were replaced with explicit chunk and index formats within those packages (and it's tests) 2. If these variables were used outside it's own packages say by ingester, compactor, etc. Then we extract correct chunk and index versions from the `schema` config. 2. Add two methods to `periodConfig`. (1) `ChunkFormat()` returning chunk and head format tied to schema (2) `TSDBFormat()` returning tsdb format tied to schema. 2. Other ideas I thought of doing but didn't end up doing is make `ChunkFormat` and `IndexFormat` as separate type (rather than `byte` and `int` currently. Similar to `HeadBlockFmt` type). But didnt' do it eventually to keep the PR small and don't want to complicate with lots of changes. 4. Moved couple of test cases from `chunkenc` to `config` package, because the test case was actually testing methods on `schemaconfig` and it was creating cycling dependencies. --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Trevor Whitney	4d2a94ebd3	Return intersecting labels when aggregating volume by label (#10248 ) This PR changes the behavior the `index/volume` endpoint to return intersecting labels when requesting volume aggregating by label. Previously it only returned labels that matched selectors in the query. The old behavior is still possible using the `targetLabels` argument.	2 years ago
Trevor Whitney	a12311c5c8	Allow volume to be aggregated by label (#9988 ) Add a `aggregateBy` query parameter to the `series_volume` endpoint, and rename that endpoint to just `volume`. This allows users to get volumes aggregated into top level labels, rather label+value pairs. --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Trevor Whitney	9e19ff006c	Add targetLabels to SeriesVolume requests (#9878 ) Adds optional `targetLabels` parameter to `series_volume` and `series_volume_range` requests that controls how volumes are aggregated. When provided, volumes are aggregated into the intersections of the provided `targetLabels` only.	3 years ago
Travis Patterson	4da0f63789	Remove unused Value field (#9774 ) We didn't end up needing the `Value` field because we can express everything we need to as selectors	3 years ago
Trevor Whitney	dbc3040739	Convert SeriesVolume response to prometheus-style (#9703 ) Changes the response type of the label volume stats endpoint to return volumes as prometheus-style timeseries metrics. It currently only supports instant queries, but is a necessary step to eventually supporting range queries.	3 years ago
Dylan Guedes	c0d9a2b4ba	Log push failures per-entry instead of in batches (#9720 ) What this PR does / why we need it: Modify how we log push errors to do it per-entry instead of accumulating all errors and then logging them all in a single pass. Which issue(s) this PR fixes: N/A	3 years ago
Travis Patterson	db97058a84	Series volume endpoint (#9704 ) This changes the `label_volume` endpoint to the `series_volume` endpoint. The new endpoint still returns volumes but now it does it for the requested streams defined by the selector names passed rather than individual labels. All relevant non-requested labels are aggregated into the returned results ex: Assume we have the following streams: ``` {cluster="prod", team="A", component="foo"} {cluster="prod", team="B", component="foo"} {cluster="dev", team="A", component="foo"} {cluster="dev", team="B", component="foo"} ``` - requesting `{cluster="prod"}` returns one result for all streams containing `{cluster="prod"}` - requesting `{cluster=~".+"}` returns two results for the streams containing `{cluster="prod"}` and `{cluster="dev"}` - requesting `{cluster=~".+", team=".+"}` returns four results for the streams containing: ``` {cluster="prod", team="A"} {cluster="prod", team="B"} {cluster="dev", team="A"} {cluster="dev", team="B"} ``` --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	3 years ago
Periklis Tsirakidis	69392de8b6	Fix duplicate label values from ingester streams (#9629 )	3 years ago
Travis Patterson	6d8a184334	Revert "revert count unflused chunks (#9504 )" (#9615 ) Now that the bug where huge values is fixed and known not to be this. Let's readd it. Original: https://github.com/grafana/loki/pull/9479	3 years ago
Travis Patterson	a4ef06890b	revert count unflused chunks (#9504 ) revert this commit: `aeba51ab0f` We seem to be dramatically overcounting bytes now which is causing oversharding This is the same query before and after: Before: ~20 GB ![image (2)](https://github.com/grafana/loki/assets/1413241/6d0e3d4d-56ce-40aa-b57e-d2dd592eefb4) After: ~8 EB ![image (1)](https://github.com/grafana/loki/assets/1413241/f07cca10-0e32-467f-bf96-13a83eea8ce6)	3 years ago
Travis Patterson	aeba51ab0f	Count unflushed chunks in instance stats (#9479 ) The instance wasn't counting unflushed chunks in the stats. This change checks that the chunk's `flushed time` is 0 which means it hasn't been flushed yet.	3 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205 )	3 years ago
Travis Patterson	92fca94bb5	Revert "Update Ingester Rate Calculations (#7652 )" (#7991 ) This change was supposed to smooth out the per-second rate of streams so distributors could react more uniformly to streams with variable rates. In practice, it resulted in drastically more resource usage on distributors because `GetStreamRates` now responds with all streams on an ingester rather than only the streams seen in the last second. This change also didn't seem to affect sharding precision one way or another. If we'd still like to implement exponential smoothing for stream rates, it should be implemented in distributors to keep communication overhead to a minimum.	3 years ago
Travis Patterson	343e39f67b	Update Ingester Rate Calculations (#7652 ) - Use weighted moving average for rates - Keep Streams around until they are removed Previously, the rate calculator only kept the most recent second. This led to issues where bursty streams that didn't get pushes every second resulted in undersharding those streams	3 years ago
Travis Patterson	c75b822fc6	Track rates per stream and per tenant (#7564 ) It is possible for two tenants to have streams with the same labels. This PR changes the stream tracking in the `shard_tracker`, `stream_rate_calculator`, and `rate_store` to account for tenant ID when tracking streams.	3 years ago
Travis Patterson	45d59ba571	Improve rate store refresh latency (#7492 ) Rather than iterating streams and getting rates per stream, keep all necessary information in the rate calculator and just hand back the current rates when asked. This change took p99 latencies from ~1.8s to ~400ms and p50 latences from ~1s to 6ms.	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425 )	3 years ago
Travis Patterson	3e7adb2c34	Measure Per-Stream rates (#7370 ) This PR implements the ability for ingesters to measure per-stream rates at 1/s resolution. It also fixes bugs in the kind of Limits used in the RateStore and the Ingester API	3 years ago
Dylan Guedes	b1d4efab12	Loki: Per-tenant stream sharding (#7311 ) What this PR does / why we need it: - Move stream sharding configuration to its own package to avoid cyclic imports - Change stream sharding to be a per-tenant configuration - Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding - Change stream sharding flags prefix from `distributor.shard-stream` to `shard-stream`	3 years ago
Travis Patterson	e3e4e27c8c	Implement ingester API to return stream rates (#7176 ) This PR introduces an API on the Ingester that reports all current streams in memory and their current rate in bytes/second. The reponse from the API contains a slice of `StreamRate` structs: ```go type StreamRate struct { StreamHash uint64 `protobuf:"varint,1,opt,name=streamHash,proto3" json:"streamHash,omitempty"` StreamHashNoShard uint64 `protobuf:"varint,2,opt,name=streamHashNoShard,proto3" json:"streamHashNoShard,omitempty"` Rate int64 `protobuf:"varint,3,opt,name=rate,proto3" json:"rate,omitempty"` } ``` In the interest of space on the caller and in transmission, only hashes and rates are sent. - `StreamHash` will be used to deduplicate replicas - `StreamHashNoShard` will be used to aggregate a sharded stream's total rate. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Travis Patterson	bc16450349	Revert "Hide duplicates caused by stream sharding (#7005 )" (#7031 ) @owen-d had a suggestion that was dramatically dimpler. Revert this whole thing and the simplified PR will be forthcoming	3 years ago
Travis Patterson	a98b7ebaf4	Hide duplicates caused by stream sharding (#7005 ) As streams are sharded and resharded across different ingesters, the likelihood of duplicate logs being ingested increases dramatically. This removes duplicates at query time by dynamically removing the `__shard_stream__` label for both log and sample queries so the `stream hash` used by the `MergeEntryIterator` will match when doing comparisons for duplicates. The `stream hash` is created in either the `store` or `ingester instance` at query time so the labels are removed there. Additionally, the `MergeEntryIterator` has been modified to remove duplicates in a stream rather than just across streams. For completeness, `__shard_stream__` has also been removed from Label and Series Queries.	3 years ago
Mathew Heard	de418bf6ac	Reduce to a single code path through sendBatches regardless of limit (#6216 ) * Reduce to a single code path through sendBatches regardless of limit specified * simplify sendBatches by using -1 to specify no limit * test fix	4 years ago
Owen Diehl	4903df6ae5	Tsdb/inverted index wiring (#6252 ) * multi inverted index * delete in inverted index signature * multi inverted index * inverted index testing + skip nil indices * wires up period config aware multi-inverted-idx through ingesters * better inverted index validation * apply shipper defaults to last relevant period config, not current one. * more verbose error	4 years ago
Christian Simon	7f50a26656	Move ingester metrics around (#6275 )	4 years ago
Travis Patterson	5939d5e9d8	Query filtering in the ingester and storage (#5629 ) * Add timestamp to pipeline and extractor interfaces * implement filter extractor and pipeline * implement storage filtering * implement ingester filtering * test cleanup * Refactor pipeline setup to util * linter * enable filtering based on mode * review feedback * review comments * review feedback * no need to include label filters in the test pipeline * review feedpack	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539 ) * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Kaviraj Kanagaraj	2db309271c	Makes `tailer.droppedStreams` slice bounded. (#5334 ) * Makes `tailer.droppedStreams` slice bounded. For slow receiver, this dropped streams may cause ingester memory to keep increasing. Sometimes OOMing the ingester. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make droppedStreams slice size configurable Update tests Adding logging when slice is dropped Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for dropped stream Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281 ) * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
RangerCD	b26b73d5be	Remove instance level streams lock (#5179 ) * Feat(ingester): remove instance level streams lock * Fix lint * Solve Load and Delete consistency problem while pushing to ingester	4 years ago
Karsten Jeschkies	7bbed2be49	Define Ingester interface to enable wrapping. (#4988 )	4 years ago
Dylan Guedes	d5783e86a3	Copy `cortex/pkg/querier/astmapper` package dependency into Loki (#4982 ) * Fork cortex `pkg/querier/asmapper` into Loki. * Move away from cortex `pkg/querier/astmapper`. * Fix lint issues. - Use _ in argument of method `pkg/querier/astmapper/subtree_folder.go` since it is not being used - Remove unnecessary cast from `subtree_folder.go`	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830 ) * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Karsten Jeschkies	5ece938f3c	Run all benchmarks. (#4742 ) * Run all benchmarks. * Start debugging memchunk benchmark. * Do not iterate when closed. * Pass default tenant config. * Set default limits. * Fix two more benchmarks. * Enable cron trigger again.	4 years ago
Hiroki Sakamoto	0e21f02d76	Respect shard number in series api (#4577 ) * Respect shard number in series api * Refactor codes * remove duplicate logic in instance * format with goimport * Improve complexity of getting all fingerprints	4 years ago
Owen Diehl	20515a2402	better per stream rate limits configuration options (#4236 ) * better per stream rate limits configuration options * clean up test config * pr feedback	4 years ago
Owen Diehl	668622c813	Refactor per stream rate limit (#4213 ) * uses new StreamRateLimiter * sets burst before limiter, uses requested ts * creates new non infinite rate limiters when configs change * time/rate testware * less indirection & protect rate limit access from race conditions * removes old comment * always recreate stream limiter after config change	4 years ago
Callum Styan	757eeacb88	[ingester/stream]: Add a byte stream rate limit. (#4191 ) * Add a byte stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Track discarded lines due to stream rate limit + use ByteSize for the config value. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * Another round of review fixes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix goimports issue. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix duplicate flag registration + stream rate limit test. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix stream rate/burst limit flags yaml naming. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix tests now that we have stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix some comments Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago

1 2

77 Commits (4cae003ecedd474e4c15feab4ea2ef435afff83f)