1894 Commits (35510ba4eb3615da78b15934bf6355b43378f499)

Author	SHA1	Message	Date
Wu Zhiyuan	3bcc7ccd5a	use fifocache.duration instead of fifocache.interval in warn deprecated (#7609) ... **What this PR does / why we need it**: I configured fifo cache with `validity` option (which is [deprecated](b92f113cb0/docs/sources/configuration/_index.md?plain=1#L2004) in v2.6.0, `fifocache.duration` as an flag) and loki print that: ``` level=warn ts=2022-07-08T17:01:30.134251284Z caller=fifo_cache.go:125 msg="running with DEPRECATED flag fifocache.interval, use fifocache.ttl instead" cache=frontend.fifocache ``` However, there is no flag `fifocache.interval`. As #5148 changed, the deprecated flag should be `fifocache.duration`, not `fifocache.interval`.	3 years ago
Danny Kopping	d32f3f7b68	Querier: remove expensive & low-value debug log line (#7532) ... Removes a low-value debug log line that, in one recent example, can logs tens of thousands of times in a short span of time. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	3c792294d2	Loki: Fix index_set log message (#7538) ... As of now, this log message isn't correctly showing the list of files (instead it logs the value as "unsupported value type").	3 years ago
Travis Patterson	af68dbc6cf	Keep rates in the ratestore for 10 minutes (#7599) ... This PR accounts for intermittent streams that might get dropped on a per-second update interval Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Periklis Tsirakidis	e0a7b28a61	Add single compactor http client for delete and gennumber clients (#7453)	3 years ago
Cyril Tovena	6aa85e318c	Improves performance of json responses (#7522) ... **What this PR does / why we need it**: This allows to reuse stream memory allocations between responses. Related https://github.com/grafana/loki/pull/3163 but this time this is for the encoding. ``` ❯ benchstat before.txt after.txt name old time/op new time/op delta _Encode-16 29.2µs ±12% 25.2µs ± 1% -13.85% (p=0.016 n=5+4) name old alloc/op new alloc/op delta _Encode-16 24.9kB ± 6% 16.4kB ± 8% -34.20% (p=0.008 n=5+5) name old allocs/op new allocs/op delta _Encode-16 145 ± 0% 129 ± 0% -11.03% (p=0.008 n=5+5) ```	3 years ago
Travis Patterson	198c81fc7b	Reduce number of locks and partition by tenant and stream hash (#7583) ... This PR reduces the number of locks in the rate calculator and allows it to hold an arbitrary number of rates by partitioning by tenant and stream hash Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	020631ebac	add user-id transformer for logs results cache (#7581) ... **What this PR does / why we need it**: Adding transformer to results cache same as metrics cache done in PR #7542	3 years ago
Travis Patterson	af899dcc00	Retry pr (#7579) ... This is the same PR as https://github.com/grafana/loki/pull/7152 but I botched the merge when updating the code. This one works Co-authored-by: Arseniy Antonov <arseny.antonov@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Travis Patterson	c75b822fc6	Track rates per stream and per tenant (#7564) ... It is possible for two tenants to have streams with the same labels. This PR changes the stream tracking in the `shard_tracker`, `stream_rate_calculator`, and `rate_store` to account for tenant ID when tracking streams.	3 years ago
Kaviraj Kanagaraj	ceb09efdf8	fix(loki-canary): Send to Loki after updating `totalEntries`. (#7211) ... **What this PR does / why we need it**: Fixes: #7142 (take a look for more details about the problem there) After trying few things to avoid this race and discussin it with Ed, This small changes seems to be more accurate and seems to fix the issue. This is unblocking rolling out new canary in our internal cells Update: For some reason I thought the fix didn't fix the original issue when I tested it with internal loki dev cell, almost a month ago. But after testing it for more than 2hr on same cell. I realized it is working fine (will leave it for a day like this just to confirm) Verified following: ``` {.... container="loki-canary"} |= "websocket missing" ``` and noticed it's happening no longer after the fix. Also verified the metrics ``` sum(increase(loki_canary_websocket_missing_entries_total{...}[$__range])) ``` It dropped to zero.   Update 2: After running it for whole day, looks there was two log entries missed on the websocket. My guess is since websocket is long-live connection and something can interrupt the connection loosing this message.  I also quickly checked how often this happens on other bigger envs (say ops) and looks like it's not that uncommon.  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/7142 **Special notes for your reviewer**: Tested it on internal loki dev cells **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	fd3a6dfaf1	Adds extra docs to flehs out ParseLabels implementation (#7562) ... Adding some extra comments for archaeological purposes	3 years ago
Travis Patterson	a65da7d78e	Shard streams round-robin rather in batches. (#7560) ... To avoid hot spots, keep track of shard numbers across pushes. When tracking shard numbers across pushes, it's no longer necessary to randomize shards, in the case where the desired number of shards is greater than the number of entries. Instead, we just partition the entries among the shards in round robin fashion. Over time, this should give us a more even distribution of logs in shards, particularly for large streams that are made of small individual pushes.	3 years ago
Michel Hollands	16761723f4	Add way to override userId for caching (#7542) ... **What this PR does / why we need it**: Add a way to change the userId used in caching. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Owen Diehl	6b4563cd53	Prevent IndexShipper from leaking mutex accesses. (#7548) ... # What Inverts control in the `index-shipper` so consumers don't manage accesses to mutexes. This prevents deadlocks and no longer requires consumers have internal knowledge of the `index-shipper` itself. This PR reverts a few previous attempts. Look at the last commit to see the net-additions. # Background: TSDB previously didn't use mmap, which meant we could freely pass around TSDBs as `[]byte`. Since they were immutable, we didn't need to know when an underlying index was cleaned up/closed; this coordination was unnecessary and the garbage collector would clean up dangling memory for us. After introducing TSDB, we needed to control access to avoid use-after-free bugs. A few attempts at controlling this became increasingly complex & error prone. Instead of requiring _consumers_ to prevent deadlocks on index-shipper internals, it makes more sense that references to the index-shipper's resources should not leak outside the `ForEach` functions that give temporary access to them. This _inverts control_ from the consumer (code which queries the indices) to the producer (the index-shipper). Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Travis Patterson	0a654e2357	Ensure labels on sharded streams are sorted before they're sent anywhere (#7556) ... Pushing streams with unsorted labels to ingesters results in a bug where ingesters duplicate chunks for streams. This causes a memory leak and eventual OOM crash when ingesters recover from the WAL because one of the duplicate streams is orphaned. This PR ensures that sharded streams have sorted labels to avoid this bug.	3 years ago
Callum Styan	46b40a5582	Hardcode ring.WriteNoExtend for distributor push operations (#7517) ... With zone awareness enabled we allow restarting multiple ingesters within a single zone at the same time, this means that for many more write requests there's potentially only 2 of the 3 replicas for a stream up at the time the write request is received. This lead to a large increase in 500s during rollouts. My suspicion is that the 500s on rollouts was always related to the following; the replica set ring code has something referred to as "extending the replica set". This essentially is the adding of ingesters in the ring that are not in an ACTIVE state (for example, are in LEAVING) to the set of replicas considered valid for an operation. Adding of the instance in a state other than ACTIVE can be controlled by using a different operation type, which Mimir was using but we were not. Here are the relevant functions, [ring.Get](8d6d914ef6/ring/ring.go (L335-L403)) (particularly these lines [1](8d6d914ef6/ring/ring.go (L381-L389)) and [2](8d6d914ef6/ring/ring.go (L394-L397))) and [replicationStrategy.Filter](789ec0ca4a/ring/replication_strategy.go (L22-L71)) (particularly these lines [1](789ec0ca4a/ring/replication_strategy.go (L32-L36)) and [2](789ec0ca4a/ring/replication_strategy.go (L54-L67))) The tl; dr of the above code snippets is that we seem to always have required a minSuccess from replicationFactor/2 + 1 except we overwrite replicationFactor to be the # of instances which is likely always going to be more than just 3 ingester replicas because of the replica set extension. At least for zone aware rollouts this seems to be the case. This is because many ingesters will be in the LEAVING state. We can avoid this by just using `WriteNoExtend` operations instead of `Write` operations ([here](789ec0ca4a/ring/ring.go (L83-L93))) In this PR I've hardcoded our call to the ring in `distributor.Push` to always use `ring.WriteNoExtend`. IMO this is the better option to allowing `ring.WriteNoExtend` to be used optionally and defaulting to `ring.Write`. Extending the replication set on writes feels like a footgun for write outages and there's even some feelings internally that allowing that extension is a bug. For more background information see: https://github.com/grafana/mimir/issues/1854 Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Cyril Tovena	7b01864a91	Improves performance of distributor validations (#7534) ... Most of the time we don't need to create those strings which create an alloc per entry.	3 years ago
Ed Welch	45caba4459	Loki: Remove the bypass for "limited" queries (#7510) ... **What this PR does / why we need it**: Limited queries are queries which don't have a filter expression. Now all log queries will be handled by the LogFilterTripperware which will result in them being split by time (they previously were not) Signed-off-by: Edward Welch <edward.welch@grafana.com> **Which issue(s) this PR fixes**: We've found that very large timeframe `limited` queries will be sent to queriers and then ingesters and can stall out the read path because of the large time ranges. Splitting these by time avoids this problem by keeping any subquery limited in length to the `split_by_interval`. It's important to note that this will likely increase some extra work done by limited queries as more work for these will be parallelized and can result in extra data being processed. This will be the same as how filter queries are handled now, so it will be no worse than that. In fact this was an optimization based on the premise that it's advantageous not to split/shard limited queries but we are seeing this not be the case when limited queries are run over very large time ranges matching streams with huge volumes and combined with parsers like `| json` **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Sandeep Sukhani	bb2927172a	tsdb post compaction cleanup (#7502) ... **What this PR does / why we need it**: Previously when we were reading the whole tsdb index in memory, there was not much cleanup required after we were done with the index. Now that we are mmaping the tsdb index, we need to close the mmap. I had missed calling `index.Close()` on the compacted-away multi-tenant index. This causes the compactor to retain the files on disk, leading to the compactor running away from disk space. This PR takes care of it. While doing the changes, I noticed that some of the tests failed due to `LabelValues` tsdb implementation returning strings valid only until the underlying byte slice stays valid. Since the underlying byte slice could become invalid when we close the tsdb, we should return a copy instead of yolostring. This can also happen on the query path when a sync closes the index before we get to return the response. This PR also fixes that.	3 years ago
Ed Welch	23cc448881	Loki: log when a query starts in the logql engine (#7469) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: There are cases where a query cannot finish (OOM of a querier/ruler) and most of our logs are emitted upon completion of a query. This can help us find queries which are OOM crashing a component. Note: It only covers queries executed by the engine, so only instant and range queries. Another PR will be necessary to cover labels/series endpoints. However this approach seemed the best to be able to get this information from ruler executed queries. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	70f6bf2e45	Index gw lock bugfix (#7497) ... While I'm still in favor of a longer refactor to address the complexity described in https://github.com/grafana/loki/pull/7490, I'm opening this PR as a shorter, more immediate alternative that should fix the deadlocks. This also reduces the number of total lock calls in many cases as we lock to act over entire groups rather than individual items.	3 years ago
Travis Patterson	45d59ba571	Improve rate store refresh latency (#7492) ... Rather than iterating streams and getting rates per stream, keep all necessary information in the rate calculator and just hand back the current rates when asked. This change took p99 latencies from ~1.8s to ~400ms and p50 latences from ~1s to 6ms.	3 years ago
Travis Patterson	b4ed595461	Remove groupcache (#7126) ... As discussed during the caching retro, this PR removes the experimental distributed cache based on Mailgun's fork of groupcache. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	da5378c927	Loki: Remove RatelimitedStream proto (#7235) ... **What this PR does / why we need it**: Remove an unused proto (RateLimitedStream). It was only used for an older approach to deal with rate-limited streams but since we changed approaches this isn't necessary anymore.	3 years ago
Mohamed-Amine Bouqsimi	7debe7c235	Ruler: enable Loki to configure Alertmanagers per tenant (#7359)	3 years ago
Dylan Guedes	28e192f22e	Loki: Implement random entries assignment when sharding (#7427) ... **What this PR does / why we need it**: Modify sharding to assign random shards when the number of shards is bigger than the number of entries. This helps us to avoid hot-spotting the first shards. By randomizing the entries assignment we help spread the load between all shards.	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414) ... We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Periklis Tsirakidis	4176de17d1	Add compactor delete client tls client config (#7426)	3 years ago
Periklis Tsirakidis	26d2989086	Expose ring and memberlist handlers through internal server listener (#7436)	3 years ago
Travis Patterson	c1bccac141	Results cache fix improvements (#7444) ... Move middleware to someplace more sensible and incorporate feedback missed in the review for this.	3 years ago
Owen Diehl	b59277b472	Revert "Revert "Update Thanos dependency to latest version"" (#7372) ... Reverts grafana/loki#7371 PR re re-enable Thanos dependency bump.	3 years ago
Travis Patterson	f2297d3d2a	Fix result cache misses on sharded queries (#7429) ... Headers weren't being propagated from query responses on sharded queries causing result cache misses. This PR: - Adds headers to `logproto.Result` so we can save response headers as they come back from queries - Metric-query results are turned into numbers well away from where responses are sent to the frontend: save all headers in the context so they can be pulled out when the final result is calculated - Propagate headers in the final responses	3 years ago
李国忠	8dcc4b3ec8	Do not log 'unable to read rules directory' at startup if directory hasn't been created yet (#7434)	3 years ago
Periklis Tsirakidis	9c8a827576	Fix enable of internal server for modules (#7423) ... **What this PR does / why we need it**: Makes adding the internal server modules to the module map more robust. The previous implementation accidentally dropped other modules. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	435c055488	log panic full stack in boltdb recover block (#7410)	3 years ago
Travis Patterson	8c84ee8b7a	Fix stream sharding (#7393) ... Promtail doesn't add a stream hash to push requests. As a result, we weren't accounting for the current rate in our sharding calculation and only sharding on individual large pushes. This PR ensures that streams have a hash as they're ingested. It also add a couple of metrics that were useful in debugging this issue.	3 years ago
Danny Kopping	33bf36aa21	Promtail: flush logs on exit (#7400) ... Promtail is using the line-buffered logger (https://github.com/grafana/loki/pull/6954), and log messages were not being printed on exit	3 years ago
Travis Patterson	543ea78b7a	Add metrics to the distributor about stream sharding (#7390) ... This PR adds some more metrics to the steam sharding interactions from the Distributor. 1. `rate_store_stream_count`: The total number of streams 2. `rate_store_max_stream_shard_count`: The stream with the largest number of shards 3. `rate_store_max_stream_rate_bytes`: The largest stream seen by a distributor 4. `rate_store_refresh_duration_seconds`: A histogram of the latency to query all the ingesters and process their rates	3 years ago
Travis Patterson	5dc27e8064	Fix Flaky Distributor Test (#7391) ... Use eventually for async tests	3 years ago
Travis Patterson	3e7adb2c34	Measure Per-Stream rates (#7370) ... This PR implements the ability for ingesters to measure per-stream rates at 1/s resolution. It also fixes bugs in the kind of Limits used in the RateStore and the Ingester API	3 years ago
Dennis Szczepanski	4e61c60296	Loki: Renames metric 'loki_log_messages_total' to 'loki_internal_log_messages_total' (#7361) ... **What this PR does / why we need it**: Renames the metric 'loki_log_messages_total' to a more descriptive name. Keeps and deprecates the previously existing metric to not break functionality. **Which issue(s) this PR fixes**: Fixes #7088 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Bryan Boreham	74d206bf28	chunks: improve readability of compression benchmarks (#7246) ... **What this PR does / why we need it**: ### BenchmarkWrite Use Go runtime facilities to report MB/s for each operation and compression ratio. Before: ``` BenchmarkWrite/ordered-none-4 1018 1104885 ns/op 1431962 B/op 33 allocs/op BenchmarkWrite/ordered-gzip-4 12 87333201 ns/op 3516699 B/op 696 allocs/op BenchmarkWrite/ordered-lz4-64k-4 51 21483048 ns/op 3095117 B/op 649 allocs/op ``` After: ``` BenchmarkWrite/ordered-none-4 841 1202007 ns/op 1054353.53 MB/s 101.9 %compressed 1569953 B/op 34 allocs/op BenchmarkWrite/ordered-gzip-4 13 86299699 ns/op 3357.25 MB/s 6.891 %compressed 3778896 B/op 702 allocs/op BenchmarkWrite/ordered-lz4-64k-4 49 22332214 ns/op 34107.61 MB/s 9.880 %compressed 3407522 B/op 661 allocs/op ``` Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding. Before: `4724260 maxresident KB`; after: `375804 maxresident KB` ### BenchmarkRead Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations. Add stats to the context for decoding, otherwise a new stats object is created each time round the loop. Re-order so all unsampled results come before all sampled results. Before: ``` BenchmarkRead/none_66_kB-4 278 4555157 ns/op 131744 B/op 717 allocs/op BenchmarkRead/sample_none_66_kB-4 142 8238861 ns/op 129771 B/op 717 allocs/op BenchmarkRead/gzip_66_kB-4 6 179445187 ns/op 2100498 B/op 15010 allocs/op BenchmarkRead/sample_gzip_66_kB-4 4 261040057 ns/op 2076030 B/op 15028 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 62855240 ns/op 1722979 B/op 14219 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 115577796 ns/op 1706040 B/op 14220 allocs/op BenchmarkRead/lz4-256k_66_kB-4 18 66144317 ns/op 1880851 B/op 15078 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 118405993 ns/op 1800396 B/op 15077 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 67832189 ns/op 1821806 B/op 15076 allocs/op ... none_524 kB: 1828.92 MB/s none_524 kB: 1809.33 MB/s none_262 kB: 1790.79 MB/s none_66 kB: 1673.84 MB/s ... gzip_66 kB: 647.35 MB/s ... ``` After: ``` BenchmarkRead/none_66_kB-4 286 4377585 ns/op 1725.23 MB/s 41896 B/op 600 allocs/op BenchmarkRead/gzip_66_kB-4 6 180770638 ns/op 655.82 MB/s 723426 B/op 13265 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 65380524 ns/op 1296.10 MB/s 727571 B/op 12925 allocs/op BenchmarkRead/lz4-256k_66_kB-4 16 63737292 ns/op 1409.75 MB/s 786994 B/op 13707 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 65963825 ns/op 1362.17 MB/s 770441 B/op 13707 allocs/op BenchmarkRead/lz4_66_kB-4 18 64955415 ns/op 1383.31 MB/s 1003277 B/op 13707 allocs/op BenchmarkRead/snappy_66_kB-4 20 51588959 ns/op 1273.38 MB/s 361872 B/op 5020 allocs/op BenchmarkRead/flate_66_kB-4 6 172147502 ns/op 691.74 MB/s 715706 B/op 13242 allocs/op BenchmarkRead/zstd_66_kB-4 3 359473273 ns/op 390.07 MB/s 427247248 B/op 18551 allocs/op ... BenchmarkRead/sample_none_66_kB-4 135 8533283 ns/op 885.04 MB/s 39866 B/op 600 allocs/op BenchmarkRead/sample_gzip_66_kB-4 5 243016949 ns/op 487.84 MB/s 686358 B/op 13212 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 114330032 ns/op 741.18 MB/s 707104 B/op 12926 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 117294928 ns/op 766.05 MB/s 778461 B/op 13709 allocs/op ... ``` Note "MB" in previous code was 2^^20 while Go uses 10^6. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - not user-facing - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	18da26e668	use a channel to signal completion of index read operation for tsdb and boltdb shipper (#7381) ... **What this PR does / why we need it**: In PR #7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated	3 years ago
Owen Diehl	8331785a23	Revert "Update Thanos dependency to latest version" (#7371) ... Reverts grafana/loki#7348 I expect we'll merge this again in the future, but we'll need some more time for downstream projects to be ready for this change.	3 years ago
Sandeep Sukhani	bbe68802d1	lock boltdb and tsdb index while queries are in process (#7367) ... **What this PR does / why we need it**: We noticed ingesters and index-gateways panicking when using `tsdb` as an index store with the latest build. This is caused by closing `mmap` while the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index. **Checklist** - [x] Tests updated	3 years ago
Owen Diehl	f4e2cf1993	TSDB explicit fingerprint (#7362) ... This PR does two things: * Includes the series fingerprint in TSDB WALs. Doing this ensures we recover fingerprint used in chunk addresses after restart even if loki is upgraded and the label hashing algorithm changes. * Removes much of the extra hashing in the TSDB write path + manager. Occasionally, the fingerprint may be different than the hash of labels. Historically this has happened in a few places: * When the `__name__="logs"` label was injected for non-tsdb indices (not applicable to TSDB) * When building a multitenant tsdb index and adding a synthetic `__loki_tenant__` label * In a previous bug(https://github.com/grafana/loki/pull/7355) when empty label values altered calculated fingerprints. Explicitly storing/retrieving the desired fingerprint throughout helps avoid calculating it in error.	3 years ago
Travis Patterson	7979cfbe07	Implement Distributor Rate Store (#7349) ... The `RateStore` concurrently queries all ingester `GetStreamRate` APIs and stores the results. `RateStore` deduplicates replicas and combines sharded streams so a caller of `RateFor` gets an unsharded view of a stream's rate.	3 years ago
Travis Patterson	e89c30ce6b	Fix Flaky Integration Tests (#7360) ... Allow the OS kernel to decide what's an open port instead of trying to guess	3 years ago