105 Commits (ca19e8d7941bfac86355b2853ef2ea110f32075e)

Author	SHA1	Message	Date
Bryan Boreham	f89a51fde8	Tracing: elide small traces for Stats call (#10308) ... **What this PR does / why we need it**: At work I see a lot of traces getting data dropped due to being tens of megabytes in size. Looking at what was captured, I see many many repeats of these four spans: <img width="1070" alt="image" src="https://github.com/grafana/loki/assets/8125524/4268e664-6910-49bb-8f68-69b139a5a66b"> Events on the parent trace are cheaper and, for small operations like this, just as useful. Where there was an event line I added a `function` field matching the previous span name. For `storeEntry.Stats()` I didn't add an event line as it doesn't seem to do very much. But please correct me; I'm not hugely familiar with the Loki codebase. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Trevor Whitney	4d2a94ebd3	Return intersecting labels when aggregating volume by label (#10248) ... This PR changes the behavior the `index/volume` endpoint to return intersecting labels when requesting volume aggregating by label. Previously it only returned labels that matched selectors in the query. The old behavior is still possible using the `targetLabels` argument.	3 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	3 years ago
Trevor Whitney	39be4b5e6b	Only aggregate labels or series based on requested (#10076) ... This improves the logic when calculating volumes by only aggregating series or labels based on which was requested. Previously we aggregated both, and then picked the correct set for the response, causing unnecessary additional memory consumption.	3 years ago
Trevor Whitney	a12311c5c8	Allow volume to be aggregated by label (#9988) ... Add a `aggregateBy` query parameter to the `series_volume` endpoint, and rename that endpoint to just `volume`. This allows users to get volumes aggregated into top level labels, rather label+value pairs. --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Trevor Whitney	9e19ff006c	Add targetLabels to SeriesVolume requests (#9878) ... Adds optional `targetLabels` parameter to `series_volume` and `series_volume_range` requests that controls how volumes are aggregated. When provided, volumes are aggregated into the intersections of the provided `targetLabels` only.	3 years ago
Travis Patterson	cb2b3ca9ea	Add limit to the series volume endpoint (#9833) ... To protect index gateways, this PR adds a per-tenant limit to the `series_volume` endpoint so the accumulation of series can't grow without bound.	3 years ago
Trevor Whitney	d4cfbaac4f	Implement series volume range queries (#9812) ... This PR adds a `series_volume_range` endpoint which allows for series volume queries over time with a specified step, returning timeseries data in the form of a Prometheus matrix response. The existing `series_volume` endpoint still returns Prometheus vector responses, and hardcodes the step to 0.	3 years ago
Travis Patterson	8ca035ffbf	Log Volume: Do the prometheus-format conversion in dedicated middleware (#9776) ... This PR moves the Prometheus-format conversion to it's own middleware. This keeps it all in one place and consolidates logic that was in many places. It also sets us up to make range queries as a next step.	3 years ago
Travis Patterson	4da0f63789	Remove unused Value field (#9774) ... We didn't end up needing the `Value` field because we can express everything we need to as selectors	3 years ago
Trevor Whitney	dbc3040739	Convert SeriesVolume response to prometheus-style (#9703) ... Changes the response type of the label volume stats endpoint to return volumes as prometheus-style timeseries metrics. It currently only supports instant queries, but is a necessary step to eventually supporting range queries.	3 years ago
Trevor Whitney	d74c67a9d0	Add version to TSDB index builder struct (#9566) ... Add version as a property to the TSDB Index Builder, forcing users of the builder to specify a version for each builder instance. This is the first of many PRs that will add the TSDB index version to the schema version config.	3 years ago
Dylan Guedes	24e9efb907	Loki: Report event for `GetChunkRef` instead of SpanLogging (#9748) ... **What this PR does / why we need it**: Modify our calls to `index_client.GetChunkRef` to report invocations through a span event instead of through span logging. This way we won't log a new log line for every GetChunkRef call. **Which issue(s) this PR fixes**: N/A	3 years ago
Travis Patterson	db97058a84	Series volume endpoint (#9704) ... This changes the `label_volume` endpoint to the `series_volume` endpoint. The new endpoint still returns volumes but now it does it for the requested streams defined by the selector names passed rather than individual labels. All relevant non-requested labels are aggregated into the returned results ex: Assume we have the following streams: ``` {cluster="prod", team="A", component="foo"} {cluster="prod", team="B", component="foo"} {cluster="dev", team="A", component="foo"} {cluster="dev", team="B", component="foo"} ``` - requesting `{cluster="prod"}` returns one result for all streams containing `{cluster="prod"}` - requesting `{cluster=~".+"}` returns two results for the streams containing `{cluster="prod"}` and `{cluster="dev"}` - requesting `{cluster=~".+", team=".+"}` returns four results for the streams containing: ``` {cluster="prod", team="A"} {cluster="prod", team="B"} {cluster="dev", team="A"} {cluster="dev", team="B"} ``` --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	3 years ago
Kaviraj Kanagaraj	57f9452231	Revert 9217 chaudum/tsdb chunkrefs pool (#9685) ... Revert #9217 (potential bug in query result) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Salva Corts	87a659a6db	Add span events for index stats and result cache (#9552) ... **What this PR does / why we need it**: This PR adds events to the traces to have some extra observability for how we compute the index stats. We also add some trace events to the results cache.      **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Christian Haudum	b4d0a2cd68	Force using a buffer from the ChunkRefsPool (#9217) ... **What this PR does / why we need it**: Re-using a buffer from the `ChunkRefsPool` avoids unnecessary allocations in TSDB's `GetChunkRefs` call. **Special notes for your reviewer**: Benchmarks ``` $ gotest -v -count=1 ./pkg/storage/stores/tsdb/ -test.run="^$" -test.bench=Bench -test.benchtime=10000x ``` ```console $ benchstat old.txt new.txt name old time/op new time/op delta TenantHeads/10-8 244µs ±58% 55µs ± 4% -77.49% (p=0.000 n=10+9) TenantHeads/100-8 1.75ms ± 9% 0.58ms ±26% -66.89% (p=0.000 n=9+10) TenantHeads/1000-8 16.5ms ± 6% 6.4ms ± 2% -60.89% (p=0.000 n=9+10) IndexClient_Stats-8 25.0µs ±10% 25.1µs ±13% ~ (p=0.863 n=9+9) TSDBIndex_GetChunkRefs-8 3.34ms ±12% 1.46ms ± 2% -56.41% (p=0.000 n=10+9) name old alloc/op new alloc/op delta IndexClient_Stats-8 2.37kB ±12% 2.37kB ±12% ~ (p=0.923 n=10+10) TSDBIndex_GetChunkRefs-8 6.40MB ± 0% 1.40MB ± 0% -78.08% (p=0.000 n=9+10) name old allocs/op new allocs/op delta IndexClient_Stats-8 59.0 ± 0% 59.0 ± 0% ~ (all equal) TSDBIndex_GetChunkRefs-8 42.0 ± 0% 27.0 ± 0% -35.71% (p=0.000 n=10+10) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Owen Diehl	64c6f6b8cc	Better combines stats in ingesters (#9474) ... * Reduces double counting streams when possible * proportionally add stats based on chunk overlap range * code reuse	3 years ago
Owen Diehl	8bd3581b95	Revert "Revert "start writing tsdb v3"" (#9332) ... Reverts grafana/loki#9301 Should be good to start writing v3 again after https://github.com/grafana/loki/pull/9328 is merged and when we're ready to start using tsdbv3 as the default	3 years ago
Owen Diehl	f78ceec941	fixes bug during chunk iteration using markers (#9328) ... Tricky one to find, but this does the following * correctly subtracts remaining chunk count when skipping a page (this was previously using a zero-valued marker) * correctly set previous maxt on pages that are skipped. This is important because chunks themselves are delta-encoded against the previous chunk. * removes the markers pool from the chunk decoding which isn't needed. * adds a test * adds a bit more logging on which series resulted in a decoding error during chunk iteration	3 years ago
Owen Diehl	cc6f6b9171	Revert "start writing tsdb v3" (#9301) ... Reverts grafana/loki#9276 Update main to prevent writing v3 by default while I find a bug in it.	3 years ago
Owen Diehl	08af8849ee	tsdb migrate versions (#9275) ... Update of https://github.com/grafana/loki/pull/9198 while @sandeepsukhani is on holiday :) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Owen Diehl	03531f5013	start writing tsdb v3 (#9276)	3 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	3 years ago
Owen Diehl	ec516facaf	removes 20 user test from compaction; seems redundant and hits timeouts (#9201) ... just a little cleanup to make tests run faster -- the 20 user test doesn't seem to test any additional surface area and takes a long time.	3 years ago
Owen Diehl	dbbadafca2	Tsdb/v3 (#9070) ... brief: adds page support for chunks within a series. This lets us do a few optimizations, most notably skipping chunk ranges that don't overlap with our query bounds. It also allows us to use aggregated stats for pages when computing `Stats` calls that completely overlap all chunks in a page. teaser: ``` pkg: github.com/grafana/loki/pkg/storage/stores/tsdb/index BenchmarkChunkStats/version_2/2_chunks-10 10121240 109.4 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/2_chunks-10 27591069 43.51 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/4_chunks-10 9410245 128.7 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/4_chunks-10 21808070 54.44 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/10_chunks-10 6048235 201.3 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/10_chunks-10 9517605 125.1 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/100_chunks-10 1000000 1081 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/100_chunks-10 1669972 715.4 ns/op 528 B/op 3 allocs/op BenchmarkChunkStats/version_2/1000_chunks-10 118125 10165 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/1000_chunks-10 570576 2125 ns/op 4816 B/op 6 allocs/op BenchmarkChunkStats/version_2/10000_chunks-10 10000 117447 ns/op 123014 B/op 3 allocs/op BenchmarkChunkStats/version_3/10000_chunks-10 74524 16225 ns/op 48601 B/op 9 allocs/op BenchmarkChunkStats/version_2/100000_chunks-10 842 1240380 ns/op 3211679 B/op 13 allocs/op BenchmarkChunkStats/version_3/100000_chunks-10 8494 141169 ns/op 516530 B/op 13 allocs/op ``` ``` pkg: github.com/grafana/loki/pkg/storage/stores/tsdb/index BenchmarkReadChunks/version_2/2_chunks-10 13673050 80.52 ns/op 172 B/op 0 allocs/op BenchmarkReadChunks/version_3/2_chunks-10 25713412 64.06 ns/op 201 B/op 0 allocs/op BenchmarkReadChunks/version_2/4_chunks-10 12302040 96.56 ns/op 373 B/op 0 allocs/op BenchmarkReadChunks/version_3/4_chunks-10 20512030 84.93 ns/op 346 B/op 0 allocs/op BenchmarkReadChunks/version_2/10_chunks-10 6974947 168.7 ns/op 490 B/op 0 allocs/op BenchmarkReadChunks/version_3/10_chunks-10 7436360 161.0 ns/op 632 B/op 0 allocs/op BenchmarkReadChunks/version_2/50_chunks-10 2038724 599.3 ns/op 2034 B/op 0 allocs/op BenchmarkReadChunks/version_3/50_chunks-10 2251822 619.8 ns/op 1926 B/op 0 allocs/op BenchmarkReadChunks/version_2/100_chunks-10 946430 1092 ns/op 4132 B/op 0 allocs/op BenchmarkReadChunks/version_3/100_chunks-10 1467297 1000 ns/op 3308 B/op 1 allocs/op BenchmarkReadChunks/version_2/150_chunks-10 674575 1721 ns/op 5767 B/op 0 allocs/op BenchmarkReadChunks/version_3/150_chunks-10 1000000 1373 ns/op 6171 B/op 1 allocs/op BenchmarkReadChunks/version_2/1000_chunks-10 109153 10554 ns/op 33063 B/op 0 allocs/op BenchmarkReadChunks/version_3/1000_chunks-10 158086 6963 ns/op 32829 B/op 1 allocs/op BenchmarkReadChunks/version_2/10000_chunks-10 9043 114121 ns/op 345258 B/op 0 allocs/op BenchmarkReadChunks/version_3/10000_chunks-10 22083 79896 ns/op 435493 B/op 1 allocs/op BenchmarkReadChunks/version_2/100000_chunks-10 1039 1103621 ns/op 3479873 B/op 0 allocs/op BenchmarkReadChunks/version_3/100000_chunks-10 2101 1155419 ns/op 4680266 B/op 1 allocs/op ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Salva Corts	fd16425062	Cache index stats requests (#8972) ... **What this PR does / why we need it**: As described in https://github.com/grafana/loki/issues/8973, we are substantially increasing the load of index stat requests we sent to our index gateways. Many of these requests should be easily re-used by caching them. This PR adds caching for index stat requests by reusing the results cache. Here's a demo ([source][1]):  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8973 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://ops.grafana-ops.net/d/afcaef21-e5ad-49e7-ab06-42a9d7d915eb/index-stats?orgId=1&var-datasource=dev-cortex&var-cluster=dev-eu-west-2&var-namespace=loki-dev-009&var-loki_datasource=Grafana%20Logging&from=1680259907288&to=1680260431814&var-operation=All --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	dc9ca8e19a	Tools/index analyzer (#9074) ... Adds a tool that allows downloading and querying statistics across TSDB indices. Example output: ``` analyzed 1046302 series and 7607495 chunks for an average of 7.270841 chunks per series. max chunks/series was 45695. number of series with over 1k chunks: 15 ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](ff7b462973/pkg/storage/factory.go (L188)) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Salva Corts	f9d9b471f8	Fix comment in `buildChunkMetas ` test function (#9124) ... wrt https://github.com/grafana/loki/pull/9096#discussion_r1164555980	3 years ago
Salva Corts	9783757932	Proportional TSDB index stats for chunks partially fitting in queried time range. (#9096) ... **What this PR does / why we need it**: When computing index stats for a query, we sum up the stats from all the chunks the query would read, even if some chunks will only be read partially. Moreover, if the query spans across two different TSDB tables, and there are chunks duplicated on both tables (at the end of the first one and at the beginning of the next one), the chunks' stats will be counted twice. This PR improves how we compute stats by taking care of partially matching chunks. Assuming entries and bytes are evenly distributed in the chunk, we will take the proportional number of entries and the number of bytes that fit on the span. For example, a chunk spans from t1 to t10 and has 10 entries and 10 KBs. A query from t3 through t7 would hit 50% of the chunk time span. Therefore, we estimate that the query would load 5 KBs and 5 entries from the chunk. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Salva Corts	fd5d78784f	Estimate chunk size on compaction looking at data uncompressed size (#9099) ... **What this PR does / why we need it**: At `compactedIndex.ToIndexFile` we were not computing the size of the chunk correctly. We were looking at the size of the chunk ref. Since the size of the chunk ref is always a bunk of bytes, the chunk meta KB field would always be 0. 4e893a0a88/pkg/storage/stores/tsdb/compactor.go (L367) Instead, we should get the approx KBs by looking at the chunks data uncompressed size. As we do at `store.IndexChunk`: d405162516/pkg/storage/stores/tsdb/store.go (L189)	3 years ago
Owen Diehl	363a85226e	makes test pass for 0 sharding factor (equivalent to no sharding) (#9065) ... Saw validation tests failing due to this on `main`; not sure how this hasn't borked our builds but this ensures we don't error on shard-factors of zero (equivalent to not sharding). This doesn't actually hit our code paths because we explicitly shortcut factor=0 in the shardmapper.	3 years ago
Ed Welch	7264e0ef53	Loki: Remove global singleton of the tsdb.Store and keep it scoped to the storage.Store (#8928) ... **What this PR does / why we need it**: I made this change to fix an issue with the cmd/migrate tool which I'm also sneaking a few fixes on in this PR too. We had a global singleton of the tsdb.Store which was leading to the migrate tool reading from and writing to the `source` storage.Store because it created the singleton with an object store client only pointing to the source store. This PR changes the scoping of that singleton to not be global but instead local to each instance of storage.Store **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Sandeep Sukhani	69d6ee293e	tsdb: use unique uploader name while building tsdb index filename, same as boltdb-shipper (#8975) ... **What this PR does / why we need it**: While building file names from `boltdb-shipper` and `tsdb`, we add the name of the uploader to it. To avoid collisions in filenames when running multiple ingesters with the same name, we also add a nanosecond precise timestamp to the ingester name, which is persistent as the uploader name in the active index directory. This is not being done for the tsdb index built from ingesters which can cause collisions in filenames when running multiple ingesters with the same name. I have moved the code to manage the uploader name to the index shipper config and used the same to share it with tsdb.	3 years ago
Dylan Guedes	9159c1dac3	Loki: Improve spans usage (#8927) ... **What this PR does / why we need it**: - At different places, inherit the span/spanlogger from the given context instead of instantiating a new one from scratch, which fix spans being orphaned on a read/write operation. - At different places, turn spans into events. Events are lighter than spans and by having fewer spans in the trace, trace visualization will be cleaner without losing any details. - Adds new spans/events to places that might be a bottleneck for our writes/reads.	3 years ago
Sandeep Sukhani	4e893a0a88	tsdb: sample chunk info from tsdb index to limit the amount of chunkrefs we read from index (#8742) ... **What this PR does / why we need it**: Previously we used to read the info of all the chunks from the index and then filter it out in a layer above within the tsdb code. This wastes a lot of resources when there are too many chunks in the index, but we just need a few of them based on the query range. Before jumping into how and why I went with chunk sampling, here are some points to consider: * Chunks in the index are sorted by the start time of the chunk. Since this does not tell us much about the end time of the chunks, we can only skip chunks that start after the end time of the query, which still would make us process lots of chunks when the query touches chunks that are near the end of the table boundary. * Data is written to tsdb with variable length encoding. This means we can't skip/jump chunks since each chunk info might vary in the number of bytes we write. Here is how I have implemented the sampling approach: * Chunks are sampled considering their end times from the index and stored in memory. * Here is how `chunkSample` is defined: ``` type chunkSample struct { largestMaxt int64 // holds largest chunk end time we have seen so far. In other words all the earlier chunks have maxt <= largestMaxt idx int // index of the chunk in the list which helps with determining position of sampled chunk offset int // offset is relative to beginning chunk info block i.e after series labels info and chunk count etc prevChunkMaxt int64 // chunk times are stored as deltas. This is used for calculating mint of sampled chunk } ``` * When a query comes in, we will find `chunkSample`, which has the largest "largestMaxt" that is less than the given query start time. In other words, find a chunk sample which skips all/most of the chunks that end before the query start time. * Once we have found a chunk sample which skips all/most of the chunks that end before the query start, we will sequentially go through chunks and consider only the once that overlap with the query range. We will stop processing chunks as soon as we see a chunk that starts after the end time of the query since the chunks are sorted by start time. * Sampling of chunks is done lazily for only the series that are queried, so we do not waste any resources on sampling series that are not queried. * To avoid sampling too many chunks, I am sampling chunks at `1h` steps i.e given a sampled chunk with chunk end time `t`, the next chunk would be sampled with end time >= `t + 1h`. This means typically, we should have ~28 chunks sampled for each series queried from each index file, considering 2h default chunk length and chunks overlapping multiple tables. Here are the benchmark results showing the difference it makes: ``` benchmark old ns/op new ns/op delta BenchmarkTSDBIndex_GetChunkRefs-10 12420741 4764309 -61.64% BenchmarkTSDBIndex_GetChunkRefs-10 12412014 4794156 -61.37% BenchmarkTSDBIndex_GetChunkRefs-10 12382716 4748571 -61.65% BenchmarkTSDBIndex_GetChunkRefs-10 12391397 4691054 -62.14% BenchmarkTSDBIndex_GetChunkRefs-10 12272200 5023567 -59.07% benchmark old allocs new allocs delta BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% benchmark old bytes new bytes delta BenchmarkTSDBIndex_GetChunkRefs-10 27286536 6398855 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286571 6399276 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286566 6400699 -76.54% BenchmarkTSDBIndex_GetChunkRefs-10 27286561 6399158 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286580 6399643 -76.55% ``` **Checklist** - [x] Tests updated	3 years ago
Sandeep Sukhani	162a2d0057	deletion: fix issue in processing delete requests with tsdb index (#8665) ... **What this PR does / why we need it**: When a delete request creates a new chunk from an existing chunk, we index it to the index store. In TSDB, we [drop](944b488bd2/pkg/storage/stores/tsdb/head_manager.go (L235-L237)) `__name__` label, which was being added to chunkrefs for historical reasons. However, this was not done while indexing chunks from the deletion code, causing deletion to fail with the error [chunk insertion is only allowed on existing streams](619c2212f5/pkg/storage/stores/tsdb/builder.go (L67)). This PR handles the issue by dropping the `__name__` label and adding relevant tests. **Special notes for your reviewer**: I will add an integration test for deletion with TSDB once we have PR https://github.com/grafana/loki/pull/7754 merged since it does some groundwork to run integration tests with both `boltdb-shipper` and `tsdb`. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	3 years ago
Owen Diehl	944b488bd2	logs more metadata when failing to recover head from WAL in TSDB (#8250)	3 years ago
Sandeep Sukhani	96a4fc1409	index-store: fix indexing of chunks overlapping multiple schemas (#8251)	3 years ago
Bryan Boreham	15b382294e	Remove dependency on thanos-io/thanos (#8227) ... **What this PR does / why we need it**: This avoids complications trying to match up versions between Prometheus and Thanos. **Special notes for your reviewer**: I don't know why the dependency on `github.com/efficientgo/e2e` has appeared in go.mod, but it doesn't seem to change what is in vendor. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	4 years ago
Owen Diehl	6b4563cd53	Prevent IndexShipper from leaking mutex accesses. (#7548) ... # What Inverts control in the `index-shipper` so consumers don't manage accesses to mutexes. This prevents deadlocks and no longer requires consumers have internal knowledge of the `index-shipper` itself. This PR reverts a few previous attempts. Look at the last commit to see the net-additions. # Background: TSDB previously didn't use mmap, which meant we could freely pass around TSDBs as `[]byte`. Since they were immutable, we didn't need to know when an underlying index was cleaned up/closed; this coordination was unnecessary and the garbage collector would clean up dangling memory for us. After introducing TSDB, we needed to control access to avoid use-after-free bugs. A few attempts at controlling this became increasingly complex & error prone. Instead of requiring _consumers_ to prevent deadlocks on index-shipper internals, it makes more sense that references to the index-shipper's resources should not leak outside the `ForEach` functions that give temporary access to them. This _inverts control_ from the consumer (code which queries the indices) to the producer (the index-shipper). Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Sandeep Sukhani	bb2927172a	tsdb post compaction cleanup (#7502) ... **What this PR does / why we need it**: Previously when we were reading the whole tsdb index in memory, there was not much cleanup required after we were done with the index. Now that we are mmaping the tsdb index, we need to close the mmap. I had missed calling `index.Close()` on the compacted-away multi-tenant index. This causes the compactor to retain the files on disk, leading to the compactor running away from disk space. This PR takes care of it. While doing the changes, I noticed that some of the tests failed due to `LabelValues` tsdb implementation returning strings valid only until the underlying byte slice stays valid. Since the underlying byte slice could become invalid when we close the tsdb, we should return a copy instead of yolostring. This can also happen on the query path when a sync closes the index before we get to return the response. This PR also fixes that.	4 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	4 years ago
Owen Diehl	b59277b472	Revert "Revert "Update Thanos dependency to latest version"" (#7372) ... Reverts grafana/loki#7371 PR re re-enable Thanos dependency bump.	4 years ago
Sandeep Sukhani	18da26e668	use a channel to signal completion of index read operation for tsdb and boltdb shipper (#7381) ... **What this PR does / why we need it**: In PR #7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated	4 years ago
Owen Diehl	8331785a23	Revert "Update Thanos dependency to latest version" (#7371) ... Reverts grafana/loki#7348 I expect we'll merge this again in the future, but we'll need some more time for downstream projects to be ready for this change.	4 years ago