2065 Commits (f7e13ccc770961bd2229141b0534a1eeb06d33a2)

Author	SHA1	Message	Date
Paul Rogers	1671751cbd	Augment statistics to note how many bytes are in duplicate lines due to replicas (#9400) ... **What this PR does / why we need it**: This PR is for counting the number of bytes of log lines that were marked as duplicates. This will be utilized to collect better statistics.	2 years ago
cui fliter	520e434446	fix function names in comments (#9415)	2 years ago
Travis Patterson	48ad068ce1	fix json expression parser (#9410) ... When the value parsed by the `JsonExpressionParser` is an escaped json string, the result still contains escape character and may not be valid json. This causes subsequent json parsing steps to fail. For example: `{app="foo"} | json message | line_format "{{.message}}" | json` fails when `message` remains escaped. This PR unescapes the results of extracted `JsonExpressions` so they can be parsed by subsequent json parsers	2 years ago
Dylan Guedes	f5a20e4e96	Avoid excessive gRPC logging (#9401) ... **What this PR does / why we need it**: Change the default value of `DisableRequestSuccessLog` to true. This way Loki won't log every successful gRPC call by default. The log line isn't useful either since it is redundant with existing gRPC metrics. **Which issue(s) this PR fixes**: N/A	2 years ago
Salva Corts	ca8a7accf0	Add query info to query.Exec span (#9320) ... **What this PR does / why we need it**: It's kinda difficult to see the query that is being run on a trace. The only place where I can see the query is in the `url` attribute of the `loki_api_v1_query_range` handler, but since it’s encoded it’s kinda difficult to read. If it's elsewhere it's not quite obvious. It’d make sense to have the query in the `query.Exec` span along with the stats of the query. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Owen Diehl	8bd3581b95	Revert "Revert "start writing tsdb v3"" (#9332) ... Reverts grafana/loki#9301 Should be good to start writing v3 again after https://github.com/grafana/loki/pull/9328 is merged and when we're ready to start using tsdbv3 as the default	2 years ago
Owen Diehl	f78ceec941	fixes bug during chunk iteration using markers (#9328) ... Tricky one to find, but this does the following * correctly subtracts remaining chunk count when skipping a page (this was previously using a zero-valued marker) * correctly set previous maxt on pages that are skipped. This is important because chunks themselves are delta-encoded against the previous chunk. * removes the markers pool from the chunk decoding which isn't needed. * adds a test * adds a bit more logging on which series resulted in a decoding error during chunk iteration	2 years ago
李国忠	38b298c3fe	LogQL: Introduce `distinct` (#8662) ... **What this PR does / why we need it**: Introduces `distinct` to LogQL. Usage: `{job="varlogs"} | distinct filename` It is similar to `distinct` in SQL and especially useful for sampling. Similar syntax exists in other log query languages. **Example** For the following log lines: ```json {"event": "access", "id": "1", "time": "2023-02-28 15:12:11"} {"event": "access", "id": "1", "time": "2023-02-28 15:13:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:14:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:15:11"} ``` The query below: ```logql {app="order"} | json | distinct id ``` Will return: ```json {"event": "access", `"id": "1",` "time": "2023-02-28 15:13:11"} {"event": "access", `"id": "2", `"time": "2023-02-28 15:15:11"} ``` **Example with multiple labels** ```logql {app="order"} | json | distinct id,time ``` **Which issue(s) this PR fixes**: Fixes #8649 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212) ... **What this PR does / why we need it**: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	1bd5a20595	Distributor: Modify `createShard` to create slice with more precise size (#9265) ... **What this PR does / why we need it**: Modify the `createShard` method to create the entries slice with the number of entries divided by the total number of shards instead of always creating it with len=1024.	2 years ago
Michel Hollands	475eb6a190	Remove shutdown marker file when shutting down (#9314) ... **What this PR does / why we need it**: This removes the shutdown marker file when actually shutting down. If the PVC is re-used then the marker file shouldn't be there. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Dylan Guedes	98b5e35f94	HTTP: Reuse `buffer.Bytes()` when decompressing (#9266) ... **What this PR does / why we need it**: - Modifies `decompressFromBuffer` to call `buffer.Bytes()` a single time. - Adds a new Benchmark under `http_loki_test.go` file. This is in a separate file to differentiate from the original `http_test.go` forked from Cortex and because `http_test.go` uses package `http_test` which doesn't allow us to test private functions.	2 years ago
Owen Diehl	cc6f6b9171	Revert "start writing tsdb v3" (#9301) ... Reverts grafana/loki#9276 Update main to prevent writing v3 by default while I find a bug in it.	2 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447) ... **What this PR does / why we need it**: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](a85d01c9cd/docs/sources/upgrading/_index.md) for more details. **Which issue(s) this PR fixes:** Fixes https://github.com/grafana/loki/issues/7276 **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Owen Diehl	08af8849ee	tsdb migrate versions (#9275) ... Update of https://github.com/grafana/loki/pull/9198 while @sandeepsukhani is on holiday :) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Owen Diehl	03531f5013	start writing tsdb v3 (#9276)	2 years ago
Christian Haudum	81e32dd6d6	Re-organize store related interfaces for better composability (#9249) ... **What this PR does / why we need it**: I got confused by our store related interfaces, so I split them up into smaller ones and composed them. **Special notes for your reviewer**: What I also noticed is that we sometimes prefix interface method with `Get`, e.g. `GetChunkRefs`, and sometimes we don't, e.g. `Series`. I think that re-organizing the package structure of the `pkg/storage/` package would help to better understand the individual purposes of the interfaces. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Karsten Jeschkies	7109c56bdc	Use un-escaped regexp literal for string matching. (#9252) ... **What this PR does / why we need it**: The Regexp simplification would use the original string value instead of the escaped literal. E.g. ``` | logfmt | level =~ `foo\.bar` ``` would match the label `level` against `foo\.bar` and not `foo.bar`. That is why the un-escaped literalt should be used in the equal and not-equal matchers. **Which issue(s) this PR fixes**: Fixes #9111 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Salva Corts	eba255c945	Do not use IPv6 when getting addr from ring (#9255) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/9184 changed the signature of `ring.GetInstanceAddr`. That function is now used since https://github.com/grafana/loki/pull/8067 which was not updated to use the changed signature. This PR fixes this by passing a new config flag `EnableIPv6` to the new `enableInet6` argument of `ring.GetInstanceAddr`. Similar to how it's done for the ruler: 3abd260975/pkg/ruler/base/ruler_ring.go (L80) **Which issue(s) this PR fixes**: Fixes `main` **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://grafana.com/docs/loki/latest/configuration/	2 years ago
Danny Kopping	4cd0aedc76	Fix output of limited response body when remote rule evaluation fails (#9253)	2 years ago
Dylan Guedes	24d426ce44	Distributor: Add AutoForget support (#8067) ... **What this PR does / why we need it**: - Add auto-forget capabilities to distributors - Change the distributor's ring implementation from Lifecycler to BasicLifecycler. That AutoForget delegate requires it.	2 years ago
Periklis Tsirakidis	3abd260975	Bump dskit from 528d8a7 to 1725bcc (#9184)	2 years ago
Mohamed-Amine Bouqsimi	62572b4f83	Change GeneratorURL to use configurable Grafana explore URL (#8500) ... **What this PR does / why we need it**: This PR changes the `GeneratorURL` associated with alerts generated by Loki. The new `GeneratorURL` uses a Grafana URL path.	2 years ago
Danny Kopping	af896eb5f6	Ruler: log rule evaluation detail (#9206) ... **What this PR does / why we need it**: This logs some extra detail around rule evaluation to enable more effective troubleshooting when a rule is not behaving as expected.	2 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	2 years ago
李国忠	98936437a9	[opt] s3: add s3 client http timeout for GetObject or PutObject (#8951) ... **What this PR does / why we need it**: add s3 client http timeout for GetObject or PutObject, why？ The flush queue has been increasing. This is a difficult problem. We have been investigating this problem for half a year. When the querier has a large query, the load on the s3 server is too high and the number of connections is full. It will cause the flush queue of the ingester to increase continuously and cannot be reduced. When this problem occurs, after we restart the s3 server, the flush queue will quickly become 0. We checked with our s3 (object storage compatible with the s3 protocol) team and found that it was because we used the s3 v1 sdk. The v1 sdk did not configure timeout by default, and it would wait indefinitely without configuration. The s3 v2 sdk defaults to a timeout of 30s, which only causes write failures and does not cause loki to get stuck. **Which issue(s) this PR fixes**: Fixes #<issue number> At present, as long as there are large logql queries such as 30 days, I must restart the s3 server, otherwise our ingester queue will continue to increase. We hope that we can break the stuck go routine after the 5m timeout occurs. **Special notes for your reviewer**: ingester flush queue length up 🔝  big query, 30day logql or something happen  pprof goroutine list profile snapshot  **There are other alternatives**: ``` ingester: flush_op_timeout: 10m ``` This `flush_op_timeout` configuration is more about the timeout of loki, not the timeout of s3 client. We will also have various s3 operations such as delete and getObject. We should add an independent timeout for s3, just like the cassandra client has a separate The timeout period can be configured. And the parameter `flush_op_timeout` in the early version of loki represents the timeout of writing `a batch of chunks `to s3, not a timeout of s3. Now it is indeed changed to write to s3 one by one, but it is not ruled out that in the future, it will be changed back to write a batch of chunks to s3 sequentially. In this way, `flush_op_timeout: 5m` is far from enough to write a batch of chunks to s3. ``` cassandra: timeout: 5m ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Owen Diehl	ec516facaf	removes 20 user test from compaction; seems redundant and hits timeouts (#9201) ... just a little cleanup to make tests run faster -- the 20 user test doesn't seem to test any additional surface area and takes a long time.	2 years ago
Dylan Guedes	46f044fdb7	Loki: Fix wrong association of PSRL when sharding is enabled (#9176) ... Obs: PSRL stands for `per-stream rate limit` **What this PR does / why we need it**: - Fixes the scenario where all entries errors are accounted as PSRL if sharding is enabled - Fixes invalid/rejected entries being accounted as candidates to be inserted	2 years ago
Michel Hollands	abd6131bba	Update prepare shutdown (#9175) ... **What this PR does / why we need it**: This updates the PrepareShutdown method so it supports GET and DELETE methods as well. This makes it similar to Mimir: https://github.com/grafana/mimir/pull/4718. The status is now stored in a local file. A new config setting had to be added for this file as there is no obvious place to store it. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Dylan Guedes	8580b8f8ad	Distributor: Reuse `time.Now()` for better performance (#9192) ... **What this PR does / why we need it**: Modifies `AggregateByShard` to reuse `time.Now()` to improve performance.	2 years ago
Salva Corts	56bbb43748	Logcli - Better logging when object cannot get loaded from object store (#9194) ... **What this PR does / why we need it**: This PR improves the error messaging for Logcli when an object cannot be downloaded from the store by printing the name of the object along with the error. **Which issue(s) this PR fixes**: Internal support escalation **Special notes for your reviewer**: There are other places where GetObject is called: - `indexStorageClient.GetFile` - `indexStorageClient.GetUserFile` Looks like both of them are used only by the compactor and the table manager. IIUC, these functions are not used by LogCLI. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	52655706b8	Prevent redis client from incorrectly choosing cluster mode with local address (#9185)	2 years ago
Salva Corts	422560b6b1	Flag to disable index stats cache (#9177) ... **What this PR does / why we need it**: At https://github.com/grafana/loki/pull/8972 we started caching all index stats requests. If the results cache gets overloaded, it can quickly take down the rest of the loki cell due to all the increased work. This PR adds a new flag so we can easily disable caching index stats requests. **Which issue(s) this PR fixes**: This PR is a follow up for https://github.com/grafana/loki/pull/8972 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	9c2e11d186	Ingester: Reuse allocations on `GetStreamRates` (#9180) ... **What this PR does / why we need it**: - Reuse rates mapping returned by the calculator to reduce the number of allocations	2 years ago
Dylan Guedes	bdfc0f92a0	Loki: Add debug event to ingester `GetStreamRates` (#9157) ... **What this PR does / why we need it**: - Add a new span event to our ingesters `GetStreamRates` function. - Add how many streams were updated on a `updateRates` call.	2 years ago
Owen Diehl	dbbadafca2	Tsdb/v3 (#9070) ... brief: adds page support for chunks within a series. This lets us do a few optimizations, most notably skipping chunk ranges that don't overlap with our query bounds. It also allows us to use aggregated stats for pages when computing `Stats` calls that completely overlap all chunks in a page. teaser: ``` pkg: github.com/grafana/loki/pkg/storage/stores/tsdb/index BenchmarkChunkStats/version_2/2_chunks-10 10121240 109.4 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/2_chunks-10 27591069 43.51 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/4_chunks-10 9410245 128.7 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/4_chunks-10 21808070 54.44 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/10_chunks-10 6048235 201.3 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/10_chunks-10 9517605 125.1 ns/op 0 B/op 0 allocs/op BenchmarkChunkStats/version_2/100_chunks-10 1000000 1081 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/100_chunks-10 1669972 715.4 ns/op 528 B/op 3 allocs/op BenchmarkChunkStats/version_2/1000_chunks-10 118125 10165 ns/op 24 B/op 1 allocs/op BenchmarkChunkStats/version_3/1000_chunks-10 570576 2125 ns/op 4816 B/op 6 allocs/op BenchmarkChunkStats/version_2/10000_chunks-10 10000 117447 ns/op 123014 B/op 3 allocs/op BenchmarkChunkStats/version_3/10000_chunks-10 74524 16225 ns/op 48601 B/op 9 allocs/op BenchmarkChunkStats/version_2/100000_chunks-10 842 1240380 ns/op 3211679 B/op 13 allocs/op BenchmarkChunkStats/version_3/100000_chunks-10 8494 141169 ns/op 516530 B/op 13 allocs/op ``` ``` pkg: github.com/grafana/loki/pkg/storage/stores/tsdb/index BenchmarkReadChunks/version_2/2_chunks-10 13673050 80.52 ns/op 172 B/op 0 allocs/op BenchmarkReadChunks/version_3/2_chunks-10 25713412 64.06 ns/op 201 B/op 0 allocs/op BenchmarkReadChunks/version_2/4_chunks-10 12302040 96.56 ns/op 373 B/op 0 allocs/op BenchmarkReadChunks/version_3/4_chunks-10 20512030 84.93 ns/op 346 B/op 0 allocs/op BenchmarkReadChunks/version_2/10_chunks-10 6974947 168.7 ns/op 490 B/op 0 allocs/op BenchmarkReadChunks/version_3/10_chunks-10 7436360 161.0 ns/op 632 B/op 0 allocs/op BenchmarkReadChunks/version_2/50_chunks-10 2038724 599.3 ns/op 2034 B/op 0 allocs/op BenchmarkReadChunks/version_3/50_chunks-10 2251822 619.8 ns/op 1926 B/op 0 allocs/op BenchmarkReadChunks/version_2/100_chunks-10 946430 1092 ns/op 4132 B/op 0 allocs/op BenchmarkReadChunks/version_3/100_chunks-10 1467297 1000 ns/op 3308 B/op 1 allocs/op BenchmarkReadChunks/version_2/150_chunks-10 674575 1721 ns/op 5767 B/op 0 allocs/op BenchmarkReadChunks/version_3/150_chunks-10 1000000 1373 ns/op 6171 B/op 1 allocs/op BenchmarkReadChunks/version_2/1000_chunks-10 109153 10554 ns/op 33063 B/op 0 allocs/op BenchmarkReadChunks/version_3/1000_chunks-10 158086 6963 ns/op 32829 B/op 1 allocs/op BenchmarkReadChunks/version_2/10000_chunks-10 9043 114121 ns/op 345258 B/op 0 allocs/op BenchmarkReadChunks/version_3/10000_chunks-10 22083 79896 ns/op 435493 B/op 1 allocs/op BenchmarkReadChunks/version_2/100000_chunks-10 1039 1103621 ns/op 3479873 B/op 0 allocs/op BenchmarkReadChunks/version_3/100000_chunks-10 2101 1155419 ns/op 4680266 B/op 1 allocs/op ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Ramon Cuñat	4c4f3adbf1	logql: handle negative offset (#7545) ... **What this PR does / why we need it**: Handles negative offset in logql correctly **Which issue(s) this PR fixes**: Fixes #7476 **Special notes for your reviewer**: I'm not really sure that I've implemented the best solution as it maybe tries to parse duration in many other cases. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Salva Corts	fd16425062	Cache index stats requests (#8972) ... **What this PR does / why we need it**: As described in https://github.com/grafana/loki/issues/8973, we are substantially increasing the load of index stat requests we sent to our index gateways. Many of these requests should be easily re-used by caching them. This PR adds caching for index stat requests by reusing the results cache. Here's a demo ([source][1]):  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8973 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://ops.grafana-ops.net/d/afcaef21-e5ad-49e7-ab06-42a9d7d915eb/index-stats?orgId=1&var-datasource=dev-cortex&var-cluster=dev-eu-west-2&var-namespace=loki-dev-009&var-loki_datasource=Grafana%20Logging&from=1680259907288&to=1680260431814&var-operation=All --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	dc9ca8e19a	Tools/index analyzer (#9074) ... Adds a tool that allows downloading and querying statistics across TSDB indices. Example output: ``` analyzed 1046302 series and 7607495 chunks for an average of 7.270841 chunks per series. max chunks/series was 45695. number of series with over 1k chunks: 15 ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Trevor Whitney	068a995134	upgrade go to 1.20 in nix flake (#9024) ... **What this PR does / why we need it**: Updates `flake.lock` to grab the latest `nixpkgs-unstable` in order to upgrade go version to `1.20`.	2 years ago
Ashwanth	4544f7b738	expiration: do not drop index if period is a zero value (#9156) ... **What this PR does / why we need it**: `DropFromIndex` should be return false when `retention_period` is set to a zero value. If we take chunks that overlap period boundary and are indexed in two tables, we drop these from the table on the left side of the period boundary even if they have not expired - https://github.com/grafana/loki/blob/ashwanth/drop-from-index-zero-value/pkg/storage/stores/indexshipper/compactor/retention/retention.go#L218 **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Tests updated - [x] `CHANGELOG.md` updated --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Salva Corts	8cf921a145	Pass engine opts down to middlewares (#9130) ... **What this PR does / why we need it**: The following middlewares in the query frontend uses a downstream engine: - `NewQuerySizeLimiterMiddleware` and `NewQuerierSizeLimiterMiddleware` - `NewQueryShardMiddleware` - `NewSplitByRangeMiddleware` These were all creating the downstream engine as follows: ```go logql.NewDownstreamEngine(logql.EngineOpts{LogExecutingQuery: false}, DownstreamHandler{next: next, limits: limits}, limits, logger), ``` As can be seen, the [engine options configured in Loki][1] were not being used at all. In the case of `NewQuerySizeLimiterMiddleware`, `NewQuerierSizeLimiterMiddleware` and `NewQueryShardMiddleware`, the downstream engine was created to get the `MaxLookBackPeriod`. When creating a new Downstream Engine as above, the `MaxLookBackPeriod` [would always be the default][2] (30 seconds). This PR fixes this by passing down the engine config to these middlewares, so this config is used to create the new downstream engines. **Which issue(s) this PR fixes**: Adresses some pending tasks from https://github.com/grafana/loki/pull/8670#issuecomment-1507031976. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: 1bcf683513/pkg/querier/querier.go (L52) [2]: edc6b0bff7/pkg/logql/engine.go (L136-L140)	2 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](ff7b462973/pkg/storage/factory.go (L188)) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Travis Patterson	c4261b19fd	Fix parsing of long queries (#9134) ... Loki uses a `scanner.Scanner` to consume the characters in a query. `lex.go` relies heavily on the assumption a scanner can be copied to perform speculative parsing. The problem is that sometimes speculative parsing overruns the scanners internal buffer of `1024` bytes and incurs an additional read to the underlying reader. At that point, all subsequent scanners are out of sync with their underlying readers and the query may fail to be parsed. In the case of #9132, there is a function call at line `1024`. `lex.go` tries to understand this is a function call by speculatively parsing and incurs an underlying read which consumers the rest of the reader. When the lexer tries to scan the next character with the original scanner, the reader is already consumed even though only 1024 bytes have been scanned The scanner has an internal buffer size of `1024 bytes`. This PR copies `scanner.Scanner` from the `text/scanner` package and increases the internal buffer to `maxInputSize` which guarantees that no calls to `Scan` will incur reads beyond the first one. Ideally we wouldn't pass Scanners around in such a way but the cost of that is to rethink much of Loki's lexing logic. Given we know a max input size and we already have that in memory, this solution should suffice. Fixes https://github.com/grafana/loki/issues/9132	2 years ago
Salva Corts	f9d9b471f8	Fix comment in `buildChunkMetas ` test function (#9124) ... wrt https://github.com/grafana/loki/pull/9096#discussion_r1164555980	2 years ago
Travis Patterson	7735b1e98e	Sanitize label keys in the unpack parser (#9121) ... This PR makes the `unpack` parser sanitize label keys like all the other parsers. When invalid label names are included, it causes the user to get `could not write JSON response` errors when the response is serialized to send back to the user.	2 years ago
Salva Corts	9783757932	Proportional TSDB index stats for chunks partially fitting in queried time range. (#9096) ... **What this PR does / why we need it**: When computing index stats for a query, we sum up the stats from all the chunks the query would read, even if some chunks will only be read partially. Moreover, if the query spans across two different TSDB tables, and there are chunks duplicated on both tables (at the end of the first one and at the beginning of the next one), the chunks' stats will be counted twice. This PR improves how we compute stats by taking care of partially matching chunks. Assuming entries and bytes are evenly distributed in the chunk, we will take the proportional number of entries and the number of bytes that fit on the span. For example, a chunk spans from t1 to t10 and has 10 entries and 10 KBs. A query from t3 through t7 would hit 50% of the chunk time span. Therefore, we estimate that the query would load 5 KBs and 5 entries from the chunk. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Karsten Jeschkies	ff36227d87	Ignore broken query limits header. (#9103)	2 years ago
Salva Corts	fd5d78784f	Estimate chunk size on compaction looking at data uncompressed size (#9099) ... **What this PR does / why we need it**: At `compactedIndex.ToIndexFile` we were not computing the size of the chunk correctly. We were looking at the size of the chunk ref. Since the size of the chunk ref is always a bunk of bytes, the chunk meta KB field would always be 0. 4e893a0a88/pkg/storage/stores/tsdb/compactor.go (L367) Instead, we should get the approx KBs by looking at the chunks data uncompressed size. As we do at `store.IndexChunk`: d405162516/pkg/storage/stores/tsdb/store.go (L189)	2 years ago
Dylan Guedes	b82ad11f92	Loki: Fix span event logging on rate store (#9086) ... **What this PR does / why we need it**: Fix an event logging on rate store that is missing the `event` key. `LogKV` expects a key-value pair always.	2 years ago