1833 Commits (f93b91bfb57e65f0a783d97fc3875283d7997409)

Author	SHA1	Message	Date
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	f5fbfabd84	Add 3rd target (`backend`) to SSD/Scalable mode (#7650) ... **What this PR does / why we need it**: This adds a 3rd target to SSD/Scalable mode, as well as a config flag to run the legacy `read` mode from the original 2 target configuration in order to give people time to migrate before we remove this option in Loki 3.0 (hopefully). The 3rd target has the two major advantages: 1. Allows the read path to be run as a deployment and thus auto-scaled using our existing auto-scaling logic for queriers 2. Creates consistency with Mimir since they went with a 3 target model for their SSD deployment.	2 years ago
Maxim Ivanov	b773a7a3cf	promtail: ratelimiting by label (#7597) ... Limit stage can now takes optional `by_label_name` param, which tracks distinct values of that label and ratelimit entries independently. Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.	2 years ago
Karsten Jeschkies	95ec710168	Introduce sizing tool to documentation page. (#7830) ... **What this PR does / why we need it**: The sizing tool should help to find a good cluster sizing when starting out with Loki. It's a dynamic form withing the documentation.  This pull request will be followed by help on hover. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Karsten Jeschkies <karste.jeschkies@grafana.com>	3 years ago
Sandeep Sukhani	e9c93cd0f5	consider range and offset in queries while looking for schema config for query sharding (#7880) ... **What this PR does / why we need it**: We disable query sharding when a query touches multiple schema configs since they might have different sharding factors and might handle sharding differently. This check was done purely on the start and end time of the query without considering `range` and `offset`, which would change the length of the actual data being queried. Besides being incorrect, it also causes us to fail queries when migrating between tsdb and non-tsdb stores since both handle query sharding differently. For e.g., if the prev schema is `boltdb-shipper` and starting today, `tsdb` index is being used, so if we do a query `sum(rate({foo="bar"}[24h])` even with start and end within `tsdb` range, we will fail the query with error `incompatible index shard` if dynamic sharding goes with shard factor other than `32`(default shard factor in ingester for inverted index). The reason being the `range` here is `24h` which causes us to process data from previous schema as well. This PR takes care of the issue by factoring in `range` and `offset` in the queries while looking for schema config for query sharding. **Checklist** - [x] `CHANGELOG.md` updated	3 years ago
Mohamed-Amine Bouqsimi	03a34db8eb	Fix alertmanager per tenant override config behavior (#7805) ... **Context:** #7359 implement multi tenancy for the alertmanager config. However, the behavior of the per-tenant alertmanager config is erroneous, as it tries to merge the global alertmanager config with the per-tenant config. This does not work because there is no way of knowing if certain values were set by the user or if they were undefined and have the default Golang zero value (typically booleans). This PR changes the behavior of the alertmanager config to override all the configuration instead of trying to merge the global config with the per-tenant config. It also applies default values when appropriate.	3 years ago
Dylan Guedes	2e496ddff2	Loki: Add one additional bucket to `rate_store_stream_shards` (#7856) ... **What this PR does / why we need it**: Add one additional bucket (value=1) to the `rate_store_stream_shards`. By jumping from 0 to 2 we can't distinguish between streams that were minimally sharded (shards=2) to non-sharded ones (shards=1).	3 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Travis Patterson	a1e9e4cdd0	Revert "Make some stream shard metrics per-tenant (#7838)" (#7842) ... Because these metrics will be coming from each distributor, this is going to have an adverse impact on cardinality. Consider doing this with logging	3 years ago
Travis Patterson	96fc2abfff	Make some stream shard metrics per-tenant (#7838) ... This PR just adds tenant information so some of our per-stream-rate-limit metrics.	3 years ago
Christian Haudum	feaf9c3232	Log query string on retry alongside the error (#7834) ... **What this PR does / why we need it**: For better observability of query retries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	25f4dda811	fix(memcached): use `default` branch avoid writing to closed chan (#7833) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> **What this PR does / why we need it**: Follow up to https://github.com/grafana/loki/pull/7817 This PR uses `default` branch instead of `inputCh <- work` to make sure we are writing to closed `inputCh`. Gist is, `default` run [only when none of the branch is ready](https://go.dev/tour/concurrency/6). which makes more sense rather than to have `inputCh <- work` (writing to closed channel on the branch condition) These can be explained by these two tiny snippets. * [with `default`](https://go.dev/play/p/-FspbTZd20I) * [without `default`](https://go.dev/play/p/Ag4WznOaEq0) **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: We already have test `TestMemcached_fetchKeysBatched` to catch this behaviour. In fact this test caught this probabilistic behaviour in some CI failures. [here ](https://drone.grafana.net/grafana/loki/17853/3/4) and[ here](https://drone.grafana.net/grafana/loki/17854/3/4) Thanks @DylanGuedes for catching this CI failures. **Checklist** Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Trevor Whitney	37b1c0fce0	guard against divide by 0 when splitting parallelism (#7831) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: We saw a spike in divide by zero panics in the code introduced in #7769. I was able to reproduce this error via a test that calculates `WeightedParallelism` with a start that's after the end. Not sure if this is possible, but we definitely saw this happening in our ops environment, so something is causing it, and the fix should guard against it in any case. **Special notes for your reviewer**: **Checklist** - [X] Tests updated Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Travis Patterson	eb7fbfe0b4	Improve ratestore performance (#7810) ... This PR changes the Rate Store to use multi-level maps rather than `tenant+stream-fingerprint` key. Benchmarks: *old* ``` goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/distributor cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz BenchmarkRateStore-8 117 10420080 ns/op 3201761 B/op 200025 allocs/op PASS ok github.com/grafana/loki/pkg/distributor 14.646s ``` *new* ``` goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/distributor cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz BenchmarkRateStore-8 320 3811906 ns/op 2660 B/op 26 allocs/op ``` *stats* ``` name old time/op new time/op delta RateStore-8 10.7ms ± 8% 4.1ms ± 8% -61.42% (p=0.000 n=10+10) name old alloc/op new alloc/op delta RateStore-8 3.20MB ± 0% 0.00MB ± 0% -99.92% (p=0.000 n=8+9) name old allocs/op new allocs/op delta RateStore-8 200k ± 0% 0k ± 0% -99.99% (p=0.000 n=9+10) ```	3 years ago
李国忠	63a57c7b5c	[bugfix] scheduler: Gracefully shutdown querier when using query-scheduler (#7735) ... **What this PR does / why we need it**: Gracefully shutdown querier when using query-scheduler This PR is an attempt to fix a bug that my loki cluster is unavailable for logql. The source code and ideas are from mimir of LGTM. Thanks mimir and pr author @pracucci **Which issue(s) this PR fixes**: Fixes #7722 **Special notes for your reviewer**: mimir PR https://github.com/grafana/mimir/pull/1756 https://github.com/grafana/mimir/pull/1767 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Kaviraj Kanagaraj	4efdc131be	fix(memcached): panic on `send on closed channel`. (#7817)	3 years ago
Christian Simon	2ff6f13c8b	Better error message for empty selector queries (#6936) ... This improves the error when an empty label selector is given: Before: ```sh-session # Before $ logcli query "{}" Error response from server: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER # After $ logcli query "{}" Error response from server: parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will ``` I fear that might have wider implications, so it would be great to get a lot of eyes on this. In theory I think we should remove the restriction on querying all log streams, but this should maybe discussed in a follow up PR 🙂 Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814) ... **What this PR does / why we need it**: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
Sandeep Sukhani	1410808ee9	use grpc for communicating with compactor for query time filtering of data requested for deletion (#7804) ... **What this PR does / why we need it**: Add grpc support to compactor for getting delete requests and gen number for query time filtering. Since these requests are internal to Loki, it would be good to use grpc instead of HTTP same as all the internal requests we do in Loki. I have added a new config for accepting the grpc address of the compactor. I tried having just the existing config and detecting if it is a grpc server, but it was hard to do it reliably, considering the different deployment modes we support. I think it is safe to keep it the same and eventually deprecate the existing config. **Checklist** - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated	3 years ago
Kaviraj Kanagaraj	30ddd77150	loki-canary: respect `useTLS` flag when `push` mode is enabled. (#7701) ... **What this PR does / why we need it**: Currently `useTLS` flag is ignored on the push writer. This PR consider that flag and adjust the schema (http -> https) when that flag is enabled. **Which issue(s) this PR fixes**: Fixes #7692 **Special notes for your reviewer**: **Checklist**	3 years ago
Josef Johansson	b05f4fced3	Remove hardwired '127.0.0.1' values in default addresses (#7784) ... While running all-in-one Loki instance, it's not possible to listen on anything other than IPv4 since compactor_address is then set to http://127.0.0.1. This enables listening on only IPv6. Also if frontend_address, scheduler_ring and scheduler_address is not set, the default value is to connect to '127.0.0.1:<grpc_listen_port>'. Which means that grpc_listen_address is ignored. **What this PR does / why we need it**: It's not possible to use compactor while using an all-in-one instance and listening to IPv6. Let's not ignore grpc_listen_address to not confuse users. **Special notes for your reviewer**: As noted in the first commit, the default behavior is changed from defaulting to `127.0.0.1` to `localhost`. Not sure how major that change is though. My /etc/hosts says ``` $ grep localhost /etc/hosts 127.0.0.1 localhost ``` Signed-off-by: Josef Johansson <josef@oderland.se>	3 years ago
Dylan Guedes	4567d01ac6	Loki: Add new sharding metrics (histogram based) (#7762) ... **What this PR does / why we need it**: Add histogram-version of two existing stream sharding metrics. The new metrics are: - `rate_store_stream_shards`: Will show us the distribution of shards reported by ingesters. This will help us on monitoring the number of shards we're dealing with. As of now we only have access to the max number of shards seen, which is helpful but not enough. - `rate_store_stream_rate_bytes`: Same as below: will help us understand the distribution of stream sizes instead of only showing us the max stream size seen.	3 years ago
李国忠	6d05ade6a8	[performance] metrics query: range vector support streaming agg when no overlap (#7380) ... metrics query: range vector support streaming agg when no overlap	3 years ago
Karsten Jeschkies	7ec51e8b9d	Log rule changes through API. (#7790) ... **What this PR does / why we need it**: This change should make it easier to detect changes to rules via the API. **Which issue(s) this PR fixes**: Fixes #7789 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	720dfe0476	fifocache config is deprecated. use embedded-cache instead (#7800)	3 years ago
Travis Patterson	343e39f67b	Update Ingester Rate Calculations (#7652) ... - Use weighted moving average for rates - Keep Streams around until they are removed Previously, the rate calculator only kept the most recent second. This led to issues where bursty streams that didn't get pushes every second resulted in undersharding those streams	3 years ago
Danny Kopping	a63ad06509	Querier/Ruler: query blocker (#7785) ... Block malicious or expensive queries using a per-tenant runtime configuration.	3 years ago
Owen Diehl	22089415e8	Split parallelism across Period Configs (#7769) ... One of things we watch while updating non-TSDB period configs to TSDB period configs is the difference in query parallelism. TSDB dynamically shards queries into (potentially) much smaller units of work compared to the static shard factors uses prior. To account for this, we use much higher query parallelism configurations with TSDB period configs. This creates a potential problem when querying across `non-tsdb, tsdb` period boundaries: we may want a query parallelism of 512 for the tsdb portion but only 64 for the non-tsdb portion! However, we only had one limit to specify this per tenant, meaning this would be too high when querying non-tsdb periods or too low when querying tsdb ones. This PR * Introduces `tsdb_max_query_parallelism` (default `512`) to `limits_config` * Uses `tsdb_max_query_parallelism` and `max_query_parallelism` limits to find a better parallelism _per query_ by weighting the two respective configs by the proportion of each query spent on TSDB or non-TSDB period configurations. Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708) ... **What this PR does / why we need it**: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> **Which issue(s) this PR fixes**: https://github.com/grafana/loki/issues/7696 **Special notes for your reviewer**: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	3 years ago
Karsten Jeschkies	dd098bbd6d	Return error on JSON serialization error. (#7745) ... Clean up of https://github.com/grafana/loki/pull/7721	3 years ago
Kaviraj Kanagaraj	3a0be48975	config: Improve error message loading config with ENV variables. (#7759) ... **What this PR does / why we need it**: Before: ``` Unable to parse config: ./clients/cmd/promtail/promtail-local-config.yaml: parse "http://${LOKI_URL}localhost:3100/loki/api/v1/push": invalid character "{" in host name ``` After: ``` Unable to parse config: ./clients/cmd/promtail/promtail-local-config.yaml: parse "http://${LOKI_URL}localhost:3100/loki/api/v1/push": invalid character "{" in host name. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file ``` **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] `CHANGELOG.md` updated	3 years ago
Sandeep Sukhani	d3e42959a5	fix bugs in processing delete requests with line filters and add proper tests (#7720) ... **What this PR does / why we need it**: It fixes the following issues: * Whole chunks being deleted with a line filter were skipped. * Chunks partially covered by delete requests with a line filter were not properly processed. The part that **was not covered** by the delete request was rewritten with a line filter, while the part that **was covered** by the delete request was completely deleted. The issue went unnoticed because it seems we continue to do query time filtering of data even after the delete requests are processed. **Which issue(s) this PR fixes**: Fixes #7590 **Special notes for your reviewer**: If multiple delete requests with a line filter touch the same chunk, then I am considering just the first request that marks it for deletion to keep this PR simple. I have left a to-do to see if we can make it work efficiently. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	3 years ago
Karsten Jeschkies	bd879498ad	Serialize query JSON response in a stream. (#7721) ... **What this PR does / why we need it**: Previously the query response was converted to `loghttp.ResultValue`. This requires a lot of allocations. Furthermore, the JSON response string was allocated before it was written to the response writer. This changes replaces the conversion from `parser.Value` to `loghttp.ResultValue` with a hand written JSON stream. This redueces the allocation overhead quite a bit: ``` › benchstat before.txt after.txt name old time/op new time/op delta _Encode-16 51.5µs ± 1% 24.1µs ± 5% -53.25% (p=0.008 n=5+5) name old alloc/op new alloc/op delta _Encode-16 7.68kB ± 0% 1.66kB ± 0% -78.34% (p=0.008 n=5+5) name old allocs/op new allocs/op delta _Encode-16 129 ± 0% 28 ± 0% -78.29% (p=0.008 n=5+5) ``` The correctness is validate with quick check. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	0458dc8cbb	fix: Hide this error message when ctx timeout occurs in s3.getObject (#6360) ... **What this PR does / why we need it**: ``` level=error ts=2022-01-24T10:57:05.273404635Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=error ts=2022-01-24T10:57:05.273434964Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=warn ts=2022-01-24T10:57:05.273457211Z caller=grpc_logging.go:55 method=/logproto.Querier/Query duration=16.524312361s err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" msg="gRPC\n" level=error ts=2022-01-24T10:57:05.273486368Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=error ts=2022-01-24T10:57:05.273786914Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" level=error ts=2022-01-24T10:57:05.274246531Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" level=warn ts=2022-01-24T10:57:05.274385302Z caller=grpc_logging.go:55 method=/logproto.Querier/Query duration=29.542213769s err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" msg="gRPC\n" ``` When logql timeout, do not print these logs. Since we retry, we don't want to show intermittent timeouts, and we only care about the last error. **Which issue(s) this PR fixes**: Fixes [<5221>](https://github.com/grafana/loki/issues/5221)	3 years ago
洪阿南	776ee95a00	modify usage stat metric unit to match definition (#7588) ... Output the flushedChunksLifespanStats metrics in seconds instead of hours	3 years ago
Dylan Guedes	8e38ebb2b7	Loki: Implement timeouts migration (#7555) ... **What this PR does / why we need it**: Improve the experience of migrating to our new timeouts behavior, which is the unifying of the engine:timeout and querier:query_timeout into a new configuration named limits_config:query_timeout. This PR makes sure users won't be surprised by their timeouts being shorter, but at the same time, it makes it easier to migrate to the new configuration. Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
李国忠	b3664ff255	[fix] cassandra: reconnection when k8s cassandra pods restart (#6725) ... **What this PR does / why we need it**: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. **Which issue(s) this PR fixes**: Fixes #1821 #7140	3 years ago
李国忠	6848ea37dd	fix ingester: grpc method=/logproto.Querier/Query fail. (#5970) ... When ingester's ctx times out, don't try to use grpc to return data to the querier again. This PR fixes the ingester error rate > 0 for our loki cluster. We rely on this error rate indicator for alarming Fixes #5964	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602) ... Implement `decolorize` filter that get rids of ANSI color codes in the log message. **Which issue(s) this PR fixes**: Fixes #7601	3 years ago
Wu Zhiyuan	3bcc7ccd5a	use fifocache.duration instead of fifocache.interval in warn deprecated (#7609) ... **What this PR does / why we need it**: I configured fifo cache with `validity` option (which is [deprecated](b92f113cb0/docs/sources/configuration/_index.md?plain=1#L2004) in v2.6.0, `fifocache.duration` as an flag) and loki print that: ``` level=warn ts=2022-07-08T17:01:30.134251284Z caller=fifo_cache.go:125 msg="running with DEPRECATED flag fifocache.interval, use fifocache.ttl instead" cache=frontend.fifocache ``` However, there is no flag `fifocache.interval`. As #5148 changed, the deprecated flag should be `fifocache.duration`, not `fifocache.interval`.	3 years ago
Danny Kopping	d32f3f7b68	Querier: remove expensive & low-value debug log line (#7532) ... Removes a low-value debug log line that, in one recent example, can logs tens of thousands of times in a short span of time. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	3c792294d2	Loki: Fix index_set log message (#7538) ... As of now, this log message isn't correctly showing the list of files (instead it logs the value as "unsupported value type").	3 years ago
Travis Patterson	af68dbc6cf	Keep rates in the ratestore for 10 minutes (#7599) ... This PR accounts for intermittent streams that might get dropped on a per-second update interval Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Periklis Tsirakidis	e0a7b28a61	Add single compactor http client for delete and gennumber clients (#7453)	3 years ago
Cyril Tovena	6aa85e318c	Improves performance of json responses (#7522) ... **What this PR does / why we need it**: This allows to reuse stream memory allocations between responses. Related https://github.com/grafana/loki/pull/3163 but this time this is for the encoding. ``` ❯ benchstat before.txt after.txt name old time/op new time/op delta _Encode-16 29.2µs ±12% 25.2µs ± 1% -13.85% (p=0.016 n=5+4) name old alloc/op new alloc/op delta _Encode-16 24.9kB ± 6% 16.4kB ± 8% -34.20% (p=0.008 n=5+5) name old allocs/op new allocs/op delta _Encode-16 145 ± 0% 129 ± 0% -11.03% (p=0.008 n=5+5) ```	3 years ago
Travis Patterson	198c81fc7b	Reduce number of locks and partition by tenant and stream hash (#7583) ... This PR reduces the number of locks in the rate calculator and allows it to hold an arbitrary number of rates by partitioning by tenant and stream hash Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	020631ebac	add user-id transformer for logs results cache (#7581) ... **What this PR does / why we need it**: Adding transformer to results cache same as metrics cache done in PR #7542	3 years ago
Travis Patterson	af899dcc00	Retry pr (#7579) ... This is the same PR as https://github.com/grafana/loki/pull/7152 but I botched the merge when updating the code. This one works Co-authored-by: Arseniy Antonov <arseny.antonov@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Travis Patterson	c75b822fc6	Track rates per stream and per tenant (#7564) ... It is possible for two tenants to have streams with the same labels. This PR changes the stream tracking in the `shard_tracker`, `stream_rate_calculator`, and `rate_store` to account for tenant ID when tracking streams.	3 years ago
Kaviraj Kanagaraj	ceb09efdf8	fix(loki-canary): Send to Loki after updating `totalEntries`. (#7211) ... **What this PR does / why we need it**: Fixes: #7142 (take a look for more details about the problem there) After trying few things to avoid this race and discussin it with Ed, This small changes seems to be more accurate and seems to fix the issue. This is unblocking rolling out new canary in our internal cells Update: For some reason I thought the fix didn't fix the original issue when I tested it with internal loki dev cell, almost a month ago. But after testing it for more than 2hr on same cell. I realized it is working fine (will leave it for a day like this just to confirm) Verified following: ``` {.... container="loki-canary"} |= "websocket missing" ``` and noticed it's happening no longer after the fix. Also verified the metrics ``` sum(increase(loki_canary_websocket_missing_entries_total{...}[$__range])) ``` It dropped to zero.   Update 2: After running it for whole day, looks there was two log entries missed on the websocket. My guess is since websocket is long-live connection and something can interrupt the connection loosing this message.  I also quickly checked how often this happens on other bigger envs (say ops) and looks like it's not that uncommon.  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/7142 **Special notes for your reviewer**: Tested it on internal loki dev cells **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago