1392 Commits (9d62f051b840eee03bc21aead3f40e3d29f4e687)

Author	SHA1	Message	Date
Christian Haudum	6d7422aabf	Assert that push values tuples consist of string values (#5685) ... * Assert that push values tuples consist of string values This fixes a bug in the parsing of push requests that could allowed users sending non-string "log lines" from clients, such as full JSON objects. The values were interpreted as strings, even though they weren't, leading to unexpected log line transformation because the request did not fail. Fixes #5645 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Karsten Jeschkies	705baa39f4	Speed up re-labelling in multi-tenant queries. (#5663)	3 years ago
Owen Diehl	3f28a33d93	TSDB Index reuses slices, adds pools (#5630) ... * adds a pool for ChunkMetas * index takes slice pointers for allocation reasons and adds pools for all index types * reuse slices in tsdb benchmarks * adds count to bench script * properly returns series to pool * more ergonomic index signatures while still supporting slice reuse * tsdb index documentation * aligns tsdb-map tooling with new index signatures	3 years ago
Owen Diehl	d084dc7f4b	Tsdb/hash order (#5631) ... * encode fingerprint into series in tsdb * uses series fingerprint in index reads * order tsdb series by fprint * tsdb writes fingerprint offsets table * shardannotation is part of tsdb/index, fixes testware * loads fingerprint offsets from tsdb * removes unnecessary sortedpostings * adds prefix sharding logic * sharded tsdb postings * updates benchmark to pass nil hash * runs tsdb benches with variable count	3 years ago
Karsten Jeschkies	3d5a3936d1	Support limits in multi-tenant queries. (#5626) ... **What this PR does / why we need it**: This changes adds support for the the following limits in multi-tenant queries: - `MaxQuerySeries` - `QuerySplitDuration` - `MaxEntriesLimitPerQuery` - `MinShardingLookback` - `MaxQueryLength` - `MaxQueryParallelism` - `MaxQueriersPerUser` - `MaxCacheFreshness` It always uses the more restricted limit. That means the smallest `MaxQuerySeries` but biggest `QuerySplitDuration`. The split duration is special. Zero means disabled. Also, the global default resolver is the multi tenant resolver when multi-tenant queries are enabled. This will result HTTP 400 errors in case the user sends multiple tenants to endpoints that do not support it. **Which issue(s) this PR fixes**: **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes. Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	3 years ago
Salva Corts	1f2b450f2d	Add inflight requests metric to scheduler (#5658)	3 years ago
Christian Haudum	95b885691d	Preserve interval parameter when splitting queries by time (#5622) ... * Preserve interval parameter when splitting queries by time Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests for interval param on Loki downstream requests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry for #5622 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Simon	ea62af58e9	Introduce a ChunkRef type as part of logproto (#5643) ... * Change protobuf definitions * Adapt to an embedded ChunkRef struct * Move the Checksum to the ChunkRef	3 years ago
Sandeep Sukhani	6a533e5a56	improve boltdb-shipper logging to help debug index query latency issues (#5651)	3 years ago
Cyril Tovena	59bd44a5df	Remove more chunkstore and schema version below v9 (#5650) ... * Remove more chunkstore and schema version below v9 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	34f9649c6d	Remove more unused code (#5638) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Sandeep Sukhani	8c06c546ab	debug log number of index entries cached and retrieved from the cache per index query (#5642)	3 years ago
Cyril Tovena	df1a8e3632	Remove unused code from cortex (#5633) ... * Remove unused code from cortex Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the build Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flakky test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update vendor Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Simon	b6e6003724	Remove unused storage gateway references (#5627)	3 years ago
Owen Diehl	ec809e665f	Tsdb/sharding-benchmarks (#5628) ... * adds shard to tsdb/index pkg * add sharded tsdb benchmark	3 years ago
Cyril Tovena	39101308f1	[Frontend] Log Result Cache (#5502) ... * Rough verion still require some fixes * with file saved * Finish off the code and logic Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactors metrics and hook the log cache middleware Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests, fixes and lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hash the key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Fixes typo in variable namme Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use only one ok bool Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * check array boundaries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620) ... **What this PR does / why we need it**: The original change injected the user id when the org id was required to fetch all data. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#230 **Special notes for your reviewer**: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1|2" --data-urlencode 'query={compose_service=~"log-gen.*"}' | jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Travis Patterson	88372c93cc	Query Filtering -- Querier (#5482) ... * Query Filtering -- Querier - Get deletes from the index gateway and attach them to store and ingester requests * lint * Fix test -- sometimes svc is nil * don't test for delete overlap on each selector * lint	3 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545) ... * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Danny Kopping	75328872dc	Print stacktrace to stderr for easier readability (#5616) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Kaviraj Kanagaraj	d6621f452f	Replace `PromQL` -> `LogQL` wordings in Querier (#5602)	3 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600) ... * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	3 years ago
Owen Diehl	2eb1046818	Tsdb/multi idx (#5598) ... * begins working on a multi file index for querying across TSDBs * abandons heapsort for multi-index * series struct includes fingerprint and implements rest of multi-idx methods * goimports	3 years ago
afayngelerindbx	e49c360c36	canary: Adds locking to prevent multiple concurrent invocations of `confirmMissing` from clobbering each other (#5568)	3 years ago
Owen Diehl	f6d15fba48	bounds utility for tsdb lib (#5593)	3 years ago
Sandeep Sukhani	d2ce08280f	Compactor improvements (#5587) ... * upload and cleanup index sets concurrently * reuse batch slice used for building a batch of index during compaction	3 years ago
Sandeep Sukhani	7f498e9617	cleanup common index source files at the end while compacting index with boltdb-shipper (#5585)	3 years ago
Owen Diehl	824c65598f	store tsdb bounds (#5582)	3 years ago
Sandeep Sukhani	c76544baf9	build a new batch while starting to process a bucket (#5580)	3 years ago
Owen Diehl	dee6a56df1	Tsdb/index interface (#5577) ... * begins speccing out index ifc, impl * block level labelnames/values * single file index test + bugfix * goimports * LabelNames and LabelValues support matchers * s/Interface/Index/	3 years ago
Susana Ferreira	03b734c573	Fix frontend-worker error message (#5567)	3 years ago
Susana Ferreira	ef49909430	Refactor sharding engine to downstream engine (#5546)	3 years ago
Susana Ferreira	16aa9dd55d	Update vectorAggEvaluator to fail for expressions without grouping (#5544)	3 years ago
Michel Hollands	9e0941b408	Change usagestats expvars so they can be set twice (#5515) ... * Change Edition so it can be set twice Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change Target so it can be set twice Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Test Target can be set twice Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Mention Target can be set multiple times Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add existence checks to other stats types and add tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove from target and edition Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Owen Diehl	84c97ce43b	Revert "metrics: Add `user` label to `worker_inflight_queries` (#5485)" (#5534) ... This reverts commit 45d5f28e67.	3 years ago
Karsten Jeschkies	9e16ffbd3b	Support `SelectLogs` and `SelectSamples` in multi-tenant querier. (#5517) ... **What this PR does / why we need it**: This patch adds some functionality to the `MultiTenantQuerier` stub. `SelectLogs` and `SelectSamples` fan out into multiple calls, one for each tenant. The returned iterators are then merged back together and the proper tenant label is added. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#232 **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
李国忠	2858ebde5c	[bugfix] fix nil pointer (#5527)	3 years ago
Sandeep Sukhani	593b510f0e	fix broken index gateway benchmark due to missing limits (#5520)	3 years ago
Michel Hollands	d6d7de4edc	Remove extra param in call for inflightRequests metric (#5507) ... * Remove extra param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add CHANGELOG.md entry Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove periods at end of line for older CHANGELOG.md entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Cyril Tovena	95fefa6096	Align usage report config with Grafana (#5406) ... * Align usage report config with Grafana Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes tests * Update pkg/usagestats/reporter.go Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Karsten Jeschkies	ae39676cce	Stub multi tenant querier. (#5490) ... **What this PR does / why we need it**: - Introduce Queries interface and Querier API. - Configure a multi-tenant querier that just passes requests through. **Which issue(s) this PR fixes**: Closes #231 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Kaviraj Kanagaraj	45d5f28e67	metrics: Add `user` label to `worker_inflight_queries` (#5485) ... * metrics: Add `user` label to `loki_querier_worker_inflight_queries` * Add also `querier` label to it. * Remove `querier`	3 years ago
Sandeep Sukhani	15dde37518	initialize overrides when running index gateways which is required to run them (#5496)	3 years ago
Travis Patterson	b485b81ff9	Fix potential deadlock in the table manager (#5472) ... * Fix potential deadlock in the table manager * Ensure second lock is released on return * use t.TempDir instead of os.TempDir * fix broken test Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Sandeep Sukhani	1e56dca741	per user index query readiness with limits overrides (#5484) ... * add support for configuring query readiness for user index with limits overrides config * improve tracking of duration for query time downloading of index * update changelog * use the existing context.Context we have for background operations while ensuring query readiness	3 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	3 years ago
Sandeep Sukhani	87b7dfae87	Stream query responses from boltdb index client (#5468) ... * stream query response from boltdb index client instead of sending one row per batch * add benchmark for testing index gateway query performance * lint and fix broken test * wrap error message while failing to send back response * sort imports	3 years ago
Cyril Tovena	a411bb9420	Disable sharding of count/avg when labels are mutated (#5474)	3 years ago
Cyril Tovena	6dad54fef9	Fixes ingester sample deduping (#5470) ... * Fixes ingester sample deduping * Call get hash only once * Update pkg/iter/entry_iterator.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago