1353 Commits (0dedb381ea3b7be012dfd5caeab1c636d8dcd889)

Author	SHA1	Message	Date
Michel Hollands	d6d7de4edc	Remove extra param in call for inflightRequests metric (#5507) ... * Remove extra param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add CHANGELOG.md entry Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove periods at end of line for older CHANGELOG.md entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	95fefa6096	Align usage report config with Grafana (#5406) ... * Align usage report config with Grafana Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes tests * Update pkg/usagestats/reporter.go Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Karsten Jeschkies	ae39676cce	Stub multi tenant querier. (#5490) ... **What this PR does / why we need it**: - Introduce Queries interface and Querier API. - Configure a multi-tenant querier that just passes requests through. **Which issue(s) this PR fixes**: Closes #231 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Kaviraj Kanagaraj	45d5f28e67	metrics: Add `user` label to `worker_inflight_queries` (#5485) ... * metrics: Add `user` label to `loki_querier_worker_inflight_queries` * Add also `querier` label to it. * Remove `querier`	4 years ago
Sandeep Sukhani	15dde37518	initialize overrides when running index gateways which is required to run them (#5496)	4 years ago
Travis Patterson	b485b81ff9	Fix potential deadlock in the table manager (#5472) ... * Fix potential deadlock in the table manager * Ensure second lock is released on return * use t.TempDir instead of os.TempDir * fix broken test Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Sandeep Sukhani	1e56dca741	per user index query readiness with limits overrides (#5484) ... * add support for configuring query readiness for user index with limits overrides config * improve tracking of duration for query time downloading of index * update changelog * use the existing context.Context we have for background operations while ensuring query readiness	4 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	4 years ago
Sandeep Sukhani	87b7dfae87	Stream query responses from boltdb index client (#5468) ... * stream query response from boltdb index client instead of sending one row per batch * add benchmark for testing index gateway query performance * lint and fix broken test * wrap error message while failing to send back response * sort imports	4 years ago
Cyril Tovena	a411bb9420	Disable sharding of count/avg when labels are mutated (#5474)	4 years ago
Cyril Tovena	6dad54fef9	Fixes ingester sample deduping (#5470) ... * Fixes ingester sample deduping * Call get hash only once * Update pkg/iter/entry_iterator.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	55acdbd08f	boltdb shipper index gateway client improvements (#5456) ... Index Gateway query performance improvements by bounding the number of concurrent grpc calls we make from the client.	4 years ago
RangerCD	daacfd0b70	Fix two remote_timeout configs in ingester_client block (#5418) ... * Fix two remote_timeout configs in ingester_client block * Add default value of remote_timeout in document	4 years ago
Paschalis Tsilias	cbd3b007cc	logproto: Add deprecated annotation to LegacySample and LegacyLabelPair (#5454) ... * Add deprecated annotation to LegacySample and LegacyLabelPair Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Replace annotation with more generic docstring Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>	4 years ago
Sandeep Sukhani	9f301cf87d	use the defined querypages callback in all the types implementing QueryPages function from IndexClient interface (#5463)	4 years ago
Cyril Tovena	e3db2671f4	Do not insert point when missing when sharding (#5444)	4 years ago
Owen Diehl	c36272e31e	Tsdb/idempotent builder (#5424) ... * builder chunks uses heap * idempotent index builder * remove early return * pr suggestions * more thorough equality check on ChunkMetas.finalize	4 years ago
Benoît Knecht	776e229e94	pkg/ruler/base: Add external_labels option (#5450) ... This option can be used to add a set of labels to all alerts emitted by the ruler. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>	4 years ago
Sandeep Sukhani	333d5da0d0	add a metric for tracking the duration it took to sync the tables in queriers/index-gateways (#5446) ... * add a metric for tracking the duration it took to sync the tables in queriers/index-gateways * changes suggested from PR review	4 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	4 years ago
Sandeep Sukhani	00ca1aa99f	do not send grpc responses concurrently from index-gateway (#5445) ... See https://github.com/grpc/grpc-go/blob/master/stream.go#L120-L123	4 years ago
Sas Swart	c80244a3e4	Parse duration expressions in accordance with promql (#5275) ... * Add support for Prometheus like duration units in logql * Remove redundant type conversion * deduplicate call to sb.String Add a comment to explain why both promql and time's ParseDuration methods are used * extract duration parsing to separate function. * Add tests to ensure promql duration units are supported * Sort duration rune comment and list of runes in ascending order of length.	4 years ago
Ed Welch	4ca1fd0adb	set match_max_concurrent true by default (#5435) ... * set match_max_concurrent true by default Signed-off-by: Edward Welch <edward.welch@grafana.com> * add note to upgrade guide, fix up some of the existing notes Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Cyril Tovena	07e5eb33d3	Do not use WaitGroup context for StepEvaluator (#5425)	4 years ago
Cyril Tovena	5d8bc6129e	Correctly sets hash value for headblock iterator (#5423)	4 years ago
Owen Diehl	137a28d82b	Adds bytes and entries to chunk metadata in tsdb (#5414) ... * adds bytes and entries to chunk metadata in tsdb * removes comment	4 years ago
Travis Patterson	d8780da454	Revert to working version of Azure client (#5413) ... * Revert "storage: update azure blobstore client to use new sdk (#5302)" This reverts commit 89b04a97b3. * Fix Azure SDK Issue - Revert SDK to working version - Introduce patches to prevent deadlocks * make metrics global	4 years ago
Michel Hollands	d8ed87d04c	Update dskit version (#5392) ... * Update dskit version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add debug printing to see interfaces in CI. This will be reverted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to remember to remove Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use netutil.PrivateNetworkInterfacesWithFallback for the other rings Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Move import to place indicated by linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove debug printing Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Determine local interfaces in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rearrange import to satify linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update build image in CircleCI Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change changelegog message to show customer visible changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add doc tag and use private network interfaces in common config Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment about filtered out non private IP addresses Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the documentation of the interface names settings Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	b4bb8a1a46	Fixes double metrics registration in reporting (#5407) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	1837c9e0b2	Experimental TSDB index (#5376) ... * moves encoding to own pkg * decwrap impl * forks tsdb index * renames tsdb_enc * wraps all tsdb decbufs * use loki encbuf * custom chunk meta replacing ref-id for uint32 checksum * handle overlapping chunks in tsdb encoding * tsdb builder impl * tsdb querier beginnings to use modified tsdb * fixes nonexistent reference * lint/remove unused * adds tsdb testware * lint	4 years ago
Cyril Tovena	9b6f17f168	Add the total number of subqueries to query statistics (#5397) ... * Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Travis Patterson	7378b697f8	Fix Azure issue where 404 not recognized (#5399)	4 years ago
Sas Swart	64589d0339	Logql: Correct grouping type in pkg/logql/expr.y (#5306) ... * Correct grouping type in pkg/logql/expr.y * Use loki-build-image to build loki instead of golang image * Include goyacc and other required tools to allow Makefile to run * Remove dependencies not relevant to yacc in cmd/loki/Dockerfile * Use go install instead of go get to install goyacc in cmd/loki/Dockerfile * Add goyacc to logcli and loki-canary Dockerfiles * Rerun CI * Rerun CI * Revert "Add goyacc to logcli and loki-canary Dockerfiles" This reverts commit 0ee887c57f. * Remove goyacc from cmd/loki/Dockerfile * Formatting of cmd/loki/Dockerfile	4 years ago
Kaviraj Kanagaraj	d970e737ab	Add more context to tailer-> ingester connect error. (#5394) ... NOTE: When client is tailing, Querier **has** to create grpc connections to **all** the ingesters. Sometimes when ingester is rolloing out and Querier unable to connect to ingester when client is tailing, then error message reported by Querier is confusing. This PR add some context to it. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Ed Welch	b3af2dbeef	Loki: include the component in metrics.go log line (#5379) ... * add a component key to our metrics.go line Signed-off-by: Edward Welch <edward.welch@grafana.com> * new computer... who dis Signed-off-by: Edward Welch <edward.welch@grafana.com> * one more lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Sandeep Sukhani	e65d1b8c69	add more context to logs, improve comments for clarity in boltdb-shipper code (#5341)	4 years ago
Cyril Tovena	4fafc9c234	Fixes a flaky hedging test. (#5389) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	da9b4c9369	cleanup common index set after cleaning up all the user index sets (#5372)	4 years ago
Cyril Tovena	d44b6892eb	Fixes memberlist usage report (#5369) ... * Fixes memberlist usage report Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the linter and improve comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	21fa187fa8	Improve report usage to never fail (#5364) ... * Improve report usage to never fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wait until cancelled for the service to not fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361) ... * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	4 years ago
gotjosh	02416736d0	Ruler: Rule group not found API message (#5362) ... * Ruler: Rule group not found API message We copied over the ruler code as part of #5089, and it seems that at the same time we stopped depending on the object storage implementation of Cortex. Turns out, there are (slight) differences between the clients which meant that response returned by the API when we try to get a rule group that is not found had changed. This ensures that is consistent with the error assertion that we have in the code for the ruler. * Use the method from the interfaace to determine object storage * also include DeleteObject * appease the linter	4 years ago
Sandeep Sukhani	7fafe94e1a	add a ms instead of ns to end time of the rebound chunk interval (#5351) ... * add a ms instead of ns to end time of the rebound chunk interval conversion between ms and ns loses ns precision which causes us to not include all the logs * improve error when failing to rebound chunk	4 years ago
Sandeep Sukhani	0afd113c96	fix rounding of time to milliseconds (#5352) ... Current implementation of RoundToMilliseconds is broken due to improper handling of floating point operation in go. Fixing it by using simple division and modulo operators	4 years ago
Karsten Jeschkies	8e9009a7e3	Speed up `EntrySortIterator` by 20%. (#5318) ... **What this PR does / why we need it**: This patch replace the underlying heap of the `EntrySortIterator`. Instead of using a min or max heap the entry iterators are sorted at the beginning. Once the iteration starts only the first entry iterator is re-positioned depending on its current value. This will yield the same runtime complexity as the current heap based solution. However, since the new algorithm accesses the first element it can be cached. Thus the number of access calls to `Entry().Timestamp` are halved. ``` › benchstat before.txt after.txt name old time/op new time/op delta SortIterator/merge_sort-16 4.40ms ± 4% 4.38ms ± 3% ~ (p=0.400 n=9+10) SortIterator/sort-16 3.65ms ± 2% 2.80ms ± 3% -23.25% (p=0.000 n=9+10) ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Dylan Guedes	a3bacd5739	Test `LegacySample` and `LegacyLabelPair` compatibility. (#5313) ... The expected values were built from Cortex' `cortexpb.Sample` and `cortexpb.LabelPair`. They prove that `logproto.LegacyLabelPair` and `logproto.LegacySample` are not compatible with `logproto.LabelPair` and `logproto.Sample`, requiring us to keep both implementations for now.	4 years ago
Sandeep Sukhani	e4d6767412	Fix apply retention issue (#5342) ... * call mark phase started/finished only when applying retention * correctly pass userID to IntervalMayHaveExpiredChunks call on retention and dletion expiry check	4 years ago
Danny Kopping	20041b74d6	Revert "Simplify code for write path in distributor (#5248)" (#5337) ... This reverts commit 10b93cebe5. We discovered that these changes can lead to log deduplication issues. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Kaviraj Kanagaraj	2db309271c	Makes `tailer.droppedStreams` slice bounded. (#5334) ... * Makes `tailer.droppedStreams` slice bounded. For slow receiver, this dropped streams may cause ingester memory to keep increasing. Sometimes OOMing the ingester. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make droppedStreams slice size configurable Update tests Adding logging when slice is dropped Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for dropped stream Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Bryan Boreham	a50cac7674	filters: use faster regexp package (#5315) ... * regexp-filter: improve benchmark Add a few more cases based on real-world usage. Also simplify the loop to just run the number of times requested. Go benchmarks run 'N' times to reach a duration, default 1 second. Previously the benchmark was running N*1000000 times, which took 7 seconds minimum for some cases. * regexp filter: use modified package with optimisations See https://github.com/grafana/regexp/tree/speedup#readme Includes the following changes proposed upstream: * [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711) * [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909) * [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756) * [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789) * [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294) * Add grafana/regexp to vendor directory	4 years ago