129 Commits (87b7dfae870a301ec1d57e1373d2d6cc433c345d)

Author	SHA1	Message	Date
Sandeep Sukhani	87b7dfae87	Stream query responses from boltdb index client (#5468) ... * stream query response from boltdb index client instead of sending one row per batch * add benchmark for testing index gateway query performance * lint and fix broken test * wrap error message while failing to send back response * sort imports	3 years ago
Sandeep Sukhani	55acdbd08f	boltdb shipper index gateway client improvements (#5456) ... Index Gateway query performance improvements by bounding the number of concurrent grpc calls we make from the client.	3 years ago
Sandeep Sukhani	9f301cf87d	use the defined querypages callback in all the types implementing QueryPages function from IndexClient interface (#5463)	3 years ago
Sandeep Sukhani	333d5da0d0	add a metric for tracking the duration it took to sync the tables in queriers/index-gateways (#5446) ... * add a metric for tracking the duration it took to sync the tables in queriers/index-gateways * changes suggested from PR review	3 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	3 years ago
Sandeep Sukhani	00ca1aa99f	do not send grpc responses concurrently from index-gateway (#5445) ... See https://github.com/grpc/grpc-go/blob/master/stream.go#L120-L123	3 years ago
Sandeep Sukhani	e65d1b8c69	add more context to logs, improve comments for clarity in boltdb-shipper code (#5341)	3 years ago
Sandeep Sukhani	da9b4c9369	cleanup common index set after cleaning up all the user index sets (#5372)	3 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361) ... * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Sandeep Sukhani	7fafe94e1a	add a ms instead of ns to end time of the rebound chunk interval (#5351) ... * add a ms instead of ns to end time of the rebound chunk interval conversion between ms and ns loses ns precision which causes us to not include all the logs * improve error when failing to rebound chunk	3 years ago
Sandeep Sukhani	e4d6767412	Fix apply retention issue (#5342) ... * call mark phase started/finished only when applying retention * correctly pass userID to IntervalMayHaveExpiredChunks call on retention and dletion expiry check	3 years ago
Owen Diehl	af3457b92e	less opaque chunk keys on fs with v12 (#5291) ... * less opaque chunk keys on fs with v12 * lint	3 years ago
Sandeep Sukhani	40e05198f9	fix a panic in index-gateway caused by double closing of a channel (#5298)	3 years ago
Ed Welch	efffe0ce50	add some timing to the logs for downloading and opening index tables (#5042) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	f083aab09d	Remove `cortexpb` dependency from Loki packages. Use only `logproto` now. (#5264) ... * Getting rid of `cortexpb` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Get rid of `cortex/pkg/tenants` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * go mod vendor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename some import alias Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Sandeep Sukhani	37fcbf9428	use default retention period to check user index may have expired chunks when user does not have custom retention (#5261)	3 years ago
Dylan Guedes	4a70f8ec84	Copy `cortex/pkg/ingester/client` package dependency into Loki (#5164) ... * Fork ingester client. * Document magic numbers.	3 years ago
Eng Zer Jun	392fde1e27	refactor: use `T.TempDir` to create temporary test directory (#5253) ... * refactor: use `T.TempDir` to create temporary test directory The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: update CHANGELOG.md Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Sandeep Sukhani	9fdcacff15	v12 chunk format handling in retention (#5254) ... Also add some tests and safety mechanism to avoid dropping whole table when we cant find any chunks	3 years ago
Callum Styan	bfef7bad44	Start refactor of storage (chunks) clients metrics into structs (#5057) ... * Refactor azure metrics to be a struct (+ add an overall client metrics struct), add an azure client metric for total egress bytes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add changelog entry for new azure metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixes based on Kavi's review. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix one more merge conflict that I missed in the merge commit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Sandeep Sukhani	0426870f3f	fix broken boltdb-shipper test (#5222)	3 years ago
Sandeep Sukhani	d87f2a1c8a	per user index changes on the write path (#5193) ... * per user index changes on the write path - add a flag to enable building per user index from the ingesters - when enabled, ingesters would build boltdb files with buckets per tenant	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Sandeep Sukhani	88b5a767bc	changes on read path for supporting per user index with boltdb-shipper (#5073)	3 years ago
Sandeep Sukhani	49ffe5291d	add objects list caching for boltdb-shipper index store to reduce object storage list api calls (#5160)	3 years ago
Piggy	e5dcadee8a	Fix test in retention/expiration_test.go on slow machines (#5137) ... Signed-off-by: Piggy NL <piggynl@outlook.com>	3 years ago
JordanRushing	291360ddcb	Simpler new chunk key v12 (#5054) ... * starts hacking new chunk paths * Create new chunk key for S3; update parsing and add basic test and benchmark Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Fix `TestGrpcStore` to work with new chunk key structure Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Add a function to SchemaConfig that returns the correct PeriodConfig for a given time. Signed-off-by: Callum Styan <callumstyan@gmail.com> Add schema config to object client so it we can use the right external key function based on the schema version. Signed-off-by: Callum Styan <callumstyan@gmail.com> Fix failing compactor tests by passing SchemaConfig to chunkClient Remove empty conditional from SchemaForTime Define schema v12; wire-in ChunkPathShardFactor and ChunkPathPeriod as configurable values Add PeriodConfig test steps for new values ChunkPathShardFactor and ChunkPathPeriod in v12 Update test for chunk.NewExternalKey(); remove completed TODO Set defaults for new ChunkPathPeriod and ChunkPathShardFactor SchemaConfig values; change FSObjectClient to use IdentityEncoder instead of Base64Encoder Use IdentityEncoder everywhere we use FSObjectClient for Chunks Add ExternalKey() function to SchemaConfig; update ObjectClient for Chunks * Finish plumbing through the chunk.ExternalKey -> schemaConfig.ExternalKey change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up lint failures Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Clean up chunk.go and fix tests Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Break SchemaConfig.ExternalKey() into smaller functions Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Quickly fix variable name in SchemaConfig.newerExternalKey() Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Clean up ExternalKey conditional logic; add better comments; add tests Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Fix a bug where we are prepending userID to legacy Chunk keys but never parsing it; refactor key tests and benchmarks Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Add small SchemaConfig test Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Update docs and CHANGELOG.md Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Correctly return an error when failing to parse a chunk external key Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Revert IdentityEncoder to Base64Encoder after testing Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Fix assignment in test for linter Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Remove leftover comments from development Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Change v12 version comment format; remove redundant login in `ParseExternalKey()` Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Change remaining v12 comment style Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Pass chunk.SchemaConfig to new parallel chunk client * Simplify chunk external key prefixes for schema v12 * Fix broken benchmark; add benchmarks for old parsing conditional Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Remove unneeded lines from upgrading doc * Add benchmarks for root chunk external key parsing functions Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * memoizes schema number calculations * Resolve linter issue with PeriodConfig receiver name Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Sandeep Sukhani	ff47d7419a	compactor changes for building per user index files in boltdb shipper (#5026) ... * compactor changes for building per user index files in boltdb shipper * update comments * lint and changes suggested from PR review * fix broken code after merging main	3 years ago
Kaviraj	14afb13d36	Chore: Copy `cortex/util/math` into Loki (#5036) ... * Copy `cortex/pkg/util/math` into Loki. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Move `MinUint32` from `util` package into `util/math` package Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove NewEWMARate and related helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted spaces Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix spaces in import paths Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Cyril Tovena	8b71a44a86	Pre-allocate chunk buffer size (#4884) ... * Pre-allocate chunk buffer size Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add doc	3 years ago
Kaviraj	fe1d28a545	Chore: use `spanlogger` from `dskit` (#5029) ... * Move `cortex/pkg/spanlogger` into `loki`. It uses spanlogger from `dskit/pkg/spanlogger` with small wrapper created on the `pkg/util/spanlogger` in loki repo. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove `cortex/util/spanlogger` -> `loki/util/spanlogger` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter (with goimports) and `FromContext` Run goimports with `-local` flag `FromContext` takes util logger to keep the API same as currently used inside loki Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Cyril Tovena	03be8dac1e	Allow to disable http2 for GCS. (#4942) ... * Allow to disable http2 for GCS. This also tweaks our iddle connection pooling accross the board and reduce the insane parallelism when fetching chunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/chunk/gcp/gcs_object_client.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * update comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Reduce MaxIdleConns to a more standard value. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Allow to configure max parallelism for fetching chunks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests with no default config Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944)	4 years ago
Christian Haudum	e573a4d1a2	Change error responses from plain text to JSON (#4845) ... * Return HTTP errors consistently as JSON response The API returned errors in plain text where the error message was the response body. However, the content type header was set to `application/json`. To make the API responses consistent, errors are now returned as JSON as well. ``` { "message": string, "status": string, "code": int } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return HTTP errors consistently as JSON response Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Sandeep Sukhani	3ac9818f82	recreate compacted boltdb files from compactor when they are more than 12 hours old (#4853)	4 years ago
Cyril Tovena	54dc737583	Reduce tracing spans (#4842) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830) ... * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Sandeep Sukhani	a707ced67f	index cleanup fixes while applying retention (#4741) ... * Add metric name to the label if missing for building index entries for deletion * fix calculation of table interval from table name * add changelog * add test for series cleanup without __name__ label	4 years ago
Sandeep Sukhani	02d88462f3	allow applying retention at different interval than compaction with a config (#4736) ... * allow applying retention at different interval than compaction with a config * add changelog entry	4 years ago
Sandeep Sukhani	20726e6ba1	fix flaky retention tests (#4697)	4 years ago
Trevor Whitney	bfc77118a1	Update local and docker config for simple scalable deployment (#4660) ... * first draft of local config * remove a few more defaults * interface names fix means we do not need lifecycler ip * update rings to use collectors/ prefix fix replication factor chunks-retain-period default to 0s fix a bug introduced with a Go upgrade? on printing the schema date. * simplify the config for the ruler, synthesize the local directory * simplify loki-docker-config.yaml * change common filesystem config to have chunks and rules directory * now that we are calling register flags on the common ring config we set more values and some of the tests needed to be tweaked. updated a few paths as well. * fix tests Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641)	4 years ago
Trevor Whitney	07ea1ed99c	Add common ring configuration (#4617) ... * add ring section to common config Co-Authored-By: Dylan Guedes <djmgguedes@gmail.com> * fix compactor test to append loopback interface it ifnames in test if present * add loopback interface to compactor ring iface names * clean-up from PR review Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> * only appy ingester config to rings with no non-default properties Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Sandeep Sukhani	2564a810c5	Logs deletion fixes (#4625) ... * check all tables when we have delete requests to process since chunks could span multiple tables * upload modified index when the it is modified without dropping the chunk * fix broken test * lint * Fix modified check Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * suggested changes from PR review * add changelog Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	308af871cf	uses ring.Write instead of ring.WriteNoExtend for compactor ring checks (#4615)	4 years ago
Ed Welch	0c9beadaaa	Loki: Add a ring to the compactor used to control concurrency when not running standalone (#4574) ... * add a ring to the compactor * wiring up the start/stop functionality * add some monitoring for multiple compactors running at the same time. * Add compactor to `read` target * fix tests/validation requirements * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * used a shared ring_config between scheduler and compactor to avoid some duplication * fix lint * better handling of the tokens file with a way to easily tell all the rings to persist tokens * fix query scheduler replication factor * removing "scheduler" from log line which may be called by components other than a scheduler in the future. * update changelog Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
JordanRushing	ca67292b54	Introduce `overrides-exporter` module to Loki (#4520) ... * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Use reflection; add more limits to export * Fix import to satisfy linter * refactors TenantLimits for clarity & efficiency. Also fixes tenantLimits nil check * Fix breaking test with nil check in runtime_config Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484) ... * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Arve Knudsen	cdde2cf050	Chore: Use dskit/grpcclient (#4312)	4 years ago