1523 Commits (0470e4e7d38e7b364acfd230b20f4237c8ef05d0)

Author	SHA1	Message	Date
Travis Patterson	1c5e094e16	Only delete data when WholeStreamDeletion or FilterAndDelete (#6286) ... * Only delete data when WholeStreamDeletion or FilterAndDelete * remove broken snyn workflow	3 years ago
Ed Welch	f52fc9d907	Loki: add a run-once flag to the compactor (#6281) ... * add a run once flag to the compactor to allow it to cleanup index files with a single manual invocation. Useful when using migrate tool. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	b5f0557e79	fix boltdb shipper local query in logcli and support `fake` tenant (#6282) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Simon	7f50a26656	Move ingester metrics around (#6275)	3 years ago
Sandeep Sukhani	2d4ff73cc2	add metrics for tracking index shipper operations (#6278)	3 years ago
Bayan Taani	50ef4f2ebd	Add duration filter unit test (#6271)	3 years ago
Sandeep Sukhani	7369b3785c	use tsdb config for initializing index gateway client (#6267) ... * use tsdb config for initializing index gateway client * review feedback Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	9b2786bde3	Fix delete updates (#6194) ... * Fix delete updates * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_client.go Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> * Add compactor address to the storage config * review comments * Review feedback Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Steve Hipwell	7cc6aa4021	feat: Add support for Azure user assigned identity (#5891) ... Signed-off-by: Steve Hipwell <steve.hipwell@gmail.com>	3 years ago
Kaviraj Kanagaraj	a795b93897	chore(cache): Make exprimental messages more clear for cache. (#6219) ... Currently we see following message when any of the experimental feature is used for cache (say fifocache or redis). But the problem it doesn't say __what_kind__ of cache is it exactly. So this leads to confusion to track what kind of cache is the source of the experiment feature. This PR fixes it by adding more clear message. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Aditya C S	410c1a0d38	feat(ruler): support alert relabeling (#6220)	3 years ago
Sandeep Sukhani	6ac955cc7d	make boltdb-shipper use the generic indexshipper for managing index in object storage (#6226)	3 years ago
Callum Styan	3d2282745b	Runtime reloadable config; ring migration setup (#6214) ... * Clean up setting of memberlist and multikv config for Loki services that use the ring. Also sets the multi client runtime config function for all services that use the ring. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add a test for the multi kv setup Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Owen Diehl	c032da9639	Tsdb/inverted index (#6233) ... * begins adding a factor-of-two dynamically sharded inverted index * bit prefix inverted index tests * goimports	3 years ago
Travis Patterson	e313d8a873	Remove __name__ from label calls (#6229)	3 years ago
Cyril Tovena	67e7285bf5	Improve labels computation during LogQL pipeline (#6110) ... * Improve labels computation during LogQL pipeline * Fixes a bug that was mutating the base labels	3 years ago
Susana Ferreira	0dfd2cbbcc	Fix label extraction pushdown on instant queries (#6215) ... * Add downstream label extraction tests * Fix label extraction downstream pushdown	3 years ago
Sandeep Sukhani	1923668a5d	run store queries in ingester when using tsdb as index store (#6209)	3 years ago
Owen Diehl	f14f9dd8c4	prevents sharding avg and avg_over_time when their children arent shardable. (#6204)	3 years ago
Dylan Guedes	440fb7d74c	Loki: Modifies TableManager to use IndexGateway ring (#5972) ... * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Use boundaries to skip users in TableManager. - Separate the assigning of indexClient in the IndexGateway to allow initializing the shipper with the IndexGateway ring - Add new TenantBoundariesClient entity, that answers if a given tenantID should be ignored or not - Use the TenantBoundariesClient implemented by the IndexGateway in the downloads TableManager * Add IndexGateway configuration docs. * Add tests for pkg/util/ring TokenFor() and IsAssignedKey() Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Small whitespace fix in pkg/util/ring_test.go Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Apply suggestions from code review Rewrite QueryIndex error phrase. Co-authored-by: JordanRushing <rushing.jordan@gmail.com> * Join users list in a single string. - This is necessary since go-kit doesn't support array type. * Tweak queryReadiness log messages. - As suggested by Ed on https://github.com/grafana/loki/pull/5972#discussion_r859734129 and https://github.com/grafana/loki/pull/5972#discussion_r859736072 * Implement new RingManager. - Adds a new entity to indexgateway named RingManager, responsible for managing the ring and the lifecycler used by the indexgateway. The ringManager is guaranteed to be initiatiated before the Shipper and before the IndexGateway. - Remove the readiness logic from the IndexGateway. - Pass the RingManager as the TenantBoundaries implementation of the Shipper * Return non-empty ringmanager for all modes. * Fix lint regarding error not being check. * Fix lint due to wrong import order. * Implement support for client and server mode for the ring manager. * Fix ring manager services registration. * Add option to configure whether or not to log gateway requests. * Check if IndexGateway is enabled instead of active. * Tune RingManager behavior. - Instantiate ring buffers inside IsAssignedKey instead of reusing same buffer to avoid issues with concurrency. - Remove unnecessary details from IndexGateway mode docs. - Return error when wrongly instantiating a RingManager when IndexGateway is in simple mode. * Rewrite `TenantInBoundaries` as a func instead of interface. * Fix lint. - Fix YAML tag - Remove else clause * Use distributor instead of querier in test. - Since the querier needs to wait for the index gateway ring, it isn't suitable for this test anymore. * Add ring mode guard clause. * Return code erased by mistake. * Rename TenantInBoundaries to IndexGatewayOwnsTenant. * Log len of users instead of all of them to avoid log msgs. * Add docstrings to new public functions. * Apply suggestions from code review Document that tenant filtering is only applied during query readiness. Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Modify IsAssignedKey to expect address directly instead of lifecycler. - Also removes HasDedicatedAddress, since it isn't necessary anymore * Log error message when IsInReplicationSet fails. * Modify IsAssignedKey to return true by default. * Remove wrong assigning of Service for clientMode. * Log gateway requests before error checks. * Remove unnecessary endpoint registration. * Fix lint. * Update pkg/util/ring.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Karen Miller	b90c460769	Rename fudge_duplicate_timestamp to be increment_duplicate_timestamp (#6120) ... * Rename fudge_duplicate_timestamp to be increment_duplicate_timestamp * run `gofmt -d -w pkg/validation/limits.go` Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Owen Diehl	3c334001cd	creates a single tsdb for entire wal recovery (#6203) ... * creates a single tsdb for entire wal recovery * tsdb wal replay builds TSDBs with unrounded timestamps to avoid clobbering	3 years ago
Sandeep Sukhani	ef4bc8f1ed	copy all the fixes/improvements that went into boltdb-shipper that are missing from generic index-shipper (#6200)	3 years ago
Christian Haudum	b073ec9017	security: Redact credentials when marshalled to YAML (#6186)	3 years ago
Christian Haudum	8518560cf2	Fix flaky gRPC server shutdown in tests (#6191) ... * Use `t.Cleanup(cleanup)` instead of `defer cleanup()` Which seems to reduce the likelyhood of the flaky gRPC server shutdown Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Do not fail test if gRPC server graceful shutdown fails Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sandeep Sukhani	62f5e598dd	tsdb index gateway (#6158) ... * enable new grpc calls on index gateway and add GetSeries call * use index gateway for tsdb index store queries * add missing file * lint * disable IndexGatewayClientStore for stores other than tsdb until we are ready to enable it again	3 years ago
Owen Diehl	4f87db7f38	strips out tenant label from LabelNames queries (#6181) ... * strips out tenant label from LabelNames queries * prevent tenant labelvalues query	3 years ago
Daniel Kovacs	e9d2a30733	Fixed reader to support both authentication and tenant-id at the same time. (#5719)	3 years ago
Owen Diehl	6b97de765e	tsdb creation supports fingerprint overrides (#6178)	3 years ago
Sandeep Sukhani	ead8c345e6	skip empty object key returned by some s3 compatible storage while listing boltdb-shipper index (#6123)	3 years ago
Kaviraj Kanagaraj	e5a89fa6da	fix(querier): Record `result` length after response validation. (#6176) ... Without this fix, it causes panic, if series or label endpoint returns error. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	e3a43c6972	query-scheduler: Now proper support for `querier_forget_delay` (#6174) ... * query-scheduler: Now proper support for `querier_forget_delay` Since now shuffle sharding is supported on the `query-path` in Loki, it makes sense to enable this config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update pkg/scheduler/scheduler.go Co-authored-by: Christian Simon <simon@swine.de> * Update docs/sources/configuration/_index.md Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Periklis Tsirakidis	6e9517e4b5	ruler: Add support for alertmanager header authorization (#6136)	3 years ago
Aditya C S	c6af640798	feat: support simple selection of labels in json expr parser (#6151)	3 years ago
Justin Burnham	97c7d7ca3c	LogQL: add the `default` sprig template function to logql label/line formatter (#6163) ... * add the `default` sprig template function to logql label/line formatter With the `default` sprig template function, we will be able to make it easier to output a default value if the source string is otherwise empty. Useful for json fields that can be missing, like HTTP headers in a log line that aren't required as in the following: ``` {job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}` ``` Previously this could only be done (that I found) with the much more verbose ``` {{regexReplaceAllLiteral "^$" .http_request_headers_x_forwarded_for "-"}} ``` * update changelog entry * add pr number * enhance docs * Update docs/sources/logql/template_functions.md Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	3 years ago
Marco Cadetg	73b39a97bc	specify logcli authorization header by flag (#6164) ... * specify logcli authorization header by flag #6144 * default AuthHeader to Authorization in client lib * use constant for default auth header * for consistency use LOKI_AUTH_HEADER	3 years ago
Yusuf Papurcu	dc782de864	Removed double space in Log (#6166)	3 years ago
Sandeep Sukhani	6e3bf3f7d0	tsdb query fixes (#6142) ... * query both downloads and uploads manager when non-nil * fix reading of index file with nodename having a `-` * remove __name__ matcher at a layer above to avoid passing it during tsdb head queries	3 years ago
Ed Welch	4bf1648da1	Loki: When processing logs during tailing, don't reuse the query pipeline (#6152) ... * when tailing, don't reuse the query pipeline because it has an unbounded label cache Signed-off-by: Edward Welch <edward.welch@grafana.com> * update changelog Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	ee4ab1b394	Fix certain binary expression queries by optimizing through push down (#6132) ... * Add test cases that fail Instant queries of the following type fail and return an `unimplemented` error: ``` sum(count_over_time({foo="bar"} | logfmt | duration > 2s [3s])) / sum(count_over_time({foo="bar"} [3s])) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix certain binary expressions for instant queries If either the left hand side or the right hand side of a binary expression is a noop, we need to return the original expression so the whole expression is a noop as well, and thus not executed using the downstream engine. Otherwise, a binary expression that has a noop on either side, results in an `unimplemented` error when executed using the downstream engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Optimize instant vector aggregations with log extraction stage This change optimizes queries that use a vector aggregation without grouping around a range aggregation with a label extraction stage such as `json` or `logfmt`. Since the vector aggregation can be pushed down to the downstream query, the downstream query does not create a massive amount of streams, even though it contains a generic label extraction stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	24f8b19e91	Remove unused grpc health check endpoints (#6113) ... We have generic `grpc_healthcheck` endpoint that get registered for all the components during initialization stage. Looks like these health checks that are part of actual component's struct itself came from legacy. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Owen Diehl	729bf3fc91	loads previously created TSDBs into shipper on startup (#6117) ... * loads previously created TSDBs into shipper on startup * noopTSDBManager impl * info logs for initial tsdb loading	3 years ago
Owen Diehl	756fc19577	fix mutability bug with labels during tsdb flushing (#6116)	3 years ago
Owen Diehl	55ea89f9fd	Tsdb/head wal feedback (#6107) ... * pr feedback, improved mutex unlocking * tenant heads use sync.Map * tenant heads bench * Revert "tenant heads use sync.Map" This reverts commit 64adebc535. * removes old code the removed TSDBs outside the shipper + minor fixes	3 years ago
Sandeep Sukhani	06e309bae6	do not increment sync operation count metric on per index set (#6109)	3 years ago
arcosx	e2fdc3437b	Feat: add Baidu Cloud BOS as storage backends for Loki #4788 (#5848) ... * feat: add baidu bce bos storage support Signed-off-by: arcosx <arcosx@outlook.com> * add baidu bce bos as chunk storage backend Signed-off-by: arcosx <arcosx@outlook.com> * fix: some doc error && rewrite bad code Signed-off-by: arcosx <arcosx@outlook.com> * fix: add the `BceServiceError` source link Signed-off-by: arcosx <arcosx@outlook.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: arcosx <arcosx@outlook.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Christian Haudum	18058d2b55	Serve correct status of ingester/distributor in gRPC healthcheck endpoint (#6094) ... * Serve correct status of ingester in gRPC healthcheck endpoint The gRCP healthcheck endpoint is used by clients that determin whether a connection to the service is healthy or not. In case the ingester is not in "running" state and is not "active" in the ring, this healthcheck should return "not serving". Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Serve correct status of distributor in gRPC healthcheck endpoint	3 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049) ... * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Ed Welch	4f75939efb	Revert "Loki: Change live tailing to only allow mutating the log line not the number of streams. (#6063)" (#6097) ... This reverts commit 98fda9b0a0.	3 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971) ... * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago