3481 Commits (67e7285bf5f3f22d298b92b2a715760d66225460)

 

All Branches   

Search

Author	SHA1	Message	Date
Cyril Tovena	67e7285bf5	Improve labels computation during LogQL pipeline (#6110) ... * Improve labels computation during LogQL pipeline * Fixes a bug that was mutating the base labels	3 years ago
Susana Ferreira	0dfd2cbbcc	Fix label extraction pushdown on instant queries (#6215) ... * Add downstream label extraction tests * Fix label extraction downstream pushdown	3 years ago
Periklis Tsirakidis	fd627f2a49	operator: Add rules support (#5986)	3 years ago
Michael Sverdlik	5206e3acd0	Docs: Hint about potential high cardinality metrics (#5603) ... * Hint about potential high cardinality metrics Add warning about potential high cardinality metrics due to the file path being included in some of the metrics. Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus. This can result in increased storage, slow down of queries, extra costs and so on. Address what has been discussed in #5553. * Reword observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Gábor Farkas	0075085e9a	docs: describe ordering in http query results (#6024) ... * docs: describe ordering in http query results * improved text Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * better text Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * better text Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Joan López de la Franca Beltran	5e465ad74e	Docs: Fix design docs index page (#6008) ... * Fix design docs index * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Sandeep Sukhani	1923668a5d	run store queries in ingester when using tsdb as index store (#6209)	3 years ago
Owen Diehl	f14f9dd8c4	prevents sharding avg and avg_over_time when their children arent shardable. (#6204)	3 years ago
Christian Haudum	d5c337c4d3	docs: Fix broken link from unpack parser to pack stage (#6207) ... and make hyperlinks more obvious Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	a58dc3f6a8	doc(storage-example-configs): AWS single store configs (#6196) ... * doc(storage-example-configs): AWS single store configs Our example configs are bit misleading here. Particularly AWS deployment. https://grafana.com/docs/loki/latest/storage/#examples As a new user, I come here and all I can conclude is, we cannot use s3 as a single store under AWS deployment. Doc seems to recommend AWS deployment always needs S3 + DynamoDB. (in fact it was real confusion for new user from the community) https://grafana.slack.com/archives/CEPJRLQNL/p1652867888720979 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update schema_config.configs.store Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	440fb7d74c	Loki: Modifies TableManager to use IndexGateway ring (#5972) ... * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Use boundaries to skip users in TableManager. - Separate the assigning of indexClient in the IndexGateway to allow initializing the shipper with the IndexGateway ring - Add new TenantBoundariesClient entity, that answers if a given tenantID should be ignored or not - Use the TenantBoundariesClient implemented by the IndexGateway in the downloads TableManager * Add IndexGateway configuration docs. * Add tests for pkg/util/ring TokenFor() and IsAssignedKey() Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Small whitespace fix in pkg/util/ring_test.go Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Apply suggestions from code review Rewrite QueryIndex error phrase. Co-authored-by: JordanRushing <rushing.jordan@gmail.com> * Join users list in a single string. - This is necessary since go-kit doesn't support array type. * Tweak queryReadiness log messages. - As suggested by Ed on https://github.com/grafana/loki/pull/5972#discussion_r859734129 and https://github.com/grafana/loki/pull/5972#discussion_r859736072 * Implement new RingManager. - Adds a new entity to indexgateway named RingManager, responsible for managing the ring and the lifecycler used by the indexgateway. The ringManager is guaranteed to be initiatiated before the Shipper and before the IndexGateway. - Remove the readiness logic from the IndexGateway. - Pass the RingManager as the TenantBoundaries implementation of the Shipper * Return non-empty ringmanager for all modes. * Fix lint regarding error not being check. * Fix lint due to wrong import order. * Implement support for client and server mode for the ring manager. * Fix ring manager services registration. * Add option to configure whether or not to log gateway requests. * Check if IndexGateway is enabled instead of active. * Tune RingManager behavior. - Instantiate ring buffers inside IsAssignedKey instead of reusing same buffer to avoid issues with concurrency. - Remove unnecessary details from IndexGateway mode docs. - Return error when wrongly instantiating a RingManager when IndexGateway is in simple mode. * Rewrite `TenantInBoundaries` as a func instead of interface. * Fix lint. - Fix YAML tag - Remove else clause * Use distributor instead of querier in test. - Since the querier needs to wait for the index gateway ring, it isn't suitable for this test anymore. * Add ring mode guard clause. * Return code erased by mistake. * Rename TenantInBoundaries to IndexGatewayOwnsTenant. * Log len of users instead of all of them to avoid log msgs. * Add docstrings to new public functions. * Apply suggestions from code review Document that tenant filtering is only applied during query readiness. Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Modify IsAssignedKey to expect address directly instead of lifecycler. - Also removes HasDedicatedAddress, since it isn't necessary anymore * Log error message when IsInReplicationSet fails. * Modify IsAssignedKey to return true by default. * Remove wrong assigning of Service for clientMode. * Log gateway requests before error checks. * Remove unnecessary endpoint registration. * Fix lint. * Update pkg/util/ring.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Karen Miller	b90c460769	Rename fudge_duplicate_timestamp to be increment_duplicate_timestamp (#6120) ... * Rename fudge_duplicate_timestamp to be increment_duplicate_timestamp * run `gofmt -d -w pkg/validation/limits.go` Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Owen Diehl	3c334001cd	creates a single tsdb for entire wal recovery (#6203) ... * creates a single tsdb for entire wal recovery * tsdb wal replay builds TSDBs with unrounded timestamps to avoid clobbering	3 years ago
Sandeep Sukhani	ef4bc8f1ed	copy all the fixes/improvements that went into boltdb-shipper that are missing from generic index-shipper (#6200)	3 years ago
Gerard Vanloo	5eb4c0d66a	operator: Changing GCP secret file path (#6199) ... * Changing GCP secret file path * Update changelog	3 years ago
irizzant	bb2eccce3c	[loki microservices]: allow custom gateway server configuration (#6138) ... * feat(loki)!: allow custom server configuration Signed-off-by: irizzant <i.rizzante@gmail.com> * doc(loki): fix documentation * refactor: move server_snippet under gateway.libsonnet Signed-off-by: irizzant <i.rizzante@gmail.com>	3 years ago
Michel Hollands	e5eff96266	Change deletion_enabled setting to deletion_mode (#6190) ... * Change deletion_enabled setting to deletion_mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Empty commit	3 years ago
Dylan Guedes	fe7ced807b	Fix CHANGELOG entries. (#6193) ... - The compactor address entry wasn't part of v2.5.0, so I moved it to the main section instead - Two PRs were linking to a wrong PR link	3 years ago
Christian Haudum	b073ec9017	security: Redact credentials when marshalled to YAML (#6186)	3 years ago
Christian Haudum	8518560cf2	Fix flaky gRPC server shutdown in tests (#6191) ... * Use `t.Cleanup(cleanup)` instead of `defer cleanup()` Which seems to reduce the likelyhood of the flaky gRPC server shutdown Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Do not fail test if gRPC server graceful shutdown fails Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sandeep Sukhani	62f5e598dd	tsdb index gateway (#6158) ... * enable new grpc calls on index gateway and add GetSeries call * use index gateway for tsdb index store queries * add missing file * lint * disable IndexGatewayClientStore for stores other than tsdb until we are ready to enable it again	3 years ago
Owen Diehl	4f87db7f38	strips out tenant label from LabelNames queries (#6181) ... * strips out tenant label from LabelNames queries * prevent tenant labelvalues query	3 years ago
Michel Hollands	b6ff9d5889	Add integration test for delete request (#6188) ... * Add integration test for delete request Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add line filter Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Daniel Kovacs	e9d2a30733	Fixed reader to support both authentication and tenant-id at the same time. (#5719)	3 years ago
Alan Conway	3547d8948c	Fix typo in operator/docs/hack_operator_make_run.md (#6180)	3 years ago
Owen Diehl	6b97de765e	tsdb creation supports fingerprint overrides (#6178)	3 years ago
Sandeep Sukhani	ead8c345e6	skip empty object key returned by some s3 compatible storage while listing boltdb-shipper index (#6123)	3 years ago
Kaviraj Kanagaraj	e5a89fa6da	fix(querier): Record `result` length after response validation. (#6176) ... Without this fix, it causes panic, if series or label endpoint returns error. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	e3a43c6972	query-scheduler: Now proper support for `querier_forget_delay` (#6174) ... * query-scheduler: Now proper support for `querier_forget_delay` Since now shuffle sharding is supported on the `query-path` in Loki, it makes sense to enable this config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update pkg/scheduler/scheduler.go Co-authored-by: Christian Simon <simon@swine.de> * Update docs/sources/configuration/_index.md Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Periklis Tsirakidis	6e9517e4b5	ruler: Add support for alertmanager header authorization (#6136)	3 years ago
irizzant	6fb86c4def	fix(tanka): add customizable common env vars (#6124) ... Signed-off-by: Ivan Rizzante <i.rizzante@gmail.com>	3 years ago
Sergio Rua	069156380f	Update SSL config for cassandra (#6145) ... Documentation was missing the references to the SSL key and cert	3 years ago
Aditya C S	c6af640798	feat: support simple selection of labels in json expr parser (#6151)	3 years ago
dependabot[bot]	c6b7752bbb	Bump golangci/golangci-lint-action from 3.1.0 to 3.2.0 (#6170) ... Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.1.0 to 3.2.0. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.1.0...v3.2.0) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Justin Burnham	97c7d7ca3c	LogQL: add the `default` sprig template function to logql label/line formatter (#6163) ... * add the `default` sprig template function to logql label/line formatter With the `default` sprig template function, we will be able to make it easier to output a default value if the source string is otherwise empty. Useful for json fields that can be missing, like HTTP headers in a log line that aren't required as in the following: ``` {job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}` ``` Previously this could only be done (that I found) with the much more verbose ``` {{regexReplaceAllLiteral "^$" .http_request_headers_x_forwarded_for "-"}} ``` * update changelog entry * add pr number * enhance docs * Update docs/sources/logql/template_functions.md Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	3 years ago
Marco Cadetg	aa3f06f36b	update auth header env variable usage help (#6169)	3 years ago
Michel Hollands	f39552c7fa	Add docs for updated deletion api (#5763) ... * Add docs for updated deletion api Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix parameter name Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clarify that only the delete requests for the current tenant are shown Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update docs/sources/api/_index.md Co-authored-by: Christian Simon <simon@swine.de> * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Specify all delete endpoints use authentication Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add description of delete modes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Marco Cadetg	73b39a97bc	specify logcli authorization header by flag (#6164) ... * specify logcli authorization header by flag #6144 * default AuthHeader to Authorization in client lib * use constant for default auth header * for consistency use LOKI_AUTH_HEADER	3 years ago
psytester	40b2373398	corrected copy&paste error in list of labels (#6146) ... response_size was twice mentioned	3 years ago
Yusuf Papurcu	dc782de864	Removed double space in Log (#6166)	3 years ago
Sandeep Sukhani	6e3bf3f7d0	tsdb query fixes (#6142) ... * query both downloads and uploads manager when non-nil * fix reading of index file with nodename having a `-` * remove __name__ matcher at a layer above to avoid passing it during tsdb head queries	3 years ago
Sashank Agarwal	301ea42314	operator: Add method to get authenticated from GCP (#6125)	3 years ago
Ed Welch	4bf1648da1	Loki: When processing logs during tailing, don't reuse the query pipeline (#6152) ... * when tailing, don't reuse the query pipeline because it has an unbounded label cache Signed-off-by: Edward Welch <edward.welch@grafana.com> * update changelog Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	ee4ab1b394	Fix certain binary expression queries by optimizing through push down (#6132) ... * Add test cases that fail Instant queries of the following type fail and return an `unimplemented` error: ``` sum(count_over_time({foo="bar"} | logfmt | duration > 2s [3s])) / sum(count_over_time({foo="bar"} [3s])) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix certain binary expressions for instant queries If either the left hand side or the right hand side of a binary expression is a noop, we need to return the original expression so the whole expression is a noop as well, and thus not executed using the downstream engine. Otherwise, a binary expression that has a noop on either side, results in an `unimplemented` error when executed using the downstream engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Optimize instant vector aggregations with log extraction stage This change optimizes queries that use a vector aggregation without grouping around a range aggregation with a label extraction stage such as `json` or `logfmt`. Since the vector aggregation can be pushed down to the downstream query, the downstream query does not create a massive amount of streams, even though it contains a generic label extraction stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Karen Miller	31ca108a3c	Docs: redo PR 5109 (#6119)	3 years ago
Irina	c2e43dd8f5	Bump k8s.io/api, k8s.io/client-go, k8s.io/apimachinery from 0.22.7 to 0.23.6 (#6037)	3 years ago
Kaviraj Kanagaraj	24f8b19e91	Remove unused grpc health check endpoints (#6113) ... We have generic `grpc_healthcheck` endpoint that get registered for all the components during initialization stage. Looks like these health checks that are part of actual component's struct itself came from legacy. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Owen Diehl	729bf3fc91	loads previously created TSDBs into shipper on startup (#6117) ... * loads previously created TSDBs into shipper on startup * noopTSDBManager impl * info logs for initial tsdb loading	3 years ago
Owen Diehl	756fc19577	fix mutability bug with labels during tsdb flushing (#6116)	3 years ago
Periklis Tsirakidis	41647d1302	operator: Ruler enhancement proposal (#5985)	3 years ago