3588 Commits (f91f6a354b5602a686d8db7e871de47467a38f17)

 

All Branches   

Search

Author	SHA1	Message	Date
Christian Haudum	ee4ab1b394	Fix certain binary expression queries by optimizing through push down (#6132) ... * Add test cases that fail Instant queries of the following type fail and return an `unimplemented` error: ``` sum(count_over_time({foo="bar"} | logfmt | duration > 2s [3s])) / sum(count_over_time({foo="bar"} [3s])) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix certain binary expressions for instant queries If either the left hand side or the right hand side of a binary expression is a noop, we need to return the original expression so the whole expression is a noop as well, and thus not executed using the downstream engine. Otherwise, a binary expression that has a noop on either side, results in an `unimplemented` error when executed using the downstream engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Optimize instant vector aggregations with log extraction stage This change optimizes queries that use a vector aggregation without grouping around a range aggregation with a label extraction stage such as `json` or `logfmt`. Since the vector aggregation can be pushed down to the downstream query, the downstream query does not create a massive amount of streams, even though it contains a generic label extraction stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Karen Miller	31ca108a3c	Docs: redo PR 5109 (#6119)	3 years ago
Irina	c2e43dd8f5	Bump k8s.io/api, k8s.io/client-go, k8s.io/apimachinery from 0.22.7 to 0.23.6 (#6037)	3 years ago
Kaviraj Kanagaraj	24f8b19e91	Remove unused grpc health check endpoints (#6113) ... We have generic `grpc_healthcheck` endpoint that get registered for all the components during initialization stage. Looks like these health checks that are part of actual component's struct itself came from legacy. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Owen Diehl	729bf3fc91	loads previously created TSDBs into shipper on startup (#6117) ... * loads previously created TSDBs into shipper on startup * noopTSDBManager impl * info logs for initial tsdb loading	3 years ago
Owen Diehl	756fc19577	fix mutability bug with labels during tsdb flushing (#6116)	3 years ago
Periklis Tsirakidis	41647d1302	operator: Ruler enhancement proposal (#5985)	3 years ago
Owen Diehl	55ea89f9fd	Tsdb/head wal feedback (#6107) ... * pr feedback, improved mutex unlocking * tenant heads use sync.Map * tenant heads bench * Revert "tenant heads use sync.Map" This reverts commit 64adebc535. * removes old code the removed TSDBs outside the shipper + minor fixes	3 years ago
Danny Kopping	8838530976	Document `/services` API endpoint (#6111) ... * Documented /services API endpoint Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Grammar Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Sandeep Sukhani	06e309bae6	do not increment sync operation count metric on per index set (#6109)	3 years ago
JordanRushing	45dd588d8f	Update consistent hash ring docs with new index gateway ring (#6041) ... * Update consistent hash ring docs with new index gateway ring Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Update docs/sources/fundamentals/architecture/rings.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
arcosx	e2fdc3437b	Feat: add Baidu Cloud BOS as storage backends for Loki #4788 (#5848) ... * feat: add baidu bce bos storage support Signed-off-by: arcosx <arcosx@outlook.com> * add baidu bce bos as chunk storage backend Signed-off-by: arcosx <arcosx@outlook.com> * fix: some doc error && rewrite bad code Signed-off-by: arcosx <arcosx@outlook.com> * fix: add the `BceServiceError` source link Signed-off-by: arcosx <arcosx@outlook.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: arcosx <arcosx@outlook.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Christian Haudum	18058d2b55	Serve correct status of ingester/distributor in gRPC healthcheck endpoint (#6094) ... * Serve correct status of ingester in gRPC healthcheck endpoint The gRCP healthcheck endpoint is used by clients that determin whether a connection to the service is healthy or not. In case the ingester is not in "running" state and is not "active" in the ring, this healthcheck should return "not serving". Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Serve correct status of distributor in gRPC healthcheck endpoint	3 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102) ... * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	3 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049) ... * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Travis Patterson	03153e89bf	clean up the pr diff message (#6098) ... * clean up the pr diff message * add dependency to clone-main to avoid possible race * don't start tests before everything is cloned * formatting change	3 years ago
Ed Welch	4f75939efb	Revert "Loki: Change live tailing to only allow mutating the log line not the number of streams. (#6063)" (#6097) ... This reverts commit 98fda9b0a0.	3 years ago
n4mine	14b0873f2b	Revert "Docs: fix pic location in `sources/fundamentals/architecture/deployment-modes.md` (#6089)" (#6093) ... This reverts commit 1d25e17b32.	3 years ago
Gábor Farkas	0a68bd2d4e	docs: update the labels timespan info (#6019) ... * docs: update the labels timespan info * updated docs Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * updated docs Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Travis Patterson	d97724e8dd	Introduce coverage to PR pipelines (#5357) ... * Add code coverage diff to CI * remove test package from coverage diff * review feedback * lint * Env variable test * faster experiment * drone vars * drone vars * how does the shell work? * add pr comment * escape json * report diff in comment * properly format json * quote post body * review feedback * add querier/queryrange to coverage	3 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971) ... * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Patrick Mansfield	b3f6f6b415	table_manager: Change some level.Info() logging to level.Debug() (#6076) ... There is excessive logging of "creating table" and "synching tables" messages, change them to use level.Debug() instead of level.Info() to limit the output. Fixes #1786	3 years ago
n4mine	1d25e17b32	Docs: fix pic location in `sources/fundamentals/architecture/deployment-modes.md` (#6089)	3 years ago
Kaviraj Kanagaraj	0a4bfbe4f7	Remove nosiy "inflight" requests log entries (#6074) ... We have enough metrics now (queue time, queue length and metrics.go statistics) that makes these log entries obselete. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Karen Miller	2264b73311	Docs: remove Cortex references (#6079) ... * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update	3 years ago
Sashank Agarwal	c76565af53	fix labeler (#6087)	3 years ago
Owen Diehl	93de7a7061	renames skip -> ok in ProcessString (#6064) ... * renames skip -> ok in ProcessString Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * streampipeline matches convention Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Kaviraj Kanagaraj	8b15632dd8	Fix(query-frontend): LokiCodec encode/decode works correctly with label values endpoint (#6084) ... When encode/decode `http.Request` into `querybase.Request` we re-create request.Path. Before this PR, both `/labels/` and `/labels/{name}/values` endpoint was decoding into same URL path `/labels/` before forwarding to the querier. So it always returns label names as response. This bug became visible after enabling all the middleware for `/labels/{name}/values` endpoint as well via #6072 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Sashank Agarwal	6392ddb71b	Auto add labels on new pull request (#6065) ... * add auto labeler github action * add auto labeler github action	3 years ago
Christian Simon	213bb86802	Add integration tests which test happy path (#5968) ... * Add integration tests which test happy path This PR adds simple go based integration tests, which can run Loki in two modes: - single-binary - micro-services with index-gateway After standing up it performs the most basic duties like: - ingest logs - run a range query - query label names and label values This is meant to catch more significant problem as early as possible in the PR. We should aim to keep this as simple as possible. * Replace config generation with template Makes configuration much more readable and avoids easy mistakes * Apply suggestions from code review Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christian Haudum	7fff660144	Add UDP protocol support to Promtail's syslog target (#5790) ... Until now, the syslog target in Promtail did only support TCP. This PR adds support for sending syslog messages to Promtail (via ng-syslog or rsyslog) via UDP. UPD for syslogs can be used if you prefer speed over guaranteed delivery. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Ed Welch	98fda9b0a0	Loki: Change live tailing to only allow mutating the log line not the number of streams. (#6063) ... * don't create more streams when live tailing logs, process the log line with the pipeline to allow mutating the content of the line but don't split it into new streams. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update changelog Signed-off-by: Edward Welch <edward.welch@grafana.com> * don't mutate the incoming streams Signed-off-by: Edward Welch <edward.welch@grafana.com> * do not reuse the pipeline because it caches labels Signed-off-by: Edward Welch <edward.welch@grafana.com> * add note to upgrade guide Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	bb2e952d6f	Fix(query-frontend): `/label/<name>/values` endpoint to use right set of middlewares (#6072) ... * Fix(query-frontend): `/label/<name>/values` endpoint to use right middleware Previously, only `/label/` and `/labels` endpoint were using correct tripperware middleware on the query frontend(e.g: stats collector middleware) This PR fixes it for `/label/<name>/values` endpoint as well. This is useful to support `metrics.go` stats middleware for `label/{name}/values` endpoint as well. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove `unhandledpath` test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unused test helper Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
dependabot[bot]	00e210aa51	Bump actions/checkout from 2 to 3 (#5558) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Corentin Altepe	d9327ecd0c	Added NLog-Targets-Loki as unofficial C# client to Loki (#6061) ... Added a link to the HTTP C# client [NLog-Targets-Loki](https://github.com/corentinaltepe/nlog.loki) in Loki's clients documentation in the unofficial clients sections, right next to its Serilog counterpart.	3 years ago
W.T. Chang	9c19b0f548	Fix broken link on deployment-modes.md (#6068) ... The configuration link should go another parent level further in order to get `docs/loki/latest/configuration/`, currently it points to `docs/loki/latest/fundamentals/configuration/`.	3 years ago
Sandeep Sukhani	8cd4694adb	allow more time for boltdb-shipper index syncs to finish (#6071) ... * retains index files with ingester for 20 mins instead of 12 mins after uploading * sets GetChunkIDs lookback to 2h41m instead of 2h28m	3 years ago
Paschalis Tsilias	179d741c5f	promtail: Add support for exclusion patterns in `static_config` (#5943) ... * PR base Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add CHANGELOG entry; updated documentation for new label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Follow logic from t.path in exlusion case; always handle matching Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make error message unique Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Only run if t.pathExclude is set Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Revert "Only run if t.pathExclude is set" This reverts commit 17abb006ce141071c4bca4b7d133bea2aa314594.	3 years ago
Susana Ferreira	38b1d3b914	Update Loki reads/writes resources dashboards (#5765)	3 years ago
Bryan Boreham	241bde1271	Add dot to some DNS address to reduce lookups (#5789) ... Kubernetes sets up a DNS search path like `namespace.svc.cluster.local, svc.cluster.local, cluster.local, google.internal`; any unqualified name is first tried as a prefix to each of these. These requests, which will never succeed, add latency and increase load on the DNS service. Adding a `.` at the end makes the name fully-qualified, so Go's DNS client will skip the search path and use the name as-is. Note only changes a subset of addresses in the config; others didn't seem to be causing a problem.	3 years ago
Evangelos Foutras	093fe91ce9	Bump inet.af/netaddr for Go 1.18 compatibility (#5872) ... Fixes panic caused by outdated go4.org/unsafe/assume-no-moving-gc. [1] [1] assume-no-moving-gc commit 538ce61f45ea5c2c48d7133d44b3e52c3db5c31e	3 years ago
Zach Leslie	cce2d8ef2f	Promoto policyRule to v1 (#6003)	3 years ago
Alex Ferm	d7f03be212	Update libsystemd-dev from bullseye-backports for arm32 docker images, fixes (#5938) (#6012)	3 years ago
dependabot[bot]	8a5877ccf1	Bump github.com/minio/minio-go/v7 from 7.0.10 to 7.0.24 (#6007) ... Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.10 to 7.0.24. - [Release notes](https://github.com/minio/minio-go/releases) - [Commits](https://github.com/minio/minio-go/compare/v7.0.10...v7.0.24) --- updated-dependencies: - dependency-name: github.com/minio/minio-go/v7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Dylan Guedes	162e82ebfb	Use quiet flag in logcli tail.go. (#6033)	3 years ago
Dylan Guedes	47ce070b73	Add missing changelog entry for change introduced by PR 6034. (#6040) ... * Add missing changelog entry regarding PR 6034. * Move changelog entry to promtail subsection	3 years ago
Sandeep Sukhani	2758dc659a	boltdb shipper index list cache improvements (#6054) ... * bypass index list cache when doing query time downloading of index * detect and refresh stale index list cache during sync	3 years ago
Karen Miller	a5b9a9a91c	Docs: Improve wording of getting started guide (#6060) ... - Add the -d option to all docker-compose up commands - Make the wording consistent about the CWD	3 years ago
Travis Patterson	eb242f305d	Add tailing metrics (#6059) ... * Add tailing metrics * fix test * fix test	3 years ago
Owen Diehl	309592a8cf	ignores reporting cancellations as errors in the usage-reporter module (#6058)	3 years ago