apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744 ) What this PR does / why we need it: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. Which issue(s) this PR fixes: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (LID 0002: Remote Rule Evaluation).	3 years ago
Paul Tirk	540380fa94	Promtail: fix GeoIP typo (#8783 ) What this PR does / why we need it: Fixes a typo for GeoIP stage. From `geoip_continet_name` to `geoip_continent_name`.	3 years ago
Ed Welch	0352b75f0d	Loki: Add new logql template functions `bytes`, `duration`, `unixEpochMillis`, `unixEpochNanos`, `toDateInZone`, `b64Enc`, and `b64Dec` (#8774 ) What this PR does / why we need it: This adds several new template functions to the logql to be used in `line_format` and `label_format` - bytes - duration (and duration_seconds) - unixEpochMillis - unixEpochNanos - toDateInZone - b64Enc and b64Dec `bytes` and `duration` adds parity with the `unwrap` pipeline stage in logql, allowing you to convert human readable versions of bytes and times into their respective primitives of bytes and seconds. This makes it possible to do math and comparisons of these extracted values in `line_format` and `label_format` `unixEpochMillis` and `unixEpochNanos` extends the existing `unixEpoch` function which can only generate seconds level precision to allow having millisecond and nanosecond level precision. `toDateInZone` allows for specifying a specific timezone when parsing a date. `b64Enc` and `b64Dec` allow base64 encoding and decoding values. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: @JStickler I made a lot of changes to the template_functions.md doc page to try and create some more consistency as well as remove some notices that really aren't relevant anymore: * Tried to make sure every function had a `Signature` and `Examples` block * Removed a lot of `NOTE` about functions being introduced in a specific version * Removed documentation for some deprecated string functions (they still exist in code but I removed the docs for them, their exist better replacements for all of them) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761 ) What this PR does / why we need it: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	eadb9d2c00	Loki: Change the behavior of the 0 value for retention_period to disable retention. (#8753 ) What this PR does / why we need it: Issue #8359 exposed a very unfortunate inconsistency in a Loki config where the zero value did not disable a feature. By convention in Loki the zero value for a config typically "disables" it. This PR changes the retention_period to honor the zero value of `0` or `0s` to disable retention. Which issue(s) this PR fixes: Fixes #8359 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
J Stickler	673bfd603b	Fixing broken link (404) (#8688 ) This was reported to the docs team via Slack. Fixing broken link from Promtail Scraping file to Promtail Configuration file.	3 years ago
Kaviraj Kanagaraj	371455770f	doc(best-practices): Update default value of `max_chunk_age`. (#8685 )	3 years ago
e0031374	2dcaba3176	Loki Getting Started: Fix example query that was not working (#8668 ) What this PR does / why we need it: Previously example query was `evaluate-loki_flog_1` this was not the label used in flog. This results in query not returning any logs which beginners may be unsure of the cause of the issue. This might also be an issue if flog was running in the background and logs are not displayed. Changed to `evaluate-loki-flog-1`. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Trivial fix to "Getting Started" documentation Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Grot (@grafanabot)	215f37cfa3	[CI/CD] Update yaml file `./production/helm/loki/values.yaml` (+1 other) (#8655 ) Here is a summary of the updates contained in this PR: * Update attribute `$.enterprise.version` in yaml file `./production/helm/loki/values.yaml` to the following value: `v1.6.2` * Bump version of Helm Chart Add changelog entry to `./production/helm/loki/CHANGELOG.md` Re-generate docs Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Christian Haudum	cec264996b	LID: Improve scheduler to control QoS across users within a tenant (#8585 ) Even though Loki is built as multi-tenant system by default, there are use-cases where a Loki installation only has a very large, single tenant, e.g. dedicated Loki cells for customers in Grafana Cloud. However, there are potentially a lot of different users using the same tenant to query logs, such as users accessing Loki from Grafana or via CLI or HTTP API. This can lead to contention between queries of different users, because they all share the same tenant. While the current implementation of the scheduler queues allows for QoS guarantees between tenants, it does not account for QoS guarantees across individual users within a single tenant. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Jan Jansen	84c5e20af1	S3: add support for s3 session token (#8474 ) We want to automate the access account rotation. This is that easy at moment with service accounts in MinIO. Our currently life time of tokens is 3 months with MinIO STS works this fine. Signed-off-by: Jan Jansen <jan.jansen@gdata.de>	3 years ago
J Stickler	5fbc747392	Fixing broken link and removing out of date blog post. (#8632 ) Which issue(s) this PR fixes: * Starts to address #8631 * Removes link to outdated blog post that says "Loki is very much alpha software and should not be used in production environments." * Fixes broken link to Configuration file.	3 years ago
Trevor Whitney	dac3b84d08	Loki cloud integration instructions (and necessary mixin changes) (#8492 ) What this PR does / why we need it: This PR is a first stab at documentation for setting up the Grafana Cloud Loki Integration to monitor a local Loki cluster, installed using the helm chart, using Grafana Cloud. In addition to instructions on how to collect the necessary kubernetes metrics, there were also some things with the Loki mixin (which provides dashboards, alerts, and rules to the integration) that needed to be fixed. For example, our dashboards rely on there being a cluster label on our recording rules, yet our mixin does not include this? Also, we are still including `cortex-gw` panels even when the mixin is compile with `internal_components: false`. Special notes for your reviewer: The changes to the mixin will not be reflected in the current version of the integration. We need to merge this PR first then update the integration. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Joseph Shanak	7204216187	chore: Fix typos in Helm documentation (#8593 )	3 years ago
Bayan Taani	68a9fd6c0a	Add logfmt selective label extraction (#6675 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR introduces extracting labels from a log line in `logfmt`, with an extra option to rename them. For example, this query: ``` {app=”foo”} \| logfmt msg,status ``` will extract the labels `msg` and `status` from the following logfmt line: ``` level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..." ``` With the following results: ``` msg="final error sending batch" status=”400” ``` -------------- Another possible scenario with label renaming: ``` {app=”foo”} \| logfmt message="msg", status ``` That produces the following results: ``` message="final error sending batch" status=”400” ``` Which issue(s) this PR fixes: Fixes #3355 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Aditya C S	b94c8aaf16	feature: geoip stage in promtail (#3493 ) Co-authored-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Christophe Collot	a013e9f342	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231 )	3 years ago
Yasmine El wahbi	c6542e61fa	Promtail: Add ready endoint to loki-push-api server (#8552 ) When trying to expose loki_push_api service (in our case using GKE ingress), we need a health check target endpoint that would return a 200 response code. This PR contains a simple addition for a HTTP handler that handles the /ready path, along with the necessary tests. Fixes #8551 Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
a5r0n	0f0553ff1c	helm: make multi tenant easy: use tenant id from nginx $remote_user (#8404 ) Fixes #8380 Co-authored-by: a5r0n <a5r0n@users.noreply.github.com>	3 years ago
Karsten Jeschkies	880de93093	Document memcached configuration for caching. (#8373 ) What this PR does / why we need it: Memcached is our default options for caching. However, the options for Memcached are not well documented. This change documents how Memcached should be configured and how Loki itself should be documented based on our ksonnet deployment. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
justcompile	ea55f285b6	S3 Storage Class Configuration (#8532 )	3 years ago
Angus Dippenaar	42522d794a	logcli: Add parallel flags (#8518 )	3 years ago
Vladyslav Diachenko	4e4359e67c	Fixed XSS on LogQL Analyzer page (#8521 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Christian Haudum	d960ee70c6	Helm: Remove unreleased setting from Helm chart configuration file (#8515 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
brettlounsbury	4c227bd268	DynamoDB: Accept a KMS key for DynamoDB tables (#8469 ) Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
J Stickler	06a6c7c9fc	Temporarily unpublish Analyzer page. (#8503 )	3 years ago
Hervé Nicol	f4119de4df	helm: values to override read/write/backend paths in gateway config (#8490 ) What this PR does / why we need it: Add configurations to override the generated read/write/backend URLs for the gateway. Which issue(s) this PR fixes: Fixes #8488	3 years ago
Tobias Abarbanell	d1d47b2fb2	Loki Documentation: corrected config example for loki.storage (#8277 )	3 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	3 years ago
J Stickler	a5fbc7cff4	Updated CONTRIBUTING file for docs. (#8431 )	3 years ago
Trevor Whitney	e6fb32b214	[helm] Bump to Loki 2.7.3 (#8445 )	3 years ago
janhorstmann	0e81d78914	Swift: Allow to configure swift with the internal endpoint (#8438 )	3 years ago
Trevor Whitney	50af51bb36	[helm] set compactor address (#8427 )	3 years ago
Trevor Whitney	71837194ba	add section to prom operator instructions about necessary label (#8379 )	3 years ago
Jan-Otto Kröpke	3c7a91b6f1	[Helm] Allow to configure publishNotReadyAddresses on memberlist service (#8446 ) Signed-off-by: Jan-Otto Kröpke <jok@cloudeteer.de>	3 years ago
Dylan Guedes	7e3dbd4907	Improve installation instructions (#8454 ) What this PR does / why we need it: Improve Helm installation instructions and add a note on how to use MinIO. Which issue(s) this PR fixes: Fixes #8440	3 years ago
Jim Gustavsson	99b4f475c4	Helm: Add podlabels to backend (#8398 ) Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Karsten Jeschkies	a5ddbf2e1f	Correct sizing tool unit in help text. (#8441 )	3 years ago
Nick Moore	85b3e5d4ab	Promtail: Add event log message stage (#8382 ) The Windows Event Log includes a field named Message that include a large amount of additional data, stored in a simple key-value format: key-value pairs are separated with Windows-style \r\n newlines, the key and value are separated with : (or just : if the key has no associated value). The key-value data in the Message field is commonly utilised when investigating Windows events, e.g., in security use-cases, but the existing windows_events pipeline stage keeps the Message data as a string, leading to requiring significant effort at the query time to extract that data to be used in field filters. This PR request introduces a new promtail stage that parses a specified Source (defaulting to message used by the windows_events) and extracts all the key-value pairs into the extracted fields, which can then be used in latter stages in the pipeline.	3 years ago
J Stickler	6f4526c3bc	Fix broken cross-reference in Helm installation. (#8430 )	3 years ago
J Stickler	e382a4175c	Fix numbering issue in second procedure. (#8429 )	3 years ago
Trevor Whitney	ef8b382c06	[helm] Single binary object storage (#8284 ) Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Christian Haudum	2139dbf55f	[LogCLI] Load tenant-specific schema config file when using `--remote-schema` (#8413 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Jack Baldry	c5725e4cae	Use portable output redirection (#8417 ) Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	3 years ago
Borna Zeba	98a2daa292	docs(monitor-and-alert): Fix typo in ServiceMonitor spelling (#7693 ) Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Hamish Forbes	027c2ef957	Jsonnet: Allows gateway tenant ID to be configured (#7624 ) Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Svilen	2d11ae2071	Fix the ruler alertmanager_client documentation (#7474 )	3 years ago
Jack King	d434e80877	Promtail: Add custom headers on push requests (#7470 ) (#7472 )	3 years ago
Witold Duranek	ee1aee0af3	feat(promtail): add more complex options of dropping logs at the drop stage (#6371 )	3 years ago

1 2 3 4 5 ...

1272 Commits (b46e392adbbb59f40fecd16c9ff925cb40456e0c)