apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958 )	9 months ago
Func	4a05964d55	fix: Use to the proper config names in warning messages (#12114 ) Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Salva Corts	5f7bde74ac	Add limits (#10552 ) What this PR does / why we need it: This PR introduces two new limits for structured metadata: - `max_structured_metadata_size`: Defaults to 64 KB. Limits the maximum size of the structured metadata attached to a log line. - `max_structured_metadata_entries_count`: Default to 128. Limits the maximum number of structured metadata entries attached to a log line. We discussed adding other limits like `max_structured_metadata_label_name_length` and `max_structured_metadata_label_value_length` but discarded this idea for now.	2 years ago
Owen Diehl	2cc80c59c4	per tenant gate for structured metadata feature (#10402 ) Adds a per tenant configuration to block access to the upcoming structured-metadata push API extension. --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Dylan Guedes	358b8965c4	Distributor: Add org_id to log error (#9475 ) What this PR does / why we need it: Add `org_id` to push error when the error is "max active streams limit exceeded". Apparently all other push errors correctly report the `org_id`, but not this one. Which issue(s) this PR fixes: N/A	2 years ago
Christian Simon	dad39f7ea6	Improve comments on sample timestamp validation errors (#8018 ) This has come up time and time again, I have tried to write it up a bit better in code first, I would also strongly suggest that we improve the error messaging and the documentation of this.	2 years ago
Owen Diehl	cd7ebf2ae0	exposes line length hist on distributors (#6309 ) * exposes line length hist on distributors + fixes some metric instantiation * adjusts buckets	3 years ago
Fanis Hatzidakis	e7a1b67c0a	Docs: Fix typo in 429 exceeded rate limit error message (#5944 ) * Fix docs typo in word "limit" * Fix docs typo "unlimited" in limits.go	3 years ago
Kaviraj	564f833a30	Improve error message if incoming logs timestamp is far too behind. (#5040 ) * Improve error message if incoming logs timestamp is far too behind. This is part of JSON response giving out as the response to HTTP /push endpoint. Old message ``` {"code":400,"status":"error","message":"entry for stream '{foo=\"bar\"}' has timestamp too old: 1970-01-01 01:00:00.5 +0100 CET"} ``` New message ``` {"code":400,"status":"error","message":"entry for stream '{foo=\"bar\"}' has timestamp too old: 2021-12-28 01:48:45.5 +0100 CET, accepts timestamp from: 2021-12-29 09:48:45.737756651 +0100 CET"} ``` main rationale being, hard to know what is the closest timestamp that Loki is expecting without going through the config. Also config is not straight forward (its `latest stream's timestamp - ingester.max-chunk-age/2`) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Tweaks the error message Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `distributor/validator_test`. 1. Make `time.Now()` mockable. 2. Make time format consistent across error messages. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename `getValidationContextFor` to `getValidationContextForTime` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Callum Styan	822d60507b	log the user ID in the rate limit error message. (#4852 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Owen Diehl	cd80bc5e7e	introduces "entry too far behind" instrumentation for unordered writes (#4592 )	4 years ago
Callum Styan	25163470b0	Improve error message for stream rate limit. (#4207 ) * Improve error message for stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Move stream rate limit error message to validation package. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Use ByteSize for number values in stream rate limit error so we can print human readable rate limits and log line byte lengths. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't need to type assert twice. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Compare errors contents instead of doing error equals comparison directly since the rate limit error is an RPC error. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't call Limit on hot path just to log the value in errors. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Make coment about rate limit clearer. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Owen Diehl	abcf4d083f	WAL replay discard metrics (#4212 ) * adds replay discard metrics * lint	4 years ago
Owen Diehl	d42de7ee2b	fixes typo (#4199 )	4 years ago
Callum Styan	757eeacb88	[ingester/stream]: Add a byte stream rate limit. (#4191 ) * Add a byte stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Track discarded lines due to stream rate limit + use ByteSize for the config value. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * Another round of review fixes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix goimports issue. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix duplicate flag registration + stream rate limit test. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix stream rate/burst limit flags yaml naming. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix tests now that we have stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix some comments Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Owen Diehl	572bbe15ec	Discard/ooo (#4104 ) * stream includes tenant id * expose out_of_order discarded metrics	4 years ago
Travis Patterson	8a2081a39e	Distributor: Truncate rather than drop log lines (#4051 ) * Truncate rather than drop logs - When log lines are longer than maxLineSize, truncate them to maxLineSize rather than dropping them Fixes: #3747 * Review feedback: - Rename 'max_line_size_should_truncate' to 'max_line_size_truncate' - Add configurable truncated indicator - Add Truncated Line metric - Changed truncated metrics to have the name mutated_{samples,bytes}_total * Linter * Review Feedback - Make metric variable names match metric names - Add benchmarks - reduce truncation allocs by 100x * Linter * Remove truncation indicator	4 years ago
Cyril Tovena	ae23b7e4ed	Rejects push requests with streams without labels. (#3643 ) * Rejects push requests with streams without labels. Fixes #3638 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing commit Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Ed Welch	407afce0e3	Loki: fix validation error and metrics (#3307 ) * Remove the helper functions for creating errors as they cause unintended value manipulation if the value contains any valid golang printf substitute characters, e.g. %s * Improve error message for invalid labels, increment metrics for lines dropped for invalid labels.	4 years ago
Ed Welch	272a89c13d	more linter fixes (#2024 ) * more linter woes Signed-off-by: Ed Welch <edward.welch@grafana.com> * feedback Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	4f9a2fe2d9	I <3 you Linter (#2022 ) Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	3dfe89369f	Loki: refactor validation and improve error messages (#2021 ) * copies some of the validation logic from cortex and modifies it to appropriate for Loki, this gives us a single metric we can use for detetecting validation errors (previously we had to check cortex_ and loki_ metrics) as well as lets us use error messages more appropriate for logs than metrics. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fixing tests and lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * updating cause to string type Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting a function for rate limited Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting functions for the rest of the error strings Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	0ab1b28812	Loki: Improve logging and add metrics to streams dropped by stream limit (#2012 ) * Improve the log message when we drop streams because a user is hitting a stream limit. Increment the dropped samples metrics when this happens also. Signed-off-by: Ed Welch <edward.welch@grafana.com> * improving comments Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	4fdfbe8e56	Increase discarded samples when line is too long. (#1708 ) * Increase discarded samples when line is too long. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add comment on the public props Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Joe Elliott <joe.elliott@grafana.com>	5 years ago
Julien Pivotto	eb7b495565	Metrics: use Namespace everywhere when declaring metrics (#1320 ) Code linting to make things consistent. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Julien Pivotto	7ccd0ab8d0	Use tenant as label name for discarded_samples metrics discarded_samples should use the name labels as the other metrics. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Julien Pivotto	5d1fec514c	Expose discarded bytes metric (#1317 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Robert Fratto	a5765b57bf	pkg/distributor: fix metric loki_discarded_samples_total (#1279 ) loki_discarded_samples_total is supposed to be the count of samples that were discarded, but was instead being reported as the total bytes discarded. The metric has been fixed and a new metric, loki_discarded_bytes_total, has been added to report the total discarded bytes. Fixes #1255.	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948 ) * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago

22 Commits (9fb2f385529c2444be98161743aa0cd8ad420bf2)