apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
JordanRushing	9000de1aa7	fix(push): Add guard clauses to prevent negative counter values (#17056 ) Signed-off-by: Jordan Rushing <rushing.jordan@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
JordanRushing	9e9f534460	fix: prevent panic when ContentLength is negative in Distributor RequestBodyTooLarge metrics (#17054 )	1 year ago
Periklis Tsirakidis	1d99f4d86d	feat(distributor): Add MaxRecvMsgSize config for uncompressed message size limits (#16915 )	1 year ago
Dylan Guedes	ea13730e63	feat: Snapshopt stream overrides values on each request (#16523 ) What this PR does / why we need it: Caches a snapshot of resolved policies and retention temporarily to guarantee consistent results on the same request. We evaluate retention and policies in a map when emitting some metrics so having policies or retention inconsistent along the chain for the same push request can cause us to miscalculate things or worse, interact with a nil map.	1 year ago
Salva Corts	c2e1e88e85	feat(policies): Add PoliciesStreamMapping to loghttp limits interface (#16105 )	1 year ago
Travis Patterson	f2bff201d7	fix: Determine when all logs have been filtered (#16073 )	1 year ago
Salva Corts	63a2442191	fix(otlp): Write protobuf status on error (#15097 )	2 years ago
Trevor Whitney	d7ff426646	feat: ability to log stream selectors before service name detection (#14154 )	2 years ago
Salva Corts	b31e04e3f1	fix(otel): Map 500 errors to 503 (#13173 )	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Travis Patterson	b0478e61c8	feat: add the ability to inject log parsers to the distributor (#12029 )	2 years ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840 )	2 years ago
Sandeep Sukhani	e625d7f1ca	otel: add support for per tenant configuration for mapping otlp data to loki format (#11143 ) What this PR does / why we need it: In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar. Special notes for your reviewer: Opening a draft PR to get some initial feedback. I will add documentation in a separate PR. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Sandeep Sukhani	6069df8f7d	ingestion: native otlp ingestion support (#10727 ) What this PR does / why we need it: Add support for natively supporting logs ingestion in OTLP format. `/otlp/v1/logs` is the new endpoint where users can push logs in OTLP format. It accepts logs serialized in JSON or proto format. Since OTEL format is very different than what Loki storage model, here is how data in OTEL format will be mapped to Loki data model: * Index labels: The Resource Attributes map quite well to Index labels in Loki since both usually identify the source of the logs. The problem however is that Resource attributes in OTLP can have an unbounded number of values while Loki has a default limit of having up to 30 labels. Since Index labels in Loki can largely drive the kind of querying experience the users are going to have, we have chosen select attributes which would be picked as Index Labels. The ones that are not picked up as Index labels would be stored as Structured Metadata with each log entry. * Timestamp: LogRecord.TimeUnixNano * LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`). * Structured Metadata: Anything which can’t be stored in Index labels and LogLine. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold: * Resource Attributes not stored as Index labels is replicated and stored with each log entry. * Everything under InstrumentationScope is replicated and stored with each log entry. * Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp. NOTES: * Since Loki does not support `.` or any other special characters other than `_` in label names, we replace all non-supported characters with `_`. * Since Loki only supports string in values of Index Labels and Structured Metadata, all the complex types are converted as follows: * Map would be flattened into label keys using `_` as separator, same as how we do it in [json parser in LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json). * Everything else is stringified using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`) Special notes for your reviewer: I will open follow-up PRs for: * Documentation * Make blessed attributes list configurable per tenant. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label	3 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	dc5d458c14	Failures logging: avoid logging generic errors (#9796 ) What this PR does / why we need it: Modify our write path to not log generic messages with the `writefailures.Manager`. With generic messages I mean: contextCancel errors and contextDeadlineExceeded errors. Which issue(s) this PR fixes: N/A	3 years ago
Dylan Guedes	c0d9a2b4ba	Log push failures per-entry instead of in batches (#9720 ) What this PR does / why we need it: Modify how we log push errors to do it per-entry instead of accumulating all errors and then logging them all in a single pass. Which issue(s) this PR fixes: N/A	3 years ago
Dylan Guedes	7d67b6306f	Loki: Implement initial phase of `limited_log_push_errors` (#9556 ) What this PR does / why we need it: Adds an initial implementation of `limited_log_push_errors`. This initial implementation encompass: - Runtime per-tenant configuration - Simple per-tenant rate-limiting based on error message size Notable features that will be added in future phases: - Instead of a single error with the final string, give to the manager the list of all entries - Once it supports per-entry error, make the rate-limiting separated per-reason - Hash the entry error and avoid repeating errors by caching seen errors in memory	3 years ago
Dylan Guedes	2bb2a2c0e1	Loki: Remove ring client from the distributor (#6622 ) Removes the ring client from the distributor. - It was originally added on PR #4938 because it was the only ring implementation with HTTPHandler support (used for the ring page). Now that DSKit has support for the other ring implementations (including the lifecycler) this isn't necessary anymore. - It had an IgnoreUnhealthyInstancesReplicationStrategy attached to it, added by mistake	4 years ago
Karsten Jeschkies	45ffebd4d8	Return HTTP 400 when multiple tenants are present in push. (#5800 )	4 years ago
Ed Welch	c158b2c5f5	Loki: Revert #4845 which changed the format of errors from the API (#5772 ) * This reverts the changes from #4845, unfortunately changing the content type from plain text to JSON is a breaking change of the API and we should wait to do this at a major release. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix tests which were updated after initial change. Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724 )	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Dylan Guedes	98ecfefd89	Add distributor ring page (#4938 ) * Implement new IngestionRateStrat Enum. - This new IngestionRateStrat is used internally to represent the chosen ingestion rate strategy, to avoid using plain strings. * Add a ring to the distributor. As of now, the distributor only has a pointer to its lifecycler. Since the ring HTTP handler isn't implemented for the lifecycler but it is for the ruler, this extends the distributor by also holding a ring instance. * Implement /distributor/ring page. * Reuse existing global strat const. * Add distributor ring HTTP page tests. * Add missing CHANGELOG entry. * Document missing ring status pages. * Fork Cortex util/http.go into source code.	4 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944 )	5 years ago
Christian Haudum	e573a4d1a2	Change error responses from plain text to JSON (#4845 ) * Return HTTP errors consistently as JSON response The API returned errors in plain text where the error message was the response body. However, the content type header was set to `application/json`. To make the API responses consistent, errors are now returned as JSON as well. ``` { "message": string, "status": string, "code": int } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return HTTP errors consistently as JSON response Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973 ) This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	5 years ago
Danny Kopping	71889dcb19	Querier/Ingester: Fixing json expression parser bug (#3928 ) * Fixing json expression parser bug Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * UNRELATED CHANGE: pulling main to fix linter issue Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840 ) * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago
Ed Welch	21b901db9b	Loki: Distributor log message bodySize should always reflect the compressed size (#3572 )	5 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460 ) * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	5 years ago
Dr. Jan-Philip Gehrcke	0f881fbf78	distributor: fix snappy pb POST request handling (#3407 ) (#3408 ) Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>	5 years ago
Dmitry Ukolov	2a51fb42bf	Distributor: Loki API can receive gzipped JSON (#3291 ) * Push API recognize HTTP header "Content-Encoding: gzip" and upload gzipped JSON * Fix documentation mistake	5 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Sandeep Sukhani	d357ad9e58	fix broken json logs push path (#3261 )	5 years ago
Ed Welch	51952dafc1	Loki: Log a crude lag metric for how far behind a client is. (#3236 ) * Log the difference in time between the most recently received log entry and the current time as a crude measure of how far behind a promtail instance is. * fix lint	5 years ago
Cyril Tovena	d976079992	Fixes previous commit not using the new sized body. (#3234 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	567ccde0b6	Re-introduce https://github.com/grafana/loki/pull/3178 . (#3233 ) Lost during a merge/rebase of `6cc41f92f8` 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794 )	6 years ago
Ed Welch	b24a143043	Promtail: Loki Push API (#2296 ) * refactor to put targets in separate packages * rename scrape package to scrapeconfig * Push implementation, working with logging * adding new vendored library * adding tests * adding documentation and an additional config option. * fixing journal test * rename scrape config entry * Update docs/clients/promtail/configuration.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * PR feedback Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	6 years ago
Cyril Tovena	d438bda94a	Update cortex to bring v11 schema (#1201 ) * Update cortex to bring v11 schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes change in ParseProtoReader Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * sync go mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	7 years ago
Joe Elliott	bb2b925255	Update `/loki/api/v1/push` to use the v1 json format (#1145 ) Signed-off-by: Joe Elliott <number101010@gmail.com>	7 years ago
Tom Wilkie	f6acb87d1f	Correctly propagate errors in the distributor; should now return 4xx on OOO entries. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	da27baad2c	Support for JSON, not just protos. (#152 ) * Support for JSON, not just protos. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * s/timestamp/ts/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago

1 2

56 Commits (bfa2a1df65e233ce99fccd01df2130f440ebd166)