apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Dylan Guedes	3c16658ee0	Loki: Append loopback to ingester net interface default list (#4570 ) * Move LoopbackInterfaceName to its own package. - Create new package `util/net` and puts `LoopbackInterfaceName` there - Modify distributor_test to use `util/net` package * By default, append loopback to ingester interfaces. - Append the loopback net interface to the list of net interfaces used by the ingester ring. The append only occurs if the ingester net interfaces are default - Add CHANGELOG entry * Test ring net interface behavior. - Add tests to check the current net interface name resolution. The tests checks if the ingester ring net interface names are correctly copied to other rings	4 years ago
JordanRushing	ca67292b54	Introduce `overrides-exporter` module to Loki (#4520 ) * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Use reflection; add more limits to export * Fix import to satisfy linter * refactors TenantLimits for clarity & efficiency. Also fixes tenantLimits nil check * Fix breaking test with nil check in runtime_config Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Dylan Guedes	d3d63e1778	Loki: Override distributor's default ring KV store (#4440 ) * Override default KV store for the compactor. - Currently, the default KVstore is consul. This changes it to inmemory to make it easier for new users to run the project without dependencies. - Refactor default flag value tests. Now, it will check for a populated map instead of checking for flags during iteration. * Move distributor flag override to RegisterFlags. * Add missing CHANGELOG and upgrading guide entries.	4 years ago
Arve Knudsen	79b95d976a	Chore: Use dskit/limiter (#4327 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Arve Knudsen	ae424deac1	Upgrade dskit (#4295 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Christian Haudum	8d561f6caf	Chore: Update Cortex and use kv package from grafana/dskit (#4282 )	4 years ago
Bryan Boreham	13a3df8826	Simplify Distributor.push (#4240 ) Instead of creating another goroutine to wait for the background `sendSamples()` to finish, give the channels it is sending on a buffer so that they can send without a listener. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
李国忠	6f78ffe33c	[fix] distributor: fix goroutine leak (#4238 ) Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	4 years ago
Arve Knudsen	cfb4fc1f55	Use flagext from dskit (#4225 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196 ) * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Travis Patterson	8a2081a39e	Distributor: Truncate rather than drop log lines (#4051 ) * Truncate rather than drop logs - When log lines are longer than maxLineSize, truncate them to maxLineSize rather than dropping them Fixes: #3747 * Review feedback: - Rename 'max_line_size_should_truncate' to 'max_line_size_truncate' - Add configurable truncated indicator - Add Truncated Line metric - Changed truncated metrics to have the name mutated_{samples,bytes}_total * Linter * Review Feedback - Make metric variable names match metric names - Add benchmarks - reduce truncation allocs by 100x * Linter * Remove truncation indicator	4 years ago
Michel Hollands	c44c12b930	Update Cortex version (#3992 ) * Update Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Changed code due to new Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973 ) This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934 )	4 years ago
Danny Kopping	71889dcb19	Querier/Ingester: Fixing json expression parser bug (#3928 ) * Fixing json expression parser bug Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * UNRELATED CHANGE: pulling main to fix linter issue Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840 ) * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Cyril Tovena	ae23b7e4ed	Rejects push requests with streams without labels. (#3643 ) * Rejects push requests with streams without labels. Fixes #3638 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing commit Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632 ) Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Owen Diehl	965d48222b	adds distributor replication factor metric (#3594 )	4 years ago
Ed Welch	21b901db9b	Loki: Distributor log message bodySize should always reflect the compressed size (#3572 )	4 years ago
Michel Hollands	f49a510f38	Loki: Update cortex version and fix resulting changes (#3532 ) * Update cortex version and fix resulting changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix scrapeconfig test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/promtail/scrapeconfig/scrapeconfig_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Remove unused fmt import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446 ) * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460 ) * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	4 years ago
Dmitry Ukolov	6ea7122777	Distributor: append several tests for http parser. (#3416 )	4 years ago
Dr. Jan-Philip Gehrcke	0f881fbf78	distributor: fix snappy pb POST request handling (#3407 ) (#3408 ) Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>	4 years ago
Ed Welch	407afce0e3	Loki: fix validation error and metrics (#3307 ) * Remove the helper functions for creating errors as they cause unintended value manipulation if the value contains any valid golang printf substitute characters, e.g. %s * Improve error message for invalid labels, increment metrics for lines dropped for invalid labels.	4 years ago
Dmitry Ukolov	2a51fb42bf	Distributor: Loki API can receive gzipped JSON (#3291 ) * Push API recognize HTTP header "Content-Encoding: gzip" and upload gzipped JSON * Fix documentation mistake	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Karsten Jeschkies	287244d110	Update vendored Cortex to 0976147451ee (#3267 )	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Sandeep Sukhani	d357ad9e58	fix broken json logs push path (#3261 )	4 years ago
Ed Welch	51952dafc1	Loki: Log a crude lag metric for how far behind a client is. (#3236 ) * Log the difference in time between the most recently received log entry and the current time as a crude measure of how far behind a promtail instance is. * fix lint	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	d976079992	Fixes previous commit not using the new sized body. (#3234 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	567ccde0b6	Re-introduce https://github.com/grafana/loki/pull/3178 . (#3233 ) Lost during a merge/rebase of `6cc41f92f8` 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	82d96dec03	Improve distributors validation. (#3134 ) ```bash ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Push-16 18708519 1348876 -92.79% benchmark old allocs new allocs delta Benchmark_Push-16 44 42 -4.55% benchmark old bytes new bytes delta Benchmark_Push-16 4008391 2079 -99.95% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	4f62f36af7	Update cortex to 1 6 (#3131 ) * Update Cortex to recent master (1.5.0+, 35e698bb56d6). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore set scheduler address. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make linter happy. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Upgrade cortex to 1.6 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Work around issues in go-openapi libraries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clean up go.mod and go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	4 years ago
李国忠	bd321fb4f2	lru cache logql.ParseLabels (#3092 ) * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels	4 years ago
Cyril Tovena	e1ab2cc509	Cleanup labels parsing. (#2929 ) * Cleanup labels parsing. Make sure we only use one type of labels parsing.This just reduce the amount of code we have to maintain. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	049c7acc12	Improve logql parser allocations. (#2927 ) * Improve logql parser allocations. Also moving labels parsing into logql package, and I'm not shy to say that promql is faster for this. So we're using promql for parser, it's faster because their lexer is more sophisticated than ours. Result are impressive. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_ParseLabels-16 24394 8745 -64.15% benchmark old allocs new allocs delta Benchmark_ParseLabels-16 156 20 -87.18% benchmark old bytes new bytes delta Benchmark_ParseLabels-16 24976 2099 -91.60% ``` Combined with #2926, I think we should be able to get rid of `util.ToClientLabels`. But that's for another PR. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2303f1bfa1	Cache label strings in ingester to improve memory usage. (#2926 ) * Add benchmark for base stats. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cache labels string in ingester to improve memory usage. This reduce allocations by a lot, but assume that labels comes in sorted which is not ensure in the distributors. The distributors will take a performance hit, but that's easier to scale or improve later.(added some todos) see benchmark: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_PushInstance-16 43505 4950 -88.62% benchmark old allocs new allocs delta Benchmark_PushInstance-16 240 12 -95.00% benchmark old bytes new bytes delta Benchmark_PushInstance-16 42568 1787 -95.80% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 🤦 wrong hashing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794 )	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	21d3e69790	Enable local rules backend & disallow configdb. (#2661 ) * revendors cortex * vendoring compat * disallow configdb ruler backend * go mod tidy	5 years ago
Aurélien Rainone	21addf7935	pkg/distributor: fix test by looking for loopback interface (#2645 ) TestDistributor_PushIngestionRateLimiter failed if the system had no interface name in "eth0", "en0" or "lo0". Fix that by looking at the name of the loopback interface in the list of the system's network interfaces, then using that name in the test. Fixes #2356	5 years ago

1 2 3

103 Commits (2564a810c5872205c7385cd6d0a787b68477188f)