64 Commits (d8305ecd0a20dc530fcb073d8372c5c53449aa43)

Author	SHA1	Message	Date
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944)	3 years ago
Callum Styan	822d60507b	log the user ID in the rate limit error message. (#4852) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641)	4 years ago
Dylan Guedes	0d66319a1f	Revert distributor defaulting to inmemory. (#4638) ... - [PR 4440](https://github.com/grafana/loki/pull/4440/files) modified the distributor setup to use inmemory by default (instead of consul). That was necessary because, by changing the default rate limiting to global, a ring was necessary, and since by default the ring was set to consul, the project couldn't run with the default configs. That's not the case anymore: right now we are reusing the same config set for the ingester (which users already have properly configured otherwise they wouldn't be able to run the project). List of changes: - Remove changelog entries - Remove upgrading guide entries - Remove added tests (unnecessary checks) - Remove value overriding	4 years ago
Dylan Guedes	d3d63e1778	Loki: Override distributor's default ring KV store (#4440) ... * Override default KV store for the compactor. - Currently, the default KVstore is consul. This changes it to inmemory to make it easier for new users to run the project without dependencies. - Refactor default flag value tests. Now, it will check for a populated map instead of checking for flags during iteration. * Move distributor flag override to RegisterFlags. * Add missing CHANGELOG and upgrading guide entries.	4 years ago
Arve Knudsen	79b95d976a	Chore: Use dskit/limiter (#4327) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Bryan Boreham	13a3df8826	Simplify Distributor.push (#4240) ... Instead of creating another goroutine to wait for the background `sendSamples()` to finish, give the channels it is sending on a buffer so that they can send without a listener. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
李国忠	6f78ffe33c	[fix] distributor: fix goroutine leak (#4238) ... Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196) ... * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Travis Patterson	8a2081a39e	Distributor: Truncate rather than drop log lines (#4051) ... * Truncate rather than drop logs - When log lines are longer than maxLineSize, truncate them to maxLineSize rather than dropping them Fixes: #3747 * Review feedback: - Rename 'max_line_size_should_truncate' to 'max_line_size_truncate' - Add configurable truncated indicator - Add Truncated Line metric - Changed truncated metrics to have the name mutated_{samples,bytes}_total * Linter * Review Feedback - Make metric variable names match metric names - Add benchmarks - reduce truncation allocs by 100x * Linter * Remove truncation indicator	4 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973) ... This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840) ... * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Owen Diehl	965d48222b	adds distributor replication factor metric (#3594)	4 years ago
Michel Hollands	f49a510f38	Loki: Update cortex version and fix resulting changes (#3532) ... * Update cortex version and fix resulting changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix scrapeconfig test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/promtail/scrapeconfig/scrapeconfig_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Remove unused fmt import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446) ... * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460) ... * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	4 years ago
Ed Welch	407afce0e3	Loki: fix validation error and metrics (#3307) ... * Remove the helper functions for creating errors as they cause unintended value manipulation if the value contains any valid golang printf substitute characters, e.g. %s * Improve error message for invalid labels, increment metrics for lines dropped for invalid labels.	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215) ... * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178) ... * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	82d96dec03	Improve distributors validation. (#3134) ... ```bash ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Push-16 18708519 1348876 -92.79% benchmark old allocs new allocs delta Benchmark_Push-16 44 42 -4.55% benchmark old bytes new bytes delta Benchmark_Push-16 4008391 2079 -99.95% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
李国忠	bd321fb4f2	lru cache logql.ParseLabels (#3092) ... * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels	4 years ago
Cyril Tovena	e1ab2cc509	Cleanup labels parsing. (#2929) ... * Cleanup labels parsing. Make sure we only use one type of labels parsing.This just reduce the amount of code we have to maintain. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2303f1bfa1	Cache label strings in ingester to improve memory usage. (#2926) ... * Add benchmark for base stats. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cache labels string in ingester to improve memory usage. This reduce allocations by a lot, but assume that labels comes in sorted which is not ensure in the distributors. The distributors will take a performance hit, but that's easier to scale or improve later.(added some todos) see benchmark: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_PushInstance-16 43505 4950 -88.62% benchmark old allocs new allocs delta Benchmark_PushInstance-16 240 12 -95.00% benchmark old bytes new bytes delta Benchmark_PushInstance-16 42568 1787 -95.80% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 🤦 wrong hashing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Javier Palomo Almena	6852b1cdcc	Replace usage of sync/atomic with uber-go/atomic (#2449) ... * distributor: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * ingester: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Vendor go.uber.org/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Install faillint in the 'loki-build-image' Docker image Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * faillint: Ensure we are not using sync/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204) ... * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031) ... * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	3dfe89369f	Loki: refactor validation and improve error messages (#2021) ... * copies some of the validation logic from cortex and modifies it to appropriate for Loki, this gives us a single metric we can use for detetecting validation errors (previously we had to check cortex_ and loki_ metrics) as well as lets us use error messages more appropriate for logs than metrics. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fixing tests and lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * updating cause to string type Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting a function for rate limited Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting functions for the rest of the error strings Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Peter Štibraný	e60164bef6	Convert Loki modules to services (#1804) ... * Loki now uses module services to start and stop its work. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use services methods to initialize some components. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use Cortex' NewModuleService. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted server to a service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted distributor to service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use table manager service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * querier service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * query-frontend service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Merged stopping method into shutdown Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted ingester to a service. It now starts all background tasks in Starting state. Stopping needs little work, as does reacting on lifecycler errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Loki Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * If lifecycler fails, ingester fails too. It now doesn't call os.Exit, but shuts down gracefully and enters Failed state. That triggers Loki to shutdown completely. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore ErrStopProcess errors from services This is a signal that Loki should stop. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use single /ready handler It checks the state of all services, and asks ingester for its own check as well. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed unused value. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Tailers, not trailers. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Adds return for the healtcheck in case of error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	49f04ef4bc	update cortex to latest master (#1956)	5 years ago
Peter Štibraný	ca19683511	loki: update Cortex to master (#1799) ... * Update Cortex to master Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Integrate latest Cortex master changes into loki - Frontend worker now needs to be started explicitly - Lifecycler no longer exits on error. Distributor and Ingester do that now Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update Cortex to master. After 0.7 release, it is now showing as v0.7.1-... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785) ... * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	d6b85876ca	max line lengths (component + tenant overrides) (#1686) ... * bytesize flag support * max line limits * configurable tenant override options for max line size * bytesize yaml support * validator methods * distributor validator type * human friendly line size output, distributor test * invalid labels distributor test * docs + vendor * removes dead limits code	5 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484) ... * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Marco Pracucci	ce407d3276	Add ingestion rate global limit support (#1486) ... * Added ingestion rate global limit support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed code comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated config doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated doc Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Robert Fratto	a5765b57bf	pkg/distributor: fix metric loki_discarded_samples_total (#1279) ... loki_discarded_samples_total is supposed to be the count of samples that were discarded, but was instead being reported as the total bytes discarded. The metric has been fixed and a new metric, loki_discarded_bytes_total, has been added to report the total discarded bytes. Fixes #1255.	6 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088) ... * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948) ... * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
sh0rez	3467440b30	fix(distributor): parseError BadRequest (#975) ... When the label validation in the distributor failed, it was reported as 500 Internal Server Error instead of 400 BadRequest. Fixes that.	6 years ago
Sandeep Sukhani	b687ec6e5b	vendoring: update cortex to latest master (#938) ... * update cortex to latest master using v3.4.0-rc.1 for go.etcd.io/etcd v0.0.2 for github.com/prometheus/procfs v1.0.0 for github.com/prometheus/client_golang latest master for github.com/weaveworks/common with changes from https://github.com/weaveworks/common/pull/153 * fixed failing tests * use large instance for test and lint jobs in CircleCI * running only 6 test binaries in parallel * removed resource type change for CircleCI * changed GOGC to 10 for lint makefile target	6 years ago
Aditya C S	b5d8c77b99	Readiness probe for querier (#854)	6 years ago
Aditya C S	f72d2fba10	Readiness probe for Distributor	6 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541) ... * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	6 years ago
Sandeep Sukhani	f7f09e2226	Limits: Reject entries based on age set in limits (#631) ... * Reject entries based on age set in limits	6 years ago
Cyril Tovena	8d7efddde2	switch to golangci-lint linter (#539) ... * switch to golangci-lint linter * fixes for review comments	6 years ago
Tom Wilkie	e1c5c4be4a	Basic distributor & ingester tests. ... Proves that they can accept multiple samples with the same timestamp. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	7527487f31	Track the right number of streams, and cancel the push when the request is cancelled. ... This was broken in a422f394bb. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	a422f394bb	Wait for the right number of streams. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago