apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	e1ab2cc509	Cleanup labels parsing. (#2929 ) * Cleanup labels parsing. Make sure we only use one type of labels parsing.This just reduce the amount of code we have to maintain. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2303f1bfa1	Cache label strings in ingester to improve memory usage. (#2926 ) * Add benchmark for base stats. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cache labels string in ingester to improve memory usage. This reduce allocations by a lot, but assume that labels comes in sorted which is not ensure in the distributors. The distributors will take a performance hit, but that's easier to scale or improve later.(added some todos) see benchmark: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_PushInstance-16 43505 4950 -88.62% benchmark old allocs new allocs delta Benchmark_PushInstance-16 240 12 -95.00% benchmark old bytes new bytes delta Benchmark_PushInstance-16 42568 1787 -95.80% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 🤦 wrong hashing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Javier Palomo Almena	6852b1cdcc	Replace usage of sync/atomic with uber-go/atomic (#2449 ) * distributor: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * ingester: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Vendor go.uber.org/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Install faillint in the 'loki-build-image' Docker image Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * faillint: Ensure we are not using sync/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204 ) * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031 ) * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	3dfe89369f	Loki: refactor validation and improve error messages (#2021 ) * copies some of the validation logic from cortex and modifies it to appropriate for Loki, this gives us a single metric we can use for detetecting validation errors (previously we had to check cortex_ and loki_ metrics) as well as lets us use error messages more appropriate for logs than metrics. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fixing tests and lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * updating cause to string type Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting a function for rate limited Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting functions for the rest of the error strings Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Peter Štibraný	e60164bef6	Convert Loki modules to services (#1804 ) * Loki now uses module services to start and stop its work. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use services methods to initialize some components. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use Cortex' NewModuleService. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted server to a service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted distributor to service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use table manager service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * querier service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * query-frontend service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Merged stopping method into shutdown Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted ingester to a service. It now starts all background tasks in Starting state. Stopping needs little work, as does reacting on lifecycler errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Loki Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * If lifecycler fails, ingester fails too. It now doesn't call os.Exit, but shuts down gracefully and enters Failed state. That triggers Loki to shutdown completely. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore ErrStopProcess errors from services This is a signal that Loki should stop. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use single /ready handler It checks the state of all services, and asks ingester for its own check as well. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed unused value. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Tailers, not trailers. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Adds return for the healtcheck in case of error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	49f04ef4bc	update cortex to latest master (#1956 )	5 years ago
Peter Štibraný	ca19683511	loki: update Cortex to master (#1799 ) * Update Cortex to master Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Integrate latest Cortex master changes into loki - Frontend worker now needs to be started explicitly - Lifecycler no longer exits on error. Distributor and Ingester do that now Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update Cortex to master. After 0.7 release, it is now showing as v0.7.1-... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785 ) * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	d6b85876ca	max line lengths (component + tenant overrides) (#1686 ) * bytesize flag support * max line limits * configurable tenant override options for max line size * bytesize yaml support * validator methods * distributor validator type * human friendly line size output, distributor test * invalid labels distributor test * docs + vendor * removes dead limits code	5 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484 ) * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Marco Pracucci	ce407d3276	Add ingestion rate global limit support (#1486 ) * Added ingestion rate global limit support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed code comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated config doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated doc Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Robert Fratto	a5765b57bf	pkg/distributor: fix metric loki_discarded_samples_total (#1279 ) loki_discarded_samples_total is supposed to be the count of samples that were discarded, but was instead being reported as the total bytes discarded. The metric has been fixed and a new metric, loki_discarded_bytes_total, has been added to report the total discarded bytes. Fixes #1255.	6 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088 ) * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948 ) * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
sh0rez	3467440b30	fix(distributor): parseError BadRequest (#975 ) When the label validation in the distributor failed, it was reported as 500 Internal Server Error instead of 400 BadRequest. Fixes that.	6 years ago
Sandeep Sukhani	b687ec6e5b	vendoring: update cortex to latest master (#938 ) * update cortex to latest master using v3.4.0-rc.1 for go.etcd.io/etcd v0.0.2 for github.com/prometheus/procfs v1.0.0 for github.com/prometheus/client_golang latest master for github.com/weaveworks/common with changes from https://github.com/weaveworks/common/pull/153 * fixed failing tests * use large instance for test and lint jobs in CircleCI * running only 6 test binaries in parallel * removed resource type change for CircleCI * changed GOGC to 10 for lint makefile target	6 years ago
Aditya C S	b5d8c77b99	Readiness probe for querier (#854 )	6 years ago
Aditya C S	f72d2fba10	Readiness probe for Distributor	6 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541 ) * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	6 years ago
Sandeep Sukhani	f7f09e2226	Limits: Reject entries based on age set in limits (#631 ) * Reject entries based on age set in limits	6 years ago
Cyril Tovena	8d7efddde2	switch to golangci-lint linter (#539 ) * switch to golangci-lint linter * fixes for review comments	6 years ago
Tom Wilkie	e1c5c4be4a	Basic distributor & ingester tests. Proves that they can accept multiple samples with the same timestamp. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	7527487f31	Track the right number of streams, and cancel the push when the request is cancelled. This was broken in `a422f394bb`. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	a422f394bb	Wait for the right number of streams. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	ab4c5bebab	Validate labels in the distributor. (#251 ) Also, make ErrOutOfOrder a HTTP 4xx error and continue to append even when we encourter an out of order entry. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Goutham Veeramachaneni	9b67d6cb7a	Rename instance to tenant to avoid clash with Prom (#164 ) Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Goutham Veeramachaneni	7bcdf0e0b1	Tenant metrics (#135 ) * Fix stream tracking metric Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Track lines and bytes received per user Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use instance instead of org Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Nuke dead code Usage has been removed as part of perf-optimisation: `3006e3901d (diff-180f56d9aaf5a9aa079e6fc9cfcf1bc8L365)` Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36 ) * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27 ) * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3b69cf127d	Don't repeat client config. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	0ffb432f30	Update for cortex changes. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3f62b4ec36	s/logish/tempo/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	cc7d4e5aa9	Fix NPE. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	9b2c27d213	Various small nits. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	58b0f3a1d4	Add ingester lifecycle.	7 years ago
Tom Wilkie	01fd5c63d7	Basic queier boilerplate.	7 years ago
Tom Wilkie	c0af4c06c8	Build distributor image.	7 years ago
Tom Wilkie	ca6dd2c8d5	Add scaffold and distributor implementation.	7 years ago

41 Commits (5572366d9363b33b7e6fe530be8bfe73f30cd53f)