apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	e1ab2cc509	Cleanup labels parsing. (#2929 ) * Cleanup labels parsing. Make sure we only use one type of labels parsing.This just reduce the amount of code we have to maintain. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	049c7acc12	Improve logql parser allocations. (#2927 ) * Improve logql parser allocations. Also moving labels parsing into logql package, and I'm not shy to say that promql is faster for this. So we're using promql for parser, it's faster because their lexer is more sophisticated than ours. Result are impressive. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_ParseLabels-16 24394 8745 -64.15% benchmark old allocs new allocs delta Benchmark_ParseLabels-16 156 20 -87.18% benchmark old bytes new bytes delta Benchmark_ParseLabels-16 24976 2099 -91.60% ``` Combined with #2926, I think we should be able to get rid of `util.ToClientLabels`. But that's for another PR. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2303f1bfa1	Cache label strings in ingester to improve memory usage. (#2926 ) * Add benchmark for base stats. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cache labels string in ingester to improve memory usage. This reduce allocations by a lot, but assume that labels comes in sorted which is not ensure in the distributors. The distributors will take a performance hit, but that's easier to scale or improve later.(added some todos) see benchmark: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_PushInstance-16 43505 4950 -88.62% benchmark old allocs new allocs delta Benchmark_PushInstance-16 240 12 -95.00% benchmark old bytes new bytes delta Benchmark_PushInstance-16 42568 1787 -95.80% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 🤦 wrong hashing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794 )	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	21d3e69790	Enable local rules backend & disallow configdb. (#2661 ) * revendors cortex * vendoring compat * disallow configdb ruler backend * go mod tidy	5 years ago
Aurélien Rainone	21addf7935	pkg/distributor: fix test by looking for loopback interface (#2645 ) TestDistributor_PushIngestionRateLimiter failed if the system had no interface name in "eth0", "en0" or "lo0". Fix that by looking at the name of the loopback interface in the list of the system's network interfaces, then using that name in the test. Fixes #2356	5 years ago
Owen Diehl	164f5cd0ae	Ruler/loki rule validator (#2589 ) * revendors cortex * loki multitenantmanager * vendoring compat * go mod	5 years ago
Sandeep Sukhani	501f93fa92	revendor cortex to latest master (#2543 )	5 years ago
Javier Palomo Almena	6852b1cdcc	Replace usage of sync/atomic with uber-go/atomic (#2449 ) * distributor: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * ingester: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Vendor go.uber.org/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Install faillint in the 'loki-build-image' Docker image Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * faillint: Ensure we are not using sync/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	5 years ago
Ed Welch	b24a143043	Promtail: Loki Push API (#2296 ) * refactor to put targets in separate packages * rename scrape package to scrapeconfig * Push implementation, working with logging * adding new vendored library * adding tests * adding documentation and an additional config option. * fixing journal test * rename scrape config entry * Update docs/clients/promtail/configuration.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * PR feedback Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204 ) * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Aditya C S	199e334126	Update loki to cortex master (#2030 )	5 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031 ) * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	272a89c13d	more linter fixes (#2024 ) * more linter woes Signed-off-by: Ed Welch <edward.welch@grafana.com> * feedback Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	3dfe89369f	Loki: refactor validation and improve error messages (#2021 ) * copies some of the validation logic from cortex and modifies it to appropriate for Loki, this gives us a single metric we can use for detetecting validation errors (previously we had to check cortex_ and loki_ metrics) as well as lets us use error messages more appropriate for logs than metrics. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fixing tests and lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * updating cause to string type Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting a function for rate limited Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting functions for the rest of the error strings Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Peter Štibraný	e60164bef6	Convert Loki modules to services (#1804 ) * Loki now uses module services to start and stop its work. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use services methods to initialize some components. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use Cortex' NewModuleService. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted server to a service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted distributor to service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use table manager service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * querier service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * query-frontend service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Merged stopping method into shutdown Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted ingester to a service. It now starts all background tasks in Starting state. Stopping needs little work, as does reacting on lifecycler errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Loki Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * If lifecycler fails, ingester fails too. It now doesn't call os.Exit, but shuts down gracefully and enters Failed state. That triggers Loki to shutdown completely. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore ErrStopProcess errors from services This is a signal that Loki should stop. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use single /ready handler It checks the state of all services, and asks ingester for its own check as well. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed unused value. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Tailers, not trailers. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Adds return for the healtcheck in case of error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	49f04ef4bc	update cortex to latest master (#1956 )	5 years ago
Peter Štibraný	ca19683511	loki: update Cortex to master (#1799 ) * Update Cortex to master Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Integrate latest Cortex master changes into loki - Frontend worker now needs to be started explicitly - Lifecycler no longer exits on error. Distributor and Ingester do that now Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update Cortex to master. After 0.7 release, it is now showing as v0.7.1-... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785 ) * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
talham7391	fa8bf2acba	Clearer error response from push endpoint when labels are malformed (#1750 ) * Clearer error response from push endpoint when labels are malformed * Concise error formatting	5 years ago
Cyril Tovena	4fdfbe8e56	Increase discarded samples when line is too long. (#1708 ) * Increase discarded samples when line is too long. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add comment on the public props Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Joe Elliott <joe.elliott@grafana.com>	5 years ago
Aditya C S	f9ff2a4b02	Add goimport local-prefixes configuration to .golangci.yml (#1710 )	5 years ago
Owen Diehl	d6b85876ca	max line lengths (component + tenant overrides) (#1686 ) * bytesize flag support * max line limits * configurable tenant override options for max line size * bytesize yaml support * validator methods * distributor validator type * human friendly line size output, distributor test * invalid labels distributor test * docs + vendor * removes dead limits code	5 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661 ) * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484 ) * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Marco Pracucci	ce407d3276	Add ingestion rate global limit support (#1486 ) * Added ingestion rate global limit support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed code comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated config doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated doc Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Robert Fratto	a5765b57bf	pkg/distributor: fix metric loki_discarded_samples_total (#1279 ) loki_discarded_samples_total is supposed to be the count of samples that were discarded, but was instead being reported as the total bytes discarded. The metric has been fixed and a new metric, loki_discarded_bytes_total, has been added to report the total discarded bytes. Fixes #1255.	6 years ago
Cyril Tovena	d438bda94a	Update cortex to bring v11 schema (#1201 ) * Update cortex to bring v11 schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes change in ParseProtoReader Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * sync go mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Joe Elliott	bb2b925255	Update `/loki/api/v1/push` to use the v1 json format (#1145 ) Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088 ) * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948 ) * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
sh0rez	3467440b30	fix(distributor): parseError BadRequest (#975 ) When the label validation in the distributor failed, it was reported as 500 Internal Server Error instead of 400 BadRequest. Fixes that.	6 years ago
Sandeep Sukhani	b687ec6e5b	vendoring: update cortex to latest master (#938 ) * update cortex to latest master using v3.4.0-rc.1 for go.etcd.io/etcd v0.0.2 for github.com/prometheus/procfs v1.0.0 for github.com/prometheus/client_golang latest master for github.com/weaveworks/common with changes from https://github.com/weaveworks/common/pull/153 * fixed failing tests * use large instance for test and lint jobs in CircleCI * running only 6 test binaries in parallel * removed resource type change for CircleCI * changed GOGC to 10 for lint makefile target	6 years ago
Aditya C S	b5d8c77b99	Readiness probe for querier (#854 )	6 years ago
Aditya C S	f72d2fba10	Readiness probe for Distributor	6 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541 ) * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	6 years ago
Sandeep Sukhani	f7f09e2226	Limits: Reject entries based on age set in limits (#631 ) * Reject entries based on age set in limits	6 years ago
Cyril Tovena	8d7efddde2	switch to golangci-lint linter (#539 ) * switch to golangci-lint linter * fixes for review comments	6 years ago
Tom Wilkie	e1c5c4be4a	Basic distributor & ingester tests. Proves that they can accept multiple samples with the same timestamp. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	7527487f31	Track the right number of streams, and cancel the push when the request is cancelled. This was broken in `a422f394bb`. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	f6acb87d1f	Correctly propagate errors in the distributor; should now return 4xx on OOO entries. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	a422f394bb	Wait for the right number of streams. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	ab4c5bebab	Validate labels in the distributor. (#251 ) Also, make ErrOutOfOrder a HTTP 4xx error and continue to append even when we encourter an out of order entry. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Goutham Veeramachaneni	9b67d6cb7a	Rename instance to tenant to avoid clash with Prom (#164 ) Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	da27baad2c	Support for JSON, not just protos. (#152 ) * Support for JSON, not just protos. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * s/timestamp/ts/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	7bcdf0e0b1	Tenant metrics (#135 ) * Fix stream tracking metric Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Track lines and bytes received per user Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use instance instead of org Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Nuke dead code Usage has been removed as part of perf-optimisation: `3006e3901d (diff-180f56d9aaf5a9aa079e6fc9cfcf1bc8L365)` Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36 ) * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27 ) * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3b69cf127d	Don't repeat client config. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago

1 2

60 Commits (a62f81f1cf15d49e02958ae9a6097e84986ea336)