756 Commits (a62f81f1cf15d49e02958ae9a6097e84986ea336)

Author	SHA1	Message	Date
Peter Štibraný	e4a5360586	Update vendored Cortex to master (1.1+) (#2149) ... * Updated Cortex to master (post 1.1) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use new modules system from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Treat query errors reported by chunk store as client errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed obsolete tests. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix race for ct variable. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Aditya C S	38ee57c7d5	Feature: Replace stage in pipeline (#2060)	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927) ... * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit 55d41b9519. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago
Jérôme Loyet	665712ff18	Add Entry variable to template (#2105) ... The new Entry variable is the log message that could be needed in template. For instance, if I want to convert all my log line to lowercase: - template: source: message template: '{{ ToLower .Entry }}'	5 years ago
Sebastian Widmer	d175fa58c8	Update go-syslog dependency to accept any encoding in message (#2127)	5 years ago
Aditya C S	ff0a4c35f4	Set JournalTarget Priority value to keyword (#2087)	5 years ago
Aditya C S	f984e71192	Fix Promtail journal seeking known position (#2111)	5 years ago
Aditya C S	360d9a7f97	Support querying labels on time range in logcli (#2083)	5 years ago
Cyril Tovena	f9a2c436ac	Adds backtick for the quoted string token lexer. (#2095) ... * Fixes unit in stats request log. (#2093) * Fix unit conversion in stats log. 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix the tests because you know ! you know ? Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add backtick for the quoted string token lexer. This allows to pass unescaped string to Loki. For example we now support this : `` {name="cassandra"} |~ `error=\w+` `` `` {name!~`mysql-\d+`} `` As you can see no escape is required when using ` quoted strings. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes newlines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add back missing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c81be6603e	Fixes unit in stats request log. (#2093) ... * Fix unit conversion in stats log. 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix the tests because you know ! you know ? Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	0520a0ac1c	allow no encoding/compression on chunks (#2088) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Owen Diehl	e0dacd2795	removes yolostring (#2078)	5 years ago
Cyril Tovena	0de1a42406	Fixes a panic in dry-run when using external labels. (#2075) ... Fixes #1972. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	9803eab947	Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper (#2073) ... * Allow configuring query_store_max_look_back_period when running a filesystem store, which allows for boltdb-shipper to horizontally scale filesystem store. Signed-off-by: Ed Welch <edward.welch@grafana.com> * Update docs/operations/storage/boltdb-shipper.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * add some more precision to the timestamp used to make the name unique just to make sure there are not collisions if for some reason the ID is empty (which can happen if the hostname doesn't resolve for some reason) Signed-off-by: Ed Welch <edward.welch@grafana.com> * Allow the ingester to query the store for labels if query store is enabled. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make queriers aware of the fact that ingesters may be querying the store so they don't duplicate query efforts Signed-off-by: Ed Welch <edward.welch@grafana.com> * improve the docs Signed-off-by: Ed Welch <edward.welch@grafana.com> * add tests to the calculateMaxLookBack Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Ed Welch	7dd109763c	expand the match_all to also optionally allow counting bytes per stream (#2066) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	3a28a17946	some metrics for measuring performance and failures in boltdb shipper (#2034) ... * some metrics for measuring performance and failures in boltdb shipper * counting both failures and success in upload and download operation in boltdb shipper * changes suggested from PR review	5 years ago
Ed Welch	f206165c4f	output a debug log if the requested path in the scrape config doesn't match any files (#2069) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Aditya C S	11937d3c3a	Targets not required in promtail config (#2026) ... * Targets not required in promtail config * changing the defaulted value from `dummy` to `localhost`, I think this will be more intuitive for people, also re-worded the docs a bit. Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Aditya C S	199e334126	Update loki to cortex master (#2030)	5 years ago
Cyril Tovena	a8b94aec3d	Reverse entry iterator pool (#2064) ... * wi Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031) ... * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	bce4470a5b	Recover from panic in http and grpc handlers. (#2059) ... * Recover from panic in http and grpc handlers. I don't see any good reason to crash any component during a bad request. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add alerts to the mixin for panics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 😡 gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	fcb4c763dc	Fix a bug in range vector skipping data. (#2058) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	83873d1c6f	Fix promtail client default values (#2049)	5 years ago
Arve Knudsen	d975b1fdd8	Fix spelling of per second (#2055) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Arve Knudsen	cc98d10d74	Chore: Correct typo "per seconds" (#2050) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Goutham Veeramachaneni	06b50590a7	Fix bug in logql parsing that leads to crash. (#2046) ... * Fix bug in logql parsing that leads to crash. ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x20ae356] goroutine 39530021 [running]: github.com/grafana/loki/pkg/logql.andFilter.Filter(0x0, 0x0, 0x3beffc0, 0xc01f065bc0, 0xc01aaaec00, 0x13d, 0x200, 0xffffffffffffffff) /src/loki/pkg/logql/filter.go:66 +0x26 github.com/grafana/loki/pkg/chunkenc.(*bufferedIterator).Next(0xc00a48fba0, 0x0) /src/loki/pkg/chunkenc/memchunk.go:616 +0x1cb github.com/grafana/loki/pkg/iter.(*nonOverlappingIterator).Next(0xc00130ae40, 0x0) /src/loki/pkg/iter/iterator.go:438 +0x135 github.com/grafana/loki/pkg/iter.(*timeRangedIterator).Next(0xc00130ae80, 0x0) /src/loki/pkg/iter/iterator.go:497 +0x48 github.com/grafana/loki/pkg/iter.(*reverseIterator).load(0xc00072ac40) /src/loki/pkg/iter/iterator.go:555 +0x70 github.com/grafana/loki/pkg/iter.(*reverseIterator).Next(0xc00072ac40, 0x0) /src/loki/pkg/iter/iterator.go:563 +0x2f github.com/grafana/loki/pkg/iter.(*nonOverlappingIterator).Next(0xc00130af40, 0x10) /src/loki/pkg/iter/iterator.go:438 +0x135 ``` This is caused by people doing `|= ""` which would result in parsing of TrueFilter but would be returned as nil. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Add tests for the panic Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	5 years ago
Cyril Tovena	4fd670dc7a	Support Post request in the frontend queryrange handler. (#2023) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	4906fb2065	logcli: Query needs to be stored into url.RawQuery, and not url.Path (#2027) ... * Query needs to be stored into url.RawQuery, and not url.Path Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * QueryStringBuilder deals with queries, not paths. Removed unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Ed Welch	272a89c13d	more linter fixes (#2024) ... * more linter woes Signed-off-by: Ed Welch <edward.welch@grafana.com> * feedback Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	4f9a2fe2d9	I <3 you Linter (#2022) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	3dfe89369f	Loki: refactor validation and improve error messages (#2021) ... * copies some of the validation logic from cortex and modifies it to appropriate for Loki, this gives us a single metric we can use for detetecting validation errors (previously we had to check cortex_ and loki_ metrics) as well as lets us use error messages more appropriate for logs than metrics. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fixing tests and lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * updating cause to string type Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting a function for rate limited Signed-off-by: Ed Welch <edward.welch@grafana.com> * extracting functions for the rest of the error strings Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Josh Soref	0d9cf4946a	Spelling (#2014) ... * spelling: access * spelling: actual * spelling: asynchronously * spelling: circleci * spelling: circle * spelling: cognitive * spelling: comment * spelling: computation * spelling: correct * spelling: daemon * spelling: elliott * spelling: enabled * spelling: encbuf * spelling: extracted * spelling: explicitly * spelling: independent * spelling: languages * spelling: loose * spelling: message * spelling: metrics * spelling: mismatch * spelling: nonexistent * spelling: output * spelling: pipeline * spelling: pipe line * spelling: position * spelling: prepend * spelling: query * spelling: receive * spelling: received * spelling: reducible * spelling: registry * spelling: response * spelling: retryable * spelling: scanning * spelling: sourced * spelling: supported * spelling: template * spelling: timeout * spelling: unsupported * spelling: adapter * spelling: deferring * spelling: expirable	5 years ago
Ed Welch	872a998fbb	make `loki_ingester_memory_streams` Gauge per tenant. (#2019) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	0ab1b28812	Loki: Improve logging and add metrics to streams dropped by stream limit (#2012) ... * Improve the log message when we drop streams because a user is hitting a stream limit. Increment the dropped samples metrics when this happens also. Signed-off-by: Ed Welch <edward.welch@grafana.com> * improving comments Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	5e34df8caf	Fix a bad rebase between #1970 and #1965. (#2011) ... * Fix a bad rebase between #1970 and #1965. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fmt Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	34f9c601c9	Adds config to disable HTTP and GRPC server in Promtail. (#2004) ... Fixes #1991. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	dc5cf8f57c	Allow to aggregate binary operations. (#1970) ... * Allow to aggregate binary operations. For example: `sum by (job) (count_over_time({namespace="tns"}[5m] |= "level=error") / count_over_time({namespace="tns"}[5m]))` would have failed previously. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing () rules in yacc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	989fb106a0	table client for boltdb shipper to enforce retention (#2001) ... * table client for boltdb shipper to enforce retention	5 years ago
Cyril Tovena	d70fc0e45f	Improve URL building in the logcli to strip trailing /. (#2000) ... Fixes #1985. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	577d8eb168	Ensure trace propagation in our logs. (#1977) ... * Ensure trace propagation in our logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing import. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix function usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	c7a1a1fb2e	Loki: Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval (#1965) ... * Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval Signed-off-by: Ed Welch <edward.welch@grafana.com> * lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * extract function for parsing parameter Signed-off-by: Ed Welch <edward.welch@grafana.com> * Clarify the experimental nature of interval in the docs. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make min time a constant Signed-off-by: Ed Welch <edward.welch@grafana.com> * Expand the docs around the experimental nature of interval a little more Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	21d4ca41e2	make boltdb shipper singleton and some other minor refactoring (#1995) ... * make boltdb shipper singleton and some other minor refactoring	5 years ago
Cyril Tovena	47b79f3ea5	Update cortex to latest. (#1982) ... * Update cortex to latest. This brings fixes for traceID in the logger and swift. I've also added a bit of configuration for swift. Fixes https://github.com/grafana/loki/issues/905 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update go.sum Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	46cfdaede0	Add a missing method to facade which is called by the metrics storage client in cortex (#1987) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Aditya C S	c0a28cbf30	Log error message for invalid checksum (#1713)	5 years ago
Ed Welch	b89a1f81dd	Update cortex vendoring to latest master (#1975) ... * Update cortex vendoring to latest master checksum `36496a074bc4530fb1d7c74136cab1566fbd9583` Signed-off-by: Ed Welch <edward.welch@grafana.com> * handle all the breaking changes from cortes Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * rebasing on master, re-runing go mod vendor seemed to bring in some more changes... Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Peter Štibraný	e60164bef6	Convert Loki modules to services (#1804) ... * Loki now uses module services to start and stop its work. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use services methods to initialize some components. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use Cortex' NewModuleService. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted server to a service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted distributor to service. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use table manager service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * querier service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * query-frontend service Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Merged stopping method into shutdown Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Converted ingester to a service. It now starts all background tasks in Starting state. Stopping needs little work, as does reacting on lifecycler errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Loki Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * If lifecycler fails, ingester fails too. It now doesn't call os.Exit, but shuts down gracefully and enters Failed state. That triggers Loki to shutdown completely. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore ErrStopProcess errors from services This is a signal that Loki should stop. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use single /ready handler It checks the state of all services, and asks ingester for its own check as well. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed unused value. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Lint Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Tailers, not trailers. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Adds return for the healtcheck in case of error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	bd4d516262	chunk bounds metric now records 8h range in 1h increments (#1964)	5 years ago
Sandeep Sukhani	fad3b61be7	flush boltdb to object store (#1837) ... * flush boltdb to object store files are stored in folder per periodic table and are named after ingester flushed every 15 mins to make index available to other services files are also flushed before ingester stops to avoid any data loss new stores can be implemented easily ingester to also query store when using boltdb * persisting uploader name across restarts, detecting objectstore type from periodic config, other refactorings * updated cli flag for active directory * add tests for boltdb shipper and update vendor * syncing boltdb files to disk during update * sleep in tests to let mtime of boltdb file be changed * add documentation for boltdb shipper and remove unwanted config	5 years ago