756 Commits (a62f81f1cf15d49e02958ae9a6097e84986ea336)

Author	SHA1	Message	Date
Robert Fratto	3451dd7c39	add RegisterFlagsWithPrefix to config structs (#2414) ... All config structs now have a RegisterFlagsWithPrefix function that allows specifying a prefixed string to append to all flag names. This may be used by clients to namespace support for Promtail in a larger application (e.g., grafana/agent). When implementing this PR, I noticed a few config structs were incorrectly registering to the global flagset rather than the flagset passed to RegisterFlags. This PR also includes a fix for that and ensures every option is registered to the flagset in the parameter. Related to #2405.	5 years ago
Sandeep Sukhani	807193d373	disable index dedupe when rf > 1 and current or upcoming index type is boltdb-shipper (#2206)	5 years ago
Ed Welch	204e61afe3	Loki-Canary: Backoff retries on query failures, add histograms for query performance. (#2413) ... * Canary needs to wait before spot-checking entries some reasonable amount of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present. Adding two histograms for tracking the performance of the spot check and metric test queries * Adding backoff to queries if they fail specifically waiting 5mins when a 429 is received from Loki * fix order of parameters	5 years ago
Vít Listík	2e1fd42540	Added support for tail to query frontend (#2032) ... * Added support for tail to query frontend * cleanup WsTailProxy * used httputil.NewSingleHostReverseProxy insted of custom implementation, added tailproxy config * proxy all unknown requests * rename proxy to tailProxy * default handler, if tailproxy url is set up default proxy is tailproxy * embedded lokifrontend conf which adds tail_proxy_url * fixed tail_proxy_url flag	5 years ago
Cyril Tovena	604ad1632e	Add regex function to promtail template stage. (#2386) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	76ceec45d9	If an entry is missing on the websocket, start checking after `wait` but continue checking until `max-wait` (#2369) ... Refactored out a useful pruneList function	5 years ago
Ed Welch	2b145da7d7	Canary tweaks (#2350) ... * tweak the canary a bit, add a metric for websocket pings, fix some logging, improve the mutex locking a bit, add a read timeout on the websocket, add a lastmessage time checker to restart websocket * fix constructor * fix counter name change metric count reporting to two separate metrics, expected vs actual * add note to upgrade guide for metric name changes * fixed tests.... * Update docs/operations/upgrade.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Aditya C S	8809de6087	refactor promtail target manager code (#2345)	5 years ago
Aditya C S	0885a49370	configuration doc improvements (#2318)	5 years ago
Ed Welch	81b9decfb5	Loki-Canary: Add query spot checking and metric count checking (#2344) ... * Add spot check capability to canary to query certain values over time to make sure they are still in Loki. Add a context timeout to Loki queries * Updating canary to use /loki/api/v1 endpoints adding a metric test query * fixing metric names to have _total for counters * fix parsing of timestamp from log line and don't use entry timestamp * Tweak the metric test query to adjust the range to the time the canary has been running to get accurate results until the requested range has been reached. Add a total spot check entry counter to get a ratio of success * Add some debugging logs to troubleshoot missing entries * add more tests, fix some log lines * update docs, make the spot-check timer separate from prune entries * just how many paramaters can I add to the constructor.... * run prune entries on a separate thread, only run one at a time.	5 years ago
Cyril Tovena	ce76ad099a	Fixes LogQL grouping (#2346) ... * Work in progress ! debugging this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * It was indeed the missing sort before processing aggregation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c9f89df930	Improve object key parsing for boltdb shipper. (#2323) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	0be64fcb34	Improve metric queries by computing samples at the edges. (#2293) ... * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	0b5996021f	Loki: add -print-config-stderr flag to dump loki's runtime config to stderr (#2336) ... * Loki: add -print-config-stderr flag to dump loki's runtime config to stderr * add ability to print inline config for better readability in grafana * add an option to log the config in reverse order to make viewing in Grafana easier. Also added this capability to Promtail	5 years ago
Aditya C S	b82d2e89d2	fix bug in ingester series query (#2264)	5 years ago
Aditya C S	01040c1a4d	Fix data race in ingester (#2327)	5 years ago
Ed Welch	635e469e9a	Loki: Use a new context to update the ring state after a failed chunk transfer (#2330) ... * Loki: Use a new context to update the ring state after a failed chunk transfer * Update pkg/ingester/transfer.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/transfer.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	edaf0ed422	transfers one chunk at a time per series instead of all (#2328)	5 years ago
Ed Welch	b24a143043	Promtail: Loki Push API (#2296) ... * refactor to put targets in separate packages * rename scrape package to scrapeconfig * Push implementation, working with logging * adding new vendored library * adding tests * adding documentation and an additional config option. * fixing journal test * rename scrape config entry * Update docs/clients/promtail/configuration.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * PR feedback Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	cd74043db9	Improve entry deduplication. (#2302) ... * Improve entry deduplication. This PR removes mostcommon and sort insert function in the heap iterator. I discovered while working on #2293 that those are actually not helping since we're deduping those lines anyways. There were no tests checking if deduping was correctly working so I did added those. Bonus point this means deduping will run faster and the code is less complex. The only side effect is that the order of entries that are at the same timestamp, before the most common entry would appear first, now we keep the same order as we stored them, which I think is better. I also change the label ordering, I think whether we are forward or backward we should keep the same aphabetical labels ordering not sure why direction was altering this before. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve heap iterator backward test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c149099955	Fixes buffered iterator skipping very long lines. (#2306) ... Funny enough this was triggered only with gzip and only with a certain size. But thanks for investigation here #2272 I was able to reproduce this with a test and indeed this was incorrect. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	68ac695fd8	Removes arm32 for now as we're migrating the CI. (#2307) ... * Removes arm32 for now as we're migrating the CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes flaky test, its being rewritten anyway. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Felix Ruess	616771aeff	Promtail: support unix timestamps with fractional seconds (#2301) ... * support unix timestamps with fractional seconds see #2193 * add test for parsing unix timestamp with fractions * mention Unix timestamp with fractions in docs * return error if timestamp is not split into two parts * don't capitalize error Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Roman Grebennikov	de644b52c3	promtail: fix high CPU usage on large kubernetes clusters. (#1118) ... * promtail: add api-level node filter for kubernetes discovery * promtail: fix hardcoded host value in k8s node filter	5 years ago
Periklis Tsirakidis	bc4f1f5833	Add support for memberlist dns-based discovery (#2288) ... * Bump cortex to 0ea5a8b5 to enable memberlist dns-based discovery * Update docs for memberlist config DNS-based discovery support * Fix mock chunk store client DeleteChunk signature	5 years ago
Ed Welch	aee064d0a8	Loki: Series API will return all series with no match or empty matcher (#2254) ... * Allow series queries with no match or an empty {} matcher which will return all series for the provided timeframe. * fix test * fix lint	5 years ago
Owen Diehl	48b8504c30	lock fix for flaky test (#2268)	5 years ago
Lukas Grimm	17390fd45d	Canary: make stream configurable (#2259) ... * tried to fix canary in non kubernetes env * update doc	5 years ago
Owen Diehl	70900be57d	avoids further time splitting in querysharding mware (#2252) ... * avoids further time splitting in querysharding mware * removes duplicate utility fns * linting	5 years ago
Ed Welch	60bf23d588	Loki: remove redundant log line on stream limit, the GRPC error logger already logs this. (#2250)	5 years ago
Owen Diehl	15378cf0e6	avoids recording stats in the sharded engine (#2249)	5 years ago
Cyril Tovena	6ab832b2ca	Cache overlapping blocks (#2239) ... * Caches block iteration that are overlapping together to avoid reprocessing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Final touch. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Calle Pettersson	baa264392b	Set user agent on outgoing http requests (#2242) ... Signed-off-by: Calle Pettersson <calle@cape.nu>	5 years ago
Sandeep Sukhani	9ab3800522	boltdb shipper download failure handling and some refactorings (#2156)	5 years ago
Cyril Tovena	91158abec1	Reuse slice for the range vector allocations. (#2219) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	721256403c	Reuse buffer for hash computation in the engine. (#2218) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	314531cb04	Improve point allocations for each steps in the logql engine. (#2216) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	1729592f38	Add Error method for step evaluators (#2223)	5 years ago
Periklis Tsirakidis	815c47533e	Replace memberlist service in favor of cortex provided service (#2224) ... * Bump cortex deps to include non-experimental memberlist service * Replace memberlist service in favor of cortex provided service * Fix route method param for querytee tool * Register memberlist flags * Add documentation and example for memberlist	5 years ago
Peter Štibraný	9bf1130f08	Remove single newline from the line before printing. (#2235) ... Lines are printed with Println, so this avoids having extra empty line between printed log lines. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204) ... * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Cyril Tovena	d08ceef16b	Fixes iterators boundaries. (#2136) ... The start should be inclusive and the end is supposed to be exclusive. If start and end are equal, then only one entry can be returned which need to also be equal to start. Fixes #2124 Those issues were edge cases where the boundaries of a block or iterator would be equal to the start. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Periklis Tsirakidis	6e55277552	Add flag to disable tracing (#2191)	5 years ago
Owen Diehl	61fc175361	Fix vector-scalar comparisons (#2189) ... * fmts shards in instance logger * vector-scalar comparisons now filter by default * gofmt	5 years ago
Owen Diehl	1530120c2e	Logql comparison ops (#2182) ... * comparison binops * starts comparison operator docs * implements bool modifier for comparison binops * vector-vector binop merging cases, including bool modifier * logql docs formatting * resets protos * linter fixes * goimports local * gofmt	5 years ago
Cyril Tovena	cb76da6c1d	Fixes path prefix in the querier. (#2178) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	b95db8d1fd	Fix error message for max tail connections. (#2161) ... * Fix error message for max tail connections. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	5b8d9b4ed6	Support series query in query-frontend (#2123)	5 years ago
Cyril Tovena	18f1b0640d	Bytes aggregations (#2150) ... * Add bytes_rate and bytes_over_time. Still need to works on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More lint fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad conflic resolution. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4c8199ec21	Validate local storage config for the logcli (#2126) ... * Allows to pass inlined pipeline stages to the docker driver. Fixes #1296 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add validation before using storage config for the logcli. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago