740 Commits (6e79cee85d8d5b62e524f08ae54a7d0f2f55a10b)

Author	SHA1	Message	Date
Ed Welch	b4c6065b97	implement shutdown for the no-op server (#2542)	5 years ago
Sandeep Sukhani	048e8bdb9c	minor fixes and improvements for boltdb shipper (#2530) ... 1. Continue uploading of other tables during failures. 2. Remove .temp files while loading dbs during startup. 3. Remove empty table folders while loading tables during startup.	5 years ago
Owen Diehl	6fe414eb25	consistent chunk metrics (#2534)	5 years ago
Ed Welch	b6d9fd5a00	restart the tailer if we fail to read and upate current position (#2532)	5 years ago
Owen Diehl	6b6b33f9eb	adds batch based metrics (#2510) ... * adds batch based metrics * changing metric names a bit Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	82845e4f78	Adds --analyize-labels to logcli series command. (#2497) ... Changes the series command to use the common matcher input found in the query and instant-query commands, instead of `logcli series --matcher='{foo="bar"}'` it's now `logcli series '{foo="bar"}'`	5 years ago
Ed Welch	e69b64f6bf	Promtail: Drop stage (#2496) ... * Adding a drop stage, it's very broken but want feedback on the design in the docs. * documentation fixes * documentation fixes * implemented and added tests * add a metric to track dropped logs with a configurable reason label.	5 years ago
Ward Bekker	8ff8b42899	Added Sha3 as alternative (#2480) ... * Added Sha3 as alternative * Added crypto x dependency to modules.txt * CI not picking up explicit module def * CI not picking up explicit module def * CI not picking up explicit module def * add x/crypto/sha3/ to vendor dir * Implemented review feedback * doc fix * Update docs/sources/clients/promtail/stages/replace.md Co-authored-by: Ed Welch <ed@oqqer.com> * Update docs/sources/clients/promtail/stages/template.md Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	5 years ago
Sandeep Sukhani	cf64b70db3	add some spans for boltdb shipper index queries (#2495)	5 years ago
Sandeep Sukhani	f1d00807eb	fix a bug in ingester doing store queries when using boltdb shipper (#2493)	5 years ago
Ed Welch	44f8dc3e46	enable the MustExist flag when starting a tailer on a fail, this will cause the tail to fail fast and allow promtail to attempt to re-open the file. (#2491)	5 years ago
Ed Welch	80693cae6f	Logcli: automatically batch requests (#2482) ... * first stab at batching logcli requests * fix other calls to printResult * working batching code * refactor the logcli Client into an interface so that it can be mocked for testing. * fixing some bugs * refactoring the output interface to make testing easier * starting to add tests * adding a bunch of tests fixing up how the MockQuerier works. * updating docs * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/getting-started/logcli.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>	5 years ago
Ed Welch	7418d42141	Promtail: force the log level on any Loki Push API target servers to match Promtail's log level. (#2475) ... * force the log level on any Loki Push API target servers to match Promtail's log level. * also include logformat	5 years ago
Ed Welch	b8a61357bb	Loki: fix batch iterator error when all chunks overlap and chunk time ranges are greater than query time range (#2484) ... * Fixes a bug where if all chunks are fetched before finding a non-overlapping chunk, and the chunk from/through times are outside of the query request times, we would incorrectly set the iterator from/to bounds leading to missing logs in the query. * uncomment `includesOverlap` which was commented during testing and not fixed by mistake.	5 years ago
Sandeep Sukhani	c5bc41628a	download boltdb files parallelly during reads (#2483) ... * download boltdb files parallelly during reads * changes suggested from PR review	5 years ago
Aditya C S	68568f141e	colored labels output for logcli (#2470) ... * colored labels output for logcli * add period to all cli description	5 years ago
Ed Welch	36164bbe1a	Promtail: use --client.external-labels for all clients (#2474) ... * --client.external-labels command line argument will now apply the labels to all clients configured in the clients config section of the config file. * do better Ed	5 years ago
Owen Diehl	f8ee408180	series endpoint uses normal splits (#2472)	5 years ago
Owen Diehl	05e0e91d6e	Fix/promtail yaml config (#2471) ... * promtail config cloneable * config parsing compile time checks against cloneable	5 years ago
Owen Diehl	7530bf6def	avoid mutating config while parsing -config.file (#2392) ... * avoid mutating config while parsing -config.file * minimal config mutation test case * logcli local query config file compat	5 years ago
Owen Diehl	cb20afaa26	BatchIter edge cases (#2466) ... * sampleIter uses correct cache * minimal lazy chunk iterator test * memchunk Blocks() inclusivity * fixes a few edge cases in the batchiterator batching * fixes bad metric name in test * due to later chunks len check, resetting nextChunk is unnecessary * lazychunks pop test * safe starting of batchChunkIterator * batchiter rudimentary safe start test	5 years ago
Sandeep Sukhani	4a358bc9ea	revendor cortex to latest master (#2463)	5 years ago
Ed Welch	096976511f	Make a clone of the LabelSet before handling, the handler contains a full pipeline and can modify the labels uniquely per each line which will lead to some concurrent map access panics if you don't pass a copy of the labels to each Handle invocation. (#2464)	5 years ago
Aditya C S	0c4f704deb	fix panic in cassandra storage while registering metrics (#2457)	5 years ago
Ed Welch	03e9059ecb	Loki: Improve error messages on query timeout or cancel (#2453) ... * Instead of returning "context deadline exceeded" on timeout and "context cancelled" when the query is cancelled, return a more descriptive error message. * fixing tests	5 years ago
Javier Palomo Almena	6852b1cdcc	Replace usage of sync/atomic with uber-go/atomic (#2449) ... * distributor: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * ingester: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Vendor go.uber.org/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Install faillint in the 'loki-build-image' Docker image Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * faillint: Ensure we are not using sync/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	5 years ago
Aditya C S	7eaaab608a	fix runtime_config panic (#2450)	5 years ago
Sandeep Sukhani	d5c840bd66	enforce requirment for periodic config for index tables to be 24h when using boltdb shipper (#2166)	5 years ago
Cyril Tovena	9ad98dfa7c	Split label names queries in the frontend. (#2441) ... * Split the Series API by 24h. We currently split too much by using the same split duration for query range. Let's start with 24h, just tested in ops-tools and it works better. We could later introduce a config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Split label name queries in the frontend. Build on top of #2436 and adds the ability to split label names queries. Those queries are heavy to run for long period and so splitting by day should improve query performance. The frontend also support the old legacy label names API which has a different response. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Robert Rimoczi	830174554c	allow empty replace values for replace stage (#2432)	5 years ago
Robert Fratto	b5f0c75c03	pkg/promtail: propagate a logger rather than using util.Logger globally (#2438) ... * pkg/promtail: propagate a logger rather than using util.Logger globally This commit allows for creating promtail with a custom log.Logger instance which will be propagated and used consistently throughout the Promtail package. This allows for clients to provide a Promtail-specific logger. Address a portion of #2405. * remove applyOptions function	5 years ago
Ward Bekker	0fa98d81ca	Template: Added a sha256 template function for obfuscating / anonymize PII data in e.g. the replace stage (#2422) ... * Added sha256 template method * updated hash result * Add replace with hashing example * use static hash function	5 years ago
Owen Diehl	4a45ca5e3a	Revendor cortex (#2427) ... * vendoring bump * vendoring compat	5 years ago
Sandeep Sukhani	e59adccfa0	create smaller unique files from boltdb shipper and other code refactorings (#2261) ... * rename package local to shipper * reorganized files to separate out uploads and downloads * create smaller unique files from boltdb shipper and other improvements * cancellation of downloading of initial files when service is being stopped * migration of existing files to dedicated folders and cleanup of index from ingesters * some more improvements to uploads and downloads code * tests for uploads and downloads package in shipper Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * test cleanup fix and some other minor changes * refactor getting uploader from object key * better naming * changes suggested from PR review * Fixes goroutine loop. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Simplify getoradd table. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes loop in upload manager. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * some more tests, logs and changes suggested from PR review * move ready state check for downloaded table inside query function * add info log for successfully finishing uploading of a table * fix merge conflict * fix ingester name and mode config Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Robert Fratto	3451dd7c39	add RegisterFlagsWithPrefix to config structs (#2414) ... All config structs now have a RegisterFlagsWithPrefix function that allows specifying a prefixed string to append to all flag names. This may be used by clients to namespace support for Promtail in a larger application (e.g., grafana/agent). When implementing this PR, I noticed a few config structs were incorrectly registering to the global flagset rather than the flagset passed to RegisterFlags. This PR also includes a fix for that and ensures every option is registered to the flagset in the parameter. Related to #2405.	5 years ago
Sandeep Sukhani	807193d373	disable index dedupe when rf > 1 and current or upcoming index type is boltdb-shipper (#2206)	5 years ago
Ed Welch	204e61afe3	Loki-Canary: Backoff retries on query failures, add histograms for query performance. (#2413) ... * Canary needs to wait before spot-checking entries some reasonable amount of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present. Adding two histograms for tracking the performance of the spot check and metric test queries * Adding backoff to queries if they fail specifically waiting 5mins when a 429 is received from Loki * fix order of parameters	5 years ago
Vít Listík	2e1fd42540	Added support for tail to query frontend (#2032) ... * Added support for tail to query frontend * cleanup WsTailProxy * used httputil.NewSingleHostReverseProxy insted of custom implementation, added tailproxy config * proxy all unknown requests * rename proxy to tailProxy * default handler, if tailproxy url is set up default proxy is tailproxy * embedded lokifrontend conf which adds tail_proxy_url * fixed tail_proxy_url flag	5 years ago
Cyril Tovena	604ad1632e	Add regex function to promtail template stage. (#2386) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	76ceec45d9	If an entry is missing on the websocket, start checking after `wait` but continue checking until `max-wait` (#2369) ... Refactored out a useful pruneList function	5 years ago
Ed Welch	2b145da7d7	Canary tweaks (#2350) ... * tweak the canary a bit, add a metric for websocket pings, fix some logging, improve the mutex locking a bit, add a read timeout on the websocket, add a lastmessage time checker to restart websocket * fix constructor * fix counter name change metric count reporting to two separate metrics, expected vs actual * add note to upgrade guide for metric name changes * fixed tests.... * Update docs/operations/upgrade.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Aditya C S	8809de6087	refactor promtail target manager code (#2345)	5 years ago
Aditya C S	0885a49370	configuration doc improvements (#2318)	5 years ago
Ed Welch	81b9decfb5	Loki-Canary: Add query spot checking and metric count checking (#2344) ... * Add spot check capability to canary to query certain values over time to make sure they are still in Loki. Add a context timeout to Loki queries * Updating canary to use /loki/api/v1 endpoints adding a metric test query * fixing metric names to have _total for counters * fix parsing of timestamp from log line and don't use entry timestamp * Tweak the metric test query to adjust the range to the time the canary has been running to get accurate results until the requested range has been reached. Add a total spot check entry counter to get a ratio of success * Add some debugging logs to troubleshoot missing entries * add more tests, fix some log lines * update docs, make the spot-check timer separate from prune entries * just how many paramaters can I add to the constructor.... * run prune entries on a separate thread, only run one at a time.	5 years ago
Cyril Tovena	ce76ad099a	Fixes LogQL grouping (#2346) ... * Work in progress ! debugging this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * It was indeed the missing sort before processing aggregation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c9f89df930	Improve object key parsing for boltdb shipper. (#2323) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	0be64fcb34	Improve metric queries by computing samples at the edges. (#2293) ... * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	0b5996021f	Loki: add -print-config-stderr flag to dump loki's runtime config to stderr (#2336) ... * Loki: add -print-config-stderr flag to dump loki's runtime config to stderr * add ability to print inline config for better readability in grafana * add an option to log the config in reverse order to make viewing in Grafana easier. Also added this capability to Promtail	5 years ago
Aditya C S	b82d2e89d2	fix bug in ingester series query (#2264)	5 years ago
Aditya C S	01040c1a4d	Fix data race in ingester (#2327)	5 years ago