13 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)

Author	SHA1	Message	Date
Ed Welch	b9301045cb	introduce randomness to the first call for the metric test query to reduce the likelyhood that large numbers of canaries will all make this request at the same time. (#6562) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
afayngelerindbx	e49c360c36	canary: Adds locking to prevent multiple concurrent invocations of `confirmMissing` from clobbering each other (#5568)	4 years ago
Callum Styan	3ee8ed3b1d	Allow for loki-canary to generate a percentage of out of order log lines (#4126) ... * Allow for loki-canary to generate a percentage of log lines with out of order timestamps, percentage and out of order time range is all configurable. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Switch OoO min/max flags to time duration and properly calculate OoO percentage check. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change some variable names + add comments to make it more obvious what various parts of the canary code are actually doing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update pkg/canary/writer/writer.go * Add notes about usage of the out-of-order-percentage flag and the requirement to use pipeline stages. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	204e61afe3	Loki-Canary: Backoff retries on query failures, add histograms for query performance. (#2413) ... * Canary needs to wait before spot-checking entries some reasonable amount of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present. Adding two histograms for tracking the performance of the spot check and metric test queries * Adding backoff to queries if they fail specifically waiting 5mins when a 429 is received from Loki * fix order of parameters	6 years ago
Ed Welch	76ceec45d9	If an entry is missing on the websocket, start checking after `wait` but continue checking until `max-wait` (#2369) ... Refactored out a useful pruneList function	6 years ago
Ed Welch	2b145da7d7	Canary tweaks (#2350) ... * tweak the canary a bit, add a metric for websocket pings, fix some logging, improve the mutex locking a bit, add a read timeout on the websocket, add a lastmessage time checker to restart websocket * fix constructor * fix counter name change metric count reporting to two separate metrics, expected vs actual * add note to upgrade guide for metric name changes * fixed tests.... * Update docs/operations/upgrade.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	6 years ago
Ed Welch	81b9decfb5	Loki-Canary: Add query spot checking and metric count checking (#2344) ... * Add spot check capability to canary to query certain values over time to make sure they are still in Loki. Add a context timeout to Loki queries * Updating canary to use /loki/api/v1 endpoints adding a metric test query * fixing metric names to have _total for counters * fix parsing of timestamp from log line and don't use entry timestamp * Tweak the metric test query to adjust the range to the time the canary has been running to get accurate results until the requested range has been reached. Add a total spot check entry counter to get a ratio of success * Add some debugging logs to troubleshoot missing entries * add more tests, fix some log lines * update docs, make the spot-check timer separate from prune entries * just how many paramaters can I add to the constructor.... * run prune entries on a separate thread, only run one at a time.	6 years ago
Joe Elliott	383eb6e70b	Addition of a `/suspend` endpoint to Loki Canary (#1891) ... * First pass loki canary pause Signed-off-by: Joe Elliott <number101010@gmail.com> * Write suspending msg to stderr Signed-off-by: Joe Elliott <number101010@gmail.com> * Added resume endpoint Signed-off-by: Joe Elliott <number101010@gmail.com> * Added documentation on new endpoints Signed-off-by: Joe Elliott <number101010@gmail.com> * Pass both signals to the same channel Signed-off-by: Joe Elliott <number101010@gmail.com> * lint Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed return on startCanary Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Ed Welch	8a95972f73	Add some additional logging to the canary on queries (#1137) ... * log what was missing from the websocket connection and what we received from a query to Loki to troubleshoot missing logs * removing unnecessary unused return variables from print statements * update test to handle the new debug output	6 years ago
Ed	173e8ea2a4	fix(canary): Fix Flaky Tests (#889) ... * make the missing entry lookup synchronous during tests to hopefully avoid races around the entry order and test verifications * playing around with timings some more to get all tests to pass even on a really slow machine	7 years ago
Marco Pracucci	b9185324f4	Improved Tailer loop (dropped entries, logic refactoring, introduced tests)	7 years ago
Edward Welch	e6f431ad46	finishing up loki-canary move into loki repo	7 years ago
Edward Welch	cd5368a44d	preparing for move into loki repo	7 years ago