apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	4a358bc9ea	revendor cortex to latest master (#2463 )	5 years ago
Sandeep Sukhani	d5c840bd66	enforce requirment for periodic config for index tables to be 24h when using boltdb shipper (#2166 )	5 years ago
Owen Diehl	4a45ca5e3a	Revendor cortex (#2427 ) * vendoring bump * vendoring compat	5 years ago
Sandeep Sukhani	e59adccfa0	create smaller unique files from boltdb shipper and other code refactorings (#2261 ) * rename package local to shipper * reorganized files to separate out uploads and downloads * create smaller unique files from boltdb shipper and other improvements * cancellation of downloading of initial files when service is being stopped * migration of existing files to dedicated folders and cleanup of index from ingesters * some more improvements to uploads and downloads code * tests for uploads and downloads package in shipper Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * test cleanup fix and some other minor changes * refactor getting uploader from object key * better naming * changes suggested from PR review * Fixes goroutine loop. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Simplify getoradd table. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes loop in upload manager. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * some more tests, logs and changes suggested from PR review * move ready state check for downloaded table inside query function * add info log for successfully finishing uploading of a table * fix merge conflict * fix ingester name and mode config Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	0885a49370	configuration doc improvements (#2318 )	5 years ago
Cyril Tovena	0be64fcb34	Improve metric queries by computing samples at the edges. (#2293 ) * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	aee064d0a8	Loki: Series API will return all series with no match or empty matcher (#2254 ) * Allow series queries with no match or an empty {} matcher which will return all series for the provided timeframe. * fix test * fix lint	5 years ago
Cyril Tovena	6ab832b2ca	Cache overlapping blocks (#2239 ) * Caches block iteration that are overlapping together to avoid reprocessing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Final touch. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927 ) * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit `55d41b9519`. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago
Sandeep Sukhani	3a28a17946	some metrics for measuring performance and failures in boltdb shipper (#2034 ) * some metrics for measuring performance and failures in boltdb shipper * counting both failures and success in upload and download operation in boltdb shipper * changes suggested from PR review	5 years ago
Aditya C S	199e334126	Update loki to cortex master (#2030 )	5 years ago
Sandeep Sukhani	989fb106a0	table client for boltdb shipper to enforce retention (#2001 ) * table client for boltdb shipper to enforce retention	5 years ago
Sandeep Sukhani	21d4ca41e2	make boltdb shipper singleton and some other minor refactoring (#1995 ) * make boltdb shipper singleton and some other minor refactoring	5 years ago
Ed Welch	b89a1f81dd	Update cortex vendoring to latest master (#1975 ) * Update cortex vendoring to latest master checksum `36496a074bc4530fb1d7c74136cab1566fbd9583` Signed-off-by: Ed Welch <edward.welch@grafana.com> * handle all the breaking changes from cortes Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * rebasing on master, re-runing go mod vendor seemed to bring in some more changes... Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	fad3b61be7	flush boltdb to object store (#1837 ) * flush boltdb to object store files are stored in folder per periodic table and are named after ingester flushed every 15 mins to make index available to other services files are also flushed before ingester stops to avoid any data loss new stores can be implemented easily ingester to also query store when using boltdb * persisting uploader name across restarts, detecting objectstore type from periodic config, other refactorings * updated cli flag for active directory * add tests for boltdb shipper and update vendor * syncing boltdb files to disk during update * sleep in tests to let mtime of boltdb file be changed * add documentation for boltdb shipper and remove unwanted config	5 years ago
Owen Diehl	3347871154	only fetches one chunk per series in /series (#1914 ) * only fetches one chunk per series in /series * Update pkg/storage/store.go * chunksBySeries naming * sorts series response, adds tests for chunks < batchsize	5 years ago
Cyril Tovena	f1b8d4d8ad	Improve logql query statistics collection. (#1573 ) * Improve logql query statistics collection. This also add information about ingester queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad copy/past in the result log. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ingester tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve headchunk efficiency. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad commit on master. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes new interface of ingester grpc server. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentations of fields. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Joe Elliott	d3972852ee	Minor typo fixes (#1545 ) Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088 ) * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948 ) * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
Cyril Tovena	7f3db9d7f0	LogQL: Vector and Range Vector Aggregation. - adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector	6 years ago
Cyril Tovena	399c990ae6	comments adjustment	6 years ago
Cyril Tovena	4d48972939	Query storage by iterating through chunks by batches. Also adds more tests for the store	6 years ago
Cyril Tovena	4bf77662d3	rounds nanoseconds boundaries to milliseconds (#771 ) * rounds nanoseconds boundaries to milliseconds * convert also store query	6 years ago
Cyril Tovena	3346ce1b40	Storage memory improvement (#713 ) * add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator	6 years ago
Cyril Tovena	922f1daf5d	Remove label __name__ from store querier (#648 )	6 years ago
Sandeep Sukhani	3d3c05717a	Update cortex vendor (#610 ) * Update cortex vendor Use query max look back from cortex Update config changes from cortex Fixed breaking code due to updating cortex vendor * fixed linter error	6 years ago
Cyril Tovena	2ad98007ff	fixes chunks lazy loading (#595 )	6 years ago
Tom Wilkie	80bbd04778	Add an extra filter step if the user specifies a regex using old API. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	e73024d3a4	Remove RegexpIter, we don't need it. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	bfb34e4070	Wire up LogQL filters in the ingester. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	5e3630815b	Errors and wiring. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	b16da1d7c3	Parse pipelines of regex matches. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Cyril Tovena	8d7efddde2	switch to golangci-lint linter (#539 ) * switch to golangci-lint linter * fixes for review comments	6 years ago
Goutham Veeramachaneni	27ca58e79f	Lazy load chunks (#435 ) * Lazy load chunks for queries. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Add context to the lazy chunk calls Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Make sure labels are plumbed through Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Pre-load non-overlapping chunk iterators Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	6 years ago
Tom Wilkie	d6e8788eae	Don't forget to apply regexp filters to chunk entries. (#124 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	9f257c591a	[WIP] Persistence (#44 ) Persistence chunks using the Cortex chunk store.	7 years ago

1 2

75 Commits (c4261b19fd8b154d8562e12a8dff4bb9de3ed085)