986 Commits (7ed63ea7f493dab6c81200ddb7a0e6f26dc41bec)

Author	SHA1	Message	Date
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031) ... * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Josh Soref	0d9cf4946a	Spelling (#2014) ... * spelling: access * spelling: actual * spelling: asynchronously * spelling: circleci * spelling: circle * spelling: cognitive * spelling: comment * spelling: computation * spelling: correct * spelling: daemon * spelling: elliott * spelling: enabled * spelling: encbuf * spelling: extracted * spelling: explicitly * spelling: independent * spelling: languages * spelling: loose * spelling: message * spelling: metrics * spelling: mismatch * spelling: nonexistent * spelling: output * spelling: pipeline * spelling: pipe line * spelling: position * spelling: prepend * spelling: query * spelling: receive * spelling: received * spelling: reducible * spelling: registry * spelling: response * spelling: retryable * spelling: scanning * spelling: sourced * spelling: supported * spelling: template * spelling: timeout * spelling: unsupported * spelling: adapter * spelling: deferring * spelling: expirable	5 years ago
Sandeep Sukhani	989fb106a0	table client for boltdb shipper to enforce retention (#2001) ... * table client for boltdb shipper to enforce retention	5 years ago
Cyril Tovena	577d8eb168	Ensure trace propagation in our logs. (#1977) ... * Ensure trace propagation in our logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing import. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix function usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	21d4ca41e2	make boltdb shipper singleton and some other minor refactoring (#1995) ... * make boltdb shipper singleton and some other minor refactoring	5 years ago
Aditya C S	c0a28cbf30	Log error message for invalid checksum (#1713)	5 years ago
Ed Welch	b89a1f81dd	Update cortex vendoring to latest master (#1975) ... * Update cortex vendoring to latest master checksum `36496a074bc4530fb1d7c74136cab1566fbd9583` Signed-off-by: Ed Welch <edward.welch@grafana.com> * handle all the breaking changes from cortes Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * rebasing on master, re-runing go mod vendor seemed to bring in some more changes... Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	fad3b61be7	flush boltdb to object store (#1837) ... * flush boltdb to object store files are stored in folder per periodic table and are named after ingester flushed every 15 mins to make index available to other services files are also flushed before ingester stops to avoid any data loss new stores can be implemented easily ingester to also query store when using boltdb * persisting uploader name across restarts, detecting objectstore type from periodic config, other refactorings * updated cli flag for active directory * add tests for boltdb shipper and update vendor * syncing boltdb files to disk during update * sleep in tests to let mtime of boltdb file be changed * add documentation for boltdb shipper and remove unwanted config	5 years ago
Ed Welch	b5b3e78424	Loki Update the cut block size counter when creating a memchunk from byte slice (#1946) ... * Update the cut block size counter when creating a memchunk from byte slice * Refactor how we create memchunks to require specifying the block size and target size	5 years ago
Peter Štibraný	6d5408c100	Update Cortex to latest master (#1869) ... * Update Cortex to master Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix compilation errors after Cortex update. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update to Cortex v1.0.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added missing vendored file. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Vendor Cortex 1.0.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Configuration changes. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Mention Cortex 1.0 changes and link to docs. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix jsonnet files, and mention fields that were removed. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	3347871154	only fetches one chunk per series in /series (#1914) ... * only fetches one chunk per series in /series * Update pkg/storage/store.go * chunksBySeries naming * sorts series response, adds tests for chunks < batchsize	5 years ago
Cyril Tovena	d34b0a9d30	Regexp simplification (#1787) ... * Refactor line filter to support custom parsing for perl regexp. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support alternate, concat regex simplification Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add benchmark between normal and simplified regex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working through all possible concat operation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes concat operations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds more type of capture group Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve test robustness Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support for nexted concat. git push Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bug with anychar. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add more not supported cases. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor the concat alternates function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentation and refactor code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all code using line filter. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests to check nil filter. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure benchmark is not optimized by the compilation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785) ... * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Aditya C S	f9ff2a4b02	Add goimport local-prefixes configuration to .golangci.yml (#1710)	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615) ... * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	f1b8d4d8ad	Improve logql query statistics collection. (#1573) ... * Improve logql query statistics collection. This also add information about ingester queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad copy/past in the result log. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ingester tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve headchunk efficiency. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad commit on master. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes new interface of ingester grpc server. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentations of fields. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484) ... * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Joe Elliott	d3972852ee	Minor typo fixes (#1545) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
Ed Welch	35dd07c4e0	fixes a panic where prefetch could try to operate on a nil object (#1499) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	33f70b70ce	Add duplicates info and remove timing informations. (#1496) ... * Add duplicates info and remove timing informations. Timing information in the chunkenc iterator are too eating too much resources. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	7c4906786d	pre-fetch the next chunk in parallel to processing the current chunk such that we aren't waiting for fetches from GCS and spend all our CPU time decompressing and processing (#1495) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	fb76416b27	Add decompression tracing instrumentation. (#1445) ... * Add decompression tracing instrumentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix time conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use span logger so we also get a log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve reusability and add fetched chunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	768bf914e0	Changes the encoding used for the benchmark in the store. (#1448) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Ed Welch	5aa82ba199	allow configuring a target chunk size in compressed bytes (#1406) ... * allow configuring a target chunk size in compressed bytes Signed-off-by: Edward Welch <edward.welch@grafana.com> * addressing feedback Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Cyril Tovena	1fc0ffde8a	Logql benchmark and performance improvement. (#1371) ... * Add logql benchmark test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes store benchmark limits. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add LogQL improvements. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add back logging in the storage benchmark Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	27f17bc1de	Fixes linter warning from the yacc file. (#1328) ... * Fixes linter warning from the yacc file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves sed to work on POSIX Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Putra Sattvika, I Gusti Ngurah	c89df1a795	storage: fix missing logs with batched chunk iterator (#1299) ... * fix batched chunk iterator * fix expected series labels in iterator_test.go * add query range boundary check * add __name__ label matcher to test cases * trim __name__ without mutating chunks	6 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088) ... * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948) ... * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
Cyril Tovena	7f3db9d7f0	LogQL: Vector and Range Vector Aggregation. ... - adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector	6 years ago
Cyril Tovena	2b40392084	reuse overlapping array in batch iterator	6 years ago
Goutham Veeramachaneni	ee1091211a	Add comments for tricky code ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	6 years ago
Cyril Tovena	399c990ae6	comments adjustment	6 years ago
Cyril Tovena	4d48972939	Query storage by iterating through chunks by batches. ... Also adds more tests for the store	6 years ago
Cyril Tovena	4bf77662d3	rounds nanoseconds boundaries to milliseconds (#771) ... * rounds nanoseconds boundaries to milliseconds * convert also store query	6 years ago
Cyril Tovena	3346ce1b40	Storage memory improvement (#713) ... * add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator	6 years ago