apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Travis Patterson	e15da437f0	Shard deletes requests based on time ranges (#6903 ) * Refactor the store API to be in terms of DeleteRequest * backfill remove tests * backfill GetAll tests * Make the delete request store deal in groups of requests instead of single ones * Make cancelation aware of request groups * remove cancelation deadline * shard deletes based on a query parameter * merge request group on gets * global configuration for max query range * sort delete requests by start time so batches are grouped to similar chunks * Fix status when some deletes haven't happened yet * fix tests * Review feedback * review feedback * make max_interval > deletion interval a validation error * list partially completed deletes * list partially completed deletes * fix flaky test * clarify naming * disallow cancelation of partial deletes unless the 'force' flag is passed * more comment specificity for the delete request hash	4 years ago
Danny Kopping	449dd81981	Restoring metrics removed by #6957 , simplifying implementation (#6962 ) Adding all available metrics from Go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	2321da49b5	Upgrade prometheus/client_golang and reconfigure to restore `go_sched.` metrics (#6957 ) Upgrading prometheus/client_golang to restore go_sched.* metrics These metrics were removed unintentionally in #6403 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test to match interface correctly Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	87e20d9399	Setting sane defaults for ruler in SSD "read" mode (#6551 ) Also removing support for configdb, since this is not supported but was the default storage type Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	c340f3e202	[TSDB] disables bloom filters for tsdb index on ingesters (#6939 ) * disables bloom filters for tsdb index on ingesters Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * faillint exception for proto sync/atomic Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Susana Ferreira	3f3f4eda1d	Fix topk and bottomk operations with int <= 0 (#6937 ) * Fix topk and bottomk operations with int <= 0 * Fix VectorAggregationExpr String method * Update rangemapper_test expression * Add tests * Update CHANGELOG.md	4 years ago
Dylan Guedes	2d7ef07494	Fix `increment_duplicate_timestamps` for multiple dupped timestamps (#6924 )	4 years ago
Owen Diehl	72d8627a55	log query which causes panic in logql.clone (#6934 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Michel Hollands	25e8365d05	Improve error message (#6913 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	f470943433	reintroduces now deprecated+noop deletion configs for compatibility (#6917 ) * reintroduces now deprecated+noop deletion configs for compatibility Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * better deprecation logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes import Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Christian Simon	953c36e3ed	Remove unnecessary spin up of mock (#6911 ) This removes the unnecessary spinning up of a mock for TestDoubleRegistration. The client itself is not used at all and that causes it to be stopped before it is actually started. Example flake: https://drone.grafana.net/grafana/loki/14724/2/5	4 years ago
Christian Simon	ee2fe6514d	Propagate error detail properly in all cases (#6890 ) The error handling was missing appending the error detail properly in all cases. Fixes #6265	4 years ago
Kaviraj Kanagaraj	693f9d7070	fix(stats_collector): Do not collect stats for failed requests. (#6878 ) * fix(stats_collector): Do not collect stats for failed requests. When any type of query fails, do not collect statistics. Rationale being when core http handler returns error, it's reponse will be nil and there won't be any statistics to collect and log. Fixes: 6231 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Sandeep Sukhani	3c5fdb749a	add metric for measuring overall index query latency (#6880 )	4 years ago
Owen Diehl	3cba86faf2	[bugfix] fixes issues sharding series by storageref vs fingerprint in stripeSeries (#6882 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879 ) * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	4 years ago
Travis Patterson	afedabf0e9	Remove unused code (#6872 )	4 years ago
Travis Patterson	983ab80e7c	Add delete api validations (#6860 ) * break out the middleware * Add validation to the API - Time must be in RFC3339 or a 10-digit-unix-seconds timestamp - Start time must always exist * lint * docs * improve tests * cleanup * cleanup * access runtime config via function on validation	4 years ago
Michel Hollands	7c680e4ce0	update dskit and fix (#6868 ) * update dskit and fix Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use a separate registry to avoid duplicate registration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix runtime config initialization Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update vendor/modules.txt Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Travis Patterson	bd3d80ad80	Add ability to limit the impact of pathological delete requests. (#6854 ) * make per-tenant override the mode rather than binary on/off * Rely on overrides to configure deletion - Remove 'deletion_mode' flag * add ability to delete requests at any time * don't load requests from disabled tenants * review comments * fix tests * loadDeleteRequestsToProcess cleanup * lint * docs * review feedback * lint	4 years ago
Sandeep Sukhani	11a34fb48c	avoid creating multiple tenant heads in tsdb during a race (#6866 ) It causes us to have same ref ids for different streams since the refid in each head starts with 1	4 years ago
Kaviraj Kanagaraj	7e3d2433ed	embedded-cache: Bring `fifocache` and `groupcache` into single tent. (#6821 ) * chore(groupcache): Rename groupcache into memcache{distributed: true} 1. Introduced new config called memorycache config. 2. It has flag `distributed`. By enabling it, we use groupcache Going forward. We will bring fifocache into memcache config(with distributed:false) This PR is just a POC. To show how we can simplify in-built memory cache config and bring it into single config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Memorycache -> EmbeddedCache * Bring fifocache into embeddedcache tent Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix edge case with enabling fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix missing flags for embedded cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Warning message for fifocache config Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix listenport issue Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * idk. review remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. update changelog and upgrade guide 2. s/Embeddedcache/EmbeddedCache/g 3. /groupcache/ handler -> /embedded-cache/ 4. some typos 5. Fix some cache prefixes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fixing corresponding tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Support extra timeouts on embedded-cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted single struct field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix(stats-collector): Let cache.New() wrap stats collector for distributed cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Deprecate max_size_items from fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks about the flag description Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Owen Diehl	001a627f67	fix context propagation for child spans in shard resolver (#6853 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	57cec8e496	[TSDB] Ingester Index Sampling (#6852 ) * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * query ingesters for index stats Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * compatibility during rollouts for idx-gw+ingester-querier post proto refactor Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto compatibility with old indexgateway pkg path Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * test compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	bab049ca6f	Only process deletes in batches (#6851 ) * Only process deletes in batches * better logging	4 years ago
Sandeep Sukhani	7c66a363f7	Delete requests processing improvements (#6837 ) * return early while checking whether a chunk is deleted if user does not have any delete requests * track oldest start and latest end time of delete requests for each user * add some tests * Update pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_manager.go Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	4 years ago
Owen Diehl	f9d78e1748	[TSDB] Indexgateway Protobuf refactor (#6841 ) * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Christian Simon	19c7315bcc	Improve error message when max error message is hit (#6834 ) before: `entry with timestamp 2022-08-02 12:32:37 +0000 UTC ignored, reason: 'entry too far behind' for stream: {foo="bar2"}` after: `entry with timestamp 2022-08-02 12:32:37 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2022-08-03T15:25:57Z' for stream: {foo="bar2"}`	4 years ago
Travis Patterson	3054a0ead5	Add groupcache timeouts (#6808 ) * Add groupcache timeouts * Made groupcache timeouts configurable	4 years ago
Kaviraj Kanagaraj	7b89f20378	fix(logql): Make `LabelSampleExtractor` work as expected when label doesn't exist in logline (#6766 ) * fix(logql): Make `StringLabelfilter` work as expected Consider Log lines ``` msg=hello subqueries=5 msg=hello subqueries=0 msg=hello ``` If you run following logql ``` {foo="bar"}\|logfmt\|subqueries!=0 ``` it returns ``` msg=hello subqueries=5 ``` But if you run following logql ``` {foo="bar"}\|logfmt\|subqueries!="0" # NOTE: "0" as string. ``` it returns ``` msg=hello subqueries=5 msg=hello ``` NOTE: it also returns log lines that doesn't contain `subqueries` label in the first place. This cause subtle error if used in metric queries using that `label` in unwrap. e.g: following logql on above log lines returns ``` sum_over_time({foo="bar"}\|logfmt\|subqueries!="0"\|unwrap subqueries[1m]) ``` Returns `pipeline error: 'SampleExtractionErr'`. Because, the lines without `subqueries` labels are not skipped during `ProcessLine` and during metric extraction, our extractor sets this `SampleExtractionerror`. Fixes: https://github.com/grafana/loki/issues/6713 * Add it on changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more tests and fix the edge case with equality to empty strings Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * handle this edge case during metric extraction. Not in label filter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove commented code Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Christian Simon	49a6c03c09	Treat version flag at any position (#6827 ) The behaviour was changed as part of #6802. This should restore the previous behaviour and allow the version to be displayed without config file.	4 years ago
Danny Kopping	66fdb3c572	Log stream flushes to see how many chunks we flush at a time (#6819 ) * Log stream flushes to see how many chunks we flush at a time Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Smuggling in an error wrapping which aides debugging Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Ed Welch <ed@oqqer.com> * Removing unnecessary label Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Ed Welch <ed@oqqer.com>	4 years ago
Owen Diehl	46ae0efc0f	Simplify shard membership calculations (#6810 )	4 years ago
Kaviraj Kanagaraj	9f06bc0083	chore(groupcache): Groupcache doesn't need global server handler anymore (#6820 ) Remove it from the constructor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Michel Hollands	03e4db46eb	Fix log messages for deletion requests (#6727 ) * Fix log messages for deletion requests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Store deleted lines in log message instead of as a metric Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Test for equal values Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Config for running with deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add back per user counter of deleted lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert example config changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change test name Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Travis Patterson	3bbc5a92f1	Allow fifocache to be configured with groupcache (#6806 )	4 years ago
Kaviraj Kanagaraj	acc3a2fe23	chore(groupcache): Fix panic with stats collector wrapper (#6798 ) * chore(groupcache): Fix panic with stats collector wrapper Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * update Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Kaviraj Kanagaraj	e67dd6c91d	chore(groupcache): Support per cache group capacity (#6793 ) * chore(groupcache): Make per cache group capacity Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted handler got merged during rebase Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Some standard renaming Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Travis Patterson	4ca7676272	Groupcache improvements (#6752 ) * Groupcache communicates with own server * Don't distribute the index cache * Fix tests * go mod fix	4 years ago
Kaviraj Kanagaraj	ef304780cb	chore(groupcache): disable hotcache for chunks (#6784 ) * chore(groupcache): disable hotcache for chunks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * GetCacheType() Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Dylan Guedes	d38c208aa7	Rephrase error message (indexgateway->groupcache). (#6763 )	4 years ago
Travis Patterson	cc9c19846f	Fix cache instrumentation (#6753 )	4 years ago
Kaviraj Kanagaraj	fb9a1989d8	chore(groupcache): Disable hotcache during set (#6740 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Owen Diehl	8ac859c5cd	[Enhancement] Removes memcached failure log (#6720 ) * removed memcached failure log Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * remove memcached fetch error log Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	26fc489d41	tsdb store refactoring (#6724 ) * do not make global tsdb store instance * upload index to storage during ingester shutdown	4 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704 ) * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	4 years ago
Owen Diehl	8daa694d12	[Fix] Furthers gcs latency buckets (#6721 ) * furthers gcs latency buckets Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * adjusts bucket upper bound Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	7f343d248a	Fix goroutine leak when using http2 (#6718 )	4 years ago
Danny Kopping	fa42c12f4c	Adding "start_delta" entry to metrics.go line (#6550 ) * Adding "start_delta" entry to metrics.go line This will help us determine if a 3h query (derived from "length") is for some random 3h period in the past, or within the last 3h Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter; for once it was actually useful Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding "end_delta" as well Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing rounding Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Ed Welch	d7345c1c4c	Loki: Do not store exact log line duplicates (now including unordered inserts) (#6642 ) * do not insert log lines which are an exact duplicate in both timestamp and log content Signed-off-by: Edward Welch <edward.welch@grafana.com> * update test to now ignore dupes pushed out of order Signed-off-by: Edward Welch <edward.welch@grafana.com> * make the test unordered to be more thorough Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago

1 2 3 4 5 ...

1662 Commits (e15da437f09ecd698ecd9a303546ff4cfa3adbe2)