apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879 ) * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	3 years ago
Travis Patterson	afedabf0e9	Remove unused code (#6872 )	3 years ago
Travis Patterson	983ab80e7c	Add delete api validations (#6860 ) * break out the middleware * Add validation to the API - Time must be in RFC3339 or a 10-digit-unix-seconds timestamp - Start time must always exist * lint * docs * improve tests * cleanup * cleanup * access runtime config via function on validation	3 years ago
Michel Hollands	7c680e4ce0	update dskit and fix (#6868 ) * update dskit and fix Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use a separate registry to avoid duplicate registration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix runtime config initialization Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update vendor/modules.txt Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Travis Patterson	bd3d80ad80	Add ability to limit the impact of pathological delete requests. (#6854 ) * make per-tenant override the mode rather than binary on/off * Rely on overrides to configure deletion - Remove 'deletion_mode' flag * add ability to delete requests at any time * don't load requests from disabled tenants * review comments * fix tests * loadDeleteRequestsToProcess cleanup * lint * docs * review feedback * lint	3 years ago
Sandeep Sukhani	11a34fb48c	avoid creating multiple tenant heads in tsdb during a race (#6866 ) It causes us to have same ref ids for different streams since the refid in each head starts with 1	3 years ago
Kaviraj Kanagaraj	7e3d2433ed	embedded-cache: Bring `fifocache` and `groupcache` into single tent. (#6821 ) * chore(groupcache): Rename groupcache into memcache{distributed: true} 1. Introduced new config called memorycache config. 2. It has flag `distributed`. By enabling it, we use groupcache Going forward. We will bring fifocache into memcache config(with distributed:false) This PR is just a POC. To show how we can simplify in-built memory cache config and bring it into single config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Memorycache -> EmbeddedCache * Bring fifocache into embeddedcache tent Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix edge case with enabling fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix missing flags for embedded cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Warning message for fifocache config Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix listenport issue Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * idk. review remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. update changelog and upgrade guide 2. s/Embeddedcache/EmbeddedCache/g 3. /groupcache/ handler -> /embedded-cache/ 4. some typos 5. Fix some cache prefixes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fixing corresponding tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Support extra timeouts on embedded-cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted single struct field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix(stats-collector): Let cache.New() wrap stats collector for distributed cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Deprecate max_size_items from fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks about the flag description Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Owen Diehl	001a627f67	fix context propagation for child spans in shard resolver (#6853 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	57cec8e496	[TSDB] Ingester Index Sampling (#6852 ) * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * query ingesters for index stats Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * compatibility during rollouts for idx-gw+ingester-querier post proto refactor Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto compatibility with old indexgateway pkg path Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * test compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	bab049ca6f	Only process deletes in batches (#6851 ) * Only process deletes in batches * better logging	3 years ago
Sandeep Sukhani	7c66a363f7	Delete requests processing improvements (#6837 ) * return early while checking whether a chunk is deleted if user does not have any delete requests * track oldest start and latest end time of delete requests for each user * add some tests * Update pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_manager.go Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Owen Diehl	f9d78e1748	[TSDB] Indexgateway Protobuf refactor (#6841 ) * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Simon	19c7315bcc	Improve error message when max error message is hit (#6834 ) before: `entry with timestamp 2022-08-02 12:32:37 +0000 UTC ignored, reason: 'entry too far behind' for stream: {foo="bar2"}` after: `entry with timestamp 2022-08-02 12:32:37 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2022-08-03T15:25:57Z' for stream: {foo="bar2"}`	3 years ago
Travis Patterson	3054a0ead5	Add groupcache timeouts (#6808 ) * Add groupcache timeouts * Made groupcache timeouts configurable	3 years ago
Kaviraj Kanagaraj	7b89f20378	fix(logql): Make `LabelSampleExtractor` work as expected when label doesn't exist in logline (#6766 ) * fix(logql): Make `StringLabelfilter` work as expected Consider Log lines ``` msg=hello subqueries=5 msg=hello subqueries=0 msg=hello ``` If you run following logql ``` {foo="bar"}\|logfmt\|subqueries!=0 ``` it returns ``` msg=hello subqueries=5 ``` But if you run following logql ``` {foo="bar"}\|logfmt\|subqueries!="0" # NOTE: "0" as string. ``` it returns ``` msg=hello subqueries=5 msg=hello ``` NOTE: it also returns log lines that doesn't contain `subqueries` label in the first place. This cause subtle error if used in metric queries using that `label` in unwrap. e.g: following logql on above log lines returns ``` sum_over_time({foo="bar"}\|logfmt\|subqueries!="0"\|unwrap subqueries[1m]) ``` Returns `pipeline error: 'SampleExtractionErr'`. Because, the lines without `subqueries` labels are not skipped during `ProcessLine` and during metric extraction, our extractor sets this `SampleExtractionerror`. Fixes: https://github.com/grafana/loki/issues/6713 * Add it on changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more tests and fix the edge case with equality to empty strings Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * handle this edge case during metric extraction. Not in label filter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove commented code Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Christian Simon	49a6c03c09	Treat version flag at any position (#6827 ) The behaviour was changed as part of #6802. This should restore the previous behaviour and allow the version to be displayed without config file.	3 years ago
Danny Kopping	66fdb3c572	Log stream flushes to see how many chunks we flush at a time (#6819 ) * Log stream flushes to see how many chunks we flush at a time Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Smuggling in an error wrapping which aides debugging Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Ed Welch <ed@oqqer.com> * Removing unnecessary label Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Ed Welch <ed@oqqer.com>	3 years ago
Owen Diehl	46ae0efc0f	Simplify shard membership calculations (#6810 )	3 years ago
Kaviraj Kanagaraj	9f06bc0083	chore(groupcache): Groupcache doesn't need global server handler anymore (#6820 ) Remove it from the constructor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Michel Hollands	03e4db46eb	Fix log messages for deletion requests (#6727 ) * Fix log messages for deletion requests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Store deleted lines in log message instead of as a metric Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Test for equal values Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Config for running with deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add back per user counter of deleted lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert example config changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change test name Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Travis Patterson	3bbc5a92f1	Allow fifocache to be configured with groupcache (#6806 )	3 years ago
Kaviraj Kanagaraj	acc3a2fe23	chore(groupcache): Fix panic with stats collector wrapper (#6798 ) * chore(groupcache): Fix panic with stats collector wrapper Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * update Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	e67dd6c91d	chore(groupcache): Support per cache group capacity (#6793 ) * chore(groupcache): Make per cache group capacity Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted handler got merged during rebase Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Some standard renaming Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	4ca7676272	Groupcache improvements (#6752 ) * Groupcache communicates with own server * Don't distribute the index cache * Fix tests * go mod fix	3 years ago
Kaviraj Kanagaraj	ef304780cb	chore(groupcache): disable hotcache for chunks (#6784 ) * chore(groupcache): disable hotcache for chunks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * GetCacheType() Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	d38c208aa7	Rephrase error message (indexgateway->groupcache). (#6763 )	3 years ago
Travis Patterson	cc9c19846f	Fix cache instrumentation (#6753 )	3 years ago
Kaviraj Kanagaraj	fb9a1989d8	chore(groupcache): Disable hotcache during set (#6740 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Owen Diehl	8ac859c5cd	[Enhancement] Removes memcached failure log (#6720 ) * removed memcached failure log Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * remove memcached fetch error log Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	26fc489d41	tsdb store refactoring (#6724 ) * do not make global tsdb store instance * upload index to storage during ingester shutdown	3 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704 ) * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	3 years ago
Owen Diehl	8daa694d12	[Fix] Furthers gcs latency buckets (#6721 ) * furthers gcs latency buckets Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * adjusts bucket upper bound Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	7f343d248a	Fix goroutine leak when using http2 (#6718 )	3 years ago
Danny Kopping	fa42c12f4c	Adding "start_delta" entry to metrics.go line (#6550 ) * Adding "start_delta" entry to metrics.go line This will help us determine if a 3h query (derived from "length") is for some random 3h period in the past, or within the last 3h Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter; for once it was actually useful Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding "end_delta" as well Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing rounding Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Ed Welch	d7345c1c4c	Loki: Do not store exact log line duplicates (now including unordered inserts) (#6642 ) * do not insert log lines which are an exact duplicate in both timestamp and log content Signed-off-by: Edward Welch <edward.welch@grafana.com> * update test to now ignore dupes pushed out of order Signed-off-by: Edward Welch <edward.welch@grafana.com> * make the test unordered to be more thorough Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	1a75bb8e22	Loki: Return an __error_details__ label for any line which incurs a __error__ while being processed (#6543 ) * adds an __error_details__ label which includes any details available about a logql error. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint, changed a test a little Signed-off-by: Edward Welch <edward.welch@grafana.com> * reset error details in Reset() function Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	934d56fe5e	Add function to reset boltDBIndexClientWithShipper singleton In a downstream project I have a test case that creates multiple stores with different BoltDB shipper schema configurations. Since the client for the store is a singleton and is re-used, there needs to be a function to reset it as well. This function must only be used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Danny Kopping	7446f2498f	Add configurable groupcache capacity (#6678 ) * Add configurable groupcache capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Improving the verbiage for clarity - thanks Travis! Changed the default size to 100MB Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding config docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removed test - it was pretty redundant, plus groupcache panics if you try init its http server more than once Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Travis Patterson	cd2ebeb279	Fix per-tenant delete client (#6696 ) - Don't make the request for a disabled tenant even if golbal deletion is allowed	3 years ago
Owen Diehl	d2c099ac1d	query_frontend_shard_factor uses exponential buckets (#6689 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	25bd0a4a55	Fix TIME_WAIT connection leak from queriers to compactor (#6681 ) * Fix TIME_WAIT connection leak from queriers to compactor * lint	3 years ago
Owen Diehl	580a5693b5	stripes frontend requests in progress (#6679 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Danny Kopping	f560dcc5b1	Set log level for GetChunkRefs log (#6677 ) Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	b3f1965ff1	Introduce an "enable_groupcache" config to control initialization better (#6673 ) * Introduce an "enable_groupcache" config to control initialization better We initialize groupcache as a module, which happens after the config is parsed. A function named applyFIFOCacheConfig is called to enable fifocache if no cache is configured, and at the time of its calling the modules have not been initialized. This new "enable_groupcache" value is set if groupcache is configured in common, or this value is set manually. With this setting, we can prevent automatic fifocache settings. Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Add tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Simplifying condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Sandeep Sukhani	5db578bfe8	fix the issue of overwriting tsdb files during rollouts (#6668 )	3 years ago
Owen Diehl	254461ce8b	[TSDB] store uncompressed chunk size in index (#6665 ) * store uncompressed chunk size in index Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * chunk data UncompressedSize for old metric compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	47b17b590c	dont skip writing to TSDB index when chunk exists in cache (#6664 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Dylan Guedes	2bb2a2c0e1	Loki: Remove ring client from the distributor (#6622 ) Removes the ring client from the distributor. - It was originally added on PR #4938 because it was the only ring implementation with HTTPHandler support (used for the ring page). Now that DSKit has support for the other ring implementations (including the lifecycler) this isn't necessary anymore. - It had an IgnoreUnhealthyInstancesReplicationStrategy attached to it, added by mistake	3 years ago
Travis Patterson	77fa734368	Fix inconsistent coverage in the distributor package (#6645 )	3 years ago
Travis Patterson	68141444f8	Introduce Groupcache (#6644 ) * implement groupcache * use mailgun groupcache * use memberlist * implement peer discovery with the ring * implement retrieval and storage * use a ticker * Correctly configuring chunk cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Enabling results cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing groupcache ring page Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * groupcache-poc: Add support for index cache. (#6535) * Hacked in prometheus metrics (#6544) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * add http to addresses * always make a group for each cache * fix peer discovery * Fix groupcache config - Allow it to be enabled/disabled - Move all config to 'common' config - Always write to hot cache * Add http/2 support for groupcache (#6596) * Add groupcache latency metrics (#6619) * Add groupcache latency metrics * lint * lint * lint * Add groupcache test (#6620) * Add groupcache test * lint * Improve 'Fetch' allocs 11 -> 8 (#6623) * Move constants to the file where they're used * remove irrelevant comments Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago

1 2 3 4 5 ...

1647 Commits (3170edf3e3cf9cc354c41cf01556eb6024016d4f)