1622 Commits (ab2d50a2b2a1ea75d013bc1f35f42e10cac1dd46)

Author	SHA1	Message	Date
Dylan Guedes	d38c208aa7	Rephrase error message (indexgateway->groupcache). (#6763)	4 years ago
Travis Patterson	cc9c19846f	Fix cache instrumentation (#6753)	4 years ago
Kaviraj Kanagaraj	fb9a1989d8	chore(groupcache): Disable hotcache during set (#6740) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Owen Diehl	8ac859c5cd	[Enhancement] Removes memcached failure log (#6720) ... * removed memcached failure log Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * remove memcached fetch error log Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	26fc489d41	tsdb store refactoring (#6724) ... * do not make global tsdb store instance * upload index to storage during ingester shutdown	4 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704) ... * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	4 years ago
Owen Diehl	8daa694d12	[Fix] Furthers gcs latency buckets (#6721) ... * furthers gcs latency buckets Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * adjusts bucket upper bound Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	7f343d248a	Fix goroutine leak when using http2 (#6718)	4 years ago
Danny Kopping	fa42c12f4c	Adding "start_delta" entry to metrics.go line (#6550) ... * Adding "start_delta" entry to metrics.go line This will help us determine if a 3h query (derived from "length") is for some random 3h period in the past, or within the last 3h Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter; for once it was actually useful Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding "end_delta" as well Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing rounding Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Ed Welch	d7345c1c4c	Loki: Do not store exact log line duplicates (now including unordered inserts) (#6642) ... * do not insert log lines which are an exact duplicate in both timestamp and log content Signed-off-by: Edward Welch <edward.welch@grafana.com> * update test to now ignore dupes pushed out of order Signed-off-by: Edward Welch <edward.welch@grafana.com> * make the test unordered to be more thorough Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	1a75bb8e22	Loki: Return an __error_details__ label for any line which incurs a __error__ while being processed (#6543) ... * adds an __error_details__ label which includes any details available about a logql error. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint, changed a test a little Signed-off-by: Edward Welch <edward.welch@grafana.com> * reset error details in Reset() function Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Christian Haudum	934d56fe5e	Add function to reset boltDBIndexClientWithShipper singleton ... In a downstream project I have a test case that creates multiple stores with different BoltDB shipper schema configurations. Since the client for the store is a singleton and is re-used, there needs to be a function to reset it as well. This function must only be used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Danny Kopping	7446f2498f	Add configurable groupcache capacity (#6678) ... * Add configurable groupcache capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Improving the verbiage for clarity - thanks Travis! Changed the default size to 100MB Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding config docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removed test - it was pretty redundant, plus groupcache panics if you try init its http server more than once Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Travis Patterson	cd2ebeb279	Fix per-tenant delete client (#6696) ... - Don't make the request for a disabled tenant even if golbal deletion is allowed	4 years ago
Owen Diehl	d2c099ac1d	query_frontend_shard_factor uses exponential buckets (#6689) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	25bd0a4a55	Fix TIME_WAIT connection leak from queriers to compactor (#6681) ... * Fix TIME_WAIT connection leak from queriers to compactor * lint	4 years ago
Owen Diehl	580a5693b5	stripes frontend requests in progress (#6679) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Danny Kopping	f560dcc5b1	Set log level for GetChunkRefs log (#6677) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	b3f1965ff1	Introduce an "enable_groupcache" config to control initialization better (#6673) ... * Introduce an "enable_groupcache" config to control initialization better We initialize groupcache as a module, which happens after the config is parsed. A function named applyFIFOCacheConfig is called to enable fifocache if no cache is configured, and at the time of its calling the modules have not been initialized. This new "enable_groupcache" value is set if groupcache is configured in common, or this value is set manually. With this setting, we can prevent automatic fifocache settings. Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Add tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Simplifying condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Sandeep Sukhani	5db578bfe8	fix the issue of overwriting tsdb files during rollouts (#6668)	4 years ago
Owen Diehl	254461ce8b	[TSDB] store uncompressed chunk size in index (#6665) ... * store uncompressed chunk size in index Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * chunk data UncompressedSize for old metric compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	47b17b590c	dont skip writing to TSDB index when chunk exists in cache (#6664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Dylan Guedes	2bb2a2c0e1	Loki: Remove ring client from the distributor (#6622) ... Removes the ring client from the distributor. - It was originally added on PR #4938 because it was the only ring implementation with HTTPHandler support (used for the ring page). Now that DSKit has support for the other ring implementations (including the lifecycler) this isn't necessary anymore. - It had an IgnoreUnhealthyInstancesReplicationStrategy attached to it, added by mistake	4 years ago
Travis Patterson	77fa734368	Fix inconsistent coverage in the distributor package (#6645)	4 years ago
Travis Patterson	68141444f8	Introduce Groupcache (#6644) ... * implement groupcache * use mailgun groupcache * use memberlist * implement peer discovery with the ring * implement retrieval and storage * use a ticker * Correctly configuring chunk cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Enabling results cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing groupcache ring page Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * groupcache-poc: Add support for index cache. (#6535) * Hacked in prometheus metrics (#6544) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * add http to addresses * always make a group for each cache * fix peer discovery * Fix groupcache config - Allow it to be enabled/disabled - Move all config to 'common' config - Always write to hot cache * Add http/2 support for groupcache (#6596) * Add groupcache latency metrics (#6619) * Add groupcache latency metrics * lint * lint * lint * Add groupcache test (#6620) * Add groupcache test * lint * Improve 'Fetch' allocs 11 -> 8 (#6623) * Move constants to the file where they're used * remove irrelevant comments Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	4 years ago
Marco Pracucci	e7fb10f456	Loki: Update `grafana/dskit` and `weaveworks/common` (#6643) ... * Upgrade dskit and weaveworks/common Signed-off-by: Marco Pracucci <marco@pracucci.com> * 'go mod vendor' again Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Owen Diehl	094f8f3b20	[TSDB] fingerprint offsets is now correctly 16 byte aligned (#6630) ... * simpler shard factor calculations, disallows identity factor of 1 Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * log interval span as soon as it finishes Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * another shard bounds test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexclient span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more index span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * correctly match single shard Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings maxOffset is properly exclusive Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * minor posting idiomatic-ness Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * store 16 byte aligned offsets for fingerprint table Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	f18edcf9d8	use the configured index prefixes while building table names for tsdb index (#6436) ... * use the configured index prefixes while building table names for tsdb index * do not fail compaction when schema for a table can't be found	4 years ago
Sandeep Sukhani	619c2212f5	build and hook tsdb compaction code to the generic index compactor (#6627)	4 years ago
Sandeep Sukhani	9590e5ca4e	Tsdb tenant index querying (#6598) ... * update per tenant index identifier and relevant changes * add a flag to ForEach index callback to tell whether it is multitenant index or not	4 years ago
Travis Patterson	7f24b0f6ab	Remove ourdated comment (#6597)	4 years ago
Travis Patterson	a92e048ec3	avoid making delete requests if deletes are disabled for a user (#6583)	4 years ago
Chris Hodges	c582c5a727	loki-canary: Fix resource leak when using client-side certs (#6579) ... * loki-canary: Fix resource leak when using client-side certs * swap panic for os.Exit when failing to create httpClient	4 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537) ... * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	4 years ago
Owen Diehl	2e348218e2	log addtl info about bytes/shard in planning (#6573) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	b9301045cb	introduce randomness to the first call for the metric test query to reduce the likelyhood that large numbers of canaries will all make this request at the same time. (#6562) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Dylan Guedes	389eee4cc7	Stop compaction interval timer to avoid leak. (#6570)	4 years ago
Michel Hollands	527b658cc1	logcli: add flag to read remote schema configs (#6539) ... * Add flag to read remote schema Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix lint imports Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Trigger CI * Trigger CI * Addresss review comment Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Christian Haudum	a1745c7b47	Do not fail queries if compactor returns unexpected status code (#6552) ... To avoid showing deleted data, the querier originally failed when deletes were unavailable. If the compactor becomes unavailable this can cause Loki to stop responding to queries altogether. This PR changes the querier to log an error but continue servicing the request when it fails to get deletes. Extends PR #6368 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Ed Welch	33761194d9	Loki: Add the ability to use the log line timestamp in label and line formatter (#6517) ... * Change Stage interface Process method to include the timestamp of the line Signed-off-by: Edward Welch <edward.welch@grafana.com> * allow using log line timestamp in formatter Signed-off-by: Edward Welch <edward.welch@grafana.com> * update formatting in test Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Michel Hollands	695fa9d7a8	Add client type as label (#6469) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Susana Ferreira	79b2d01970	Add label_replace and literal expressions to instant query split by range (#6515) ... * Add label_replace logic to instant query split by range * Add label_replace and literal expressions support * Support binary operations with literal expressions * Remove Literal expression as splittable * Fix lint * Address comments	4 years ago
Owen Diehl	8eed9a4dde	dont call getinstance twice in checkpointing (#6516) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Karsten Jeschkies	d9a522e5e6	Disable index gateway client on read target. (#6490)	4 years ago
Dylan Guedes	bcb57f100c	During query readiness, log duration of all object-store operations (#6489) ... * During query readiness, log duration of all object-store operations. * Fix duration assignment.	4 years ago
joe miller	6d11a1507b	logcli: add --proxy-url flag for http/https proxy support (#6484) ... Add `--proxy-url` flag to `logcli` to support connecting to Loki instances running behind either HTTP or HTTPS CONNECT-style proxies. Examples: http proxy: ``` logcli \ --addr="http://loki-distributed-querier.loki.svc.cluster.local.:3100/" \ --proxy-url "http://envoy.dom.tld" \ query '{app"foo"}' ``` https proxy with mTLS auth: ``` logcli \ --addr="https://loki-distributed-querier.loki.svc.cluster.local.:3100/" \ --proxy-url "https://envoy.dom.tld" \ --cert tls.crt \ --key tls.key \ --ca-cert ca.crt \ query '{app"foo"}' ``` Note that tail (`-f`) support and https proxies only works with a fork of the `gorilla/websocket` lib. There are open PRs that promise to add support for https proxies such as: https://github.com/gorilla/websocket/pull/740 Add the following to `go.mod` to use the fork with https support: ``` replace github.com/gorilla/websocket v1.4.2 => github.com/philipatl/websocket v1.4.3-0.20211206152948-d16969baa130 ```	4 years ago
Christian Simon	b8c9672de6	Update loki's default port to 3100 (#6349) ... * Update loki's default port to 3100 I was suprised this was not already the case. I think it would be a good change to make, but I am also contious of the breaking nature of this. * Add note to the changelong * Add upgrade note * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Dylan Guedes	c69605f2af	Loki: Fix flaky IndexGatewayClient test (#6481) ... * Stop gRPC clients in tests.	4 years ago
Michel Hollands	61e3e52d4f	Handle duplicate registration of deletion metrics (#6446) ... * Handle duplicate registration of deletion metrics Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Create metrics struct at Loki level Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Michel Hollands	f80e487dcf	Remove whole stream deletion mode (#6435) ... * Remove whole-stream-deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove whole-stream-deletion from docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Sort changelog entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove link to wrong configuration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set default deletion mode to disabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extra white line in documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value in docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add DeletionEnabled method on mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago