apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Dylan Guedes	440fb7d74c	Loki: Modifies TableManager to use IndexGateway ring (#5972 ) * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Use boundaries to skip users in TableManager. - Separate the assigning of indexClient in the IndexGateway to allow initializing the shipper with the IndexGateway ring - Add new TenantBoundariesClient entity, that answers if a given tenantID should be ignored or not - Use the TenantBoundariesClient implemented by the IndexGateway in the downloads TableManager * Add IndexGateway configuration docs. * Add tests for pkg/util/ring TokenFor() and IsAssignedKey() Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Small whitespace fix in pkg/util/ring_test.go Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Apply suggestions from code review Rewrite QueryIndex error phrase. Co-authored-by: JordanRushing <rushing.jordan@gmail.com> * Join users list in a single string. - This is necessary since go-kit doesn't support array type. * Tweak queryReadiness log messages. - As suggested by Ed on https://github.com/grafana/loki/pull/5972#discussion_r859734129 and https://github.com/grafana/loki/pull/5972#discussion_r859736072 * Implement new RingManager. - Adds a new entity to indexgateway named RingManager, responsible for managing the ring and the lifecycler used by the indexgateway. The ringManager is guaranteed to be initiatiated before the Shipper and before the IndexGateway. - Remove the readiness logic from the IndexGateway. - Pass the RingManager as the TenantBoundaries implementation of the Shipper * Return non-empty ringmanager for all modes. * Fix lint regarding error not being check. * Fix lint due to wrong import order. * Implement support for client and server mode for the ring manager. * Fix ring manager services registration. * Add option to configure whether or not to log gateway requests. * Check if IndexGateway is enabled instead of active. * Tune RingManager behavior. - Instantiate ring buffers inside IsAssignedKey instead of reusing same buffer to avoid issues with concurrency. - Remove unnecessary details from IndexGateway mode docs. - Return error when wrongly instantiating a RingManager when IndexGateway is in simple mode. * Rewrite `TenantInBoundaries` as a func instead of interface. * Fix lint. - Fix YAML tag - Remove else clause * Use distributor instead of querier in test. - Since the querier needs to wait for the index gateway ring, it isn't suitable for this test anymore. * Add ring mode guard clause. * Return code erased by mistake. * Rename TenantInBoundaries to IndexGatewayOwnsTenant. * Log len of users instead of all of them to avoid log msgs. * Add docstrings to new public functions. * Apply suggestions from code review Document that tenant filtering is only applied during query readiness. Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Modify IsAssignedKey to expect address directly instead of lifecycler. - Also removes HasDedicatedAddress, since it isn't necessary anymore * Log error message when IsInReplicationSet fails. * Modify IsAssignedKey to return true by default. * Remove wrong assigning of Service for clientMode. * Log gateway requests before error checks. * Remove unnecessary endpoint registration. * Fix lint. * Update pkg/util/ring.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Owen Diehl	3c334001cd	creates a single tsdb for entire wal recovery (#6203 ) * creates a single tsdb for entire wal recovery * tsdb wal replay builds TSDBs with unrounded timestamps to avoid clobbering	4 years ago
Sandeep Sukhani	ef4bc8f1ed	copy all the fixes/improvements that went into boltdb-shipper that are missing from generic index-shipper (#6200 )	4 years ago
Christian Haudum	b073ec9017	security: Redact credentials when marshalled to YAML (#6186 )	4 years ago
Christian Haudum	8518560cf2	Fix flaky gRPC server shutdown in tests (#6191 ) * Use `t.Cleanup(cleanup)` instead of `defer cleanup()` Which seems to reduce the likelyhood of the flaky gRPC server shutdown Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Do not fail test if gRPC server graceful shutdown fails Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Sandeep Sukhani	62f5e598dd	tsdb index gateway (#6158 ) * enable new grpc calls on index gateway and add GetSeries call * use index gateway for tsdb index store queries * add missing file * lint * disable IndexGatewayClientStore for stores other than tsdb until we are ready to enable it again	4 years ago
Owen Diehl	4f87db7f38	strips out tenant label from LabelNames queries (#6181 ) * strips out tenant label from LabelNames queries * prevent tenant labelvalues query	4 years ago
Owen Diehl	6b97de765e	tsdb creation supports fingerprint overrides (#6178 )	4 years ago
Sandeep Sukhani	ead8c345e6	skip empty object key returned by some s3 compatible storage while listing boltdb-shipper index (#6123 )	4 years ago
Sandeep Sukhani	6e3bf3f7d0	tsdb query fixes (#6142 ) * query both downloads and uploads manager when non-nil * fix reading of index file with nodename having a `-` * remove __name__ matcher at a layer above to avoid passing it during tsdb head queries	4 years ago
Owen Diehl	729bf3fc91	loads previously created TSDBs into shipper on startup (#6117 ) * loads previously created TSDBs into shipper on startup * noopTSDBManager impl * info logs for initial tsdb loading	4 years ago
Owen Diehl	756fc19577	fix mutability bug with labels during tsdb flushing (#6116 )	4 years ago
Owen Diehl	55ea89f9fd	Tsdb/head wal feedback (#6107 ) * pr feedback, improved mutex unlocking * tenant heads use sync.Map * tenant heads bench * Revert "tenant heads use sync.Map" This reverts commit `64adebc535`. * removes old code the removed TSDBs outside the shipper + minor fixes	4 years ago
Sandeep Sukhani	06e309bae6	do not increment sync operation count metric on per index set (#6109 )	4 years ago
arcosx	e2fdc3437b	Feat: add Baidu Cloud BOS as storage backends for Loki #4788 (#5848 ) * feat: add baidu bce bos storage support Signed-off-by: arcosx <arcosx@outlook.com> * add baidu bce bos as chunk storage backend Signed-off-by: arcosx <arcosx@outlook.com> * fix: some doc error && rewrite bad code Signed-off-by: arcosx <arcosx@outlook.com> * fix: add the `BceServiceError` source link Signed-off-by: arcosx <arcosx@outlook.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: arcosx <arcosx@outlook.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049 ) * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971 ) * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Patrick Mansfield	b3f6f6b415	table_manager: Change some level.Info() logging to level.Debug() (#6076 ) There is excessive logging of "creating table" and "synching tables" messages, change them to use level.Debug() instead of level.Info() to limit the output. Fixes #1786	4 years ago
Karen Miller	2264b73311	Docs: remove Cortex references (#6079 ) * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update	4 years ago
Christian Simon	213bb86802	Add integration tests which test happy path (#5968 ) * Add integration tests which test happy path This PR adds simple go based integration tests, which can run Loki in two modes: - single-binary - micro-services with index-gateway After standing up it performs the most basic duties like: - ingest logs - run a range query - query label names and label values This is meant to catch more significant problem as early as possible in the PR. We should aim to keep this as simple as possible. * Replace config generation with template Makes configuration much more readable and avoids easy mistakes * Apply suggestions from code review Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	4 years ago
Sandeep Sukhani	2758dc659a	boltdb shipper index list cache improvements (#6054 ) * bypass index list cache when doing query time downloading of index * detect and refresh stale index list cache during sync	4 years ago
Cyril Tovena	9cb78bdad8	Improve index entries deduping and concurrency (#6055 )	4 years ago
Cyril Tovena	5d51e5a54f	Improve batching in boltdb index (#6044 ) * Improve batching in boltdb index * review feedback	4 years ago
Christian Haudum	4891e28bff	Decrease log level for "failed to put to memcached" (#6018 ) Since failing to put to memcache does not have an operational impact on the write path, this log message should not be an error. Make it a warning instead. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Christian Simon	6c1a62d829	Ensure proto definitions are formatted [2/2] (#6000 ) * Update build image to 0.10.4 * Add a fmt target for proto defintions This uses buf to ensure the proto defintions are more consistently formatted. * Update formatted files	4 years ago
Michel Hollands	477a0e71df	Add filter parameter to rebound so lines can be deleted by the compactor (#5879 ) * Add filter parameter to rebound Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issues Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add filter function to delete request Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issues Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Enable api for filter and delete mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add settings for retention Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use labels to check and add test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Simplify filter function Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Also enable filter mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove test settings in config file for docker Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add extra (unused) param for ProcessString in filter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Empty commit to trigger CI again Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix flapping test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove commented out unit tests and add some more Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add extra test case for delete request without line filter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use chunk bounds Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * check if the log selector has a filter if the whole chunk is selected Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * fix lint issue: use correct go-kit import Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	d73df0834a	Improve series store index queries (#6045 )	4 years ago
Dylan Guedes	cd02d6a478	Add better observability to queryReadiness (#5946 ) * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Join users list in a single string. - This is necessary since go-kit doesn't support array type. * Tweak queryReadiness log messages. - As suggested by Ed on https://github.com/grafana/loki/pull/5972#discussion_r859734129 and https://github.com/grafana/loki/pull/5972#discussion_r859736072 * Ensure queryReadinessDuration metric. - It is redundant with a recently added log line. * noop	4 years ago
Kaviraj Kanagaraj	0cc3fe7e54	Disable calling new index-gateway client's API. (#6025 ) * Disable calling new index-gateway client's API. * Remove infinite recursive call * Correct mock setup for test	4 years ago
Trevor Whitney	2a04ac9af7	Remove references to cortex (#6015 )	4 years ago
Christian Haudum	b6ce3826b4	Use new build image 0.20.3 (#5904 )	4 years ago
Travis Patterson	9cef86b162	Invalidate caches on delete (#5661 ) * Generate cache invalidation numbers in the delete store * Get cache generation numbers from the store on request * changlog * rename tombstones to something more meaningful * User invisible module * query frontend relies on a compactor to get the cache generation number * fix serialization * source -> name * lint errors * lint errors * log non-200 responses * add jsonnet changes * lint * review feedback * review feedback * client rename	4 years ago
Christian Simon	337659602e	Add string representation of boltdb shipper mode (#5982 )	4 years ago
Christian Haudum	db238873f9	Call correct method on index store (#5979 ) when index gateway does not implement the new API. This bug did not yield an error but only returned incorrect results because both functions `c.IndexStore.LabelNamesForMetricName` and `c.IndexStore.LabelValuesForMetricName` return `([]string, error)`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Christian Haudum	38e5e410fe	Do not parse string of empty matchers (#5980 ) This fixes a bug in the index gateway when querying values for a label. Since the gRPC handler for LabelValuesForMetricName in the index gateway allows empty matchers in the LabelValuesForMetricNameRequest, we need to check if the matchers string is an empty matcher (`{}`) before we parse the string. This bug was introduced with the index gateway api refactoring in #5892 Fixes: #5965 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
afayngelerindbx	c0f51b9851	redact s3 credential values when printing config to logs (#5961 )	4 years ago
Michel Hollands	ef56b73758	Add query filtering to the tailer (#5969 ) * Add some comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use deletes in tail Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change error message Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Sandeep Sukhani	8f6a75cd0d	fix index gateway chunk refs response (#5957 )	4 years ago
Sandeep Sukhani	6bdecd6a50	common index set should not expire before any of the user index sets (#5954 )	4 years ago
Kaviraj Kanagaraj	3d579fb329	chore: Fix flaky test with `TestSingleIdx`. (#5947 )	4 years ago
Sandeep Sukhani	b3c818e132	build generic index shipper to be shared between boltdb-shipper and tsdb-shipper (#5705 ) * build generic index shipper to be shared between boltdb-shipper and tsdb-shipper * changes suggested from pr review * Update pkg/storage/stores/indexshipper/downloads/index_set.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * update comments on trying to open corrupt files * fix references broken due to storage refactoring * lint Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Travis Patterson	5939d5e9d8	Query filtering in the ingester and storage (#5629 ) * Add timestamp to pipeline and extractor interfaces * implement filter extractor and pipeline * implement storage filtering * implement ingester filtering * test cleanup * Refactor pipeline setup to util * linter * enable filtering based on mode * review feedback * review comments * review feedback * no need to include label filters in the test pipeline * review feedpack	4 years ago
Sandeep Sukhani	d1aff7b29f	store: fix bug in getting chunk ids from ingesters when using boltdb-shipper (#5945 ) * initialize async store in store factory function itself * add a comment about QueryIngestersWithin	4 years ago
Owen Diehl	4c3959b573	Implements TSDB Head (#5864 ) * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting	4 years ago
Cyril Tovena	681195f806	Fixes the index-gateway when using the ring (#5923 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Dylan Guedes	a41926ec50	Only run boltdb-shipper when appropriate. (#5917 )	4 years ago
Cyril Tovena	1331214939	Fixes a panic when response is nil in the index-gw (#5910 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	f691e1bcc3	Introduce a more scalable index-gateway API. (#5892 ) * Introduce a more scalable index-gateway API. Because of the storage refactoring (https://github.com/grafana/loki/pull/5833) we can now introduce a better index-gateway API that fits more our index interface. Such as : ```proto rpc GetChunkRef(GetChunkRefRequest) returns (GetChunkRefResponse) {}; rpc LabelNamesForMetricName(LabelNamesForMetricNameRequest) returns (LabelResponse) {}; rpc LabelValuesForMetricName(LabelValuesForMetricNameRequest) returns (LabelResponse) {}; ``` This will avoid sending thousands of index queries to the index-gateway but instead them just a single request. The index caching, parsing and filtering is now happening all on the index-gateway side. Loki queriers will first check if the new API exists before using it, this way update can be done transparently. However the check happens only on startup, so if you want to start using the new API you need to restart queriers after fully rolling out index-gateways. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Attempt to use the new API on every call. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <christian.simon@grafana.com> * Add test for the GRPC fallback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <christian.simon@grafana.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * review feedback Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the parameters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <christian.simon@grafana.com>	4 years ago
Dylan Guedes	c7e18f535c	Do not use boltdb-shipper if no ring or address are given. (#5902 )	4 years ago
JordanRushing	1dc0d978e4	Add a Ring to IndexGateway (#5358 ) * Begin to add a Ring to IndexGateway Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Implement missing methods for IndexGateway. - Implement missing methods for the IndexGateway to be used as a BasicLifecyclerDelegate. The methods are stored in a lifecycle file and follow the same approach of the Ruler ring - Make Gateway public and add missing parameters to the IndexGateway's initialization method * Fix failing linter Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Implement IndexGateway support for dynamic configs. - Add memberlist as a dependency of the IndexGateway - Add IndexGateway support for the common configuration section - Modify Loki to register IndexGateway flags. This fixes the default flag values for the IndexGateway - Make IndexGateway configurations public. Without this, we can't use them. * Implement NewBasicService for the IndexGateway. - If IndexGateway is running with a ring, it uses the IdleService. Otherwise, it uses the BasicService - Implement IndexGateway support to handle ring HTTP pages * Test IndexGateway dynamic configuration. * Implement new IndexGatewayGRPCPool entity. * Make IndexGateway attributes public. * Implement IndexGatewayRing reader. - Implement new IndexGatewayRing service - Add IndexGatewayRing as a Store dependency - Modify store to pass IndexGatewayRing as a parameter - Implement IndexGatewayClient ring mode * Implement Ring mode in the IndexGatewayClient. - Moves IndexGateway configuration to the IndexGatewayClient struct within the boltdb-shipper - Reuse the IndexGatewayClient everywhere - Implement IndexGateway gRPC pool * Add new ring index gateway parameter to new Store calls. * Use errors.Wrap instead of fmt.Errorf. * Extract tenantID from context instead of iterating on queries. * Remove indexGateway ring param. - Instead, add it to the store configuration and bubble it down to deeper modules. * Split IndexGateway server from client implementation. * Fix imports order. * Remove ring as parameter from IndexGateway-related funcs. * Fix default flag value and IndexQuerier type. * Remove additional mode field and reuse it from cfg. * Remove redundant service init. * Add sanity check for IndexGateway client constructor. * Move mode assigning to initStore method. * Reorder IndexGateway constructor. * Rewrite indexClient chunk.IndexClient as querier Index.Querier. * Fix flag registration for IndexGateway server. - In a previous PR I changed how it works and forgot to update how the flag registration occurs. * Fix flag registration for test. * Keep only one reference to indexQuerier. * Add guard-clause on IndexGatewayRing service. - Without this, we would be initializing ring-related things even if the index gateway is in simple mode. * Move IndexGatewayClientCfg to gateway_client file. * Update CHANGELOG.md for `IndexGateway` support for `RingMode` Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Update GatewayClient to use dskit tenant package Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Add listenport configuration for IndexGateway and Ring Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Make IndexGateway replication factor configurable. - Add `replication_factor` flag to the IndexGateway config struct - Modify Index Gateway ring config struct to be inline with a new struct, since it doesn't expose a replication factor config - Modify dynamic config wrapper to reuse a common replication factor on the Index Gateway ring * Randomize replication set access. - If we don't randomize access, we'll always access same Index Gateway instances in same order for the same tenant * Remove unwanted merge HEAD tags. * Move away from stores/chunk package. * Pass util_log in factory. * Change index gateway client ring to ignore replicas. - This is accomplished by using `NewIgnoreUnhealthyInstancesReplicationStrategy` - It is already done by the server * Refactor where the common replication factor is applied. * Housekeeping config_wrapper IndexGateway configs. - Remove unnecessary/wrong comments - Only set replication factor at a single place Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago

1 2 3 4 5 ...

350 Commits (0dfd2cbbcc15d888466b520ee9345be9b7c8ee8a)