apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953 ) Reverts grafana/loki#9920	2 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920 ) What this PR does / why we need it: Updated Loki with the latest version of dskit Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Christian Haudum	ce81895241	Add shuffle sharding to index gateway (#9710 ) This PR introduces shuffle sharding of the index files across index gateways where the shuffle sharding key is the tenant ID. Instead of having a global replication factor on the index gateways which affects all tenants the same, whether they are small or large tenants, the shuffle sharding allows to specify a shard factor per tenant (on top of the replication factor). This a) reduces the amount of indexes that are downloaded on individual index gateways in case of a high replication factor (which is currently the de-factor standard to deal with high load) and b) allows to scale tenants individually. The shuffle sharding algorithm is part of dskit's ring implementation and is not part of this commit. If you have set a high RF for the index gateways, it is required to reduce the RF and instead set the shard factor for the tenants. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Kaviraj Kanagaraj	4c0e01b93f	revert(9476): Bringback async store like before for the queriers (#9715 ) This seems to cause some issue with `querier`, not getting chunks from ingesters correctly via `async_store`. I narrowed down to this PR. But hard `revert` seems not possible anymore. So created this manually. https://github.com/grafana/loki/pull/9476 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Ashwanth	98d1307c7d	config: ensure storage config defaults apply to named stores (#9650 ) What this PR does / why we need it: Since named store config does not register any flags, storage configs defined under it do not get the defaults. For example [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config) sets the default `storage_class` to `STANDARD`, but the same doesn't get applied by default when using named stores. This PR ensures that named storage configs are always assigned default values when they are unmarshalled by implementing `yaml.Unmarshaler` interface	2 years ago
Travis Patterson	7ff8463def	Fix query behavior in Single Binary and SSD deployments (#9476 ) To keep ingesters from calling themselves, the AsyncStore was only being used when the module was `Querier`, `Ruler`, or `Read`. This means that, for some operations, ingesters were never queries when the deployment mode was single-binary or `Backend` (where the ruler now lives). This change scopes the async store to just the querier rather than all storage access and always includes in it queriers. Because the `AsyncStore` isn't automatically instantiated, now, I've also moved `IngesterQuerier` to be a dependency to the store. That way anything that wants to make an `AsyncStore` is guaranteed to have what it needs.	2 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754 ) Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> What this PR does / why we need it: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](`ff7b462973/pkg/storage/factory.go (L188)`) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). Note: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. Which issue(s) this PR fixes: Fixes #7276 Special notes for your reviewer: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826 ) What this PR does / why we need it: Add support for IBM cloud object storage as storage client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	2 years ago
Danny Kopping	2d93952929	(chore): remove incorrect cache config preample (#8859 ) With the addition of the [config generation tool](https://github.com/grafana/loki/pull/7916), a comment prefix was added for all cache types: `Cache config for index entry writing.` https://grafana.com/docs/loki/next/configuration/#cache_config This appears in the description of each cache config option, and is misleading.	2 years ago
Callum Styan	9a2a038f43	Allow passing of context to query related limits functions (#8689 ) In this PR we're allowing for passing of a `context.Context` via the Limits interfaces (some of which are new, to clean up hardcoding/embedding of `validation.Overrides`) This is based on work/ideas by @jeschkies . Fixes #8694 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	2 years ago
Ashwanth	b3e038c7a8	config_wrapper: Add support for named stores (#7946 ) Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> What this PR does / why we need it: This PR adds support for named stores which allows users to define multiple object store configurations for a storage type. Named stores can be referred to from other sections of the config similar to how provider type is being used `i.e aws, gcs...`. Named store reference should be of the format: `<store_type>.<key_name>` Below is an example configuration with two named stores defined for aws `store-1, store-2`. Schema config is referring to one of them as `aws.store-1` ``` storage_config: aws: endpoint: s3://common-bucket region: us-east1 access_key_id: abc123 secret_access_key: def789 named_stores: aws: store-1: endpoint: s3://foo-bucket region: us-west1 access_key_id: 123abc secret_access_key: 789def store-2: endpoint: s3://bar-bucket region: us-west2 access_key_id: 456def secret_access_key: 789abc schema_config: configs: - from: "2020-07-31" index: period: 24h prefix: loki_prod_index_ object_store: aws.store-1 schema: v11 store: boltdb-shipper ``` Which issue(s) this PR fixes: Fixes #7276 Special notes for your reviewer: Configuration defined in common storage config has no effect on the named stores. Common storage config only overrides storageConfig if it's not explicitly set - same behavior as before. This should be a non-intrusive change. Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916 ) What this PR does / why we need it: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](`5550cd65ec/docs/sources/configuration/_index.md`) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. NOTE: this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/83 Special notes for your reviewer: Files: * [docs/sources/configuration/index.template](`5550cd65ec/docs/sources/configuration/index.template`): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](`c32e5d0acb/docs/sources/configuration/_index.md`): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](`5550cd65ec/tools/doc-generator`) directory with documentation generation tool. Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Christian Haudum	177f37700a	Make TSDB store singleton resettable (#6950 ) This changes allows downstream projects to reset the TSDB store singleton. This must only be called in test cases where a new store instances cannot be explicitly created. Also add a `nil` check to the reset function for the BoltDB shipper singleton instance. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704 ) * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	3 years ago
Christian Haudum	934d56fe5e	Add function to reset boltDBIndexClientWithShipper singleton In a downstream project I have a test case that creates multiple stores with different BoltDB shipper schema configurations. Since the client for the store is a singleton and is re-used, there needs to be a function to reset it as well. This function must only be used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sandeep Sukhani	82da01ad43	make index shipper read path handle range of tables by type of index (#6304 )	3 years ago
Sandeep Sukhani	7369b3785c	use tsdb config for initializing index gateway client (#6267 ) * use tsdb config for initializing index gateway client * review feedback Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Sandeep Sukhani	6ac955cc7d	make boltdb-shipper use the generic indexshipper for managing index in object storage (#6226 )	3 years ago
Dylan Guedes	440fb7d74c	Loki: Modifies TableManager to use IndexGateway ring (#5972 ) * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Add two metrics to the IndexGateway. - Add a new `query_readiness_duration_seconds` metric, that reports query readiness duration of a tablemanager/index gateway instance. We should use it later to report performance against the ring mode - Add a new `usersToBeQueryReadyForTotal` metric, that reports number of users involved in the query readiness operation. We should use it later to correlate number of users with the query readiness duration. * Remove `usersToBeQueryReadyForTotal`. - It will report all users always for now, so it isn't too helpful the way it is. * Rename metric help text to not mislead people. * Log queryReadiness duration. * Fix where log message and duration and triggered. * Use boundaries to skip users in TableManager. - Separate the assigning of indexClient in the IndexGateway to allow initializing the shipper with the IndexGateway ring - Add new TenantBoundariesClient entity, that answers if a given tenantID should be ignored or not - Use the TenantBoundariesClient implemented by the IndexGateway in the downloads TableManager * Add IndexGateway configuration docs. * Add tests for pkg/util/ring TokenFor() and IsAssignedKey() Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Small whitespace fix in pkg/util/ring_test.go Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Apply suggestions from code review Rewrite QueryIndex error phrase. Co-authored-by: JordanRushing <rushing.jordan@gmail.com> * Join users list in a single string. - This is necessary since go-kit doesn't support array type. * Tweak queryReadiness log messages. - As suggested by Ed on https://github.com/grafana/loki/pull/5972#discussion_r859734129 and https://github.com/grafana/loki/pull/5972#discussion_r859736072 * Implement new RingManager. - Adds a new entity to indexgateway named RingManager, responsible for managing the ring and the lifecycler used by the indexgateway. The ringManager is guaranteed to be initiatiated before the Shipper and before the IndexGateway. - Remove the readiness logic from the IndexGateway. - Pass the RingManager as the TenantBoundaries implementation of the Shipper * Return non-empty ringmanager for all modes. * Fix lint regarding error not being check. * Fix lint due to wrong import order. * Implement support for client and server mode for the ring manager. * Fix ring manager services registration. * Add option to configure whether or not to log gateway requests. * Check if IndexGateway is enabled instead of active. * Tune RingManager behavior. - Instantiate ring buffers inside IsAssignedKey instead of reusing same buffer to avoid issues with concurrency. - Remove unnecessary details from IndexGateway mode docs. - Return error when wrongly instantiating a RingManager when IndexGateway is in simple mode. * Rewrite `TenantInBoundaries` as a func instead of interface. * Fix lint. - Fix YAML tag - Remove else clause * Use distributor instead of querier in test. - Since the querier needs to wait for the index gateway ring, it isn't suitable for this test anymore. * Add ring mode guard clause. * Return code erased by mistake. * Rename TenantInBoundaries to IndexGatewayOwnsTenant. * Log len of users instead of all of them to avoid log msgs. * Add docstrings to new public functions. * Apply suggestions from code review Document that tenant filtering is only applied during query readiness. Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Modify IsAssignedKey to expect address directly instead of lifecycler. - Also removes HasDedicatedAddress, since it isn't necessary anymore * Log error message when IsInReplicationSet fails. * Modify IsAssignedKey to return true by default. * Remove wrong assigning of Service for clientMode. * Log gateway requests before error checks. * Remove unnecessary endpoint registration. * Fix lint. * Update pkg/util/ring.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
arcosx	e2fdc3437b	Feat: add Baidu Cloud BOS as storage backends for Loki #4788 (#5848 ) * feat: add baidu bce bos storage support Signed-off-by: arcosx <arcosx@outlook.com> * add baidu bce bos as chunk storage backend Signed-off-by: arcosx <arcosx@outlook.com> * fix: some doc error && rewrite bad code Signed-off-by: arcosx <arcosx@outlook.com> * fix: add the `BceServiceError` source link Signed-off-by: arcosx <arcosx@outlook.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: arcosx <arcosx@outlook.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049 ) * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Sandeep Sukhani	d1aff7b29f	store: fix bug in getting chunk ids from ingesters when using boltdb-shipper (#5945 ) * initialize async store in store factory function itself * add a comment about QueryIngestersWithin	3 years ago
Dylan Guedes	a41926ec50	Only run boltdb-shipper when appropriate. (#5917 )	3 years ago
Dylan Guedes	c7e18f535c	Do not use boltdb-shipper if no ring or address are given. (#5902 )	3 years ago
JordanRushing	1dc0d978e4	Add a Ring to IndexGateway (#5358 ) * Begin to add a Ring to IndexGateway Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Implement missing methods for IndexGateway. - Implement missing methods for the IndexGateway to be used as a BasicLifecyclerDelegate. The methods are stored in a lifecycle file and follow the same approach of the Ruler ring - Make Gateway public and add missing parameters to the IndexGateway's initialization method * Fix failing linter Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Implement IndexGateway support for dynamic configs. - Add memberlist as a dependency of the IndexGateway - Add IndexGateway support for the common configuration section - Modify Loki to register IndexGateway flags. This fixes the default flag values for the IndexGateway - Make IndexGateway configurations public. Without this, we can't use them. * Implement NewBasicService for the IndexGateway. - If IndexGateway is running with a ring, it uses the IdleService. Otherwise, it uses the BasicService - Implement IndexGateway support to handle ring HTTP pages * Test IndexGateway dynamic configuration. * Implement new IndexGatewayGRPCPool entity. * Make IndexGateway attributes public. * Implement IndexGatewayRing reader. - Implement new IndexGatewayRing service - Add IndexGatewayRing as a Store dependency - Modify store to pass IndexGatewayRing as a parameter - Implement IndexGatewayClient ring mode * Implement Ring mode in the IndexGatewayClient. - Moves IndexGateway configuration to the IndexGatewayClient struct within the boltdb-shipper - Reuse the IndexGatewayClient everywhere - Implement IndexGateway gRPC pool * Add new ring index gateway parameter to new Store calls. * Use errors.Wrap instead of fmt.Errorf. * Extract tenantID from context instead of iterating on queries. * Remove indexGateway ring param. - Instead, add it to the store configuration and bubble it down to deeper modules. * Split IndexGateway server from client implementation. * Fix imports order. * Remove ring as parameter from IndexGateway-related funcs. * Fix default flag value and IndexQuerier type. * Remove additional mode field and reuse it from cfg. * Remove redundant service init. * Add sanity check for IndexGateway client constructor. * Move mode assigning to initStore method. * Reorder IndexGateway constructor. * Rewrite indexClient chunk.IndexClient as querier Index.Querier. * Fix flag registration for IndexGateway server. - In a previous PR I changed how it works and forgot to update how the flag registration occurs. * Fix flag registration for test. * Keep only one reference to indexQuerier. * Add guard-clause on IndexGatewayRing service. - Without this, we would be initializing ring-related things even if the index gateway is in simple mode. * Move IndexGatewayClientCfg to gateway_client file. * Update CHANGELOG.md for `IndexGateway` support for `RingMode` Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Update GatewayClient to use dskit tenant package Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Add listenport configuration for IndexGateway and Ring Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Make IndexGateway replication factor configurable. - Add `replication_factor` flag to the IndexGateway config struct - Modify Index Gateway ring config struct to be inline with a new struct, since it doesn't expose a replication factor config - Modify dynamic config wrapper to reuse a common replication factor on the Index Gateway ring * Randomize replication set access. - If we don't randomize access, we'll always access same Index Gateway instances in same order for the same tenant * Remove unwanted merge HEAD tags. * Move away from stores/chunk package. * Pass util_log in factory. * Change index gateway client ring to ignore replicas. - This is accomplished by using `NewIgnoreUnhealthyInstancesReplicationStrategy` - It is already done by the server * Refactor where the common replication factor is applied. * Housekeeping config_wrapper IndexGateway configs. - Remove unnecessary/wrong comments - Only set replication factor at a single place Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago

27 Commits (2880b5055a71f337e2a0f8bfe724faac08c73294)