apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	856c11dad2	feat: Add support for storing chunk deletion markers in object storage instead of local disk (#19689 )	2 months ago
Sandeep Sukhani	9e0f50c362	feat(compactor-hs): add a metric to track the number of jobs left to conclude ongoing unit of work (#19506 )	3 months ago
Sandeep Sukhani	000f90d74b	feat(compactor HS): Fixes and changes for getting horizontally scalable compactor to work well (#18526 )	6 months ago
Sandeep Sukhani	bd293206cd	feat(compactor HS): Wire things up and add some metrics (#18421 )	6 months ago
Sandeep Sukhani	1ea8d48fc4	chore: expose tables manager interface from compactor (#18344 )	6 months ago
Sandeep Sukhani	364660748d	feat(compactor HS): refactor interfaces and build a link between delete requests manager and compactor for HS of compactor (#18326 ) Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	6 months ago
Sandeep Sukhani	ba3c67076a	feat(compactor HS): add support for applying storage updates after finishing processing of deletion manifest (#18294 )	6 months ago
Footur	ba8830b080	docs: Document the default value for the working_directory for compactor (#17075 ) Signed-off-by: Footur <3769085+Footur@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	9 months ago
Sandeep Sukhani	db59d0e9b2	fix: wire up deletion series progress tracking (#17099 )	9 months ago
Sandeep Sukhani	e3e1f096ea	feat: store details of processed streams while processing delete requests (#16825 )	10 months ago
Sandeep Sukhani	3277321e21	feat: when using sqlite for storing delete requests, use the stored completion time to reduce the requests considered for querytime filtering (#16600 )	10 months ago
Sandeep Sukhani	5b33e65bcd	feat: add support for using sqlite for storing delete requests (#16437 )	10 months ago
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438 ) Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	10 months ago
Cyril Tovena	dbf2befc1d	feat: Distributed Operational UI (#16097 )	11 months ago
Sandeep Sukhani	bd0ca9387d	chore: refactor delete requests store interface to prepare for replacing boltdb with sqlite for storing delete requests (#16181 )	11 months ago
Vladyslav Diachenko	31368806a9	feat: add backoff mechanism to the retention process (#14182 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Ed Welch	705379cc81	chore: remove experimental flags from features and configs which are no longer so. (#12481 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Sandeep Sukhani	0b7ff48175	chore: delete request processing improvements (#12259 )	2 years ago
Salva Corts	63c88489d0	refactor: Add RF and Tokens to Loki ring Cfg and allow overwriting docs. (#12142 ) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Cyril Tovena	19f5f3fd44	fix: Fixes compactor shutdown log (#12195 )	2 years ago
Owen Diehl	928e3f34f8	Blooms: Prepare compactor refactor (#11847 ) Prepares `bloomcompactor` pkg for integration with new logic, primarily removes/thins out existing code.	2 years ago
Sandeep Sukhani	3a7940af1c	compactor" track number of times table compactions were skipped consecutively (#11604 ) What this PR does / why we need it: Earlier we had a counter for the number of times tables were skipped by compaction due to them being locked by retention. Counter makes it hard to reliably write an alert without making it noisy. I am changing the counter to gauge which would count the number of times tables were consecutively skipped by compaction. This would make it easier to write alerts. Checklist - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Sandeep Sukhani	162bbb11be	do not run retention loop when retention is not enabled (#11280 ) What this PR does / why we need it: Do not run retention loop when retention is not enabled	2 years ago
Sandeep Sukhani	6a62b8cf42	compaction: Separate metrics for tracking retention and compaction (#11263 ) What this PR does / why we need it: In PR #9884, we separated the retention loop from compaction to avoid blocking compaction for too long due to some intensive delete requests. Currently, we track retention and compaction using the same metrics. This PR adds separate metrics for monitoring retention operation. I have also updated the Retention dashboard to use the new metrics.	2 years ago
Sandeep Sukhani	c716e498ad	compactor: do not block compation when retention is taking too long (#9884 ) What this PR does / why we need it: Currently, we perform compaction and apply retention in the same loop. Although we have a flag for not applying retention every time we perform compaction, we still see compaction getting blocked when processing some intensive delete requests(processed while applying retention). This PR separates out the compaction and retention to run in a separate loop. I have added a table-locking feature to avoid compaction and retention from processing the same tables at a time. However, compaction and retention would treat locked tables differently, as explained below: * When compaction sees a table is locked: It would skip the table and move on to the following table. However, before skipping, it would check if the table has any uncompacted files and increment the newly added counter called `loki_compactor_skipped_compacting_locked_tables_total` to track how often we are skipping compacting tables which have uncompacted files. * When retention sees a table is locked: It would wait for the lock to be released since we can't skip any tables while processing delete requests. Special notes for your reviewer: * The check for tables with uncompacted files looks for count > 1 because initially, we did not support per tenant index in `boltdb-shipper`, so a table can have a single compacted multi-tenant index file. In a rare case where we have a single file which was supposed to be compacted away, it is okay to have a single uncompacted file for a while. The aim here is to not block compaction for too long in a large cell with too many uncompacted files. * Retention only works on the compacted index, so we first compact down any uncompacted files while applying retention. Checklist - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151 ) What this PR does / why we need it: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` Special notes for your reviewer: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields Checklist* - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
April Schleck	4b750cc0cc	Storage: Allow setting a constant prefix for all created keys (#10096 ) What this PR does / why we need it: Adds a new option under the aws stanza named key_prefix. This is useful if you have many Loki installations and do not want to create a million buckets. This is different than `compactor.shared_store_key_prefix` because it also affects eg the `loki_cluster_seed.json` file. Which issue(s) this PR fixes: Fixes #5889 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Ed Welch <edward.welch@grafana.com>	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Ashwanth	a118e994d8	Remove shared_store and shared_store_key_prefix from shipper and compactor (#10840 ) What this PR does / why we need it: #### Removes `shared_store` and `shared_store_key_prefix` from index shipper and compactor configs and their corresponding CLI flags. - `-tsdb.shipper.shared-store` - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` `shared_store` has been a confusing option allowing users to easily misconfigure Loki. Going forward `object_store` setting in the [period_config](https://grafana.com/docs/loki/latest/configure/#period_config) (which already configured the store for chunks) will be used to configure store for the index. And the newly added `path_prefix` option under the `index` key in `period_config` will configure the path under which index tables are stored. This change enforces chunks and index files for a given period reside together in the same storage bucket. More details in the upgrade guide. --- `-compactor.delete-request-store` has to be explicitly configured going forward. Without setting this, loki wouldn't know which object store to use for storing delete requests. Path prefix for storing deletes is decided by `-compactor.delete-request-store.key-prefix` which defaults to `index/`. Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Christian Haudum	2abb472e7e	Unify ring managers across components (#10931 ) What this PR does / why we need it: This PR creates a re-usable `RingManager` implementation that is then used by * Query Scheduler ring * Index Gateway ring * Bloom Gateway ring * Bloom Compactor ring Which issue(s) this PR fixes: Every component had its own implementation of the ring manager which only really differed in the log messages containing the component name. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	8bb615c2cc	config: loki better defaults (#10793 ) What this PR does / why we need it: Updates some of the configuration defaults to provide a better experience for users out of the box. I tried to add notes for non-trivial changes explaining the reason. \| configuration \| new default \| old default \| notes \| \| ------------------------------------------------------ \| ----------- \| ----------- \| -------- \| `compactor.delete-max-interval` \| 24h \| 0 \| splits the delete requests into intervals no longer than `delete_max_interval` \| \| `distributor.max-line-size` \| 256KB \| 0 \| - \| \| `ingester.sync-period` \| 1h \| 0 \| ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. \| \| `ingester.sync-min-utilization` \| 0.1 \| 0 \| ^ \| \| `frontend.max-querier-bytes-read` \| 150GB \| 0 \| - \| \| `frontend.max-cache-freshness` \| 10m \| 1m \| avoid caching results for data that is still in churn \| \| `frontend.max-stats-cache-freshness` \| 10m \| 0 \| avoid caching results for data that is still in churn \| \| `memcached.batchsize` \| 256 \| 1024 \| - \| \| `memcached.parallelism` \| 10 \| 100 \| - \| \| `querier.tsdb-max-query-parallelism` \| 128 \| 512 \| avoid over parallelising queries for smaller installations \| \| `querier.split-queries-by-interval` \| 1h \| 30m \| avoid over parallelising queries for smaller installations \| \| `querier.compress-http-responses` \| true \| false \| compress response if the request accepts gzip encoding \| \| `query-scheduler.max-outstanding-requests-per-tenant` \| 32000 \| 100 \| current default is too small, opening a big dashboard could hit this limit \| \| `validation.max-label-names-per-series` \| 15 \| 30 \| reducing this to avoid blowing up the series count and index size. Also enforces users to think about what labels are being added. \| Special notes for your reviewer: regarding `TestQueryTSDB_WithCachedPostings`: removed the assert on get calls as this is hard to get right (splits, querySize mw making stats calls) validating additions to the cache and misses is more important here I think. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Ashwanth	91cae03fe1	chore: restructure compactor package (#10775 ) What this PR does / why we need it: This pr moves the compactor pkg from `pkg/storage/stores/shipper/indexshipper/compactor` to `pkg/compactor` - same level as other high level components (querier, distributor...) table compaction code however resides in the respective store pkg closer to other store specific code. `pkg/storage/stores/shipper/indexshipper/boltdb/compactor/table_compactor.go` `pkg/storage/stores/shipper/indexshipper/tsdb/compactor.go` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729 ) What this PR does / why we need it: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │ └── compactor ├── compactor │ ├── client │ │ └── grpc │ ├── deletion │ ├── deletionmode │ ├── generationnumber │ └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │ ├── index │ ├── testdata │ └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... Special notes for your reviewer: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	18482c67d6	compactor: deprecate -boltdb.shipper.compactor. prefix in favor of -compactor. (#10344 ) What this PR does / why we need it: Compactor config use `-boltdb.shipper.compactor.` as prefix which is misleading as the config options apply to tsdb index as well. This pr introduces `-compactor.` prefix which can now be used to configure compactor configuration options and deprecates `-boltdb.shipper.compactor.` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091 )	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953 ) Reverts grafana/loki#9920	3 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920 ) What this PR does / why we need it: Updated Loki with the latest version of dskit Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Sandeep Sukhani	2ed3c15c22	index: allow refreshing index file names cache for specific tables (#9798 ) What this PR does / why we need it: We have a cache for index files available in the object storage, which helps reduce the number of list calls we make to the object storage. Currently, we only support refreshing the whole index cache, which includes all the files for all tables and users. However, in a large cell with many tables and tenants, it is less likely that all the tables are being referenced. Also, most iterative operations would only need a fresh listing of just a few tables at a time so it is wasteful to always refresh the whole index cache. This PR adds support for refreshing the index cache for only specific tables. I am doing it as part of some compactor improvements to run compaction and retention in separate goroutines to avoid blocking compaction when some expensive delete requests are being processed. To make those changes, I needed a way to cost-effectively refresh the index files in the table being picked up for processing. Checklist - [x] Tests updated	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212 ) What this PR does / why we need it: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447 ) What this PR does / why we need it: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](`a85d01c9cd/docs/sources/upgrading/_index.md`) for more details. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/7276 Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826 ) What this PR does / why we need it: Add support for IBM cloud object storage as storage client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	3 years ago
Callum Styan	9a2a038f43	Allow passing of context to query related limits functions (#8689 ) In this PR we're allowing for passing of a `context.Context` via the Limits interfaces (some of which are new, to clean up hardcoding/embedding of `validation.Overrides`) This is based on work/ideas by @jeschkies . Fixes #8694 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Sandeep Sukhani	b2d0481cbe	deletion: fix log deletion with line filters (#8151 )	3 years ago
Sandeep Sukhani	dfff86f1cd	skip compacting tables that do not fit in start and end date of desired schema config (#8127 ) Before compacting tables, we look up the schema config for them. If we do not find a schema config for a table, we skip them. However, the schema lookup code was considering the start time of the schema and not the end time. This can cause the compactor to try compacting tables with the same prefix as one of the previous schemas, but the table interval falls into one of the subsequent schemas with a different config. This PR fixes the issue and simplifies the code by reusing existing schema lookup code. I have also updated test cases to verify the fix works.	3 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916 ) What this PR does / why we need it: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](`5550cd65ec/docs/sources/configuration/_index.md`) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. NOTE: this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/83 Special notes for your reviewer: Files: * [docs/sources/configuration/index.template](`5550cd65ec/docs/sources/configuration/index.template`): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](`c32e5d0acb/docs/sources/configuration/_index.md`): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](`5550cd65ec/tools/doc-generator`) directory with documentation generation tool. Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	1410808ee9	use grpc for communicating with compactor for query time filtering of data requested for deletion (#7804 ) What this PR does / why we need it: Add grpc support to compactor for getting delete requests and gen number for query time filtering. Since these requests are internal to Loki, it would be good to use grpc instead of HTTP same as all the internal requests we do in Loki. I have added a new config for accepting the grpc address of the compactor. I tried having just the existing config and detecting if it is a grpc server, but it was hard to do it reliably, considering the different deployment modes we support. I think it is safe to keep it the same and eventually deprecate the existing config. Checklist - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated	3 years ago
Michel Hollands	accb5e1ca2	Log the interval of a delete request (#7120 ) What this PR does / why we need it: An interval is generated automatically for a delete request: this is the max size of a shard of a deletion request. This value is now logged. Additionally a setting name that should have been renamed was fixed. Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
afayngelerindbx	4b5b16569d	compactor: adds tunable to control upload parallelism (#6817 ) What this PR does / why we need it: This can help with compaction latency. I patched the compactor to set this to 100. Which issue(s) this PR fixes: Helps with #6775 Checklist - [x] Documentation added	3 years ago

33 Commits (main)