12 Commits (dea5d7883a05856c99e18db976ff88e25a40ac31)

Author	SHA1	Message	Date
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Dylan Guedes	30fbe38341	Revert "Introduces cache to TSDB postings (#9621)" (#11157) ... This reverts commit 1221658fc5. **What this PR does / why we need it**: We decided to discontinue the postings cache a few releases ago for not finding any evidences that postings was the culprit for the bad igw performance. Plus, we noticed a major issue with it: it doesn't bubble down shards correctly, so every query end up processing every shard.	2 years ago
Ashwanth	a118e994d8	Remove shared_store and shared_store_key_prefix from shipper and compactor (#10840) ... **What this PR does / why we need it**: #### Removes `shared_store` and `shared_store_key_prefix` from index shipper and compactor configs and their corresponding CLI flags. - `-tsdb.shipper.shared-store` - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` `shared_store` has been a confusing option allowing users to easily misconfigure Loki. Going forward `object_store` setting in the [period_config](https://grafana.com/docs/loki/latest/configure/#period_config) (which already configured the store for chunks) will be used to configure store for the index. And the newly added `path_prefix` option under the `index` key in `period_config` will configure the path under which index tables are stored. This change enforces chunks and index files for a given period reside together in the same storage bucket. More details in the upgrade guide. --- `-compactor.delete-request-store` has to be **explicitly configured** going forward. Without setting this, loki wouldn't know which object store to use for storing delete requests. Path prefix for storing deletes is decided by `-compactor.delete-request-store.key-prefix` which defaults to `index/`. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729) ... **What this PR does / why we need it**: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │   └── compactor ├── compactor │   ├── client │   │   └── grpc │   ├── deletion │   ├── deletionmode │   ├── generationnumber │   └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │   ├── index │   ├── testdata │   └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... **Special notes for your reviewer**: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	02ae366451	tool for building bloom filters from log contents (#10594) ... This adds a tool which builds scalable bloom filters from the contents of logs themselves & records metrics. --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Đurica Yuri Nikolić	37ec68eebb	Remove unnecessary code after dskit log.Interface has been dereleased (#10434) ... This PR removes unnecessary code used to convert the dereleased `dslog.Interface` (see https://github.com/grafana/dskit/pull/359) into `go-kit.Logger`. Additionally, it modifies the signature and implementation of `pkg/util/log/log.go`'s `InitLogger` as follows: - previous signature ``` InitLogger(*server.Config, prometheus.Registerer, bool, bool) ``` has been replaced with ``` InitLogger(*server.Config, prometheus.Registerer, bool, bool) log.Logger ``` Namely, the new `InitLogger` initialises the global logger and returns it. - the new implementation initializes the global logger, but it does not override the default logger for the server. This is now done by the callers of `InitLogger`. Previously, it was done inside `InitLogger` for convenience, since the types of the 2 loggers were not the same. Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Owen Diehl	dc9ca8e19a	Tools/index analyzer (#9074) ... Adds a tool that allows downloading and querying statistics across TSDB indices. Example output: ``` analyzed 1046302 series and 7607495 chunks for an average of 7.270841 chunks per series. max chunks/series was 45695. number of series with over 1k chunks: 15 ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago