apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Quentin DUPUY	c5ec1ac27f	fix/docs: Fixed a typo in the ingestor section (#7285 )	3 years ago
Karsten Jeschkies	c53c6072cf	[Mixin] Rename `logs` template variable to `loki_datasource`. (#7278 ) What this PR does / why we need it: Grafana Cloud integration have the convention to name data source variables after `<type>_datasource`. That means `$logs` should become `loki_datasource`.	3 years ago
Gerard Vanloo	bcdf17b0a5	Configurable TLS MinVersion and CipherSuites (#7227 )	3 years ago
Dylan Guedes	2dc5a71a67	Tests: Reduce flakiness of GCPLog and integration tests (#7273 ) What this PR does / why we need it: - Reduce flakiness of GCPLog tests by reusing context and stopping GCP topic - Reduce flakiness of integration tests by: - using port as they're assigned - reusing context - adding timeout to calls (hence, if something get stuck, tests fail earlier)	3 years ago
Michel Hollands	77c10da81d	Remove deletion filter debug log messages (#7277 ) These debug messages are a slightly misleading and appear too frequent. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Dylan Guedes	f3b72c8b59	Promtail: Change max support compressed line size to be 2MB (#7268 ) What this PR does / why we need it: Change Promtail compression support max line size to 2MB instead of 4096 bytes (~4KB). This however doesn't change the default buffer size. 4096 bytes means we can't ever have a buffer size bigger than ~4KB but since the default mem page size is 4096 ([source](https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space)), we end up ignoring all lines.	3 years ago
Pablo	64fe0af9dd	Expose tenant ID through relabel in push promtail targets (#7250 ) What this PR does / why we need it: Some Promtail targets that support pushing logs through HTTP, allow using a specific header for writing logs for multiple tenants. This log is propagated through a Promtail internal label, that is then consumed by the remote write client. If Promtail is used in a multi-tenant manner, it'd be nice to somehow be able to track how much is each tenant's share from the logs received. This PR exposes the received tenant id value through relabeling. This allows for example, keeping it as a label and exposing a metric though the `metrics` pipeline stage, which can be segregated by tenant. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067	3 years ago
Dylan Guedes	73bea7e4fa	Promtail: Add compressed files support (#6708 ) What this PR does / why we need it: Adds to Promtail the ability to read compressed files. It works by: 1. Infer which compression format to use based on the file extension 2. Uncompress the file with the native `golang/compress` packages 3. Iterate over uncompressed lines and send them to Loki Its usage is the same as our current file tailing. Which issue(s) this PR fixes: Fixes #5956 Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Bryan Boreham	5ee0d09389	chunks: update klauspost/compress package to v1.15.11 (#7263 ) What this PR does / why we need it: Bring in various improvements from upstream, particularly https://github.com/klauspost/compress/pull/668 Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Benchmarks (using changes from #7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-flate-4 81.5ms ± 5% 78.7ms ± 2% -3.39% (p=0.016 n=5+5) Read/none_66_kB-4 4.16ms ± 6% 3.93ms ± 4% -5.51% (p=0.032 n=5+5) Read/flate_66_kB-4 168ms ± 4% 179ms ±11% +6.50% (p=0.032 n=5+5) Read/zstd_66_kB-4 334ms ±16% 152ms ± 3% -54.57% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 3.81ms ± 2% -2.91% (p=0.008 n=5+5) Read/zstd_262_kB-4 369ms ± 8% 131ms ± 1% -64.62% (p=0.008 n=5+5) Read/zstd_524_kB-4 466ms ±17% 146ms ± 2% -68.68% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 420ms ±18% 224ms ± 1% -46.69% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 214ms ± 1% 218ms ± 1% +1.90% (p=0.016 n=5+5) Read/sample_zstd_262_kB-4 418ms ±21% 200ms ± 2% -52.15% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 576ms ±35% 234ms ± 1% -59.42% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-gzip-4 267MB/s ±10% 284MB/s ± 2% +6.51% (p=0.008 n=5+5) Write/ordered-flate-4 274MB/s ± 4% 291MB/s ± 2% +6.49% (p=0.008 n=5+5) Read/none_66_kB-4 1.82GB/s ± 6% 1.92GB/s ± 4% +5.78% (p=0.032 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 698MB/s ± 1% +4.30% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 930MB/s ± 3% +119.12% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 1.99GB/s ± 2% +3.00% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 721MB/s ± 2% +2.96% (p=0.032 n=5+5) Read/zstd_262_kB-4 357MB/s ± 8% 1009MB/s ± 1% +182.89% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 763MB/s ± 1% +1.74% (p=0.008 n=5+5) Read/zstd_524_kB-4 365MB/s ±18% 1149MB/s ± 2% +215.34% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 341MB/s ±17% 616MB/s ± 9% +80.67% (p=0.008 n=5+5) Read/sample_zstd_262_kB-4 318MB/s ±18% 658MB/s ± 2% +107.08% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 521MB/s ± 1% 530MB/s ± 1% +1.72% (p=0.032 n=5+5) Read/sample_zstd_524_kB-4 299MB/s ±28% 718MB/s ± 1% +140.23% (p=0.008 n=5+5) name old %compressed new %compressed delta Write/ordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.000 n=5+4) Write/ordered-flate-4 6.89 ± 0% 6.69 ± 0% -2.84% (p=0.008 n=5+5) Write/ordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) Write/unordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.008 n=5+5) Write/unordered-flate-4 6.89 ± 0% 6.70 ± 0% -2.81% (p=0.000 n=4+5) Write/unordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.8MB ± 1% -5.28% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 735kB ± 1% +2.99% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 733kB ± 0% +2.57% (p=0.008 n=5+5) Read/zstd_66_kB-4 423MB ± 0% 1MB ± 0% -99.65% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 204kB ± 0% +3.07% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 169kB ± 0% -0.07% (p=0.000 n=5+4) Read/flate_262_kB-4 197kB ± 0% 205kB ± 0% +3.77% (p=0.008 n=5+5) Read/zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.92% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/flate_524_kB-4 95.4kB ± 0% 98.7kB ± 0% +3.46% (p=0.008 n=5+5) Read/zstd_524_kB-4 786MB ± 0% 1MB ± 6% -99.93% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 703kB ± 0% +2.48% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 710kB ± 1% +3.08% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 423MB ± 0% 2MB ± 0% -99.59% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 200kB ± 0% +3.10% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 194kB ± 0% 201kB ± 0% +3.77% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.91% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 97.9kB ± 0% +2.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 98.8kB ± 0% +3.71% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 786MB ± 0% 1MB ± 0% -99.90% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Write/ordered-gzip-4 701 ± 0% 718 ± 1% +2.40% (p=0.008 n=5+5) Write/ordered-lz4-256k-4 656 ± 0% 657 ± 0% +0.21% (p=0.048 n=5+5) Write/ordered-flate-4 626 ± 0% 634 ± 1% +1.37% (p=0.008 n=5+5) Write/ordered-zstd-4 2.07k ± 0% 1.44k ± 0% -30.49% (p=0.008 n=5+5) Write/unordered-gzip-4 418k ± 0% 428k ± 0% +2.39% (p=0.008 n=5+5) Write/unordered-flate-4 418k ± 0% 430k ± 0% +2.89% (p=0.008 n=5+5) Write/unordered-zstd-4 469k ± 0% 468k ± 0% -0.12% (p=0.008 n=5+5) Read/gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 13.7k ± 0% 13.7k ± 0% -0.01% (p=0.029 n=4+4) Read/flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.72% (p=0.008 n=5+5) Read/zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.19% (p=0.008 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.76k ± 0% +3.17% (p=0.016 n=4+5) Read/flate_262_kB-4 4.61k ± 0% 4.78k ± 0% +3.62% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 2.54k ± 0% -64.22% (p=0.016 n=5+4) Read/gzip_524_kB-4 2.62k ± 0% 2.69k ± 0% +2.36% (p=0.016 n=4+5) Read/flate_524_kB-4 2.62k ± 0% 2.70k ± 0% +3.09% (p=0.008 n=5+5) Read/zstd_524_kB-4 6.13k ± 2% 1.63k ± 0% -73.40% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.88% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.00% (p=0.008 n=5+5) Read/sample_gzip_262_kB-4 4.63k ± 0% 4.77k ± 0% +3.16% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 4.62k ± 0% 4.79k ± 0% +3.61% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 7.12k ± 1% 2.54k ± 0% -64.28% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.69k ± 0% +2.36% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.72k ± 0% +3.31% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 6.11k ± 2% 1.64k ± 0% -73.23% (p=0.008 n=5+5) ```	3 years ago
Michel Hollands	391b9cea10	Logcli: load remote schema before validation (#7258 ) The --remote-schema option didn't work as validation would be called before the remote schema was loaded. An error about a missing schema section is the result. This was tested by running against a remote bucket containing a schemaconfig.yaml file with a local config file that does not contain a schema config section. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Dylan Guedes	dcfba366de	Loki: Shard streams based on desired rate (#7199 ) What this PR does / why we need it: Modify our sharding mechanism to be dictated by the desired rate and ingestion rate. Once the number of shards is calculated, the stream entries will be distributed across the shards. If the number of entries is lower than number of shards, we report a metric and use number of entries as the number of shards.	3 years ago
Shweta Padubidri	eb949e2907	Operator: Config and bundle manifest changes for Lokistack API ObjectstorageTLS CA certificate key addition (#7256 )	3 years ago
Owen Diehl	dab6a24541	correct heartbeat timeout to distributor lifecycler for ring page (#7253 ) Passes the correct `heartbeat_timeout` to the created lifecycler config in distributor ring construction. When we didn't include this, it would create a visual bug incorrectly marking many distributors as unhealthy when in fact they were in fine condition. Including The _lifecycler_ `heartbeat_timeout` is _only_ referenced in the status page [here](`74c8cf03ba/vendor/github.com/grafana/dskit/ring/lifecycler.go (L892-L894)`). The _ring_ `heartbeat_timeout` continues to be updated correctly.	3 years ago
Michel Hollands	74c8cf03ba	Update deletions dashboard (#7244 ) Add a panel for the deletions in progress (which is received - processed). This is similar to the existing 'Number of pending requests' only panel but seems more reliable. I'll remove one later after this has been running for a while. Add some compactor CPU, memory and processing duration statistics so you can see how the compactor is doing. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Vladyslav Diachenko	3e7e44400a	bumped GEL version in Loki helm chart to v1.5.2 (#7241 ) GEL release 1.5.2	3 years ago
李国忠	4089b73bc3	[Optimization] logql client: When logql fails to execute, return the failure information to the caller (#6671 ) …ail msg to the client caller. When logql fails to execute, return the failure information to the caller. Solve the problem that grafana's loki alarm cannot see the cause of failure. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Shweta Padubidri	1489c17312	operator: Add support for certificates to allow external S3 storage connections (#7070 ) Signed-off-by: Shweta Padubidri <spadubid@redhat.com> This PR supports bringing up Loki in a cluster that runs behind a proxy It also adds support to provide proxy certificates to support secure traffic behind a proxy in a cluster.	3 years ago
Pablo	42bfc22d23	Fix promtail `metric` stage causing failure when getting extracted metrics (#7216 ) What this PR does / why we need it: Promtail allows ingested log entries to be modified before pushing them to Loki with the use of pipelines. One of the available pipeline "stages" is called `metrics`, and allows extracting metrics from incoming logs. These metrics are labeled with the labels available in the processed `Entry`. There's some edge cases in which if either a entry label is invalid, or it starts with the prometheus internal label prefix, `__`, calling the promtail `/metrics` fails. One way to easily reproduce this is by having the following configuration: ```yaml pipeline_stages: - static_labels: good_label: "1" - tenant: value: "2" - metrics: loki_count: type: Counter description: "should count all entries" config: match_all: true action: inc ``` Which exposes a `promtail_custom_loki_count` metrics that counts processed log entries. In this case, the step before the `metrics` one injects an internal label `__tenant_id__` to be used by the remote write client. Since this label starts with the prometheus reserved prefix, this will make calling the `metrics` endpoint fail with an error like the following: ``` An error has occurred while serving metrics: 1 error(s) occurred: * "__tenant_id__" is not a valid label name for metric "promtail_custom_loki_count" ``` This PR attempts to fix that by, before creating the corresponding metrics, dropping invalid or internal labels. That way promtail will just use labels that are valid. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067 Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
tonistuulNG	277d8261eb	Documentation: Fix broken link (#7136 ) There is currently a broken link leading to 404 in Loki documentation.	3 years ago
Periklis Tsirakidis	5acf651b59	operator: Fix ruler grpc tls client configuration (#7214 )	3 years ago
Yahya	6bb73ef97e	Docs: remove redundant section in Template functions (#6458 ) Pipelines are already defined in L18-19	3 years ago
Garrett Guillotte	7eadddbcfc	Docs: Populate empty index pages (#7200 ) Three docs index pages are dead ends - they are either blank or have no links. In lieu of more detailed content, this PR populates these pages with `{{< section >}}` shortcodes that automatically generate TOC-like links to subpages.	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212 ) The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
李国忠	557fdf2206	[bugfix] logql: vectorExpr run fail when query_range: parallelise_shardable_queries: true (#7045 ) …hardable_queries: true <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: A grafana error occurred when I deployed the vector(0) function to our online loki cluster. It was finally found because online loki has a configuration `unexpected expr type (syntax.VectorExpr) for ASTMapper type (logql.ShardMapper) ` ```yaml query_range: parallelise_shardable_queries: true ``` Which issue(s) this PR fixes: Fixes # Special notes for your reviewer: error snapssnapshot ![image](https://user-images.githubusercontent.com/9583245/188357957-e573e589-a789-4e5e-a1c3-b91c019ceb62.png) with this pr，success snapshot ![image](https://user-images.githubusercontent.com/9583245/188358036-5ca0d9be-c2b6-4516-807c-c027445ee62d.png) <!-- Note about CHANGELOG entries, if a change adds: an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	2af3d680cb	tsdb supports data deletion, this PR takes care of todo to enable delete request client for tsdb (#7223 ) What this PR does / why we need it: `tsdb` index store supports deletion of data, but we had missed enabling the creation of delete requests client, which is required for query time filtering of data requested for deletion. This PR takes care of it. I have also removed some unused code and adjusted the tests accordingly. Checklist - [x] Tests updated	3 years ago
Danny Kopping	16a7dd19a2	Jsonnet: fix overrides exporter to use correct CPU limits (#7221 ) Currently the overrides exporter uses 200 vCPUs due to a missing `m` unit indicating "milli-CPUs".	3 years ago
Alexandru Gologan	5e134f3b6d	Helm: add support for configuring ruler (#7196 ) What this PR does / why we need it: Enables configuring various `ruler` options like alertmanager url. `rulerConfig` is added second in the template so users can override the `storage` config if necessary. Have seen questions in community Slack asking how to get filesystem support and alertmanager back when migrating from the old loki helm which should now be possible.	3 years ago
Alexandru Gologan	f4967c46da	Helm: expose Prometheus-compatible endpoints in gateway (#7195 ) What this PR does / why we need it: The gateway in non-enterprise mode doesn't expose Prometheus endpoint for alerts/rules so Grafana fails to query and display them in alerting section. Note: When comparing nginxConfig enterprise vs non-enterprise there are a couple of other endpoints which are missing from the gateway config.	3 years ago
Trevor Whitney	afd63c5981	[helm] Add Loki Canary to Helm Chart (#7173 ) What this PR does / why we need it: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. Which issue(s) this PR fixes: Fixes #7019 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Jarjar	9265b8eef5	Helm: Fix ExternalLicence (#7189 ) What this PR does / why we need it: Update references from: ``` .Values.useExternalLicense .Values.externalLicenseName ``` to ``` .Values.enterprise.useExternalLicense .Values.enterprise.externalLicenseName ``` Which issue(s) this PR fixes: Fixes #7171 Signed-off-by: CAJNA Jarod <contact@jarodcajna.fr>	3 years ago
gerhard-strauss-jambit	f29a03833e	Helm: Make automountServiceAccountToken configurable (#7114 ) What this PR does / why we need it: just a simple modifikation to set automountServiceAccountToken on Container Level. ServiceAccount already has this setting	3 years ago
Sandeep Sukhani	7f298ff72f	avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209 ) What this PR does / why we need it: Avoid using bloomfilters for chunks deduplication in tsdb `Stats` calls by avoiding fetching duplicate entries. The idea is to split and align queries by [ObjectStorageIndexRequiredPeriod](`61794710a7/pkg/storage/config/schema_config.go (L47)`) and make each split process chunks with a start time >= start time of the table interval. In other terms, table interval that contains start time of the chunk, owns it. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-12, 11-13. Query with range 6-15 would be split into 6-10, 10-15. query1 would process chunks 5-7, 8-12 and query2 would process chunks 11-13. This check is not applied for the first split so that we do not eliminate any chunks that overlaps the original query intervals but starts at the previous table. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-13, 14-13. Query with range 11-12 should process chunk 8-13 even though its start time <= start time of table we will query for index. The caveat here is that we will overestimate the data we will be processing if the index is not compacted yet since it could have duplicate chunks when RF > 1. I think it is okay since the Stats call is just an estimation and need not be accurate. Removing all the extra processing saves us quite a bit of CPU and memory, as seen from the benchmark comparison between the two implementations: ``` name old time/op new time/op delta IndexClient_Stats-10 187µs ± 0% 34µs ± 1% -82.00% (p=0.008 n=5+5) name old alloc/op new alloc/op delta IndexClient_Stats-10 61.5kB ± 4% 12.5kB ± 2% -79.69% (p=0.008 n=5+5) name old allocs/op new allocs/op delta IndexClient_Stats-10 1.46k ± 0% 0.48k ± 0% -67.28% (p=0.008 n=5+5) ``` Checklist - [x] Tests updated	3 years ago
Owen Diehl	bac52165d3	highly sharded subqueries are ellided in expr.String (#7203 ) This is mostly a quality of life improvement. Highly sharded subqueries can be burdensome to view and they're logged on every request. This PR will only display the first four subqueries in a `Concat{Log,Sample}Expr`, which is easier on the eyes when viewing logs and is more efficient.	3 years ago
Sandeep Sukhani	4e7105d018	add index requests panel to reads/writes dashboard (#7163 ) What this PR does / why we need it: In PR #7154, I refactored the code also to measure the latency on index write requests. This PR adds the index requests panel to the Loki Reads/Writes dashboard.	3 years ago
Enzo Nocera	4f9c85df5d	Guidelines: Update CONTRIBUTING.md (#6757 ) For a new contributor (which I am 🎉), the `CONTRIBUTING.md` file might seem a bit vague in regard to why we need to use the `GOPATH`-scoped Loki repository to contribute. The rationale is now given (If I've missed any other reasons, please tell me). * Fixed wrong level title, * Added information about local workflow when dealing with locally modified dependencies, * Added references to Go 1.18 workspaces. Signed-off-by: Raskyld <enzo@nocera.eu>	3 years ago
Christian Haudum	67b12f2357	Update distributor push tests (#6641 ) Adds a test case for concurrent pushes (which would have spotted a bug in https://github.com/grafana/loki/pull/5248) and a test case for tolerance regarding replication factor. The PR also replaces the mock ring with an actual in-memory kv store ring. And variables and function names were changed to meet Loki/logs terminology. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Ryan Garcia	d8eb8b2a9f	Helm: Adding tolerations value support for tokengen job (#7122 ) What this PR does / why we need it: Adding toleration support for the enterprise tokengen job template. Which issue(s) this PR fixes: Fixes #7091 Signed-off-by: garcia.ryan <garcia.ryan@solute.us>	3 years ago
Roelof Blom	a1d77bf5ea	[helm] Storage: Read support for Azure Object Storage (#7162 ) What this PR does / why we need it: Using Azure Object Storage worked before Helm chart version 3.0. Which issue(s) this PR fixes: Fixes #7090	3 years ago
Danny Kopping	72aec6a38b	Document validation & rate-limit errors (#7145 ) This PR adds documentation for the various validation & rate-limit errors that can occur when sending logs to Loki.	3 years ago
Robert Jacob	f1fbb7bb3f	operator: Write configuration for per-tenant retention (#7201 )	3 years ago
nniehoff	a5c1557d03	docs: Change AWS EKS example eventrouter rbac resources from v1beta1 apis to v1 (#6483 ) RBAC Resource apiVersion `rbac.authorization.k8s.io/v1beta1` was deprecated with Kubernetes 1.22 (see the [deprecation notice](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#rbac-resources-v122)). Now that EKS supports 1.22 the docs need to be updated.	3 years ago
Jeremy Wu	161fcac646	support custom label selector in kubernetes_sd_configs (#5920 ) make promtail scrape logs of pod with specific field or label selector, so that it will watch less logs and make fewer calculations and get a better performance in some scenarios. Signed-off-by: Zhimin Wu <hoozecn@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Ashwanth	3245625c40	ingester_querier: early exit queries on reaching quorum (#7172 ) Currently `forGivenIngesters` and its derivatives don't allow passing down context to the work function as a result of which we are not making use of the tracking functionality provided by `replicationSet.Do` to exit early and relieve pressure on the ingesters. This change ensures that Unary rpc calls receive ctx populated by the replication set tracker to exit early on reaching quorum or max failures. The same cannot be applied to stream rpc calls given how the code is structured at the moment where we are processing the streams outside of the work func	3 years ago
Trevor Whitney	8d552878cd	Bring addition of server config into existing style (#7174 ) What this PR does / why we need it: A recent PR (https://github.com/grafana/loki/pull/7137) was merged to make the server settings more configurable in the helm chart. This PR fixes a few style nits with that PR, mainly: * Name the config block the same as the header in the actual loki config file (in this case, `server` not `serverConfig`) * Use `with`, and wrap around the header in the config file so the entire section is omitted if empty, rather than setting an empty object under the header.	3 years ago
Jennifer Villa	9467a341f6	Update README.md (#7190 ) Typo fix on helm chart readme	3 years ago
Jennifer Villa	4310ffe0bd	Update README.md (#7191 ) Fix typo on README	3 years ago
Dylan Guedes	bf1b5a5ced	Project: Change PR template and CONTRIBUTING guide (#7116 ) What this PR does / why we need it: - Move contributing instructions from the PR template to the CONTRIBUTING guide - Rewrite the contributing guide - Make the PR template slim by getting rid of sections that are optional but often left blank by most contributors/maintainers - We're making the PR template slim as we're using the merged PR body as the merge commit message. Because the comments are often left there by contributors, they're prone to be merged by maintainers, hence making our commit log history dirty with a lot of undesired comments.	3 years ago
Salva Corts	df4461d712	Evenly spread memcached across nodes (#7183 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR enables the [TopologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) for Memcached chunks cache introduced at https://github.com/grafana/jsonnet-libs/pull/832. Now the chunks cache will be evenly spread across nodes but more than one cache can run on the same node. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Karsten Jeschkies	a1dce32d9a	Abide to Mixin linting rules. (#7010 ) Cloud integrations need to pass linting rules such as proper dashboard names. It should be `Data Source` not `datasource`. See also grafana/cloud-onboarding#2228	3 years ago
Owen Diehl	12ebd33f94	TSDB indexBuckets helper logs instead of errors (#7187 ) indexBuckets will return the list of overlapping TSDB buckets for a time range. Historically, this errored when there was no overlap, which is generally impossible. While mutating period configurations as a test (which should never be done in production), I realized this would preemptively error us when there were old TSDB WALs from a bucket that no longer belonged to a TSDB period (i.e. by replacing a `tsdb` index with a `boltdb-shipper` one). This PR changes this to a log line instead as we don't need to error and stop computation; it's better to return an empty overlap instead. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago

1 2 3 4 5 ...

3990 Commits (c5ec1ac27f2d3172d8195b8ebf064cbae1c9b61f) All Branches Search

3990 Commits (c5ec1ac27f2d3172d8195b8ebf064cbae1c9b61f)

All Branches