apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Jacob	766d4b7b5d	operator: Configure default node affinity for all pods (#7000 )	3 years ago
Trevor Whitney	5ca7c29e2c	Helm: test migration from `grafana/loki-simple-scalable` (#6986 ) What this PR does / why we need it: Verify the upgrade path from `grafana/loki-simple-scalable` -> `grafana/loki` works. Fixes #6874	3 years ago
李国忠	25cc9402b7	fix test increase timeout (#7022 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: increase timeout, fix test fail. ``` --- FAIL: TestInstanceFor (0.01s) downstreamer_test.go:205: Error Trace: downstreamer_test.go:205 downstreamer_test.go:256 Error: lock couldn't be acquired Test: TestInstanceFor FAIL ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	6f4671a572	[new feature] syntax: support "vector" grama (#7007 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: Implement vector() function. The "vector function" feature is divided into multiple PRs to avoid excessively large PRs resulting in too long review time. Which issue(s) this PR fixes: Fixes #6946 Special notes for your reviewer: The test for syntax has passed ![image](https://user-images.githubusercontent.com/9583245/187588436-7c2c401f-1d88-4bff-a41a-965d36454ebe.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
sardismart	821301b95f	azblob: Make azblob support private cloud endpoint suffix (#6842 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The endpoint suffix of private cloud(azure stack hub) azblob is different from Azure. The loki configuration has already support customize endpoint suffix but not use when select blob/container URL. This PR introduce following change: * Consider endpoint suffix from configuration when select blob or container URL format Which issue(s) this PR fixes: Fixes #6863 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	a98b7ebaf4	Hide duplicates caused by stream sharding (#7005 ) As streams are sharded and resharded across different ingesters, the likelihood of duplicate logs being ingested increases dramatically. This removes duplicates at query time by dynamically removing the `__shard_stream__` label for both log and sample queries so the `stream hash` used by the `MergeEntryIterator` will match when doing comparisons for duplicates. The `stream hash` is created in either the `store` or `ingester instance` at query time so the labels are removed there. Additionally, the `MergeEntryIterator` has been modified to remove duplicates in a stream rather than just across streams. For completeness, `__shard_stream__` has also been removed from Label and Series Queries.	3 years ago
Karsten Jeschkies	c23e541fe7	Generate Helm chart README. (#7012 ) What this PR does / why we need it: The Helm CI would break because the Helm chart README was not generated. This change * updates the README * introduces a filter that will only run the Helm CI on changes to the Helm chart directory * removes trailing whitespace * corrects the Minio chart repository	3 years ago
Karsten Jeschkies	1c3f5d0985	Helm: Upgrade Minio chart dependency to 4.0.12. (#6974 ) What this PR does / why we need it: Changes to a supported Minio chart. Minio is also enabled by default. Which issue(s) this PR fixes: Fixes #6973 Checklist - [x] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	562ccd6932	[bugfix] fix flush.go nil pointer panic (#6993 ) What this PR does / why we need it: Fix a panic when reporting chunk statistics. The chunk would already be garbage-collected as it is marked as flushed before reporting its statistics. This fix it by first reporting its statistics and only then marking it as flushed. Panic description: ```go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x177d158] goroutine 14455 [running]: github.com/grafana/loki/pkg/chunkenc.(MemChunk).Size(0xc00441c750?) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/chunkenc/memchunk.go:611 +0x18 github.com/grafana/loki/pkg/ingester.(Ingester).reportFlushedChunkStatistics(0xc00087a000, 0xc00f8efe80, 0xc0519d0870, {0x36b7008, 0xc01a593860}, {0x36b7008, 0xc01a5938c0}, {0x307e2f9, 0x7}) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/ingester/flush.go:395 +0x2f2 ```	3 years ago
Dylan Guedes	bba6c14459	Loki: Modify ingesters to return rate-limited streams to distributors (#6977 ) What this PR does / why we need it: Modify ingesters to return a list of rate-limited streams on its reply to `(distributor).Push`. This is done by using the gRPC details feature. I've updated the tests and apparently, the returned error message is exactly the same so this shouldn't be a breaking change. Also modifies `(stream).Push` to return a list of all entries with errors instead of returning the final gRPC error. This will allow us in the future to easily return to distributors the indices of all reject entries, so that distributors won't have to always retry all entries. This work is necessary for the new stream sharding feature.	3 years ago
Owen Diehl	30db6c120e	[TSDB] 600MB max per shard in query planning (#7004 ) This PR gives a little better understanding into shard query planning and targets a range of `(300MB,600MB]` per shard. Doubling this also allows us to halve query parallelism configs and still schedule the same amount of work in parallel, which should make for easier schema upgrades as the parallelism differences between TSDB and prior period configs will be less. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Kaviraj Kanagaraj	5a6d09f83e	doc(community): Update community call timings. (#7002 )	3 years ago
afayngelerindbx	36109eb09a	compactor: prefetch existing compacted user index files in parallel (#6803 ) What this PR does / why we need it: Before this change, these files are fetched while [holding a table global lock](`a1e75da970/pkg/storage/stores/shipper/index/compactor/table_compactor.go (L195)`), serializing fetches. I chose the approach of prefetching index files over attempting to change the locking pattern to allow parallel, on-demand fetches because the complexity of parallelizing on-demand fetches outweighed the benefits of avoiding extra fetches. More specifically, this patch trades off fetching idle, non-compactable index files in favor of avoiding blocking during compactions. The extra fetches are parallelized and so should incur a negligible wall clock penalty even for large numbers of tenants(e.g. 1k idle tenants at 50 threads and 500ms to fetch is ~10s). Which issue(s) this PR fixes: This should help with #6775, without completely resolving it. Checklist This change does not introduce any settings, metrics etc. It should be transparent from a user perspective. Therefore, I don't believe it merits documentation/changelog updates. - [x] Tests updated	3 years ago
Gerard Vanloo	1bb535a730	operator: Fixing logcli pod image value for operator addons (#6997 )	3 years ago
Christian Haudum	177f37700a	Make TSDB store singleton resettable (#6950 ) This changes allows downstream projects to reset the TSDB store singleton. This must only be called in test cases where a new store instances cannot be explicitly created. Also add a `nil` check to the reset function for the BoltDB shipper singleton instance. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Callum Styan	da6fd01444	[docs] document logfmt pipeline stage (#6238 ) I noticed earlier while writing a design doc that the `logfmt` pipeline stage is not documented. It was originally added here: https://github.com/grafana/loki/pull/4346 I've also fixed a minor typo in the json pipeline stage docs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Trevor Whitney	c83b6c543d	Helm: Add helm ci (#6988 ) Tests Helm chart when changed as part of PR CI process.	3 years ago
Dylan Guedes	c6dc0d2649	Docs: Fix default max_chunk_age (#6996 ) What this PR does / why we need it: Fix a doc stating a wrong `max_chunk_age` value. Which issue(s) this PR fixes: Fixes #6919	3 years ago
Vladyslav Diachenko	615941234a	Logql Analyzer (#6171 ) * created logql-debugger page and API endpoint that returns information for each stage of the query * added docker-compose setup for local testing * refactored analyzer and added unit tests * removed docs page to release it in a separate PR when back-end API is published * changes structure of results object in response to return origin log line needed for case when only stream selector is sent in query * merged the latest main and resolved conflicts * added endpoint for readiness probe and configured CI * fixed drone.yml discrepancy * fixed method signature * fixed path to Dockerfile * added clean step	3 years ago
Owen Diehl	9e84648f3e	querier deployment rollingupdate works as a percentage to allow faste… (#6990 ) …r rollouts in large environments	3 years ago
Trevor Whitney	cffcb468f2	Remove deprecated helm chart source code (#6978 ) What this PR does / why we need it: All of these helm charts have been moved out of this repo to `grafana/helm-charts`. As we are now moving `grafana/loki` back into this repo, this deprecated charts are confusing. I have opted to include `README`s with the deprecation notice, as well as a link to the current sha of `main` so that people can easily find the old source code.	3 years ago
Ed Welch	bf5a25dd5b	Docs: update `split_queries_by_interval` documentation to have the correct explanation for how to disable. (#6715 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: I noticed when responding to #5123 the docs did not correctly explain how to disable splitting queries by time. I searched through the code and confirmed `0` is the correct value to disable this feature Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Ed Welch	6d54505485	Loki: add `__line__` and `__timestamp__` to label_format (#6983 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: We have the `__line__` and `__timestamp__` functions currently available in the `line_format` stage but not in the `label_format` stage. This PR fixes that. Specifically `__timestamp__` can be valuable in a label because you can do things like ``` label_format hour_of_day=`{{__timestamp__ \| date "15"}}` ``` To be able to conditionaly select log lines based on the hour of the day they were created, this can be useful for conditional alerting as an example. Which issue(s) this PR fixes: Fixes #6402 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979 ) Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977 ) What this PR does / why we need it: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. Which issue(s) this PR fixes: Fixes #5978 Special notes for your reviewer: n/a Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	e3cd312c37	[fix panic] s3: fix object nil point (#6970 ) What this PR does / why we need it: fix a s3 panic. ```go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14fec7b] goroutine 6271245 [running]: github.com/grafana/loki/pkg/storage/chunk/client.(*client).getChunk(_, {_, _}, _, {{0xb130cf997b272f99, {0xc05077e6a0, 0x1b}, 0x182d1aefb2d, 0x182d1b47984, 0x9fab6388}, ...}) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/storage/chunk/client/object_client.go:171 +0x2fb ```	3 years ago
Kim Nylander	a12b0803f0	[Doc] Fixed comma and missing "of" in observability.md (#6976 ) What this PR does / why we need it: Correcting two mistakes that @osg-grafana found in PR#6971. Checklist - [X] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
afayngelerindbx	4b5b16569d	compactor: adds tunable to control upload parallelism (#6817 ) What this PR does / why we need it: This can help with compaction latency. I patched the compactor to set this to 100. Which issue(s) this PR fixes: Helps with #6775 Checklist - [x] Documentation added	3 years ago
Trevor Whitney	11caa49221	Support helm migration from current grafana/loki chart (#6958 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: These are changes to the helm chart and documentation need to migration from the current version of the `grafana/loki` helm chart to the new one in this repo. The current plan (as outlined in the README) is to support 3 upgrade paths: 1. From the existing `grafana/loki` chart using filesystem storage. This path will result in a single binary deployment using local filesystem storage with the new chart. This PRs includes changes to make sure this path works. 2. ~From the existing `grafana/loki` chart using object storage. This path will result in an SSD deployment with 1 read and 1 write), and has yet to be tested/documented.~ (Edit: I no longer think we need to support this use case) 3. From the existing `grafana/loki-simple-scalable` chart using object storage. This path will result in an SSD reployment with 3 read and 3 write replicas, and has yet to be tested/documented. By clarifying the three modes, and that single binary will only support a single, filesystem backed instance, it made possible some refactors. For example, the user no longer needs to specify a `deploymentMode`, and we were able to reorganize the templates a bit into a flatter folder structure. Checklist - [X] Documentation added - [ ] Tests updated (will do in subsequent PR) - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
dmitri-lerko	e06c0790ba	promtail: GCPLog fixing `failed to receive pubsub messages` error (#6965 ) 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: Upgrades PubSub to the latest. Myself and other users observe frequent disconnects from PubSub which was addressed upstream. High load production testing has proven that this fixes the problem. As an extra bonus, PubSub seems to ingest ~33% faster with the latest library. Which issue(s) this PR fixes: Fixes #6505 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ursula Kallio	8da290cd2d	Use backticks for code syntax. (#6967 ) * Use backticks for code syntax. * Fix for real. :)	3 years ago
Dylan Guedes	f6f49ec6bb	Return Querier/QueryTimeout YAML configuration. (#6972 ) What this PR does / why we need it: On PR #6835 I mistakenly removed the querier:query_timeout YAML configuration. This PR returns it and uses it if it is still configured while logging a warning message to alert users of the upcoming changes.	3 years ago
Michel Hollands	3839522eb7	Add deletion metrics to the documentation (#6971 ) * Add deletion metrics Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix metric types for some entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add missing dot at end of line Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
afayngelerindbx	1e1c2470b5	compactor: prioritize newer tables when compacting. add operator flags to compac… (#6804 ) * prioritize newer tables when compacting. add operator flags to compact time ranges of tables Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * reroll doc and PR changes after rebase Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Dylan Guedes	5d8c48bd10	Loki: Implement stream sharding (#6952 ) * introduce 'StreamSharder' interface to decouple sharding from managing shards * make the streamsharder take a whole stream instead of just and ID * Implement stream sharding usage. * move streamShard to distributor * Implement the stream sharder and shard iter (#6941) * Modify sharding to not duplicate all entries for all shards (#6942) * Modify sharding to not duplicate all entries for all shards. * Rename ShardIter and get rid of NextShardId method. * Get rid of ShardIter and ShardStats. * Implement a StreamSharder mock and use it in tests. * Rewrite stubbing and remove flags. * Add auto-splitting. * Only shard stream when approppriate. * Make sharding mode an enum. * Bench shardStream function. * Modify shard comparison to be <= 1. * Add a guard clause for when lowerbound>upperbound. * Apply suggestions from code review Avoid using `fmt.Sprintf`. Co-authored-by: Danny Kopping <danny.kopping@grafana.com> * Update pkg/distributor/distributor.go Rename metric name. Co-authored-by: Danny Kopping <danny.kopping@grafana.com> * Avoid double-negative by Danny's suggestion. * Move TODO to its right place. * Log scenario where upperbound < lowerbound. * Reuse compiled regex. * Add docstring explaining why we use max(shards2, 2). make sharder use a RWMutex * simplify sharding config * replace doc string * Revert "Add auto-splitting" for standalone PR. This reverts commit `e3886a6210`. * shard loop refactor * move shard creation into it's own function * fix after refactor * make config more explicit and document it * reduce allocations * review feedback Co-authored-by: Travis Patterson <travis.patterson@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835 ) What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Travis Patterson	e15da437f0	Shard deletes requests based on time ranges (#6903 ) * Refactor the store API to be in terms of DeleteRequest * backfill remove tests * backfill GetAll tests * Make the delete request store deal in groups of requests instead of single ones * Make cancelation aware of request groups * remove cancelation deadline * shard deletes based on a query parameter * merge request group on gets * global configuration for max query range * sort delete requests by start time so batches are grouped to similar chunks * Fix status when some deletes haven't happened yet * fix tests * Review feedback * review feedback * make max_interval > deletion interval a validation error * list partially completed deletes * list partially completed deletes * fix flaky test * clarify naming * disallow cancelation of partial deletes unless the 'force' flag is passed * more comment specificity for the delete request hash	3 years ago
Salva Corts	7961ef6a4e	Evenly spread distributors across available nodes (#6964 ) * Evenly spread distributors across available nodes * Add note about reliability implications	3 years ago
liukch	59be5e036c	Fix Loki distributor in production Nomad job and docs. (#6963 )	3 years ago
Trevor Lauder	e125252ee7	lambda-promtail: allow numbers on LB name regex (#6961 )	3 years ago
Danny Kopping	449dd81981	Restoring metrics removed by #6957 , simplifying implementation (#6962 ) Adding all available metrics from Go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	2321da49b5	Upgrade prometheus/client_golang and reconfigure to restore `go_sched.` metrics (#6957 ) Upgrading prometheus/client_golang to restore go_sched.* metrics These metrics were removed unintentionally in #6403 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test to match interface correctly Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Alex Close	288e644894	docs: Fixed examples within Loki and Promtail docs (#6591 ) * Updated the promtail install example to both fix the Daemonset yaml file and provide a more concrete example for deploying to kubernetes * Removed the sidecar documentation for promtail install as it was both broken and didn't do what it said it did * Updated the simple scalable chart docs to include an example of using a overrides values file and to point to a working example of deploying promtail in a k8s cluster * Updated promtail and loki SSD install docs based on PR feedback * Update docs/sources/clients/promtail/installation.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Trevor Whitney	c6f7a41225	Bring helm chart back into Loki repo (#6829 ) What this PR does / why we need it: This is the first step of the process to bring the helm chart back into the Loki repo. We currently have ~7 public helm charts. This creates a lot of confusion for the community on which ones to start with, and is also a lot of additional surface area that is hard for Loki contributors to keep track of. By bringing the helm chart that we use ourselves back into the Loki repo, it will be a more clear starting point for helm users. It will also increase visibility for Loki contributors into helm PRs and issues. This will _not_ affect where the helm chart is published. It will still be published to the `grafana/helm-charts` following the same pattern as the Mimir helm chart. The helm chart that we are currently using and most actively supporting is the `loki-simple-scalable` chart. Part of this effort is to rename that chart to just `loki`, which means it needs to be expanded to support the single binary deployment topology currently deployed by the existing `loki` helm chart. This PR is the first attempt to merge those two charts. This chart currently works for a basic install of each deployment topology. There is more work to be done before we publish it. This work will include: * Making sure there is a migration path from the existing `loki` chart to this new one * Making sure there is a migration path from the existing `loki-simple-scalable` chart to this new one * Making sure the single binary deployment works in both `filesystem` and minio-backed multi-replica mode * Migrating CI from `grafana/helm-charts` for testing helm chart changes * Integrating with the github action in the `grafana/helm-charts` repo (this will publish it, so we will tackle this last) A question I have for reviewers is what we should name each deployment mode (single binary vs. simple scalable)? We have received feedback from the community that some did not think "simple scalable" was suitable for production loads since it had "simple" in the name. This might be an opportunity to adjust that perception. Some options I've thought of include: * `single-binary` and `simple-scalable` (currently how it's coded) * `basic` and `scalable` (leaving open the option for `advanced` if we ever choose to support micro-services topology in this chart as well) * `ha`, `redundant` or similar vs. not Fixes #6873	3 years ago
Jonathan	1b6d0bf3a0	feat: add kms ecryption (#6926 )	3 years ago
Danny Kopping	87e20d9399	Setting sane defaults for ruler in SSD "read" mode (#6551 ) Also removing support for configdb, since this is not supported but was the default storage type Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Owen Diehl	c340f3e202	[TSDB] disables bloom filters for tsdb index on ingesters (#6939 ) * disables bloom filters for tsdb index on ingesters Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * faillint exception for proto sync/atomic Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Ed Welch	ded2614823	Remove comparisons page, it's not very acurate and the information is outdated. (#6938 ) Signed-off-by: Edward Welch <edward.welch@grafana.com> Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Salva Corts	83081653d1	Add operational guide for autoscaling (#6801 ) * Add operational guide for autoscaling * Add promql tags to code snippets * Apply feedback from Karen * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * CR feedback * Add note about needing to use the query-scheduler. * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * CR changes * Apply suggestions from code review Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/operations/autoscaling_queriers.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * CR feedback Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>	3 years ago
André Tadeu de Carvalho	9694776537	docs:Add documentation for TLS configuration parameters for HTTP and gRPC (#6933 ) * Add documentation for TLS configuration parameters for HTTP and gRPC. * Removed possible values from client-auth parameters.	3 years ago

1 2 3 4 5 ...

3865 Commits (766d4b7b5d2475d9424ca03e2a746e4c0e46660f) All Branches Search

3865 Commits (766d4b7b5d2475d9424ca03e2a746e4c0e46660f)

All Branches