1894 Commits (35510ba4eb3615da78b15934bf6355b43378f499)

Author	SHA1	Message	Date
Travis Patterson	ef47e0ee8a	Fix errors loading deletes after force-removing partially-completed delete (#7125) ... When sharded deletes that are partially completed are force-removed, everything is removed except the shards that are currently being deleted. The shards being deleted aren't necessarily in numerical order so there's no way to know assume which shards have and haven't been deleted. The Delete store assumed it could look up requests in sequential order and returned empty requests where there were gaps in the stored data. This change takes the `sequence number` from the stored key of the delete requests it finds and uses that to look up details.	3 years ago
Michel Hollands	accb5e1ca2	Log the interval of a delete request (#7120) ... **What this PR does / why we need it**: An interval is generated automatically for a delete request: this is the max size of a shard of a deletion request. This value is now logged. Additionally a setting name that should have been renamed was fixed. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Kaviraj Kanagaraj	2c9fa05c29	feat(loki-canary): Add support to push logs directly to Loki. (#7063) ... Add additional `push` mode to loki-canary, which pushes the logs directly to given Loki URL as it generates logs. The real function of Loki Canary is to act like a tenant and help us know the whether Loki installation is working as perceived by a real tenant. Main rationale for this additional push mode is to make canary more standalone without needing for `promtail` (or `grafana-agent`) to scrape it's logs and send to loki, with this change, Loki canary happily tests Loki behavior without needing any other dependencies. **NOTES**: 1. If you run Loki behind any proxy that has different authorization policies to READ or WRITE to Loki, then important change that canary operator need to be aware of it, now the user credentials they pass via `-user` and `-pass` to access loki endpoints need to have both `read' and `write` permissions (previously canary just query the logs where as promtail is the one pushes the logs, so just READ permissions was sufficient). 2. There will be follow up PR(s) to cleanup, particularly `reader` and `comparitor` component in terms of logging with proper logger. Rationale is to keep the changes small per PR to make it easy to review. 3. This changes were tested it in one of the internal Loki dev cell. 4. **This PR is a no-op if this new `push` flag is disabled (it's disabled by default)** Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Mohamed-Amine Bouqsimi	3d4788fb51	Add ability for Ruler to remote write to multiple clients (#6823)	3 years ago
Michel Hollands	08cf2153d5	Add sequence num to log output for delete requests (#7087) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com> **What this PR does / why we need it**: Print the sequence number of the delete request so the messages are not all the same. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Periklis Tsirakidis	1a41682bb0	Fix internal server initialization (#7085)	3 years ago
Periklis Tsirakidis	c30fb9450b	Add support for custom internal server (#7069) ... Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Adam	eaf7f34f49	bugfix:add cases.NoLower option (#7052) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: see #7048 if we don't apply cases.NoLower, it will upper first alphabet lower the others ```go package main import ( "fmt" "golang.org/x/text/cases" "golang.org/x/text/language" "strings" ) func main() { source := "testAPI" fmt.Println(cases.Title(language.Und).String(source)) // out: Testapi fmt.Println(cases.Title(language.Und, cases.NoLower).String(source)) // out: TestAPI fmt.Println(strings.Title(source)) // out: TestAPI } ``` **Which issue(s) this PR fixes**: **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: wujw39640 <wujw39640@hundsun.com> Signed-off-by: wujunwei <wjw3323@live.com> Co-authored-by: wujw39640 <wujw39640@hundsun.com>	3 years ago
Ashwanth	53e01a795a	tsdb: build tsdb from tenantheads (#7036) ... **What this PR does / why we need it**: refactors tsdb manager code to expose a new method `BuildFromHead` which comes in handy for building tsdb from tenant heads after rotation. Currently we build it by recovering head from WAL even when we have access to the in-mem tenant heads. **Special notes for your reviewer**: using `loki_tsdb_head_rotations_*` metric to track loop failures which could be caused either by rotation failures or if rotation is blocked by prev head failing to build **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Michel Hollands	34c2c0c4a9	Update metrics descriptions (#6982) ... **What this PR does / why we need it**: The docs team reviewed and fixed the descriptions of the deletion metrics added to the documentation in https://github.com/grafana/loki/pull/6971. The fix is in a separate PR https://github.com/grafana/loki/pull/6976. This PR fixes that same description in the code. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Adam	d69d3b664c	Deprecation Fix: Replace strings.Title with cases.Title().String() (#7048) ... Signed-off-by: wujunwei <wjw3323@live.com> <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: [Golang 1.18 deprecates the `strings.title function`](https://github.com/golang/go/issues/48367) use `cases.Title(language.Und, cases.NoLower).String()` instead. **Which issue(s) this PR fixes**: **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: wujw39640 <wujw39640@hundsun.com>	3 years ago
Trevor Whitney	c0eb73c34a	Enable build in env w/o internet or GCP creds (#7029) ... **What this PR does / why we need it**: The build is currently dependent upon you building in an environment with internet connectivity and GCP credentials. This PR fixes the few parts of our code and test that rely on these external dependencies so it can be built in an environment without internet or GCP credentials (such as when building a [nix flake](https://github.com/grafana/loki/pull/7020)) **Which issue(s) this PR fixes**: Fixes #7028	3 years ago
Ashwanth	5ea38fbd51	tsdb: build rotated heads asynchronously (#6956) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: Currently `Append` call to the head has to wait for block creation to complete when rotating head. And since we rotate head every 15m (default period), this would result in latency spikes at these times. This PR moves the TSDB build step to an async routine. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
李国忠	3296c6dbe8	[new feature] logql engine support exec `vector(0)` grama (#7023) ... **What this PR does / why we need it**: logql engine support exec `vector(0)` grama. new PR of :https://github.com/grafana/loki/pull/7007 **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: test passed. test1: vectorExpr only  test2: binExpr and vectorExpr  **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	bc16450349	Revert "Hide duplicates caused by stream sharding (#7005)" (#7031) ... @owen-d had a suggestion that was dramatically dimpler. Revert this whole thing and the simplified PR will be forthcoming	3 years ago
李国忠	25cc9402b7	fix test increase timeout (#7022) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: increase timeout, fix test fail. ``` --- FAIL: TestInstanceFor (0.01s) downstreamer_test.go:205: Error Trace: downstreamer_test.go:205 downstreamer_test.go:256 Error: lock couldn't be acquired Test: TestInstanceFor FAIL ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	6f4671a572	[new feature] syntax: support "vector" grama (#7007) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: Implement vector() function. The "vector function" feature is divided into multiple PRs to avoid excessively large PRs resulting in too long review time. **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: The test for syntax has passed  <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
sardismart	821301b95f	azblob: Make azblob support private cloud endpoint suffix (#6842) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: The endpoint suffix of private cloud(azure stack hub) azblob is different from Azure. The loki configuration has already support customize endpoint suffix but not use when select blob/container URL. This PR introduce following change: * Consider endpoint suffix from configuration when select blob or container URL format **Which issue(s) this PR fixes**: Fixes #6863 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	a98b7ebaf4	Hide duplicates caused by stream sharding (#7005) ... As streams are sharded and resharded across different ingesters, the likelihood of duplicate logs being ingested increases dramatically. This removes duplicates at query time by dynamically removing the `__shard_stream__` label for both log and sample queries so the `stream hash` used by the `MergeEntryIterator` will match when doing comparisons for duplicates. The `stream hash` is created in either the `store` or `ingester instance` at query time so the labels are removed there. Additionally, the `MergeEntryIterator` has been modified to remove duplicates in a stream rather than just across streams. For completeness, `__shard_stream__` has also been removed from Label and Series Queries.	3 years ago
李国忠	562ccd6932	[bugfix] fix flush.go nil pointer panic (#6993) ... **What this PR does / why we need it**: Fix a panic when reporting chunk statistics. The chunk would already be garbage-collected as it is marked as flushed before reporting its statistics. This fix it by first reporting its statistics and only then marking it as flushed. Panic description: ```go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x177d158] goroutine 14455 [running]: github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Size(0xc00441c750?) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/chunkenc/memchunk.go:611 +0x18 github.com/grafana/loki/pkg/ingester.(*Ingester).reportFlushedChunkStatistics(0xc00087a000, 0xc00f8efe80, 0xc0519d0870, {0x36b7008, 0xc01a593860}, {0x36b7008, 0xc01a5938c0}, {0x307e2f9, 0x7}) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/ingester/flush.go:395 +0x2f2 ```	3 years ago
Dylan Guedes	bba6c14459	Loki: Modify ingesters to return rate-limited streams to distributors (#6977) ... **What this PR does / why we need it**: Modify ingesters to return a list of rate-limited streams on its reply to `(*distributor).Push`. This is done by using the gRPC details feature. I've updated the tests and apparently, the returned error message is exactly the same so this shouldn't be a breaking change. Also modifies `(*stream).Push` to return a list of all entries with errors instead of returning the final gRPC error. This will allow us in the future to easily return to distributors the indices of all reject entries, so that distributors won't have to always retry all entries. This work is necessary for the new stream sharding feature.	3 years ago
Owen Diehl	30db6c120e	[TSDB] 600MB max per shard in query planning (#7004) ... This PR gives a little better understanding into shard query planning and targets a range of `(300MB,600MB]` per shard. Doubling this also allows us to halve query parallelism configs and still schedule the same amount of work in parallel, which should make for easier schema upgrades as the parallelism differences between TSDB and prior period configs will be less. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
afayngelerindbx	36109eb09a	compactor: prefetch existing compacted user index files in parallel (#6803) ... **What this PR does / why we need it**: Before this change, these files are fetched while [holding a table global lock](a1e75da970/pkg/storage/stores/shipper/index/compactor/table_compactor.go (L195)), serializing fetches. I chose the approach of prefetching index files over attempting to change the locking pattern to allow parallel, on-demand fetches because the complexity of parallelizing on-demand fetches outweighed the benefits of avoiding extra fetches. More specifically, this patch trades off fetching idle, non-compactable index files in favor of avoiding blocking during compactions. The extra fetches are parallelized and so should incur a negligible wall clock penalty even for large numbers of tenants(e.g. 1k idle tenants at 50 threads and 500ms to fetch is ~10s). **Which issue(s) this PR fixes**: This should help with #6775, without completely resolving it. **Checklist** This change does not introduce any settings, metrics etc. It should be transparent from a user perspective. Therefore, I don't believe it merits documentation/changelog updates. - [x] Tests updated	3 years ago
Christian Haudum	177f37700a	Make TSDB store singleton resettable (#6950) ... This changes allows downstream projects to reset the TSDB store singleton. This must only be called in test cases where a new store instances cannot be explicitly created. Also add a `nil` check to the reset function for the BoltDB shipper singleton instance. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Vladyslav Diachenko	615941234a	Logql Analyzer (#6171) ... * created logql-debugger page and API endpoint that returns information for each stage of the query * added docker-compose setup for local testing * refactored analyzer and added unit tests * removed docs page to release it in a separate PR when back-end API is published * changes structure of results object in response to return origin log line needed for case when only stream selector is sent in query * merged the latest main and resolved conflicts * added endpoint for readiness probe and configured CI * fixed drone.yml discrepancy * fixed method signature * fixed path to Dockerfile * added clean step	3 years ago
Ed Welch	6d54505485	Loki: add `__line__` and `__timestamp__` to label_format (#6983) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: We have the `__line__` and `__timestamp__` functions currently available in the `line_format` stage but not in the `label_format` stage. This PR fixes that. Specifically `__timestamp__` can be valuable in a label because you can do things like ``` label_format hour_of_day=`{{__timestamp__ | date "15"}}` ``` To be able to conditionaly select log lines based on the hour of the day they were created, this can be useful for conditional alerting as an example. **Which issue(s) this PR fixes**: Fixes #6402 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
李国忠	e3cd312c37	[fix panic] s3: fix object nil point (#6970) ... **What this PR does / why we need it**: fix a s3 panic. ```go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14fec7b] goroutine 6271245 [running]: github.com/grafana/loki/pkg/storage/chunk/client.(*client).getChunk(_, {_, _}, _, {{0xb130cf997b272f99, {0xc05077e6a0, 0x1b}, 0x182d1aefb2d, 0x182d1b47984, 0x9fab6388}, ...}) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/storage/chunk/client/object_client.go:171 +0x2fb ```	3 years ago
afayngelerindbx	4b5b16569d	compactor: adds tunable to control upload parallelism (#6817) ... **What this PR does / why we need it**: This can help with compaction latency. I patched the compactor to set this to 100. **Which issue(s) this PR fixes**: Helps with #6775 **Checklist** - [x] Documentation added	3 years ago
Dylan Guedes	f6f49ec6bb	Return Querier/QueryTimeout YAML configuration. (#6972) ... What this PR does / why we need it: On PR #6835 I mistakenly removed the querier:query_timeout YAML configuration. This PR returns it and uses it if it is still configured while logging a warning message to alert users of the upcoming changes.	3 years ago
afayngelerindbx	1e1c2470b5	compactor: prioritize newer tables when compacting. add operator flags to compac… (#6804) ... * prioritize newer tables when compacting. add operator flags to compact time ranges of tables Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * reroll doc and PR changes after rebase Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Dylan Guedes	5d8c48bd10	Loki: Implement stream sharding (#6952) ... * introduce 'StreamSharder' interface to decouple sharding from managing shards * make the streamsharder take a whole stream instead of just and ID * Implement stream sharding usage. * move streamShard to distributor * Implement the stream sharder and shard iter (#6941) * Modify sharding to not duplicate all entries for all shards (#6942) * Modify sharding to not duplicate all entries for all shards. * Rename ShardIter and get rid of NextShardId method. * Get rid of ShardIter and ShardStats. * Implement a StreamSharder mock and use it in tests. * Rewrite stubbing and remove flags. * Add auto-splitting. * Only shard stream when approppriate. * Make sharding mode an enum. * Bench shardStream function. * Modify shard comparison to be <= 1. * Add a guard clause for when lowerbound>upperbound. * Apply suggestions from code review Avoid using `fmt.Sprintf`. Co-authored-by: Danny Kopping <danny.kopping@grafana.com> * Update pkg/distributor/distributor.go Rename metric name. Co-authored-by: Danny Kopping <danny.kopping@grafana.com> * Avoid double-negative by Danny's suggestion. * Move TODO to its right place. * Log scenario where upperbound < lowerbound. * Reuse compiled regex. * Add docstring explaining why we use max(shards*2, 2). * make sharder use a RWMutex * simplify sharding config * replace doc string * Revert "Add auto-splitting" for standalone PR. This reverts commit e3886a6210. * shard loop refactor * move shard creation into it's own function * fix after refactor * make config more explicit and document it * reduce allocations * review feedback Co-authored-by: Travis Patterson <travis.patterson@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835) ... What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Travis Patterson	e15da437f0	Shard deletes requests based on time ranges (#6903) ... * Refactor the store API to be in terms of DeleteRequest * backfill remove tests * backfill GetAll tests * Make the delete request store deal in groups of requests instead of single ones * Make cancelation aware of request groups * remove cancelation deadline * shard deletes based on a query parameter * merge request group on gets * global configuration for max query range * sort delete requests by start time so batches are grouped to similar chunks * Fix status when some deletes haven't happened yet * fix tests * Review feedback * review feedback * make max_interval > deletion interval a validation error * list partially completed deletes * list partially completed deletes * fix flaky test * clarify naming * disallow cancelation of partial deletes unless the 'force' flag is passed * more comment specificity for the delete request hash	3 years ago
Danny Kopping	449dd81981	Restoring metrics removed by #6957, simplifying implementation (#6962) ... Adding all available metrics from Go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	2321da49b5	Upgrade prometheus/client_golang and reconfigure to restore `go_sched.*` metrics (#6957) ... * Upgrading prometheus/client_golang to restore go_sched.* metrics These metrics were removed unintentionally in #6403 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test to match interface correctly Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	87e20d9399	Setting sane defaults for ruler in SSD "read" mode (#6551) ... Also removing support for configdb, since this is not supported but was the default storage type Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Owen Diehl	c340f3e202	[TSDB] disables bloom filters for tsdb index on ingesters (#6939) ... * disables bloom filters for tsdb index on ingesters Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * faillint exception for proto sync/atomic Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Susana Ferreira	3f3f4eda1d	Fix topk and bottomk operations with int <= 0 (#6937) ... * Fix topk and bottomk operations with int <= 0 * Fix VectorAggregationExpr String method * Update rangemapper_test expression * Add tests * Update CHANGELOG.md	3 years ago
Dylan Guedes	2d7ef07494	Fix `increment_duplicate_timestamps` for multiple dupped timestamps (#6924)	3 years ago
Owen Diehl	72d8627a55	log query which causes panic in logql.clone (#6934) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Michel Hollands	25e8365d05	Improve error message (#6913) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Owen Diehl	f470943433	reintroduces now deprecated+noop deletion configs for compatibility (#6917) ... * reintroduces now deprecated+noop deletion configs for compatibility Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * better deprecation logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes import Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Simon	953c36e3ed	Remove unnecessary spin up of mock (#6911) ... This removes the unnecessary spinning up of a mock for TestDoubleRegistration. The client itself is not used at all and that causes it to be stopped before it is actually started. Example flake: https://drone.grafana.net/grafana/loki/14724/2/5	3 years ago
Christian Simon	ee2fe6514d	Propagate error detail properly in all cases (#6890) ... The error handling was missing appending the error detail properly in all cases. Fixes #6265	3 years ago
Kaviraj Kanagaraj	693f9d7070	fix(stats_collector): Do not collect stats for failed requests. (#6878) ... * fix(stats_collector): Do not collect stats for failed requests. When any type of query fails, do not collect statistics. Rationale being when core http handler returns error, it's reponse will be nil and there won't be any statistics to collect and log. Fixes: 6231 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Sandeep Sukhani	3c5fdb749a	add metric for measuring overall index query latency (#6880)	3 years ago
Owen Diehl	3cba86faf2	[bugfix] fixes issues sharding series by storageref vs fingerprint in stripeSeries (#6882) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879) ... * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	3 years ago
Travis Patterson	afedabf0e9	Remove unused code (#6872)	3 years ago
Travis Patterson	983ab80e7c	Add delete api validations (#6860) ... * break out the middleware * Add validation to the API - Time must be in RFC3339 or a 10-digit-unix-seconds timestamp - Start time must always exist * lint * docs * improve tests * cleanup * cleanup * access runtime config via function on validation	3 years ago