apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Totalus	01f0ded7fc	Update swift_object_client.go : Fetch all objects in List() function (#9463 ) What this PR does / why we need it: Allow the Swift Client `List()` function to return all objects that match the query. Which issue(s) this PR fixes: Fixes #9360 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205 )	2 years ago
李国忠	98936437a9	[opt] s3: add s3 client http timeout for GetObject or PutObject (#8951 ) What this PR does / why we need it: add s3 client http timeout for GetObject or PutObject, why？ The flush queue has been increasing. This is a difficult problem. We have been investigating this problem for half a year. When the querier has a large query, the load on the s3 server is too high and the number of connections is full. It will cause the flush queue of the ingester to increase continuously and cannot be reduced. When this problem occurs, after we restart the s3 server, the flush queue will quickly become 0. We checked with our s3 (object storage compatible with the s3 protocol) team and found that it was because we used the s3 v1 sdk. The v1 sdk did not configure timeout by default, and it would wait indefinitely without configuration. The s3 v2 sdk defaults to a timeout of 30s, which only causes write failures and does not cause loki to get stuck. Which issue(s) this PR fixes: Fixes #<issue number> At present, as long as there are large logql queries such as 30 days, I must restart the s3 server, otherwise our ingester queue will continue to increase. We hope that we can break the stuck go routine after the 5m timeout occurs. Special notes for your reviewer: ingester flush queue length up 🔝 ![image](https://user-images.githubusercontent.com/9583245/228786401-cf07dc32-dcc0-428f-9558-561b588ea2c4.png) big query, 30day logql or something happen ![image](https://user-images.githubusercontent.com/9583245/228789991-57185bc3-e628-45a5-82df-e4682590b786.png) pprof goroutine list profile snapshot ![image](https://user-images.githubusercontent.com/9583245/228787732-6170b8ec-db86-4ca7-8ea0-a7f19b7d6141.png) There are other alternatives: ``` ingester: flush_op_timeout: 10m ``` This `flush_op_timeout` configuration is more about the timeout of loki, not the timeout of s3 client. We will also have various s3 operations such as delete and getObject. We should add an independent timeout for s3, just like the cassandra client has a separate The timeout period can be configured. And the parameter `flush_op_timeout` in the early version of loki represents the timeout of writing `a batch of chunks `to s3, not a timeout of s3. Now it is indeed changed to write to s3 one by one, but it is not ruled out that in the future, it will be changed back to write a batch of chunks to s3 sequentially. In this way, `flush_op_timeout: 5m` is far from enough to write a batch of chunks to s3. ``` cassandra: timeout: 5m ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Salva Corts	56bbb43748	Logcli - Better logging when object cannot get loaded from object store (#9194 ) What this PR does / why we need it: This PR improves the error messaging for Logcli when an object cannot be downloaded from the store by printing the name of the object along with the error. Which issue(s) this PR fixes: Internal support escalation Special notes for your reviewer: There are other places where GetObject is called: - `indexStorageClient.GetFile` - `indexStorageClient.GetUserFile` Looks like both of them are used only by the compactor and the table manager. IIUC, these functions are not used by LogCLI. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	9159c1dac3	Loki: Improve spans usage (#8927 ) What this PR does / why we need it: - At different places, inherit the span/spanlogger from the given context instead of instantiating a new one from scratch, which fix spans being orphaned on a read/write operation. - At different places, turn spans into events. Events are lighter than spans and by having fewer spans in the trace, trace visualization will be cleaner without losing any details. - Adds new spans/events to places that might be a bottleneck for our writes/reads.	2 years ago
Suruthi-G-K	e7b0d4d26e	feat(cos): Support authentication with trusted profiles (#8939 ) What this PR does / why we need it: Add trusted profile authentication in COS client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com>	2 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826 ) What this PR does / why we need it: Add support for IBM cloud object storage as storage client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	2 years ago
André Águas	44f1d8d7f6	azure: respect retry config before cancelling the context (#8732 ) What this PR does / why we need it: On GET blob operations a context timeout is set to `RequestTimeout`, however this value is not the full timeout. This is only the first timeout and afterwards there are retries. As a result, the retry configuration is never used because the context is immediately cancelled. To fix we should set the context timeout to a value larger than the Azure timeout after retries. Since the backoff is exponential and randomized we need to take the value of `MaxRetryDelay` to set an upper bound.	2 years ago
Trevor Whitney	b46e392adb	gofmt is needed for the build (#8678 ) What this PR does / why we need it: `gofmt` is needed for the build, `make format` fails without it.	2 years ago
Jan Jansen	84c5e20af1	S3: add support for s3 session token (#8474 ) We want to automate the access account rotation. This is that easy at moment with service accounts in MinIO. Our currently life time of tokens is 3 months with MinIO STS works this fine. Signed-off-by: Jan Jansen <jan.jansen@gdata.de>	2 years ago
justcompile	ea55f285b6	S3 Storage Class Configuration (#8532 )	2 years ago
brettlounsbury	4c227bd268	DynamoDB: Accept a KMS key for DynamoDB tables (#8469 ) Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	2 years ago
janhorstmann	0e81d78914	Swift: Allow to configure swift with the internal endpoint (#8438 )	2 years ago
Bryan Boreham	15b382294e	Remove dependency on thanos-io/thanos (#8227 ) What this PR does / why we need it: This avoids complications trying to match up versions between Prometheus and Thanos. Special notes for your reviewer: I don't know why the dependency on `github.com/efficientgo/e2e` has appeared in go.mod, but it doesn't seem to change what is in vendor. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Nils	e4fbd3c949	Loki: Add support for Azure Workload Identity authentication (#7540 )	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916 ) What this PR does / why we need it: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](`5550cd65ec/docs/sources/configuration/_index.md`) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. NOTE: this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/83 Special notes for your reviewer: Files: * [docs/sources/configuration/index.template](`5550cd65ec/docs/sources/configuration/index.template`): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](`c32e5d0acb/docs/sources/configuration/_index.md`): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](`5550cd65ec/tools/doc-generator`) directory with documentation generation tool. Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
李国忠	b3664ff255	[fix] cassandra: reconnection when k8s cassandra pods restart (#6725 ) What this PR does / why we need it: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. Which issue(s) this PR fixes: Fixes #1821 #7140	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340 ) I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243 ) Upgrade Loki and all related components to Go 1.19.2.	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212 ) The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Owen Diehl	85e8f00958	fixes azure client secret duplicate CLI registration (#7185 ) Fixes duplicate key registration from https://github.com/grafana/loki/pull/7179/files#diff-665c7ce7ef736bdf9f9da76a8e389d5b6fcdc90099db913ce36b4c0da2a97ef2R121-R122	3 years ago
Vladyslav Diachenko	7f629c8849	[loki] Azure auth using Service Principal (#7179 ) This PR adds the ability to use Service Principal credentials to authenticate through Azure OAuth. It might be needed when we want to access Azure Blob storage from another tenant. Previously we could use only StorageAccount key that can not be used cross-tenant.	3 years ago
sardismart	821301b95f	azblob: Make azblob support private cloud endpoint suffix (#6842 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The endpoint suffix of private cloud(azure stack hub) azblob is different from Azure. The loki configuration has already support customize endpoint suffix but not use when select blob/container URL. This PR introduce following change: * Consider endpoint suffix from configuration when select blob or container URL format Which issue(s) this PR fixes: Fixes #6863 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	e3cd312c37	[fix panic] s3: fix object nil point (#6970 ) What this PR does / why we need it: fix a s3 panic. ```go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14fec7b] goroutine 6271245 [running]: github.com/grafana/loki/pkg/storage/chunk/client.(*client).getChunk(_, {_, _}, _, {{0xb130cf997b272f99, {0xc05077e6a0, 0x1b}, 0x182d1aefb2d, 0x182d1b47984, 0x9fab6388}, ...}) /loki@v1.6.2-0.20220816085209-65ed7b60fbd4/pkg/storage/chunk/client/object_client.go:171 +0x2fb ```	3 years ago
Danny Kopping	2321da49b5	Upgrade prometheus/client_golang and reconfigure to restore `go_sched.` metrics (#6957 ) Upgrading prometheus/client_golang to restore go_sched.* metrics These metrics were removed unintentionally in #6403 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test to match interface correctly Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	66fdb3c572	Log stream flushes to see how many chunks we flush at a time (#6819 ) * Log stream flushes to see how many chunks we flush at a time Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Smuggling in an error wrapping which aides debugging Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Ed Welch <ed@oqqer.com> * Removing unnecessary label Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Ed Welch <ed@oqqer.com>	3 years ago
Owen Diehl	8daa694d12	[Fix] Furthers gcs latency buckets (#6721 ) * furthers gcs latency buckets Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * adjusts bucket upper bound Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Marco Pracucci	e7fb10f456	Loki: Update `grafana/dskit` and `weaveworks/common` (#6643 ) * Upgrade dskit and weaveworks/common Signed-off-by: Marco Pracucci <marco@pracucci.com> * 'go mod vendor' again Signed-off-by: Marco Pracucci <marco@pracucci.com>	3 years ago
Wu Zhiyuan	f61cbe4288	GCS: Add service_account option (#6366 ) * gcs client: add service_account support * Update docs for GCS's service_account * Fixed lint check failed	3 years ago
Evangelos Foutras	a877febf75	Use fake credentials in storage/chunk/storage tests (#6187 ) Enable insecure mode for the GCS client so fake credentials are used when running the storage tests (see commit `c0cc004333` ("gcs client: use fake credentials in unit test"). Fixes: https://github.com/grafana/loki/issues/6185	3 years ago
Steve Hipwell	7cc6aa4021	feat: Add support for Azure user assigned identity (#5891 ) Signed-off-by: Steve Hipwell <steve.hipwell@gmail.com>	3 years ago
Sandeep Sukhani	6ac955cc7d	make boltdb-shipper use the generic indexshipper for managing index in object storage (#6226 )	3 years ago
Christian Haudum	b073ec9017	security: Redact credentials when marshalled to YAML (#6186 )	3 years ago
arcosx	e2fdc3437b	Feat: add Baidu Cloud BOS as storage backends for Loki #4788 (#5848 ) * feat: add baidu bce bos storage support Signed-off-by: arcosx <arcosx@outlook.com> * add baidu bce bos as chunk storage backend Signed-off-by: arcosx <arcosx@outlook.com> * fix: some doc error && rewrite bad code Signed-off-by: arcosx <arcosx@outlook.com> * fix: add the `BceServiceError` source link Signed-off-by: arcosx <arcosx@outlook.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: arcosx <arcosx@outlook.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update pkg/storage/chunk/client/baidubce/bos_storage_client.go Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Karen Miller	2264b73311	Docs: remove Cortex references (#6079 ) * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update	3 years ago
Cyril Tovena	5d51e5a54f	Improve batching in boltdb index (#6044 ) * Improve batching in boltdb index * review feedback	3 years ago
Christian Simon	6c1a62d829	Ensure proto definitions are formatted [2/2] (#6000 ) * Update build image to 0.10.4 * Add a fmt target for proto defintions This uses buf to ensure the proto defintions are more consistently formatted. * Update formatted files	3 years ago
Trevor Whitney	2a04ac9af7	Remove references to cortex (#6015 )	3 years ago
afayngelerindbx	c0f51b9851	redact s3 credential values when printing config to logs (#5961 )	3 years ago
Karsten Jeschkies	c9f3868287	Update dskit to v0.0.0-20220331160727-49faf69f72ca. (#5881 )	3 years ago
Bryan Boreham	e2d7ee2cff	GCS: refactor creation of http client (#5838 ) Separate the function into one part which creates the RoundTripper and one which wraps it in instrumentation.	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago

1 2

95 Commits (d9c441efcc159e91a4251c9aca8f8914633c8505)