apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ashwanth	758364c777	fix: remove usage of unsafe string in label adapter unmarshal (#14216 )	1 year ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365 )	2 years ago
dependabot[bot]	ff2063bcaf	chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /pkg/push (#12694 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
renovate[bot]	bdad1d331d	chore(deps): update module google.golang.org/protobuf to v1.33.0 [security] (main) (#12137 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
renovate[bot]	6665685888	fix(deps): update golang.org/x/exp digest to a685a6e (main) (#11155 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
renovate[bot]	0695424f7d	fix(deps): update module google.golang.org/grpc [security] (main) (#11031 ) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) \| require \| minor \| `v1.53.0` -> `v1.56.3` \| \| [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) \| require \| patch \| `v1.58.2` -> `v1.58.3` \| ### GitHub Vulnerability Alerts #### [GHSA-m425-mq94-257g](https://togithub.com/grpc/grpc-go/security/advisories/GHSA-m425-mq94-257g) ### Impact In affected releases of gRPC-Go, it is possible for an attacker to send HTTP/2 requests, cancel them, and send subsequent requests, which is valid by the HTTP/2 protocol, but would cause the gRPC-Go server to launch more concurrent method handlers than the configured maximum stream limit. ### Patches This vulnerability was addressed by #6703 and has been included in patch releases: 1.56.3, 1.57.1, 1.58.3. It is also included in the latest release, 1.59.0. Along with applying the patch, users should also ensure they are using the `grpc.MaxConcurrentStreams` server option to apply a limit to the server's resources used for any single connection. ### Workarounds None. ### References #6703 --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.56.3`](https://togithub.com/grpc/grpc-go/releases/tag/v1.56.3): Release 1.56.3 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.56.2...v1.56.3) ### Security - server: prohibit more than MaxConcurrentStreams handlers from running at once (CVE-2023-44487) In addition to this change, applications should ensure they do not leave running tasks behind related to the RPC before returning from method handlers, or should enforce appropriate limits on any such work. ### [`v1.56.2`](https://togithub.com/grpc/grpc-go/releases/tag/v1.56.2): Release 1.56.2 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.56.1...v1.56.2) - status: To fix a panic, `status.FromError` now returns an error with `codes.Unknown` when the error implements the `GRPCStatus()` method, and calling `GRPCStatus()` returns `nil`. ([#6374](https://togithub.com/grpc/grpc-go/issues/6374)) ### [`v1.56.1`](https://togithub.com/grpc/grpc-go/releases/tag/v1.56.1): Release 1.56.1 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.56.0...v1.56.1) - client: handle empty address lists correctly in addrConn.updateAddrs ### [`v1.56.0`](https://togithub.com/grpc/grpc-go/releases/tag/v1.56.0): Release 1.56.0 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.55.1...v1.56.0) ### New Features - client: support channel idleness using `WithIdleTimeout` dial option ([#6263](https://togithub.com/grpc/grpc-go/issues/6263)) - This feature is currently disabled by default, but will be enabled with a 30 minute default in the future. - client: when using pickfirst, keep channel state in TRANSIENT_FAILURE until it becomes READY ([gRFC A62](https://togithub.com/grpc/proposal/blob/master/A62-pick-first.md)) ([#6306](https://togithub.com/grpc/grpc-go/issues/6306)) - xds: Add support for Custom LB Policies ([gRFC A52](https://togithub.com/grpc/proposal/blob/master/A52-xds-custom-lb-policies.md)) ([#6224](https://togithub.com/grpc/grpc-go/issues/6224)) - xds: support pick_first Custom LB policy ([gRFC A62](https://togithub.com/grpc/proposal/blob/master/A62-pick-first.md)) ([#6314](https://togithub.com/grpc/grpc-go/issues/6314)) ([#6317](https://togithub.com/grpc/grpc-go/issues/6317)) - client: add support for pickfirst address shuffling ([gRFC A62](https://togithub.com/grpc/proposal/blob/master/A62-pick-first.md)) ([#6311](https://togithub.com/grpc/grpc-go/issues/6311)) - xds: Add support for String Matcher Header Matcher in RDS ([#6313](https://togithub.com/grpc/grpc-go/issues/6313)) - xds/outlierdetection: Add Channelz Logger to Outlier Detection LB ([#6145](https://togithub.com/grpc/grpc-go/issues/6145)) - Special Thanks: [@s-matyukevich](https://togithub.com/s-matyukevich) - xds: enable RLS in xDS by default ([#6343](https://togithub.com/grpc/grpc-go/issues/6343)) - orca: add support for application_utilization field and missing range checks on several metrics setters - balancer/weightedroundrobin: add new LB policy for balancing between backends based on their load reports ([gRFC A58](https://togithub.com/grpc/proposal/blob/master/A58-client-side-weighted-round-robin-lb-policy.md)) ([#6241](https://togithub.com/grpc/grpc-go/issues/6241)) - authz: add conversion of json to RBAC Audit Logging config ([#6192](https://togithub.com/grpc/grpc-go/issues/6192)) - authz: add support for stdout logger ([#6230](https://togithub.com/grpc/grpc-go/issues/6230) and [#6298](https://togithub.com/grpc/grpc-go/issues/6298)) - authz: support customizable audit functionality for authorization policy ([#6192](https://togithub.com/grpc/grpc-go/issues/6192) [#6230](https://togithub.com/grpc/grpc-go/issues/6230) [#6298](https://togithub.com/grpc/grpc-go/issues/6298) [#6158](https://togithub.com/grpc/grpc-go/issues/6158) [#6304](https://togithub.com/grpc/grpc-go/issues/6304) and [#6225](https://togithub.com/grpc/grpc-go/issues/6225)) ### Bug Fixes - orca: fix a race at startup of out-of-band metric subscriptions that would cause the report interval to request 0 ([#6245](https://togithub.com/grpc/grpc-go/issues/6245)) - xds/xdsresource: Fix Outlier Detection Config Handling and correctly set xDS Defaults ([#6361](https://togithub.com/grpc/grpc-go/issues/6361)) - xds/outlierdetection: Fix Outlier Detection Config Handling by setting defaults in ParseConfig() ([#6361](https://togithub.com/grpc/grpc-go/issues/6361)) ### API Changes - orca: allow a ServerMetricsProvider to be passed to the ORCA service and ServerOption ([#6223](https://togithub.com/grpc/grpc-go/issues/6223)) ### [`v1.55.1`](https://togithub.com/grpc/grpc-go/releases/tag/v1.55.1): Release 1.55.1 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.55.0...v1.55.1) - status: To fix a panic, `status.FromError` now returns an error with `codes.Unknown` when the error implements the `GRPCStatus()` method, and calling `GRPCStatus()` returns `nil`. ([#6374](https://togithub.com/grpc/grpc-go/issues/6374)) ### [`v1.55.0`](https://togithub.com/grpc/grpc-go/releases/tag/v1.55.0): Release 1.55.0 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.54.1...v1.55.0) ### Behavior Changes - xds: enable federation support by default ([#6151](https://togithub.com/grpc/grpc-go/issues/6151)) - status: `status.Code` and `status.FromError` handle wrapped errors ([#6031](https://togithub.com/grpc/grpc-go/issues/6031) and [#6150](https://togithub.com/grpc/grpc-go/issues/6150)) - Special Thanks: [@psyhatter](https://togithub.com/psyhatter) ### New Features - xds/xdsclient: support `ignore_resource_deletion` server feature as per gRFC [A53](https://togithub.com/grpc/proposal/blob/master/A53-xds-ignore-resource-deletion.md) ([#6035](https://togithub.com/grpc/grpc-go/issues/6035)) - security/advancedtls: add min/max TLS version selection options ([#6007](https://togithub.com/grpc/grpc-go/issues/6007)) - Special Thanks: [@joeljeske](https://togithub.com/joeljeske) ### Bug Fixes - xds: stop routing RPCs to deleted clusters ([#6125](https://togithub.com/grpc/grpc-go/issues/6125)) - client: fix race between stream creation and GOAWAY receipt, which could lead to spurious UNAVAILABLE stream errors ([#6142](https://togithub.com/grpc/grpc-go/issues/6142)) ### Performance Improvements - server: improve stream handler goroutine worker allocation when [`NumStreamWorkers`](https://pkg.go.dev/google.golang.org/grpc#NumStreamWorkers) is used ([#6004](https://togithub.com/grpc/grpc-go/issues/6004)) - Special Thanks: [@SaveTheRbtz](https://togithub.com/SaveTheRbtz) ### [`v1.54.1`](https://togithub.com/grpc/grpc-go/releases/tag/v1.54.1): Release 1.54.1 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.54.0...v1.54.1) ### Bug Fixes - credentials/alts: revert a change that causes a crash in the handshaker ### [`v1.54.0`](https://togithub.com/grpc/grpc-go/releases/tag/v1.54.0): Release 1.54.0 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.53.0...v1.54.0) ### Behavior Changes - xds: remove support for xDS v2 transport API ([#6013](https://togithub.com/grpc/grpc-go/issues/6013)) ### New Features - server: expose `SetSendCompressor` API to set send compressor name ([#5744](https://togithub.com/grpc/grpc-go/issues/5744)) - Special Thanks: [@jronak](https://togithub.com/jronak) - xdsclient: include `Node` proto only in the first discovery request message, to improve performance ([#6078](https://togithub.com/grpc/grpc-go/issues/6078)) ### Bug Fixes - metadata: fix validation logic and properly validate metadata appended via `AppendToOutgoingContext` ([#6001](https://togithub.com/grpc/grpc-go/issues/6001)) - Special Thanks: [@ktalg](https://togithub.com/ktalg) - transport: do not close connections when we encounter I/O errors until after all data is consumed ([#6110](https://togithub.com/grpc/grpc-go/issues/6110)) - ringhash: ensure addresses are consistently hashed across updates ([#6066](https://togithub.com/grpc/grpc-go/issues/6066)) - xds/clusterimpl: fix a bug causing unnecessary closing and re-opening of LRS streams ([#6112](https://togithub.com/grpc/grpc-go/issues/6112)) - xds: NACK route configuration if sum of weights of weighted clusters exceeds uint32\_max ([#6085](https://togithub.com/grpc/grpc-go/issues/6085)) ### Documentation - resolver: update `Resolver.Scheme()` docstring to mention requirement of lowercase scheme names ([#6014](https://togithub.com/grpc/grpc-go/issues/6014)) - resolver: document expected error handling of `UpdateState` errors ([#6002](https://togithub.com/grpc/grpc-go/issues/6002)) - Special Thanks: [@fho](https://togithub.com/fho) - examples: add example for ORCA load reporting ([#6114](https://togithub.com/grpc/grpc-go/issues/6114)) - examples: add an example to illustrate authorization (authz) support ([#5920](https://togithub.com/grpc/grpc-go/issues/5920)) - Special Thanks: [@KenxinKun](https://togithub.com/KenxinKun) </details> --- ### Configuration 📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 Immortal: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zMS41IiwidXBkYXRlZEluVmVyIjoiMzcuMzEuNSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Salva Corts	52a3f16039	Flag categorize labels on streams response (#10419 ) We recently introduced support for ingesting and querying structured metadata in Loki. This adds a new dimension to Loki's labels since now we arguably have three categories of labels: _stream_, _structured metadata_, and _parsed_ labels. Depending on the origin of the labels, they should be used in LogQL expressions differently to achieve optimal performance. _stream_ labels should be added to stream matchers, _structured metadata_ labels should be used in a filter expression before any parsing expression, and _parsed_ labels should be placed after the parser expression extracting them. The Grafana UI has a hard time dealing with this same problem. Before https://github.com/grafana/grafana/pull/73955, the filtering functionality in Grafana was broken since it was not able to distinguish between _stream_ and _structured metadata_ labels. Also, as soon as a parser expression was added to the query, filters added by Grafana would be appended to the end of the query regardless of the label category. The PR above implements a workaround for this problem but needs a better API on Loki's end to mitigate all corner cases. Loki currently returns the following JSON for log queries: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "info", "traceID": "68810cf0c94bfcca" }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n" ], ... }, { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "debug", "traceID": "a7116cj54c4bjz8s" }, "values": [ [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n" ], ... }, ... ``` As can be seen, there is no way we can distinguish the category of each label. This PR introduces a new flag `X-Loki-Response-Encoding-Flags: categorize-labels` that makes Loki return categorized labels as follows: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n", { "structuredMetadata": { "traceID": "68810cf0c94bfcca" }, "parsed": { "level": "info" } } ], [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n", { "structuredMetadata": { "traceID": "a7116cj54c4bjz8s" }, "parsed": { "level": "debug" } } ], ... }, ... ``` Note that this PR only supports log queries, not metric queries. From a UX perspective, being able to categorize labels in metric queries doesn't have any benefit yet. Having said that, supporting this for metric queries would require some minor refactoring on top of what has been implemented here. If we decide to do that, I think we should do it on a separate PR to avoid making this PR even larger. I also decided to leave out support for Tail queries to avoid making this PR even larger. Once this one gets merged, we can work to support tailing. --- Note to reviewers This PR is huge since we need to forward categorized all over the codebase (from parsing logs all the way to marshaling), fortunately, many of the changes come from updating tests and refactoring iterators. Tested out in a dev cell with query `'{stream="stdout"} \| label_format new="text"`. - Without the new flag: ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED { "data": { "result": [ { "stream": { "new": "text", "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n" ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n" ], ... ``` - With the new flag ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED X-Loki-Response-Encoding-Flags:categorize-labels { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n", { "parsed": { "new": "text" } } ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n", { "parsed": { "new": "text" } } ], ... ```	2 years ago
renovate[bot]	ab94c70c4c	chore(deps): update module golang.org/x/net to v0.17.0 [security] (main) (#10831 ) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| golang.org/x/net \| indirect \| minor \| `v0.7.0` -> `v0.17.0` \| \| golang.org/x/net \| indirect \| minor \| `v0.8.0` -> `v0.17.0` \| ### GitHub Vulnerability Alerts #### [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978) Text nodes not in the HTML namespace are incorrectly literally rendered, causing text which should be escaped to not be. This could lead to an XSS attack. #### [CVE-2023-39325](https://togithub.com/golang/go/issues/63417) A malicious HTTP/2 client which rapidly creates requests and immediately resets them can cause excessive server resource consumption. While the total number of requests is bounded by the http2.Server.MaxConcurrentStreams setting, resetting an in-progress request allows the attacker to create a new request while the existing one is still executing. With the fix applied, HTTP/2 servers now bound the number of simultaneously executing handler goroutines to the stream concurrency limit (MaxConcurrentStreams). New requests arriving when at the limit (which can only happen after the client has reset an existing, in-flight request) will be queued until a handler exits. If the request queue grows too large, the server will terminate the connection. This issue is also fixed in golang.org/x/net/http2 for users manually configuring HTTP/2. The default stream concurrency limit is 250 streams (requests) per HTTP/2 connection. This value may be adjusted using the golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams setting and the ConfigureServer function. #### [CVE-2023-44487](https://togithub.com/apple/swift-nio-http2/security/advisories/GHSA-qppj-fm5r-hxr3) swift-nio-http2 is vulnerable to a denial-of-service vulnerability in which a malicious client can create and then reset a large number of HTTP/2 streams in a short period of time. This causes swift-nio-http2 to commit to a large amount of expensive work which it then throws away, including creating entirely new `Channel`s to serve the traffic. This can easily overwhelm an `EventLoop` and prevent it from making forward progress. swift-nio-http2 1.28 contains a remediation for this issue that applies reset counter using a sliding window. This constrains the number of stream resets that may occur in a given window of time. Clients violating this limit will have their connections torn down. This allows clients to continue to cancel streams for legitimate reasons, while constraining malicious actors. --- ### Configuration 📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy44LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNy4xOS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Arve Knudsen	489ca25bab	Upgrade to Prometheus@4b9c19fe5510 (#10843 ) What this PR does / why we need it: Upgrade to Prometheus@4b9c19fe5510. Since also golang.org/x/exp gets upgraded, calls to e.g. `slices.SortFunc` need updating (due to changed comparison function signature). I'm not aware of any functional changes from Prometheus (or otherwise), but please review carefully. Special notes for your reviewer: The Mimir team needs for Loki to be compatible with the latest Prometheus, so we can vendor mimir-prometheus@main into GEM. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [na] Documentation added - [x] Tests updated - [na] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [na] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [na] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432 ) Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Salva Corts	15af77ba83	Remove Prometheus dependency for push pkg (#9937 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/9694, we modified the `push` pkg to import Prometheus as a dependency to use the `labels.Labels` type for the entries' non-indexed labels. Having Prometheus as a dependency is problematic since any project importing the `push` pkg will need to import Prometheus as a result. In fact, this is one of the reasons why the `push` pkg was extracted from Loki in the first place (https://github.com/grafana/loki/pull/8259). This PR removes the dependency of Prometheus from the `push` pkg by copying some bits of the implementation for Prometheus' `labels.Labels`. We copy: - The Labels struct definition - The JSON Marshaling and Unmarshaling methods for the labels. We need this so labels are encoded as maps instead of an array of objects. --- Notes for reviewers: - To implement the JSON Marshaling and Unmarshaling methods the `push` pkg now depends on `golang.org/x/exp`. I think it should be fine for projects importing the `push` pkg to also depend on `golang.org/x/exp`, right?	3 years ago
Salva Corts	aae13c376d	Add metadata to push payload (#9694 ) What this PR does / why we need it: We are adding support for attaching labels to each log line. This is one of the series of the PRs broken up to make it easier to review changes. This PR updates the push payload to send labels with each log entry optionally. The log labels are supposed to be in the same format as the stream labels. Just to put it out, here is how it would look for proto and json push payload with same data: proto(`endpoint`: `(/loki/api/v1/push\|/api/prom/push)`, `Content-Type`: `application/x-protobuf`)(payload built using [push.Stream](`4cd1246b88/pkg/push/types.go (L12)`)): ``` push.Stream{ Entries: []logproto.Entry{ { Timestamp: time.Unix(0, 1688515200000000000), Line: "log line", Labels: `{foo="bar"}`, }, }, Labels: `{app="test"}`, } ``` v1(`endpoint`: `/loki/api/v1/push`, `Content-Type`: `application/json`): ```json { "streams": [{ "stream": { "app": "test" }, "values": [ ["1688515200000000000", "log line", { "foo": "bar" }] ] }] } ``` legacy-json(`/api/prom/push`, `Content-Type`: `application/json`): ```json { "streams": [{ "labels": "{app=\"test\"}", "entries": [{ "ts": "2023-07-05T00:00:00.000000000Z", "line": "log line", "labels": "{foo=\"bar\"}" }] }] } ``` Which issue(s) this PR fixes: Special notes for your reviewer: We may need to add more thoughtful tests. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
dependabot[bot]	9ee918d359	Bump google.golang.org/grpc from 1.52.0 to 1.53.0 in /pkg/push (#9871 ) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.52.0 to 1.53.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-go/releases">google.golang.org/grpc's releases</a>.</em></p> <blockquote> <h2>Release 1.53.0</h2> <h1>API Changes</h1> <ul> <li>balancer: support injection of per-call metadata from LB policies (<a href="https://redirect.github.com/grpc/grpc-go/issues/5853">#5853</a>)</li> <li>resolver: remove deprecated field <code>resolver.Target.Endpoint</code> and replace with <code>resolver.Target.Endpoint()</code> (<a href="https://redirect.github.com/grpc/grpc-go/issues/5852">#5852</a>) <ul> <li>Special Thanks: <a href="https://github.com/kylejb"><code>@kylejb</code></a></li> </ul> </li> </ul> <h1>New Features</h1> <ul> <li>xds/ringhash: introduce <code>GRPC_RING_HASH_CAP</code> environment variable to override the maximum ring size. (<a href="https://redirect.github.com/grpc/grpc-go/issues/5884">#5884</a>)</li> <li>rls: propagate headers received in RLS response to backends (<a href="https://redirect.github.com/grpc/grpc-go/issues/5883">#5883</a>)</li> </ul> <h1>Bug Fixes</h1> <ul> <li>transport: drain client transport when streamID approaches MaxStreamID (<a href="https://redirect.github.com/grpc/grpc-go/issues/5889">#5889</a>)</li> <li>server: after GracefulStop, ensure connections are closed when final RPC completes (<a href="https://redirect.github.com/grpc/grpc-go/issues/5968">#5968</a>)</li> <li>server: fix a few issues where grpc server uses RST_STREAM for non-HTTP/2 errors (<a href="https://redirect.github.com/grpc/grpc-go/issues/5893">#5893</a>)</li> <li>xdsclient: fix race which can happen when multiple load reporting calls are made at the same time. (<a href="https://redirect.github.com/grpc/grpc-go/issues/5927">#5927</a>)</li> <li>rls: fix a data race involving the LRU cache (<a href="https://redirect.github.com/grpc/grpc-go/issues/5925">#5925</a>)</li> <li>xds: fix panic involving double close of channel in xDS transport (<a href="https://redirect.github.com/grpc/grpc-go/issues/5959">#5959</a>)</li> <li>gcp/observability: update method name validation (<a href="https://redirect.github.com/grpc/grpc-go/issues/5951">#5951</a>)</li> </ul> <h1>Documentation</h1> <ul> <li>credentials/oauth: mark <code>NewOauthAccess</code> as deprecated (<a href="https://redirect.github.com/grpc/grpc-go/issues/5882">#5882</a>) <ul> <li>Special Thanks: <a href="https://github.com/buzzsurfr"><code>@buzzsurfr</code></a></li> </ul> </li> </ul> <h2>Release 1.52.3</h2> <h1>Bug Fixes</h1> <ul> <li>Fix user-agent version</li> </ul> <h2>Release 1.52.2</h2> <h1>Bug Fixes</h1> <ul> <li>xds: fix panic involving double close of channel in xDS transport (<a href="https://redirect.github.com/grpc/grpc-go/issues/5959">#5959</a>)</li> </ul> <h2>Release 1.52.1</h2> <h1>Bug Fixes</h1> <ul> <li>grpclb: rename grpclbstate package back to state (<a href="https://redirect.github.com/grpc/grpc-go/issues/5963">#5963</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="`dba26e15a0`"><code>dba26e1</code></a> Change version to 1.53.0 (<a href="https://redirect.github.com/grpc/grpc-go/issues/5983">#5983</a>)</li> <li><a href="`2a1e9348ff`"><code>2a1e934</code></a> server: after GracefulStop, ensure connections are closed when final RPC comp...</li> <li><a href="`e2d69aa076`"><code>e2d69aa</code></a> tests: fix spelling of variable (<a href="https://redirect.github.com/grpc/grpc-go/issues/5966">#5966</a>)</li> <li><a href="`a6376c9893`"><code>a6376c9</code></a> xds/resolver: cleanup tests to use real xDS client 3/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5953">#5953</a>)</li> <li><a href="`bf8fc46fa6`"><code>bf8fc46</code></a> xds/resolver: cleanup tests to use real xDS client 5/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5955">#5955</a>)</li> <li><a href="`3930549b38`"><code>3930549</code></a> resolver: replace resolver.Target.Endpoint field with Endpoint() method (<a href="https://redirect.github.com/grpc/grpc-go/issues/5852">#5852</a>)</li> <li><a href="`894816c487`"><code>894816c</code></a> grpclb: rename <code>grpclbstate</code> package back to <code>state</code> (<a href="https://redirect.github.com/grpc/grpc-go/issues/5962">#5962</a>)</li> <li><a href="`e5a0237a46`"><code>e5a0237</code></a> encoding: fix duplicate compressor names (<a href="https://redirect.github.com/grpc/grpc-go/issues/5958">#5958</a>)</li> <li><a href="`4adb2a7a00`"><code>4adb2a7</code></a> xds/resolver: cleanup tests to use real xDS client 2/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5952">#5952</a>)</li> <li><a href="`52a8392f37`"><code>52a8392</code></a> gcp/observability: update method name validation (<a href="https://redirect.github.com/grpc/grpc-go/issues/5951">#5951</a>)</li> <li>Additional commits viewable in <a href="https://github.com/grpc/grpc-go/compare/v1.52.0...v1.53.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=google.golang.org/grpc&package-manager=go_modules&previous-version=1.52.0&new-version=1.53.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grafana/loki/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
dependabot[bot]	31f8bf9a73	Bump golang.org/x/net from 0.4.0 to 0.7.0 in /pkg/push (#8562 ) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. <details> <summary>Commits</summary> <ul> <li><a href="`8e2b117aee`"><code>8e2b117</code></a> http2/hpack: avoid quadratic complexity in hpack decoding</li> <li><a href="`547e7edf38`"><code>547e7ed</code></a> http2: avoid referencing ResponseWrite.Write parameter after returning</li> <li><a href="`39940adcaa`"><code>39940ad</code></a> html: parse comments per HTML spec</li> <li><a href="`87ce33ecb4`"><code>87ce33e</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="`415cb6d518`"><code>415cb6d</code></a> all: fix some comments</li> <li><a href="`7e3c19ca52`"><code>7e3c19c</code></a> all: correct typos in comments</li> <li><a href="`296f09aa38`"><code>296f09a</code></a> http2: case insensitive handling for 100-continue</li> <li><a href="`f8411da775`"><code>f8411da</code></a> nettest: fix tests on dragonfly and js/wasm</li> <li><a href="`8e0e7d8d38`"><code>8e0e7d8</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="`7805fdc37d`"><code>7805fdc</code></a> http2: rewrite inbound flow control tracking</li> <li>Additional commits viewable in <a href="https://github.com/golang/net/compare/v0.4.0...v0.7.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/net&package-manager=go_modules&previous-version=0.4.0&new-version=0.7.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grafana/loki/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Ed Welch	fe904c5aee	Loki: add apache2 license to newly refactored push module (#8305 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Irina	4cd1246b88	Logproto: Extract push.proto from logproto package to the separate module (#8259 ) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago

19 Commits (d7c0f2bfc136155bbdde79fe481ebddee4e84023)