179 Commits (e382a4175cf1491c5d37c707850ccfcf4c5bf00a)

Author	SHA1	Message	Date
Trevor Whitney	ef8b382c06	[helm] Single binary object storage (#8284) ... Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Trevor Whitney	9920a27f06	Update GEL plugin in k3d env to latest (#8378)	3 years ago
Trevor Whitney	cc39334391	[helm] Fix helm dashboards (#8261) ... Fix the dashboards shipped with the helm chart. Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Danny Kopping	ebfeba5de3	Ruler: implement rule-based and shuffle sharding (#8092)	3 years ago
Christian Haudum	1728d09cba	Use Loki build image 0.27.1 in CI (#8279) ... Second part of updating the Loki build image Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	58e29de988	Update golang to 1.19.5 (#8278) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Taehyun Hwang	b2ea1214fc	Loki: Fix version info issue that shows wrong version (#7669, #8055) (#8232)	3 years ago
Trevor Whitney	71846b0430	[helm] add local k3d loki cluster (#8218)	3 years ago
Trevor Whitney	40ed217620	Make secret job fail when not logged in (#8215) ... Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Karsten Jeschkies	44e8422dc4	Enalbe multi tenant queries on Helm test chart. (#8228) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Danny Kopping	d0fd4e8ad1	Do not exit when using `stream_lag_labels` setting (#8243)	3 years ago
Trevor Whitney	b6004050a6	[helm] Fix self-monitoring tenant secret (#8198) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/8081 introduced a bug that caused the `provisioner` job to create an incorrect secret for the self-monitoring tenant used by the canary. This fixes that bug, as well as adding a readme on how to use the `helm-cluster` k3d env. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Whitney	eb39e2645f	[helm] allow provisioned tokens to be created in different namespaces (#8081) ... **What this PR does / why we need it**: _Note_: This PR bumps the chart to 4.0, so it includes a few other breaking changes we've been waiting to make. The new helm chart has a few different ways to create tokens when running in enterprise mode: * The tokengen job creates the initial admin token * The provisioner job creates the self-monitoring tenant if enabled * The provisioner job can also create additional tenants if configured The problem with these jobs is they put the generated read/write tokens for these tenants in secrets in the same namespace as the loki deploiyment. This is problematic if you need these tokens for a specific application, for example a Grafana datasource, that is deployed in a separate namespace. This PR introduces two breaking changes, one of which is breaking as it significantly changes how both the self monitoring and additional tenants are configured. First it adds the value `enterprise.adminToken.additionalNamespaces` and moves `enterprise.adminTokenSecret` to `enterprise.adminToken.secret`. If `additionalNamespaces` are included, additional secrets containing the GEL admin token are created in those namespaces as well as the release namespace. Second, it changes the structure of defining a tenant from just a string name, to an object containing a `name` and `secretNamespace` field. The `secretNamespace` is the namespace the secret containing that tenants token(s) will be placed in. In the case of the self-monitoring token, a secret will still be created in the default namespace in addition to the `secretNamespace` specified as that token is required by multiple components.	3 years ago
Luis Felipe Vaca	d317e659c7	lambda-promtail-: fix parse kinesis timestamp (#7158)	3 years ago
Christophe Collot	ec33a891df	lambda-promtail: remove duplicate key in terraform config sample (#8137) ... **What this PR does / why we need it**: This PR removes a duplicate key in the terraform configuration sample **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: This is a trivial PR	3 years ago
Trevor Whitney	712bfdd0e0	[helm] add third scalable target `backend` (#7920) ... **What this PR does / why we need it**: Adds the new 3rd scalable deployment mode target (backend) to the helm chart. This new target was added in #7650	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868) ... This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Trevor Whitney	3f9b9cc961	[helm] fix race between minio bucket job and enterprise tokengen job (#8064) ... **What this PR does / why we need it**: This PR replaces https://github.com/grafana/loki/pull/7985. It solves the same problem, which is the race condition between the minio create bucket job and the enterprise tokengen job, but without removing minio.	3 years ago
Thomas Belián	6fcc4a2712	Lambda-promtail: Add skip tls verify option (#8059) ... **What this PR does / why we need it**: This PR sets a possibility to use lambda-promtail with self signed certificates sometimes used in development systems. This setting is configurable via environment variable. **Which issue(s) this PR fixes**: Fixes #8013 Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
Ashwanth	b3e038c7a8	config_wrapper: Add support for named stores (#7946) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: This PR adds support for named stores which allows users to define multiple object store configurations for a storage type. Named stores can be referred to from other sections of the config similar to how provider type is being used `i.e aws, gcs...`. Named store reference should be of the format: `<store_type>.<key_name>` Below is an example configuration with two named stores defined for aws `store-1, store-2`. Schema config is referring to one of them as `aws.store-1` ``` storage_config: aws: endpoint: s3://common-bucket region: us-east1 access_key_id: abc123 secret_access_key: def789 named_stores: aws: store-1: endpoint: s3://foo-bucket region: us-west1 access_key_id: 123abc secret_access_key: 789def store-2: endpoint: s3://bar-bucket region: us-west2 access_key_id: 456def secret_access_key: 789abc schema_config: configs: - from: "2020-07-31" index: period: 24h prefix: loki_prod_index_ object_store: aws.store-1 schema: v11 store: boltdb-shipper ``` **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: Configuration defined in common storage config has no effect on the named stores. Common storage config only overrides storageConfig if it's not explicitly set - same behavior as before. This should be a non-intrusive change. Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Trevor Whitney	98fd01aef8	Add empty k3d cluster for testing helm chart (#7984) ... **What this PR does / why we need it**: Add a new k3d cluster which is a cluster that has all Loki dependencies for purposes of testing the helm charts using `helm install`. The other environments use the helm charts, but deploy them using `tanka` and therefore can't catch certain race conditions or issues that occur with helm installs.	3 years ago
irizzant	ff7b462973	updated versions to the latest release v2.7.1 (#7940) ... Signed-off-by: irizzant <i.rizzante@gmail.com> **What this PR does / why we need it**: This PR updates Loki version to v2.7.1 **Which issue(s) this PR fixes**: Fixes #7768 Signed-off-by: irizzant <i.rizzante@gmail.com>	3 years ago
Susana Ferreira	d086c237df	Add missing period_config root block to doc generator (#7981) ... **What this PR does / why we need it**: Update doc generator tool to validate if slice element is a root block. Add missing `period_config` root block to doc generator tool. **Special notes for your reviewer**: **Checklist** - [ ] Documentation updated	3 years ago
Susana Ferreira	d36dd9edc4	Add README file to doc-generator tool (#7983) ... **What this PR does / why we need it**: Add README file to doc-generator tool	3 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Kaviraj Kanagaraj	c44fb2e882	chore(tools): Update promtail image version (#7770) ... **What this PR does / why we need it**: update promtail version on `tools/promtail.sh` which generates promtail config for kubernetes **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: This image is part of latest loki 2.7.0 release and[ it does exists ](https://hub.docker.com/layers/grafana/promtail/2.7.0/images/sha256-d54b599ea154df295486068f1545731bd5ab9790602d8518816c9285eec3df5e?context=repo) **Checklist**	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	3 years ago
Trevor Whitney	6c53113be1	[helm] Add a loki canary test to the helm chart (#7229) ... **What this PR does / why we need it**: Add a helm chart test that utilizes the Loki canary to confirm the Loki cluster is functioning correctly. **Which issue(s) this PR fixes**: Fixes #7228 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Karsten Jeschkies	9bc6e85604	Helm: Allow users to define remote write for metrics. (#7399)	3 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243) ... Upgrade Loki and all related components to Go 1.19.2.	3 years ago
Thomas Belian	b795672bba	feat(lambda-promtail): add bearer token support (#7333) ... It adds bearer token support for lambda-promtail, as it supports only http basic auth. Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
Sandeep Sukhani	b23687ad96	update regex for matching nlb logs while using lambda promtail (#7305) ... **What this PR does / why we need it**: Regex in lambda promtail looks for specific file patterns for reading and pushing logs from it. It currently does not read logs generated by nlb i.e having `_net` in the name. This PR updates the regex. Cross checked it with was docs here https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Trevor Whitney	afd63c5981	[helm] Add Loki Canary to Helm Chart (#7173) ... **What this PR does / why we need it**: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. **Which issue(s) this PR fixes**: Fixes #7019 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Trevor Whitney	4b6df86333	Add local k3d environment for enterprise-logs (#7146) ... **What this PR does / why we need it**: This adds a local k3d based development cluster for testing out the enterprise logs side of the helm chart, which can be enabled by setting: ``` enterprise: enabled: true ``` It requires both the `gex-plugins` and `enterprise-logs` repos to be checked out locally (though the latter soon will not be needed once the provisioner is published). We need this environment when developing enterprise features for the helm chart.	3 years ago
Jonathan	6b9bf0b5f4	feat: add kms and ec2 permission (#7058) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR will help to increase a necessary policy (kms and ec2) to resource log group and lambda. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Karsten Jeschkies	c23e541fe7	Generate Helm chart README. (#7012) ... **What this PR does / why we need it**: The Helm CI would break because the Helm chart README was not generated. This change * updates the README * introduces a filter that will only run the Helm CI on changes to the Helm chart directory * removes trailing whitespace * corrects the Minio chart repository	3 years ago
Karsten Jeschkies	1c3f5d0985	Helm: Upgrade Minio chart dependency to 4.0.12. (#6974) ... **What this PR does / why we need it**: Changes to a supported Minio chart. Minio is also enabled by default. **Which issue(s) this PR fixes**: Fixes #6973 **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979) ... Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977) ... **What this PR does / why we need it**: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. **Which issue(s) this PR fixes**: Fixes #5978 **Special notes for your reviewer**: n/a **Checklist** - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Lauder	e125252ee7	lambda-promtail: allow numbers on LB name regex (#6961)	3 years ago
Jonathan	1b6d0bf3a0	feat: add kms ecryption (#6926)	3 years ago
Owen Diehl	ba5d218e3e	detects issuer via useragent in querytee reqs (#6914) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Simon	88e2705034	Update alpine and golang base images (#6897) ... * Upgrade base images to alpine 3.15.6 everywhere https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html * Update golang images to 1.18.5	3 years ago
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879) ... * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	3 years ago
Julien Duchesne	7b70d3d939	deb and rpm packaging for all binaries (#6456) ... * deb and rpm packaging for all binaries There is currently deb and rpm packages being published for logcli amd64 but they aren't signed In this PR: - Add RPM and deb packages for `logcli`, `loki-canary` `loki` and `promtail` for arm, arm64 and amd64 - Sign them with the Grafana GPG key (packages.grafana.com/gpg.key). This will be the same key for all Grafana products - Add CI steps to test the RPM and deb packages. This launches a docker image with systemd installed, then it installs both loki and promtail and does a `logcli` query on the Loki instance * Remove specific version in test scripts * Bump build image to 0.22.0 * Add restart policy for promtail's service	3 years ago
Karsten Jeschkies	51b09d771f	Open access to Minio console in dev setup. (#6807)	3 years ago
Sandeep Sukhani	2e95a64cb4	query-tee: support for multi backend response comparison and disabling query proxy for select backends (#6736) ... * add support for comparing responses with multiple secondary backends * allow disabling read proxy for select secondary backends * Update tools/querytee/proxy.go Co-authored-by: Salva Corts <salva.corts@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	4 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704) ... * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	4 years ago
Kaviraj Kanagaraj	0d99d80acd	upgrade `go:1.17.9` to `go:1.18.4` (#6726)	4 years ago