apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	b23687ad96	update regex for matching nlb logs while using lambda promtail (#7305 ) What this PR does / why we need it: Regex in lambda promtail looks for specific file patterns for reading and pushing logs from it. It currently does not read logs generated by nlb i.e having `_net` in the name. This PR updates the regex. Cross checked it with was docs here https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212 ) The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Trevor Whitney	afd63c5981	[helm] Add Loki Canary to Helm Chart (#7173 ) What this PR does / why we need it: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. Which issue(s) this PR fixes: Fixes #7019 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Trevor Whitney	4b6df86333	Add local k3d environment for enterprise-logs (#7146 ) What this PR does / why we need it: This adds a local k3d based development cluster for testing out the enterprise logs side of the helm chart, which can be enabled by setting: ``` enterprise: enabled: true ``` It requires both the `gex-plugins` and `enterprise-logs` repos to be checked out locally (though the latter soon will not be needed once the provisioner is published). We need this environment when developing enterprise features for the helm chart.	3 years ago
Jonathan	6b9bf0b5f4	feat: add kms and ec2 permission (#7058 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR will help to increase a necessary policy (kms and ec2) to resource log group and lambda. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Karsten Jeschkies	c23e541fe7	Generate Helm chart README. (#7012 ) What this PR does / why we need it: The Helm CI would break because the Helm chart README was not generated. This change * updates the README * introduces a filter that will only run the Helm CI on changes to the Helm chart directory * removes trailing whitespace * corrects the Minio chart repository	3 years ago
Karsten Jeschkies	1c3f5d0985	Helm: Upgrade Minio chart dependency to 4.0.12. (#6974 ) What this PR does / why we need it: Changes to a supported Minio chart. Minio is also enabled by default. Which issue(s) this PR fixes: Fixes #6973 Checklist - [x] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979 ) Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977 ) What this PR does / why we need it: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. Which issue(s) this PR fixes: Fixes #5978 Special notes for your reviewer: n/a Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Lauder	e125252ee7	lambda-promtail: allow numbers on LB name regex (#6961 )	3 years ago
Jonathan	1b6d0bf3a0	feat: add kms ecryption (#6926 )	3 years ago
Owen Diehl	ba5d218e3e	detects issuer via useragent in querytee reqs (#6914 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Simon	88e2705034	Update alpine and golang base images (#6897 ) * Upgrade base images to alpine 3.15.6 everywhere https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html * Update golang images to 1.18.5	3 years ago
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879 ) * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	3 years ago
Julien Duchesne	7b70d3d939	deb and rpm packaging for all binaries (#6456 ) * deb and rpm packaging for all binaries There is currently deb and rpm packages being published for logcli amd64 but they aren't signed In this PR: - Add RPM and deb packages for `logcli`, `loki-canary` `loki` and `promtail` for arm, arm64 and amd64 - Sign them with the Grafana GPG key (packages.grafana.com/gpg.key). This will be the same key for all Grafana products - Add CI steps to test the RPM and deb packages. This launches a docker image with systemd installed, then it installs both loki and promtail and does a `logcli` query on the Loki instance * Remove specific version in test scripts * Bump build image to 0.22.0 * Add restart policy for promtail's service	3 years ago
Karsten Jeschkies	51b09d771f	Open access to Minio console in dev setup. (#6807 )	3 years ago
Sandeep Sukhani	2e95a64cb4	query-tee: support for multi backend response comparison and disabling query proxy for select backends (#6736 ) * add support for comparing responses with multiple secondary backends * allow disabling read proxy for select secondary backends * Update tools/querytee/proxy.go Co-authored-by: Salva Corts <salva.corts@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	3 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704 ) * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	3 years ago
Kaviraj Kanagaraj	0d99d80acd	upgrade `go:1.17.9` to `go:1.18.4` (#6726 )	3 years ago
Vladyslav Diachenko	b39e4900ec	updated versions to the latest release v2.6.1 (#6703 )	4 years ago
Conor Evans	3d49c09e34	fix(lambda-promtail): update default lambda_vpc_subnets / lambda_vpc_security_groups values to be appropriate empty list(string) value (#6453 ) Signed-off-by: Conor Evans <coevans@tcd.ie>	4 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537 ) * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	4 years ago
Joan López de la Franca Beltran	bf3bca3e99	tools: add lambda-promtail missing errchecks (#6541 ) * tools: lambda-promtail missing errchecks * undo undesired changes	4 years ago
Julien Duchesne	6a9557a1e0	Add Drone pipeline to test packaging (#6482 ) * Add Drone pipeline to test packaging Currently, the packaging process only happens on tags, which could lead to surprises Especially considering I'm working on deb and RPM packaging (lots more steps than now) I also moved the publishing from Circle to Drone because: 1. I'll need some new secrets from Vault 2. It should be faster * Fix make target * Remove `-parallel` arg for `gox`	4 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102 ) * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	4 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049 ) * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Travis Patterson	03153e89bf	clean up the pr diff message (#6098 ) * clean up the pr diff message * add dependency to clone-main to avoid possible race * don't start tests before everything is cloned * formatting change	4 years ago
Travis Patterson	d97724e8dd	Introduce coverage to PR pipelines (#5357 ) * Add code coverage diff to CI * remove test package from coverage diff * review feedback * lint * Env variable test * faster experiment * drone vars * drone vars * how does the shell work? * add pr comment * escape json * report diff in comment * properly format json * quote post body * review feedback * add querier/queryrange to coverage	4 years ago
Christian Simon	084a0804f1	docker-compose: Disable auth in Grafana (#5998 )	4 years ago
Christian Haudum	b3bc8d3063	Other: Improve check-drone-drift script (#5925 ) * Improve check-drone-drift script Instead of checking the line count of the current branch an main, run `make drone` and check if there are changes to the target `.drone/drone.yml` file. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Generate .drone/drone.yml using `make drone` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Generate .drone/drone.yml using `make drone` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Christian Simon	9d0fd3a7e3	Upgrade golang version to v1.17.9 (#5899 ) ``` git ls-files '[Dd]ockerfile' \| grep -v ^vendor/ \| xargs sed -i "s/golang:[0-9\\.]*/golang:1.17.9/g" \| ag 'golang:' ```	4 years ago
Cyril Tovena	8c1095a554	Moves TSDB to the stores package. (#5852 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cristian Greco	02876514b3	lambda-promtail: fix typos in CloudFormation and Terraform files. (#5749 ) This fixes a typo in the CloudFormation template file (`Descripton`) which makes it unusable, as well as some other minor typos.	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	52bc0c64ae	2.5 changelog, release notes, and upgrade guide (#5819 ) * update changelog Signed-off-by: Ed Welch <edward.welch@grafana.com> * update the upgrade guide Signed-off-by: Ed Welch <edward.welch@grafana.com> Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * add release notes Signed-off-by: Ed Welch <edward.welch@grafana.com> Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * udpate things to 2.5.0 Signed-off-by: Ed Welch <edward.welch@grafana.com> * Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * cleanout wierd changelog union stuff Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Trevor Whitney	7b214e2ea6	fix missing file during rpm and deb packaging step (#5818 )	4 years ago
Owen Diehl	0c826b834a	Tsdb/compaction (#5783 ) * ChunkMetas.finalize uses pool * slight refactoring. Uses pools during chunkmetas.finalize and adds an InclusiveBounds helper * multi idx explicitly wraps tsdb files * TSDB Checksum helper + requires at least 2 indices for MultiIndex constructor * tsdb compaction impl * skips compacting a single file * compaction + testing * tsdb compaction tests * passes reference to &index.ChunkMetasPool for avoiding nocopy * linting * gitignore .vscode * s/Combine/FilePath/g	4 years ago
Karsten Jeschkies	7193f540c2	Return bad request for multi-tenant tail. (#5792 ) Closes #5753	4 years ago
Christian Simon	3d941ccccb	Update alpine images to 3.15.4 (#5780 )	4 years ago
Ed Welch	b136d0dab3	Increase flush_op_timeout to 10m, 10s is often much too short for a big WAL replay which can generate a lot of chunks very quickly. (#5768 ) Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago
Owen Diehl	51fbc95bbd	fixes existing test case and adds more (#5687 )	4 years ago
Owen Diehl	3f28a33d93	TSDB Index reuses slices, adds pools (#5630 ) * adds a pool for ChunkMetas * index takes slice pointers for allocation reasons and adds pools for all index types * reuse slices in tsdb benchmarks * adds count to bench script * properly returns series to pool * more ergonomic index signatures while still supporting slice reuse * tsdb index documentation * aligns tsdb-map tooling with new index signatures	4 years ago
Owen Diehl	d084dc7f4b	Tsdb/hash order (#5631 ) * encode fingerprint into series in tsdb * uses series fingerprint in index reads * order tsdb series by fprint * tsdb writes fingerprint offsets table * shardannotation is part of tsdb/index, fixes testware * loads fingerprint offsets from tsdb * removes unnecessary sortedpostings * adds prefix sharding logic * sharded tsdb postings * updates benchmark to pass nil hash * runs tsdb benches with variable count	4 years ago
Owen Diehl	ec809e665f	Tsdb/sharding-benchmarks (#5628 ) * adds shard to tsdb/index pkg * add sharded tsdb benchmark	4 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620 ) What this PR does / why we need it: The original change injected the user id when the org id was required to fetch all data. Which issue(s) this PR fixes: Fixes grafana/loki-private#230 Special notes for your reviewer: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1\|2" --data-urlencode 'query={compose_service=~"log-gen."}' \| jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. Checklist* - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600 ) * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	4 years ago
Owen Diehl	67f961e4dc	more tsdb benchmarks + updates script to work with incompatible branches (#5590 ) * more tsdb benchmarks + updates script to work with incompatible branches * shellcheck	4 years ago
Owen Diehl	614912181e	updates clusterrole and clusterrolebinding apiVersions in promtail.sh (#5547 )	4 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Thomas Freyss	9d1f351713	Lambda-promtail: add basic auth support (#5503 ) * fix/lambda-promtail: re-add basic auth support Feature got lost in `699fffe9e5` refactoring. Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com> * fix/lambda-promtail: remove non existant make target Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com>	4 years ago

1 2 3

148 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)