128 Commits (2f192bbb25131b1b9b414973de3e778578c84e84)

Author	SHA1	Message	Date
Conor Evans	3d49c09e34	fix(lambda-promtail): update default lambda_vpc_subnets / lambda_vpc_security_groups values to be appropriate empty list(string) value (#6453) ... Signed-off-by: Conor Evans <coevans@tcd.ie>	3 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537) ... * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	3 years ago
Joan López de la Franca Beltran	bf3bca3e99	tools: add lambda-promtail missing errchecks (#6541) ... * tools: lambda-promtail missing errchecks * undo undesired changes	3 years ago
Julien Duchesne	6a9557a1e0	Add Drone pipeline to test packaging (#6482) ... * Add Drone pipeline to test packaging Currently, the packaging process only happens on tags, which could lead to surprises Especially considering I'm working on deb and RPM packaging (lots more steps than now) I also moved the publishing from Circle to Drone because: 1. I'll need some new secrets from Vault 2. It should be faster * Fix make target * Remove `-parallel` arg for `gox`	3 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102) ... * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	3 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049) ... * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Travis Patterson	03153e89bf	clean up the pr diff message (#6098) ... * clean up the pr diff message * add dependency to clone-main to avoid possible race * don't start tests before everything is cloned * formatting change	3 years ago
Travis Patterson	d97724e8dd	Introduce coverage to PR pipelines (#5357) ... * Add code coverage diff to CI * remove test package from coverage diff * review feedback * lint * Env variable test * faster experiment * drone vars * drone vars * how does the shell work? * add pr comment * escape json * report diff in comment * properly format json * quote post body * review feedback * add querier/queryrange to coverage	3 years ago
Christian Simon	084a0804f1	docker-compose: Disable auth in Grafana (#5998)	3 years ago
Christian Haudum	b3bc8d3063	Other: Improve check-drone-drift script (#5925) ... * Improve check-drone-drift script Instead of checking the line count of the current branch an main, run `make drone` and check if there are changes to the target `.drone/drone.yml` file. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Generate .drone/drone.yml using `make drone` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Generate .drone/drone.yml using `make drone` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Simon	9d0fd3a7e3	Upgrade golang version to v1.17.9 (#5899) ... ``` git ls-files '*[Dd]ockerfile*' | grep -v ^vendor/ | xargs sed -i "s/golang:[0-9\\.]*/golang:1.17.9/g" | ag 'golang:' ```	3 years ago
Cyril Tovena	8c1095a554	Moves TSDB to the stores package. (#5852) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cristian Greco	02876514b3	lambda-promtail: fix typos in CloudFormation and Terraform files. (#5749) ... This fixes a typo in the CloudFormation template file (`Descripton`) which makes it unusable, as well as some other minor typos.	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833) ... * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Ed Welch	52bc0c64ae	2.5 changelog, release notes, and upgrade guide (#5819) ... * update changelog Signed-off-by: Ed Welch <edward.welch@grafana.com> * update the upgrade guide Signed-off-by: Ed Welch <edward.welch@grafana.com> Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * add release notes Signed-off-by: Ed Welch <edward.welch@grafana.com> Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * udpate things to 2.5.0 Signed-off-by: Ed Welch <edward.welch@grafana.com> * Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/release-notes/v2-5.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * cleanout wierd changelog union stuff Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Trevor Whitney	7b214e2ea6	fix missing file during rpm and deb packaging step (#5818)	3 years ago
Owen Diehl	0c826b834a	Tsdb/compaction (#5783) ... * ChunkMetas.finalize uses pool * slight refactoring. Uses pools during chunkmetas.finalize and adds an InclusiveBounds helper * multi idx explicitly wraps tsdb files * TSDB Checksum helper + requires at least 2 indices for MultiIndex constructor * tsdb compaction impl * skips compacting a single file * compaction + testing * tsdb compaction tests * passes reference to &index.ChunkMetasPool for avoiding nocopy * linting * gitignore .vscode * s/Combine/FilePath/g	3 years ago
Karsten Jeschkies	7193f540c2	Return bad request for multi-tenant tail. (#5792) ... Closes #5753	3 years ago
Christian Simon	3d941ccccb	Update alpine images to 3.15.4 (#5780)	3 years ago
Ed Welch	b136d0dab3	Increase flush_op_timeout to 10m, 10s is often much too short for a big WAL replay which can generate a lot of chunks very quickly. (#5768) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	51fbc95bbd	fixes existing test case and adds more (#5687)	3 years ago
Owen Diehl	3f28a33d93	TSDB Index reuses slices, adds pools (#5630) ... * adds a pool for ChunkMetas * index takes slice pointers for allocation reasons and adds pools for all index types * reuse slices in tsdb benchmarks * adds count to bench script * properly returns series to pool * more ergonomic index signatures while still supporting slice reuse * tsdb index documentation * aligns tsdb-map tooling with new index signatures	3 years ago
Owen Diehl	d084dc7f4b	Tsdb/hash order (#5631) ... * encode fingerprint into series in tsdb * uses series fingerprint in index reads * order tsdb series by fprint * tsdb writes fingerprint offsets table * shardannotation is part of tsdb/index, fixes testware * loads fingerprint offsets from tsdb * removes unnecessary sortedpostings * adds prefix sharding logic * sharded tsdb postings * updates benchmark to pass nil hash * runs tsdb benches with variable count	3 years ago
Owen Diehl	ec809e665f	Tsdb/sharding-benchmarks (#5628) ... * adds shard to tsdb/index pkg * add sharded tsdb benchmark	3 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620) ... **What this PR does / why we need it**: The original change injected the user id when the org id was required to fetch all data. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#230 **Special notes for your reviewer**: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1|2" --data-urlencode 'query={compose_service=~"log-gen.*"}' | jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600) ... * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	3 years ago
Owen Diehl	67f961e4dc	more tsdb benchmarks + updates script to work with incompatible branches (#5590) ... * more tsdb benchmarks + updates script to work with incompatible branches * shellcheck	3 years ago
Owen Diehl	614912181e	updates clusterrole and clusterrolebinding apiVersions in promtail.sh (#5547)	3 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Thomas Freyss	9d1f351713	Lambda-promtail: add basic auth support (#5503) ... * fix/lambda-promtail: re-add basic auth support Feature got lost in 699fffe9e5 refactoring. Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com> * fix/lambda-promtail: remove non existant make target Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com>	3 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	3 years ago
Salva Corts	651ac093b6	Fix circular dependencies (#5420)	3 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	3 years ago
Jonathan Bédard Schami	776d5c41c7	Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value (#5359) ... * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.	3 years ago
Karsten Jeschkies	ec1ae42083	Verify that drone.yml was updated. (#5396)	3 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065) ... * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	3 years ago
Trevor Whitney	c477e29c3f	Add tooling for local k3d clusters under tools/dev (#5272) ... * add k3d folder with loki-distributed env * add readme * shellcheck create_cluster.sh * lint jsonnet * Update grafana image to latest	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Dylan Guedes	66bf9f6474	Copy `cortex/tools/querytee` package dependency into Loki (#5165) ... * Fork Cortex `tools/querytee`. * Use forked querytee instead of Cortex one. * Add `loki/tools` to list of possible paths. * Add opts to querytee proxy.	3 years ago
Karsten Jeschkies	b03e2783d6	Package logcli as rpm and deb. (#4667) ... * Package logcli as rpm, deb and apk. * Package logcli on release builds.	3 years ago
Ed Welch	34a4b242bf	Update to 2.4.2 release (#5121) ... * update project to 2.4.2 release Signed-off-by: Edward Welch <edward.welch@grafana.com> * go.mod cleanup Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Susana Ferreira	2fdd24038b	Move querier.split-queries-by-interval to a per-tenant configuration (#5023)	3 years ago
Hiroki Sakamoto	1557dab34f	Add SASL&mTLS authentication support for Kafka in Promtail (#4663) ... * Add xdg-go module * Add mTLS, SASL/PLAIN, SASL/SCRAM authentication support * docs: Add authentication configuration * Add test tools for kafka authentication * Add test stacks for SSL, SASL/PLAIN, SASL/SCRAM, SASL over TLS authentication * Fix shelllint	4 years ago
Ed Welch	413ffab235	update everything for 2.4.1 (#4701)	4 years ago
Ed Welch	fbebf67588	Loki: check new Read target when initializing boltdb-shipper store (#4681) ... * make sure to check for new `Read` target when setting up boltdb-shipper * also need to make sure ingester querier is added as a dep for Read target * improve the comment on why we shorten the cache validity. * fix shellcheck errors * apply smart defaults to chunk retain period to avoid query gaps. * typo	4 years ago
Ed Welch	5602a49489	update references to new 2.4.0 image (#4672)	4 years ago
Cyril Tovena	c8f3df33cf	Promtail Kafka target (#4568) ... * Adds a kafka target manager in promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar test for the fanout client. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * final adjustement Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding topics discovery. Still needs to finish tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off testing it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Revert config changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add group id as discovered label Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tools for running kafka and testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted shell Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update sarama to compile in ARM. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for kafka target. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve code comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add a few s's * Better cancellation support. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * adds `__meta_kafka_` suffix to discovered labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Callum Styan	ea78cde659	Add drone build job for lambda-promtail images. (#4480) ... * Add drone build job for lambda-promtail images. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix image tag portion of drone job for lambda-promtail build Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review feedback. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Regenerate drone yaml file after some changes. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	1493278ff2	Add a a parameter to keep/drop the stream label from cloudwatch. (#4494) ... * Add a a parameter to keep/drop the stream label from cloudwatch. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Cyril's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure the keep stream value is properly passed in as an environment variable when deploying via CloudFormatioEnsure the keep stream value is properly passed in as an environment variable when deploying via CloudFormationn Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago