101 Commits (614912181e6f3988b2b22791053278cfb64e169c)

Author	SHA1	Message	Date
Owen Diehl	614912181e	updates clusterrole and clusterrolebinding apiVersions in promtail.sh (#5547)	3 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Thomas Freyss	9d1f351713	Lambda-promtail: add basic auth support (#5503) ... * fix/lambda-promtail: re-add basic auth support Feature got lost in 699fffe9e5 refactoring. Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com> * fix/lambda-promtail: remove non existant make target Signed-off-by: Thomas Freyss <2027875+IppX@users.noreply.github.com>	3 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	3 years ago
Salva Corts	651ac093b6	Fix circular dependencies (#5420)	3 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	3 years ago
Jonathan Bédard Schami	776d5c41c7	Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value (#5359) ... * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.	3 years ago
Karsten Jeschkies	ec1ae42083	Verify that drone.yml was updated. (#5396)	3 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065) ... * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	3 years ago
Trevor Whitney	c477e29c3f	Add tooling for local k3d clusters under tools/dev (#5272) ... * add k3d folder with loki-distributed env * add readme * shellcheck create_cluster.sh * lint jsonnet * Update grafana image to latest	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Dylan Guedes	66bf9f6474	Copy `cortex/tools/querytee` package dependency into Loki (#5165) ... * Fork Cortex `tools/querytee`. * Use forked querytee instead of Cortex one. * Add `loki/tools` to list of possible paths. * Add opts to querytee proxy.	3 years ago
Karsten Jeschkies	b03e2783d6	Package logcli as rpm and deb. (#4667) ... * Package logcli as rpm, deb and apk. * Package logcli on release builds.	3 years ago
Ed Welch	34a4b242bf	Update to 2.4.2 release (#5121) ... * update project to 2.4.2 release Signed-off-by: Edward Welch <edward.welch@grafana.com> * go.mod cleanup Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Susana Ferreira	2fdd24038b	Move querier.split-queries-by-interval to a per-tenant configuration (#5023)	3 years ago
Hiroki Sakamoto	1557dab34f	Add SASL&mTLS authentication support for Kafka in Promtail (#4663) ... * Add xdg-go module * Add mTLS, SASL/PLAIN, SASL/SCRAM authentication support * docs: Add authentication configuration * Add test tools for kafka authentication * Add test stacks for SSL, SASL/PLAIN, SASL/SCRAM, SASL over TLS authentication * Fix shelllint	4 years ago
Ed Welch	413ffab235	update everything for 2.4.1 (#4701)	4 years ago
Ed Welch	fbebf67588	Loki: check new Read target when initializing boltdb-shipper store (#4681) ... * make sure to check for new `Read` target when setting up boltdb-shipper * also need to make sure ingester querier is added as a dep for Read target * improve the comment on why we shorten the cache validity. * fix shellcheck errors * apply smart defaults to chunk retain period to avoid query gaps. * typo	4 years ago
Ed Welch	5602a49489	update references to new 2.4.0 image (#4672)	4 years ago
Cyril Tovena	c8f3df33cf	Promtail Kafka target (#4568) ... * Adds a kafka target manager in promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar test for the fanout client. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * final adjustement Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding topics discovery. Still needs to finish tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off testing it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Revert config changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add group id as discovered label Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tools for running kafka and testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted shell Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update sarama to compile in ARM. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for kafka target. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve code comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add a few s's * Better cancellation support. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * adds `__meta_kafka_` suffix to discovered labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Callum Styan	ea78cde659	Add drone build job for lambda-promtail images. (#4480) ... * Add drone build job for lambda-promtail images. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix image tag portion of drone job for lambda-promtail build Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review feedback. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Regenerate drone yaml file after some changes. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	1493278ff2	Add a a parameter to keep/drop the stream label from cloudwatch. (#4494) ... * Add a a parameter to keep/drop the stream label from cloudwatch. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Cyril's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure the keep stream value is properly passed in as an environment variable when deploying via CloudFormatioEnsure the keep stream value is properly passed in as an environment variable when deploying via CloudFormationn Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	096cf0cfaf	Update golang and loki-build-image image versions. (#4481) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	05bb3cec4c	Rewrite lambda-promtail to use subscription filters. (#4315) ... * Rewrite lambda-promtail to use subscription filters. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update lambda library vendoring. * Fix small terraform comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Allow for lambda-promtail binary to pull basic auth credentials via env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * remove mistakenly committed binary and zip Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update terraform to allow passing of variousn configuration values, like the write endpoint and log groups to subscribe to. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update cloudformation template to also use parameters/env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * don't show password in template parameters + include examples of deploy commands in readme Signed-off-by: Callum Styan <callumstyan@gmail.com> * We can use terraform vars, with empty defaults, to allow for specifying VPC config for promtail-lambda deployment. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Ed Welch	e1aea518ce	update jsonnet for 2.3.0 (#4109)	4 years ago
Karsten Jeschkies	c3aaebe509	Support frontend V2 with query scheduler. (#4071) ... * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Remove unused querier option. * Use new query-scheduler module in Loki. * Remove unused option. * Document `max_queriers_per_tenant`.	4 years ago
Daniel Wright	c00af4d38a	Update lambda-promtail documentation example (#4098) ... Closes #4097	4 years ago
Karsten Jeschkies	5a5bcb16e2	Document 0 meaning for retries. (#4096)	4 years ago
Karsten Jeschkies	17f4a73a24	Filter instant queries and shard them. (#3984) ... * Filter instan queries and shard them. * [WIP] Test instant query sharding * Trace casting error. * WRap LokiInstantRequest for Params interface. * Convert paras to request. * Convert vector. * Use proper query time stamp. * Assert number of calls. * Convert vector to multiple samples. * Format code. * Use type switch. * Format code. * Rename Matrix to proto conversion. * Remvoe comment on caching. * Update pkg/querier/queryrange/roundtrip_test.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove nil middlware. * Handle LokiInstantRequest correctly. * Assert reponse type. * Return LokiPromResponse. * Verify err in tests. * Set shards. * Correct result type. * Use limits middleware. * Use shars parameter in querier. * Return vector. * Unify params. * Change direction to FORWARD. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Lize Cai	8370ed3b04	fix typo in loki docker-compose (#4016)	4 years ago
Cyril Tovena	86000dd320	Development Docker Compose Setup (#3941) ... * Adds the first version of a docker compose dev stack. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add index-gateway and code source. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better docker ignore. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better bash comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update tools/dev/loki-boltdb-storage-s3/README.md Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * lint bash scripts. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	4 years ago
Karsten Jeschkies	0cde54c4dc	Lint script files. (#3944) ... * Lint script files. This patch adds a Drone CI step that runs shellcheck on all Bash script files. * Install Bash in shellcheck container. * Fix a few shelcheck errors. * Fix build.sh * Fix remaining shellcheck errors.	4 years ago
Julien Duchesne	bbb82b5781	Deploy loki with Drone plugin instead of CircleCI job (#3708) ... We are deprecating the CircleCI job Also, I moved secrets to Vault instead of Drone repository secrets Note that this will require Flux in all environments that are updated Issue: grafana/deployment_tools#5307	4 years ago
Ed Welch	a9d85de4aa	Re-license to AGPLv3 (#3630) ... For a rationale as to why this was done, please see * https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/ * https://grafana.com/blog/2021/04/20/qa-with-our-ceo-on-relicensing/ Signed-off-by: Richard Hartmann <richih@richih.org> Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Brett Jones	f23aa6512a	add some tests	4 years ago
Ed Welch	9c470d4f9a	update for 2.2 release (#3477)	4 years ago
Ward Bekker	e383472d90	Don't auto-include pod labels as loki labels as a sane default (#3390) ... Don't auto-include pod labels as loki labels as a sane default. Folks regularly run into stream limits when they have a bunch of labels on their pods, so better to remove this to get them a nicer first onboarding experience.	4 years ago
Robert Fratto	3e4566daf9	remove unneeded prune-ci-tags drone job (#3412)	4 years ago
Owen Diehl	d4fdccbf6f	defaults promtail to 2.1.0 in install script (#3321)	4 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	2a596a77d7	Lambda-Promtail (#2282) ... * specs out lambda-promtail * init fn * udpates readme/template * lambda-promtail docs * lambda promtail includes source timestamp and uses context * non markdown links * new doc structure	5 years ago
Josh Soref	0d9cf4946a	Spelling (#2014) ... * spelling: access * spelling: actual * spelling: asynchronously * spelling: circleci * spelling: circle * spelling: cognitive * spelling: comment * spelling: computation * spelling: correct * spelling: daemon * spelling: elliott * spelling: enabled * spelling: encbuf * spelling: extracted * spelling: explicitly * spelling: independent * spelling: languages * spelling: loose * spelling: message * spelling: metrics * spelling: mismatch * spelling: nonexistent * spelling: output * spelling: pipeline * spelling: pipe line * spelling: position * spelling: prepend * spelling: query * spelling: receive * spelling: received * spelling: reducible * spelling: registry * spelling: response * spelling: retryable * spelling: scanning * spelling: sourced * spelling: supported * spelling: template * spelling: timeout * spelling: unsupported * spelling: adapter * spelling: deferring * spelling: expirable	5 years ago
Owen Diehl	fee58f1a71	delete tag script filters by prefix instead of substring (#1942)	5 years ago
Ed Welch	ab92ec8863	Update the CD deploy task name (#1780) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Edward Welch	c715f862db	strip the leading 'v' in front of docker images based on repo tags, grafana/loki:v1.4.0 -> grafana/loki:1.4.0 ... unpin the drone manifest plugin Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Robert Fratto	784c4ce232	ci: print error when getting tags fails (#1640) ... This commit also removes fatal logs. The pruning process isn't essential and should generally never cause the CI to report failure.	5 years ago
Robert Fratto	1fe5dc45b3	ci: prune docker tags prefixed with "master-" older than 90 days on merge (#1639)	5 years ago
Edward Welch	92eea517d3	updating to 1.3.0 release ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago