3256 Commits (46956d0bf6367aa320de22eb42345b3edb0166ac)

 

All Branches   

Search

Author	SHA1	Message	Date
Sandeep Sukhani	40e05198f9	fix a panic in index-gateway caused by double closing of a channel (#5298)	3 years ago
Christian Haudum	3fb572abda	Fix Promtail watching deadlock (#5283) ... * Reuse channel from fileEventWatcher map if already exists Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove sync() call in FileTarget constructor The constructor `NewFileTarget` is called from within a locked code block by the FileTargetManager. However, the constructor also calls `sync()` which starts and stops watching new directories. These methods `startWatching` and `stopWatching` both send to the `targetEventHandler` channel, causing the `FileTargetManager` to receive the event in a goroutine (`watch()` function) running in the background. Handling the event calls `sendFileCreateEvent` on the syncers, which subsequently tries to write to the `fileEventWatcher` channel which is never read because of the locked mutex. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sashank Agarwal	c88734b72c	Link `replace` pipeline stage to doc (#5200)	3 years ago
Karsten Jeschkies	b92b544ca2	Write Promtail position file atomically. (#5266) ... Fixes #5218	3 years ago
Karsten Jeschkies	54bc2dc40e	Restart Docker target if stopped. (#5280) ... Summary: The Docker target would not reconnect to a container if it restarted and kept the same ID. This should introduces a `startIfNotRunning` method that will be called to restart scraping the container's logs. Fixes #5259	3 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281) ... * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284) ... * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	96ff7f900b	Revert "Adding a docs page that helps users understand memberlist a bit more (#5186)" (#5286) ... This reverts commit 7e372cbc56.	3 years ago
Dylan Guedes	ad3c8d07b7	Cleanup `util/validation` code. (#5285) ... * Cleanup `util/validation` code. * Remove `errors.go` file.	3 years ago
Ed Welch	efffe0ce50	add some timing to the logs for downloading and opening index tables (#5042) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	57b384df8f	Auto-expire old items from FIFO cache (#5148) ... * Remove validity from FIFO cache We can remove the validity setting of 1h in 2.4 for the chunks cache because it doesn't reduce memory usage and instead leads to valid chunks being ignore when querying the cache. Chunks are immutable so it doesn't make sense to not return them if they are present in the cache. Closes #4922 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add periodic task to prune old FIFO cache elements As described in #4921 this PR adds a periodic task that prunes expired items from the FIFO cache to free up memory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Jennifer Villa	7e372cbc56	Adding a docs page that helps users understand memberlist a bit more (#5186) ... * Added some new docs on memberlist. * Added a page with more memberlist info.	3 years ago
Trevor Whitney	926a0b2a4e	Update local configs to fix issues with incorrect addresses (#5207) ... * Update local configs to fix issues with incorrect addresses * Add back instance_addr for local config	3 years ago
Dan Poltawski	379ca6b409	Promtail docs: Fix link to `unpack` LogQL documentation (#5274)	3 years ago
Christian Haudum	8d2b3f7aa5	Return early if push payload does not contain data (#5229) ... Optimize for: * payload does not contain any streams * a stream does not contain any entries * none of the streams contain any entries Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	531905c066	Remove `cortex` references from `go.mod` and makefile (#5269) ... * Remove cortex references from `go.mod` and makefile Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove from `modules.txt` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	7856db62bb	Copy `cortex/pkg/storage/bucket` package dependency into Loki (#5258) ... * Fork cortex `storage/bucket` into Loki. * Replace storage/bucket usage from Cortex to Loki one. * Remove redundant error check. * Revert SwiftConfig struct and remove cortex pkgs alias. * Add missing test files. * Replace `cortex/pkg/util` with our own implementation. * Remove unused cortex packages.	3 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Lydia Lee	a4cfab4b84	feat(promtail): fix TargetManager.run() not exit (#5238)	3 years ago
Kaviraj Kanagaraj	f083aab09d	Remove `cortexpb` dependency from Loki packages. Use only `logproto` now. (#5264) ... * Getting rid of `cortexpb` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Get rid of `cortex/pkg/tenants` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * go mod vendor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename some import alias Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Ed Welch	09a5a2cee3	Logstash: Add config option which allows seting up an allowlist for labels to be mapped to Loki (#5244) ... * add a config called `include_fields` which lets you allowlist a specific set of fields which will be mapped to labels and sent to Loki. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * restore path for Dockerfile so it works in CI Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Michel Hollands	87edab0867	Ingester: remove the labelFilter (#5262) ... * Remove the labelFilter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Sandeep Sukhani	37fcbf9428	use default retention period to check user index may have expired chunks when user does not have custom retention (#5261)	3 years ago
Cyril Tovena	cde9a711f3	fluent-bit: Attempt to unmarshal nested json. (#5223) ... * fluent-bit: Attempt to unmarshal nested json. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Also support nested json array. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Owen Diehl	91d837e79c	Refactor/remove global splitby (#5243) ... * cleans up internal limits struct, removes unnecessary logic * moves querier.split-queries-by-interval to limits, simplifies resulting code. * docs, changelog * flag supports model.duration instead of time.duration * lint and adds new error for deprecated field * reregisters queryrange flags * comment lint * ensures we validate queryrange config and simplifies embedded struct validate() * 0 still disables * simplify cache key generation and protect against divide by zero	3 years ago
Dylan Guedes	3d135e5338	Copy `cortex/pkg/configs` package dependency into Loki (#5139) ... * Fork cortex `configs` into Loki. * Use new forked `configs` package. * Remove duplicated import. * Fix lint issues. * Use new forked util/logger. * Drop support for promql rules v1. - This allow us to remove the `legacy_promql` implementation - If a rulesconfig is received in v1 format, we output a specific error, so we can differentiate between a unknown version and a v1 one. * Modify tests to expect error for legacy rules format. * Add changelog and upgrading guide. - Added them as we dropped support for legacy promql rules configuration which might be a breaking change for some environments. * Update vendor/modules after rebasing.	3 years ago
Dylan Guedes	4a70f8ec84	Copy `cortex/pkg/ingester/client` package dependency into Loki (#5164) ... * Fork ingester client. * Document magic numbers.	3 years ago
Dylan Guedes	ba6a2fd41e	Get rid of Cortex chunk. (#5239)	3 years ago
3JIou_from_home	863408684c	Push: add deflate compression in post requests (#5249) ... * deflate-compression-5247 - add deflate compression in post requests * deflate-compression-5247 - add deflate tests by example gzip	3 years ago
Eng Zer Jun	392fde1e27	refactor: use `T.TempDir` to create temporary test directory (#5253) ... * refactor: use `T.TempDir` to create temporary test directory The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: update CHANGELOG.md Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Sandeep Sukhani	9fdcacff15	v12 chunk format handling in retention (#5254) ... Also add some tests and safety mechanism to avoid dropping whole table when we cant find any chunks	3 years ago
Travis Patterson	8646436b18	Fix docker plugin build (#5240) ... - Rebuild the rootfs before each plugin build	3 years ago
Michel Hollands	a3fcd3cc75	Update Go version to latest and bump build image version (#5250) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Florian Klink	ca2132d7cb	Update _index.md (#5241) ... We also fixes typo.	3 years ago
Roger Steneteg	bc88f298cb	skip glob search if path is an existing file and not a directory (#5246) ... Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	3 years ago
RangerCD	87cbe50b59	Lock consistencyMtx outside locking chunkMtx (#5210) ... * Lock consistencyMtx outside locking chunkMtx * Remove unnecessary RLock	3 years ago
Nabeel Saad	d0c6e3d016	Update _index.md (#5159) ... Adding default value for line_format so customers know which is picked up if the option is not provided.	3 years ago
RangerCD	4ee686c22c	Remove ErrStreamMissing (#5234)	3 years ago
JordanRushing	849f45ea8c	Update loki-mixin gcs queries in the Loki operational dashboard (#5232) ... Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com>	3 years ago
Callum Styan	bfef7bad44	Start refactor of storage (chunks) clients metrics into structs (#5057) ... * Refactor azure metrics to be a struct (+ add an overall client metrics struct), add an azure client metric for total egress bytes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add changelog entry for new azure metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixes based on Kavi's review. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix one more merge conflict that I missed in the merge commit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Kaviraj Kanagaraj	7713ff78e2	Fix `cortexpb` -> `logproto` rename in some tests (#5231) ... * Fix `cortexpg` -> `logproto` rename in some tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix TimestampMs rename Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * queryrange test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests on queryrangebase pacakage\ Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix some typos Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Callum Styan	f24b84c15a	[querier] Abort a query in limitedRoundTripper if parallelism is 0 (#5190) ... * Abort a query in limitedRoundTripper if parallelism is 0. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests related to fakeLimits change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change limitedRoundTripper parallelism error message. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Cyril Tovena	fa25e54e26	Bump logstash plugin gem (#5230) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Karsten Jeschkies	4716eea0db	Provide Docker target and discovery in Promtail. (#4911) ... **What this PR does / why we need it**: This patch adds support to fetch Docker container logs through the Docker daemon API. This should be more robust than the Loki Docker driver or scraping the logs files. The new Docker target will also collect meta information of the scraped containers. **Which issue(s) this PR fixes**: Addresses #2361 Closes #4703 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Christian Haudum	2e6e050453	Improve docs on how to install logcli from source (#5126) ... * Improve docs on how to install logcli from source Closes #5108 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Improve docs on how to install logcli from source Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	db283da088	Add `MaxSurge` and `MaxUnavailable` strategy to all Loki k8 workloads. (#5227) ... * Add `MaxSurge` and `MaxUnavailable` strategy to all loki k8 workloads. This fixes couple of issues. 1. By default these configs are 25% in k8, meaning during rollout 25% of pods are allowed to shutdown immediately. 2. Due to (1), during graceful shutdown process, 25% of all the pods access consul to `unregister()` from shared key value. (2) makes CAS rate of underlying KV store high (leads to lots of retry and failing) sometimes failing to unregister leaving the ring "unhealthy" Also this PR make these configs consistent across all k8 workloads. More details: https://github.com/grafana/dskit/issues/117 * Remove it from statefulset workloads Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	3f496e7336	Copy `cortex/pkg/cortexpb` package dependency into Loki (#5151) ... * Fork cortexpb into Cortex `pkg/logproto`. * Use `pkg/logproto` instead of Cortex `pkg/cortexpb`. * Rename `cortexpb` usage to `logproto`. * Fix encoding/decoding test. * Replace cortexpb usage in the ruler/base package. * Remove exemplars from logproto.	3 years ago
Dylan Guedes	3f78d876ff	Copy `cortex/pkg/prom1` package dependency into Loki (#5196) ... * Fork cortex `prom1`. * Use forked `prom1` instead of Cortex one.	3 years ago
aengusrooneygrafana	c9d49cea6f	Update to Unofficial clients for JS (#5180) ... Came across these packages for JS with help from Danny - I thought they would be useful to include in our docs for posterity. CC @dannykopping for info!	3 years ago