150 Commits (b3a2cec7c010bf1f5341a515e64c287ab3297e0e)

Author	SHA1	Message	Date
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842) ... The chunk package has been forked from this commit for future reference and licensing. 09fe17c0e6	4 years ago
Oleg Zaytsev	b4086dfe33	Upgrade Prometheus and Cortex (#4049) ... This bumps prometheus up to the changes from https://github.com/prometheus/prometheus/pull/9083 and Cortex to https://github.com/cortexproject/cortex/pull/4380 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	4 years ago
Arve Knudsen	3e385464c6	Chore: Upgrade Prometheus and Cortex (#4017) ... * Chore: Upgrade Prometheus Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Undo unnecessary changes Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Kaviraj	bfb24e6a4c	Ip matcher for LogQL (#3986) ... * IP matcher and filter via netaddr.IP package Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Adding IP filter into label filter gramer 1. ast tests 2. parse tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Label filters all test cases passes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename ipFilter -> ipLabelFilter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * More tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove testlogs.txt Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more cases to the benchmark Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. support NEQ 2. reduce allocation. 3. Minor tweaks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make IP filter work as line filter 1. Add necessary interface implementation 2. More tests for parser and ast Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Bug with filter matcher Its wierd bug. Not handing string() method breaks the parser :( Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Split Line and Label filter to different struct. It got out of hands to handle both in same struct. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add basic doc for IP matcher Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add docs feedback from Karen * Label filter add patter error report * PR remarks 1. Indendation fix 2. Remove string() interface for ip label filte * Linter happy * PR remaks * Fix ip label matcher string() method if pattern is invalid * Fix label filter tests * Have local copy of `pattern` in label filter to use in string() * Rebase conflicts * ast.go fixes	4 years ago
Cyril Tovena	fed9115627	Improve decoding of JSON responses. (#4019) ... * Improves frontend allocations Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves also labels Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests and review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Danny Kopping	ac63ea8500	Promtail: adding pipeline stage inspector (#4011) ... * Adding promtail pipeline stage inspector Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding library Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing old version of go-cmp to pass check-mod step Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making screenshot narrower and correcting link Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Christian Simon	2fd633cded	Restrict path segments in TenantIDs (#4020) ... Updates cortex dependency to prevent paths derived from TenantIDs to become vulnerable to path traversal attacks. CVE-2021-36156 CVE-2021-36157 Signed-off-by: Christian Simon <simon@swine.de>	4 years ago
Michel Hollands	c44c12b930	Update Cortex version (#3992) ... * Update Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Changed code due to new Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	23ad7f2e61	Unordered head block (#3957) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting	4 years ago
Trevor Whitney	de93a1bd54	Promtail: add consul agent service discovery (#3834) ... * working prototype Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * update test for new responses and metadata labels Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix linting issues Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * clarify license/copyright/attribution to prometheus code Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * undo changes to go.sum when adding consul dep Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add documentation around consul and consulagent scrape configs Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Bpaddyswk	7d666cceb2	lz4: update lz4 version to v4.1.7 to avoid possibly panic (#3835) ... * lz4: update lz4 version to v4.1.7 to avoid possibly panic * lz4: add lz4 v4.1.7 to vendor	4 years ago
Periklis Tsirakidis	3cc024a591	Fix minor glitches and tidy modules (#61)	4 years ago
Sashank Agarwal	32734d284e	added service monitor for ocp prometheus to scrape metrics (#58)	4 years ago
Sandeep Sukhani	d765921735	revendor cortex to latest master (#3771)	4 years ago
Gerard Vanloo	8415b37568	Creating telemetry metrics for loki operator based on complete build spec (#32)	4 years ago
Cyril Tovena	15d417efe1	Replace satori.uuid with gofrs/uuid (#3743) ... * Replace satori.uuid with gofrs/uuid Fixes #3742 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix new import. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672) ... Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	d88f3996ea	revendor compatibility: various prom+k8s+cortex (#3664) ... * revendor compatibility: various prom+k8s+cortex * fix logcli client instantiation * go mod tidy	4 years ago
Michel Hollands	9137ef921b	Update cortex to 1.8 (#3627) ... * Update Cortex to 1.8.1 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Brett Jones	f975a5dd2f	Add the scaffolding for tshirt sizing (#21)	4 years ago
Brett Jones	2ee3b4649c	LOG-1251/convert to statefulsets (#16)	4 years ago
Michel Hollands	f49a510f38	Loki: Update cortex version and fix resulting changes (#3532) ... * Update cortex version and fix resulting changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix scrapeconfig test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/promtail/scrapeconfig/scrapeconfig_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Remove unused fmt import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Brett Jones	f23aa6512a	add some tests	4 years ago
Brett Jones	3a80bdcddd	cleanup	4 years ago
Kaviraj	14b2c093c1	Add unique promtail_instance id to labels for gcptarget (#3501) ... * Add unique promtail_instance id to labels for gcptarget Rationale: To make labelset unique per promtail instance to avoid out-of-order errors * Update docs/sources/clients/promtail/scraping.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	4 years ago
Brett Jones	d301867259	working with struct definitions	4 years ago
Aditya C S	b1ea8ac050	support math functions in line_format and label_format (#3434) ... * support math functions in line_format and label_format * fix lint and tests * added docs and more tests * update doc * remove toString and int64 * doc adjustement. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Aditya C S	bf4fa6613a	support max_message_length configuration for syslog parser (#3400) ... * support max_message_length configuration for syslog parser * address review comments	4 years ago
Brett Jones	eeafa067b9	wip	4 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466) ... * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Brett Jones	803ff5890a	WIP	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446) ... * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Brett Jones	cc756b2e67	init	4 years ago
Cyril Tovena	7e9ac16dbf	Add fgfprof to Loki and Promtail. (#3424) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Brett Jones	af23e36bc7	init	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Cyril Tovena	72b3acb925	Windows events (#3246) ... * First commit for windows event targets. I had to add couple of new go modules and hook the windows log in. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the ability to save where we are. * Finished setup tests and comments. * nits * adding documentation * go tidy * lint windows * nope * add manager for non windows * don't lint forked package * mod check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/promtail/targets/windows/bookmark.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Use passed-in Prometheus registerer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * testing out windows drone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * type exec. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * docker image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go/go.exe Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * build windows on container. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * trying another image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo in image Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixed local timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests with timezone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes timezone tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update logstash since the last image is not working anymore 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Karsten Jeschkies	287244d110	Update vendored Cortex to 0976147451ee (#3267)	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256) ... * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215) ... * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163) ... * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083) ... * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Allenzhli	06de835f6e	update github.com/gorilla/websocket to fixes a potential DoS (#3152) ... Signed-off-by: allenzhli <allenzhli@tencent.com>	4 years ago
Cyril Tovena	f1c4ba2f59	Add zstd and flate compressions algorithms. (#3064) ... * Add zstd and flate compressions algorithms. zstd has the highest ratio of all while flate is very simalar to gzip. I'm adding those mainly for posterity and to keep them benchmarked. This also update gzip and lz4 library which both have been improved. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing vendor change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	4f62f36af7	Update cortex to 1 6 (#3131) ... * Update Cortex to recent master (1.5.0+, 35e698bb56d6). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore set scheduler address. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make linter happy. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Upgrade cortex to 1.6 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Work around issues in go-openapi libraries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clean up go.mod and go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	4 years ago
Keith Byrne	5793c4964e	Configuration: Support environment expansion in configuration (#2837) ... * Support environment expansion in configuration Inspired by https://github.com/cortexproject/cortex/pull/2147 Fixes #2311 Adds -config.expand-env as flag to loki and promtail. * Disable env expansion on logcli * Small nit updates	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877) ... * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794)	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755) ... * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Ed Welch	8ea6c381cb	Promtail: Fix deadlock on tailer shutdown. (#2717) ... * to avoid a deadlock on shutdown, leave the goroutine running which reads the Lines channel from the tailer and it will exit when the tailer closes the channel. * remove the flag I added a few weeks ago and instead when updating the position information of the file gracefully fail if the file no longer exists. * update the vendoring of the tail library, more re-work on how tailing shutdown and errors are handled. * fix mistake * updating tail lib to merged commit hash * go mod tidy * update tail library with additional mutex lock * break up the tailer goroutine into two separate goroutines to make it a little more clear and remove some special case logic. * don't forget to close your channels! * make the stop method in the tailer only execute at most one time. * update hp-tail again with improvements to the mutex locking in closeFile * replace the more complicated channel based single execution code with a sync.Once	5 years ago