apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	9e33f8f6b7	Fixes races in multiline stage and promtail. (#3302 ) * Fixes a race in multiline code. Not fully sure how this was happening, but I have a test showing it's possible and a fix for it. I think it should fix #3120. I also remove race within some tests of promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Reset config Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	15b6235cae	Fixes step encoding in logcli. (#3325 ) Fixes #3318 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	47e7ec5ec6	adds some flushing instrumentation/logs (#3340 )	4 years ago
Owen Diehl	862cfa4e1f	adds Start() method to WAL interface to delay checkpointing until after replay (#3339 )	4 years ago
Sandeep Sukhani	76e713f57d	reduce number of list calls on shared object store when using boltdb-shipper (#3283 )	4 years ago
Owen Diehl	8a9b94a8bc	removes recover flag (#3326 )	4 years ago
Ed Welch	407afce0e3	Loki: fix validation error and metrics (#3307 ) * Remove the helper functions for creating errors as they cause unintended value manipulation if the value contains any valid golang printf substitute characters, e.g. %s * Improve error message for invalid labels, increment metrics for lines dropped for invalid labels.	4 years ago
Danny Kopping	feb7fb470b	LogQL: Simple JSON expressions (#3280 ) * New approach, still rough Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding benchmark Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding more tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding documentation Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Docs fixup Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing unnecessary condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding extra tests from suggestion in review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding JSONParseErr Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding test to cover invalid JSON line Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding equivalent benchmarks for JSON and JSONExpression parsing Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding suffix if label would be overridden Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reparenting jsonexpr directory to more appropriate location Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Setting empty label on non-matching expression, to retain parity with label_format Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding statement about returned complex JSON types Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Added check for valid label name Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making json expressions shardable Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Gregory Brzeski	6c8fdd68aa	fix typo in config variable name: BookmarkPath (#3298 )	4 years ago
Cyril Tovena	82cead2af7	Increase the windows bookmark buffer. (#3303 ) See this discussion https://github.com/grafana/loki/pull/3246#discussion_r568674040 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	5f0c06ac3a	Logql/absent label optimization (#3299 ) * absent_over_time does not extract unnecessary labels * rebase fix	4 years ago
Dmitry Ukolov	2a51fb42bf	Distributor: Loki API can receive gzipped JSON (#3291 ) * Push API recognize HTTP header "Content-Encoding: gzip" and upload gzipped JSON * Fix documentation mistake	4 years ago
Kaviraj	59a34f9867	Make incoming labels from gcp into Loki internal labels. (#3285 ) * Make incoming labels from gcp into Loki internal labels. If these labels are significant, we can always use relabeling. * Add support for rebeling in gcptarget * Test for SnakeCase	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Kaviraj	d667dd20f2	Avoid putting all the GCP labels into loki labels (#3284 )	4 years ago
Cyril Tovena	3bd725e826	Fixes logfmt parser hints. (#3279 ) Recently introduce a bug where label where tested for extraction against non sanitized version. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Chance Zibolski	054b8fcc91	logcli: Refactor logcli Client interface to use time.Time and time.Duration for LiveTailQueryConn method (#3271 )	4 years ago
Owen Diehl	4d1c6e0169	adds another test (#3278 )	4 years ago
Cyril Tovena	72b3acb925	Windows events (#3246 ) * First commit for windows event targets. I had to add couple of new go modules and hook the windows log in. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the ability to save where we are. * Finished setup tests and comments. * nits * adding documentation * go tidy * lint windows * nope * add manager for non windows * don't lint forked package * mod check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/promtail/targets/windows/bookmark.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Use passed-in Prometheus registerer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * testing out windows drone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * type exec. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * docker image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go/go.exe Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * build windows on container. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * trying another image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo in image Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixed local timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests with timezone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes timezone tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update logstash since the last image is not working anymore 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	e8cce09de2	Fix a bug with metric queries and label_format. (#3263 ) * Fix a bug whith metric queries and label_format. In my previous PR where I introduced hints, I forgot to implement label format label required. This would cause label format label to be skipped during parsing. I've also took the opportunity to improve the code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor and add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	287244d110	Update vendored Cortex to 0976147451ee (#3267 )	4 years ago
Chance Zibolski	91974e0cfe	logcli: Fix handling of logcli query using --since/--from and --tail (#3270 ) * pkg/logcli/client: Correctly set "start" query param for tail API instead of "from" * pkg/logcli/client: Change Client method variable names to match API query parameters Instead of using from/through, use start/end like the Loki API does.	4 years ago
李国忠	0986d8dafe	[ingester_query] change var "clients" to "reps" (#3274 )	4 years ago
Ed Welch	b4eeddc710	Loki migrate-tool (#3151 ) * basics working, needs some more cleanup and more flags for config and a Dockerfile Signed-off-by: Edward Welch <edward.welch@grafana.com> * add label matchers start end time params and ability to remap the chunk to a different user Signed-off-by: Edward Welch <edward.welch@grafana.com> * adding dockerfile and makefile instructions Signed-off-by: Edward Welch <edward.welch@grafana.com> * push out some of the limits show errors from index query * for testing have docker container not run migrate command but instead stay running by tailing /dev/null * big overhaul to add parallelization * adding better stats * adding retry logic * tweaking retry logic * tweaking retry logic * work around changes in latest cortex Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix up imports and changes since the last time this was merged added a readme fixed up the makefile added some comments * fixing some of my garbage * fixing more of my garbage * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * tabs not spaces Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Sandeep Sukhani	d357ad9e58	fix broken json logs push path (#3261 )	4 years ago
Ed Welch	51952dafc1	Loki: Log a crude lag metric for how far behind a client is. (#3236 ) * Log the difference in time between the most recently received log entry and the current time as a crude measure of how far behind a promtail instance is. * fix lint	4 years ago
Cyril Tovena	1a36874e1c	Ensure we have parenthetis for bin ops. (#3251 ) This ensure we correctly keep the order of execution when we stringify a bin operation. This has recently introduced a bug because we now use stringify to deep clone a LogQL expression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	db14ff193c	limit query range in async store for ingesters when query-ingesters-within flag is set (#3201 ) * limit query range in async store for ingesters when query-ingesters-within flag is set * do not reduce ingester query start time	4 years ago
Cyril Tovena	3e46067cbb	Fixes a bug where slice of Entries where not zeroed before decoding json. (#3249 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	224e68a50b	WAL backpressure (#3218 ) * adds WAL replay backpressure * backpressure fn doesnt lock during passed fn for concurrency * replayController * replayController supports a RemoveFlushedChunks fn * custom wal flusher * encoding test determinism * standardizes recoveredBytesTotal to replayControllers ownership * linting * default wal memory threshold to 4gb, adds docs * fix test after v3 chunk schema change * more lenient expectCheckpoint function * replayController protects against flush race condition * adds replay_memory_ceiling to wal jsonnet * replay ceiling help msg	4 years ago
Thibault Deutsch	2bbf4b21c3	logcli: support --include-label when not using --tail (#3229 )	4 years ago
Cyril Tovena	8716e24518	Allocate entries array with correct size while decoding WAL entries. (#3241 ) Small optimization I found while deployment the new version. We might want to have a pool of entries here. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_DecodeWAL-16 2350024 1591633 -32.27% benchmark old allocs new allocs delta Benchmark_DecodeWAL-16 20043 20005 -0.19% benchmark old bytes new bytes delta Benchmark_DecodeWAL-16 4946246 1763270 -64.35% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a5cb38d54c	Fixes unmarshalling of tailing responses. (#3237 ) * Fixes unmarshalling of tailing responses. Some websocket were still using old json package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes missing update of generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Jeroen Veltman	5d6343c1d0	Fix(pkg/promtail) CVE-2020-11022 JQuery vulnerability (#3224 ) * Fix CVE-2020-11022: Upgraded to jquery to 3.5.1 from 3.3.1 Backup jquery(3.3.1) to jquery.min.js, added jquery-3.5.1.js, updated reference in base template * Fix CVE-2020-11022: changes in assets_vfsdata.go executed make check-generated-files	4 years ago
Michel Hollands	2afebbc064	Add config endpoint (#3228 ) * Add config route Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	d976079992	Fixes previous commit not using the new sized body. (#3234 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	567ccde0b6	Re-introduce https://github.com/grafana/loki/pull/3178 . (#3233 ) Lost during a merge/rebase of `6cc41f92f8` 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a70e68862b	Rename checkpoint proto package to avoid conflict with cortex. (#3217 ) This will fix the warning in all our binaries. ``` 2021-01-21 16:09:11.947264 I \| proto: duplicate proto type registered: ingester.Series ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Robert Fratto	da54d33e33	promtail: pass registerer to gcplog (#3191 ) * promtail: pass registerer to gcplog * enforce gcplog NewMetrics registerer to be non-nil * reformat gcplog/targetmanager.go	4 years ago
Cyril Tovena	fd619b6931	Fixes tail api marshalling for v1. (#3211 ) Introduce a bug by removing the default marshalling (#3163) but the tail api was using the default json. This fixes it by forcing the usage of jsoniter package. Added a missing test so that it doesn't happen again. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	e74eb549ba	Reverts flush buffer pooling. (#3210 ) Currently this requires more work, as the cache is asynchronously using the buffer when flushing chunks. We still benefits from other improvements like computing the right size for a chunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ba0df61531	Fixes panic when using multiple client with the same prometheus registry. (#3207 ) Introduce by #3175. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ea488f9940	Improve ingester flush memory usage. (#3200 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add benchmark for flush loop. ``` goos: darwin goarch: amd64 pkg: github.com/grafana/loki/pkg/ingester Benchmark_FlushLoop-16 16 70040359 ns/op 241901661 B/op 1112 allocs/op PASS ok github.com/grafana/loki/pkg/ingester 4.829s ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Re-use buffer when flushing chunks. benchcmp ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_FlushLoop-16 104723243 9233780 -91.18% benchmark old allocs new allocs delta Benchmark_FlushLoop-16 1115 568 -49.06% benchmark old bytes new bytes delta Benchmark_FlushLoop-16 241857243 989590 -99.59% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	07ece2bc2a	Improve checkpoint series iterator. (#3193 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	0a81f70800	Improves checkpointerWriter memory usage (#3188 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set a base benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves checkpointwriter memory usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Kavi feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	b861757b34	Ignore flushed chunks during checkpointing (#3195 ) * dont checkpoint flushed chunks * commenting	4 years ago
Cyril Tovena	5eee25b560	Fixes unwrap expressions from last optimization. (#3194 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago

1 2 3 4 5 ...

854 Commits (9e33f8f6b7fe35a395ef8fec08fc9554e3ad2346)