apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	e8cce09de2	Fix a bug with metric queries and label_format. (#3263 ) * Fix a bug whith metric queries and label_format. In my previous PR where I introduced hints, I forgot to implement label format label required. This would cause label format label to be skipped during parsing. I've also took the opportunity to improve the code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor and add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	287244d110	Update vendored Cortex to 0976147451ee (#3267 )	4 years ago
Chance Zibolski	91974e0cfe	logcli: Fix handling of logcli query using --since/--from and --tail (#3270 ) * pkg/logcli/client: Correctly set "start" query param for tail API instead of "from" * pkg/logcli/client: Change Client method variable names to match API query parameters Instead of using from/through, use start/end like the Loki API does.	4 years ago
李国忠	0986d8dafe	[ingester_query] change var "clients" to "reps" (#3274 )	4 years ago
Ed Welch	b4eeddc710	Loki migrate-tool (#3151 ) * basics working, needs some more cleanup and more flags for config and a Dockerfile Signed-off-by: Edward Welch <edward.welch@grafana.com> * add label matchers start end time params and ability to remap the chunk to a different user Signed-off-by: Edward Welch <edward.welch@grafana.com> * adding dockerfile and makefile instructions Signed-off-by: Edward Welch <edward.welch@grafana.com> * push out some of the limits show errors from index query * for testing have docker container not run migrate command but instead stay running by tailing /dev/null * big overhaul to add parallelization * adding better stats * adding retry logic * tweaking retry logic * tweaking retry logic * work around changes in latest cortex Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix up imports and changes since the last time this was merged added a readme fixed up the makefile added some comments * fixing some of my garbage * fixing more of my garbage * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * tabs not spaces Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Sandeep Sukhani	d357ad9e58	fix broken json logs push path (#3261 )	4 years ago
Ed Welch	51952dafc1	Loki: Log a crude lag metric for how far behind a client is. (#3236 ) * Log the difference in time between the most recently received log entry and the current time as a crude measure of how far behind a promtail instance is. * fix lint	4 years ago
Cyril Tovena	1a36874e1c	Ensure we have parenthetis for bin ops. (#3251 ) This ensure we correctly keep the order of execution when we stringify a bin operation. This has recently introduced a bug because we now use stringify to deep clone a LogQL expression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	db14ff193c	limit query range in async store for ingesters when query-ingesters-within flag is set (#3201 ) * limit query range in async store for ingesters when query-ingesters-within flag is set * do not reduce ingester query start time	4 years ago
Cyril Tovena	3e46067cbb	Fixes a bug where slice of Entries where not zeroed before decoding json. (#3249 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	224e68a50b	WAL backpressure (#3218 ) * adds WAL replay backpressure * backpressure fn doesnt lock during passed fn for concurrency * replayController * replayController supports a RemoveFlushedChunks fn * custom wal flusher * encoding test determinism * standardizes recoveredBytesTotal to replayControllers ownership * linting * default wal memory threshold to 4gb, adds docs * fix test after v3 chunk schema change * more lenient expectCheckpoint function * replayController protects against flush race condition * adds replay_memory_ceiling to wal jsonnet * replay ceiling help msg	4 years ago
Thibault Deutsch	2bbf4b21c3	logcli: support --include-label when not using --tail (#3229 )	4 years ago
Cyril Tovena	8716e24518	Allocate entries array with correct size while decoding WAL entries. (#3241 ) Small optimization I found while deployment the new version. We might want to have a pool of entries here. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_DecodeWAL-16 2350024 1591633 -32.27% benchmark old allocs new allocs delta Benchmark_DecodeWAL-16 20043 20005 -0.19% benchmark old bytes new bytes delta Benchmark_DecodeWAL-16 4946246 1763270 -64.35% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a5cb38d54c	Fixes unmarshalling of tailing responses. (#3237 ) * Fixes unmarshalling of tailing responses. Some websocket were still using old json package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes missing update of generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Jeroen Veltman	5d6343c1d0	Fix(pkg/promtail) CVE-2020-11022 JQuery vulnerability (#3224 ) * Fix CVE-2020-11022: Upgraded to jquery to 3.5.1 from 3.3.1 Backup jquery(3.3.1) to jquery.min.js, added jquery-3.5.1.js, updated reference in base template * Fix CVE-2020-11022: changes in assets_vfsdata.go executed make check-generated-files	4 years ago
Michel Hollands	2afebbc064	Add config endpoint (#3228 ) * Add config route Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	d976079992	Fixes previous commit not using the new sized body. (#3234 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	567ccde0b6	Re-introduce https://github.com/grafana/loki/pull/3178 . (#3233 ) Lost during a merge/rebase of `6cc41f92f8` 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a70e68862b	Rename checkpoint proto package to avoid conflict with cortex. (#3217 ) This will fix the warning in all our binaries. ``` 2021-01-21 16:09:11.947264 I \| proto: duplicate proto type registered: ingester.Series ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Robert Fratto	da54d33e33	promtail: pass registerer to gcplog (#3191 ) * promtail: pass registerer to gcplog * enforce gcplog NewMetrics registerer to be non-nil * reformat gcplog/targetmanager.go	4 years ago
Cyril Tovena	fd619b6931	Fixes tail api marshalling for v1. (#3211 ) Introduce a bug by removing the default marshalling (#3163) but the tail api was using the default json. This fixes it by forcing the usage of jsoniter package. Added a missing test so that it doesn't happen again. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	e74eb549ba	Reverts flush buffer pooling. (#3210 ) Currently this requires more work, as the cache is asynchronously using the buffer when flushing chunks. We still benefits from other improvements like computing the right size for a chunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ba0df61531	Fixes panic when using multiple client with the same prometheus registry. (#3207 ) Introduce by #3175. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ea488f9940	Improve ingester flush memory usage. (#3200 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add benchmark for flush loop. ``` goos: darwin goarch: amd64 pkg: github.com/grafana/loki/pkg/ingester Benchmark_FlushLoop-16 16 70040359 ns/op 241901661 B/op 1112 allocs/op PASS ok github.com/grafana/loki/pkg/ingester 4.829s ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Re-use buffer when flushing chunks. benchcmp ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_FlushLoop-16 104723243 9233780 -91.18% benchmark old allocs new allocs delta Benchmark_FlushLoop-16 1115 568 -49.06% benchmark old bytes new bytes delta Benchmark_FlushLoop-16 241857243 989590 -99.59% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	07ece2bc2a	Improve checkpoint series iterator. (#3193 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	0a81f70800	Improves checkpointerWriter memory usage (#3188 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set a base benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves checkpointwriter memory usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Kavi feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	b861757b34	Ignore flushed chunks during checkpointing (#3195 ) * dont checkpoint flushed chunks * commenting	4 years ago
Cyril Tovena	5eee25b560	Fixes unwrap expressions from last optimization. (#3194 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Michel Hollands	56bf5e7073	Make httpAuthMiddleware field public (#3159 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Robert Fratto	628c1fae95	pass a prometheus registerer to promtail components (#3175 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
Cyril Tovena	2a90c4ceb5	Improve wal entries encoding. (#3153 ) Little trick to avoid memory allocation with regards to bytes slice and string. benchmp: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_EncodeEntries-16 1699362 1055627 -37.88% benchmark old allocs new allocs delta Benchmark_EncodeEntries-16 20025 25 -99.88% benchmark old bytes new bytes delta Benchmark_EncodeEntries-16 5625393 4665376 -17.07% ``` This originated from an investigation on CPU usage of ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	6dc8bee9b0	ingester/shutdown -> ingester/flush_shutdown (#3146 )	4 years ago
Cyril Tovena	5070605335	Fixes absent_over_time to work with all log selector. (#3143 ) * Fixes absent_over_time to work with all log selector. I made a mistake forcing absent_over_time to work only for unwrap, which is not the most useful usecase. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * actually add a test that cover the previous mistake :) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	95052943c4	swaps mutex for atomic (#3141 )	4 years ago
Owen Diehl	ed649eec37	Don't fail writes due to full WAL disk (#3136 ) * dont fail writes on full wal disk * wal full failure will cause flush on shutdown * logs the first full WAL failure	4 years ago
Owen Diehl	fd94b8d30d	label_format no longer shardable and introduces the Shardable() metho… (#3137 ) * label_format no longer shardable and introduces the Shardable() method to Expr ifc * unused llint	4 years ago
Cyril Tovena	f1c4ba2f59	Add zstd and flate compressions algorithms. (#3064 ) * Add zstd and flate compressions algorithms. zstd has the highest ratio of all while flate is very simalar to gzip. I'm adding those mainly for posterity and to keep them benchmarked. This also update gzip and lz4 library which both have been improved. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing vendor change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	82d96dec03	Improve distributors validation. (#3134 ) ```bash ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Push-16 18708519 1348876 -92.79% benchmark old allocs new allocs delta Benchmark_Push-16 44 42 -4.55% benchmark old bytes new bytes delta Benchmark_Push-16 4008391 2079 -99.95% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	c9b85b33ac	Integrates label replace into sharding code (#3132 ) * avoid sharding label replace * updates sharding tests * matrix stepper end-inclusive for rangeVectorIterator parity * Revert "matrix stepper end-inclusive for rangeVectorIterator parity" This reverts commit `2f34deab98`. * removes sharding equivalence tests due to interference by non-sharding bug in the control case	4 years ago
Danny Kopping	fcabfec470	Implementing line comments (#3126 ) * Implementing line comments Adding tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasement Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests to ensure that a # within a string is not incorrectly lexed as a comment Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding documentation Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Michel Hollands	4f62f36af7	Update cortex to 1 6 (#3131 ) * Update Cortex to recent master (1.5.0+, 35e698bb56d6). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore set scheduler address. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make linter happy. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Upgrade cortex to 1.6 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Work around issues in go-openapi libraries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clean up go.mod and go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	4 years ago
Cyril Tovena	cdf4a73238	Improve tailer matching by using the index. (#3090 ) * Improve tailer matching by using the index. ``` ❯ benchcmp before.txt after benchmark old ns/op new ns/op delta Benchmark_instance_addNewTailer-16 537731 1479 -99.72% benchmark old allocs new allocs delta Benchmark_instance_addNewTailer-16 244 3 -98.77% benchmark old bytes new bytes delta Benchmark_instance_addNewTailer-16 22098 121 -99.45% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	c971e8e65a	Optimizes SampleExpr to remove unnecessary line_format. (#3065 ) * Optimizes SampleExpr to remove unnessary line_format. When doing count_over_time and rate(), it doesn't make sense to have a line_format since you're counting line, you don't really care about what the line contains. At the same time this set the foundation of for further optimization, like parsing only required labels during label parsing. I've also found a bug where `topk by(foo) (1...` where not correctly parsed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * For all range vector operations. Except for bytes one which count bytes in lines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logql/optimize_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
李国忠	bd321fb4f2	lru cache logql.ParseLabels (#3092 ) * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels	4 years ago
Owen Diehl	72666dc799	self documenting pipeline process ifc (#3122 )	4 years ago

1 2 3 4 5 ...

835 Commits (e8cce09de29dc58c38b333e72dfe11613d49fbb3)