apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Owen Diehl	224e68a50b	WAL backpressure (#3218 ) * adds WAL replay backpressure * backpressure fn doesnt lock during passed fn for concurrency * replayController * replayController supports a RemoveFlushedChunks fn * custom wal flusher * encoding test determinism * standardizes recoveredBytesTotal to replayControllers ownership * linting * default wal memory threshold to 4gb, adds docs * fix test after v3 chunk schema change * more lenient expectCheckpoint function * replayController protects against flush race condition * adds replay_memory_ceiling to wal jsonnet * replay ceiling help msg	4 years ago
Thibault Deutsch	2bbf4b21c3	logcli: support --include-label when not using --tail (#3229 )	4 years ago
Cyril Tovena	8716e24518	Allocate entries array with correct size while decoding WAL entries. (#3241 ) Small optimization I found while deployment the new version. We might want to have a pool of entries here. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_DecodeWAL-16 2350024 1591633 -32.27% benchmark old allocs new allocs delta Benchmark_DecodeWAL-16 20043 20005 -0.19% benchmark old bytes new bytes delta Benchmark_DecodeWAL-16 4946246 1763270 -64.35% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Andreas Plesner	4c6f8e8149	docs: point to latest release for docker installation (#3235 ) Signed-off-by: Andreas Plesner <apj@mutt.dk>	4 years ago
Cyril Tovena	a5cb38d54c	Fixes unmarshalling of tailing responses. (#3237 ) * Fixes unmarshalling of tailing responses. Some websocket were still using old json package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes missing update of generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6856209093	Terraform script to automate GCP provisioning for gcplog (#3206 ) * Terraform script to automate GCP provisioning for gcplog * PR remarks	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Leo Takaoka	931fdcbb93	fix: Insert a space in the error message (#3225 )	4 years ago
Jeroen Veltman	5d6343c1d0	Fix(pkg/promtail) CVE-2020-11022 JQuery vulnerability (#3224 ) * Fix CVE-2020-11022: Upgraded to jquery to 3.5.1 from 3.3.1 Backup jquery(3.3.1) to jquery.min.js, added jquery-3.5.1.js, updated reference in base template * Fix CVE-2020-11022: changes in assets_vfsdata.go executed make check-generated-files	4 years ago
Michel Hollands	2afebbc064	Add config endpoint (#3228 ) * Add config route Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	d976079992	Fixes previous commit not using the new sized body. (#3234 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	567ccde0b6	Re-introduce https://github.com/grafana/loki/pull/3178 . (#3233 ) Lost during a merge/rebase of `6cc41f92f8` 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
李国忠	0208071bbd	[doc] fix err. "loki_frontend" is invalid (#3220 )	4 years ago
Michał Sochoń	3bb7f52f1f	Fix: Update docs for logcli (#3212 ) * Update docs for logcli Fix description for logcli, use proper flag names. Extend docs with example using timestamp. Signed-off-by: Michał Sochoń <kaszpir@gmail.com> * Update cmd/logcli/main.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	a70e68862b	Rename checkpoint proto package to avoid conflict with cortex. (#3217 ) This will fix the warning in all our binaries. ``` 2021-01-21 16:09:11.947264 I \| proto: duplicate proto type registered: ingester.Series ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Robert Fratto	da54d33e33	promtail: pass registerer to gcplog (#3191 ) * promtail: pass registerer to gcplog * enforce gcplog NewMetrics registerer to be non-nil * reformat gcplog/targetmanager.go	4 years ago
Cyril Tovena	fd619b6931	Fixes tail api marshalling for v1. (#3211 ) Introduce a bug by removing the default marshalling (#3163) but the tail api was using the default json. This fixes it by forcing the usage of jsoniter package. Added a missing test so that it doesn't happen again. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	e74eb549ba	Reverts flush buffer pooling. (#3210 ) Currently this requires more work, as the cache is asynchronously using the buffer when flushing chunks. We still benefits from other improvements like computing the right size for a chunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ba0df61531	Fixes panic when using multiple client with the same prometheus registry. (#3207 ) Introduce by #3175. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ea488f9940	Improve ingester flush memory usage. (#3200 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add benchmark for flush loop. ``` goos: darwin goarch: amd64 pkg: github.com/grafana/loki/pkg/ingester Benchmark_FlushLoop-16 16 70040359 ns/op 241901661 B/op 1112 allocs/op PASS ok github.com/grafana/loki/pkg/ingester 4.829s ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Re-use buffer when flushing chunks. benchcmp ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_FlushLoop-16 104723243 9233780 -91.18% benchmark old allocs new allocs delta Benchmark_FlushLoop-16 1115 568 -49.06% benchmark old bytes new bytes delta Benchmark_FlushLoop-16 241857243 989590 -99.59% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	07ece2bc2a	Improve checkpoint series iterator. (#3193 ) * Add basic benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves memory usage of checkpointer series iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * make lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better size computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test ordering flakyness. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	0a81f70800	Improves checkpointerWriter memory usage (#3188 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set a base benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves checkpointwriter memory usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Kavi feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	b861757b34	Ignore flushed chunks during checkpointing (#3195 ) * dont checkpoint flushed chunks * commenting	4 years ago
Cyril Tovena	5eee25b560	Fixes unwrap expressions from last optimization. (#3194 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	3054d64377	doc(gcplog): Fix titles for Cloud provisioning for GCP logs (#3190 )	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Hui Kang	1ffd4f641e	Doc: Add missing wal in configuration (#3162 ) * Doc: Add missing wal in configuration * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Michel Hollands	56bf5e7073	Make httpAuthMiddleware field public (#3159 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	40a637e897	remaining sts use parallel mgmt policy (#3181 )	4 years ago
Owen Diehl	c175ad0862	moves boltdb flags to config file (#3180 )	4 years ago
Owen Diehl	5b6d64c89e	sts rulers (#3179 )	4 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	c883f02eb0	changes ingester default from 5Gi to 10Gi, prior to PR 3079 correctly fixed the ingester deployment to use this value, however the hard coded value was 10Gi and this default was 5Gi. This is a dissalowed change in kubernetes and causes deployment failures. (#3156 ) Changing this back to 10Gi isn't ideal however because anyone who also deployed with the 5Gi is now in the same situation. I think however more people are going to find themselves impacted by this change as it existed for a lot longer as 10Gi than 5Gi.	4 years ago
Robert Fratto	628c1fae95	pass a prometheus registerer to promtail components (#3175 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
李国忠	54d1d3bea7	[doc] fix:querier do not handle "/flush" api (#3165 )	4 years ago
Owen Diehl	322e4bc6fc	updates alerting docs post 2.0 (#3164 ) * updates alerting docs post 2.0 * 1s -> 10s	4 years ago
Hui Kang	1f6fd5e45e	Doc: add missing type supported by table manager (#3148 )	4 years ago
Allenzhli	06de835f6e	update github.com/gorilla/websocket to fixes a potential DoS (#3152 ) Signed-off-by: allenzhli <allenzhli@tencent.com>	4 years ago
Cyril Tovena	2a90c4ceb5	Improve wal entries encoding. (#3153 ) Little trick to avoid memory allocation with regards to bytes slice and string. benchmp: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_EncodeEntries-16 1699362 1055627 -37.88% benchmark old allocs new allocs delta Benchmark_EncodeEntries-16 20025 25 -99.88% benchmark old bytes new bytes delta Benchmark_EncodeEntries-16 5625393 4665376 -17.07% ``` This originated from an investigation on CPU usage of ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
John Losito	cfbc5f84e6	Allow dependabot to keep github actions up-to-date (#3149 )	4 years ago
Owen Diehl	6dc8bee9b0	ingester/shutdown -> ingester/flush_shutdown (#3146 )	4 years ago
Cyril Tovena	5070605335	Fixes absent_over_time to work with all log selector. (#3143 ) * Fixes absent_over_time to work with all log selector. I made a mistake forcing absent_over_time to work only for unwrap, which is not the most useful usecase. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * actually add a test that cover the previous mistake :) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	b72198c80a	better boltdb errmsg (#3139 )	4 years ago
Owen Diehl	95052943c4	swaps mutex for atomic (#3141 )	4 years ago
Karsten Jeschkies	776c9df9e3	Give another example for multiline. (#3138 ) * Give another example for multiline. Summary: This is a new example I've tested. It uses the zero-width space character `U+200B` to identify the first line of a multline block. * Update docs/sources/clients/promtail/stages/multiline.md Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	ed649eec37	Don't fail writes due to full WAL disk (#3136 ) * dont fail writes on full wal disk * wal full failure will cause flush on shutdown * logs the first full WAL failure	4 years ago
Owen Diehl	fd94b8d30d	label_format no longer shardable and introduces the Shardable() metho… (#3137 ) * label_format no longer shardable and introduces the Shardable() method to Expr ifc * unused llint	4 years ago
Cyril Tovena	f1c4ba2f59	Add zstd and flate compressions algorithms. (#3064 ) * Add zstd and flate compressions algorithms. zstd has the highest ratio of all while flate is very simalar to gzip. I'm adding those mainly for posterity and to keep them benchmarked. This also update gzip and lz4 library which both have been improved. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing vendor change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	82d96dec03	Improve distributors validation. (#3134 ) ```bash ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Push-16 18708519 1348876 -92.79% benchmark old allocs new allocs delta Benchmark_Push-16 44 42 -4.55% benchmark old bytes new bytes delta Benchmark_Push-16 4008391 2079 -99.95% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago

1 2 3 4 5 ...

2004 Commits (224e68a50b822edc9b63c834c79636b0eaf15f73) All Branches Search

2004 Commits (224e68a50b822edc9b63c834c79636b0eaf15f73)

All Branches