apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Owen Diehl	b861757b34	Ignore flushed chunks during checkpointing (#3195 ) * dont checkpoint flushed chunks * commenting	4 years ago
Cyril Tovena	5eee25b560	Fixes unwrap expressions from last optimization. (#3194 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	3054d64377	doc(gcplog): Fix titles for Cloud provisioning for GCP logs (#3190 )	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Hui Kang	1ffd4f641e	Doc: Add missing wal in configuration (#3162 ) * Doc: Add missing wal in configuration * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Michel Hollands	56bf5e7073	Make httpAuthMiddleware field public (#3159 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	40a637e897	remaining sts use parallel mgmt policy (#3181 )	4 years ago
Owen Diehl	c175ad0862	moves boltdb flags to config file (#3180 )	4 years ago
Owen Diehl	5b6d64c89e	sts rulers (#3179 )	4 years ago
Cyril Tovena	28c0b310c7	Logs PushRequest data. (#3178 ) * Logs PushRequest data. This will allows to find information about received size and total entries per tenant. Example of a log from my dev testing: ``` level=debug ts=2021-01-15T11:16:21.735663076Z caller=http.go:67 org_id=3927 traceID=11c4774c6ec4bbf4 msg="push request parsed" path=/loki/api/v1/push content-type=application/x-protobuf body-size="11 kB" streams=5 entries=298 streamLabelsSize="1.9 kB" entriesSize="45 kB" totalSize="47 kB" ``` Of course this means we can use LogQL on this. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move metrics to avoid request traversal twice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	c883f02eb0	changes ingester default from 5Gi to 10Gi, prior to PR 3079 correctly fixed the ingester deployment to use this value, however the hard coded value was 10Gi and this default was 5Gi. This is a dissalowed change in kubernetes and causes deployment failures. (#3156 ) Changing this back to 10Gi isn't ideal however because anyone who also deployed with the 5Gi is now in the same situation. I think however more people are going to find themselves impacted by this change as it existed for a lot longer as 10Gi than 5Gi.	4 years ago
Robert Fratto	628c1fae95	pass a prometheus registerer to promtail components (#3175 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
李国忠	54d1d3bea7	[doc] fix:querier do not handle "/flush" api (#3165 )	4 years ago
Owen Diehl	322e4bc6fc	updates alerting docs post 2.0 (#3164 ) * updates alerting docs post 2.0 * 1s -> 10s	4 years ago
Hui Kang	1f6fd5e45e	Doc: add missing type supported by table manager (#3148 )	4 years ago
Allenzhli	06de835f6e	update github.com/gorilla/websocket to fixes a potential DoS (#3152 ) Signed-off-by: allenzhli <allenzhli@tencent.com>	4 years ago
Cyril Tovena	2a90c4ceb5	Improve wal entries encoding. (#3153 ) Little trick to avoid memory allocation with regards to bytes slice and string. benchmp: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_EncodeEntries-16 1699362 1055627 -37.88% benchmark old allocs new allocs delta Benchmark_EncodeEntries-16 20025 25 -99.88% benchmark old bytes new bytes delta Benchmark_EncodeEntries-16 5625393 4665376 -17.07% ``` This originated from an investigation on CPU usage of ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
John Losito	cfbc5f84e6	Allow dependabot to keep github actions up-to-date (#3149 )	4 years ago
Owen Diehl	6dc8bee9b0	ingester/shutdown -> ingester/flush_shutdown (#3146 )	4 years ago
Cyril Tovena	5070605335	Fixes absent_over_time to work with all log selector. (#3143 ) * Fixes absent_over_time to work with all log selector. I made a mistake forcing absent_over_time to work only for unwrap, which is not the most useful usecase. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * actually add a test that cover the previous mistake :) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	b72198c80a	better boltdb errmsg (#3139 )	4 years ago
Owen Diehl	95052943c4	swaps mutex for atomic (#3141 )	4 years ago
Karsten Jeschkies	776c9df9e3	Give another example for multiline. (#3138 ) * Give another example for multiline. Summary: This is a new example I've tested. It uses the zero-width space character `U+200B` to identify the first line of a multline block. * Update docs/sources/clients/promtail/stages/multiline.md Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	ed649eec37	Don't fail writes due to full WAL disk (#3136 ) * dont fail writes on full wal disk * wal full failure will cause flush on shutdown * logs the first full WAL failure	4 years ago
Owen Diehl	fd94b8d30d	label_format no longer shardable and introduces the Shardable() metho… (#3137 ) * label_format no longer shardable and introduces the Shardable() method to Expr ifc * unused llint	4 years ago
Cyril Tovena	f1c4ba2f59	Add zstd and flate compressions algorithms. (#3064 ) * Add zstd and flate compressions algorithms. zstd has the highest ratio of all while flate is very simalar to gzip. I'm adding those mainly for posterity and to keep them benchmarked. This also update gzip and lz4 library which both have been improved. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing vendor change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	82d96dec03	Improve distributors validation. (#3134 ) ```bash ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Push-16 18708519 1348876 -92.79% benchmark old allocs new allocs delta Benchmark_Push-16 44 42 -4.55% benchmark old bytes new bytes delta Benchmark_Push-16 4008391 2079 -99.95% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	c9b85b33ac	Integrates label replace into sharding code (#3132 ) * avoid sharding label replace * updates sharding tests * matrix stepper end-inclusive for rangeVectorIterator parity * Revert "matrix stepper end-inclusive for rangeVectorIterator parity" This reverts commit `2f34deab98`. * removes sharding equivalence tests due to interference by non-sharding bug in the control case	4 years ago
Danny Kopping	fcabfec470	Implementing line comments (#3126 ) * Implementing line comments Adding tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasement Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests to ensure that a # within a string is not incorrectly lexed as a comment Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding documentation Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Sam Coren	ca13e79f04	Update getting-in-touch.md (#3123 ) This page wasn't updated yet to reflect the community support channel shift from Grafana Community Slack -> Grafana Community Forums. Also added clarity re: support for Grafana Cloud-hosted Loki users and included community call information.	4 years ago
Michel Hollands	4f62f36af7	Update cortex to 1 6 (#3131 ) * Update Cortex to recent master (1.5.0+, 35e698bb56d6). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore set scheduler address. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make linter happy. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Upgrade cortex to 1.6 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Work around issues in go-openapi libraries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clean up go.mod and go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	4 years ago
Cyril Tovena	cdf4a73238	Improve tailer matching by using the index. (#3090 ) * Improve tailer matching by using the index. ``` ❯ benchcmp before.txt after benchmark old ns/op new ns/op delta Benchmark_instance_addNewTailer-16 537731 1479 -99.72% benchmark old allocs new allocs delta Benchmark_instance_addNewTailer-16 244 3 -98.77% benchmark old bytes new bytes delta Benchmark_instance_addNewTailer-16 22098 121 -99.45% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	fea5db12ed	Fixes LogQL documentation links. (#3128 ) * Fixes LogQL documentation ref links. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix old link. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
WuJie	ca344ca31a	fix time duration unit (#3124 ) time.ParseDuration does not support unit `d`	4 years ago
Cyril Tovena	c971e8e65a	Optimizes SampleExpr to remove unnecessary line_format. (#3065 ) * Optimizes SampleExpr to remove unnessary line_format. When doing count_over_time and rate(), it doesn't make sense to have a line_format since you're counting line, you don't really care about what the line contains. At the same time this set the foundation of for further optimization, like parsing only required labels during label parsing. I've also found a bug where `topk by(foo) (1...` where not correctly parsed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * For all range vector operations. Except for bytes one which count bytes in lines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logql/optimize_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
李国忠	bd321fb4f2	lru cache logql.ParseLabels (#3092 ) * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels * lru cache logql.ParseLabels	4 years ago
Val	5572366d93	Include instruction to enable variable expansion (#3115 )	4 years ago
Owen Diehl	72666dc799	self documenting pipeline process ifc (#3122 )	4 years ago
Owen Diehl	ad803af9dc	Wal/recover corruption (#3117 ) * simplifies recovery code and continues after errors * does not exit on wal recovery failure * log line parity * less scary error msgs * more descriptive errors * Update pkg/ingester/ingester.go Co-authored-by: Ed Welch <ed@oqqer.com> * Update pkg/ingester/ingester.go Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	4 years ago
Cyril Tovena	7682f133b6	Improve JSON parser and add labels parser hints. (#3080 ) * Improve JSON parser and add labels parser hints. This is a rework of the json parser, that now uses iteration over reflection allowing to avoid allocations. I've also implemented a way to guess and hints what labels needs to be parsed when doing metric queries, this means we can extract only the few labels required from the log line. benchcmp ``` ❯ benchcmp before.txt after.txt5 benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 9889 3281 -66.82% Benchmark_Parser/logfmt/no_labels_hints-16 1624 1671 +2.89% Benchmark_Parser/logfmt/labels_hints-16 1601 790 -50.66% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 139 56 -59.71% Benchmark_Parser/logfmt/no_labels_hints-16 31 31 +0.00% Benchmark_Parser/logfmt/labels_hints-16 31 5 -83.87% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 3671 912 -75.16% Benchmark_Parser/logfmt/no_labels_hints-16 464 464 +0.00% Benchmark_Parser/logfmt/labels_hints-16 464 144 -68.97% ``` I've experienced a 2x to 4x improvement in my cluster. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typos. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	58987553a1	Disables the stream limiter until wal has recovered (#3114 ) * disables the stream limiter until wal has recovered * limiter disable/enable methods	4 years ago
Kaviraj	3edcda744e	mention the use of `config.expand-env` flag in the doc. (#3078 )	4 years ago
Owen Diehl	ee9c629f51	wal docs (#3026 )	4 years ago
Andrea Scarpino	3640d2d4a8	Update reference to old helm chart repo (#3084 )	4 years ago
nileshcs	92c7bb5513	Update docs for downstream_url (#3109 )	4 years ago
Cyril Tovena	e83f809dd7	Small fixes for the documentation. (#3088 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
coffee coder	03aaa34de8	Assign appropriate pvc size and class to compactor and ingester (#3074 )	4 years ago
Rajat Vig	001668b79d	Fix ingester PVC data declaration to use configured value (#3079 )	4 years ago
Benjamin Huo	52afadc027	Fix storage guide links (#3094 ) Signed-off-by: Benjamin <benjamin@yunify.com>	4 years ago

1 2 3 4 5 ...

1982 Commits (b861757b346fced7de2a258dd8ad89aad7e24b7b) All Branches Search

1982 Commits (b861757b346fced7de2a258dd8ad89aad7e24b7b)

All Branches