apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
atxviking	f6dcd3f3a4	Fix document links for api push example (#3690 )	4 years ago
3Xpl0it3r	89bdc9c10a	add debug information for extracted data (#3711 ) * fix the promtail-default-config download link in doc * add debug information for extreacted in stage * add Debug variable * format	4 years ago
jadd	38e06015ba	Update troubleshooting.md (#3715 ) fix some typo. change utf8 em dash to ascii dash	4 years ago
Fredrik Ekre	11052c597e	[docs/clients] fix header for "Unofficial clients" (#3720 ) and add a reference to said section.	4 years ago
Diana Payton	6a92eb3f07	Update _index.md (#3723 )	4 years ago
3Xpl0it3r	ccb5cc0a9e	remove duplicated code (#3727 ) * fix the promtail-default-config download link in doc * remove duplicated code	4 years ago
Roger Steneteg	0679e50d6a	correct logcli instant query timestamp param name (#3739 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	4 years ago
Cyril Tovena	e1a3ab841c	Improve retention mark files. (#3706 ) * Improve retention mark files. This PR is two fold: 1) Rotate mark files when they reached 100k marks, this is to ensure we max out marks file and don't create file that may be too big. 2) Instead of inserting marks using the chunk id as the key, we insert using a natural sequence. This has two benefits: - Keys are ordered, and so insertion are faster. - Allows us to use boltdb fill percent to 100%, this means boltdb won't over allocate for inserting key in between unordered data. Why ? I realize that when inserting huge amount of marks, this operation can take up to hours since boltdb has to re-allocate pages over and over. This is mainly because chunkid arrive without specific order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve key buf sizing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/stores/shipper/compactor/retention/marker.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Copy the value so that it still available for the whole boltdb tx. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Sandeep Sukhani	2b7a6f201c	Logs deletion support for boltdb-shipper store (#3688 ) * add support for slicing the chunks for partial chunk deletion * add delete requests store and handler for requests * add delete requests manager and a way to check for chunk expiry based on delete requests * add chunk rewriter for partially deleted chunks * hook delete requests manager into compactor and update expiration checker * use chunk client in sweeper for deleting chunks * upload delete requests table during shutdown and some code refactoring * add tests and metrics * fixes post rebase * lint * changes suggested from PR review * avoid allocation when getting userid from chunkid * use bytes.IndexByte instead	4 years ago
Ed Welch	fa247353ed	Add a flag which allows the queriers to run completely standalone and only query stored data. (#3700 )	4 years ago
Cyril Tovena	210dc355cb	Fixes a goroutine leak in the store when doing cancellation. (#3733 ) Found out that we might have no listener and trying to send the information of cancellation. I've moved the error information down the respective log and sample iterators so that we don't need to send it via the channel. This seems to have cause some pretty big memory leak on our side. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	58bf983518	Fixes doc w/r/t grpc compression. (#3551 ) * Fixes doc w/r/t grpc compression. Fixes #3537. Due to a change in upstream we didn't catch that breaking change in the config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update _index.md * Update _index.md * Update _index.md	4 years ago
Danny Kopping	a00b19240b	Modifying issue templates (#3732 ) Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Cyril Tovena	cba373cce5	Update publish.yml (#3731 )	4 years ago
Cyril Tovena	7a5fe3dd1c	Fixes docker driver that would panic when closed. (#3709 ) This is happening because it can still received log, but the client might be already closed. Fixes #3705 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Diana Payton	74ee19c39c	Update CODEOWNERS (#3722 )	4 years ago
Cyril Tovena	d9380eaac9	Add a target to find dead link in our documentation. (#3713 ) This is not part of the CI because it can be throttle and it's not super fast ~2min. But still usefull to check that no link are down if we make some big structure changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	ae5548c09d	Fluent-bit git repo link fix (#3714 )	4 years ago
Joaquin Menchaca	405e815008	Update docker.md for 2.2.1 (#3703 ) * Update docker.md for 2.2.1 * update to correspond to recent release of 2.2.1 * Version Bump docker-compose.yaml docker-compose.yaml has fallen several versions behind, this refreshes to recent release.	4 years ago
Julien Duchesne	bbb82b5781	Deploy loki with Drone plugin instead of CircleCI job (#3708 ) We are deprecating the CircleCI job Also, I moved secrets to Vault instead of Drone repository secrets Note that this will require Flux in all environments that are updated Issue: grafana/deployment_tools#5307	4 years ago
Cyril Tovena	f408e05ad4	Add the ability to wrap the roundtripper of the logcli client. (#3678 ) I actually did that on promtail on the previous PR, but logcli was required. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	03aab3deca	better operational dashboard annotations via diff logger (#3697 )	4 years ago
Cyril Tovena	14fa7d8f3a	Removes file sync syscall for compaction. (#3693 ) Because we mostly download, read and upload but also because all those operations are idempotent we don't need to try to outsmart the OS by forcefully syncing the file to disk. The OS will use cache on files and we'll most likely not suffer from high disk io. This was a conclusion I made after some digging of why the operation of compaction+retention was a bit slow sometimes. (+5s) I've tried the PR in dev and the whole process execute now below 40s whereas because it was 4min. strace before: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 75.75 12.333637 2122 5812 561 futex 10.81 1.760698 242 7267 nanosleep 7.69 1.252697 88 14287 epoll_pwait 2.86 0.464888 22 21608 write 1.08 0.175313 19479 9 fsync <---------- 1.06 0.172173 15 11249 4334 read 0.40 0.064422 2147 30 15 unlinkat 0.15 0.023945 19 1246 madvise 0.09 0.015288 1529 10 munmap 0.04 0.006098 23 263 tgkill 0.03 0.004134 16 263 getpid ``` strace after: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 77.01 58.085485 4599 12631 840 futex 9.81 7.395901 230 32145 nanosleep 8.34 6.287953 175 35882 epoll_pwait 3.33 2.512512 18 141141 write 0.79 0.596351 16 37911 13536 read 0.51 0.382579 3985 96 48 unlinkat 0.08 0.058465 1720 34 munmap 0.04 0.031904 26 1219 tgkill 0.03 0.023962 22 1082 madvise 0.02 0.013831 11 1219 getpid 0.01 0.011042 9 1219 12 rt_sigreturn 0.01 0.005348 111 48 fdatasync 0.01 0.003785 36 105 openat 0.00 0.003603 8 435 pwrite64 0.00 0.003005 13 229 200 epoll_ctl 0.00 0.001828 19 96 34 newfstatat ``` As you can see the fsync and write are less intensive after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	5c4ba10fb9	Use model.Duration for easy yaml/json marshalling. (#3687 ) * Use model.Duration for easy yaml/json marshalling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * test fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
3Xpl0it3r	554a7413d3	fix the promtail-default-config download link in doc (#3647 )	4 years ago
Kris Budde	33313158f0	added timezone to promtai dry-run logger output #3679" (#3683 ) Signed-off-by: Kris Budde <Kris@budd.ee>	4 years ago
Roger Steneteg	735e8dc17d	Remove targets that dont have any tails, in order for the target to be removed and recreated during next sync (#3587 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	4 years ago
Cyril Tovena	7dcc2eae6f	Fixes a panic with the frontend when use with downstream URL. (#3686 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	1995eaa426	Pass in the now value to the retention. (#3673 ) This allows to call model.Time only once per table which is reasonable for accuracy and this will save some cpu cycle. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	aa544a9dff	cleanup boltdb files failing to open during loading tables which are possibly corrupt (#3605 )	4 years ago
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677 ) * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672 ) Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	dbeb9840d1	Add a dashboard for retention to the loki-mixin. (#3658 ) * Add a dashboard for retention to the loki-mixin. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update dashboard to reflects last changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665 ) * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	d88f3996ea	revendor compatibility: various prom+k8s+cortex (#3664 ) * revendor compatibility: various prom+k8s+cortex * fix logcli client instantiation * go mod tidy	4 years ago
Cyril Tovena	04f4f82b8e	Adds the ability to provide a tripperware to Promtail client. (#3654 ) Fixes #3608 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642 ) * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ruppsn	771d33b69e	Update labels.md in Loki Docs (#3648 ) Complement to https://github.com/grafana/loki/pull/3641	4 years ago
Trevor Whitney	2f29e8e583	add note about wildcard log patterns and log rotation (#3655 ) Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Periklis Tsirakidis	63e25c9c15	Add Red Hat to adopters (#3644 ) * Add Red Hat to adopters * Update ADOPTERS.md	4 years ago
Cyril Tovena	ae23b7e4ed	Rejects push requests with streams without labels. (#3643 ) * Rejects push requests with streams without labels. Fixes #3638 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing commit Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	e71cb454ca	removes naming collision (#3641 )	4 years ago
Michel Hollands	9137ef921b	Update cortex to 1.8 (#3627 ) * Update Cortex to 1.8.1 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Ursula Kallio	f0715a2c8c	Fix wget link. (#3633 )	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632 ) Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	4 years ago
Ed Welch	a9d85de4aa	Re-license to AGPLv3 (#3630 ) For a rationale as to why this was done, please see * https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/ * https://grafana.com/blog/2021/04/20/qa-with-our-ceo-on-relicensing/ Signed-off-by: Richard Hartmann <richih@richih.org> Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Kaviraj	3c78579676	Add json struct tags to limits. (#3628 )	4 years ago
Michel Hollands	515f82ee1e	Add interceptor override and make ingester and cfg public (#3618 ) * Add interceptor override and make ingester and cfg public Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extraneous comment Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Ed Welch	31bb757c3b	update CI for branch rename to main (#3625 )	4 years ago
Ed Welch	0a02aecdb9	fix drone dependencies for manifest (#3624 )	4 years ago

1 2 3 4 5 ...

2213 Commits (f6dcd3f3a44fdb9dbba3e1e0a3881a8e655ba4e0) All Branches Search

2213 Commits (f6dcd3f3a44fdb9dbba3e1e0a3881a8e655ba4e0)

All Branches