apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677 ) * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Lashan	e0f0ee0585	Update README.md (#3552 ) With Go 1.11 it's not possible to build the Fluent Bit plugin with the following error. ``` build github.com/sercand/kuberesolver: cannot find module for path github.com/sercand/kuberesolver ``` Needed to use Go 1.16.2 to resolve the issue.	4 years ago
Ed Welch	36ec92ce5c	Update Go and Alpine versions (#3481 ) * Upgrade go to 1.16 and alpine to 3.13 * update drone to use new build image version * gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446 ) * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460 ) * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	4 years ago
Filip Poboril	8c98ee267c	Fix fluentd plugin when kubernetes labels were missing (#3358 ) When kubernetes labels were missing in the received data, plugin failed with error "undefined method `each_key' for nil:NilClass".	4 years ago
Ed Welch	0d70e340f2	Tooling: Update chunks-inspect to understand the new chunk format as well as new compression algorithms (#3377 ) * Update chunks-inspect to understand the new chunk format as well as new compression algorithms * forgot to add go mod files * Update cmd/chunks-inspect/loki.go Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Cyril Tovena	d1b9912602	Add pprof endpoint to docker-driver. (#3331 ) * Add pprof endpoint to docker-driver. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	bca29c011f	Embed timezone data in Promtail. (#3332 ) Fixes #2995 Fixes #1361 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	8a9b94a8bc	removes recover flag (#3326 )	4 years ago
Simon Baier	9a58f83a90	fix fluent-bit output plugin generating invalid JSON (#3240 ) * fix fluent-bit output plugin generating invalid JSON * fix golint error	4 years ago
Kaviraj	59a34f9867	Make incoming labels from gcp into Loki internal labels. (#3285 ) * Make incoming labels from gcp into Loki internal labels. If these labels are significant, we can always use relabeling. * Add support for rebeling in gcptarget * Test for SnakeCase	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Chance Zibolski	054b8fcc91	logcli: Refactor logcli Client interface to use time.Time and time.Duration for LiveTailQueryConn method (#3271 )	4 years ago
Cyril Tovena	72b3acb925	Windows events (#3246 ) * First commit for windows event targets. I had to add couple of new go modules and hook the windows log in. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the ability to save where we are. * Finished setup tests and comments. * nits * adding documentation * go tidy * lint windows * nope * add manager for non windows * don't lint forked package * mod check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/promtail/targets/windows/bookmark.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Use passed-in Prometheus registerer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * testing out windows drone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * type exec. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * docker image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go/go.exe Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * build windows on container. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * trying another image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo in image Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixed local timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests with timezone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes timezone tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update logstash since the last image is not working anymore 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	b4eeddc710	Loki migrate-tool (#3151 ) * basics working, needs some more cleanup and more flags for config and a Dockerfile Signed-off-by: Edward Welch <edward.welch@grafana.com> * add label matchers start end time params and ability to remap the chunk to a different user Signed-off-by: Edward Welch <edward.welch@grafana.com> * adding dockerfile and makefile instructions Signed-off-by: Edward Welch <edward.welch@grafana.com> * push out some of the limits show errors from index query * for testing have docker container not run migrate command but instead stay running by tailing /dev/null * big overhaul to add parallelization * adding better stats * adding retry logic * tweaking retry logic * tweaking retry logic * work around changes in latest cortex Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix up imports and changes since the last time this was merged added a readme fixed up the makefile added some comments * fixing some of my garbage * fixing more of my garbage * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * tabs not spaces Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Leo Takaoka	931fdcbb93	fix: Insert a space in the error message (#3225 )	4 years ago
Michał Sochoń	3bb7f52f1f	Fix: Update docs for logcli (#3212 ) * Update docs for logcli Fix description for logcli, use proper flag names. Extend docs with example using timestamp. Signed-off-by: Michał Sochoń <kaszpir@gmail.com> * Update cmd/logcli/main.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163 ) * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Robert Fratto	628c1fae95	pass a prometheus registerer to promtail components (#3175 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
Heds Simons	d08871d4c7	fluent-bit: Rename Fluent Bit plugin output name. (#2974 ) This prevents a clash with the Loki plugin now included with the Fluent Bit agent by default. The output plugin is now named `grafana-loki`. Signed-off-by: Heds Simons <heds@whaleway.net> (cherry picked from commit `b195e94f39`)	4 years ago
Heds Simons	b195e94f39	fluent-bit: Rename Fluent Bit plugin output name. (#2974 ) This prevents a clash with the Loki plugin now included with the Fluent Bit agent by default. The output plugin is now named `grafana-loki`. Signed-off-by: Heds Simons <heds@whaleway.net>	5 years ago
Cyril Tovena	c4faa579e3	Asynchronous Promtail stages (#2996 ) * Introducing go pipelines to promtail. Based off @jeschkies idea. WIP Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More tests and code cleanup. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip breaking things. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests, adding Stop to the interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar fixes for tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Close correctly client before reading. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use defer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes lint issues Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Attempt to fix journald without linux. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journald json test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing stop in the filetarget. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logentry/stages/match_test.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * First set of feeback review fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use newEntry as suggested by Karsten. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Kaviraj	410b9d9adc	fix(docker-driver): Propagate promtail's `client.Stop` properly (#2898 ) * impr(Makefile): add support to take docker-driver image via ENV * impr(promtail/client): Add cancelling mechanism for batchSend stores `context` and its `cancel` in client and uses it to make upstream calls that need `ctx`. Also properly cancel's all the upstream calls that using `ctx` when client's Stop is called. * Add `StopNow()` method to stop the client without retries. Use `StopNow()` from the docker-driver * Add test for StopNow * reverting back the client.Timeout	5 years ago
Owen Diehl	4d9865acd4	Adds WAL support (experimental) (#2981 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * prevent shared access bug with tailers and entry pool * removes stream push entry pool optimization	5 years ago
Cyril Tovena	221fd382a9	Move CI to helm3. (#2986 ) * Move CI to helm3. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rollback to v3.2.3 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	0b84ef8868	Promtail: Update debian image and use a newer libsystemd (#2957 ) * use a newer libsystemd which understands compression needed in some of the newer OS distros * add comments	5 years ago
Yuichi Sawada	27dddef475	Fix logcli logql document url (#2968 )	5 years ago
Tom Wilkie	1c8dd3e11a	Add a few more instructions to logcli --help. (#2948 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	5 years ago
Keith Byrne	5793c4964e	Configuration: Support environment expansion in configuration (#2837 ) * Support environment expansion in configuration Inspired by https://github.com/cortexproject/cortex/pull/2147 Fixes #2311 Adds -config.expand-env as flag to loki and promtail. * Disable env expansion on logcli * Small nit updates	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877 ) * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	6bacfa1f3f	Bump fluentd plugin (#2825 ) so we can push it. ``` gem push fluent-plugin-grafana-loki-1.2.15.gem /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin19/rbconfig.rb:229: warning: Insecure world writable dir /Users/ctovena/go/src/github.com/grafana/loki/cmd/logcli in PATH, mode 040777 Pushing gem to https://rubygems.org... Repushing of gem versions is not allowed. Please use `gem yank` to remove bad gem releases. ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Anders Ingemann	081cc024d6	fluent-plugin: Improve escaping in key_value format (#2434 ) The key_value format does not seem to be well specified. This change assumes "logfmt" as the format and escapes values accordingly: The code itself is lifted from here: `e279d43cde/lib/stringify.js` Grafana seems to handle values escaped in this manner much better than before. e.g.: query="{program="loki"}") was shown as (IIRC) query="{program= and is now displayed correctly. As an added bonus only values that require quoting will actually be quoted, improving readability and derived fields regexes.	5 years ago
Ed Welch	af6e8cb6c9	Update local and docker default config files to use boltdb-shipper with a few other config changes (#2803 ) * change the default local and docker config files to use boltdb-shipper also change them to use chunk_target_size * create ruler local filesystems directories if they don't exist change the ruler paths to be a little more consistent * don't need to create temp dir, also checking for the storage type * add compactor config to docker config file.	5 years ago
Dr. Jan-Philip Gehrcke	4e661cde0e	FluentD loki plugin: add support for bearer_token_file parameter (#2739 ) * fluentd plugin: improve param descriptions * fluentd plugin: make code comments more correct * fluentd plugin: rename things for clarity * fluentd plugin: add bearer_token_file parameter * fluentd plugin: read and set bearer token * fluentd plugin: improve error message * fluentd-plugin: bump version * fluentd plugin: fix header, log success * fluentd plugin: improve log messages	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
dlemel8	9a3592c42c	add "verify-config" flag that load and verify configuration and exits with proper exit code (#2737 )	5 years ago
Aditya C S	772b7ff087	Logstash cpu usage fix (#2607 ) * Replace concurrent-ruby-edge with agent lib * Fix logstash cpu consumption * increment gemspec version * increment gemspec version * fix tests	5 years ago
Davor Kapsa	fffcfa6e81	fix some misspells (#2725 )	5 years ago
Ed Welch	d3bf21e774	Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710 ) * change the max batch size to 1MB for all the defaults including helm and fluent-bit, attempt to centralize this config a little where possible. * fix test	5 years ago
Jiajin Zheng	1b2bae46ba	fluent-bit plugin support tls (#2568 )	5 years ago
Owen Diehl	0d64d2d912	fixes ruler docs & includes ruler configs in cmd/configs + docker img (#2657 )	5 years ago
Ryan Lahfa	e9c9faf481	Docker logging driver: Add a keymod for the extra attributes from the Docker logging driver (#2459 ) * docker: add a keymod to replace . by _, making them "prometheus"-friendly * docker: fix the code * docker: replace also - by _	5 years ago
Sandeep Sukhani	65b539c2ab	add compactor details and other boltdb-shipper doc improvments (#2622 )	5 years ago
Sandeep Sukhani	143672e643	add support for building querytee (#2602 )	5 years ago
Tharun Rajendran	ab2dc9e2d3	Added logcli docker image (#2561 ) Signed-off-by: Tharun <rajendrantharun@live.com>	5 years ago
Sandeep Sukhani	501f93fa92	revendor cortex to latest master (#2543 )	5 years ago

1 2 3 4 5 ...

302 Commits (966bed72d3247a15c7f94a14ca13b536edb30f74)