apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	7fff660144	Add UDP protocol support to Promtail's syslog target (#5790 ) Until now, the syslog target in Promtail did only support TCP. This PR adds support for sending syslog messages to Promtail (via ng-syslog or rsyslog) via UDP. UPD for syslogs can be used if you prefer speed over guaranteed delivery. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Paschalis Tsilias	179d741c5f	promtail: Add support for exclusion patterns in `static_config` (#5943 ) * PR base Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add CHANGELOG entry; updated documentation for new label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Follow logic from t.path in exlusion case; always handle matching Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make error message unique Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Only run if t.pathExclude is set Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Revert "Only run if t.pathExclude is set" This reverts commit 17abb006ce141071c4bca4b7d133bea2aa314594.	4 years ago
Christian Haudum	5ecbbbe5f5	Mark GELF support in Promtail as experimental (#5697 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Cyril Tovena	7f71c5f051	Promtail pull cloudflare logs (#4813 )	4 years ago
Hiroki Sakamoto	d61dd1872a	Expose Kafka message key in labels (#4745 ) * expose kafka message key in labels * add changelog entry * improve performance * goimported * fix label name in test * add empty message key test case * pass actual value in message key	4 years ago
James Callahan	21556efa87	Promtail: improve GCP labels (#4628 ) * Promtail: GCP resource label prefix is now '__gcp_resource_labels_' * Promtail: turn GCP 'resource_type' into the internal label '__gcp_resource_type' * Promtail: add internal label with logName * Add upgrade notes for gcplog changes	4 years ago
Cyril Tovena	81e67f89b3	Add GELF support for Promtail. (#4744 ) * Add Gelf support for Promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move change to MAIN Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update graylog to grafana remote. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test and check mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add log messages around startup and shutdown Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	3fef08ca1a	2.4 release notes, changelog, upgrade guide (#4658 ) * update changelog, release notes, and upgrade guide for 2.4 * tweak release notes a little * make path relative for kafka config * update upgrade guide * more docs fix the cache size not applied to results cache. * fix prefix name in ring_config	4 years ago
Cyril Tovena	c8f3df33cf	Promtail Kafka target (#4568 ) * Adds a kafka target manager in promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar test for the fanout client. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * final adjustement Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding topics discovery. Still needs to finish tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off testing it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Revert config changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add group id as discovered label Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tools for running kafka and testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted shell Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update sarama to compile in ARM. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for kafka target. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve code comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add a few s's * Better cancellation support. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * adds `__meta_kafka_` suffix to discovered labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
James Callahan	a92541bd84	Promtail: no need for GCP promtail_instance label now that loki supports out-of-order writes (#4556 ) Closes #4474	4 years ago
Karen Miller	24accf6a5b	Docs: correctly represent product name (#4416 ) The product name is Grafana Loki. The first use of the product name on each web page should be the full product name. After that, within each page, it is fine to use just Loki. This PR also set up some frontmatter with weights where it was missing on a few files, and correct where "Loki" was not capitalized.	4 years ago
Karen Miller	11a0d28b61	Docs: first draft, Loki accepts out-of-order writes (#4237 ) * Docs: first draft, Loki accepts out-of-order writes * Update docs/sources/architecture/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Karsten Jeschkies	29d2394962	Link Kubernetes service discovery configuration. (#4206 ) * Link Kubernetes service discovery configuration. * Use `configuration`.	4 years ago
Karen Miller	475560ba13	Corrected capitalization and formatting of "Promtail" (#3860 )	5 years ago
Julien Duchesne	6f5e6b637b	Fix broken link in `Windows Event Log` scraping docs (#3828 )	5 years ago
Charlie N	54bc1468b4	Added docs for GCP internal labels. (#3529 ) Signed-off-by: Whyeasy <Whyeasy@users.noreply.github.com>	5 years ago
Cyril Tovena	594b596d67	Fixes broken link in the documentation. (#3516 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Kaviraj	14b2c093c1	Add unique promtail_instance id to labels for gcptarget (#3501 ) * Add unique promtail_instance id to labels for gcptarget Rationale: To make labelset unique per promtail instance to avoid out-of-order errors * Update docs/sources/clients/promtail/scraping.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	5 years ago
Cyril Tovena	72b3acb925	Windows events (#3246 ) * First commit for windows event targets. I had to add couple of new go modules and hook the windows log in. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the ability to save where we are. * Finished setup tests and comments. * nits * adding documentation * go tidy * lint windows * nope * add manager for non windows * don't lint forked package * mod check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/promtail/targets/windows/bookmark.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Use passed-in Prometheus registerer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * testing out windows drone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * type exec. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * docker image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go/go.exe Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * build windows on container. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * trying another image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo in image Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixed local timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests with timezone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes timezone tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update logstash since the last image is not working anymore 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083 ) * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Douglas Hall	e64409540b	Doc: Vague link text. (#2698 ) * Fixes remove vague link text #2680 * Fix remaining 'docs' links. * Accept suggestion. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Change HTTP API to Ruler API. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Correct typo. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Capitalize Github. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Capitalize Promtail. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Add copy-edit suggestions from @achatterjee-grafana. * Incorporate @owen-d's changes. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>	5 years ago
Madhav Jivrajani	356e2765ff	[DOCS] change format of unordered lists in tech docs (#2724 )	5 years ago
Ward Bekker	3f7bc33aaa	DOCS: clarified globbing behaviour of __path__ of the doublestar library (#2437 ) * clarified globbing behaviour of __path__ of the doublestar library * Update docs/sources/clients/promtail/scraping.md Co-authored-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Robby Milo	5a8bc848db	Sync docs to website (#2378 ) * update docs location * add workflow * update docs content * update content * cleanup	5 years ago
Edward Welch	a85ce664ea	improve systemd field docs Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Robert Fratto	874d016bd7	promtail: add support for passing through journal entries as JSON This commit adds an extra configuration field (`json`) to the journal target configuration schema. When set to true, messages from the journal target will be passed through as JSON, containing all of the fields from the entry as keys and their values as the value. For backwards compatibility, the value for `json` defaults to false. Closes #1288.	6 years ago
Sebastian Widmer	f0f6f24926	pkg/promtail: IETF Syslog (RFC5424) Support (#1275 ) * IETF syslog (rfc5254) support * Upgrade go-syslog to correctly released v2.0.1 * Incorporate feedback from new linter * Update documentation with relevant RFC sections. * Incorporate feedback from review * Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections * Incorporate further feedback from review * Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing() * Switch to "nontransparent" parser * Further improvements to the documentation	6 years ago
Robert Fratto	715f215c50	pkg/promtail: remove journal target forced path (#1298 ) * pkg/promtail: remove journal target forced path The forced default path for the journal target (/var/log/journal) did not apply to all Linux systems: systemd actually defaults to reading from _two_ default paths: /var/log/journal and /run/log/journal. Not setting an explicit default will enable both of those paths to be searched for journal logs. The comment mentioning the local machine ID was technically accurate, but is normally dealt with by bind mounting /etc/machine-id. That's the standard approach for running fluent-bit in a container, and should be the standard approach for Promtail as well. Fixes #1286.	6 years ago
Robert Fratto	91bbe86a73	Document systemd journal scraping (#1261 )	6 years ago
Björn Rabenstein	1b251e54a4	Simplify regexp's (#1202 ) This is mostly to remove redundant anchoring (as it makes people believe that Prometheus's regexp's aren't anchored by default, which will backfire badly at some point), but it also removes useless grouping (again to avoid confusion). Signed-off-by: beorn7 <beorn@grafana.com>	6 years ago
Robert Fratto	65ba42a6e7	Documentation Rewrite (#982 ) * docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)	6 years ago

24 Commits (d2c099ac1db23fb0c29e55b6df17e177f356fda3)