From 42fb6aed0760668cf932b4e25b062fea583e86f8 Mon Sep 17 00:00:00 2001 From: Daniel Lee Date: Mon, 10 Dec 2018 22:55:25 +0100 Subject: [PATCH 1/3] readme: adds docker instructions and updates the intro section. Signed-off-by: Daniel Lee --- README.md | 73 ++++++++++++++++++++++++++++---- docs/docker-compose.yaml | 18 ++++++++ docs/promtail-docker-config.yaml | 19 +++++++++ 3 files changed, 102 insertions(+), 8 deletions(-) create mode 100644 docs/docker-compose.yaml create mode 100644 docs/promtail-docker-config.yaml diff --git a/README.md b/README.md index 2023e07733..bd7340b04a 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,68 @@ [![CircleCI](https://circleci.com/gh/grafana/loki/tree/master.svg?style=svg&circle-token=618193e5787b2951c1ea3352ad5f254f4f52313d)](https://circleci.com/gh/grafana/loki/tree/master) [Design doc](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit) -Loki is a horizontally-scalable, highly-available, multi-tenant, log aggregation -system inspired by Prometheus. It is designed to be very cost effective, as it does -not index the contents of the logs, but rather a set of labels for each log stream. +Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream. -## Run it locally +Compared to other log aggregation systems, Loki's main features are: + +- does not parse incoming logs, or do full text indexing. By storing plain text logs and only indexing the metadata, Loki is simpler to operate and cheaper to run. +- indexes and groups log streams using the same labels you’re already using with Prometheus. +- especially good fit for storing Kubernetes logs and seamless to switch between metrics and logs using the same Kubernetes labels that you’re already using with Prometheus. +- native support in Grafana (already in the nightly builds, will be included in Grafana 6.0). + +Loki consists of 3 components: + +- `loki` is the main server, responsible for storing logs and processing queries. +- `promtail` is the agent, responsible for gathering logs and sending them to loki. +- Grafana for the UI. + +## Install + +Currently there are two ways to install Loki, docker and building from source (precompiled binaries coming soon). + +### Install Using Docker + +The Docker images for [Loki](https://hub.docker.com/r/grafana/loki/) and [Promtail](https://hub.docker.com/r/grafana/promtail/) are available on DockerHub. + +To test locally using `docker run`: + +1. Create a docker network that the docker containers can share: + ```bash + docker network create loki + ``` +2. Start the Loki server: + ```bash + docker run --name loki --network=loki -p 3100:3100 --volume "$PWD/docs:/etc/loki" grafana/loki:master-8fa9461 -config.file=/etc/loki/loki-local-config.yaml + ``` +3. Then start the Promtail agent. The default config polls the contents of your `/var/log` directory. + ```bash + docker run --name promtail --network=loki --volume "$PWD/docs:/etc/promtail" --volume "/var/log:/var/log" --network="container:loki" grafana/promtail:make-images-static-26a87c9 -config.file=/etc/promtail/promtail-local-config.yaml + ``` +4. If you also want to run Grafana in docker: + ```bash + docker pull grafana/grafana:master + docker run --name grafana --network=loki -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master +5. Follow the steps for configuring the datasource in Grafana in the section below and set the URL field to: `http://loki:3100` + +Another option is to use the docker-compose file in the docs directory: + +1. git clone this repo locally (or just copy the contents of the docker-compose file locally into a file named `docker-compose.yaml`) +2. `cd loki/docs` +3. `docker-compose up` + +### Configuring the Loki Datasource in Grafana + +Grafana ships with built-in support for Loki in the [latest nightly builds](https://grafana.com/grafana/download). Loki support will be officially released in Grafana 6.0. + +1. Open the side menu by clicking the Grafana icon in the top header. +2. In the side menu under the Dashboards link you should find a link named Data Sources. +3. Click the `+ Add data source` button in the top header. +4. Choose Loki from the list. +5. The http URL field should be the address of your Loki server e.g. `http://localhost:3100` + +Read more about the Explore feature in the [Grafana docs](http://docs.grafana.org/features/explore) and on how to search and filter logs with Loki. + +### Build and Run Loki Locally Loki can be run in a single host, no-dependencies mode using the following commands. @@ -16,7 +73,7 @@ and sending them to loki and `grafana` as the UI. To run loki, use the following commands: -``` +```bash $ go build ./cmd/loki $ ./loki -config.file=./docs/loki-local-config.yaml ... @@ -24,7 +81,7 @@ $ ./loki -config.file=./docs/loki-local-config.yaml To run promtail, use the following commands: -``` +```bash $ go build ./cmd/promtail $ ./promtail -config.file=./docs/promtail-local-config.yaml ... @@ -32,8 +89,8 @@ $ ./promtail -config.file=./docs/promtail-local-config.yaml Grafana is Loki's UI, so you'll also want to run one of those: -``` +```bash $ docker run -ti -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master ``` -In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Logging" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy! \ No newline at end of file +In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Logging" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy! diff --git a/docs/docker-compose.yaml b/docs/docker-compose.yaml new file mode 100644 index 0000000000..368f325b6a --- /dev/null +++ b/docs/docker-compose.yaml @@ -0,0 +1,18 @@ +version: "2" + +services: + loki: + image: grafana/loki:master-8fa9461 + ports: + - "3100:3100" + volumes: + - $PWD:/etc/loki + command: -config.file=/etc/loki/loki-local-config.yaml + + promtail: + image: grafana/promtail:make-images-static-26a87c9 + volumes: + - $PWD:/etc/promtail + - /var/log:/var/log + command: + -config.file=/etc/promtail/promtail-docker-config.yaml diff --git a/docs/promtail-docker-config.yaml b/docs/promtail-docker-config.yaml new file mode 100644 index 0000000000..8bc5f24ca7 --- /dev/null +++ b/docs/promtail-docker-config.yaml @@ -0,0 +1,19 @@ +server: + http_listen_port: 0 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +client: + url: http://loki:3100/api/prom/push + +scrape_configs: +- job_name: system + entry_parser: raw + static_configs: + - targets: + - localhost + labels: + job: varlogs + __path__: /var/log From ab46d576aa5daf4069b462c14d511a08d0d6990b Mon Sep 17 00:00:00 2001 From: Daniel Lee Date: Mon, 10 Dec 2018 23:48:59 +0100 Subject: [PATCH 2/3] adds grafana to docker-compose example Signed-off-by: Daniel Lee --- docs/docker-compose.yaml | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/docker-compose.yaml b/docs/docker-compose.yaml index 368f325b6a..0cd32791ef 100644 --- a/docs/docker-compose.yaml +++ b/docs/docker-compose.yaml @@ -1,4 +1,7 @@ -version: "2" +version: "3" + +networks: + loki: services: loki: @@ -8,6 +11,8 @@ services: volumes: - $PWD:/etc/loki command: -config.file=/etc/loki/loki-local-config.yaml + networks: + - loki promtail: image: grafana/promtail:make-images-static-26a87c9 @@ -15,4 +20,15 @@ services: - $PWD:/etc/promtail - /var/log:/var/log command: - -config.file=/etc/promtail/promtail-docker-config.yaml + -config.file=/etc/promtail/promtail-docker-config.yaml + networks: + - loki + + grafana: + image: grafana/grafana:master + ports: + - "3000:3000" + environment: + GF_EXPLORE_ENABLED: "true" + networks: + - loki From c52aa211a8de358498fa78d5f898bdc342615d96 Mon Sep 17 00:00:00 2001 From: Daniel Lee Date: Tue, 11 Dec 2018 10:00:30 +0100 Subject: [PATCH 3/3] readme changes Signed-off-by: Daniel Lee --- README.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index bd7340b04a..af9ee2a253 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,12 @@ Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream. -Compared to other log aggregation systems, Loki's main features are: +Compared to other log aggregation systems, Loki: -- does not parse incoming logs, or do full text indexing. By storing plain text logs and only indexing the metadata, Loki is simpler to operate and cheaper to run. +- does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing the metadata, Loki is simpler to operate and cheaper to run. - indexes and groups log streams using the same labels you’re already using with Prometheus. -- especially good fit for storing Kubernetes logs and seamless to switch between metrics and logs using the same Kubernetes labels that you’re already using with Prometheus. -- native support in Grafana (already in the nightly builds, will be included in Grafana 6.0). +- ia an especially good fit for storing Kubernetes logs. It is seamless to switch between metrics and logs using the same Kubernetes labels that you’re already using with Prometheus. +- has native support in Grafana (already in the nightly builds, will be included in Grafana 6.0). Loki consists of 3 components: @@ -41,8 +41,8 @@ To test locally using `docker run`: ``` 4. If you also want to run Grafana in docker: ```bash - docker pull grafana/grafana:master docker run --name grafana --network=loki -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master + ``` 5. Follow the steps for configuring the datasource in Grafana in the section below and set the URL field to: `http://loki:3100` Another option is to use the docker-compose file in the docs directory: @@ -51,6 +51,18 @@ Another option is to use the docker-compose file in the docs directory: 2. `cd loki/docs` 3. `docker-compose up` +If you have have an older cached version of the grafana/grafana:master container then start by doing either: + +```bash +docker pull grafana/grafana:master +``` + +Or for docker-compose: + +```bash +docker-compose pull +``` + ### Configuring the Loki Datasource in Grafana Grafana ships with built-in support for Loki in the [latest nightly builds](https://grafana.com/grafana/download). Loki support will be officially released in Grafana 6.0. @@ -93,4 +105,4 @@ Grafana is Loki's UI, so you'll also want to run one of those: $ docker run -ti -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master ``` -In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Logging" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy! +In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Loki" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy!