apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
4688 Commits
845 Branches
311 Tags
655 MiB
Tag: Branch: Tree: dac3b84d08
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'dac3b84d08'
${ noResults }
loki/docs/sources/operations/scalability.md

33 lines
2.3 KiB
Raw Normal View History Unescape

Sync docs to website (#2378) * update docs location * add workflow * update docs content * update content * cleanup
5 years ago
---
title: Scalability
Fix documentation linter errors (#8229)
2 years ago
description: Scaling with Grafana Loki
Doc updates: (#3823) - Revise section titles - Add weight metadata to specify ordering of subsections - Remove TOC info from content prose, as it is already in the right side navigation panel - Remove a redundant section about upgrading.
4 years ago
weight: 30
Sync docs to website (#2378) * update docs location * add workflow * update docs content * update content * cleanup
5 years ago
---
Fix documentation linter errors (#8229)
2 years ago
# Scalability
Documentation Rewrite (#982) * docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)
6 years ago
Fix documentation linter errors (#8229)
2 years ago
See [Loki: Prometheus-inspired, open source logging for cloud natives](/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)
Docs: correctly represent product name (#4416) The product name is Grafana Loki. The first use of the product name on each web page should be the full product name. After that, within each page, it is fine to use just Loki. This PR also set up some frontmatter with weights where it was missing on a few files, and correct where "Loki" was not capitalized.
4 years ago
for a discussion about Grafana Loki's scalability.
Documentation Rewrite (#982) * docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)
6 years ago
When scaling Loki, operators should consider running several Loki processes
partitioned by role (ingester, distributor, querier) rather than a single Loki
Sync docs to website (#2378) * update docs location * add workflow * update docs content * update content * cleanup
5 years ago
process. Grafana Labs' [production setup](https://github.com/grafana/loki/blob/master/production/ksonnet/loki)
Documentation Rewrite (#982) * docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)
6 years ago
contains `.libsonnet` files that demonstrates configuring separate components
and scaling for resource usage.
Document operation with the query scheduler. (#4100) * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Document operation with the query scheduler. * Replace Cortex with Loki image. * Remove unused Corte config. * Correct merger errors. * Clear up confusing sentence. * Update docs/sources/operations/scalability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
4 years ago
## Separate Query Scheduler
Docs: remove Cortex references (#6079) * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update
3 years ago
The Query frontend has an in-memory queue that can be moved out into a separate process similar to the
Fix documentation linter errors (#8229)
2 years ago
[Grafana Mimir query-scheduler](/docs/mimir/latest/operators-guide/architecture/components/query-scheduler/). This allows running multiple query frontends.
Document operation with the query scheduler. (#4100) * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Document operation with the query scheduler. * Replace Cortex with Loki image. * Remove unused Corte config. * Correct merger errors. * Clear up confusing sentence. * Update docs/sources/operations/scalability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
4 years ago
[DOC] Fix broken links in docs (#8065) **What this PR does / why we need it**: This PR fixes several broken links in the Promtail, operations, and install sections of the doc.
2 years ago
To run with the Query Scheduler, the frontend needs to be passed the scheduler's address via `-frontend.scheduler-address` and the querier processes needs to be started with `-querier.scheduler-address` set to the same address. Both options can also be defined via the [configuration file]({{< relref "../configuration" >}}).
Document operation with the query scheduler. (#4100) * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Document operation with the query scheduler. * Replace Cortex with Loki image. * Remove unused Corte config. * Correct merger errors. * Clear up confusing sentence. * Update docs/sources/operations/scalability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
4 years ago
[DOC] Fix broken links in docs (#8065) **What this PR does / why we need it**: This PR fixes several broken links in the Promtail, operations, and install sections of the doc.
2 years ago
It is not valid to start the querier with both a configured frontend and a scheduler address.
Document operation with the query scheduler. (#4100) * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Document operation with the query scheduler. * Replace Cortex with Loki image. * Remove unused Corte config. * Correct merger errors. * Clear up confusing sentence. * Update docs/sources/operations/scalability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
4 years ago
Docs: remove Cortex references (#6079) * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update
3 years ago
The query scheduler process itself can be started via the `-target=query-scheduler` option of the Loki Docker image. For instance, `docker run grafana/loki:latest -config.file=/mimir/config/mimir.yaml -target=query-scheduler -server.http-listen-port=8009 -server.grpc-listen-port=9009` starts the query scheduler listening on ports `8009` and `9009`.
Add the option to configure memory ballast for Loki (#5081) * Add memory ballast * Add changelog entry * Add documentation Add additional information to Flag description * Ensure that ballast_bytes can be configured from config file Add documentation * Add Operational scalability documentation related to memory ballast * Restore config.ballast-bytes cli flag * Move config.ballast-bytes cli flag declaration Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
3 years ago
Docs: Update memory ballast documentation (#5106) * Add memory ballast * Add changelog entry * Add documentation Add additional information to Flag description * Ensure that ballast_bytes can be configured from config file Add documentation * Add Operational scalability documentation related to memory ballast * Restore config.ballast-bytes cli flag * Move config.ballast-bytes cli flag declaration * Update memory ballast documentation as per https://github.com/grafana/loki/pull/5081\#pullrequestreview-848460132 Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
3 years ago
## Memory ballast
Add the option to configure memory ballast for Loki (#5081) * Add memory ballast * Add changelog entry * Add documentation Add additional information to Flag description * Ensure that ballast_bytes can be configured from config file Add documentation * Add Operational scalability documentation related to memory ballast * Restore config.ballast-bytes cli flag * Move config.ballast-bytes cli flag declaration Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
3 years ago
Docs: Update memory ballast documentation (#5106) * Add memory ballast * Add changelog entry * Add documentation Add additional information to Flag description * Ensure that ballast_bytes can be configured from config file Add documentation * Add Operational scalability documentation related to memory ballast * Restore config.ballast-bytes cli flag * Move config.ballast-bytes cli flag declaration * Update memory ballast documentation as per https://github.com/grafana/loki/pull/5081\#pullrequestreview-848460132 Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
3 years ago
In compute-constrained environments, garbage collection can become a significant performance factor. Frequently-run garbage collection interferes with running the application by using CPU resources. The use of memory ballast can mitigate the issue. Memory ballast allocates extra, but unused virtual memory in order to inflate the quantity of live heap space. Garbage collection is triggered by the growth of heap space usage. The inflated quantity of heap space reduces the perceived growth, so garbage collection occurs less frequently.
Configure memory ballast using the ballast_bytes configuration option.