apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ed Welch	2ac409c23e	Build: simplify how protos are built (#4639 ) * we always seem to be chasing our tails with how protos are generated and Makes use of timestamps to determine if files should be recompiled. Instead of touching files and altering timestamps always delete the compiled proto files when calling `protos` to make sure they are compiled every time. Removed targets to build protos, yaccs, and ragel files when trying to build loki or the canary. This isn't necesary, if you are changing these files you would know you need to build them and the `check-generated-files` should catch any changes to them not committed. * rm -rf * we have a race between check-generated-files and our other steps, so let that run first. also removing `check-generated-files` from the `all` target because its redundant with a separate step and could also race with the parallel lint * remove TOUCH_PROTOS * more cleanup of TOUCH_PROTOS	4 years ago
Callum Styan	096cf0cfaf	Update golang and loki-build-image image versions. (#4481 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	3ee8ed3b1d	Allow for loki-canary to generate a percentage of out of order log lines (#4126 ) * Allow for loki-canary to generate a percentage of log lines with out of order timestamps, percentage and out of order time range is all configurable. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Switch OoO min/max flags to time duration and properly calculate OoO percentage check. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change some variable names + add comments to make it more obvious what various parts of the canary code are actually doing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update pkg/canary/writer/writer.go * Add notes about usage of the out-of-order-percentage flag and the requirement to use pipeline stages. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	6b775ac31c	allow setting tenant id for querying logs from loki (#4185 )	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Ed Welch	36ec92ce5c	Update Go and Alpine versions (#3481 ) * Upgrade go to 1.16 and alpine to 3.13 * update drone to use new build image version * gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	221fd382a9	Move CI to helm3. (#2986 ) * Move CI to helm3. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rollback to v3.2.3 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877 ) * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	691d27c8c7	Update Loki build image (#2467 ) * updating build-image version * updating drone config * updating a few other references to the build image	5 years ago
Ed Welch	204e61afe3	Loki-Canary: Backoff retries on query failures, add histograms for query performance. (#2413 ) * Canary needs to wait before spot-checking entries some reasonable amount of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present. Adding two histograms for tracking the performance of the spot check and metric test queries * Adding backoff to queries if they fail specifically waiting 5mins when a 429 is received from Loki * fix order of parameters	5 years ago
Ed Welch	76ceec45d9	If an entry is missing on the websocket, start checking after `wait` but continue checking until `max-wait` (#2369 ) Refactored out a useful pruneList function	5 years ago
Ed Welch	2b145da7d7	Canary tweaks (#2350 ) * tweak the canary a bit, add a metric for websocket pings, fix some logging, improve the mutex locking a bit, add a read timeout on the websocket, add a lastmessage time checker to restart websocket * fix constructor * fix counter name change metric count reporting to two separate metrics, expected vs actual * add note to upgrade guide for metric name changes * fixed tests.... * Update docs/operations/upgrade.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Ed Welch	81b9decfb5	Loki-Canary: Add query spot checking and metric count checking (#2344 ) * Add spot check capability to canary to query certain values over time to make sure they are still in Loki. Add a context timeout to Loki queries * Updating canary to use /loki/api/v1 endpoints adding a metric test query * fixing metric names to have _total for counters * fix parsing of timestamp from log line and don't use entry timestamp * Tweak the metric test query to adjust the range to the time the canary has been running to get accurate results until the requested range has been reached. Add a total spot check entry counter to get a ratio of success * Add some debugging logs to troubleshoot missing entries * add more tests, fix some log lines * update docs, make the spot-check timer separate from prune entries * just how many paramaters can I add to the constructor.... * run prune entries on a separate thread, only run one at a time.	5 years ago
Owen Diehl	7682b32cba	Revert "Revert "Update go from 1.13 to 1.14. (#2013 )"" (#2029 ) * Revert "Revert "Update go from 1.13 to 1.14. (#2013)"" This reverts commit `e8fece69e3`. * Update cortex to latest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update go.sum and doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Lukas Grimm	17390fd45d	Canary: make stream configurable (#2259 ) * tried to fix canary in non kubernetes env * update doc	5 years ago
Owen Diehl	2d31d07680	Revert "Update go from 1.13 to 1.14. (#2013 )" (#2028 ) This reverts commit `8d6dc2c4ca`.	5 years ago
Cyril Tovena	8d6dc2c4ca	Update go from 1.13 to 1.14. (#2013 ) * Update go from 1.13 to 1.14. See https://golang.org/doc/go1.14 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * modules.txt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Joe Elliott	383eb6e70b	Addition of a `/suspend` endpoint to Loki Canary (#1891 ) * First pass loki canary pause Signed-off-by: Joe Elliott <number101010@gmail.com> * Write suspending msg to stderr Signed-off-by: Joe Elliott <number101010@gmail.com> * Added resume endpoint Signed-off-by: Joe Elliott <number101010@gmail.com> * Added documentation on new endpoints Signed-off-by: Joe Elliott <number101010@gmail.com> * Pass both signals to the same channel Signed-off-by: Joe Elliott <number101010@gmail.com> * lint Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed return on startCanary Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615 ) * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	29e4ad8cf8	TOUCH_PROTOS build arg for dockerfile (#1479 ) * TOUCH_PROTOS build arg for dockerfile * echos touch protos in dockerfile * touch protos correct var * make touch-protos options in all dockerfiles, run for amd builds	5 years ago
Xiang Dai	bb0a64a64b	fix all misspell (#1437 ) Signed-off-by: Xiang Dai <764524258@qq.com>	6 years ago
Ed Welch	191f87a376	Adding `zip` to the build image (#1364 ) Adding `drone` to the build image and updating the make target to run the drone command in the build image Re-adding `GO111MODULE=on` to the dependency import as it is still required and was removed in error. (Also updated the comment a little to reflect why it is still needed)	6 years ago
Peter Štibraný	49d0d0f0da	Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases. (#1095 ) * Extracted -mod=vendor into a variable * Use ?= to set variable only if not set yet. This enables "MOD_VENDOR= make all" use from command line * Enable/disable vendoring with VENDOR=true or false. Enabled by default * Make go generate use -mod=vendor depending on VENDOR flag * Use GOMOD to specify module version to use. GOMOD can be set to empty string, "vendor", "readonly" and in Go 1.14 to "mod". If not defined, defaults to "vendor". Non empty values are passed to -mod parameter to Go tools, and --modules-download-mode parameter for golangci-cli linting tool. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix versioning info when not using vendoring. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to use Golintci-cli v1.21.0 and Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to version 0.8.0. Changs: Golintci-cli v1.21.0, Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Renamed MOD_VENDOR to MOD_FLAG to better describe its function. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix version info in all command line tools. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use loki-build-image:0.8.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Marco Pracucci	de026a00ac	Fix compiled logproto protobuf (#1183 ) * Enforce recompile of protobuf sources on 'make check-generated-files' * Pinned version of protobuf in loki-build-image * Pinned version of goyacc in loki-build-image * Added 'make build-image-push' to Makefile * Documented how to release loki-build-image	6 years ago
Xiang Dai	b6faee205a	bump up golangci to 1.20 (#1133 ) * bump up golangci to 1.20 Signed-off-by: Xiang Dai <764524258@qq.com> * Built and published 0.7.3. Updated all references Signed-off-by: Joe Elliott <number101010@gmail.com> Signed-off-by: Xiang Dai <764524258@qq.com> * Removed concurrency limitation and added 5m timeout Signed-off-by: Joe Elliott <number101010@gmail.com> * Upping concurrency to try to reduce lint time Signed-off-by: Joe Elliott <number101010@gmail.com> Co-authored-by: Xiang Dai <764524258@qq.com> Co-authored-by: Joe Elliott <number101010@gmail.com>	6 years ago
Joe Elliott	9faa8bcf1d	Go 1.13 (#1089 ) Updated everything to 1.13 Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Joe Elliott	fc6c5c063f	dep => go mod (#1062 ) * go mod files added. dep removed Signed-off-by: Joe Elliott <number101010@gmail.com> * Magically got prometheus version to stick Signed-off-by: Joe Elliott <number101010@gmail.com> * Cortex updated and prometheus updated Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated prometheus dependency Signed-off-by: Joe Elliott <number101010@gmail.com> * Added additional deps Signed-off-by: Joe Elliott <number101010@gmail.com> * Added two replaces from Gopkg.toml. Tests passing Signed-off-by: Joe Elliott <number101010@gmail.com> * Added deps Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated build image to 1.12 Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated golangci-lint to use vendored dependencies Signed-off-by: Joe Elliott <number101010@gmail.com> * Added check-mod makefile step and referenced in drone and circle ci configs Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated loki-build-image to 1.12 Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed linting error. Force go111module on for linting Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod vendor Signed-off-by: Joe Elliott <number101010@gmail.com> * Forced the inclusion of modtimevfs Signed-off-by: Joe Elliott <number101010@gmail.com> * Pin client-go due to issue with v12 go.mod Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy Signed-off-by: Joe Elliott <number101010@gmail.com> * Added check-mod to drone Signed-off-by: Joe Elliott <number101010@gmail.com> * Re-readded correct golang client Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy Signed-off-by: Joe Elliott <number101010@gmail.com> * Pinned golang/x/net to avoid proxy errors Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed check-mod from all. Not necessary for tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated go.sum to match new pin Signed-off-by: Joe Elliott <number101010@gmail.com> * Upgraded proto to match build image Signed-off-by: Joe Elliott <number101010@gmail.com> * Force check-mod to wait til after test and lint are successful Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod vendor to go builds to force usage of vendored dependencies Signed-off-by: Joe Elliott <number101010@gmail.com> * Turn on gomodules on all builds Signed-off-by: Joe Elliott <number101010@gmail.com> * Revert "Added mod vendor to go builds to force usage of vendored dependencies" This reverts commit 65865a24c9a23133e0fa52942f2828ead7c22147. * Moved builds out of the gopath to enforce vendor usage Signed-off-by: Joe Elliott <number101010@gmail.com> * Revert "Turn on gomodules on all builds" This reverts commit b5847f0158e928e935e0b3c1b1d4eaba840ca3dc. * Explicitly choose build image for docker driver Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated golang.org/x/sys to match prometheus's version to fix promtail windows compilation Signed-off-by: Joe Elliott <number101010@gmail.com> * Added fluentbit dependency Signed-off-by: Joe Elliott <number101010@gmail.com> * Added dependency management clause Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated go version in contributing Signed-off-by: Joe Elliott <number101010@gmail.com> * Added phony makefile targets Signed-off-by: Joe Elliott <number101010@gmail.com> * Circle CI is increasingly failing linting Signed-off-by: Joe Elliott <number101010@gmail.com> * Force vendored deps on builds Signed-off-by: Joe Elliott <number101010@gmail.com> * Added logcli to gitignore Signed-off-by: Joe Elliott <number101010@gmail.com> * Reduced concurrency to help CircleCI Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved drone builds out of GOPATH to force use of go module Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod vendor to prevent go clean from redownloading all packages Signed-off-by: Joe Elliott <number101010@gmail.com> * Added to test as well Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod-vendor to final go commands Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved fluent-bit out of GO_PATH to force go modules Signed-off-by: Joe Elliott <number101010@gmail.com> * Pass mod vendor to the last holdout: go generate Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated ugorji/go to 1.17 to avoid version regression Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy and go mod vendor for ugorji/go Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
sh0rez	794d7da845	feat: -version flag (#1036 ) While we were already setting the version on the built binaries, this now also exposes them as a user-facing flag (`-version`). The already used Prometheus package `github.com/prometheus/common/version` is used for generated the printed string.	6 years ago
sh0rez	45bdebf8b2	chore(ci/cd): build containers using drone.io (#891 ) Because drone.io provides us with native ARM runners, we evaluate the platform to figure out if it is a suitable replacement for CircleCI and qemu. Configuration is done using .drone/drone.jsonnet, the .drone/drone.yml is generated from that. Images are built on amd64, arm64 and arm, natively and without emulation. This allows to get rid of cross-compilation, which in turn allows more advanced builds (CGO, etc.) Hopefully this will make the CI faster and more reliable in the future. Let's see!	6 years ago
Ed	173e8ea2a4	fix(canary): Fix Flaky Tests (#889 ) * make the missing entry lookup synchronous during tests to hopefully avoid races around the entry order and test verifications * playing around with timings some more to get all tests to pass even on a really slow machine	6 years ago
Ed	4eddcc1c8c	chore(packaging): avoid running containers from Dockerfile	6 years ago
sh0rez	e4b99f2729	chore(packaging): wrong executable format (#886 ) Due to an invalid usage of bash, the binary was always built for amd64. This fixes that and builds the correct one.	6 years ago
Robert Fratto	297f0e9d03	Makefile: build promtail with CGO_ENABLED if GOHOSTOS=GOOS=linux (#809 ) This commit modifies the Makefile to build promtail with CGO_ENABLED if both GOHOSTOS and GOOS are set to linux. To facilitate the Docker container utilizing the journal support, the container for promtail had to be modified to be based on top of debian. The sd-journal library depends on libsystemd.so being present at runtime. As alpine currently does not have a package for systemd, the easiest approach is to change the base image to debian:stretch-slim. A future solution for providing an alpine-based build will be considered in the future and discussion for the approach will be tracked in #838. This commit also updates the Dockerfile to default to using the latest loki-build-image. If a specific version is required, it should be provided as a build arg to docker build.	6 years ago
Ed	fc12a2355d	style(loki-canary): spelling (#832 )	6 years ago
sh0rez	b887602e36	chore(packaging/docker): Support multiple architectures (#762 ) This finally adds support for multiple architectures! 🎉 Images are now built using BuildKit (actually img), Dockerfiles were parameterized to allow this. Containers are still built on alpine, using QEMU and binfmt as a kernel compatibility layer to allow RUN steps in these. CircleCI has been fully revamped, to speed up builds and de-duplicate the configuration. Makefile has been extended with additional logic to shadow the multi-arch to non-CI users. Happy hacking on Raspberry Pi and friends :)	6 years ago
Ed	573673a0ab	Simplify our makefile as much as possible (#753 ) refactor(makefile): simplify makefile by removing the autogenerated targets and make building binaries and dockerfiles separate operations.	6 years ago
Edward Welch	e6f431ad46	finishing up loki-canary move into loki repo	6 years ago
Edward Welch	cd5368a44d	preparing for move into loki repo	6 years ago
Edward Welch	eedd3d9fc2	prune interval is configurable canary will suspend all operations on SIGINT but not exit, allowing you to shutdown the canary without it being restarted by docker/kubernetes SIGTERM will shutdown everything and end the process	6 years ago
Edward Welch	bcccbbfaaf	changing the default prune interval to 60 seconds, at the previous 1 second it would cause some way too aggressive querying when scaled out over all our clusters and loki had a hiccup	6 years ago
Edward Welch	02aed71a28	refactoring things so that the comparator can use the reader to make a direct query to loki to look for logs not received over the websocket before reporting them as missing	6 years ago
Edward Welch	5aac00f4d9	changing response_latency buckets some more, making exponential and configurable	6 years ago
Edward Welch	9dde2c4e5f	fixing how auth is applied	6 years ago
Edward Welch	34bb9db758	using an environment variable passed into an arg instead of reading the pod name from a file and the downardApi added a makefile	6 years ago
Edward Welch	45a7556da2	improving metrics improving tests config via flags added ksonnet config	6 years ago
Edward Welch	8340232b04	initial checkin mostly working, need to extract out the config needs a proper ksonnet or helm deployment or both needs a makefile	6 years ago

46 Commits (e8d14157256918e623c1ebe232b8db7fb67b9284)