apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Travis Patterson	3f0ecd3c7b	Implement all-or-nothing stream ingestion (#7094 ) Per-stream sharding will introduce automatic stream sharding on rate-limited streams. The ingester's current line-by-line behavior may result in duplicates when a partially-accepted stream is resharded and sent to other ingesters. This PR makes the ingester either rate limit the entire stream before attempting to write lines to a chunk. The PR 1. Separates validation, storage, and wal replay into their own functions 2. Implements a flag to rate limit the whole stream or not 3. Checks rate limits for the whole stream rather than line-by-line 4. Returns an error with information on the whole stream when rate limited	3 years ago
Travis Patterson	bc16450349	Revert "Hide duplicates caused by stream sharding (#7005 )" (#7031 ) @owen-d had a suggestion that was dramatically dimpler. Revert this whole thing and the simplified PR will be forthcoming	3 years ago
Travis Patterson	a98b7ebaf4	Hide duplicates caused by stream sharding (#7005 ) As streams are sharded and resharded across different ingesters, the likelihood of duplicate logs being ingested increases dramatically. This removes duplicates at query time by dynamically removing the `__shard_stream__` label for both log and sample queries so the `stream hash` used by the `MergeEntryIterator` will match when doing comparisons for duplicates. The `stream hash` is created in either the `store` or `ingester instance` at query time so the labels are removed there. Additionally, the `MergeEntryIterator` has been modified to remove duplicates in a stream rather than just across streams. For completeness, `__shard_stream__` has also been removed from Label and Series Queries.	3 years ago
Owen Diehl	57cec8e496	[TSDB] Ingester Index Sampling (#6852 ) * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * query ingesters for index stats Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * compatibility during rollouts for idx-gw+ingester-querier post proto refactor Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto compatibility with old indexgateway pkg path Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * test compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Haudum	c12a1f4f43	feature: Add ingester handler for shutdown and forget tokens (#6179 ) * Update grafana/dskit to 07166f9 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add /ingester/shutdown handler This handler replaces the deprecated /ingester/flush_shutdown handler and can be used to gracefully shut down a Loki instance and delete the file that persists the tokens of the ingester ring. In production environments you usually want to persist ring tokens so that during a restart of an ingester instance, or during rollout, the tokens from that instance are not re-distributed to other instances, but instead kept so that the same streams end up on the same instance once it is up and running again. For that, the tokens are written to a file that can be specified via the `-ingester.tokens-file-path` argument. In certain cases, however, you want to forget the tokens and re-distribute them when shutting down an ingester instance. This was already possible by calling `/ingester/flush_shutdown`, deleting the tokens file and terminating the process. The new handler `/ingester/shutdown` combines these manual steps into a single handler. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Mathew Heard	de418bf6ac	Reduce to a single code path through sendBatches regardless of limit (#6216 ) * Reduce to a single code path through sendBatches regardless of limit specified * simplify sendBatches by using -1 to specify no limit * test fix	4 years ago
Owen Diehl	4903df6ae5	Tsdb/inverted index wiring (#6252 ) * multi inverted index * delete in inverted index signature * multi inverted index * inverted index testing + skip nil indices * wires up period config aware multi-inverted-idx through ingesters * better inverted index validation * apply shipper defaults to last relevant period config, not current one. * more verbose error	4 years ago
Sandeep Sukhani	1923668a5d	run store queries in ingester when using tsdb as index store (#6209 )	4 years ago
Kaviraj Kanagaraj	24f8b19e91	Remove unused grpc health check endpoints (#6113 ) We have generic `grpc_healthcheck` endpoint that get registered for all the components during initialization stage. Looks like these health checks that are part of actual component's struct itself came from legacy. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Christian Haudum	18058d2b55	Serve correct status of ingester/distributor in gRPC healthcheck endpoint (#6094 ) * Serve correct status of ingester in gRPC healthcheck endpoint The gRCP healthcheck endpoint is used by clients that determin whether a connection to the service is healthy or not. In case the ingester is not in "running" state and is not "active" in the ring, this healthcheck should return "not serving". Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Serve correct status of distributor in gRPC healthcheck endpoint	4 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049 ) * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Christian Haudum	b6ce3826b4	Use new build image 0.20.3 (#5904 )	4 years ago
Travis Patterson	7299e3bdcd	Fix query filtering (#5951 ) - milli -> nano conversion in the querier - add deletes to store requests from the ingester	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	b136d0dab3	Increase flush_op_timeout to 10m, 10s is often much too short for a big WAL replay which can generate a lot of chunks very quickly. (#5768 ) Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724 )	4 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539 ) * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Michel Hollands	d8ed87d04c	Update dskit version (#5392 ) * Update dskit version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add debug printing to see interfaces in CI. This will be reverted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to remember to remove Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use netutil.PrivateNetworkInterfacesWithFallback for the other rings Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Move import to place indicated by linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove debug printing Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Determine local interfaces in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rearrange import to satify linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update build image in CircleCI Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change changelegog message to show customer visible changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add doc tag and use private network interfaces in common config Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment about filtered out non private IP addresses Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the documentation of the interface names settings Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	4 years ago
Kaviraj Kanagaraj	2db309271c	Makes `tailer.droppedStreams` slice bounded. (#5334 ) * Makes `tailer.droppedStreams` slice bounded. For slow receiver, this dropped streams may cause ingester memory to keep increasing. Sometimes OOMing the ingester. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make droppedStreams slice size configurable Update tests Adding logging when slice is dropped Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for dropped stream Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281 ) * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257 ) * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Michel Hollands	87edab0867	Ingester: remove the labelFilter (#5262 ) * Remove the labelFilter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Trevor Whitney	3091ccd3eb	improve default config values (#5077 ) * improve default config values * change defaults for upstream query range package * update changelog * remove max_tranfer_retries fix as it is covered in another PR, 4792 * increase query ingesters within by 1h This provides an additional buffer on top of the max_chunk_age. Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * add comment reminder to remove query range config hack Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rollback change to align_queries_with_step and parallelise_shardable_queries * add upgrading docs * re-enable parallelise_shardable_queries by default * add parallelise_shardable_queries back to upgrading doc Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	e823a5037f	better defaults for flush queue parallelism (#5091 ) * better defaults for flush queue length * changelog	4 years ago
Arve Knudsen	d87dbf9f8a	Chore: Upgrade to latest Cortex (#5085 ) * Chore: Upgrade to latest Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Chore: Upgrade to Prometheus@692a54649ed7 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build after upgrading dskit Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
JordanRushing	291360ddcb	Simpler new chunk key v12 (#5054 ) * starts hacking new chunk paths * Create new chunk key for S3; update parsing and add basic test and benchmark Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Fix `TestGrpcStore` to work with new chunk key structure Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Add a function to SchemaConfig that returns the correct PeriodConfig for a given time. Signed-off-by: Callum Styan <callumstyan@gmail.com> Add schema config to object client so it we can use the right external key function based on the schema version. Signed-off-by: Callum Styan <callumstyan@gmail.com> Fix failing compactor tests by passing SchemaConfig to chunkClient Remove empty conditional from SchemaForTime Define schema v12; wire-in ChunkPathShardFactor and ChunkPathPeriod as configurable values Add PeriodConfig test steps for new values ChunkPathShardFactor and ChunkPathPeriod in v12 Update test for chunk.NewExternalKey(); remove completed TODO Set defaults for new ChunkPathPeriod and ChunkPathShardFactor SchemaConfig values; change FSObjectClient to use IdentityEncoder instead of Base64Encoder Use IdentityEncoder everywhere we use FSObjectClient for Chunks Add ExternalKey() function to SchemaConfig; update ObjectClient for Chunks * Finish plumbing through the chunk.ExternalKey -> schemaConfig.ExternalKey change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up lint failures Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Clean up chunk.go and fix tests Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Break SchemaConfig.ExternalKey() into smaller functions Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Quickly fix variable name in SchemaConfig.newerExternalKey() Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Clean up ExternalKey conditional logic; add better comments; add tests Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Fix a bug where we are prepending userID to legacy Chunk keys but never parsing it; refactor key tests and benchmarks Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Add small SchemaConfig test Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Update docs and CHANGELOG.md Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Correctly return an error when failing to parse a chunk external key Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Revert IdentityEncoder to Base64Encoder after testing Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Fix assignment in test for linter Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Remove leftover comments from development Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Change v12 version comment format; remove redundant login in `ParseExternalKey()` Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Change remaining v12 comment style Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Pass chunk.SchemaConfig to new parallel chunk client * Simplify chunk external key prefixes for schema v12 * Fix broken benchmark; add benchmarks for old parsing conditional Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * Remove unneeded lines from upgrading doc * Add benchmarks for root chunk external key parsing functions Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> * memoizes schema number calculations * Resolve linter issue with PeriodConfig receiver name Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Karsten Jeschkies	7bbed2be49	Define Ingester interface to enable wrapping. (#4988 )	4 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944 )	4 years ago
Michel Hollands	e69ceefd84	Add LabelFilterer and Store wrapper (#4818 ) * Add LabelFilterer and Store wrapper Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Address review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830 ) * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Dylan Guedes	130878f3f6	Enrich WAL folder creation error message (#4705 ) * Enrich WAL folder creation error message. * Apply suggestions from code review Use %q and try absolute path. Co-authored-by: Mateusz Gozdek <mateusz@kinvolk.io> * Include file/path package. Co-authored-by: Mateusz Gozdek <mateusz@kinvolk.io>	4 years ago
Cyril Tovena	d81f757ecb	Refactor and simplify stats package. (#4712 ) * Refactor and simplify stats package. This PR refactors the stats package and make it smaller and simpler to work with. - It doesn't expose the context which can causes race when mutating it. - Uses concurrency primitive to ensure correctness - Replace trailers with in-batch statistics to propagate ingesters stats. Fixes #3366 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix tests and lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Trevor Whitney	bfc77118a1	Update local and docker config for simple scalable deployment (#4660 ) * first draft of local config * remove a few more defaults * interface names fix means we do not need lifecycler ip * update rings to use collectors/ prefix fix replication factor chunks-retain-period default to 0s fix a bug introduced with a Go upgrade? on printing the schema date. * simplify the config for the ruler, synthesize the local directory * simplify loki-docker-config.yaml * change common filesystem config to have chunks and rules directory * now that we are calling register flags on the common ring config we set more values and some of the tests needed to be tweaked. updated a few paths as well. * fix tests Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641 )	4 years ago
Trevor Whitney	ed025e0cd6	Change a few default config values and improve application of common storage config (#4543 ) * further config simplifications and better defaults * default ruler api to on * register defaults for storage configs in the common config (ie. signature version for S3) * enable ingester wal by default * disable chunk retries by default * allow common path prefix to include or omit trailing slash * allow partial storage config to be defined for overriding things like bucket names per component * fix tests * add changelog and upgrade docs * explain impact of wal change in upgrading.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rename add WithPrefix to RegisterFlags * fix (frontend OR worker) and scheduler boot order when both are running (#4545) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	47336cdbe6	Fixes a panic in the labels API when no parameters are supplied. (#4554 ) * Fixes the a panic in the labels API when no parameters are supplied. We were supporting this before. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove bad test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Trevor Whitney	de0c469161	default chunk target size to ~1MB~ 1.5MB (#4388 ) * default chunk target size to 1MB Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * bump up to 1.5MB Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196 ) * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Owen Diehl	9cdb1295b5	adds loki_ingester_wal_replay_active metric and records this more acc… (#4193 ) * adds loki_ingester_wal_replay_active metric and records this more accurately * adds WAL to msg	4 years ago
Owen Diehl	ff2ced09ea	Cleanup/unordered writes ingester config (#4192 ) * handle replaying unordered WAL into ordered configs. * ingester limiter metric, testware, correctly replayed unordered WAL into ordered config	4 years ago
Owen Diehl	92a4e57448	ingester.index-shards config (#4111 ) * consistent shard mapping test & better error msg * ingester.index-shards config * loki fails to start if the ingester & schema factors arent compatible. * lint * docs * update changelog	4 years ago
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842 ) The chunk package has been forked from this commit for future reference and licensing. `09fe17c0e6`	5 years ago
Owen Diehl	56256bfd5b	Unordered chunks (#4001 ) * merge feature/unordered-replay * interoperable head chunks * memchunk block interop * retain ordered memchunk optimizations when possible * tests+bench for unordered chunk reads * reorder on chunk close * [wip] ingester stream unorderd * unordered writes default in testware config, fixes OOO bug & removes unused lastChunkTimestamp var * validity window is 1/2 of max age & fixes old transfer test * more consistent headblock checking/creation * more cohesive encoding tests * unordered stream test with validity bounds * compat - unordered * reinstates memchunk defaults when rebounding & updates storage test compatibility * lint * reorder across blocks doesnt overflow * respect chunk configs during rebounding when possible * only sync checks on ordered writes	5 years ago
Vincent Miszczak	d6dc8b991c	Add ingester.autoforget-unhealthy-timeout opt-in feature (#3919 ) * Add ingester.autoforget-unhealthy-timeout opt-in feature * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Michel Hollands	78b073e022	Add a chunk filterer field to the config (#3979 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973 ) This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	5 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago

1 2 3

118 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)