1134 Commits (2564a810c5872205c7385cd6d0a787b68477188f)

Author	SHA1	Message	Date
Sandeep Sukhani	2564a810c5	Logs deletion fixes (#4625) ... * check all tables when we have delete requests to process since chunks could span multiple tables * upload modified index when the it is modified without dropping the chunk * fix broken test * lint * Fix modified check Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * suggested changes from PR review * add changelog Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	0e0d0b7897	dont try to use the scheduler ring when a downstream url is configured (#4635)	4 years ago
Christian Haudum	929c8e4923	Allow HTTP POST requests on ring pages (#4630) ... The ring pages offer the ability to "forget" instances of the ring and this is implemented by a HTTP `form` element, which does a `POST` request on itself. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Danny Kopping	5b30589acf	Documenting recording rules per-tenant WAL (#4566) ... * WIP Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Document ruler config options Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Self-review updates Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clarifying WAL benefit in response to review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding dashboard link Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Ensuring ruler and ingester default WAL paths do not conflict Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	de4993f921	explicitly define allowed HTTP methods on HTTP endpoints (#4627)	4 years ago
Trevor Whitney	62ee399e48	default ingester final sleep to 0 unless otherwise specified (#4608) ... * default ingester final sleep to 0 unless otherwise specified * add changelog and upgrading * update config docs	4 years ago
Ed Welch	fe8bc9144f	Loki: query scheduler should send shutdown to frontends when ReplicationSet changes (#4614) ... * make sure we send a shutdown message to frontends from the scheduler when the ReplicationSet changes so that inflight queries are canceled and retried in the frontends. * use Write so we find an instance incase one is unhealthy	4 years ago
Owen Diehl	308af871cf	uses ring.Write instead of ring.WriteNoExtend for compactor ring checks (#4615)	4 years ago
Kaviraj	1e0386e9e6	Fix `ip` matcher lexer to differentiate filter from identifier (#4598) ... * Fix `ip` matcher lexer to differentiate filter from identifier Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Revert small fmt fix on expr.y Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks from Cyril Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add ip line filter parser test in mix with ip as label Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Garrett	1c7df620c4	add date time sprig template functions in logql label/line formatter (#4603) ... * add date time sprig template functions in logql label/line formatter With date time sprig template functions, we will be able to filter test jobs which create time is 1 day before as the following: ``` {job="test"} | label_format nowEpoch=`{{(unixEpoch now)}}`,createDateEpoch=`{{unixEpoch (toDate "2006-01-02" .createDate)}}` | label_format dateTimeDiff="{{sub .nowEpoch .createDateEpoch}}" | dateTimeDiff > 86400 ``` * add documents for date time sprig template functions * resolve comments * add golang datetime layout doc reference * update CHANGELOG.md accordingly to include this PR	4 years ago
Owen Diehl	84220c4b02	improves scheduler & compactor ringwatcher checks (#4607)	4 years ago
Ed Welch	0c9beadaaa	Loki: Add a ring to the compactor used to control concurrency when not running standalone (#4574) ... * add a ring to the compactor * wiring up the start/stop functionality * add some monitoring for multiple compactors running at the same time. * Add compactor to `read` target * fix tests/validation requirements * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * used a shared ring_config between scheduler and compactor to avoid some duplication * fix lint * better handling of the tokens file with a way to easily tell all the rings to persist tokens * fix query scheduler replication factor * removing "scheduler" from log line which may be called by components other than a scheduler in the future. * update changelog Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Hiroki Sakamoto	1ff95fc025	Respect gRPC context error when handling errors (#4593) ... * Respect rpc status when handling http errors * Format by goimport * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Add some tests * context, rpc cancelled and another error case * context, rpc deadline exceeded and another error case Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Hiroki Sakamoto	0e21f02d76	Respect shard number in series api (#4577) ... * Respect shard number in series api * Refactor codes * remove duplicate logic in instance * format with goimport * Improve complexity of getting all fingerprints	4 years ago
Owen Diehl	0c97afadbc	cleans up redundant setting of stream.unorderedWrites=true when replaying wal (#4595)	4 years ago
Owen Diehl	633b4cab53	ignore validity window during wal replay (#4596)	4 years ago
Owen Diehl	45113cb375	Enable unordered_writes by default (#4594) ... * enable unordered_writes by default * removes out_of_orders from transfer test * update changelog	4 years ago
Owen Diehl	cd80bc5e7e	introduces "entry too far behind" instrumentation for unordered writes (#4592)	4 years ago
Danny Kopping	de7ab3fde2	Configuring query-frontend interface names with loopback device (#4586) ... * Configuring query-frontend interface names with loopback device Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding test (thanks @DylanGuedes!) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	1ef1ff0224	replaces fallthrough statement in InitFrontend (#4589)	4 years ago
Dylan Guedes	3ab7bcd17a	Change default RejectOldSamplesMaxAge from 14d to 7d. (#4569)	4 years ago
lingpeng0314	76e2a14029	add group_{left,right} to LogQL (#4510) ... * Add group_left/group_right support * Update docs/sources/logql/_index.md * Minor change for a test case * removes CardManyToMany * removes now-unused IsSetOperator helper Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Dylan Guedes	3c16658ee0	Loki: Append loopback to ingester net interface default list (#4570) ... * Move LoopbackInterfaceName to its own package. - Create new package `util/net` and puts `LoopbackInterfaceName` there - Modify distributor_test to use `util/net` package * By default, append loopback to ingester interfaces. - Append the loopback net interface to the list of net interfaces used by the ingester ring. The append only occurs if the ingester net interfaces are default - Add CHANGELOG entry * Test ring net interface behavior. - Add tests to check the current net interface name resolution. The tests checks if the ingester ring net interface names are correctly copied to other rings	4 years ago
Trevor Whitney	ed025e0cd6	Change a few default config values and improve application of common storage config (#4543) ... * further config simplifications and better defaults * default ruler api to on * register defaults for storage configs in the common config (ie. signature version for S3) * enable ingester wal by default * disable chunk retries by default * allow common path prefix to include or omit trailing slash * allow partial storage config to be defined for overriding things like bucket names per component * fix tests * add changelog and upgrade docs * explain impact of wal change in upgrading.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rename add WithPrefix to RegisterFlags * fix (frontend OR worker) and scheduler boot order when both are running (#4545) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	b4bd96073c	validate default limits config with other configs at startup (#4573) ... * validate default limits config with other configs at startup This should also parse and set the matchers in default stream retention config * directly set matchers in stream retention instead of building the whole struct again	4 years ago
Dylan Guedes	331fa6d7b0	Docs: Add virtual targets docs (#4565) ... * Add CHANGELOG entry and updated modes diagram. * Update documentation with the new hybrid mode. - Update targets with new 'read' and 'write' options - Add new docker-compose yaml that exemplifies how to use new 'read' and 'write' targets * Add missing targets to configuration docs. - Missing ones: index-gateway, ingester-querier, query-scheduler	4 years ago
Cyril Tovena	165ff31e2d	Fixes the Series function to hanlde properly sharding. (#4563) ... Before they would try to match the injected label but this label is only used by the store and not returned in the chunks causing all chunks to be filtered out. Fixes #4547 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ea8e0987b3	Fixes an edge case in the batch chunk iterator. (#4550) ... * no-cache Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * found a bug to fix. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a edge case in the batch chunk iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	47336cdbe6	Fixes a panic in the labels API when no parameters are supplied. (#4554) ... * Fixes the a panic in the labels API when no parameters are supplied. We were supporting this before. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove bad test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	a27aec37cf	only exports tenant limits which differ from defaults and export defa… (#4542) ... * only exports tenant limits which differ from defaults and export defaults under `loki_overrides_defaults` metric. * docs & changelog	4 years ago
Ed Welch	6853c387ed	Loki: Apply the ingester ring config to all other rings (distributor, ruler, query-scheduler) (#4546) ... * apply the ingester ring config to all other rings in ApplyDynamicConfig * update the test, only verify ingester config wasn't manipulated by the step to apply the common memberlist config when join_members is defined.	4 years ago
Trevor Whitney	2864a4f728	enable virtual read write targets (#4498) ... Co-Authored-By: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com>	4 years ago
李国忠	6fcd02bad4	[querier] s3: add getObject retry (#4453) ... * [querier] s3: add getObject retry * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4452 * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Dylan Guedes	77ea11724b	Loki: Enable FIFO cache by default (#4519) ... * Set default FIFO cache with reasonable defaults. Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * Remove FIFO cache feature flag. - We noticed that we always want that behavior, so the logic is better implemented in the `ApplyDinamicConfig` method - Update documentation * Move FIFO cache default behavior. - Move FIFO cache default behavior implementation to ApplyDinamicConfig * Add tests for the new FIFO cache default behavior. - The tests tests the behavior of the four existing caches (query results, query index, write dedupe, and chunk results) - Add changelog entry - Fix lint issues regarding import order - Revert moving of EnableFifoCache check * Apply suggestions from code review - Change `EnableFifoCache` description to mention that it is only applied for chunk and query results cache Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
JordanRushing	93507a00ed	Add quick nil check in TenantLimits for runtime_config (#4531)	4 years ago
Owen Diehl	91eba4fb6b	correctly sets subservicesWatcher on scheduler (#4529)	4 years ago
JordanRushing	ca67292b54	Introduce `overrides-exporter` module to Loki (#4520) ... * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Use reflection; add more limits to export * Fix import to satisfy linter * refactors TenantLimits for clarity & efficiency. Also fixes tenantLimits nil check * Fix breaking test with nil check in runtime_config Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	aec00c2507	safely checks read ring for potentially nil scheduler (#4525)	4 years ago
Trevor Whitney	6f28a5dc70	Apply better defaults when boltdb shipper is being used (#4508) ... * changes from PR review * rename object_store -> storage * add local/filesytem support to common object storage config * add better defaults for boltdb shipper * guard against missing schema configs in dynamic config application * fix alphabetical order of storage configs * return an error when more than one storage config is provided	4 years ago
Ed Welch	2b5f3000b6	Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS (#4424) ... * Fork the frontend and scheduler so we can add support for discovery via the ring. * Add a ring to the query scheduler which is then used by the frontend worker and querier workers to find the scheduler address as an alternative to using DNS * update some of the forked code to use dskit grpcclient and grpcutil since #4312 was merged after the original fork was created. * remove query scheduler dependency from querier Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * migrate logging package and make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add SafeReadRing for instances when scheduler is not enabled on same instance * Doc changes from code review A few doc fixes Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * go mod tidy on go-kit dep * update changelog * remove ReadRing in favor of SafeReadRing Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Trevor Whitney	2427fab32d	Configuration: add a common config section for object storage (#4473) ... * add common config for object storage Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * wip Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add test to compactor config override Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * update changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * changes from PR review * rename object_store -> storage * add local/filesytem support to common object storage config	4 years ago
Ed Welch	d8e6debaa5	Loki: Fix bug where items are returned to a sync.Pool incorrectly (#4518) ... * Make sure calls to the SampleIteratorWithClose Close() function are idempotent. * the linter hates my fun error messages	4 years ago
Garrett	a046d79261	make LogQL syntax scope from private to public (#4446) ... * make LogQL syntax scope from private to public The follow-up for https://github.com/grafana/loki/pull/3991, it makes using LogQL syntax outside possible, it also makes LogRange implements Expr interface. * make constructor also public * Revert "make constructor also public" This reverts commit ab9d0a942c.	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484) ... * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	7832783b1c	always expoe loki_build_info (#4482)	4 years ago
Owen Diehl	3fb8f3d3c6	restores for state at seconds(now-forDuration) (#4479) ... * restores for state at seconds(now-forDuration) * bump ci	4 years ago
Mauro Stettler	4b59556342	Update cortex to newer version (#4478) ... * update cortex and thanos to newer version * pin k8s.io/apimachinery * fixes for newer libs * add missing method to mock wal data collector * fixing test by using different gokit level pkg * make check-mod	4 years ago
Dylan Guedes	6498c8cb13	Loki: Change default limits to common values (#4415) ... * Change some default limits to reasonable values. - These new limits protect users from overwhelming their cluster with ingestion load - The new limits are: * ingestion_rate_strategy: global <- currently defaults to local * max_global_streams_per_user: 5000 <- current default is no limit * max_query_length: 721h <- current default is no limit * max_query_parallelism: 32 <- current default is 14 * max_streams_per_user: 0 <- current default is 10000 * reject_old_samples: true <-- current default is false * reject_old_samples_max_age: 168h <-- current default is 336h * Remove parameters from examples - They are not necessary anymore because of the new default values * Add new entries to CHANGELOG and upgrading guide * Apply suggestions from code review Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Change changelog mark entry from '-' to '*' Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Dylan Guedes	d3d63e1778	Loki: Override distributor's default ring KV store (#4440) ... * Override default KV store for the compactor. - Currently, the default KVstore is consul. This changes it to inmemory to make it easier for new users to run the project without dependencies. - Refactor default flag value tests. Now, it will check for a populated map instead of checking for flags during iteration. * Move distributor flag override to RegisterFlags. * Add missing CHANGELOG and upgrading guide entries.	4 years ago
Trevor Whitney	7118dc3272	Change default values for two GRPC setting we have to set so the queriers can connect to a frontend or scheduler (#4435) ... * change gprc server defaults Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add docs and changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago