41 Commits (29be929a8cde0a20fc2e639285db90cf9be88338)

Author	SHA1	Message	Date
Sandeep Sukhani	cd723c0479	Boltdb shipper query readiness (#2911) ... * add support for downloading index on startup in boltdb-shipper for query readiness * tests * pass context to load tables function to be able to cancel the operation * add missing call to tablemanager.stop when loading of tables fail	5 years ago
Sandeep Sukhani	fd451d97f4	fix flaky load tables test for boltdb-shipper uploads table-manager (#2891)	5 years ago
Owen Diehl	9249ebc022	Fix/validate compactor config (#2824) ... * default compactor config throws error when compactor is a target or subtarget * linting * moves compactor config validation to NewCompactor fn	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877) ... * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	feb66d62cf	fix flaky tests in boltdb-shipper (#2831)	5 years ago
Sandeep Sukhani	2793499309	Boltdb shipper metrics changes (#2790) ... * rename boltdb shipper metrics to have table in their name instead of files * Increment tables_sync_operation_total metric during periodic syncs as well * tweak error handling Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	7b7b1844ca	add a metric in compactor to record timestamp of last successful run (#2788)	5 years ago
Sandeep Sukhani	f60554ce91	Boltdb shipper query performance improvements (#2770) ... * boltdb-shipper query with cursor to do multiple queries in a single transaction * index deduper * use fifo cache in ingesters * snapshot dbs in ingester for reads * retain fewer files in ingesters for queries * set timeout in cache config instead of index read cache config which saves us from unmarshalling the object for checking the expiry * add comment about fifo cache invalidation * tweaking a few comments Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755) ... * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Davor Kapsa	fffcfa6e81	fix some misspells (#2725)	5 years ago
Sandeep Sukhani	79c027550c	run multiple queries per table at once with boltdb-shipper (#2656)	5 years ago
Sandeep Sukhani	5b361a0c89	improvements for boltdb-shipper compactor (#2640) ... 1. Run the compaction process on startup instead of waiting for the first tick. 2. Run the compactor service when using loki with boltdb-shipper in single binary mode without clustering.	5 years ago
Lars Lehtonen	334cc1e59a	pkg/storage/stores/shipper/uploads: fix test error (#2620) ... Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>	5 years ago
Sandeep Sukhani	e53d2032fe	fix a panic when trying to stop boltdb-shipper multiple times using sync.once (#2613)	5 years ago
Sandeep Sukhani	6dd2e9ef34	Read, Write and operational dashboard improvements (#2584) ... 1. Add bigtable, boltdb-shipper to read/write dashboards. 2. Add boltdb-shipper to operational dashboard. 3. Record latency for write operation in boltdb-shipper.	5 years ago
Sandeep Sukhani	7566aa4950	fix closing of compressed file from boltdb-shipper compactor (#2574)	5 years ago
Sandeep Sukhani	0a0ad68af2	fix dir setup based on which mode it is running (#2559)	5 years ago
Sandeep Sukhani	1bba2a89f8	add some metrics for monitoring compactor (#2548)	5 years ago
Sandeep Sukhani	4f6a4c2a93	cleanup boltdb files in queriers during startup/shutdown (#2558)	5 years ago
Sandeep Sukhani	2bb2eebace	register boltdb shipper compactor cli flags (#2546)	5 years ago
Sandeep Sukhani	dab393a62d	fix a flaky test in boltdb shipper (#2550)	5 years ago
Sandeep Sukhani	b489130828	compactor for compacting boltdb files uploaded by shipper (#2526)	5 years ago
Sandeep Sukhani	97dfb297a0	compress boltdb files to gzip while uploading from shipper (#2507)	5 years ago
Sandeep Sukhani	9b0740c0ed	upload boltdb files from shipper only when they are not expected to be modified or during shutdown (#2487) ... * upload boltdb files from shipper only when they are not expected to be modified or during shutdown * updated comment * add one more test condition to make sure files are not re-uploaded Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	048e8bdb9c	minor fixes and improvements for boltdb shipper (#2530) ... 1. Continue uploading of other tables during failures. 2. Remove .temp files while loading dbs during startup. 3. Remove empty table folders while loading tables during startup.	5 years ago
Sandeep Sukhani	cf64b70db3	add some spans for boltdb shipper index queries (#2495)	5 years ago
Sandeep Sukhani	c5bc41628a	download boltdb files parallelly during reads (#2483) ... * download boltdb files parallelly during reads * changes suggested from PR review	5 years ago
Owen Diehl	4a45ca5e3a	Revendor cortex (#2427) ... * vendoring bump * vendoring compat	5 years ago
Sandeep Sukhani	e59adccfa0	create smaller unique files from boltdb shipper and other code refactorings (#2261) ... * rename package local to shipper * reorganized files to separate out uploads and downloads * create smaller unique files from boltdb shipper and other improvements * cancellation of downloading of initial files when service is being stopped * migration of existing files to dedicated folders and cleanup of index from ingesters * some more improvements to uploads and downloads code * tests for uploads and downloads package in shipper Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * test cleanup fix and some other minor changes * refactor getting uploader from object key * better naming * changes suggested from PR review * Fixes goroutine loop. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Simplify getoradd table. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes loop in upload manager. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * some more tests, logs and changes suggested from PR review * move ready state check for downloaded table inside query function * add info log for successfully finishing uploading of a table * fix merge conflict * fix ingester name and mode config Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c9f89df930	Improve object key parsing for boltdb shipper. (#2323) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	68ac695fd8	Removes arm32 for now as we're migrating the CI. (#2307) ... * Removes arm32 for now as we're migrating the CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes flaky test, its being rewritten anyway. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	9ab3800522	boltdb shipper download failure handling and some refactorings (#2156)	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204) ... * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Ed Welch	9803eab947	Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper (#2073) ... * Allow configuring query_store_max_look_back_period when running a filesystem store, which allows for boltdb-shipper to horizontally scale filesystem store. Signed-off-by: Ed Welch <edward.welch@grafana.com> * Update docs/operations/storage/boltdb-shipper.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * add some more precision to the timestamp used to make the name unique just to make sure there are not collisions if for some reason the ID is empty (which can happen if the hostname doesn't resolve for some reason) Signed-off-by: Ed Welch <edward.welch@grafana.com> * Allow the ingester to query the store for labels if query store is enabled. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make queriers aware of the fact that ingesters may be querying the store so they don't duplicate query efforts Signed-off-by: Ed Welch <edward.welch@grafana.com> * improve the docs Signed-off-by: Ed Welch <edward.welch@grafana.com> * add tests to the calculateMaxLookBack Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	3a28a17946	some metrics for measuring performance and failures in boltdb shipper (#2034) ... * some metrics for measuring performance and failures in boltdb shipper * counting both failures and success in upload and download operation in boltdb shipper * changes suggested from PR review	5 years ago
Aditya C S	199e334126	Update loki to cortex master (#2030)	5 years ago
Sandeep Sukhani	989fb106a0	table client for boltdb shipper to enforce retention (#2001) ... * table client for boltdb shipper to enforce retention	5 years ago
Sandeep Sukhani	21d4ca41e2	make boltdb shipper singleton and some other minor refactoring (#1995) ... * make boltdb shipper singleton and some other minor refactoring	5 years ago
Ed Welch	b89a1f81dd	Update cortex vendoring to latest master (#1975) ... * Update cortex vendoring to latest master checksum `36496a074bc4530fb1d7c74136cab1566fbd9583` Signed-off-by: Ed Welch <edward.welch@grafana.com> * handle all the breaking changes from cortes Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * rebasing on master, re-runing go mod vendor seemed to bring in some more changes... Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	fad3b61be7	flush boltdb to object store (#1837) ... * flush boltdb to object store files are stored in folder per periodic table and are named after ingester flushed every 15 mins to make index available to other services files are also flushed before ingester stops to avoid any data loss new stores can be implemented easily ingester to also query store when using boltdb * persisting uploader name across restarts, detecting objectstore type from periodic config, other refactorings * updated cli flag for active directory * add tests for boltdb shipper and update vendor * syncing boltdb files to disk during update * sleep in tests to let mtime of boltdb file be changed * add documentation for boltdb shipper and remove unwanted config	5 years ago