103 Commits (master)

Author	SHA1	Message	Date
Carl Schwan	b6313f68d3	perf(s3): Expose pre-signed urls for S3 ... This is faster than going back to nextcloud to download the files. This is an opt-in setting that can be enabled by setting use_presigned_url in the object store config. Additionally add support for the proxy config which is needed in a docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431 Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	3 weeks ago
Robin Appelman	a7d130c598	Revert "fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579" ... This reverts commit 4b4b39e7ec. Signed-off-by: Robin Appelman <robin@icewind.nl>	1 month ago
Daan Selen	4b4b39e7ec	fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579 ... Signed-off-by: Daan Selen <dselen@systemec.nl>	2 months ago
Thomas Citharel	5c8bdad828	fix(objectstore): add named parameter for like query ... to avoid 'object::%' to be considered as a column (`la colonne \\u00ab object::% \\u00bb n'existe pas` on PG) Signed-off-by: Thomas Citharel <tcit@tcit.fr>	3 months ago
yemkareems	200f9dc39f	fix: small typo fix ... Signed-off-by: yemkareems <yemkareems@gmail.com>	3 months ago
provokateurin	98081dfd42	feat(ObjectStore): Add option to limit the total size ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 months ago
Robin Appelman	97efc95efc	fix: better object store write error propagation ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	83b8a390cd	fix: always do stream counting for object store upload ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	398b106f0c	fix: validate written size for s3 multipart uploads ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Ferdinand Thiessen	5981b7eb51	chore: apply new CSFixer rules ... Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de> # Conflicts: # apps/settings/lib/SetupChecks/PhpOpcacheSetup.php	6 months ago
Robin Appelman	2e1c316582	fix: remove double cache write in ObjectStoreStorage::touch ... the file_put_contents already puts all of those values into the cache Signed-off-by: Robin Appelman <robin@icewind.nl>	6 months ago
Robin Appelman	5b3f4cd8fe	perf: set the folder size in the same query as we create it ... Signed-off-by: Robin Appelman <robin@icewind.nl>	7 months ago
Louis Chemineau	1f8b823992	fix(S3): Use original folder size during copy ... This prevent having copied folders with a wrongly set size of 0KB. - Fix https://github.com/nextcloud/server/issues/51916 Signed-off-by: Louis Chemineau <louis@chmn.me>	7 months ago
Louis Chemineau	8fdf2a7eae	fix: Transfer ownership with S3 as primary ... When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error: `SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'` The `--move` option moves the entire home folder from one account to another. The error means that the move failed because the destination folder already exist in `oc_filecache`. - With S3 as primary storage, folders only exists as entries in `oc_filecache`. - With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder. - With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder. - `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112) - `oneTimeUserSetup(...)`: 687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262) - `mkdir(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135) - `moveFromStorage(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636) Delete pre-existing folder in `moveFromStorage(...)` Signed-off-by: Louis Chemineau <louis@chmn.me>	9 months ago
Robin Appelman	8aae332622	feat: store original storage id and path in object store metadata ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Robin Appelman	94114b99f7	feat: more generic way of passing metadata to object storage backends for new objects ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Robin Appelman	5896f8c37f	chore: cleanup leftover debug statement ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	6cf66f95ce	fix: get object size from stream where possible ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
provokateurin	0de4843b73	refactor(Storage): Align all Storage constructors ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago
provokateurin	f28e74b7a8	refactor(Storage): Make all parameter types strong types ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago
Robin Appelman	3e12e1e789	fix: rework move into object store to better preserve fileids ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	888d06dff9	fix: preserve fileid when moving from objectstore to non-objectstore ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	da21acfb3f	fix: ensure source folder is removed from cache when moving to objectstore ... otherwise this causes confusion down the line as it's contents will be moved to the new cache Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
provokateurin	7cdccd058f	fix(Storage): Fix IStorage return types ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago
Robin Appelman	47d281865e	perf: reuse cache info we already have when moving to object store ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
provokateurin	9836e9b164	chore(deps): Update nextcloud/coding-standard to v1.3.1 ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago
Robin Appelman	3c0c81bb94	fix: ensure parent folder exists when writing a file to object storage ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	60f6347c97	fix: cleanup objectstore file_put_content ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	3e94ee391f	fix: create intermediate directories for objectstore moveFromStorage ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	879fcc4253	fix: get source file before moving the cache item in objectstore moveFromStorage ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
provokateurin	d2d1e3284d	fix(Storage): Fix method docs inheritance ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago
Côme Chilliet	492e6997d8	chore: Fix psalm issues, put back private versions of getter in private Storage interface ... Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>	1 year ago
Christoph Fiehe	9597072ada	perf(ObjectStoreStorage): Improve (slow) move on same object bucket ... This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage. The behavior improves a lot with https://github.com/nextcloud/server/pull/46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder. Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com> Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>	1 year ago
Daniel Kesselberg	af6de04e9e	style: update codestyle for coding-standard 1.2.3 ... Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	1 year ago
Robin Appelman	bd740ac0b0	fix: write object to the correct urn when moving from another storage to object store ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Andy Scherzinger	dae7c159f7	chore: Add SPDX header ... Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>	2 years ago
Côme Chilliet	ec5133b739	fix: Apply new coding standard to all files ... Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>	2 years ago
Joas Schilling	951c330d39	Revert "don't perform an explicit setup when using the file cache"	2 years ago
Robin Appelman	4fba4cd14c	fix: fix user folder init ... Signed-off-by: Robin Appelman <robin@icewind.nl>	2 years ago
Côme Chilliet	c0ce272e9c	chore: Migrate away from OC::$server->getLogger ... Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>	2 years ago
Thomas Citharel	30798eb6c2	refactor(objectstorage): cleanup types ... Signed-off-by: Thomas Citharel <tcit@tcit.fr>	2 years ago
Thomas Citharel	3be3dbdb3b	refactor(objectstorage): move $user property from ObjectStoreStorage class to HomeObjectStoreStorage ... Signed-off-by: Thomas Citharel <tcit@tcit.fr>	2 years ago
Ferdinand Thiessen	5172baaf8b	fix(ObjectStore): Make copying behavior consistent with local storage ... Drop file permissions on copy like we do on local storage. Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>	2 years ago
Julius Härtl	88a96e4db7	fix: Only store unencrypted_size if path should be encrypted ... Signed-off-by: Julius Härtl <jus@bitgrid.net>	2 years ago
Robin Appelman	de85610385	use source cache when listing folder during recursive copy ... Signed-off-by: Robin Appelman <robin@icewind.nl>	3 years ago
Robin Appelman	cc0ad25c98	don't always check if we need to setup the object store root ... Signed-off-by: Robin Appelman <robin@icewind.nl>	3 years ago
Robin Appelman	2ea41dab93	repair -1 folder sizes for object store background scan ... Signed-off-by: Robin Appelman <robin@icewind.nl>	3 years ago
Robin Appelman	c6be96d04f	additional logging when mkdir fails for object storage ... Signed-off-by: Robin Appelman <robin@icewind.nl>	3 years ago
Robin Appelman	d63fa2d991	improve objectstore rmdir handling ... remove folder entries as they are cleared instead of in one go afterwards otherwise they stick around if some of the child entries can't be removed Signed-off-by: Robin Appelman <robin@icewind.nl>	3 years ago
Julius Härtl	e23aa8883e	feat(s3): Use multipart upload for chunked uploading ... This allows to stream file chunks directly to S3 during upload. Signed-off-by: Julius Härtl <jus@bitgrid.net>	3 years ago