331 Commits (master)

Author	SHA1	Message	Date
Carl Schwan	b6313f68d3	perf(s3): Expose pre-signed urls for S3 ... This is faster than going back to nextcloud to download the files. This is an opt-in setting that can be enabled by setting use_presigned_url in the object store config. Additionally add support for the proxy config which is needed in a docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431 Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	3 weeks ago
Daniel Kesselberg	f977a7fec6	fix(s3): make data integrity protections opt-in ... Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	4 weeks ago
provokateurin	2ab29a69e0	feat(ObjectStore): Allow overriding arguments per bucket ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 weeks ago
provokateurin	0c3a872908	feat(CertificateManager): Add option to specify the default certificates bundle path ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 weeks ago
Kent Delante	96672e1c4d	feat: emit an event when an S3 bucket is created ... Signed-off-by: Kent Delante <kent.delante@proton.me>	1 month ago
Carl Schwan	a228ffa641	refactor(s3): Readability ... Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com> Signed-off-by: Carl Schwan <carl@carlschwan.eu>	1 month ago
nfebe	3030783785	feat(objectstore): add configurable S3 retry attempts ... Add retriesMaxAttempts parameter to S3 objectstore configuration to allow customization of AWS SDK retry behavior for handling unreliable network conditions or proxy issues. Defaults to 5 retries (AWS SDK default) if not specified. Signed-off-by: nfebe <fenn25.fn@gmail.com>	1 month ago
Robin Appelman	a7d130c598	Revert "fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579" ... This reverts commit 4b4b39e7ec. Signed-off-by: Robin Appelman <robin@icewind.nl>	1 month ago
Robin Appelman	8194be795d	fix: encode s3 metadata as base64 if needed ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 month ago
Carl Schwan	67d0307e66	perf(s3): Cache whether bucket exists ... Otherwise, we call doesBucketExist all the time which does a network request to the S3 server adding some non-trivial latency when creating a S3 connection object. Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	2 months ago
Fiehe Christoph	b87add2711	fix(s3): expose request_checksum_calculation and response_checksum_validation ... Fix https://github.com/nextcloud/server/issues/56077 This commit makes the configuration settings 'request_checksum_calculation' and 'response_checksum_validation' of the S3Client from the AWS SDK for PHP configurable. Signed-off-by: Fiehe Christoph <c.fiehe@eurodata.de> Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	2 months ago
Daan Selen	4b4b39e7ec	fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579 ... Signed-off-by: Daan Selen <dselen@systemec.nl>	3 months ago
Thomas Citharel	5c8bdad828	fix(objectstore): add named parameter for like query ... to avoid 'object::%' to be considered as a column (`la colonne \\u00ab object::% \\u00bb n'existe pas` on PG) Signed-off-by: Thomas Citharel <tcit@tcit.fr>	3 months ago
Louis Chmn	00ec57efd6	fix(S3): Only append streams if non-seekable ... Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected. Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146 Signed-off-by: Louis Chmn <louis@chmn.me>	3 months ago
Carl Schwan	b0357663b9	perf(preview): Optimize migration and simplify DB layout ... * Simplify migration by not moving the actual files and just updating the DB * Don't store the storageid in the preview table as it is not needed * Start adding tests Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	3 months ago
Carl Schwan	13c35c0f17	perf(preview): Migrate previews to the new optimized table ... Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	3 months ago
Carl Schwan	18fbacdd8d	perf(preview): Split preview data to new table ... The new oc_previews table is optimized for storing previews and should decrease significantly the space taken by previews in the filecache table. This attend to reuse the IObjectStore abstraction over S3/Swift/Azure but currently only support one single bucket configuration. Signed-off-by: Carl Schwan <carl.schwan@nextclound.com>	3 months ago
yemkareems	200f9dc39f	fix: small typo fix ... Signed-off-by: yemkareems <yemkareems@gmail.com>	3 months ago
provokateurin	98081dfd42	feat(ObjectStore): Add option to limit the total size ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 months ago
Robin Appelman	0c1d5e8c54	fix: ensure all object store configuration have distict bucket names ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 months ago
Robin Appelman	02f4a82088	fix: make bucket mapper work with new multi-object-store config ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 months ago
Robin Appelman	b3c53c7436	feat: allow object store configuration aliases for easier migrations ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	2d4bba7b0c	feat: add command to get user objectstore config mappings ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	385dd36ff8	feat: allow configuring multiple objectstore configurations ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	97efc95efc	fix: better object store write error propagation ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	83b8a390cd	fix: always do stream counting for object store upload ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Robin Appelman	398b106f0c	fix: validate written size for s3 multipart uploads ... Signed-off-by: Robin Appelman <robin@icewind.nl>	5 months ago
Ferdinand Thiessen	5981b7eb51	chore: apply new CSFixer rules ... Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de> # Conflicts: # apps/settings/lib/SetupChecks/PhpOpcacheSetup.php	6 months ago
Robin Appelman	2e1c316582	fix: remove double cache write in ObjectStoreStorage::touch ... the file_put_contents already puts all of those values into the cache Signed-off-by: Robin Appelman <robin@icewind.nl>	6 months ago
Daniel Calviño Sánchez	d7ae952dc8	fix(ObjectStore): Make S3 "connect_timeout" option configurable ... The hardcoded connection timeout of 5 seconds may not be enough in some cases, so now it is got from the ObjectStore arguments in Nextcloud configuration, falling back to 5 if not set. The connection timeout is set in seconds, but decimal precision can be used for subsecond accuracy (for example, 4.2 for 4200 milliseconds). Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>	7 months ago
Kent Delante	06b5ce5859	fix(s3): retry failed multipart uploads with decreased concurrency ... Signed-off-by: Kent Delante <kent.delante@proton.me>	7 months ago
Robin Appelman	5b3f4cd8fe	perf: set the folder size in the same query as we create it ... Signed-off-by: Robin Appelman <robin@icewind.nl>	7 months ago
Louis Chemineau	1f8b823992	fix(S3): Use original folder size during copy ... This prevent having copied folders with a wrongly set size of 0KB. - Fix https://github.com/nextcloud/server/issues/51916 Signed-off-by: Louis Chemineau <louis@chmn.me>	7 months ago
Robin Appelman	7599162c7b	feat: move primary object store configuration to a single place ... Signed-off-by: Robin Appelman <robin@icewind.nl>	7 months ago
Anna Larch	49baa914cf	fix(objectstorage): add retry attempts to S3 connection ... Signed-off-by: Anna Larch <anna@nextcloud.com>	8 months ago
Louis Chemineau	8fdf2a7eae	fix: Transfer ownership with S3 as primary ... When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error: `SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'` The `--move` option moves the entire home folder from one account to another. The error means that the move failed because the destination folder already exist in `oc_filecache`. - With S3 as primary storage, folders only exists as entries in `oc_filecache`. - With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder. - With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder. - `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112) - `oneTimeUserSetup(...)`: 687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262) - `mkdir(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135) - `moveFromStorage(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636) Delete pre-existing folder in `moveFromStorage(...)` Signed-off-by: Louis Chemineau <louis@chmn.me>	9 months ago
Robin Appelman	8aae332622	feat: store original storage id and path in object store metadata ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Robin Appelman	94114b99f7	feat: more generic way of passing metadata to object storage backends for new objects ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Robin Appelman	fcde776683	feat: add command to list objects ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Robin Appelman	c3bc362f48	feat: add command to get object metadata ... Signed-off-by: Robin Appelman <robin@icewind.nl>	9 months ago
Benjamin Gaussorgues	9bdee76535	perf(files): faster query to fetch incomplete directories ... Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>	11 months ago
Hector Valcarcel	6b4c859a41	feat(object_store): Add support for session token in AWS credentials ... - Pass session token, either null or with value, to the AWS Credentials constructor Signed-off-by: Hector Valcarcel <hmvalcarcel@gmail.com>	11 months ago
Richard Steinmetz	f1c025dfd3	fix(s3): treat empty sse_c_key as not set ... Co-authored-by: Christian Becker <christian@dabecka.de> Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>	11 months ago
Ferdinand Thiessen	b48ee2e924	fix: Harden files scanner for invalid null access ... Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de> Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com> Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>	11 months ago
Robin Appelman	5896f8c37f	chore: cleanup leftover debug statement ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	6cf66f95ce	fix: get object size from stream where possible ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	ca547bb913	fix: don't perform the extra buffering in s3 stream write when the stream size is known ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	7da8951c57	fix: use php://temp instead of php://memory for multi-part upload buffer ... this should reduce potential memory issues if the limit is set very high Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Robin Appelman	df3a9e221d	fix: throw correct exception type when we can't verify if an s3 bucket exists ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
provokateurin	0de4843b73	refactor(Storage): Align all Storage constructors ... Signed-off-by: provokateurin <kate@provokateurin.de>	1 year ago