79 Commits (master)

Author	SHA1	Message	Date
Carl Schwan	b6313f68d3	perf(s3): Expose pre-signed urls for S3 ... This is faster than going back to nextcloud to download the files. This is an opt-in setting that can be enabled by setting use_presigned_url in the object store config. Additionally add support for the proxy config which is needed in a docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431 Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	3 weeks ago
Daniel Kesselberg	f977a7fec6	fix(s3): make data integrity protections opt-in ... Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	4 weeks ago
provokateurin	2ab29a69e0	feat(ObjectStore): Allow overriding arguments per bucket ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 weeks ago
provokateurin	0c3a872908	feat(CertificateManager): Add option to specify the default certificates bundle path ... Signed-off-by: provokateurin <kate@provokateurin.de>	4 weeks ago
Kent Delante	96672e1c4d	feat: emit an event when an S3 bucket is created ... Signed-off-by: Kent Delante <kent.delante@proton.me>	1 month ago
Carl Schwan	a228ffa641	refactor(s3): Readability ... Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com> Signed-off-by: Carl Schwan <carl@carlschwan.eu>	1 month ago
nfebe	3030783785	feat(objectstore): add configurable S3 retry attempts ... Add retriesMaxAttempts parameter to S3 objectstore configuration to allow customization of AWS SDK retry behavior for handling unreliable network conditions or proxy issues. Defaults to 5 retries (AWS SDK default) if not specified. Signed-off-by: nfebe <fenn25.fn@gmail.com>	1 month ago
Carl Schwan	67d0307e66	perf(s3): Cache whether bucket exists ... Otherwise, we call doesBucketExist all the time which does a network request to the S3 server adding some non-trivial latency when creating a S3 connection object. Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>	2 months ago
Fiehe Christoph	b87add2711	fix(s3): expose request_checksum_calculation and response_checksum_validation ... Fix https://github.com/nextcloud/server/issues/56077 This commit makes the configuration settings 'request_checksum_calculation' and 'response_checksum_validation' of the S3Client from the AWS SDK for PHP configurable. Signed-off-by: Fiehe Christoph <c.fiehe@eurodata.de> Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	2 months ago
Ferdinand Thiessen	5981b7eb51	chore: apply new CSFixer rules ... Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de> # Conflicts: # apps/settings/lib/SetupChecks/PhpOpcacheSetup.php	6 months ago
Daniel Calviño Sánchez	d7ae952dc8	fix(ObjectStore): Make S3 "connect_timeout" option configurable ... The hardcoded connection timeout of 5 seconds may not be enough in some cases, so now it is got from the ObjectStore arguments in Nextcloud configuration, falling back to 5 if not set. The connection timeout is set in seconds, but decimal precision can be used for subsecond accuracy (for example, 4.2 for 4200 milliseconds). Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>	6 months ago
Anna Larch	49baa914cf	fix(objectstorage): add retry attempts to S3 connection ... Signed-off-by: Anna Larch <anna@nextcloud.com>	8 months ago
Hector Valcarcel	6b4c859a41	feat(object_store): Add support for session token in AWS credentials ... - Pass session token, either null or with value, to the AWS Credentials constructor Signed-off-by: Hector Valcarcel <hmvalcarcel@gmail.com>	11 months ago
Richard Steinmetz	f1c025dfd3	fix(s3): treat empty sse_c_key as not set ... Co-authored-by: Christian Becker <christian@dabecka.de> Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>	11 months ago
Robin Appelman	df3a9e221d	fix: throw correct exception type when we can't verify if an s3 bucket exists ... Signed-off-by: Robin Appelman <robin@icewind.nl>	1 year ago
Daniel Kesselberg	af6de04e9e	style: update codestyle for coding-standard 1.2.3 ... Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>	1 year ago
Git'Fellow	c84e76a749	fix(s3): Don't wait indefinitely for S3 to return ... Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: lint Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: use AwsException Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: Throw on connection failure Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: Wrap all in try catch block Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: use RequestTimeout error message Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> log: use OCP Server class Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: Handle connect timeout only Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> fix: Handle errors more generically Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>	2 years ago
Ferdinand Thiessen	ae1f5cc56d	fix(S3): Adjust usage of guzzle promise ... `Promise\promise_for` was deprecated and is now removed and replaced with the static API (`Create::promiseFor`). Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>	2 years ago
Josh	63886147ba	fix(ObjectStore): handle empty S3 hostname ... Fixes #45637 The support for s3-accelerate added in #44496 introduced a regression in AWS S3 environments when `hostname` is blank (which is a valid configuration w/ AWS since the hostname gets auto-generated). Signed-off-by: Josh <josh.t.richards@gmail.com>	2 years ago
Andy Scherzinger	dae7c159f7	chore: Add SPDX header ... Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>	2 years ago
Robin Appelman	40d9a3fcab	feat: add option to enable s3 acceleration ... Signed-off-by: Robin Appelman <robin@icewind.nl>	2 years ago
Ferdinand Thiessen	32dee2f84a	feat(ObjectStore): Make S3 MultipartUpload concurrency configurable ... Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>	2 years ago
Ferdinand Thiessen	69c5e8ebde	fix(ObjectStore): Use common S3 trait so config and connection trait members are in sync ... Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>	2 years ago
Julius Härtl	e4054370b1	fix(s3): Add config option to disable multipart copy for certain s3 providers ... Signed-off-by: Julius Härtl <jus@bitgrid.net>	2 years ago
Robin Appelman	1043c21b35	only do a multipart s3 copy when above the regular copy limit ... Signed-off-by: Robin Appelman <robin@icewind.nl>	2 years ago
Hamid Dehnavi	ea06cf2f39	Convert isset ternary to null coalescing operator ... Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>	2 years ago
Thomas Citharel	3f28fc58aa	fix(s3): fix handling verify_bucket_exists parameter ... If 'verify_bucket_exists' is set to false in the config.php s3 configuration, it's supposed to avoid verifying that the bucket exists. However empty(falsy) will always return true, so this condition would not work. Signed-off-by: Thomas Citharel <tcit@tcit.fr>	2 years ago
Julius Härtl	159a0c8411	feat(s3): Add option to specify an SSE-C customer provided key ... Signed-off-by: Julius Härtl <jus@bitgrid.net>	3 years ago
François Ménabé	cf9bdc0274	Manage s3 storage class in objectstore ... Signed-off-by: François Ménabé <francois.menabe@gmail.com>	3 years ago
Robin Appelman	f1486890d7	only use nextcloud bundle when explicitly enabled ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 years ago
Robin Appelman	bffa67c48b	also use nextcloud certificate bundle when downloading from s3 ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 years ago
Jasper Weyne	0633a1d9f5	Disable ~/.aws/config access by S3ConnectionTrait ... This commit sets the 'use_aws_shared_config_files' option to false, in order to disable configuration loading from ~/.aws/config by the AWS SDK, specifically the S3Client. It is a continuation of #27040, as that PR only changed the behaviour of the CredentialsProvider; this change affects the ConfigurationProvider as well. Signed-off-by: Jasper Weyne <jasperweyne@gmail.com>	4 years ago
Côme Chilliet	069477e608	Migrate more classes of lib/private to LoggerInterface ... Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>	4 years ago
Robin Appelman	1156214a26	don't try to get custom certs for s3 primary storage ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 years ago
Robin Appelman	194a21f374	use the nextcloud certificate bundle for s3 ... Signed-off-by: Robin Appelman <robin@icewind.nl>	4 years ago
Julius Härtl	2aac757805	Make max size for single put uploads configurable ... Signed-off-by: Julius Härtl <jus@bitgrid.net>	4 years ago
Julius Härtl	e475dfe16c	Fix php-cs for S3ConnectionTrait.php ... Signed-off-by: Julius Härtl <jus@bitgrid.net>	4 years ago
Stephen Cuppett	4a6a2c2b48	Simplify S3ConnectionTrait to defaultProvider plus option ... When we initially added the EC2 and ECS IAM role support in #24700, we had to use a workaround by explicitly ordering the various providers due to an inconsistency in the AWS SDK for PHP. We submitted a PR there to get that squared away. Now, we've consumed that version upstream for the SDK and can update our code here to be the most concise version as well as position ourselves to pick up new methods as those become available and prevalent in AWS (for acquiring credentials). See also: https://github.com/nextcloud/server/pull/24700#issuecomment-747650892 See also: https://github.com/aws/aws-sdk-php/pull/2172 Signed-off-by: Stephen Cuppett <steve@cuppett.com>	4 years ago
Bernd Rederlechner	3866f388b1	Refactor writeObject to only use MultipartUpload when required ... Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com> Co-authored-by: Julius Härtl <jus@bitgrid.net>	4 years ago
Maxime Besson	388a458cd5	fix(proxy): use accessor ... Signed-off-by: Maxime Besson <maxime.besson@worteks.com>	5 years ago
John Molakvoæ (skjnldsv)	215aef3cbd	Update php licenses ... Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>	5 years ago
Florent	d4444f2472	Avoid reading ~/.aws/config when using S3 provider ... When using S3 storage, we sometimes have open_basedir errors due to aws-sdk-php trying to read ~/.aws/config which is out of open_basedir restrictions. Christoph Wurst already added csm=false config in #21406 but it wasn't enough, we also need to set use_arn_region=false, added in this commit. Signed-off-by: Florent <florent@coppint.com>	5 years ago
Samuel	03fe74b95e	fix(proxy): reaching s3 storage behind some http proxy ... Signed-off-by: Maxime Besson <maxime.besson@worteks.com>	5 years ago
Christoph Wurst	d89a75be0b	Update all license headers for Nextcloud 21 ... Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>	5 years ago
James Letendre	45a02ee30c	Fix failing tests ... Signed-off-by: James Letendre <james.letendre@gmail.com>	5 years ago
James Letendre	ad95e51cbe	Add web identity provider to S3 connection chain ... Signed-off-by: James Letendre <james.letendre@gmail.com>	5 years ago
James Letendre	5b756a9fb0	Replace defaultProvider with explicit calls to exclude user home directory lookup ... Signed-off-by: James Letendre <james.letendre@gmail.com>	5 years ago
James Letendre	85aa77539b	Resolves #24699, Support ES2 and ECS instance providers for S3 buckets ... Signed-off-by: James Letendre <james.letendre@gmail.com>	5 years ago
Roeland Jago Douma	65141d4864	Allow config to specify the bucket exists ... In the 99% case the bucket is just always there. And if it is not the read/write will fail hard anyways. Esp on big instances the Objectstore is not always fast and this can save a few hundered ms of each request that acess the objectstore. In short it is adding 'verify_bucket_exists' => false To the S3 config part Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>	5 years ago
Christoph Wurst	2a054e6c04	Update the license headers for Nextcloud 20 ... Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>	5 years ago