All Questions
Tagged with aws-datasync amazon-s3
20 questions
0
votes
1
answer
33
views
AWS Datasync S3 to Google Cloud Storage: Full PutObjectFailed errors
Hope you are doing great.
We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. ...
- 75
0
votes
1
answer
101
views
AWS DataSync S3 transfer failed with Cannot allocate memory
I am using AWS DataSync to copy an S3 bucket from one region to another in the same account. For a few of my buckets, the DataSync task execution is failing with the error "Cannot allocate memory&...
- 1,748
0
votes
0
answers
365
views
AWS DataSync Create Location S3 failing
I am trying to set up Data Sync to migrate data from a S3 source bucket to a S3 destination bucket in a different account by following the video below:
https://www.youtube.com/watch?v=boqaSFilGTU
I ...
- 309
0
votes
1
answer
980
views
I'm using AWS DataSync from on-premise to Amazon S3, If I delete the data in on-premise, will it sync to Amazon S3?
Does AWS DataSync sync the file that has been deleted in on-premise to Amazon S3, so from both location it is gone/deleted? If yes, can I turn off this feature? How?
I've read the documentation for ...
0
votes
1
answer
494
views
Which permissions for a cross-accounts transfert across two s3 with aws-datasync
I'm currently following the AWS official documentation tutorial for transferring a large volume of data between two S3 buckets, which can be found at this link: https://docs.aws.amazon.com/datasync/...
- 179
1
vote
1
answer
2k
views
Permission issue when creating AWS datasync task
I am trying to use AWS datasync service to copy files from one bucket in one account to another bucket in another account. Also the buckets are in a different region.
I get this error message when I ...
- 31
1
vote
1
answer
613
views
aws datasync cause double trigger for lambda event
I use AWS DataSync service to sync two buckets A and B
The DataSync defined to check integrity during transfer and copy only new data to another bucket (B).
I have lambda function that defined to ...
- 417
1
vote
1
answer
3k
views
Efficient way to Copy/Replicate S3 Objects?
I need to replicate Millions (one time) of S3 Objs by modifying the metadata (within the same bucket, and obj path)
To perform this, we've various options mentioned below, we need to choose cost-...
- 1,223
-1
votes
1
answer
255
views
Sync S3 Objects Across Bucket with Intelligent Tiering
We want to sync all data from a us-east-1 bucket to us-west-2 bucket. However, we have intelligent tiering enabled. The us-east-1 bucket has thousands of objects where a large portion of them can be ...
- 192
0
votes
1
answer
490
views
Migrating a petabyte of data from an SMB drive to s3 while maintaining file creation data metadata?
I am currently trying to migrate about a petabyte of data that currently sits on a local SMB share to AWS S3. The problem I am having is the loss of the original file creation metadata when using ...
0
votes
1
answer
4k
views
S3 to S3 Cross account data replication
I have a S3 bucket in Account A and i want to replicate it to S3 bucket in Account B. I need the whole process along with security measures explained in detail. Starting from how to setup the whole ...
- 37
1
vote
0
answers
111
views
How can I install AWS Data sync agent on Hadoop cluster?
I have my hadoop sandbox set up on a Virtual Machine on Azure Cloud. I would like to install aws data sync agent on my hadoop sandbox so that I can create a task for data copy from my Hadoop cluster ...
- 23
1
vote
1
answer
4k
views
AWS DataSync: Unable to connect to S3 endpoint
I am trying to sync two S3 buckets in different accounts. I have successfully configured the locations and created a task. However, when I run the task I get a Unable to connect to S3 endpoint error. ...
7
votes
6
answers
17k
views
AWS Datasync S3 -> S3 cross account, confused about destination role/account
I want to use Datasync to copy data from a single S3 bucket in one account to a single S3 bucket in another account. I'm following this official AWS Datasync blog: https://aws.amazon.com/blogs/storage/...
- 1,090
0
votes
0
answers
797
views
Recurrent auto-sync of assets from one S3 bucket to another in separate account
The problem:
I have an old S3 bucket: bucket A and a new S3 bucket: bucket B. These buckets are in separate accounts. Up until now, I have been serving assets from bucket A. Moving forward, I want to ...
- 1,865