40 questions
0
votes
1
answer
33
views
AWS Datasync S3 to Google Cloud Storage: Full PutObjectFailed errors
Hope you are doing great.
We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. ...
- 75
0
votes
1
answer
100
views
AWS DataSync S3 transfer failed with Cannot allocate memory
I am using AWS DataSync to copy an S3 bucket from one region to another in the same account. For a few of my buckets, the DataSync task execution is failing with the error "Cannot allocate memory&...
- 1,748
0
votes
0
answers
365
views
AWS DataSync Create Location S3 failing
I am trying to set up Data Sync to migrate data from a S3 source bucket to a S3 destination bucket in a different account by following the video below:
https://www.youtube.com/watch?v=boqaSFilGTU
I ...
- 309
0
votes
1
answer
64
views
Hi fellow developers, there is something I need to know about DataSync Start ExecutionApi
I am working with Data sync start execution Api.
I have One stack in which I am creating all the locations, Tasks, Role and I am using DataSync client with startExecutionApi to call the task in the ...
0
votes
1
answer
1k
views
AWS DataSync some errors in your task execution. No affected files in CloudWatch logs
I'm getting this error when I run a DataSync task:
Transfer and verification completed. There were some errors in your task execution. See a list of affected files in CloudWatch logs. If no files are ...
- 355
0
votes
1
answer
979
views
I'm using AWS DataSync from on-premise to Amazon S3, If I delete the data in on-premise, will it sync to Amazon S3?
Does AWS DataSync sync the file that has been deleted in on-premise to Amazon S3, so from both location it is gone/deleted? If yes, can I turn off this feature? How?
I've read the documentation for ...
0
votes
1
answer
494
views
Which permissions for a cross-accounts transfert across two s3 with aws-datasync
I'm currently following the AWS official documentation tutorial for transferring a large volume of data between two S3 buckets, which can be found at this link: https://docs.aws.amazon.com/datasync/...
- 179
0
votes
1
answer
522
views
How to use TaskToken feature with Datasync task in a step function?
I am executing a datasync task from step function using sdk and appending waitforTaskExecution at the end. But the state keeps on running even if the datasync task is completed. I came to know about ...
- 57
0
votes
1
answer
1k
views
Execute AWS Datasync task using step function
We have a requirement to create a state machine which includes to datasync tasks followed by each other. These datasync tasks are already created and now I want to execute it using step functions one ...
- 57
1
vote
1
answer
2k
views
Permission issue when creating AWS datasync task
I am trying to use AWS datasync service to copy files from one bucket in one account to another bucket in another account. Also the buckets are in a different region.
I get this error message when I ...
- 31
0
votes
0
answers
371
views
Task timed out when creating DataSync agent through boto3 in aws lambda
I'm trying to create a DataSync agent. When I use the console with the same parameters, it works fine and very quickly (a few seconds). However, if I want to create it by using the lambda function, ...
- 75
0
votes
1
answer
597
views
Airbyte Missing Columns in tables
I am considering Airbyte as a replacement for singer-io because of its user-friendly UI and code reduction features.
However, facing an issue where Airbyte is unable to detect certain columns in our ...
1
vote
1
answer
613
views
aws datasync cause double trigger for lambda event
I use AWS DataSync service to sync two buckets A and B
The DataSync defined to check integrity during transfer and copy only new data to another bucket (B).
I have lambda function that defined to ...
- 417
1
vote
1
answer
3k
views
Efficient way to Copy/Replicate S3 Objects?
I need to replicate Millions (one time) of S3 Objs by modifying the metadata (within the same bucket, and obj path)
To perform this, we've various options mentioned below, we need to choose cost-...
- 1,223
-1
votes
1
answer
255
views
Sync S3 Objects Across Bucket with Intelligent Tiering
We want to sync all data from a us-east-1 bucket to us-west-2 bucket. However, we have intelligent tiering enabled. The us-east-1 bucket has thousands of objects where a large portion of them can be ...
- 188