AWS DataSync S3 transfer failed with Cannot allocate memory

Question

I am using AWS DataSync to copy an S3 bucket from one region to another in the same account. For a few of my buckets, the DataSync task execution is failing with the error "Cannot allocate memory". It works for other buckets though. I am guessing that the failing bucket might be too big for DataSync to handle but I am not sure.

How can I fix the issue?

Alex · Accepted Answer · 2025-02-26 04:50:10Z

0

How are you transferring the data? Did you set up an EC2 instance and run the DataSync agent there? If so, you might need to allocate more memory.

If you're doing it without an EC2 instance, I think you're right — the bucket might be too large. You should consider filtering the data and transferring it step by step, especially if it's a one-time operation.

If this also does not work you may need to contact support - in my experience DataSync behaves weird sometimes and I had multiple cases where the support had to fix something for me, before it worked.

answered Feb 26 at 4:50

Alex

131 silver badge2 bronze badges

I don't think filtering would work because there is no information on the directory structure. What if the bucket is flat and has all .jpeg files in it in the format "{0-9}+.jpeg"? Furthermore, without knowing how much data DataSync can transfer, you cannot decide the filter because it is possible that the filter might again be too big. I guess best option would be to reach out to AWS Support.
– Samkit Jain
Commented Mar 4 at 8:42

Add a comment |

Collectives™ on Stack Overflow

AWS DataSync S3 transfer failed with Cannot allocate memory

1 Answer 1

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Related