How to feed large NumPy arrays to tf.fit()

Question

I have two NumPy arrays saved in .npy file extension. One contains x_train data and other contains y_train data.

The x_train.npy file is 5.7GB of size. I can't feed it to the training by loading the whole array to the memory.

Every time I try to load it to RAM and train the model, Colab crashes before starting the training.

Is there a way to feed large Numpy files to tf.fit()

files I have:

"x_train.npy" 5.7GB
"y_train.npy"

It is a Resnet type model for audio. Each input has a size of (16000,1). It can be fitted by batches — Oshan Jayawardana
– Oshan Jayawardana, Commented Mar 28, 2022 at 11:07
Then I advise loading it from disk by batches and clearing these from RAM as you iterate over the whole dataset. — Learning is a mess
– Learning is a mess, Commented Mar 28, 2022 at 11:42

Blurred0993 · Accepted Answer · 2022-03-28 10:46:35Z

0

Depending on how much RAM your device has, it may not be possible from a hardware point of view.

answered Mar 28, 2022 at 10:46

Blurred0993

1531 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Oshan Jayawardana Over a year ago

I'm using Colab. I think that is 12GB ram. Isn't there any way to work around this? I found this solution link. But it doesn't say how to use y labels

Collectives™ on Stack Overflow

How to feed large NumPy arrays to tf.fit()

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related