0

I'm working on a semantic segmentation model through U-net for classifying 11 categories. After splitting my image and label data into training and testing arrays, I turn the arrays into Unsigned Integer 8 so I have

labels = labels.astype(np.uint8, copy=False)

array([[[[2],
         [2],
         [2],
         ...,
         [9],
         [9],
         [9]]]], dtype=uint8)

I then use to_categorical to make them into categorical shapes for Keras

labels_cat = to_categorical(labels)

However, I get

---> categorical = np.zeros((batch_size, num_classes))
MemoryError: Unable to allocate ___ GiB for an array with shape (_) and data type __

Moreover, every time I re-run the cell block, I get a different value for either the memory, shape, or data type. the memory would sometimes be 26 GiB or 1 GiB or even 100 Mb, or the shape would be something like (276824064, 13) or (100, 512, 512, 3). The datatype would be float64, float32, uint8.

What could be the reason for these fluctuating errors, and how do I fix it?

Improve this question
2
  • 2
    The reason is that you are allocating too much memory, and the fix would be to use less memory. Given the amount of detail you have provided, that's about all we can say. But if your program ever tries to allocate an array with 276824064 rows, there is most certainly something very wrong with your code. Commented Jan 5 at 8:47
  • @xdurch0 I was wondering if this was maybe a common problem with using to_categorical. The classification I'm doing makes use of 512x512 RGB training images with the aim of classifying 11 categories, and so I'm hot encoding the label data. Incidentally, I actually got this to finish running one time without changing anything, so I thought it was okay, but then it started giving me errors again. What other details could I provide? Your reply is much appreciated. Commented Jan 6 at 1:17
Related questions
2
sklearn TfidfVectorizer giving MemoryError
1041
How do I solve "error: externally-managed-environment" every time I use pip 3?
0
Tensorflow tf.squared_difference outputs unexpected shape
Load 3 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.