Objective: Crop the image so only the number stays in the image

Problem: Slow Performance

I have code that crops an image. The image pixels are 0 or 255. There are no values between.

The background is 0 (black) and the letter/number is between 0 (not-inclusive) - 255 (white).

This code is being used to crop a Mnist Digit of the Mnist Dataset.

Image Source

The code does it, however, it does with 2 fors and it takes a long time! How can I optimize it?

def crop_image(data_to_crop):
    cropped_data = []

    for z in xrange(data_to_crop.shape[0]):

        img = data_to_crop[z]
        img = img.reshape(data_to_crop.shape[0], data_to_crop.shape[1])

        rx = -1
        upy = -1

        lx = -1
        by = -1

        for x in xrange(data_to_crop.shape[0]):
            for y in xrange(data_to_crop.shape[1]):

                px = img[x, y]

                if px > 0:

                    if rx == -1 or x > rx:
                        rx = x

                    if lx == -1 or x < lx:
                        lx = x

                    if upy == -1 or y > upy:
                        upy = y

                    if by == -1 or y < by:
                        by = y

        img = img[lx:rx, by:upy]

        cropped_data.append(img)

    return cropped_data