Objective: Crop the image so only the number stays in the image

Problem: Slow Performance

I have code that crops an image. The image pixels are 0 or 255. There are no values between.

The background is 0 (black) The Letter/Number is between 0(not-inclusive) - 255 (white)

This code is being used to crop a Mnist Digit of the Mnist Dataset.

Image Source: http://multithreaded.stitchfix.com/blog/2015/12/09/intro-to-chainer/

The code below does it. However, it does with 2 fors And it takes a long time!

How can I optimize it?

def crop_image(data_to_crop):
    cropped_data = []

    for z in xrange(data_to_crop.shape[0]):

        img = data_to_crop[z]
        img = img.reshape(data_to_crop.shape[0], data_to_crop.shape[1])

        rx = -1
        upy = -1

        lx = -1
        by = -1

        for x in xrange(data_to_crop.shape[0]):
            for y in xrange(data_to_crop.shape[1]):

                px = img[x, y]

                if px > 0:

                    if rx == -1 or x > rx:
                        rx = x

                    if lx == -1 or x < lx:
                        lx = x

                    if upy == -1 or y > upy:
                        upy = y

                    if by == -1 or y < by:
                        by = y

        img = img[lx:rx, by:upy]

        cropped_data.append(img)

    return cropped_data