Algorithm to find edges of groups of intersections running to slow

I have an algorithm that takes in an array of interections found between lines in an image. It then has to look for any instances where there are a group of intersections together, remove the middle ones and give me back all the edges of it.

I have it working as well as I need it too except for the performance... it takes waaaay to long. on my machine about 30-40 seconds for about 18000 intersections. I was wondering if anyone had any performance tips that could help me.

This is my code:

private Pixel[] RemoveBlobsOfIntersections(Pixel[] Intersections)
{
    Intersections = Intersections.OrderBy(i => i.X).ThenBy(i => i.Y).ToArray();

    List<Pixel> plAlreadyIteratedOver = new List<Pixel>();
    List<Pixel> plNoBlobsOfIntersections = new List<Pixel>();
    foreach (Pixel pIntersection in Intersections)
    {
        if (!plAlreadyIteratedOver.Any(p => p.X == pIntersection.X && p.Y == pIntersection.Y))
        {
            Pixel[] paBlob = FindBlob(pIntersection, Intersections);
            plAlreadyIteratedOver.AddRange(paBlob);

            plNoBlobsOfIntersections.AddRange(GetBlobEdges(paBlob));
        }
    }

    return plNoBlobsOfIntersections.ToArray();
}

private Pixel[] FindBlob(Pixel StartingPixel, Pixel[] PixelArray)
{
    List<Pixel> plBlob = new List<Pixel>() { StartingPixel };

    List<Pixel> plPixelsToCheck = new List<Pixel>(plBlob);
    while (plPixelsToCheck.Count > 0)
    {
        List<Pixel> plTempList = new List<Pixel>();
        foreach (Pixel pixel in plPixelsToCheck)
        {
            Pixel[] paSurroundingPixels = GetSurroundingPixels(pixel, PixelArray);
            foreach (Pixel p in paSurroundingPixels)
            {
                if (p != null && !plBlob.Any(bp => bp.X == p.X && bp.Y == p.Y) && !plTempList.Any(bp => bp.X == p.X && bp.Y == p.Y) && !plPixelsToCheck.Any(bp => bp.X == p.X && bp.Y == p.Y))
                    plTempList.Add(p);
            }

            plBlob.Add(pixel);
        }

        plPixelsToCheck.Clear();
        plPixelsToCheck.AddRange(plTempList);
    }

    return plBlob.ToArray();
}

private Pixel[] GetSurroundingPixels(Pixel StartingPixel, Pixel[] PixelArray)
{
    Pixel[] paSurroudingPixels = new Pixel[8];

    paSurroudingPixels[0] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X - 1) && p.Y == (StartingPixel.Y - 1));
    paSurroudingPixels[1] = PixelArray.FirstOrDefault(p => p.X == StartingPixel.X && p.Y == (StartingPixel.Y - 1));
    paSurroudingPixels[2] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X + 1) && p.Y == (StartingPixel.Y - 1));

    paSurroudingPixels[3] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X - 1) && p.Y == StartingPixel.Y);
    paSurroudingPixels[4] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X + 1) && p.Y == StartingPixel.Y);

    paSurroudingPixels[5] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X - 1) && p.Y == (StartingPixel.Y + 1));
    paSurroudingPixels[6] = PixelArray.FirstOrDefault(p => p.X == StartingPixel.X && p.Y == (StartingPixel.Y + 1));
    paSurroudingPixels[7] = PixelArray.FirstOrDefault(p => p.X == (StartingPixel.X + 1) && p.Y == (StartingPixel.Y + 1));

    return paSurroudingPixels;
}

private Pixel[] GetBlobEdges(Pixel[] PixelArray)
{
    List<Pixel> plBlobEdges = new List<Pixel>();

    IGrouping<int, Pixel>[] xGroupedPixels = PixelArray.GroupBy(p => p.X).ToArray();
    foreach (IGrouping<int, Pixel> xGroup in xGroupedPixels)
    {
        Pixel[] paOrderedXGroup = xGroup.OrderBy(b => b.X).ToArray();

        if (!plBlobEdges.Any(p => p.X == paOrderedXGroup[0].X && p.Y == paOrderedXGroup[0].Y))
            plBlobEdges.Add(paOrderedXGroup[0]);

        if (!plBlobEdges.Any(p => p.X == paOrderedXGroup[paOrderedXGroup.Length - 1].X && p.Y == paOrderedXGroup[paOrderedXGroup.Length - 1].Y))
            plBlobEdges.Add(paOrderedXGroup[paOrderedXGroup.Length - 1]);
    }

    IGrouping<int, Pixel>[] yGroupedPixels = PixelArray.GroupBy(p => p.Y).ToArray();
    foreach (IGrouping<int, Pixel> yGroup in yGroupedPixels)
    {
        Pixel[] paOrderedYGroup = yGroup.OrderBy(b => b.X).ToArray();

        if (!plBlobEdges.Any(p => p.X == paOrderedYGroup[0].X && p.Y == paOrderedYGroup[0].Y))
            plBlobEdges.Add(paOrderedYGroup[0]);

        if (!plBlobEdges.Any(p => p.X == paOrderedYGroup[paOrderedYGroup.Length - 1].X && p.Y == paOrderedYGroup[paOrderedYGroup.Length - 1].Y))
            plBlobEdges.Add(paOrderedYGroup[paOrderedYGroup.Length - 1]);
    }

    return plBlobEdges.ToArray();
}

And the Pixel class:

public class Pixel
{
    public int X { get { return _nX; } }

    public int Y { get { return _nY; } }

    public bool IsBlack { get { return _bIsBlack; } }

    public float Confidence { get { return _fConfidence; } }

    private int _nX;
    private int _nY;
    private bool _bIsBlack;
    private float _fConfidence;

    public Pixel(int x, int y, bool isBlack, float confidence)
    {
        _nX = x;
        _nY = y;
        _bIsBlack = isBlack;
        _fConfidence = confidence < 0 ? 0 : confidence;
    }
}

EDIT:

A blob is a group of pixels in an image - the pixels I am searching on are the intersections found in the image between horizontal and vertical lines. However as you can imagine if a few horizontal and vertical lines meet in the same area a group (blob) of intersections are joined together. I am trying to remove these blobs by finding the edges.

An image of the stages of the process. From top to bottom. 1: initial 2: Found lines (Green horizontal, Blue vertical) 3: With found intersections(Orange), With edges of blobs/groups(Purple).

As you can see I initially find all the lines in the image, followed by finding the intersections within it. On the one on the right however 9 intersections are found in a group and so I am trying to remove unnecessary ones by going to an outline of them (Note in the actual image the groups are much larger e.g. the largest of which is about 70 x 50 pixels is size)