Is the perfect antenna going to be (50+0j) Ω?

That will be the perfectly matched antenna, with the minimum feedline loss. It's not necessarily the perfect antenna. A dummy load is 50+j0 too, and any time you see an antenna that has a "nice flat SWR" over a broad bandwidth, it's a fair bet that it has a bit of dummy-load in its ancestry.

Is it the imaginary part that changes the SWR?

It's both, but SWR changes a lot faster with reactance than resistance, and when you're tuning a wire antenna that's not far from resonance, the reactance changes a lot faster with length than the resistance does. Those two things put together mean that the point of zero reactance tends to be awfully close to the point of minimum SWR.

How do you get a piece of copper wire to change its resistance to get to 50 Ω if its not?
If I mess with it enough, am I ever going to get close to (50+0j) Ω?
Should I even care, if it works pretty well as is?

Last question first: nope! A practical approach for a dipole is simply to tune it so that the minimum SWR is in the middle of the band (or the middle of the segment you want to operate on), and then check the values at the edges. If they're good, you're done. With a dipole, you should pretty much always end up in a place where almost any transmitter will be happy and your feedline losses won't be excessive. Your minimum SWR will probably be 1.2 or 1.3, not 1.0. That's not even remotely an issue.

But say you're dealing with some other sort of antenna design that "natively" has a feedpoint impedance further from 50 ohms. A folded dipole or a full-wave loop might be several hundred ohms at resonance. An EFHW might be several thousand. Some kinds of Yagis might be 20-25 ohms. And you'll probably never build a 160m vertical tall enough to have more than a few ohms of resistance.

In those cases, some kind of transformation or matching comes into play. A "4:1 balun" or "49:1 unun" is a transformer that steps an impedance up or down to a more convenient value. A gamma match, delta match, or shunt feed is a kind of "one-turn transformer" built in to the antenna structure that we can tap at a point that gives a feedpoint impedance we want. And then there are RC matching networks, which can be placed at the antenna (to match the feedline), or at the shack (to match whatever comes off of the feedline to 50 ohms). A combination of parallel and series elements will change the magnitude of an impedance, besides adding or subtracting reactance. Different options have different tradeoffs in efficiency, convenience, cost, size, power handling, etc.