Listing successful Nazi submarine attacks against US destroyers, this answer and the other feedback on that page, defies the presumption that "U-Boat attacks on naval vessels (especially destroyers) [were] such a rare occurrence".
It may have been rare, but it happened.
Wikipedia lists 5 Destroyers torpedoed by German U-Boats. I don't see a claim of 'unrealistic' can stand up in the face of confirmed proof.
Of note
And those are only the attacks that resulted in the ship being sunk.
I am generalizing this quesiton to all Axis attacks on Allied destroyers. See https://history.stackexchange.com/a/64283
At least 9 US destroyers or destroyer escorts were sunk by German submarines and at least 8 sunk by Japanese submarines (including midgets and Kaitens). These can be found in the easiest format here. Wiki - US Navy Losses.
And see Did a submarine ever torpedo and sink a destroyer?
From the German side, Ships sunk by German submarines in World War II lists the destroyers HMS Cossack, HMS Crusader, HMS Daring, USS Borie, USS Branch, USS Bristol, and USS Buck among other warships.
I would like to hear the other side of the story. Notwithstanding all these advantages of destroyers below, why did U-boats still attempt to torpedo destroyers?
There's big difference between merchant ships in a convoy and navy ships:
Merchant ships run at fixed and usually low speed; the speed of the slowest ship. The speed varied per convoy type.
Destroyers can dart around at 36 knots. Merchant ships couldn't quickly turn, the escorts could. (Flower class corvettes could do 16 knots, slightly slower than a surfaced Type VII sub.
Escorts (and most navy ships in general) have the means to sink a submarine. It was simply too dangerous for U-Boats to attack willy-nilly escorts ships.
Destroyers can turn on a dime, and present a small frontal target. You had to be a very brave U boat captain to target an approaching destroyer.
There is a big difference between corvettes and destroyers. A corvette was a 'just enough' solution: a converted whaler design that, as a stop gap, would be just enough to protect convoys. It was slower than U-Boats, had limited weaponry (certainly in the beginning of the war), and was better suite to keep U-Boats away rather than hunt them.
Destroyers were much (about twice) faster, heavily armed, much better equipped to find submarines. But they were also much more expensive to build, and building took more time. Hence the Flower class as stop gap solution.
Torpedoes are expensive. Better use them on merchant ships rather than elusive and dangerous escorts. U-boats could easily avoid them. They presented themselves small targets. At night an U-boat on the surface was almost invisible for other ships (the convoy).
That changed during the war when radar became more accurate and more available. When that happened U-Boats had no choice but to defend themselves or be sunk. Many of them were sunk that way.