Instead of forcing Mathematica to arrive at the desired form, we can analyse the expression. Let us first get rid of the square root, which clearly cannot be simplified:
sqrt = Sqrt[1 + (3/2 - Sqrt[2]) (1 + U^2)]
Then we look at the remaining part:
expr2 = expr*sqrt /. {U -> Sqrt[x]};
FunctionPoles[expr2, x]
(* {} *)
Thus, expr2 is a polynomial. Next we find the roots of the numerator and denominator, which can be simplified with methods for polynomial functions:
Solve[Numerator[expr3]expr3 === 0PolynomialQuotient[Numerator[expr2], Denominator[expr2], x] // FullSimplify
Solve[Denominator[expr3] == 0, x] // FullSimplify
(* {{x -> -7 - 4 Sqrt[2]}, {x -> -7 - 4 Sqrt[2]}, {x -> -6 -
4 Sqrt[2]}, {x -> -6 - 4 Sqrt[2]}, {x -> -6 -
1/4 Sqrt[2]}, {x -> -6r (- 4 Sqrt[2]}, {x -> -3 - 2 Sqrt[2] + (
2c (2 + Sqrt[2]))/c}, {x -> -3 - 2 Sqrt[2] + (2 (-2 + Sqrt[2]) x)/c}}) *)
(* {{x -> -7 - 4 Sqrt[2]}, {x -> -7 - 4 Sqrt[2]}, {x -> -6 -
4 Sqrt[2]}, {x -> -6 - 4 Sqrt[2]}, {x -> -6 -
And the final simplified expression recovered as
exprSimple = 4(expr3 Sqrt[2]},/. {x -> -6 - 4 Sqrt[2]U^2}, {x -> -3 - 2 Sqrt[2] + (
2 (2 + Sqrt[2]))/c}} *)sqrt
This leads to a lot cancellations, which can be automated.$$\frac{r \left(c \left(-\left(\left(\sqrt{2}-2\right) U^2\right)+\sqrt{2}+2\right)-4\right)}{4 \sqrt{\left(\frac{3}{2}-\sqrt{2}\right) \left(U^2+1\right)+1}}$$