I arrive at the same conclusion as @heropup, with a different path (I use trigonometry for the very last computation).

Notations : $S$ for the area of triangle $ABC$ ; $s=\frac12(a+b+c)$ for its semi-perimeter, and $b:=AC=5$ and $c:=AB=11$.

I use relationship $$S=rs \ \iff \ r^2=\frac{S^2}{s^2}\tag{1}$$ (see here)

Using Heron's formula :

$$S^2=s(s-a)(s-b)(s-c),\tag{2}$$

we have to maximize $$r^2=\underbrace{\frac{1}{s}(s-a)(s-b)(s-c)}_{f(a)}=\frac{2(a^2 -36)(16 - a)}{16 + a} \ \text{for} \ a \in [c-a,c+a]=[6,16]$$

Fig. 1 : Graphical representation of function $f$ displaying the presence of a unique maximum of $r^2$ for a certain value $a=a_0$.

Differentiation gives :

$$f'(a)=\frac{-a^3 - 16a^2 + 256a + 576}{2(16 + a)^2}$$

Equation $f'(a)=0$ has three real roots. Only one of them is positive

$$a_0 \approx 11.264804945818$$

which is therefore the extremal value of $f$ (see fig. 1).

We know now $s=\frac12(a_0+b+c) \approx 13.632402472909$

By definition of $f$, we have $$S=s \sqrt{f(a_0)}\approx 27.082086683899.$$

The final touch : using formula $S=\frac12 bc \sin(\hat{A})$, we get

$$\hat{A}=\arcsin\left(\frac{2S}{bc}\right)\frac{180}{\pi} \approx 79.998482041911°$$

(decimal degrees) : not exactly $80°$ but so near !

I arrive at the same conclusion as @heropup, with a different path (I use trigonometry for the very last computation).

Notations : $S$ for the area of triangle $ABC$ ; $s=\frac12(a+b+c)$ for its semi-perimeter, and $b:=AC=5$ and $c:=AB=11$.

I use relationship $$S=rs \ \iff \ r^2=\frac{S^2}{s^2}\tag{1}$$ (see here)

Using Heron's formula :

$$S^2=s(s-a)(s-b)(s-c),\tag{2}$$

we have to maximize $$r^2=\underbrace{\frac{1}{s}(s-a)(s-b)(s-c)}_{f(a)}=\frac{2(a^2 -36)(16 - a)}{16 + a} \ \text{for} \ a \in [c-b,c+b]=[6,16]$$

Fig. 1 : Graphical representation of function $f$ showing that $r^2$ is maximal for a unique value $a=a_0$.

Differentiation gives :

$$f'(a)=\frac{-a^3 - 16a^2 + 256a + 576}{2(16 + a)^2}$$

Equation $f'(a)=0$ has three real roots. Only one of them is positive

$$a_0 \approx 11.264804945818$$

which is therefore the extremal value of $f$ (see fig. 1).

We know now $s=\frac12(a_0+b+c) \approx 13.632402472909$

By definition of $f$, we have $$S=s \sqrt{f(a_0)}\approx 27.082086683899.$$

The final touch : using formula $S=\frac12 bc \sin(\hat{A})$, we get

$$\hat{A}=\arcsin\left(\frac{2S}{bc}\right)\frac{180}{\pi} \approx 79.998482041911°$$

(decimal degrees) : not exactly $80°$ but so near !

Remark : For any triangle with given sides $a$ and $b$, this optimal angle giving a maximal inradius cannot be less than : $$2 \arcsin(\Phi-1) \ \text{radians}$$

(where $\Phi$ is golden ratio)

$\approx 76.3454153°$, this value being reached for isosceles triangles ($b=c$).

I arrive at the same conclusion as @heropup, with a different path (I use trigonometry for the very last computation).

I use relationship $$r^2=\frac{S^2}{s^2}\tag{1}$$ (see here) where $S$ is the area of triangle $ABC$ and $s=\frac12(a+b+c)$ the semi-perimeter, with

$b=AC=5$ and $c=AB=11$.

Using Heron's formula :

$$S^2=s(s-a)(s-b)(s-c)\tag{2}$$

we have to maximize $$r^2=\underbrace{\frac{1}{s}(s-a)(s-b)(s-c)}_{f(a)}=\frac{2(a^2 -36)(16 - a)}{16 + a}$$

Differentiation gives :

$$f'(a)=\frac{-a^3 - 16a^2 + 256a + 576}{2(16 + a)^2}$$

The extremal value of $f$ is attained for the (unique) real root of $f'(a)=0$ which is $$a_0 \approx 11.264804945818$$

We know now $s=\frac12(a_0+b+c) \approx 13.632402472909$

By definition of $f$, we have $$S=s \sqrt{f(a_0)}\approx 27.082086683899.$$

The final touch : using formula $S=\frac12 bc \sin(\hat{A})$, we get

$$\hat{A}=\arcsin\left(\frac{2S}{bc}\right)\frac{180}{\pi} \approx 79.998482041911°$$

(decimal degrees) : not exactly $80°$ but so near !

I arrive at the same conclusion as @heropup, with a different path (I use trigonometry for the very last computation).

Notations : $S$ for the area of triangle $ABC$ ; $s=\frac12(a+b+c)$ for its semi-perimeter, and $b:=AC=5$ and $c:=AB=11$.

I use relationship $$S=rs \ \iff \ r^2=\frac{S^2}{s^2}\tag{1}$$ (see here)

Using Heron's formula :

$$S^2=s(s-a)(s-b)(s-c),\tag{2}$$

we have to maximize $$r^2=\underbrace{\frac{1}{s}(s-a)(s-b)(s-c)}_{f(a)}=\frac{2(a^2 -36)(16 - a)}{16 + a} \ \text{for} \ a \in [c-a,c+a]=[6,16]$$

Fig. 1 : Graphical representation of function $f$ displaying the presence of a unique maximum of $r^2$ for a certain value $a=a_0$.

Differentiation gives :

$$f'(a)=\frac{-a^3 - 16a^2 + 256a + 576}{2(16 + a)^2}$$

Equation $f'(a)=0$ has three real roots. Only one of them is positive

$$a_0 \approx 11.264804945818$$

which is therefore the extremal value of $f$ (see fig. 1).

We know now $s=\frac12(a_0+b+c) \approx 13.632402472909$

By definition of $f$, we have $$S=s \sqrt{f(a_0)}\approx 27.082086683899.$$

The final touch : using formula $S=\frac12 bc \sin(\hat{A})$, we get

$$\hat{A}=\arcsin\left(\frac{2S}{bc}\right)\frac{180}{\pi} \approx 79.998482041911°$$

(decimal degrees) : not exactly $80°$ but so near !