I do not precisely understand what you mean by "Why do the arrows ruin the left side but keep the right side smooth? ". However, this is just to say that there are many possible ways to draw smooth curves with TikZ. A rather powerful way is to employ the hobby library.
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{hobby}
\begin{document}
\begin{tikzpicture}[scale=.6,cap=round,
tangent/.style={%
in angle={(180+#1)} ,
Hobby finish ,
designated Hobby path=next , out angle=#1,
}]
\tikzset{axes/.style={}}
% The graphic
\begin{scope}[style=axes]
\draw[->] (-5,0) -- (5,0) node[below] {$x$};
\draw[->] (0,-3)-- (0,5) node[left] {$y$};
%\draw [gray!50] (-5,3) -- (-2.5,1) -- (2.5,4) -- (5,-1);
\draw [red,thick,<->,use Hobby shortcut]
([tangent=-80]-5,4) .. (-2.5,1) .. (2.5,4) .. ([tangent=-80]5,-1);
\end{scope}
\end{tikzpicture}
\end{document}

Does this come closer to what you want to achieve?
ADDENDUM: You are asking about the distortion you get when you add or remove an arrow. The effect you are referring to is clearly visible as the difference between the red and blue curves in
\documentclass{article}
\usepackage{tikz}
%\usetikzlibrary{bending}
\begin{document}
\begin{tikzpicture}[scale=.6,cap=round]
\tikzset{axes/.style={}}
% The graphic
\begin{scope}[style=axes]
\draw[->] (-5,0) -- (5,0) node[below] {$x$};
\draw[->] (0,-3)-- (0,5) node[left] {$y$};
%\draw [gray!50] (-5,3) -- (-2.5,1) -- (2.5,4) -- (5,-1);
\draw [red,thick,<->] plot [smooth,tension=1] coordinates {
(-5,4) (-2.5,1) (2.5,4) (5,-1)};
\draw [blue,thick,-] plot [smooth,tension=1] coordinates {
(-5,4) (-2.5,1) (2.5,4) (5,-1)};
\end{scope}
\end{tikzpicture}
\end{document}

This effect is due to the fact that a (standard) arrow inserts a stretch of straight (as opposed to curved) path. You can simply get rid of it by activating \usetikzlibrary{bending}, i.e. by removing the % in front of it.

Of course, if you have longer arrows, you want to make sure that you load arrows.meta as well and pass bend to the options of the arrow, e.g. `
\draw [red,thick,{Latex[bend,length=5pt]}-{Latex[bend,length=5pt]}] plot [smooth,tension=1] coordinates {
(-5,4) (-2.5,1) (2.5,4) (5,-1)};