How Create a math symbol

Without a full MWE it's difficult to be 100% sure, but I hope I've understood your intent.

What you need to do here is define a new math symbol \coexist, using your TikZ drawing, and make it responsive to different math styles (\displaystyle, \textstyle, \scriptstyle, \scriptscriptstyle) by wrapping it in \mathchoice. You can control the size of the drawing in each context manually via the scale=#1 argument in the helper macro \coexistSymbol.

This is a valid approach, and it should behave correctly in various environments — including with \big, in subscripts, superscripts, etc. If needed, you can further tweak the four scale values passed in \mathchoice to improve visual balance.

\documentclass{article}
\usepackage{tikz}
\usepackage{amsmath}
\newcommand{\coexist}{\mathrel{
  \mathchoice
    {\coexistSymbol{0.2}}% displaystyle
    {\coexistSymbol{0.1}}% textstyle
    {\coexistSymbol{0.05}}% scriptstyle
    {\coexistSymbol{0.025}}% scriptscriptstyle
}}
\newcommand{\coexistSymbol}[1]{%
  \vcenter{\hbox{%
    \begin{tikzpicture}[scale=#1, line width=0.4pt]
      \draw (-.85,-0.2) -- (-1,0) -- (0.15,1.5);
      \draw (-0.45,-0.65) -- (0.15,-1.5);
      \draw (-1.15,1.5) -- (-0.55,0.65);
      \draw (-1.15,-1.5) -- (0,0) -- (-0.15,0.2);
    \end{tikzpicture}%
  }}%
}

\begin{document}

Inline: $\langle \coexist \rangle$

\medskip

Displaystyle:
\[
\sum_{\coexist} x_{\coexist} = \big\langle \coexist \big\rangle
\]

\medskip

Scriptstyle: $f_{\coexist}$, $a^{\coexist}$

\medskip

Scriptscriptstyle: $f_{\scriptscriptstyle\coexist}$, $a^{\scriptscriptstyle\coexist}$

\medskip

$\displaystyle \prod_{i=1}^{n_{\coexist}} x_i^{\coexist}$

\end{document}

It works also with mathabx (adding it) avoiding to load after amsmath as it conflicts with the \iint command,

LaTeX Error: Command \iint already defined.
Or name \end... illegal, see p.192 of the manual.

See the comment of @cfr: Load amsmath before mathabx.

Using the answer of @egreg baseline=-\dimexpr\fontdimen22\textfont2\relax in this question Drawing the rules for the bracket polynomial, and avoiding the spaces given of \mathrel, you will have:

\documentclass{article}
\usepackage{tikz}
\usepackage{amsmath}
\newcommand{\coexist}{%
  \mathchoice
    {\coexistSymbol{0.15}}% displaystyle
    {\coexistSymbol{0.1}}% textstyle
    {\coexistSymbol{0.05}}% scriptstyle
    {\coexistSymbol{0.025}}% scriptscriptstyle
}
\newcommand{\coexistSymbol}[1]{%
  \vcenter{\hbox{%
    \begin{tikzpicture}[scale=#1, line width=0.4pt, baseline=-\dimexpr\fontdimen22\textfont2\relax]
      \draw (-.85,-0.2) -- (-1,0) -- (0.15,1.5);
      \draw (-0.45,-0.65) -- (0.15,-1.5);
      \draw (-1.15,1.5) -- (-0.55,0.65);
      \draw (-1.15,-1.5) -- (0,0) -- (-0.15,0.2);
    \end{tikzpicture}%
  }}%
}
\begin{document}

Inline: $\langle \coexist \rangle$

\medskip

Displaystyle:
\[
\sum_{\coexist} x_{\coexist} = \big\langle \coexist \big\rangle
\]

\medskip

Scriptstyle: $f_{\coexist}$, $a^{\coexist}$

\medskip

Scriptscriptstyle: $f_{\scriptscriptstyle\coexist}$, $a^{\scriptscriptstyle\coexist}$

\medskip

$\displaystyle \prod_{i=1}^{n_{\coexist}} x_i^{\coexist}$

\end{document}