$G := \{f : f:[0,1] \rightarrow [0,1]$ such that it is bijective function and strictly increasing }
Now the question is
For any $ h \in G,$does there exist $g \in G$ such that $h=g \circ g $?
Is such a $g$, if it exist , unique?
My observation :
Conjecture: if $h \in G$ has $n \in \mathbb{N}$ fixed points in (0,1) then it has $n+1$ 'square root' functions. Please help me to solve the question!
The square root always exists and, except for the special case where $f$ is the identity, there are uncountably many square roots.
Case I: $f$ has no fixed points in $(0,1)$. By the intermediate value theorem, we either have
I will consider (1) (the situation with $f(x) < x$ can be handled similarly). Choose any $x_0\in(0,1)$ and set $x_k=f^k(x_0)$, which is a strictly increasing sequence over $k\in\mathbb{Z}$. The limits of $x_k$ as $k\to\pm\infty$ are fixed points of $f$, so are equal to $1$ and $0$ respectively.
Now, choose any increasing homeomorphism $\theta\colon[0,1]\to[x_0,x_1]$. Extend $\theta$ to all of $\mathbb{R}$ by setting $$ \theta(k+x)=f^k(\theta(x)) $$ for $k\in\mathbb{Z}$ and $x\in[0,1)$. This defines a homeomorphism from $\mathbb{R}$ to $(0,1)$. Furthermore, $$ f(\theta(x))=\theta(x+1). $$ We can define a square root by $$ g(x) = \theta(\theta^{-1}(x)+1/2) $$ for $x\in(0,1)$, and $g(0)=0$, $g(1)=1$. Note that $g(x_0)=\theta(1/2)$, which can be chosen to be any value in $(x_0,x_1)$, so there are infinitely many square roots.
Case II: Now, for the general case.
Let $S\subseteq[0,1]$ be the set of fixed points of $f$. We define the square root to also be the identity on $S$. As $S$ is closed, its complement is a countable union of disjoint open intervals $(a,b)$ and, restricted to each such interval, $f$ gives a homeomorphism of $[a,b]$ with no fixed points in $(a,b)$. So, applying the construction above, there are uncountably many square roots on each such interval. So, $f$ has a square root and, except in the case where $S$ is all of $[0,1]$, there are uncountably many square roots.
Basic details on my comment above:
Suppose $g:[0,1]\rightarrow [0,1]$ is an increasing function. Let $y \in [0,1]$ be a point that satisfies $g(g(y))=y$. Then $g(y)=y$.
Suppose $y<g(y)$. Since $g$ is increasing we conclude $$ g(y) < g(g(y))=y$$ a contradiction. A similar contradiction holds if $y>g(y)$. Hence $y=g(y)$. $\Box$
It follows that fixed points of $h$ correspond to fixed points of $g$.
In particular, if $h$ is the identity function $h(x)=x$ for all $x\in[0,1]$, then its only increasing square root is also the identity function $g(x)=x$ for all $x \in [0,1]$. In this example, $h$ has an infinite number of fixed points, but only one (increasing) square root function. This example seems to disprove the conjecture given in the question.
