Circles packed between $y=1/x$ and $y=0$ in the first quadrant: What is the radius of the $n$th circle?

For dimensional homogeneity, let the hyperbola have equation $xy=s^2$.

At parameterized point $P=(sp,s/p)$ (for dimensionless $p$) on the hyperbola, a tangent vector is $\dfrac{p^2}{s}\dfrac{dP}{dp}=(p^2,-1)$, so that the unit downward-pointing normal is $$n := \left(-\frac{1}{\sqrt{1+p^4}}, -\frac{p^2}{\sqrt{1+p^4}}\right) \tag1$$ The center of the circle of radius $r$ tangent at $P$ will have center $K := P + r n$. For this circle to also be tangent to the $x$-axis, we require the $y$-coordinate to be $r$; solving for $r$ gives $$r = \frac{s}{p}\left(1+p^4-p^2\sqrt{1+p^4}\right)\tag2$$ so that the center is $$K = \left(\frac{s}{p}\left(2p^2-\sqrt{1+p^4}\right), \frac{s}{p}\left(1+p^4-p^2\sqrt{1+p^4}\right)\right) \tag3$$ (Sanity check: When $p=1$ and $s=1$, we have $K = (2-\sqrt{2},2-\sqrt{2})$, the correct center for the first circle.)

Now, if the circles for parameters $p$ and $q$ are tangent, then we have $$\left|K_p K_q\right|^2 = (r_p+r_q)^2 \tag4$$ This expands to a messy equation in $p$, $q$, $\sqrt{1+p^4}$, $\sqrt{1+q^4}$. A couple of rounds of squaring eliminates the roots, leaving a messier polynomial equation in $p$ and $q$: $$\begin{align} 0 = p^4 &- 12 p^3 q - 16 p^7 q + 38 p^2 q^2 + 76 p^6 q^2 + 64 p^{10} q^2 - 12 p q^3 - 176 p^5 q^3 \\ &- 112 p^9 q^3 + q^4 + 40 p^4 q^4 + 86 p^8 q^4 - 128 p^{12} q^4 - 176 p^3 q^5 - 56 p^7 q^5 \\ &+ 16 p^{11} q^5 + 76 p^2 q^6 + 516 p^6 q^6 + 172 p^{10} q^6 + 64 p^{14} q^6 - 16 p q^7 \\ &- 56 p^5 q^7 + 32 p^9 q^7 - 400 p^{13} q^7 + 86 p^4 q^8 + 392 p^8 q^8 + 625 p^{12} q^8 \\ &- 112 p^3 q^9 + 32 p^7 q^9 - 924 p^{11} q^9 + 64 p^2 q^{10} + 172 p^6 q^{10} + 1414 p^{10} q^{10}\\ &+ 64 p^{14} q^{10} + 16 p^5 q^{11} - 924 p^9 q^{11} - 400 p^{13} q^{11} - 128 p^4 q^{12} + 625 p^8 q^{12} \\ &+ 672 p^{12} q^{12} - 400 p^7 q^{13} - 400 p^{11} q^{13} + 64 p^6 q^{14} + 64 p^{10} q^{14} \end{align} \tag5$$ So, the general form is, to say the least, difficult to extract. However, for just the second circle, we can substitute $p=1$ into $(4)$ to get a degree-$7$ polynomial in $q$: $$\begin{align} 0 = 8 p^7 (2 - \sqrt2) &- p^6(73 - 48 \sqrt2 ) + 2 p^5 (23 - 16 \sqrt2) - p^4 ( 63 - 40 \sqrt2) \\ &+ 4 p^3 (5 - 4 \sqrt2)- 5p^2 (11 - 8 \sqrt2)+ 2p (7 - 4 \sqrt2) -1 \end{align} \tag6$$ Solving numerically, the only viable root is $q = 1.5831\ldots$, which corresponds to a radius of $0.3274\ldots$. (This result is consistent with a GeoGebra sketch.)

Consider the hyperbola equation $pq=1$. Its gradient at $\left(p,q\right)$ is $\left(q,p\right)$. If $r$ is the radius of the circle tangent at $\left(p,q\right)$ then its center is at $\left(p,q\right)-r\frac{\left(q,p\right)} {\left\Vert q,p\right\Vert}$.

The same circle is tangent at the $x$-axis when the radius $r$ is the same as the $y$-coordinate of its center. This happens when $$r=q-r\frac{p} {\left\Vert q,p\right\Vert}$$ so that $$r=\frac q {1+p/\left\Vert q,p\right\Vert}$$ and this gives the curve $$ C\left(p\right) = \left(p-r\frac q{\left\Vert q,p\right\Vert},r\right) $$ with $pq=1$, that contains all the centers of all circles tangent on the hyperbola and the $x$-axis.

For the first circle we have $p=q=1$. Then $r=\frac 1 {1+1/\sqrt2}=2-\sqrt2$.

Put $c_n=\left(p_n,q_n\right)=C\left(p_n\right)$ as the center of the $n$-th circle with radius $r_n$. Suppose $c_n$ and $r_n$ are known. The point $c_{n+1} \in C$ is the center of the $n+1$-th circle tangent to the previous one if $\left\Vert c_{n+1}-c_n\right\Vert = q_n + q_{n+1}$ since the distance between the two centers is the sum of the two radii and this is the sum of the $y$-coordinates of the centers of the circles. It is equivalent to $\left(p_n-p_{n+1}\right)^2=4 q_n q_{n+1}$ and we need $p_{n+1}>p_n$ for our problem.

This gives the conditions that let us calculate the $\left(n+1\right)$-th center and radius.

For $n=1$ we have $p_1=q_1=r_1=2-\sqrt 2 $ and the previous equation becomes $$ \left(p-\frac{p^{-2}}{\sqrt{p^{2}+p^{-2}}+p}-2+\sqrt{2}\right)^{2}=4\left(2-\sqrt{2}\right)\frac{p^{-1}\sqrt{p^{2}+p^{-2}}}{\sqrt{p^{2}+p^{-2}}+p} $$

that gives the solution $p_2\approx 1.58313$ and $r_2\approx0.327489$.

Even though there are many answers, one can consider a trigonometric alternative;

For your question, the following geometry arises,

Where the following relation of

$$\tan{ \pi - \theta} = \frac{d}{dx}\big[\frac{1}{x}\big]$$

yields,

$$\theta = \arctan{\frac{1}{x^2}}, \theta \in [0, \pi]$$

Consider, $\phi = \pi - \theta$ and the isosceles triangle involving $\phi$, the other two angles (of the mentioned isosceles) must be $\frac{\pi - (\pi - \theta)}{2} = \frac{\theta}{2}$ each.

A bit more geometry, reveals:

If the dashed side of the isosceles is $a$, then the following simultaneous equations arise:

$$\begin{cases}a^2 = \frac{1}{t^2\cos^2{\frac{\theta}{2}}} & ,t \text{ is the tangency point } \\ a^2 = 2r^2 - 2r^2\cos{(\pi - \theta)} \end{cases}$$

Solve for $r$, yields $$r = \frac{\sqrt{2}}{2\sqrt{\cos{(\theta)} + 1} \cdot \cos{\big( \frac{\theta}{2}\big) \cdot t}}$$

But $\theta = \arctan{\frac{1}{t^2}}$

Hence, we obtain the radius of the circle in terms of $t$, the intersection point of $y = \frac{1}{x}$ and some $n^{th}$ circle:

$$r(t) = \frac{\sqrt{2}\big(t^4 + 1\big)^\frac{1}{4}}{2t\sqrt{t^2 + \sqrt{t^4 + 1}}\cdot \sin{\big(\frac{\arctan{t^2}}{2} + \frac{\pi}{4}\big)}}$$

(To confirm this works, $r(1) = 2 - \sqrt{2}$ indeed).

From hereon, a method is to continue from @Blue 's answer $\to (4) \dots $

An alternative includes, looking at the radii between $2$ circles in terms of $r_{n}(t)$ and $\theta$

If what's above if true, then (for second circle):

$$\frac{1}{t_2} = \sqrt{r(t_2)^2 - \big(t_2 - 2 + \sqrt{2} - (r(t_2) + r(1))*\cos{(\arctan{t^2})}\big)^2} + r(t_2)$$

for which my CAS is still BuFfErRiNg (self evident). Again re-confirming with @Blue 's result for $t_2 \approx 1.5831$, $r(1.5831) \approx 0.3275$. Moving on, circle three would be disastrous.

For the $i$-th circle, let $(x_i, r_i)$ be the position of the center, and $r_i$ the radius.

For $ i = 1$, $r_1 = 2 - \sqrt{2} , x_1 = 1 - \dfrac{1}{\sqrt{2}} r_1 $

The next circle has $(x_{i+1}, r_{i+1})$ as the center. By distance from the previous circle,

$ (x_i - x_{i+1})^2 + (r_{i+1} - r_i)^2 = (r_{i+1} + r_i)^2$

so,

$ 4 r_{i+1} r_i = (x_i - x_{i+1})^2 \dots (1) $

If the tangency point of the new circle with $f(x)=\dfrac{1}{x}$, is $(t_{i+1}, \dfrac{1}{t_{i+1}})$, then, by distance,

$ (t_{i+1} - x_{i+1})^2 + (\dfrac{1}{t_{i+1}} - r_{i+1})^2 = r_{i+1}^2$

And this simplifes to

$ t_{i+1}^2 (t_{i+1} - x_{i+1})^2 + 1 - 2 r_{i+1} t_{i+1} = 0 \dots (2)$

Using the slope of the tangent,

$ \dfrac{t_{i+1} - x_{i+1}}{\dfrac{1}{t_{i+1}} - r_{i+1}} = \dfrac{1}{t_{i+1}^2} $

And this reduces to

$ t_{i+1}^3 (t_{i+1} - x_{i+1}) = 1 - r_{i+1} t_{i+1} \dots (3)$

The unknowns are: $ r_{i+1}, x_{i+1}, t_{i+1} $

Solving numerically for each $i$, generates the circles.

The figure below shows the result.

COMMENT.- The true question in this post is about the $n^{th}$ radius and this it is not hard to prove that if $(a_n,r_n)$ is the center of the $n^{th}$ circle we have the following: $$r_{2n}=r\left[\frac{(a_{2n}-a_{2n-1})(a_{2n-2}-a_{2n-3})\cdots(a_2-a_1)}{(a_{2n-1}-a_{2n-2})(a_{2n-3}-a_{2n-4})\cdots(a_3-a_2)(a_1-r)}\right]^2\\r_{2n+1}=\frac{1}{4r}\left[\frac{(a_{2n+1}-a_{2n})(a_{2n-1}-a_{2n-2})\cdots(a_3-a_2)(a_1-r)}{2r(a_{2n}-a_{2n-1})(a_{2n-2}-a_{2n-3})\cdots(a_2-a_1)}\right]^2$$ where $r=2-\sqrt2$ which is the radius $r_0$ of the first circle.

In any case, this is an alternative in search towards an explicit formulation (very complicated). Unfortunately calculation of the abscissas $a_i$ of the centers of the circles are not easy to get. However it is possible to step by step to calculated them so getting finitely many values as we could. In order to have something improving this, it is obvious that we must use tangency points of two circles and the other one of tangency with the hyperbola because if not then our circles are not determined. The task is difficult but I guess it is not impossible to solve.

You can use methods in the other answers to get the first few radii, say the first $k$ radii, where $r_k/r_{k-1}\approx1$, then you can easily approximate the other radii by using the following approximation formula, which I prove below:

$$r_n\approx \left({r_k}^{-2}+8(n-k)\right)^{-1/2}$$

The approximation is close even with $k=2$. In the graph below, the red circles' radii were approximated by my approximation formula with $k=2$.

Here is the proof of my approximation formula.

Consider two adjacent circles with radii $R$ and $r$, where $R>r$.

First we show that $\lim\limits_{r/R\to1}\left(\dfrac{1}{r^2}-\dfrac{1}{R^2}\right)=8$

$\lim\limits_{r/R\to1}\left(\dfrac{1}{r^2}-\dfrac{1}{R^2}\right)$
$=\lim\limits_{r/R\to1}\left(\dfrac{1}{r^2}-\dfrac{1}{R^2}\right)\left(\dfrac{CD}{AB}\right)$
$=\lim\limits_{r/R\to1}\left(\dfrac{1}{r^2}-\dfrac{1}{R^2}\right)\left(\dfrac{R+r}{(x\text{-coordinate of }B)-(x\text{-coordinate of }A)}\right)$
$=\lim\limits_{r/R\to1}\left(\dfrac{1}{r^2}-\dfrac{1}{R^2}\right)\left(\dfrac{R+r}{\frac{1}{2r}-\frac{1}{2R}}\right)$
$=\lim\limits_{r/R\to1}\left(\dfrac{R^2-r^2}{r^2R^2}\right)\left(\dfrac{R+r}{\frac{R-r}{2Rr}}\right)$
$=\lim\limits_{r/R\to1}\dfrac{2(R+r)^2}{Rr}$
$=\lim\limits_{r/R\to1}\left(\dfrac{2R}{r}+4+\dfrac{2r}{R}\right)$
$=8$

$\therefore\dfrac{1}{{(r_n)}^2}\approx \dfrac{1}{{(r_{n-1})}^2}+8 \approx \dfrac{1}{{(r_{n-2})}^2}+16 \approx \dfrac{1}{{(r_{n-3})}^2}+24 \approx... \approx \dfrac{1}{{(r_k)}^2}+8(n-k)$

$\therefore r_n\approx \left({r_k}^{-2}+8(n-k)\right)^{-1/2}$

The first circle $C_1$ is well-known. Now, let $C_{n}$, $n\geq 2$ be the circle with center $O_{n}(x_{n},r_{n})$ which is tangent to the circle $C_{n-1}$, $x$-axis and the hyperbola $y=\frac1x$. Let $P_{n}(a_{n},\frac{1}{a_n})$ be the point where $C_n$ is tangent to the hyperbola. From the slope condition $\large\frac{r_{n}-\frac{1}{a_{n}}}{x_{n}-a_{n}}=a_{n}^2$ and the condition $|O_{n}P_{n}^{}|^2=r_{n}^2$ which is $(\frac{1}{a_{n}}-r_{n})^2+(a_{n}-x_{n})^2=r_{n}^2$, we have $x_{n}=2a_{n}-\frac{1}{a_{n}}\sqrt{1+a_{n}^4}$ and $r_{n}=a_{n}^3+\frac{1}{a_{n}}-a_{n}\sqrt{1+a_{n}^4}.$ The condition $|O_{n}O_{n-1}|=r_{n}+r_{n-1}$ gives $(x_{n}-x_{n-1})^2=4r_{n}r_{n-1}.$ Now let $a_{n}=a, x_{n-1}=m$ and $r_{n-1}=n.$ Then we have $$8na^9-(16mn+9)a^8+(8nm^2+24m)a^7-(22m^2+16n^2)a^6+(8m^3+16n)a^5+(6-m^4-16nm)a^4+(8m^2n-8m)a^3+(2m^2-16n^2)a^2+8na-1=0$$ Examples: $x_2≈1.4617, r_2≈0.3275, a_2≈1.5831; x_3≈2.026, r_3≈0.2434, a_3≈2.081; x_4≈2.469, r_4≈0.201, a_4≈2.501$ Approximations: $a_{n}≈a_{n-1}+\frac{1}{a_{n-1}}+\frac{1}{4a_{n-1}^3}; x_{n}≈x_{n-1}++\frac{1}{x_{n-1}}+\frac{1}{4x_{n-1}^3}$ and $r_{n}≈\large\frac{r_{n-1}}{1+4r_{n-1}^2+2r_{n-1}^4}≈r_{n-1}-4r_{n-1}^3+14r_{n-1}^5.$