For any non-empty subset $\mathcal I$ of the natural numbers, let $\Lambda_{\mathcal I}$ denote those numbers in the unit interval whose continued fraction digits all lie in $\mathcal I$. Define the corresponding transfer operator

$$ \mathcal L_{\mathcal I, \beta} f(z) = \sum_{n \in \mathcal I} \left( \frac{1}{n + z} \right)^{2 \beta} f\left( \frac{1}{n + z} \right) $$

for $\text{Re} (\beta) > \max (0, \theta_{\mathcal I})$, where $\text{Re} (\beta) = \theta_{\mathcal I}$ is the abscissa of convergence of the series $\sum_{n \in \mathcal I}n^{-2 \beta}$.

When acting on a certain Hilbert space $\mathcal H_{\mathcal I, \beta}$, we show that the operator $\mathcal L_{\mathcal I, \beta}$ is conjugate to an integral operator $\mathcal K_{\mathcal I, \beta}$. If furthermore $\beta$ is real, then $\mathcal K_{\mathcal I, \beta}$ is selfadjoint, so that $\mathcal L_{\mathcal I, \beta} : \mathcal H_{\mathcal I, \beta} \to \mathcal H_{\mathcal I, \beta}$ has purely real spectrum. It is proved that $\mathcal L_{\mathcal I, \beta}$ also has purely real spectrum when acting on various Hilbert or Banach spaces of holomorphic functions, on the nuclear space $C^\omega [0, 1]$, and on the Fréchet space $C^\infty [0, 1]$.

The analytic properties of the map $\beta \mapsto \mathcal L_{\mathcal I, \beta}$ are investigated. For certain alphabets $\mathcal I$ of an arithmetic nature (for example, $\mathcal I = \{\text{primes}\}$, $\mathcal I = \{\text{squares}\}$, $\mathcal I$ an arithmetic progression, $\mathcal I$ the set of sums of two squares) it is shown that $\beta \mapsto \mathcal L_{\mathcal I, \beta}$ admits an analytic continuation beyond the half-plane $\text{Re} (\beta) > \theta_{\mathcal I}$.