1 Introduction
A fundamental problem in harmonic analysis is whether $E_{\Lambda }:=\{e^{-2\pi i\langle \lambda , x\rangle }: \lambda \in \Lambda \}$ forms an orthonormal basis for $L^2(\,\mu )$ , the space of all square-integrable functions with respect to a probability measure $\mu $ . A Borel probability measure $\mu $ on ${\mathbb R}^d$ is called a spectral measure if we can find a countable set $\Lambda \subset {\mathbb R}^d$ such that the set of exponential functions $E_{\Lambda }:=\{e^{-2\pi i\langle \lambda , x\rangle }: \lambda \in \Lambda \}$ forms an orthonormal basis for $L^2(\,\mu )$ . If such $\Lambda $ exists, then $\Lambda $ is called a spectrum for $\mu $ .
Spectral theory has been studied extensively since it was initiated by Fuglede [Reference Fuglede11] in 1974. Jorgensen and Pedersen [Reference Jorgensen and Pedersen13] related spectral measures to fractals and gave the first example of a singular, nonatomic, fractal spectral measure. They showed that the one-fourth Cantor measure is a spectral measure, but the one-third Cantor measure is not. Further research on the spectrality and nonspectrality of measures treats self-similar measures (see [Reference An and Wang2] for a recent example), self-affine measures (see [Reference Chen, Dong and Zhang4, Reference Dai, Fu and Yan5]) and Moran measures (see [Reference An, Fu and Lai1]).
A Sierpinski-type measure $\mu _{M,{\mathcal D}}$ is defined by
where $M=(\begin {smallmatrix}b_1 & 0 \\0 & b_2 \\\end {smallmatrix})$ with $b_1,b_2>1$ , is an expanding matrix, and ${\mathcal D}=\{(\begin {smallmatrix}0\\ 0\end {smallmatrix}),(\begin {smallmatrix}1\\0 \end {smallmatrix}),(\begin {smallmatrix}0\\1\end {smallmatrix})\}$ . The Sierpinski-type measure plays an important role in fractal geometry and geometric measure theory (see [Reference Falconer10, Reference Hutchinson12]). In [Reference Deng and Lau8], Deng and Lau considered the special case $b_1=b_2=b$ and proved that $L^2(\,\mu _{M,{\mathcal D}})$ admits an infinite orthogonal set of exponential functions if and only if $b=({p}/{q})^{{1}/{r}}$ for some $p,q,r\in {\mathbb N}$ with $3\mid p$ , and $\mu _{M,{\mathcal D}}$ is a spectral measure if and only if $3\mid b$ . Dai et al. [Reference Dai, Fu and Yan5] generalised the results under the assumption that $b_1\neq b_2$ .
The nonspectral problem on the singular measure $\mu $ may be the start of investigating the completeness of a family of exponential functions in $L^2(\,\mu )$ . For the Sierpinski-type measure defined above, Dutkay and Jorgensen [Reference Dutkay and Jorgensen9] discovered that if $b_1=b_2=b$ and $3\nmid b$ , then there exist at most $3$ mutually orthogonal exponential functions in $L^2(\,\mu _{M,{\mathcal D}})$ . Later, Li [Reference Li14] proved that if $M=(\begin {smallmatrix}a & b \\0 & c \\\end {smallmatrix})$ with $ac\not \in 3{\mathbb Z}$ , then there exist at most $3$ mutually orthogonal exponential functions in $L^2(\,\mu _{M,{\mathcal D}})$ and the number $3$ is best possible. The more general setting with $a,b,c\in {\mathbb R}$ was considered by Chen et al. [Reference Chen, Wang and Zheng3]. Recently, Liu et al. [Reference Liu, Dong and Li15] considered the matrix $M=(\begin {smallmatrix}a & b \\d & c \\ \end {smallmatrix})$ with $ac-bd\not \in 3{\mathbb Z}$ and showed that there exist at most $9$ mutually orthogonal exponential functions in $L^2(\,\mu _{M,{\mathcal D}})$ and the number $9$ is best possible. All of the known results above are concentrated on the situation when a (or $b,c,d$ ) is the rth root of a rational for $r\ge 1$ .
Motivated by the above results, we will study the nonspectrality of the planar Moran–Sierpinski measure $\mu _{M,\{{\mathcal D}_n\}}$ . Let
and
where $\sigma _n,\gamma _n\in \{-1,1\}$ for each $n\ge 1$ . Then there exists a Borel probability measure with compact support defined by the infinite convolution
where $\delta _E= ({1}/{\# E}) \sum _{e\in E}\delta _e$ for any finite set E, $\delta _e$ is the Dirac measure at the point e and the convergence is in the weak sense. The measure $\mu _{M,\{{\mathcal D}_n\}}$ is called a Moran–Sierpinski measure and its support is the Moran set
Throughout the paper, we make the convention that all fractions have the simplest form, that is, for a fraction ${q}/{p}$ , we have $\gcd (p, q) = 1$ . We denote by r the smallest integer such that $( p/q)^r\in \mathbb {Q}$ (for example, for $\rho =( {4}/{9})^{1/4}= ({2}/{3})^{1/2}$ , we take $r = 2$ ).
Our first result is the following theorem.
Theorem 1.1. Let M and ${\mathcal D}_n$ be defined by (1.1) and (1.2) and define the Moran measure $\mu _{M,\{{\mathcal D}_n\}}$ by (1.3). Then $L^2(\,\mu _{M,\{{\mathcal D}_n\}})$ admits an infinite orthonormal set of exponential functions if and only if $\rho =({q}/{p})^{{1}/{r}}$ for some $p,q,r\in {\mathbb N}$ with $3\mid p$ .
The theorem indicates some connections between number theory and spectral theory. We can conclude from Theorem 1.1 that if $\rho \neq ({q}/{p})^{{1}/{r}}$ for any $p,q,r\in {\mathbb N}$ with $3\mid p$ , then any orthogonal set of exponential functions for $L^2(\,\mu _{M,\{{\mathcal D}_n\}})$ is finite. In this case, we want to estimate the number of orthogonal exponential functions in $L^2(\,\mu _{M,\{{\mathcal D}_n\}})$ exactly. Specifically, for $\rho \in (0,1)$ , we will distinguish the following cases:
-
• $\rho =({q}/{p})^{{1}/{r}}$ with $p,q,r\in {\mathbb N}$ and $\gcd (p,3)=1$ ;
-
• $\rho $ does not have the form $({q}/{p})^{{1}/{r}}$ for any $p,q,r\in {\mathbb N}$ .
For simplicity, we call $\Lambda $ an orthogonal set (respectively a maximal orthogonal set) for $\mu _{M,\{{\mathcal D}_n\}}$ if $\{e^{-2\pi i \langle \lambda , x\rangle }: \lambda \in \Lambda \}$ is an orthonormal family (respectively a maximal orthonormal family) for $L^2(\,\mu _{M,\{{\mathcal D}_n\}})$ .
For the first case above, we obtain the following conclusion.
Theorem 1.2. Let $\rho =({q}/{p})^{{1}/{r}}$ for some $p,q,r\in {\mathbb N}$ with $\gcd (p,3)=1$ and let M, ${\mathcal D}_n$ and $\mu _{M,\{{\mathcal D}_n\}}$ be defined by (1.1), (1.2) and (1.3), respectively. If $\Lambda $ is an orthogonal set of $\mu _{M,\{{\mathcal D}_n\}}$ , then the following statements hold:
-
(i) if $\gcd (q,3)=1$ , then $\#\Lambda \leq 3$ , and $3$ is best possible;
-
(ii) if $3\mid q$ , then there may be any number of elements in an orthogonal exponential set in $L^{2}(\,\mu _{M,\{{\mathcal D}_n\}})$ .
If $\rho $ does not have the form $({q}/{p})^{{1}/{r}}$ for any $p,q,r\in {\mathbb N}$ , we introduce the concept of trinomial number (see Definition 4.1) and prove the following theorem.
Theorem 1.3. Let M, ${\mathcal D}_n$ and $\mu _{M,\{{\mathcal D}_n\}}$ be defined by (1.1), (1.2) and (1.3), respectively. Suppose $\Lambda $ is an orthogonal set of $\mu _{M,\{{\mathcal D}_n\}}$ . Suppose furthermore that $\rho $ does not have the form $({q}/{p})^{{1}/{r}}$ for any $p,q,r\in {\mathbb N}$ .
-
(i) If $\rho $ is a trinomial number with degree m, then $\#\Lambda \leq 3^{m+1}$ .
-
(ii) If $\rho $ is not a trinomial number, then $\#\Lambda \leq 3$ and $3$ is best possible.
We organise this paper as follows. In Section 2, we give the proof of Theorem 1.1. In Section 3, we prove Theorem 1.2. Finally, we prove Theorem 1.3 in Section 4.
2 The proof of Theorem 1.1
Let $\mu $ be a Borel probability measure with compact support on ${\mathbb R}^2$ . The Fourier transform of $\mu $ is defined as usual by
for any $\xi \in {\mathbb R}^2$ . We denote the zero set of $\widehat {\mu }$ by $\mathcal {Z}{(\,\widehat {\mu}\kern1.8pt)}$ , that is,
It is easy to show that $\Lambda $ is an orthogonal set for $\mu $ if and only if
As the orthogonality of the set $\Lambda $ is invariant under translations, without loss of generality, we always assume that $0\in \Lambda $ .
By the definition of Fourier transform of $\mu _{M,\{{\mathcal D}_n\}}$ and (1.3), for any $\xi \in {\mathbb R}^2$ ,
Hence,
By a simple calculation,
For convenience, we denote
and
Then (2.2) and (2.3) imply that
For the one-dimensional self-similar measure $\mu _{\rho ,m}$ ,
where $0<\rho <1$ and $m>1$ is an integer, Deng [Reference Deng7] studied when $L^2(\,\mu _{\rho ,m})$ admits an infinite orthogonal set of exponential functions and obtained the following conclusion.
Lemma 2.1 [Reference Deng7].
Let $\mu _{\rho ,\,m}$ be defined by (2.5). If m is a prime, then $L^2(\,\mu _{\rho ,\,m})$ admits an infinite orthogonal set of exponential functions if and only if $\rho =({q}/{p})^{{1}/{r}}$ for $p,q,r\in {\mathbb N}$ with $m\mid p$ .
For any $\xi \in {\mathbb R}^2$ ,
Take $\nu _i=\ast _{j=0}^{\infty }\delta _{M^{-(\kern1.5pt jr+i)}{\mathcal D}_{jr+i}}$ for $1\le i\le r$ . Then
and we have
Moreover,
Proof of Theorem 1.1.
Suppose $\Lambda $ is an infinite orthogonal set of $\mu _{M,\{{\mathcal D}_n\}}$ with $0\in \Lambda $ . Set $\Lambda =(\begin {smallmatrix} \Lambda ^{(1)}\\ \Lambda ^{(2)} \end {smallmatrix})$ , where $\Lambda ^{(1)}$ is the first coordinate of $\Lambda $ and $\Lambda ^{(2)}$ is the second coordinate. By the orthogonality of $\Lambda $ , we have $(\Lambda -\Lambda )\setminus \{0\}\subset \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ . This, together with (2.3) and (2.4), implies that
It follows that $(\Lambda ^{(i)}-\Lambda ^{(i)})\setminus \{0\}\subset \mathcal {Z}(\,\widehat {\mu }_{\rho ,3})$ for $i=1,2$ . Therefore, $\Lambda ^{(i)}\ (i=1,2)$ is an orthogonal set of $\mu _{\rho ,3}$ .
We now claim that $\Lambda ^{(i)}$ is infinite for $i=1,2$ . It is enough to prove that $\Lambda ^{(1)}$ is infinite, since the proof for $\Lambda ^{(2)}$ is similar. Suppose to the contrary that $\Lambda ^{(1)}$ is finite. By the pigeonhole principle, there exist two distinct elements $\lambda ,\lambda '\in \Lambda $ with $\lambda =(\begin {smallmatrix} \lambda _1\\ \lambda _2 \end {smallmatrix}),\,\lambda '=(\begin {smallmatrix} \lambda ^{\prime }_1\\ \lambda ^{\prime }_2 \end {smallmatrix})$ , such that $\lambda _1=\lambda ^{\prime }_1$ . Then
This is a contradiction. Hence, the claim follows and we conclude that $\Lambda ^{(i)}\,(i=1,2)$ is an infinite orthogonal set of $\mu _{\rho ,3}$ . By Lemma 2.1, $\rho =({q}/{p})^{{1}/{r}}$ for some $p,q,r\in {\mathbb N}$ with $3\mid p$ .
For the converse, suppose that $\rho =({q}/{p})^{{1}/{r}}$ for some $p,q,r\in {\mathbb N}$ with $3\mid p$ . Fix $i\in \{1,2,\ldots ,r\}$ . By the pigeonhole principle, there exists an infinite set $\mathcal {T}$ such that for any distinct $j,j'\in \mathcal {T}$ , we have ${\mathcal D}_{jr+i}={\mathcal D}_{j'r+i}$ . Without loss of generality, assume that ${\mathcal D}_{jr+i}=\{(\begin {smallmatrix}0 \\0 \end {smallmatrix}),\,(\begin {smallmatrix}1\\0\end {smallmatrix}),(\begin {smallmatrix}0 \\1 \end {smallmatrix})\}$ for $j\in \mathcal {T}$ . Set
It is clear that $\Lambda \setminus \{0\}\subset \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ and $\Lambda $ is an infinite set. Now, it is enough to prove that $\Lambda $ is an orthogonal set of $\mu _{M,\{{\mathcal D}_n\}}$ . For any distinct $\lambda _1,\lambda _2\in \Lambda $ , we can write
with $j_1>j_2$ . Then
Since $\gcd (3,q)=1$ , we have $(p^{\,j_1-j_2}q^{\,j_2}-q^{\,j_2})a\in \mathcal {Z}(\widehat {\delta }_{{\mathcal D}_{{j_2}r+i}})$ , and thus $\lambda _1-\lambda _2\in \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ . Hence,
Therefore, $\Lambda $ is an infinite orthogonal set of $\mu _{M,\{{\mathcal D}_n\}}$ .
3 The proof of Theorem 1.2
We now turn to Theorem 1.2. To prove it, we need the following lemmas.
Lemma 3.1 [Reference Deng and Lau8].
Suppose that $b\in {\mathbb R}$ admits a minimal integer polynomial $qx^{r}-p$ ( $r>1$ ) and satisfies $a_{1}b^{l}+a_{2}b^{m}=a_{3}b^{n}$ , where $a_{1},a_{2},a_{3}\in \mathbb {Z}\setminus \{ 0\} $ and $l,m,n$ are nonnegative integers. Then $l\equiv m\equiv n\pmod r$ .
Lemma 3.2. If $b\in {\mathbb R}$ has a minimal integer polynomial $qx^{r}-p$ with $r>1$ and satisfies
where $(\begin {smallmatrix} a_{i1}\\ a_{i2}\end {smallmatrix})\in {\mathcal A}_1\cup {\mathcal A}_2\cup {\mathcal A}_3\cup {\mathcal A}_4$ for $1\leq i\leq 3$ and $n_1,n_2,n_3$ are nonnegative integers, then $n_1\equiv n_2\equiv n_3 \pmod r$ .
Proof. From (3.1),
It follows from (2.3) that $3a_{ij}\neq 0$ for all $i=1,2,3$ and $j=1,2$ . Applying Lemma 3.1, we have $n_1\equiv n_2\equiv n_3 \pmod r$ .
Proof of Theorem 1.2.
(i) Assume for contradiction’s sake that $\#\Lambda>3$ . Let $\Lambda =\{0,\lambda _1,\lambda _2,\lambda _3\}$ be an orthogonal set for $\mu _{M,\{{\mathcal D}_n\}}$ . We claim that
for some i with $1\leq i \leq r$ . Indeed, for any two distinct $\lambda _i,\lambda _j\in \Lambda $ , we can write
where $n_i,n_j\geq 1$ and $(\begin {smallmatrix} \lambda _{i1}\\ \lambda _{i2}\end {smallmatrix}),(\begin {smallmatrix} \lambda _{j1}\\ \lambda _{j2}\end {smallmatrix})\in {\mathcal A}_1\cup {\mathcal A}_2\cup {\mathcal A}_3\cup {\mathcal A}_4$ . From the orthogonality of $\Lambda $ , there exists $\rho ^{-m}(\begin {smallmatrix} \lambda _{1}\\ \lambda _{2}\end {smallmatrix})$ with $m\geq 1$ and $(\begin {smallmatrix} \lambda _{1}\\ \lambda _{2}\end {smallmatrix}) \in {\mathcal A}_1\cup {\mathcal A}_2\cup {\mathcal A}_3\cup {\mathcal A}_4$ such that
By Lemma 3.2, $n_i\equiv n_j\equiv m\pmod r$ . Therefore, $\lambda _i,\lambda _j,\lambda _i-\lambda _j\in \mathcal {Z}(\,\widehat {\nu }_i)$ for some i with $1\leq i \leq r$ and the claim follows.
It follows from the above claim that
for some $i\in \{1,2,\ldots ,r\}$ . This together with (2.6) implies that we can rewrite $\lambda _k$ as
where $n_k\geq 0$ and $(\begin {smallmatrix}\lambda _{k1}\\ \lambda _{k2} \end {smallmatrix}) \in 3({\mathcal A}_1\cup {\mathcal A}_2\cup {\mathcal A}_3\cup {\mathcal A}_4)$ for $k=1,2,3$ . Introduce $N=\max \{n_k:k=1,2,3\}$ . Then
Set
Since $3\nmid p^{n_k}q^{N-n_k}\lambda _{k1}$ for $k=1,2,3$ , by the pigeonhole principle, there exist $j\neq l\in \{1,2,3\}$ such that
That is, $3\mid (p^{n_j}q^{N-n_j}\lambda _{j1}-p^{n_l}q^{N-n_l}\lambda _{l1})$ . Then
which contradicts (3.2). Hence, $\#\Lambda \leq 3$ .
Next, we construct an appropriate orthogonal set to show that $3$ is best possible. Let
for some $j\geq 0$ and $i\in \{1,2,\ldots ,r\}$ . It is obvious that $(\Lambda _0-\Lambda _0)\setminus \{0\}\subset \mathcal {Z}(\,\widehat {\nu }_i)$ and $\#\Lambda =3$ . Hence, the number $3$ is best possible.
(ii) For any $n\ge 1$ , either $\mathcal {Z}(\widehat {\delta }_{{\mathcal D}_n})={\mathcal A}_1\cup {\mathcal A}_{2}$ or $\mathcal {Z}(\widehat {\delta }_{{\mathcal D}_n})={\mathcal A}_{3}\cup {\mathcal A}_{4}$ . Applying the pigeonhole principle, there exists an infinite set $\mathcal {T}$ and $k\in \{1,3\}$ such that $\mathcal {Z}(\widehat {\delta }_{{\mathcal D}_n})= {\mathcal A}_k\cup {\mathcal A}_{k+1}$ for all $n\in \mathcal {T}$ . Without loss of generality, we assume that $\mathcal {Z}(\widehat {\delta }_{{\mathcal D}_n})= {\mathcal A}_1\cup {\mathcal A}_{2}$ for all $n\in \mathcal {T}$ . By the pigeonhole principle again, there exist $i\in \{1,2,\ldots ,r\}$ and an infinite set $\mathcal {T}'$ such that $\mathcal {Z}(\widehat {\delta }_{{\mathcal D}_{jr+i}})={\mathcal A}_1\cup {\mathcal A}_{2}$ for all $j\in \mathcal {T}'$ . Set $\mathcal {T}'=\{\kern1.5pt j_n\}_{n=1}^{\infty }$ with $j_1<j_2<\cdots $ . For any $N\geq 1$ , define
As $3\nmid p$ , we have $\Lambda _N\setminus \{0\}\subset \mathcal {Z}(\widehat {\nu }_{i})\subset \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ by (2.6) and (2.7). For any distinct elements $\lambda _n,\lambda _m\in \Lambda _N$ , we can write
with $n<m$ . Then
Since $3\nmid p$ and $3\mid q$ ,
It follows that $\lambda _n-\lambda _m\in \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ . Hence, $\Lambda _N$ is an orthogonal set of $\mu _{M,\,\{{\mathcal D}_n\}}$ . By the arbitrariness of N, the proof is completed.
4 The proof of Theorem 1.3
In this section, we prove Theorem 1.3. We begin with the important concept of a trinomial number.
Definition 4.1. We say that $\rho \in (0,1)$ is a trinomial number if there exist $\alpha ,\beta ,\gamma \in {\mathbb Z}\setminus 3{\mathbb Z}$ and $m,n\in {\mathbb N}$ with $m>n>0$ , such that
The smallest m satisfying (4.1) is called the degree of the trinomial number $\rho $ .
The next lemma is inspired by [Reference Dai and Zhu6, Lemma 4.1].
Lemma 4.2. Let $P(x)$ be an integer polynomial with all its coefficients divisible by 3 except for one, and let $Q(x)$ be an integer polynomial whose first and last coefficients are not divisible by 3. Then $P(x)$ and $Q(x)$ are coprime.
Proof. Note that $Q(x)$ is not monomial. We divide the proof into two cases.
Case I: $Q(x)$ is irreducible in ${\mathbb Q}[x]$ . We argue by contradiction. Suppose that $P(x)$ and $Q(x)$ are not coprime. Then there exists $H(x)\in {\mathbb Z}[x]$ , such that ${P(x)=H(x)Q(x)$} . Set ${\mathbb Z}_3={\mathbb Z}\setminus 3{\mathbb Z}$ . Denote by $P'(x),Q'(x),H'(x)\in {\mathbb Z}_3[x]$ the polynomials whose respective coefficients are congruent to the coefficients of $P(x),Q(x),H(x)$ modulo $3$ . Then
The assumption of Lemma 4.2 implies that $P'(x)$ is a monomial but $Q'(x)$ is not, which gives a contradiction. Thus, $P(x)$ and $Q(x)$ are coprime.
Case II: $Q(x)$ is reducible. Then we can write $Q(x)=Q_1(x)Q_2(x)\cdots Q_k(x)$ , where each $Q_i(x)$ is irreducible for $1\leq i\leq k$ . By Case I, $P(x)$ and $Q_i(x)$ are coprime for ${i=1,2,\ldots ,k}$ . Therefore, $P(x)$ and $Q(x)$ are coprime.
Now we have all ingredients for the proof of Theorem 1.3.
Proof of Theorem 1.3.
(i) Assume that $\#\Lambda>3^{m+1}$ . Let $\Lambda =\{0,\lambda _1,\ldots ,\lambda _{3^{m+1}}\}$ be an orthogonal set for $\mu _{M,\{{\mathcal D}_n\}}$ . Since $\rho $ is a trinomial number with degree m, there exist $\alpha ,\beta ,\gamma \in {\mathbb Z}\setminus 3{\mathbb Z}$ and $m,n\in {\mathbb N}$ with $m>n>0$ , such that
Now we claim that for any $k\in {\mathbb N}$ , there exist $\{c_{k,0},c_{k,1},\ldots ,c_{k,m-1}\}\subset {\mathbb Z}$ such that
where $c_{k,s}\in {\mathbb Z}\setminus 3{\mathbb Z}\cup \{0\}$ . Indeed, if $k<m$ , then (4.3) obviously holds. If $k\geq m$ , we denote $k=s_1m+t_1\,(0 \leq t_1\leq m-1)$ , then (4.2) implies that
If $s_1n+t_1<m$ , then (4.3) follows. If $s_1n+t_1\geq m$ , we set $s_1n+t_1=s_2m+t_2$ with $0 \leq t_2\leq m-1)$ . Then
After finitely many steps, we reach $r\in {\mathbb N}$ , such that $s_r n + t_r<m$ . Then the claim follows. By the pigeonhole principle, there exist mutually different $k_i,k_j,k_l\in \{1,\ldots ,3^{m+1}\}$ , such that
for $s=0,1,\ldots ,m-1$ . Denote the corresponding $\lambda _i,\lambda _j,\lambda _l$ by
where $a_i,a_j,a_l\in \{(\begin {smallmatrix}1\\2\end {smallmatrix}), (\begin {smallmatrix}2\\ 1 \end {smallmatrix}), (\begin {smallmatrix}1\\1\end {smallmatrix}), (\begin {smallmatrix}2\\2\end {smallmatrix})\}+3{\mathbb Z}^2$ . Let $N=\max \{k_i,k_j,k_l\}$ . Denote by $a_i^{(1)},a_j^{(1)},a_l^{(1)}$ the first coordinates of $a_i,a_j,a_l$ , respectively. Applying the pigeonhole principle again, there exist two elements of $\{\lambda _i,\lambda _j,\lambda _l\}$ (which we might as well denote as $\lambda _i,\lambda _j$ ), such that
By the orthogonality of $\Lambda $ , there exists $\tfrac 13\rho ^{-k_{ij}}a_{ij}\in \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ with $k_{ij}\ge 1$ and $a_{ij}\in \{(\begin {smallmatrix}1\\2\end {smallmatrix}), (\begin {smallmatrix}2\\ 1 \end {smallmatrix}), (\begin {smallmatrix}1\\1\end {smallmatrix}), (\begin {smallmatrix}2\\2\end {smallmatrix})\}+3{\mathbb Z}^2$ , such that
Applying (4.3), we have
Combining this with (4.6) gives
Define
From (4.4) and (4.5), $c_{k_i,\,s}\alpha ^{N-k_i}a_i^{(1)}-c_{k_j,\,s}\alpha ^{N-k_j}a_j^{(1)}\in 3{\mathbb Z}$ for $0\le s\le m-1$ . Then by Lemma 4.2, $P(x)$ and $Q(x)$ are coprime. However, $P(\kern1.2pt\rho ^{-1})=0$ and $Q(\kern1.2pt \rho ^{-1})=0$ . This gives a contradiction. Hence, $\#\Lambda \leq 3^{m+1}$ .
(ii) We argue by contradiction. Suppose that $\#\Lambda \ge 4$ , and let $\Lambda =\{0,\lambda _1,\lambda _2,\lambda _3\}$ be an orthogonal set for $\mu _{M,\{{\mathcal D}_n\}}$ . By (2.1) and (2.4), we can write
for $i=1,2,3$ . Applying the pigeonhole principle, there exist distinct $\lambda _i,\,\lambda _j\in \Lambda $ , such that $a_i^{(1)}\equiv a_j^{(1)}\pmod 3$ and $k_i\ge k_j$ . By the orthogonality of $\Lambda $ , there exist $\tfrac 13\rho ^{-k_{ij}}a_{ij}\in \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ with $a_{ij}\in \{(\begin {smallmatrix}1\\2\end {smallmatrix}), (\begin {smallmatrix}2\\ 1 \end {smallmatrix}), (\begin {smallmatrix}1\\1\end {smallmatrix}), (\begin {smallmatrix}2\\2\end {smallmatrix})\}+3{\mathbb Z}^2$ , such that
It follows that
Now we distinguish three cases.
Case I: $k_i,k_j,k_{ij}$ are mutually different. We might as well assume that $k_i>k_j>k_{ij}$ since the proof in the other cases is similar. Then (4.7) implies that
This means that $\rho $ is a trinomial number, which is a contradiction.
Case II: Only two elements of $\{k_i,k_j,k_{ij}\}$ are equal. We might as well assume that $k_i>k_j=k_{ij}$ . From (4.7),
It follows that $\rho =({a_i^{(1)}}/{(a_j^{(1)}+a_{ij}^{(1)})})^{{1}/{(k_i-k_{ij})}}$ , which contradicts the fact that $\rho $ is not of the form $({q}/{p})^{{1}/{r}}$ for $p,q,r\in {\mathbb N}$ .
Case III: $k_i=k_j=k_{ij}$ . Then $a_i^{(1)}-a_j^{(1)}=a_{ij}^{(1)}$ . As $a_i^{(1)}\equiv a_j^{(1)}\pmod 3$ , we have $a_{ij}^{(1)}\equiv ~0\pmod 3$ . This is impossible. Hence, $\#\Lambda \leq 3$ .
Finally, let
It is easy to verify that $(\Lambda _0-\Lambda _0)\setminus \{0\} \subset \mathcal {Z}(\,\widehat {\mu }_{M,\{{\mathcal D}_n\}})$ and $\#\Lambda _0=3$ . Therefore, the number $3$ is best possible.