2.1 Local case

We assume first that $F$ is a local field and $E/F$ is a separable quadratic field extension. We let $E_0$ denote the $F$-subspace of trace $0$ elements in $E$ and let $E_1 \subset E^{\times }$ denote the subgroup of norm 1 elements. Fix a non-trivial additive character $\psi$ of $F$ and let $\sigma \in {\rm Gal}(E/F)$ be the non-trivial automorphism of $E/F$.

For a skew-Hermitian space $V$ over $E$ of dimension $n$, we recall that

\begin{align*} \det(V) \in \begin{cases} F^{\times}/ N_{E/F}(E^{\times}), & \text{if $n$ is even;} \\ E_0^{\times}/N_{E/F}(E^{\times}), & \text{if $n$ is odd.} \end{cases} \end{align*}

If $F$ is non-archimedean, there are precisely two skew-Hermitian spaces of dimension $n$, distinguished by their determinants. When $F$ is archimedean, there are many more skew-Hermitian spaces, distinguished by their signatures.

Without loss of generality, we may assume that all these skew-Hermitian spaces (of a given dimension) have the same underlying vector space $V$ over $E$, equipped with non-isomorphic skew-Hermitian forms. Thus the unitary groups ${\rm U}(V) \subset {\rm GL}(V)={\rm Aut}_E(V) = {\rm GL}_n(E)$ are all subgroups of a fixed ambient group ${\rm GL}_n(E)$.

For each skew-Hermitian space $V$ over $E$ and a conjugate-symplectic character $\mu$ of $E^{\times }$, we have the associated Weil representation $\omega _{V, \psi, \mu }$ of ${\rm U}(V)$. Now for an irreducible representation $\Pi$ of ${\rm GL}(V) \cong {\rm GL}_n(E)$, we consider the Hom space

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi, \mu}). \end{align*}

We now state our main local conjecture in this case.

Conjecture 2.1 (i) For any $\Pi \in {\rm Irr}({\rm GL}(V))$,

\begin{align*} \dim {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi, \mu}) \leq 1. \end{align*}

(ii) If $\Pi \in {\rm Irr}({\rm GL}(V))$ is generic, then

\begin{align*} \sum_V \dim {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) = 1. \end{align*}

where the sum is over the equivalence classes of skew-Hermitian structures on $V$.

(iii) For generic $\Pi \in {\rm Irr}({\rm GL}(V))$, the unique skew-Hermitian space $V$ which gives a non-zero contribution to the above sum satisfies

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, \Pi \times {}^\sigma\Pi ^\vee {\times} \mu^{-1}, \psi_E) \cdot \omega_{\Pi}(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

where ${}^\sigma \Pi ^{\vee }$ is the conjugate-dual representation of $\Pi$ and $\omega _{\Pi }$ is the central character of $\Pi$.

As noted in the introduction, the ratio of the two sides of condition (iii) is a priori $\pm 1$. When $F$ is non-archimedean, condition (iii) in the conjecture uniquely determines the summand with non-zero contribution to the sum in condition (ii). When $F = \mathbb {R}$ and $E = \mathbb {C}$, one needs to be more specific about the $V$ which gives non-zero contribution. We shall consider this archimedean case in greater detail in § 6. Note that if we define the discriminant of $V$ by

\begin{align*} {\rm disc}(V) = (-1)^{n(n-1)/2} \cdot \det(V), \end{align*}

then the formula in condition (iii) can be expressed more succinctly as

\begin{align*} \mu({\rm disc}(V)) = \epsilon( 1/2, \Pi \times {}^\sigma \Pi^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\Pi}(-1)^n, \end{align*}

taking note of the fact that $\mu (-1) = \omega _{E/F}(-1)$. We shall provide some evidence for this conjecture in the next two sections, verifying it for $\dim V \leq 2$ and for unitary principal series representations of ${\rm GL}(V)$ for $V$ of arbitrary dimension over $E$.

In the above formulation, the conjecture does not require the local Langlands correspondence, as the local root number in condition (iii) can be interpreted as the Rankin–Selberg local root number defined by Jacquet, Piatetski-Shapiro, and Shalika [Reference Jacquet, Piatetski-Shapiro and ShalikaJPSS83].

Let $M$ denote the Langlands parameter of $\Pi$, so that $M$ is an $n$-dimensional representation of the Weil–Deligne group $WD_E$ of $E$ with $\det (M)$ corresponding to the central character $\omega _{\Pi }$ under the local class field theory. We have noted in the introduction that $M \otimes {}^\sigma \!M ^\vee \otimes \mu ^{-1}$ is a conjugate-symplectic representation of $WD_E$. Then Conjecture 2.1(iii) can be written as

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \det(M)(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}. \end{align*}

Note that, for $e \in E_0^{\times }$,

\begin{align*} \det(M \otimes {}^\sigma \!M ^\vee)(e) = \det(M)(e)^n / \det(M)(e^{\sigma})^n = \det(M)(-1)^n, \end{align*}

and

\begin{align*} \omega_{E/F}(-1) = \omega_{K/F}(e^2) = (e^2, e^2) \quad \text{(Hilbert symbol)}. \end{align*}

Hence, the above identity can be expressed as (for $E=K$)

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \det(M \otimes {}^\sigma \!M ^\vee)(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2}, \end{align*}

and it is this last statement that generalizes well when we deal with the general case (where $E \ne K$) later.

2.2 Global case

Consider now the case when $E/ F$ is a quadratic extension of global fields with adele rings $\mathbb {A}_E$ and $\mathbb {A}_F$. Fix a non-trivial additive character $\psi$ of $F \backslash \mathbb {A}_F$. We shall consider all skew-Hermitian structures on a vector space $V$ of dimension $n$ over $E$.

Let $\Pi \cong \bigotimes _v \Pi _v$ be a cuspidal automorphic representation of ${\rm GL}(V)(\mathbb {A}_F) = {\rm GL}(V\otimes _F\mathbb {A}_F) = {\rm GL}(V\otimes _E \mathbb {A}_E)$ so that $\Pi _v$ are generic representations for each place $v$ of $E$. For a conjugate-symplectic Hecke character $\mu$ of $\mathbb {A}_E^{\times }$, we may consider the automorphic Weil representation $\omega _{V,\psi, \mu }$ of ${\rm U}(V)(\mathbb {A}_F)$ (see [Reference Gan, Gross and PrasadGGP12b]). Now we consider the global period integral

\begin{align*} \mathcal{P}_V : \Pi\otimes \overline{\omega_{V,\psi, \mu}} \longrightarrow \mathbb{C} \end{align*}

defined by

\begin{align*} \mathcal{P}_V(f, \phi) = \int_{[{\rm U}(V)]} f(g) \cdot \overline{\phi(g)} \, dg \quad \text{for $f \in \Pi$ and $\phi \in \omega_{V,\psi, \mu}$,} \end{align*}

where we have written $[{\rm U}(V)]$ for the automorphic quotient ${\rm U}(V)(F) \backslash {\rm U}(V)(\mathbb {A}_F)$ with $dg$ the Tamagawa measure on it.

Globally, we are interested in characterizing the non-vanishing of this period integral. Our global conjecture is the following.

Observe that if we are given a collection of local skew-Hermitian spaces $\{V_v\}$ for all places $v$ of $F$ (of a fixed dimension $n \geq 1$), then the adelic skew-Hermitian space $\bigotimes _v V_v$ is coherent over $F$, i.e. the family of local skew-Hermitian spaces $V_v$ comes from a global skew-Hermitian space $V$, if and only if

\begin{align*} \prod_v \mu_v\big( \det(V_v) \big) = 1. \end{align*}

Therefore, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$, if the local skew-Hermitian spaces $\{V_v\}$ are those for which ${\rm Hom}_{{\rm U}(V_v)}( \Pi _v, \omega _{V_v, \psi _v, \mu _v}) \ne 0$ for all places $v$ of $F$, then part (iii) of our local Conjecture 2.1 implies that this collection of local skew-Hermitian spaces $\{V_v\}$ is coherent over $F$ if and only if

\begin{align*} \epsilon(1/2, \Pi \times {}^\sigma\Pi^\vee \times \mu^{-1}) = 1. \end{align*}

Therefore, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$ for which the global period integral on $[U(V)]$ is non-zero (hence, ${\rm Hom}_{{\rm U}(V_v)}( \Pi _v, \omega _{V_v, \psi _v, \mu _v}) \ne 0$ for all places $v$ of $F$), then $\epsilon (1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) = 1$. Thus, a necessary condition for the non-vanishing of $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1})$ is satisfied if the global period integral on $[U(V)]$ is non-zero. Conversely, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$ for which $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) \not = 0$ and hence $\epsilon (1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) = 1$, we have a global skew-Hermitian space $V$, unique up to isomorphism, for which Conjecture 2.2 implies non-vanishing of period integral on $[{\rm U}(V)]$.

2.3 A refined global conjecture

Not surprisingly, one expects to be able to refine the above global conjecture to a precise formula relating the global period integral to the central $L$-value.

For $\Pi \cong \bigotimes _v \Pi _v$, a cuspidal automorphic representation of ${\rm GL}(V)(\mathbb {A}_F) = {\rm GL}(V\otimes _F\mathbb {A}_F) = {\rm GL}(V\otimes _E \mathbb {A}_E)$, and $\omega _{V, \psi, \mu } \cong \bigotimes _v \omega _{V_v, \psi _v, \mu _v}$, the Weil representation of ${\rm U}(V)(\mathbb {A}_F)$, $f_v,f'_v \in \Pi _v$ and $\phi _v,\phi '_v \in \omega _{V_v, \psi _v, \mu _v}$, we may consider the following integral of matrix coefficients for each place $v$ of $F$:

(2.3)\begin{equation} \mathcal{I}_v ( f_v, f'_v, \phi_v, \phi'_v) := \int_{{\rm U}(V)(F_v)} \langle g_v \cdot f_v, f'_v \rangle \cdot \overline{ \langle g_v \cdot \phi_v, \phi'_v\rangle} \, dg_v. \end{equation}

As in [Reference XueXue16], it is not hard to see that if $\Pi _v$ is tempered, this integral is absolutely convergent, so that it defines a ${\rm U}(V_v) \times {\rm U}(V_v)$-equivariant linear functional

\begin{align*} \mathcal{I}_v : \Pi_v \otimes \overline{\Pi}_v \otimes \overline{\omega_{V_v, \psi_v, \mu_v}} \otimes \omega_{V_v, \psi_v,\mu_v} \longrightarrow \mathbb{C}. \end{align*}

Now one would like to:

• • show that $\mathcal {I}_v$ is non-zero if and only if ${\rm Hom}_{{\rm U}(V)(F_v)}( \Pi _v , \omega _{V_v, \psi _v,\mu _v}) \ne 0$;

• • compute this integral at almost all places $v$ of $F$ where all data involved are unramified.

Without having done this work, we may nonetheless venture a guess here, in analogy with the original GGP case [Reference XueXue16].

Conjecture 2.4 Suppose that:

• • $E_v/F_v$ is an unramified quadratic extension of residue characteristic not $2$ and $\psi _v$ has conductor $\mathcal {O}_{F_v}$;

• • $\mu _v$ is unramified;

• • $V_v$ contains a selfdual lattice $\Lambda _v$ whose stabilizer in ${\rm U}(V_v)$ is a hyperspecial maximal compact subgroup $K_v$, contained in $\tilde {K}_v = {\rm GL}(\Lambda _v) \subset {\rm GL}(V_v)$;

• • $dg_v$ is the Haar measure on ${\rm U}(V_v)$ which gives $K_v$ volume $1$;

• • $\Pi _v$ is $\tilde {K}_v$-unramified and $f_v = f'_v$ is a $\tilde {K}_v$-spherical vector of norm 1;

• • $\phi _v = \phi '_v$ is a $K_v$-spherical vector of norm 1 in the Weil representation $\omega _{V_v, \psi _v, \mu _v}$.

Then

\begin{align*} \mathcal{I}_v ( f_v, f'_v, \phi_v, \phi'_v) = \frac{L(1, M_{{\rm GL}(V_v)}^{\vee})}{L(1, M^{\vee}_{{\rm U}(V_v)})} \cdot \frac{ L(1/2, \Pi_v \times {}^\sigma\Pi^\vee_v \times \mu_v^{-1})}{L(1, \Pi_v, {\rm Ad})}, \end{align*}

where

\begin{align*} L(1, M_{{\rm GL}(V_v)}^{\vee}) = \prod_{k=1}^n \zeta_{E_v}(k) \quad \text{and} \quad L(1, M^{\vee}_{{\rm U}(V_v)}) = \prod_{k=1}^n L(k, \omega_{E_v/F_v}^k) \end{align*}

are the values at $s=1$ of the $L$-functions of the dual motives of ${\rm GL}(V)$ and ${\rm U}(V)$, respectively. (One may observe that the expression for $\mathcal {I}_v ( f_v, f'_v, \phi _v, \phi '_v)$ given above implies, in particular, that it is non-zero.)

Given this, it is natural to define a normalized local period integral:

(2.5)\begin{equation} \mathcal{I}_v^{\#} = \frac{L(1, M^{\vee}_{{\rm U}(V_v)})}{L(1, M_{{\rm GL}(V_v)}^{\vee})} \cdot \frac{L(1, \Pi_v, {\rm Ad})}{L(1/2, \Pi_v \times {}^\sigma\!\Pi^\vee_v \times \mu_v^{-1})} \cdot \mathcal{I}_v. \end{equation}

We also note that if $E_v = F_v \times F_v$, the analog of the above conjecture holds, and has already been considered in the original formulation of the refined GGP conjecture for skew-Hermitian spaces in [Reference Gan, Gross and PrasadGGP12b].

Coming back to the global setting, for each of the groups ${\rm GL}(V)$ or ${\rm U}(V)$, we will fix a decomposition of the Tamagawa measures $dg = \prod _v dg_v$, so that for almost all $v$, the local Haar measures $dg_v$ give a hyperspecial maximal compact subgroup volume $1$. We will also fix a decomposition of the global Petersson inner product (defined by integrating over ${\rm GL}_n(E) \backslash {\rm GL}_n({\mathbb{A}}_E)^1$, where ${\rm GL}_n({\mathbb{A}}_E)^1 \,{:=}\, \{ g \in {\rm GL}_n({\mathbb{A}}_E)\ {\rm with}\ |{\rm det}(g)| =1 \} $) as a product of local pairings:

(2.6)\begin{equation} \langle- , - \rangle_{{\rm Pet}} = \prod_v \langle-, - \rangle_v, \end{equation}

and use these $dg_v$ and $\langle -,-\rangle _v$ in the definition of the local period integrals $\mathcal {I}_v$ introduced above. We can now state the following.

Conjecture 2.7 Given a (tempered) cuspidal automorphic representation $\Pi$ of ${\rm GL}(V)$,

\begin{align*} \mathcal{P} \otimes \overline{\mathcal{P}}= \frac{ L(1/2, \Pi \times {}^\sigma\Pi^\vee \times \mu^{-1})}{L(1, M_{{\rm U}(V)}^{\vee})} \cdot \bigg( \frac{L(s, M_{{\rm GL}(V)}^{\vee})}{L(s, \Pi, {\rm Ad})} \bigg)\bigg|_{s=1} \cdot \prod_v \mathcal{I}_v^{\#} . \end{align*}

as linear functionals on $\Pi \otimes \overline {\Pi } \otimes \overline {\omega _{V,\psi, \mu }} \otimes \omega _{V, \psi, \mu }$.

Here, note that $L(s, M_{{\rm GL}(V)}^{\vee })$ and $L(s, \Pi, {\rm Ad})$ both have a simple pole at $s=1$, so that their ratio is holomorphic and non-zero at $s=1$.

2.4 Finite fields

We conclude this section by highlighting the restriction problem for skew-Hermitian spaces over a finite field $F = \mathbb {F}_q$. In the finite field setting, only the case $E = K$ can occur. In this setting, a naive first guess is that for any irreducible generic representation $\Pi$ of ${\rm GL}_n(\mathbb {F}_{q^2})$,

\begin{align*} \dim {\rm Hom}_{{\rm U}_n(\mathbb{F}_q)} (\Pi, \omega) =1, \end{align*}

where $\omega$ is the Weil representation of ${\rm Sp}_{2n}(\mathbb {F}_q)$, restricted to the subgroup ${\rm U}_n(\mathbb {F}_q)$. However, an examination of the case $n=1$ shows that this cannot literally be the case because $\dim \omega = q$ but ${\rm U}_1(\mathbb {F}_q)$ has $q+1$ characters. Indeed, the unique non-trivial quadratic character of ${\rm U}_1(\mathbb {F}_q)$ is missing from $\omega$. Moreover, experience with the usual GGP problem over finite fields shows that the above branching multiplicity can be larger than 1. Nonetheless, the naive expectation should be generically true for cuspidal Deligne–Lusztig representations and it is an interesting question to quantify the extent of its failure.

Over finite fields, we can also consider this restriction problem for symplectic groups. For any irreducible generic representation $\Pi$ of ${\rm Sp}_{2n}(\mathbb {F}_{q^2})$, one would thus like to determine

\begin{align*} \dim {\rm Hom}_{{\rm Sp}_{2n}(\mathbb{F}_q)}(\Pi, \omega) . \end{align*}

It is curious that since the two fold cover of ${\rm Sp}_{2n}(E)$ splits over ${\rm Sp}_{2n}(F)$, there is no analogous problem for non-archimedean local fields. Perhaps, one could go to four fold cover of ${\rm Sp}_{2n}(E)$ (if the 4th roots of unity are there in $E$) to study the analogous branching problem?

A first study of these branching problems over finite fields has been conducted by Nhat Hoang Le.

3.1 Rank-one case

We begin by examining the case when $\dim V =1$, so that ${\rm GL}(V) = E^{\times } \supset {\rm U}(V) = E_1$, where $E_1$ denotes the subgroup of norm one elements. Given a character $\chi$ of $E^{\times }$, we are thus interested in understanding ${\rm Hom}_{E_1}( \chi, \omega _{V,\psi, \mu })$. This is addressed by a theorem of Moen [Reference MoenMoe87] and Rogawski [Reference RogawskiRog92].

Theorem 3.1 If $\chi$ is a character of $E^{\times }$, then

\begin{align*} \dim {\rm Hom}_{E_1}( \chi, \omega_{V,\psi, \mu}) \leq 1 \end{align*}

and equality holds if and only if

\begin{align*} \mu(\det(V)) = \chi(-1) \cdot \epsilon(1/2, \chi^{\sigma}/\chi \cdot \mu^{-1}, \psi_E). \end{align*}

This is precisely what Conjecture 2.1 asserts in the case $\dim V =1$.

3.2 Rank-two case

Suppose now that $\dim V = 2$. Skew-Hermitian spaces of rank two can be described using quaternion $F$-algebras, as we have exploited in [Reference Gan, Gross and PrasadGGP12a]. More precisely, for a quaternion $F$-algebra $B$, fix an $F$-algebra embedding $i: E \hookrightarrow B$ and write $B = E \oplus E \cdot x$ where $x$ is an element of $B$ such that $x e x^{-1} = e^{\sigma }$. Thus, $B$ is a two-dimensional $E$-vector space (by left multiplication), and we may identify ${\rm GL}_E(B)$ with ${\rm GL}_2(E)$ with respect to the basis $\{1, x\}$.

Now fix a trace 0 element $\delta \in E^{\times }$ and set

\begin{align*} \langle b_1, b_2 \rangle = \delta \cdot (\text{projection of $b_1 \cdot \bar{b}_2$ onto $E$}). \end{align*}

Then $\langle -, -\rangle$ is a skew-Hermitian form on $B$; we shall denote this skew-Hermitian space by $V_B$. The isomorphism class of $V_B$ is independent of $x,\delta$, and $V_B$ is split if and only if $B$ is split.

The unitary similitude group ${\rm GU}(V_B) \subset {\rm GL}(V_B) = {\rm GL}_2(E)$ can be described by the isomorphism

\begin{align*} \iota: (B^{\times} \times E^{\times}) / \Delta F^{\times} \stackrel{\cong} {\longrightarrow} {\rm GU}(V_B) \subset {\rm GL}(V_B) \end{align*}

given by sending $(b,e) \in B^{\times } \times E^{\times }$ to the element of ${\rm GL}(V_B)$ whose action on $B$ is

\begin{align*} (b,e): y \mapsto e \cdot y \cdot b^{-1}. \end{align*}

The similitude character is

\begin{align*} {\rm sim}(b, e) = N_{E/F}(e) \cdot N_B(b)^{-1}. \end{align*}

Hence, the unitary group is

\begin{align*} {\rm U}(V_B) \cong \{ (b, e) \in (B^{\times} \times E^{\times}) / \Delta F^{\times} = {\rm GU}(V_B): N_{E/F}(e) = N_B(b) \}. \end{align*}

This is contained in the subgroup

\begin{align*} {\rm GU}(V_B)^+ \cong \{ (b,e) \in (B^{\times} \times E^{\times}) / \Delta F^{\times} ={\rm GU}(V_B): N_B(b) \in N_{E/F}(E^{\times}) \}, \end{align*}

which has index 2 in ${\rm GU}(V_B)$. Moreover, if $Z = E^{\times }$ denotes the center of ${\rm GL}(V_B)$, then

\begin{align*} {\rm GU}(V_B)^+ = Z \cdot {\rm U}(V_B). \end{align*}

Thus, when working with irreducible representations of ${\rm U}(V_B)$, there is no difference in working with ${\rm GU}(V_B)^+$ instead.

Let us explicate the Weil representation of ${\rm U}(V_B)$ in this framework. The Weil representation $\omega _{\psi,\mu, B}$ is reducible but admits a central character decomposition:

\begin{align*} \omega_{\psi, \mu, B} = \bigoplus_{\lambda} \omega_{\psi,\mu, B}[\lambda], \end{align*}

where the sum runs over the characters of $Z({\rm U}(V_B))= E_1$ and each summand is irreducible or $0$. We can describe $\omega _{\psi,\mu, B}[\lambda ]$ in terms of the description of ${\rm U}(V_B)$ given above. More precisely, suppose that $\lambda = \chi |_{E_1}$ for a character $\chi$ of $E^{\times }$. Consider the $L$-parameter

\begin{align*} N = {\rm Ind}_{W_E}^{W_F} (\mu \cdot \chi^{-1}) \quad \text{ of ${\rm GL}_2(F)$,} \end{align*}

and let $\Sigma _{B,N}$ be the associated representation of $B^{\times }$. This gives a representation

\begin{align*} \Sigma_{B,N} \boxtimes \chi \quad \text{ of $B^{\times}\times E^{\times}$}, \end{align*}

which is trivial on $\Delta F^{\times }$, i.e. a representation of ${\rm GU}(V_B)$. This representation of ${\rm GU}(V_B)$ decomposes into the sum of two irreducible summands when restricted to ${\rm GU}(V_B)^+$. One of these summands is the representation $\omega _{\psi,\mu, B} [\chi |_{E_1}]$ whereas the other is $\omega _{\psi ',\mu, B} [\chi |_{E_1}]$, with $\psi '$ in a different $N(E^{\times })$-orbit as $\psi$.

Now suppose that $\Pi$ is an irreducible generic representation of ${\rm U}(V_B \otimes _F E) \cong {\rm GL}(V_B)$ with $L$-parameter $M$. The embedding ${\rm U}(V_B) \hookrightarrow {\rm U}(V_B \otimes _F E)$ is the natural embedding ${\rm U}(V_B) \subset {\rm GL}(V_B)$. Pulling $\Pi$ back via $\iota$, and with $\chi := \omega _{\Pi }$, we see that

\begin{align*} {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}) &= {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}[\chi|_{E_1}]) \\ &\cong {\rm Hom}_{(B^{\times})^+}( \iota^*(\Pi), \omega_{\psi,\mu, B}[\chi|_{E_1}]) \\ &\cong {\rm Hom}_{B^{\times}}( \iota^*(\Pi), \Sigma_{B,N}). \end{align*}

Now it is important to note that the embedding $\iota : B^{\times} \hookrightarrow {\rm GL}(V_B) = {\rm GL}_2(E)$ is not the natural embedding $B^{\times } \hookrightarrow (B \otimes _F E)^{\times } \cong {\rm GL}_2(E)$, but rather differs from it by the outer automorphism $b \mapsto \bar {b}^{-1}$. Indeed, $\iota$ is the inverse on the central $F^{\times }$. Taking this into account, we see that the last Hom space above is the space

\begin{align*} {\rm Hom}_{B^{\times}}(\Pi^{\vee} \otimes \Sigma_{B,N}^{\vee}, \mathbb{C}) \end{align*}

of twisted trilinear forms, where $B^{\times } \hookrightarrow (B \otimes _F E)^{\times } \cong {\rm GL}_2(E)$, with the last isomorphism induced by an $E$-algebra isomorphism $B \otimes _F E \cong {\rm M}_2(E)$.

By a result of the third author [Reference PrasadPra92], one has

\begin{align*} \dim {\rm Hom}_{B^{\times}}(\Pi^{\vee} \otimes \Sigma_{B,N}^{\vee}, \mathbb{C}) \leq 1 \end{align*}

with equality if and only if

\begin{align*} \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes N^{\vee}, \psi_E) \cdot \omega_{E/F}(-1) = \mu(\det(V_B)), \end{align*}

where $As^+$ is the Asai lift of $M$ from $E$ to $F$. We refer the reader to § 8.4 for the definition and properties of $As^+$. Now let us explicate the local root number:

\begin{align*} \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes N^{\vee}, \psi_E) &= \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes {\rm Ind}_E^F (\mu^{-1} \cdot \chi),\psi) \\ &= \epsilon(1/2, {\rm Ind}_E^F ( M^{\vee} \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1} \otimes \chi), \psi) \\ &= \epsilon(1/2, {\rm Ind}_E^F ( M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}),\psi) \\ & = \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E), \end{align*}

where in the second last equality, we have noted that $\chi = \omega _{\Pi } = \det M$, so that $M^{\vee } \otimes \chi \cong M$ (since $\dim M = 2$), and in the last equality, we have used the fact that epsilon factors are inductive in dimension zero together with the fact that $\dim (M \otimes {}^\sigma \!M ^\vee ) =4$.

To conclude, we have shown the following.

Proposition 3.2 For an irreducible generic representation $\Pi$ of ${\rm U}(V_B \otimes _F E) \cong {\rm GL}(V_B)$, with $L$-parameter $M$ (a two-dimensional representation of $WD_E$),

\begin{align*} {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}) \ne 0 \iff \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \omega_{E/F}(-1) = \mu(\det(V_B)). \end{align*}

This is precisely what Conjecture 2.1 says in the case $n =2$.

3.3 Global conjecture: rank-one case

Finally, we can also verify the global Conjecture 2.7 when $\dim _E V = 1$. Let $\chi$ be a Hecke character of ${\rm GL}(V)(\mathbb {A}_E) = \mathbb {A}_E^{\times }$, so that we are considering the global period integral

\begin{align*} \mathcal{P}: \mathbb{C} \chi \otimes \overline{\omega_{V,\psi,\mu}} \longrightarrow \mathbb{C} \end{align*}

defined by

\begin{align*} \mathcal{P}(\phi) = \int_{[E_1]} \chi(x) \cdot \overline{\phi(x)} \, {{d} x}. \end{align*}

Observe that this is simply the (conjugate of the) global theta lifting of $\chi$ for the dual pair

\begin{align*} {\rm U}_1 \times {\rm U}_1 = {\rm U}(V) \times {\rm U}(W), \end{align*}

evaluated at the identity element. Here, $V$ is equipped with its given skew Hermitian structure and $W$ is the rank-one Hermitian space $\langle 1 \rangle$. The non-vanishing of $\mathcal {P}$ is thus equivalent to the non-vanishing of the global theta lift $\Theta _{V,W,\psi,\mu }(\chi )$ of $\chi$. Moreover, when this global theta lift is non-zero, it is isomorphic to the representation $\chi$ of ${\rm U}(W) = E_1$. Then we have

\begin{align*} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} \cdot \mu([E_1]) = \langle \Theta(\phi_2,\chi), \Theta(\phi_1,\chi) \rangle_{\rm Pet}, \end{align*}

where $\mu ([E_1]) =2$ is the Tamagawa measure of ${\rm U}(W)$. Now the Petersson inner product of the global theta lift on the right-hand side is computed by the Rallis inner product formula. This was first done by Tonghai Yang [Reference YangYan97] and a convenient reference is [Reference XueXue16, Theorem A.4.2]. One has

\begin{align*} \langle \Theta(\phi_2,\chi), \Theta(\phi_1,\chi) \rangle_{\rm Pet} = \frac{L(1/2, \chi^{\sigma} \chi^{-1} \cdot\mu^{-1})}{L(1, \omega_{E/F})} \cdot Z^*(\phi_2, \phi_1), \end{align*}

where

\begin{align*} Z^*(\phi_2,\phi_1)= \int^*_{\mathbb{A}_E^1} \overline{\langle g \phi_1,\phi_2\rangle}\cdot \langle g \chi, \chi \rangle_{{\rm U}(V), {\rm Pet}} \, dg \end{align*}

is the normalized global doubling zeta integral. Since the Tamagawa measure of ${\rm U}(V)$ is $2$, one has

\begin{align*} \langle g \cdot \chi, \chi \rangle_{{\rm U}(V), {\rm Pet}} = 2 \cdot \chi(g) \end{align*}

so that

\begin{align*} Z^*(\phi_1,\phi_2) = 2 \cdot \int^*_{\mathbb{A}_E^1} \overline{\langle g \phi_1,\phi_2\rangle}\cdot \chi(g) \, dg = \prod_v \mathcal{I}_v^{\#}(\chi,\chi, \phi_1,\phi_2), \end{align*}

where the local factors $\mathcal {I}_v^{\#}$ are as defined in (2.3) and (2.5). Hence, we conclude that

(3.3)\begin{equation} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} = \frac{L(1/2, \chi^{\sigma} \chi^{-1} \cdot\mu^{-1})}{L(1, \omega_{E/F})} \cdot \prod_v \mathcal{I}_v^{\#}(\chi,\chi, \phi_1,\phi_2). \end{equation}

This is precisely what Conjecture 2.7 says.

For the case when $\dim _E V = 2$, Conjecture 2.7 should reduce to Ichino's formula [Reference IchinoIch08] relating the (twisted) triple product period integral and the (twisted) triple product $L$-value. We leave the verification of this to the interested reader.

4.1 Principal series

Let $V$ be a vector space of dimension $n$ over $E$. For a partition $n = a+b,$ with $0< a \leq b \in \mathbb {Z}$, let

\begin{align*} V = V_a \oplus V_b \end{align*}

with $\dim V_a = a$ and $\dim V_b = b$. Consider the maximal parabolic subgroup

\begin{align*} P = P_{a,b} = M \cdot N \end{align*}

of ${\rm GL}(V)$ stabilizing $V_a$, with Levi factor

\begin{align*} M= {\rm GL}(V_a) \times {\rm GL}(V_b). \end{align*}

Let $\pi = \pi _1 \boxtimes \pi _2$ be a representation of ${\rm GL}(V_a)\times {\rm GL}(V_b)$ and consider the (normalized) parabolically induced representation

(4.1)\begin{equation} \pi = \pi_1 \times \pi_2 = {\rm Ind}_P^{{\rm GL}(V)} (\pi_1 \boxtimes \pi_2). \end{equation}

These are the principal series representations we will consider.

4.2 Skew-Hermitian structures

Recall that there are two inequivalent skew-Hermitian structures on $V$, distinguished by their determinants in $F^{\times }/NE^{\times }$ or $E_0^{\times }/ NE^{\times }$ (depending on whether $n = \dim V$ is even or odd). For such a class $\delta$, we let $V_{\delta }$ denote the skew-Hermitian structure on $V$ with determinant $\delta$, so that ${\rm U}(V_{\delta }) \subset {\rm GL}(V)$. We often drop $\delta$ from $V_\delta$ when a particular skew-Hermitian structure is fixed on $V$. We also let ${\rm rk}(V)$ denote the dimension of a maximal isotropic subspace of the skew-Hermitian space $V$; this is sometimes called the Witt index of $V$.

On the other hand, $V$ with a roman subscript, such as $V_a$, will denote either just a vector space over $E$ or a skew-Hermitian space over $E$ of dimension $a$.

4.3 Mackey theory

For a fixed skew-Hermitian space $V = V_{\delta }$, the goal of this section is to compute

\begin{align*} {\rm Hom}_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi,\mu}), \end{align*}

where $\omega _{V, \psi, \mu }$ denotes the Weil representation of ${\rm U}(V)$ associated to $(\psi,\mu )$. In fact, we will consider the more general

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi,\mu}). \end{align*}

This will be achieved by using Mackey theory, which requires the determination of the orbits of ${\rm U}(V)$ on the partial flag variety ${\rm GL}(V)/P$. In this analysis, each orbit gives rise to a certain induced representation of ${\rm U}(V)$ arising from the restriction of the inducing data to the stabilizer of a point in the orbit.

Thus, the representation $\pi = \pi _1 \times \pi _2$ when restricted to ${\rm U}(V)$ comes equipped with a certain finite filtration by ${\rm U}(V)$-modules in which the open orbits contribute as submodules, and the non-open orbits contribute as subquotients.

4.4 Orbits

The following lemma, the proof of which is omitted, summarizes the orbit structure of ${\rm U}(V)$ on ${\rm GL}(V) / P$ and is a direct consequence of Witt's theorem.

4.5 Stabilizers

Let $[X]$ be an ${\rm U}(V)$-orbit in ${\rm GL}(V)/P_{a,b}$, represented by an $E$-subspace $X \subset V$ of dimension $a$ with $\dim (X \cap X^\perp ) =d$. Let us first determine the stabilizer $S=S_X$ of $X$ in ${\rm U}(V)$.

Observe that $S_X$ preserves the flag

\begin{align*} 0 \subset X \cap X^\perp \subset X \subset ( X \cap X^\perp)^\perp = X+X^\perp \subset V, \end{align*}

and note that

\begin{align*} X/( X \cap X^\perp ) \subset (X+X^\perp) / ( X \cap X^\perp) =: V_{n-2d} \end{align*}

are non-degenerate skew-Hermitian spaces of dimension $a-d$ and $n-2d$, respectively. Hence, $S_X$ is contained in the maximal parabolic subgroup $Q_d$ of ${\rm U}(V)$ stabilizing the isotropic space $X \cap X^{\perp }$. The parabolic subgroup $Q_d =M_d \cdot N_d$ can be depicted in matrix form as

\begin{align*} Q_d = \Bigg(\begin{array}{@{}ccc@{}} {\rm GL}(X \cap X^{\perp}) & *_1 & *_d \\ 0 & {\rm U}(V_{n-2d}) & *_2 \\ 0 & 0 & * \end{array} \Bigg), \end{align*}

with Levi factor

\begin{align*} M_d = {\rm GL}(X \cap X^{\perp}) \times {\rm U}(V_{n-2d}) \end{align*}

and unipotent radical $N_d$. The center of $N_d$ is the subgroup $Z_d$ consisting of matrices with $*_1=*_2 =0$.

It follows that, as a subgroup of $Q_d$, $S_X$ has the form:

\begin{align*} S_X = \left(\begin{array}{@{}cccc@{}} g & *_{12} & *_{13} & *_4 \\ 0 & {\rm U}_{a-d} & 0 & *_{24} \\ 0 & 0 & {\rm U}_{b-d} & *_{34} \\ 0 & 0 & 0 & (g^*)^{-1} \\ \end{array} \right), \end{align*}

where:

• • $g \in {\rm GL}(X \cap X^{\perp }) \cong {\rm GL}_d(E)$;

• • the entries $*_{12}$ and $*_{34}$ are arbitrary matrices with entries in $E$ of appropriate sizes which determines $*_{24}$, $*_{13}$;

• • the entry $*_4$ is an arbitrary skew-Hermitian matrix of size $d \times d$.

Let us highlight certain natural subgroups or quotients of $S_X$.

• • The unipotent radical $N(S_X)$ of $S_X$ consists of those matrices which have the identity matrix on each diagonal block. Observe that $N(S_X)$ is, in fact, the unipotent radical $N_d$ of the maximal parabolic subgroup $Q_d$.

• • The center $Z(S_X)$ of $N(S_X)$ is the subgroup consisting of elements whose only non-zero entry in the upper triangular blocks is $*_4$, so that $Z(S_X) =Z_d$.

• • The Levi factor $S_X/ N(S_X)$ is isomorphic to

  \begin{align*} {\rm GL}(X \cap X^{\perp}) \times {\rm U}_{a-d} \times {\rm U}_{b-d}. \end{align*}

4.6 Modules

In what follows, we use ${\rm Ind}$ for the usual normalized induction, and ${\rm ind}$ for the usual normalized induction with compact support, whereas we will use $\mathfrak {I} nd$ and $\mathfrak {i} nd$ for the corresponding un-normalized induction. Thus, for example,

\begin{align*} \pi = \pi_1 \times \pi_2 = {\rm Ind}_P^{{\rm GL}(V)} (\pi_1 \otimes \pi_2) = \mathfrak{I} nd_P^{{\rm GL}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_P^{1/2}). \end{align*}

By Mackey theory, the restriction of the principal series representation $\pi = \pi _1 \times \pi _2$ to ${\rm U}(V)$ has a finite equivariant filtration indexed by the ${\rm U}(V)$-orbits given in Lemma 4.2. For each such ${\rm U}(V)$-orbit $[X]$, let $\pi _X$ denote the associated ${\rm U}(V)$-subquotient of $\pi$. The following proposition determines the representation $\pi _X$.

Proposition 4.3 For a ${\rm U}(V)$-orbit $[X]$ on ${\rm GL}(V)/P$, with $\dim X \cap X^{\perp } = d$ and stabilizer $S= S_X$, one has

\begin{align*} \pi_X \cong \mathfrak{i} nd_{S}^{{\rm U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_P^{1/2})|_{S} = {\rm ind}_{S}^{{\rm U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})|_{S}, \end{align*}

where we have written $\delta _{P/S} = \delta _P \delta _S^{-1}$.

We note that the representation $\pi _1 \otimes \pi _2 \otimes \delta _{P/S}^{1/2}$ is non-trivial on the unipotent radical $N(S)$ of $S$, but it is trivial on the center $Z(S)$ of $N(S)$.

4.7 Branching for $\pi _X$

We are now ready to consider the branching problem

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi, \mu}). \end{align*}

Since, as a ${\rm U}(V)$-module, $\pi _1 \times \pi _2$ has a finite filtration with subquotients $\pi _X$ as given in Proposition 4.3, it is natural to first consider

\begin{align*} {\rm Ext}^i_{{\rm U}(V)} (\pi_X, \omega_{V, \psi, \mu}). \end{align*}

The result of this key computation is given by the following proposition.

Proposition 4.4 For an orbit $[X]$ of ${\rm U}(V)$ on ${\rm GL}(V)/P$, with $\dim (X \cap X^{\perp }) =d$, corresponding stabilizer $S = S_X$ and associated ${\rm U}(V)$-module $\pi _X$, one has

\begin{align*} & {\rm Ext}^i_{{\rm U}(V)} \big( \pi_X, \omega_{V,\psi,\mu}\big)\\ &\quad \cong {\rm Ext}^i_{S/N(S)}\Big( \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d},\, \delta_{S}^{1/2} \cdot |{\det}_{{\rm GL}_d}|^{-1/2}\mu \cdot \omega_{V_{n-2d},\psi,\mu} \Big) \end{align*}

where we note:

• • $S/ N(S) \cong {\rm GL}_d(E) \times {\rm U}_{a-d} \times {\rm U}_{b-d}$;

• • $(\pi _1)_{d,a-d}$ denotes the unnormalized Jacquet module of $\pi _1$ with respect to the $(d,a-d)$ parabolic subgroup in ${\rm GL}(V_a) \cong {\rm GL}_a(E)$, regarded as a representation of ${\rm GL}_d(E) \times {\rm U}_{a-d} \subset {\rm GL}_d(E) \times {\rm GL}_{a-d}(E)$ by restriction;

• • likewise, $(\pi _2)_{b-d,d}$ is the unnormalized Jacquet module of $\pi _2$ with respect to the $(b-d,d)$-parabolic subgroup in ${\rm GL}(V_b) \cong {\rm GL}_b(E)$, regarded as a representation of ${\rm U}_{b-d} \times {\rm GL}_d(E) \subset {\rm GL}_{b-d}(E) \times {\rm GL}_d(E)$ by restriction and taking contragredient on the ${\rm GL}_d(E)$ factor;

• • the characters $\delta _{P/S}$ and $\delta _S$ are trivial on ${\rm U}_{a-d} \times {\rm U}_{b-d}$ and are given on ${\rm GL}_d(E)$ by

  \begin{align*} \delta_{P/S} = |\!\det\!|^d \quad \text{and} \quad \delta_S= |\!\det\!|^{n-d}. \end{align*}

In particular, for the two open orbits $X$ corresponding to $d=0$, we have

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] \cong \sum_{\substack{i=j+k\\ V= V_a\oplus V_b}} {\rm Ext}^j_{{\rm U}(V_a)} [\pi_1|_{{\rm U}(V_a)}, \omega_{V_a, \psi,\mu}] \otimes {\rm Ext}^k_{{\rm U}(V_b)}[ \pi_2|_{{\rm U}(V_b)}, \omega_{V_b,\psi,\mu}] , \end{align*}

where $X = V_a$ are the isomorphism classes of non-degenerate subspaces of $V$ of dimension $a$ with orthogonal complement $X^{\perp } = V_b$.

Lemma 4.5 Let $U \subset H$ be $p$-adic groups with $U$ a normal subgroup of $H$ which is a union of compact open subgroups. Let $\pi _1,\pi _2$ be two smooth representations of $H$ such that $U$ acts trivially on $\pi _2$. Then

\begin{align*} {\rm Ext}^i_H[\pi_1,\pi_2] \cong {\rm Ext}^i_{H/U}[ \pi_{1, U}, \pi_2]. \end{align*}

Lemma 4.6 Let $G$ be any $p$-adic group, $Z\subset G$ a closed central subgroup. If $\pi _1$ and $\pi _2$ are two smooth representations of $G$ on which $Z$ operates by different characters, then

\begin{align*} {\rm Ext}_G^i[\pi_1,\pi_2] = 0\quad \text{for all}\ i \geq 0. \end{align*}

Lemma 4.7 Let $V=V_n$ be a skew-Hermitian space over $E$ of dimension $n$, and let $\omega _{V,\psi,\mu }$ be a Weil representation of ${\rm U}(V)$. Let $Q_d = M_d N_d$ be the maximal parabolic subgroup of ${\rm U}(V)$ stabilizing a $d$-dimensional isotropic space (see § 4.5), so that $M_d \cong {\rm GL}_d(E) \times {\rm U}(V_{n-2d})$. Then for $Z_d$, the center of $N_d$, we have

\begin{align*} (\omega_{V,\psi,\mu})_{Z_d} = (\omega_{V,\psi, \mu})_{N_d} \cong (\mu \cdot |-|^{1/2} \circ \det) \otimes\omega_{{V_{n-2d}},\psi,\mu} , \end{align*}

as $M_d$-modules.

With these preliminaries in place, we now compute

\begin{alignat}{2} &\hskip -17pt{\rm Ext}^i_{\mathrm{U}(V)}[ \pi_X, \omega_{V,\psi, \mu}] & \notag \\ \quad\cong &\hskip 5pt {\rm Ext}^i_{\mathrm{U}(V)}[ {\rm ind}_{S}^{\mathrm{U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2}), \omega_{V,\psi, \mu}], &\text{ (by Proposition 4.3)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{\mathrm{U}(V)}[ \omega_{V,\psi^-, \mu^{-1}}, {\rm Ind}_{S}^{\mathrm{U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by (a), (b) and (c))} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S}[ \delta_{S}^{-1/2}\omega_{V,\psi^-, \mu^{-1}}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by Frobenius reciprocity)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \delta_{S}^{-1/2}(\omega_{V,\psi^-, \mu^{-1}})_{Z(S)}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by Lemma 4.5)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \delta_{S}^{-1/2} \cdot \mu^{-1} \cdot |\!\det\!|^{1/2} \cdot \omega_{{V_{n-2d}},\psi^-, \mu^{-1}}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ] & \text{ (by Lemma 4.7)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2}, \,\, \delta_{S}^{1/2} |\!\det\!|^{-1/2}\cdot \mu \cdot \omega_{V_{n-2d},\psi,\mu}] & \hskip 5pt \text{(by (a))} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/N(S)}[ (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} \otimes \delta_{P/S}^{1/2},\,\, \delta_{S}^{1/2} |\!\det\!|^{-1/2} \cdot \mu \cdot \omega_{V_{n-2d},\psi,\mu}] & \notag \end{alignat}

where $(\pi _1)_{d,a-d}$ denotes the unnormalized Jacquet module of $\pi _1$ with respect to the $(d,a-d)$ parabolic subgroup in ${\rm GL}_a(E)$; similarly for $(\pi _2)_{b-d,d}$. Here we have applied Lemma 4.5 (taking $U = N(S)/Z(S)$) for the last isomorphism for which it is important to note that $N(S)/Z(S)$ maps isomorphically to the product of the unipotent radicals of the $(d,a-d)$-parabolic subgroup of ${\rm GL}_a(E)$ and the $(b-d,d)$-parabolic subgroup of ${\rm GL}_b(E)$.

For the final assertion in the proposition regarding the open orbits corresponding to $d=0$, it suffices to observe that for the direct sum of non-degenerate skew-Hermitian spaces $V = V_a \oplus V_b$, we have the tensor product decomposition of their Weil representations:

\begin{align*} \omega_{V,\psi,\mu} \cong \omega_{V_a,\psi,\mu} \otimes \omega_{V_b,\psi,\mu}, \end{align*}

as representations of ${\rm U}(V_a) \times {\rm U}(V_b) \subset {\rm U}(V)$. Thus, the final assertion is a direct consequence of the Kunneth theorem [Reference PrasadPra18, Theorem 3.1], completing the proof of Proposition 4.4.

4.8 Branching for $\pi _1 \times \pi _2$

We can now assemble the results of Proposition 4.4 for the various ${\rm U}(V)$-orbits $[X]$ on ${\rm GL}(V)/ P$ to understand the branching problem

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi, \mu}). \end{align*}

The result is most definitive when only the open orbits have non-zero contribution. The following theorem, which is the main result of this section, gives a simple sufficient condition (temperedness of $\pi _1$ and $\pi _2$) for this to happen.

Theorem 4.8 Suppose that $\pi _1$ and $\pi _2$ are tempered representations of ${\rm GL}(V_a)$ and ${\rm GL}(V_b)$ (with unitary central characters), so that $\pi = \pi _1 \times \pi _2$ is a tempered principal series representation of ${\rm GL}(V)$ for $V=V_a+ V_b$. If $[X]$ is a non-open orbit of ${\rm U}(V)$ on ${\rm GL}(V)/P$, then for $\pi _X$, the subquotient of $\pi$ supported on the orbit $[X]$, we have

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] = 0, \end{align*}

for all $i \geq 0$.

As a consequence, for all $i \geq 0$, one has

(4.10)\begin{align} &\bigoplus_{\delta} {\rm Ext}^i_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi, \mu}] \nonumber\\ &\quad \cong \bigoplus_{i=j+k} \bigg(\bigoplus_{\delta'} {\rm Ext}^j_{{\rm U}(V_{a,\delta'})} [ \pi_1, \omega_{V_{a, \delta'},\psi, \mu}] \bigg) \otimes \bigg(\bigoplus_{\delta''} {\rm Ext}^k_{{\rm U}(V_{b,\delta''})}[ \pi_2, \omega_{V_{b,\delta''},\psi, \mu}] \bigg), \end{align}

where the sums over $\delta$, $\delta '$, and $\delta ''$ run over $F^{\times }/NE^{\times }$ or $E_0^{\times }/ NE^{\times }$ according to the parity of $n, a,b$, respectively.

Hence, for $i=0$, one has

(4.11)\begin{align} &{\rm Hom}_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi}]\nonumber\\ &\quad \cong \bigoplus_{(\delta', \delta''): V_{a,\delta'} \oplus V_{b,\delta''} \cong V_{\delta} } {\rm Hom}_{{\rm U}(V_{a,\delta'})}[ \pi_1, \omega_{V_{a,\delta'},\psi}] \otimes {\rm Hom}_{{\rm U}(V_{b,\delta''})} [ \pi_2, \omega_{V_{b,\delta''},\psi}] . \end{align}

In particular,

(4.12)\begin{align} &\bigoplus_{\delta} {\rm Hom}_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi}]\nonumber\\ &\quad \cong \bigg(\bigoplus_{\delta'} {\rm Hom}_{{\rm U}(V_{a,\delta'})}[ \pi_1, \omega_{V_{a,\delta'},\psi}] \bigg)\otimes \bigg(\bigoplus_{\delta''} {\rm Hom}_{{\rm U}(V_{b,\delta''})} [ \pi_2, \omega_{V_{b,\delta''},\psi}] \bigg). \end{align}

Proof. By Proposition 4.4, ${\rm Ext}^i_{{\rm U}(V)}[ \pi _X, \omega _{V,\psi,\mu }]$ is equal to

\begin{align*} {\rm Ext}^i_{S/N(S)}[ \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} , \delta_{S}^{1/2} \cdot |\!\det\!|^{-1/2} \mu\cdot \omega_{V_{n-2d},\psi,\mu}] . \end{align*}

Since $\pi _1$ is tempered, it follows by Casselman's temperedness criterion that the central exponents of $(\pi_1)_{d, a-d}$ have the form $\delta_{P_{d, a-d}}^{(1 + \alpha)/2}$ with $\alpha \geq 0$. Moreover, for $(g,h) \in {\rm GL}_d(E) \times {\rm U}_{a-d}(F)$,

\begin{align*} \delta_{P_{d,a-d}}(g,h)^{(1+\alpha)/2} = |\!\det(g)|^{(a-d+\epsilon)/2 }, \quad \text{with $\epsilon=\alpha \cdot (a-d)$.} \end{align*}

Similarly, the central exponents of $(\pi_2)_{b-d,d}$ have the form

\begin{align*} \delta_{P_{b-d,d}}^{(1 + \alpha')/2} \quad \text{with}\ \alpha' \geq 0,\end{align*}

and for $(h, (g^*) ^{-1} ) \in {\rm U}_{b-d}(F) \times {\rm GL}_d(E)$,

\begin{align*} \delta_{P_{b-d,d}}(h, (g^*)^{-1})^{(1+\alpha')/2} = |\!\det(g)|^{(b-d+\epsilon')/2}, \quad \text{with $\epsilon' = \alpha' \cdot (b-d)$.} \end{align*}

Summarizing, we have:

1. (1) the representation

   \begin{align*} A = \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} \end{align*}
   has central exponents of the form
   \begin{align*} |\!\det \!|^{(a-d+\epsilon)/2} \cdot |\!\det \!|^{(b-d+\epsilon')/2} |\!\det \!|^{d/2} = |\!\det \!|^{(n-d+\epsilon+\epsilon')/2} \end{align*}
   with $\epsilon$ and $\epsilon'$ non-negative;

2. (2) the representation

   \begin{align*} B= \delta_{S}^{1/2} \cdot |\!\det \!|^{-1/2} \mu\cdot \omega_{V_{n-2d},\psi,\mu} \end{align*}
   is the twist of a unitary representation of ${\rm GL}_d(E) \times {\rm U}(V_{n-2d})$ by the character
   \begin{align*} |\!\det\!|^{(n-d)/2} \cdot |\!\det\! |^{-1/2} = |\!\det\!|^{(n-d-1)/2} \end{align*}
   of ${\rm GL}_d(E)$.

Thus, when $d > 0$, the actions of the center of ${\rm GL}_d(E)$ in

\begin{align*} S/N(S)={\rm GL}_d(E) \times {\rm U}(V_{a-d}) \times {\rm U}(V_{b-d}) \end{align*}

on the two representations $A$ and $B$ are different. Therefore, by Lemma 4.6,

\begin{align*} {\rm Ext}^i_{S/N(S)}[A,B] = 0, \end{align*}

for all $i \geq 0$ (as long as $d\not = 0$). This completes the proof that for a non-open ${\rm U}(V)$-orbit $[X] \subset {\rm GL}(V)/P$, the associated subquotient $\pi _X$ of the ${\rm U}(V)$-module $\pi$ satisfies

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi}] = 0 \quad \text{for all $i \geq 0$.} \end{align*}

As a consequence of the vanishing of ${\rm Ext}^i$, $i \geq 0$, for all non-open orbits, we deduce that

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] = \bigoplus_X {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}], \end{align*}

where now $X$ runs over the two open orbits of ${\rm U}(V)$ on ${\rm GL}(V)/P$. Since Proposition 4.4 calculates ${\rm Ext}^i_{{\rm U}(V)}[ \pi _X, \omega _{V,\psi }]$ for the open orbits, the proof of Theorem 4.8 is complete.

5.1 Inductive argument

Let us first record the following consequence of Theorem 4.8.

Proof. The vanishing statement in part (ii) follows from Theorem 4.8, especially (4.9). Likewise, (4.11) imply that Conjecture 2.1(i) and (ii) for $\pi _1 \times \pi _2$ follows from the corresponding statements for $\pi _1$ and $\pi _2$. Thus, it remains to verify that the unique skew-Hermitian space $V_{\delta }$ for which ${\rm Hom}_{{\rm U}(V_{\delta })}(\pi _1 \times \pi _2 , \omega _{V_{\delta }, \psi, \mu })$ is non-zero is as predicted by Conjecture 2.1(iii).

Assume then that

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\pi_1 \times \pi_2 , \omega_{V, \psi, \mu}) \ne 0. \end{align*}

By (4.10),

\begin{align*} {\rm Hom}_{{\rm U}(V_a)}(\pi_1 , \omega_{V_a, \psi, \mu}) \ne 0 \quad \text{and} \quad {\rm Hom}_{{\rm U}(V_b)}(\pi_2 , \omega_{V_b, \psi, \mu}) \ne 0 \end{align*}

for a unique pair of skew-Hermitian spaces $V_a$ and $V_b$ satisfying $V_a \oplus V_b \cong V$. As we have assumed that Conjecture 2.1(iii) holds for $\pi _1$ and $\pi _2$, we have

\begin{align*} \mu(\det(V_a)) = \epsilon( 1/2, \pi_1 \times {}^\sigma\pi_1 ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi_1}(-1)^a \cdot \omega_{E/F}(-1)^{a(a-1)/2}, \end{align*}

and

\begin{align*} \mu(\det(V_b)) = \epsilon( 1/2, \pi_2 \times {}^\sigma\pi_2 ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi_2}(-1)^b \cdot \omega_{E/F}(-1)^{b(b-1)/2}. \end{align*}

This implies that for $\pi = \pi _1 \times \pi _2$,

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, \pi \times {}^\sigma\pi ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi}(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

using the facts that $\mu ={}^\sigma \mu ^{-1}$, $\mu (-1)= \omega _{E/F}(-1)$ and

\begin{align*} \epsilon (\Pi + {}^\sigma\Pi ^\vee, \psi_E) = \det \Pi(-1) \end{align*}

for any representation $\Pi$ of ${\rm GL}_m(E)$. This completes the proof of the corollary.

5.2 Reduction to the discrete-series case

Corollary 5.1 allows one to reduce Conjecture 2.1 for tempered representations to the case of discrete-series representations.

5.3 Borel–Principal series

In addition, by applying Corollary 5.1 inductively, we deduce the following.

5.4 An alternative proof

We now give another proof of Corollary 5.3 as it brings out an interesting structure of the open orbits of ${\rm U}(V)$ on ${\rm GL}(V)/B$.

Suppose that

\begin{align*} \Pi = {\rm Ind}_B^{{\rm GL}(V)} (\chi_1 \otimes \cdots \otimes \chi_n) \quad \text{(normalized induction),} \end{align*}

so that its $L$-parameter is

\begin{align*} M = \bigoplus_i \chi_i . \end{align*}

On restriction to ${\rm U}(V)$, Theorem 4.8 inductively implies that only the open ${\rm U}(V)$-orbits on the flag variety ${\rm GL}(V)/B$ will contribute to the Hom space ${\rm Hom}_{{\rm U}(V)}[ \Pi, \omega _{V,\psi }]$. Moreover, using Lemma 4.2 inductively, the open orbits can be described as follows. Given an ordered collection

\begin{align*} \mathcal{L} = \{L_1,\ldots , L_n \} \end{align*}

of non-degenerate orthogonal lines in $V$, the ${\rm U}(V)$-orbit of the flag

\begin{align*} \mathfrak{F}_{\mathcal{L}}: L_1 \subset L_1 \oplus L_2 \subset \cdots \end{align*}

is an open orbit, and the stabilizer of $\mathfrak {F}_{\mathcal {L}}$ in ${\rm U}(V)$ is the subgroup

\begin{align*} {\rm U}(\mathcal{L}):= \prod_i {\rm U}(L_i). \end{align*}

Moreover, all open orbits are given by such an ordered collection $\{ L_i \}$ of isomorphism classes of (non-degenerate) skew-Hermitian $E$-spaces of dimension 1, subject to the condition that $\bigoplus _i L_i \cong V$ as skew-Hermitian $E$-spaces; we say that such an $\mathcal {L}$ is $V$-relevant. There are thus $2^{n-1}$ open orbits, indexed by $V$-relevant $\mathcal {L}$. This can also be gleaned from a Galois cohomological argument: having fixed an open orbit over $F$ with stabilizer ${\rm U}(\mathcal {L})$ in ${\rm U}(V)$ and noting that there is exactly one open orbit over $\overline {F}$, the number of open ${\rm U}(V)$-orbits is given by

\begin{align*} {\rm Ker}\big( H^1(F, {\rm U}(\mathcal{L})) \rightarrow H^1(F, {\rm U}(V))\big) = {\rm Ker} \big( (F^{\times}/ N(E^{\times}))^n \rightarrow F^{\times}/ N(E^{\times}) \big). \end{align*}

Hence, by an inductive application of Theorem 4.8, we have

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) \cong \bigoplus_{\mathcal{L}} {\rm Hom}_{{\rm U}(V)} ({\rm ind}_{{\rm U}(\mathcal{L})}^{{\rm U}(V)} ( \boxtimes_i \chi_i ), \omega_{V,\psi,\mu}), \end{align*}

where the sum runs over $V$-relevant $\mathcal {L}$. By Frobenius reciprocity, and the fact that

\begin{align*} \omega_{V,\psi, \mu} |_{{\rm U}(\mathcal{L})} \cong \bigotimes_i \omega_{L_i,\psi, \mu}, \end{align*}

one deduces that

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) \cong \bigoplus_{\mathcal{L}} \bigotimes_i {\rm Hom}_{{\rm U}(L_i)}( \chi_i, \omega_{L_i,\psi, \mu}). \end{align*}

Now by Theorem 3.1 (the theorem of Moen and Rogawski),

\begin{align*} {\rm Hom}_{{\rm U}(L_i)}( \chi_i, \omega_{L_i,\psi, \mu}) \ne 0 \Longleftrightarrow \epsilon(1/2, \chi_i/ \chi_i^{\sigma} \cdot \mu^{-1}, \psi_E) \cdot \chi_i(-1) = \mu(\det(L_i)). \end{align*}

Hence, at most one term in the sum over $\mathcal {L}$ has non-zero contribution, and this unique $\mathcal {L}$ exists if and only if

\begin{align*} \mu(\det(V)) = \prod_i \epsilon(1/2, \chi_i/ \chi_i^{\sigma} \cdot \mu^{-1}, \psi_E) \cdot \chi_i(-1). \end{align*}

To prove Conjecture 2.1, we explicate as follows:

\begin{align*} &\epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \cdot \mu^{-1}, \psi_E) \\ &\quad = \prod_i \epsilon(1/2, \chi_i/\chi_i^{\sigma} \cdot \mu^{-1},\psi_E) \cdot \prod_{i< j} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big). \end{align*}

For $i < j$, observe that

\begin{align*} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big) &= \epsilon ( 1/2, \chi_i/ \chi_j^{\sigma}\cdot \mu^{-1}, \psi_E) \epsilon ( 1/2, \chi^\sigma_j/ \chi_i\cdot \mu^{-1}, \psi_E) \\ & = \chi_i(-1) \cdot \chi_j(-1) \cdot \omega_{E/F}(-1), \end{align*}

where we have used the following standard properties of the epsilon factor:

1. (1) $\epsilon (1/2, W, \psi _E) \cdot \epsilon (1/2, W^\vee, \psi _E) = \det (W)(-1)$;

2. (2) $\epsilon (1/2, W, \psi _E) = \epsilon (1/2, W^\sigma, \psi _E)$.

It follows that

\begin{align*} \prod_{i< j} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big) = \det(M)(-1)^{n-1} \cdot \omega_{E/F}(-1)^{n(n-1)/2}. \end{align*}

Putting everything together, we see that ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{V,\psi,\mu }) \ne 0$ if and only if

\begin{align*} \mu(\det(V)) = \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \cdot \mu^{-1}, \psi_E) \cdot \det(M)(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

as desired.

5.5 Reduction to the supercuspidal case

We have seen that Conjecture 2.1 for tempered representations can be reduced to the case of discrete-series representations by a Mackey theory argument. In the rest of the section, we investigate whether the same argument can be used to reduce Conjecture 2.1 for discrete-series representations to the case of supercuspidal representations. It turns out that this can be done under a certain hypothesis. While we cannot prove this hypothesis in general, it can be shown in some situations. This will allow us to prove Conjecture 2.1 for the Steinberg representation, for example.

Let us first set up some notation and formulate the relevant hypothesis. Suppose that $\pi$ is a supercuspidal representation (with unitary central character) of ${\rm GL}(V) = {\rm GL}_m(E)$. The parabolically induced representation

\begin{align*} \pi |\!\det\!|^{({n-1})/{2}} \times \pi|\!\det\!|^{({n-3})/{2}} \times \cdots\times \pi|\!\det\!|^{-({n-1})/{2}} \end{align*}

of ${\rm GL}(V^{\oplus n}) \cong {\rm GL}_{mn}(E)$ is a standard module and, thus, has a unique irreducible quotient ${\rm Sp}(\pi,n)$, which is often called a Speh representation and is non-tempered (if $n>1$). This parabolically induced representation also has a unique irreducible submodule ${\rm St}(\pi,n)$; this is the ‘generalized Steinberg’ representation, which is a discrete-series representation. All the irreducible (unitary) discrete-series representations of general linear groups are of the form ${\rm St}(\pi, n)$. The supercuspidal ones are precisely those with $n=1$.

If

\begin{align*} \phi_\pi: W_E\rightarrow {\rm GL}_m(\mathbb{C}) \end{align*}

is the $L$-parameter of $\pi$, then the $L$-parameter of ${\rm St}(\pi,n)$ is the representation $\phi _{\pi } \otimes {\rm Sym}^{n-1}(\mathbb {C}^2)$ of the Weil–Deligne group $WD_E= W_E \times {\rm SL}_2(\mathbb {C})$. We also write $[n]={\rm Sym}^{n-1}(\mathbb {C}^2)$ for the unique irreducible $n$-dimensional representation of ${\rm SL}_2(\mathbb {C})$, and write $\phi _{\pi }[n]$ for the $L$-parameter of ${\rm St}(\pi,n)$.

To deal with the generalised Steinberg representations, we will need to make an assumption. In this, $V,W$ are the two isomorphism classes of skew-Hermitian spaces over $E$ of dimension $m$. Then we make the following assumption:

\begin{align*} ({\bf Assumption}) \quad \begin{cases} {\rm Hom}_{{\rm U}(V+ V )}[{\rm Sp}(\pi,2), \omega_{V+V,\psi, \mu}] = 0, \\ {\rm Hom}_{{\rm U}(V+ W )}[{\rm Sp}(\pi,2), \omega_{V+W,\psi, \mu}] = 0. \end{cases} \end{align*}

We remark that this assumption is a case of the non-tempered twisted GGP conjecture formulated in Conjecture 7.2.

With this assumption formulated, our result is as follows.

Proof of Theorem 5.4(b) Recall from [Reference TateTat79] that for an irreducible representation $\lambda \otimes [n]$ of $WD_E=W_E \times {\rm SL}_2(\mathbb {C})$, one has

\begin{align*} \epsilon( \lambda \otimes [n]) = \epsilon(\lambda)^n \cdot \det (-F, \lambda^I)^{n-1}, \end{align*}

where $\lambda ^I$ denotes the subspace of $\lambda$ fixed by the inertia group $I$ and $F$ denotes the Frobenius element of $W_E/I$.

On the other hand, by the Clebsch–Gordon theorem,

\begin{align*} [n] \otimes [n] = [2n-1] \oplus [2n-3] \oplus \cdots \oplus [1]. \end{align*}

In particular, only odd integers $(2d+1)$ appear in this decomposition. It is easy to see that in the expression

\begin{align*} \epsilon( \lambda \otimes [2d+1]) = \epsilon(\lambda)^{2d+1} \cdot \det (-F, \lambda^I)^{2d}, \end{align*}

the factor $\det (-F, \lambda ^I)^{2d}$ is trivial for $\lambda$ a conjugate selfdual representation of $W_E$. Hence, we find that

\begin{align*} \epsilon( \lambda \otimes [2d+1]) = \epsilon(\lambda)^{2d+1} \end{align*}

for $\lambda$ a conjugate selfdual representation of $W_E$. These considerations, applied to the conjugate selfdual representation $\lambda$ of $W_E$ associated to $\pi \times {}^\sigma \pi ^\vee \times \mu ^{-1}$, allow one to prove identity $(2)$ from identity $(1)$; we leave the simple and pleasant computation to the reader.

Proof. (i) The fact that the discrete-series representation ${\rm St}(\pi, n)$ of ${\rm GL}_{mn}(E)$ appears as a quotient of the principal series $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ is clear, since ${\rm St}(\pi, n)$ is a quotient of the principal series representation

\begin{align*} \nu^{-(n-1)/2}\pi \times \nu^{-(n-3)/2} \pi \times \cdots \times \nu^{(n-1)/2}\pi . \end{align*}

It is well-known from Zelevinski [Reference ZelevinskyZel80] that the principal series $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ has length 2, so that $K_n$ is irreducible and part (i) is proved.

(ii) Since $K_n$ is irreducible, it suffices to prove that there is a non-zero ${\rm GL}_{mn}(E)$-equivariant homomorphism from the principal series $\nu ^{-(n-2)/2}{\rm Sp}(\pi,2) \times \nu {\rm St}(\pi,n-2)$ to $K_n$. By the second adjointness theorem, this boils down to proving that the normalized Jacquet functor of $K_n$ for the opposite parabolic of the maximal standard parabolic with Levi ${\rm GL}_{2m}(E) \times {\rm GL}_{(n-2)m}(E)$ contains the irreducible representation $\nu ^{-(n-2)/2}{\rm Sp}(\pi,2) \otimes \nu {\rm St}(\pi,n-2)$ of ${\rm GL}_{2n}(E) \times {\rm GL}_{(m-2)n}(E)$ as a submodule. We leave this simple calculation to the reader.

Next, we apply Proposition 4.4 to the two principal series representations appearing in Lemma 5.5. We do not perform the explicit calculation here, but simply summarize the results.

With the above two lemmas at hand, let us now return to the proof of Theorem 5.4(a). By Lemma 5.5(ii), combined with Lemma 5.6(i), we deduce that

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}[K_n,\omega_{V^{ n},\psi}] = 0. \end{align*}

Therefore, by Lemma 5.5(i),

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}({\rm St}(\pi, n) , \omega_{V^{ n},\psi}) \cong {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}). \end{align*}

Furthermore, from Lemma 5.6(ii),

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}) \end{align*}

is contributed by the submodule of the principal series representation

\begin{align*} \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) \end{align*}

supported on the open orbits.

Observe that the open orbits of the action of ${\rm U}(V^n)$ on ${\rm GL}_{mn}(E)/P_{m,m(n-1)}$ are parametrized by the isomorphism classes of the skew-Hermitian subspaces of $V^n$ of dimension $m=\dim (V)$. Thus, there are exactly two orbits, represented by the skew-Hermitian spaces $V$ and $W$, with stabilizer in ${\rm U}(V^n)$ being ${\rm U}(V) \times {\rm U}(V^{n-1})$ and ${\rm U}(W) \times {\rm U}(W+V^{n-2})$. Therefore,

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}({\rm St}(\pi, n) , \omega_{V^{ n},\psi}) \cong {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}) \end{align*}

is the sum $A + B$ of the contributions coming from these two open orbits, with

\begin{align*} A & = {\rm Hom}_{{\rm U}(V)}(\pi, \omega_{V,\psi}) \otimes {\rm Hom}_{{\rm U}(V^{n-1})}({\rm St}(\pi, n-1) , \omega_{V^{(n-1)},\psi}), \\ B & = {\rm Hom}_{{\rm U}(W)}(\pi, \omega_{W,\psi}) \otimes {\rm Hom}_{{\rm U}(W+V^{n-2})}({\rm St}(\pi, n-1) , \omega_{W+V^{(n-2)},\psi}). \end{align*}

Now as ${\rm Hom}_{{\rm U}(W)}(\pi, \omega _{W,\psi })=0$, we conclude that $B = 0$. This completes the proof of part (a) of Theorem 5.4, and hence the proof of Theorem 5.4 is complete.

5.6 The Steinberg representation

At the moment, we do not know how to prove the (Assumption) that

\begin{align*} {\rm Hom}_{{\rm U}(V^{ 2})}[{\rm Sp}(\pi,2), \omega_{V^{ 2},\psi}] = {\rm Hom}_{{\rm U}(V+W)}[{\rm Sp}(\pi,2), \omega_{V+W,\psi}] = 0, \end{align*}

except when $m=\dim V = \dim W =1$, i.e. when $\pi$ is a character of $E^\times$. In this case, the (Assumption) is equivalent to saying that the Weil representation of ${\rm U}(2)$ does not contain any one-dimensional representation of ${\rm U}(2)$ where ${\rm U}(2)$ is either of the two unitary groups in two variables. The next lemma establishes this.

Lemma 5.7 For $V$ be a skew-Hermitian space over $E$, a non-archimedean local field, of dimension $d \geq 2$, one has

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \chi, \omega_{V,\psi, \mu}] = 0 \end{align*}

for any one-dimensional character $\chi$ of ${\rm U}(V)$.

Proof. If $V = V_1\oplus V_2$, a direct sum of skew-Hermitian spaces, one knows that as representations of ${\rm U}(V_1) \times {\rm U}(V_2) \subset {\rm U}(V)$,

\begin{align*} \omega_{V,\psi, \mu}= \omega_{V_1,\psi,\mu} \otimes \omega_{V_2,\psi,\mu}. \end{align*}

Therefore, the proof of the lemma reduces to the case of $d=2$.

When $d =2$, we have seen in the discussion in § 3.2 that $\omega _{V, \psi,\mu }$ is a direct sum of irreducible summands (with different central characters), each of which has dihedral $L$-parameters. Hence, $\omega _{V, \psi,\mu }$ does not contain one-dimensional characters of ${\rm U}(V)$. (This uses the requirement for $E$ to be a non-archimedean local field.)

As a consequence, we obtain the following.

5.7 Ext vanishing

The results of this section also prove the following theorem on the vanishing of Ext groups for tempered representations.

Theorem 5.9 Let $F$ be a non-archimedean local field and $E$ a separable quadratic algebra over $F$. Let $V$ be a skew-Hermitian space over $E$, with corresponding unitary group ${\rm U}(V) \subset {\rm GL}(V)$. For any irreducible tempered representation $\Pi$ of ${\rm GL}(V)$ and any Weil representation $\omega _{V, \psi, \mu }$ of ${\rm U}(V)$,

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \Pi, \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{align*}

Proof. Footnote ¹ As any irreducible tempered representation of ${\rm GL}(V)$ is parabolically induced from an irreducible discrete-series representation of a Levi subgroup, by Corollary 5.1, it suffices to prove this theorem for the discrete-series representations $\Pi = {\rm St}(\pi,n)$ of ${\rm GL}_{mn}(E)$ where $\pi$ is a cuspidal representation of ${\rm GL}_m(E)$. We prove this by an induction on the integer $n$.

The base case $n=1$ is clear since a supercuspidal representation $\pi$ of ${\rm GL}_m(E)$ is a projective representation when restricted to any subgroup $H \subset {\rm GL}_m(E)$ for which the intersection of $H$ with the center of ${\rm GL}_m(E)$ is compact. In particular, this applies to $H={\rm U}_m(E)$.

For the inductive step, let us assume that the theorem holds good for $\Pi = {\rm St}(\pi,n-1)$, so that our goal is to prove it for $\Pi = {\rm St}(\pi,n)$. Recall the exact sequence from Lemma 5.5:

(5.10)\begin{equation} 0 \rightarrow K_n \rightarrow \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) \rightarrow {\rm St}(\pi, n) \rightarrow 0, \end{equation}

with $K_n$ an irreducible representation of ${\rm GL}_{mn}(E)$. Observe that

(5.11)\begin{equation} {\rm Ext}^i_{{\rm U}(V)}( \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{equation}

Indeed, by Lemma 5.6(ii), one knows the vanishing of ${\rm Ext}^i, i \geq 0$ for the subquotient of the principal series representation $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ of ${\rm GL}_{mn}(E)$ supported on a non-open orbit. For the open orbits, the vanishing of ${\rm Ext}^i, i \geq 1$ is consequence of the induction hypothesis and the Kunneth theorem.

Equipped with this vanishing of ${\rm Ext}^i_{{\rm U}(V)}( \nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1) , \omega _{V, \psi, \mu })$ for all $i \geq 1,$ the usual long exact sequence of Ext groups associated to the short exact sequence of modules in (5.10) gives us isomorphisms:

(5.12)\begin{equation} {\rm Ext}^{i+1}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^i_{{\rm U}(V)}( K_n , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{equation}

Next, we use the pinned outer automorphism $\phi$ on ${\rm GL}_{mn}(E)$ which is a conjugate of the automorphism $g \rightarrow {}^tg^{-1}$ by a (longest) Weyl group element. The outer automorphism $\phi$ takes standard parabolic subgroups to standard parabolic subgroups and, in particular, takes the parabolic $P_{m,m(n-1)}$ to $P_{m(n-1),m}$. Moreover, for an element $(g_1,g_2)\in {\rm GL}_m(E) \times {\rm GL}_{m(n-1)}(E)$ in the Levi subgroup ${\rm GL}_m(E) \times {\rm GL}_{m(n-1)}(E)$ of $P_{m,m(n-1)}$, one has

\begin{align*} \phi(g_1, g_2) = ({}^tg_2^{-1}, {}^tg_1^{-1}) \in {\rm GL}_{m(n-1)}(E) \times {\rm GL}_m(E). \end{align*}

Applying $\phi$ to the exact sequence (5.10) above, we obtain

\begin{align*} 0 \rightarrow K^\phi_n \rightarrow [\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1)]^\phi \rightarrow {\rm St}(\pi, n)^\phi \rightarrow 0. \end{align*}

By transport of structure,

\begin{align*} [\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1)]^\phi \cong [\nu^{1/2}{\rm St}(\pi, n-1)]^\phi \times [\nu^{-(n-1)/2}\pi]^\phi. \end{align*}

Now by a well-known theorem of Gelfand–Kazhdan, the action of $\phi$ on any irreducible representation of ${\rm GL}_{mn}(E)$ is just the contragredient. Thus, we obtain the exact sequence

\begin{align*} 0 \rightarrow K^\vee_n \rightarrow \nu^{-1/2} {\rm St}(\pi, n-1)^\vee \times (\nu^{(n-1)/2}\pi^\vee) \rightarrow {\rm St}(\pi, n)^\vee \rightarrow 0. \end{align*}

Taking the contragredient of this exact sequence, we get

(5.13)\begin{equation} 0 \rightarrow {\rm St}(\pi,n) \rightarrow \nu^{1/2}{\rm St}(\pi, n-1) \times (\nu^{-(n-1)/2}\pi) \rightarrow K_n \rightarrow 0. \end{equation}

Once again, we have

(5.14)\begin{equation} {\rm Ext}^i_{{\rm U}(V)}( \nu^{1/2} {\rm St}(\pi, n-1) \times \nu^{-(n-1)/2}\pi, \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{equation}

As for (5.11), this follows by Lemma 5.6(iii), which gives the vanishing of ${\rm Ext}^i$ ($i \geq 0$) for the subquotient of the principal series $\nu ^{1/2} {\rm St}(\pi, n-1) \times \nu ^{-(n-1)/2}\pi$ supported on a non-open orbit, and the vanishing of ${\rm Ext}^i$ ($i \geq 1$) for the open orbits is a consequence of the induction hypothesis and the Kunneth theorem.

Equipped with this vanishing of ${\rm Ext}^i_{{\rm U}(V)}( \nu ^{1/2} {\rm St}(\pi, n-1) \times \nu ^{-(n-1)/2}\pi, \omega _{V, \psi, \mu })$ for all $i \geq 1,$ the usual long exact sequence of Ext groups associated to the short exact sequence of modules in (5.13) gives us isomorphisms:

(5.15)\begin{equation} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^{i+1}_{{\rm U}(V)}( K_n , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{equation}

Using the isomorphisms in (5.12) and (5.15), we get

\begin{align*} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^{i+2}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{align*}

Now for any two smooth representations $\pi _1,\pi _2$ of a reductive $p$-adic group $G(F)$, one has

\begin{align*} {\rm Ext}^i_G[\pi_1,\pi_2]=0 \quad \text{for any $i> $ the $F$-rank of $G$.} \end{align*}

Hence, we deduce by (5.15) that

\begin{align*} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1, \end{align*}

completing the proof of the theorem.

7.1 Non-tempered GGP and relevance

In [Reference Gan, Gross and PrasadGGP20], we considered the problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N) = \text{$0$ or $1$ } \end{align*}

where $\pi _M$ and $\pi _N$ are, respectively, irreducible representations of ${\rm GL}_{n+1}(F)$ and ${\rm GL}_n(F)$ of Arthur type, with associated $A$-parameters

\begin{align*} M_A = \bigoplus_{i=1}^k M_i \boxtimes {\rm Sym}^{d_i-1}(\mathbb{C}^2) \quad \text{and} \quad N_A = \bigoplus_{i=1}^l N_i \boxtimes {\rm Sym}^{e_i-1}(\mathbb{C}^2). \end{align*}

Here, $M_i$ and $N_i$ are irreducible bounded admissible representations of the Weil–Deligne group $WD_F$ and ${\rm Sym}^{d-1}(\mathbb {C}^2)$ is the $d$-dimensional irreducible representation of ${\rm SL}_2(\mathbb {C})$ (the Arthur ${\rm SL}_2(\mathbb {C})$), so that $M_A$ and $N_A$ are representations of $WD_F \times {\rm SL}_2(\mathbb {C})$ of dimension $n+1$ and $n$, respectively. The associated $A$-packets are singletons, containing the irreducible unitary principal series representations:

\begin{align*} \pi_M = \times_{i=1}^r {\rm Sp}( \pi_{M_i}, d_i) \quad \text{and} \quad \pi_N = \times_{i=1}^{l} {\rm Sp}(\pi_{N_i}, e_i), \end{align*}

where $\pi _{M_i}$ refers to the irreducible representation of the appropriate ${\rm GL}$ with $L$-parameter $M_i$ and ${\rm Sp}(\pi _{M_i}, d_i)$ denotes the associated Speh representation (as introduced in § 5.5).

Remark We take this opportunity to correct a misnomer in [Reference Gan, Gross and PrasadGGP20, § 5]. In the first paragraph of [Reference Gan, Gross and PrasadGGP20, p. 2312], the representation with $A$-parameter $M_i \otimes {\rm Sym}^{d_i} (\mathbb {C}^2)$ was denoted by ${\rm Speh}(\pi _{M_i}, d_i)$. Though just a naming convention, it is more customary to denote this representation by ${\rm Speh}(\pi _{M_i}, d_i+1)$. We have followed the latter convention here.

Now the main conjecture in [Reference Gan, Gross and PrasadGGP20] (for the general linear groups) is that

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N) = 1 \end{align*}

if and only if the pair $(M_A, N_A)$ is a relevant pair of $A$-parameters. This conjecture has now been proven by Chan [Reference ChanCha22]. Our goal here is to recall the key notion of ‘relevance’ and make a couple of remarks about it, especially in the context of classical groups.

Definition 7.1 Given two $A$-parameters (of arbitrary dimensions) of ${\rm GL}$-groups

\begin{align*} M_A = \bigoplus_{i=0}^d M_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2) \quad \text{and} \quad N_A = \bigoplus_{i=0}^d N_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

we say that $(M_A, N_A)$ is a relevant pair if we have a decomposition of the respective representations of $WD_F$ as

\begin{align*} M_i = M_i^+ + M_i^- \quad \text{and} \quad N_i = N_i^+ + N_i^- \end{align*}

with the property that

\begin{align*} M_i^+ = N_{i+1}^- \quad \text{for $i \geq 0$}\quad \text{and} \quad M_i^- = N_{i-1}^+ \quad \text{for $i \geq 1$.} \end{align*}

This combinatorial definition has a more geometric interpretation which was discussed in [Reference Gan, Gross and PrasadGGP20, § 4].

7.2 Relevance for classical groups

We make a few remarks on the relevance condition for classical groups, clarifying [Reference Gan, Gross and PrasadGGP20].

$\bullet$ The first point is minor but worth noting. The typical GGP conjecture (in the context of ${\rm GL}_n \times {\rm GL}_{n+1}$ say) is formulated as the branching problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)^{\Delta}} (\pi_M \otimes \pi_N, \mathbb{C}), \quad \text{rather than } \quad \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N). \end{align*}

When formulated in this way, the non-tempered GGP conjecture would then say that

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)^{\Delta}} (\pi_M \otimes \pi_N, \mathbb{C}) = 1 \end{align*}

if and only if $(M_A, N_A^{\vee })$ is relevant, where $N_A^{\vee }$ is the dual representation of $N_A$.

$\bullet$ Second, in [Reference Gan, Gross and PrasadGGP20, § 6], we formulated the non-tempered GGP conjecture for the classical groups, asserting that the same ‘relevance’ condition plays a crucial role. We take this opportunity to explicate the relevance notion here.

For classical groups, the branching problem concerns the determination of

\begin{align*} \dim {\rm Hom}_H(\pi, \nu), \end{align*}

where $\pi$ is an irreducible representation of

\begin{align*} G= G_1 \times G_2 = {\rm U}_n \times {\rm U}_m \quad \text{(say),} \end{align*}

with $n\geq m$,

\begin{align*} H= {\rm U}_m\ltimes N \subset G \end{align*}

is a subgroup with unipotent radical $N$ and $\nu$ is a certain small representation of $H$. More precisely, $\nu$ is a one-dimensional character if $n \not \equiv m \bmod 2$; this case is referred to as the Bessel case for Hermitian spaces. On the other hand, the case when $n \equiv m \bmod 2$ is referred to as the Fourier–Jacobi case for skew-Hermitian spaces; in this case, $\nu$ is a Weil representation.

The $A$-parameters for classical groups are likewise finite-dimensional representations of $WD_E \times {\rm SL}_2(\mathbb {C})$, where $WD_E$ is the Weil–Deligne group of $E$, with appropriate (conjugate)-duality conditions. Suppose we are given $A$-parameters

\begin{align*} M_A = \bigoplus_{i=0}^d M_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

\begin{align*} N_A = \bigoplus_{i=0}^d N_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

with $M_i$ and $N_i$ satisfying appropriate (conjugate-)duality conditions. We can now summarize the relevance conditions required in each case as follows.

1. (i) Orthogonal and symplectic groups (both Bessel and Fourier–Jacobi models): an $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $M_A= M_A^{\vee }$ and $N_A= N_A^{\vee }$ form a relevant pair in the sense of Definition 7.1 for ${\rm GL}_m \times {\rm GL}_n$.

2. (ii) Hermitian case (Bessel models): an $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $M_A^\vee$ and $N_A$ form a relevant pair in the sense of Definition 7.1 for ${\rm GL}_m\times {\rm GL}_n$.

3. (iii) Skew-Hermitian case (Fourier–Jacobi model): in this case, the definition of the Weil representation $\nu$ requires an extra piece of data, namely a character

   \begin{align*} \mu: E^\times \rightarrow \mathbb{C}^\times \quad \text{with $\mu|_{F^\times} = \omega_{E/F}$.} \end{align*}
   An $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $\mu \cdot M_A^\vee$ and $N_A$ are relevant in the sense of Definition 7.1 for ${\rm GL}_m \times {\rm GL}_n$.

7.3 Non-tempered twisted GGP

We shall now formulate the extension of the non-tempered GGP conjecture of [Reference Gan, Gross and PrasadGGP20] to the twisted setting considered in this paper. Hence, with $E/F$ a quadratic extension, suppose we have a representation $\pi _M$ of ${\rm GL}(V)$ with associated $A$-parameter $M_A$. The notion of relevance is not immediately obvious in this setting, as in contrast to the situations discussed above, we do not have a pair of $A$-parameters but only a single one. Nonetheless, we have the following result.

Conjecture 7.2 Let $V$ be an $n$-dimensional $E/F$-skew-Hermitian space. Let $\pi$ be an irreducible admissible representation of ${\rm GL}(V)$ with an $A$-parameter (which is an $n$-dimensional representation of $WD_E \times {\rm SL}_2(\mathbb {C})$) of the form

\begin{align*} M_A = \bigoplus_{i=1}^r M_i \boxtimes {\rm Sym}^{d_i}(\mathbb{C}^2), \end{align*}

where $M_i$ is an irreducible $m_i$-dimensional tempered representation of $WD_E$. If

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] \not = 0, \end{align*}

then $M_A$ is a sum of a tempered $A$-parameter (i.e. with ${\rm SL}_2(\mathbb {C})$ acting trivially) and summands of the form

\begin{align*} N_i \boxtimes {\rm Sym}^{d_i}(\mathbb{C}^2) \oplus \mu\cdot N_i^\sigma \boxtimes {\rm Sym}^{d_i-1}(\mathbb{C}^2), \end{align*}

where $d_i \geq 1$, and the $N_i$ are tempered representations of $WD_E$ with $N^\sigma _i$ their conjugate under the action of ${\rm Gal}(E/F)$. Equivalently, the parameters $M_A$ and $\mu \cdot M_A^\sigma$ should be relevant in the sense of [Reference Gan, Gross and PrasadGGP20].

Conversely, if the parameters $M_A$ and $\mu \cdot M_A^\sigma$ are relevant in the sense of [Reference Gan, Gross and PrasadGGP20], then

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] = \mathbb{C} \end{align*}

for exactly one skew-Hermitian space $V$, namely the one determined as in Conjecture 2.1(iii).