2.1. The case $m \to \infty $

We start by proving the $m \to \infty $ case of Theorem 1.1. We first show that this is equivalent to

(2.4) $$ \begin{align} \mathbf{ASym}_{X_1,\ldots,X_n} & \left[ \prod_{1 \le i < j \le n} \frac{(f(X_j^{-1})-f(X_i))g(X_i) g(X_j^{-1}) X_j^2}{q(1-X_i X_j)} \prod_{i=1}^n h(X_i)^{i-1} \prod_{i=1}^n \frac{1}{1- \prod_{j=i}^n h(X_j)} \right] \nonumber\\ &\qquad= \prod_{i=1}^n f(X_i^{-1}) g(X_i^{-1}) X_i^2 \frac{\prod_{1 \le i < j \le n} \left( f(X_i)-f(X_j) \right) g(X_i) g(X_j)} {\prod_{1 \le i \le j \le n} q (1-X_i X_j)}, \end{align} $$

which is just (1.5) multiplied on both sides with $\prod _{1 \le i < j \le n} (X_j-X_j)$ . To see this, we rewrite the left-hand side of (1.7) by using the summation formula for the geometric series n times. As $m \to \infty $ , $a_{j,m,n}(Q,r;X_i)$ simplifies to

$$ \begin{align*}X^2 f(X_i)^{j-1} g(X_i)^{n-1} f(X_i^{-1}) g(X_i^{-1}) \end{align*} $$

in a formal power series sense, and

$$ \begin{align*}\det_{1 \le i,j \le n} \left( X_i^2 f(X_i)^{j-1} g(X_i)^{n-1} f(X_i^{-1}) g(X_i^{-1}) \right) \end{align*} $$

can be computed using the Vandermonde determinant evaluation, we are led to the right-hand side of (2.4) eventually.

We denote by $L_n(X_1,\ldots ,X_n)$ the left-hand side of (2.4) and observe that the following recursion is satisfied:

(2.5) $$ \begin{align} L_n(X_1,\ldots,X_n) = \sum_{k=1}^n (-1)^{k-1} & \frac{1}{1-\prod_{i=1}^n h(X_i) } L_{n-1}(X_1,\ldots,\widehat{X_k},\ldots,X_n) \nonumber\\ &\qquad \times \prod_{1 \le j \le n, \atop j \not= k} \frac{f(X_j)g(X_j) g(X_k) \left( 1 - f(X_j^{-1})^{-1} f(X_k)\right)}{q(1-X_j X_k)}, \end{align} $$

where $\widehat {X_k}$ means that we omit $X_k$ . Indeed, suppose more generally that

$$ \begin{align*}P(X_1,\ldots,X_n) = \mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i < j \le n} s(X_i,X_j) \prod_{i=1}^n t(X_i)^{i-1} \prod_{i=1}^n \frac{1}{1- \prod_{j=i}^n u(X_j)} \right], \end{align*} $$

then

(2.6) $$ \begin{align} \begin{aligned} P(X_1,\ldots,X_n) &= \sum_{k=1}^n \sum_{\sigma \in {\mathcal S}_n: \atop \sigma(1)=k} \operatorname{\mathrm{sgn}} \sigma \frac{1}{1- \prod_{j=1}^n u(X_j)} \prod_{1 \le j \le n, \atop j \not= k} s(X_k,X_j) t(X_j) \\ &\qquad \times \sigma \left[ \prod_{2 \le i < j \le n} s(X_i,X_j) \prod_{i=2}^n t(X_i)^{i-2} \prod_{i=2}^n \frac{1}{1- \prod_{j=i}^n u(X_j)} \right] \\ &= \sum_{k=1}^n (-1)^{k-1} \frac{1}{1- \prod_{j=1}^n u(X_j)} P(X_1,\ldots,\widehat{X_k},\ldots,X_n) \\ &\qquad \times \prod_{1 \le j \le n, \atop j \not= k} s(X_k,X_j) t(X_j), \end{aligned} \end{align} $$

where we use the notation

$$ \begin{align*}\sigma \left[f(X_1,\ldots,X_n) \right] = f(X_{\sigma(1)},X_{\sigma(2)},\ldots,X_{\sigma(n)}). \end{align*} $$

The last equality in (2.6) follows from the fact that the sign of $\sigma $ is the product of $ (-1)^{k-1}$ and the sign of the restriction of $\sigma $ to $\{2,3,\ldots ,n\}$ , assuming $\sigma (1)=k$ and ‘identifying’ the preimage $\{2,3,\ldots ,n\}$ as well as the image $\{1,\ldots ,n\} \setminus \{k\}$ with $\{1,2,\ldots ,n-1\}$ in the natural way.

We show (2.4) by induction with respect to n. The case $n=1$ is easy to check. It suffices to show that the right-hand side of (2.4) satisfies the recursion (2.5) – that is,

$$ \begin{align*} \prod_{i=1}^n& f(X_i^{-1}) g(X_i^{-1}) X_i^2 \frac{\prod_{1 \le i < j \le n} \left( f(X_j)-f(X_i) \right) g(X_i) g(X_j)} {\prod_{1 \le i \le j \le n} q (1-X_i X_j)} \\ & = \sum_{k=1}^n (-1)^{k-1} \frac{1}{1-\prod_{i=1}^n h(X_i) } \prod_{1 \le j \le n, \atop j \not= k} \frac{f(X_j)g(X_j)g(X_j^{-1}) g(X_k) \left( f(X_k)-f(X_j^{-1}) \right) X_j^2}{q(1-X_j X_k)} \\ & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \times \frac{\prod_{1 \le i < j \le n, i,j \not= k} \left( f(X_j)-f(X_i) \right) g(X_i) g(X_j)} {\prod_{1 \le i \le j \le n, i,j \not= k} q (1-X_i X_j)}. \end{align*} $$

We multiply by $\left (1 - \prod _{i=1}^n h(X_i) \right ) \prod _{1 \le i \le j \le n} q (1-X_i X_j)$ and obtain

(2.7) $$ \begin{align} & \left(\prod_{i=1}^{n} f(X_i^{-1}) g(X_i^{-1}) X_i^2 - \prod_{i=1}^{n} f(X_i) g(X_i) \right) \prod_{1 \le i < j \le n} (f(X_j)-f(X_i)) g(X_i) g(X_j) \nonumber\\ & = \prod_{1 \le i < j \le n} (f(X_j)-f(X_i)) g(X_i) g(X_j) \sum_{k=1}^n q (1-X_k^2) \prod_{1 \le j \le n \atop j \not= k} \frac{f(X_j) g(X_j^{-1})( f(X_k)-f(X_j^{-1})) X_j^2}{f(X_k) - f(X_j)}. \end{align} $$

Note that the sign $(-1)^{k+1}$ in the sum has disappeared as we have pulled out the factor $\prod _{1 \le i < j \le n} (f(X_j)-f(X_i))$ .

By the definitions of $f(X)$ and $g(X)$ , this is

$$ \begin{align*} \bigg( \prod_{i=1}^n (q+X_i) - \prod_{i=1}^n X_i & (1+q X_i) \bigg) \prod_{1 \le i < j \le n} q (X_j - X_i)(1+q X_i + q X_j + w X_i X_j) \\[5pt] &\quad = \prod_{1 \le i < j \le n} q (X_j - X_i)(1+q X_i + q X_j + w X_i X_j) \\[5pt] & \times \sum_{k=1}^n (q-X_k^2) \prod_{1 \le j \le n \atop j \not= k} \frac{X_j ( 1 + q X_j)(1-X_j X_k) (q + X_j + w X_k + q X_j X_k)}{(X_j-X_k)(1 + q X_j + q X_k + w X_j X_k)}. \end{align*} $$

For each $s \in \{1,2,\ldots ,n\}$ , both sides are polynomials in $X_s$ of degree not greater than $2n$ . It is not hard to see that both sides vanish for $X_s = X_t$ and $X_s =-\frac {q(1+ q X_t)}{q + w X_t}$ for any $t \in \{1,2,\ldots ,n\} \setminus \{s\}$ . Moreover, it is also not hard to see that the evaluations also agree for $X_s=0,-q^{-1}$ , which gives a total of $2n$ evaluations for each $X_s$ : due to the factors $X_j$ and $1+q X_j$ , all summands on the right-hand side vanish when setting $X_s=0,-q^{-1}$ , except for the one for $k=s$ . This summand can easily be seen to coincide with the specialization of the left-hand side.

It follows that the difference of the left-hand side and the right-hand side is up to a constant in $\mathbb {Q}(q,w)$ equal to

(2.8) $$ \begin{align} \prod_{i=1}^n X_i(1+ q X_i) \prod_{1 \le i < j \le n} (X_j - X_i)(1 + q X_i + q X_j + w X_i X_j). \end{align} $$

To show that this constant is indeed zero, we consider the following specialization

$$ \begin{align*}(X_1,X_2,X_3,X_4,\ldots) = \left(X_1,\frac{1}{X_1},X_3,\frac{1}{X_3},\ldots \right).\end{align*} $$

Note first that (2.8) does not vanish at this specialization, and therefore, it suffices to show that the left-hand side and the right-hand side of (2.7) agree on this specialization. If n is even, this is particularly easy to see because both sides vanish (on the right-hand side, all summands vanish, which is due to the factor $1-X_j X_k$ ). If n is odd, then only the last summand on the right-hand side remains, and it is not hard to see that it is equal to the left-hand side.

2.2. The general case

In order to prove Theorem 1.1, we need to show

(2.9) $$ \begin{align} \det_{1 \le i, j \le n} \left( a_{j,m,n}(q,w;X_i) \right) = \mathbf{ASym}_{X_1,\ldots,X_n} F(m;X_1,\ldots,X_n), \end{align} $$

where

$$ \begin{align*} F(m;X_1,\ldots,X_n)= \prod_{i=1}^n (1-X_i^2) \prod_{1 \le i < j \le n} q^{-1} & (f(X_j^{-1}) - f(X_i))g(X_i) g(X_j^{-1}) X_j^2 \\[5pt] &\quad \times \sum_{0 \le k_1 < k_2 < \ldots < k_n \le m} h(X_1)^{k_1} h(X_2)^{k_2} \cdots h(X_n)^{k_n}. \end{align*} $$

See also (2.2). Observe that we have the following recursion:

$$ \begin{align*} F(m;X_1,\ldots,X_n) = (1-X_1^2) \prod_{j=2}^n q^{-1} & (f(X_j^{-1}) - f(X_1))g(X_1) g(X_j^{-1}) X_j^2 \\ & \times \sum_{l=0}^m h(X_1)^{-1} \left( \prod_{i=1}^n h(X_i) \right)^{l+1} F(m-1-l;X_2,\ldots,X_n). \end{align*} $$

We set

$$ \begin{align*}A(m;X_1,\ldots,X_n) = \mathbf{ASym}_{X_1,\ldots,X_n} F(m;X_1,\ldots,X_n) \end{align*} $$

and observe that

$$ \begin{align*} A(m;X_1,\ldots,X_n) &= \sum_{k=1}^n \sum_{l=0}^m (-1)^{k+1} (1-X_k^2) h(X_k)^{-1} \left( \prod_{i=1}^n h(X_i) \right)^{l+1} \\ &\quad \times A(m-l-1;X_1,\ldots,\widehat{X_k},\ldots,X_n) \\ &\quad \times \prod_{1 \le i \le n, i \not= k} q^{-1} (f(X_i^{-1}) - f(X_k))g(X_k) g(X_i^{-1}) X_i^2, \end{align*} $$

by the same argument that has led to (2.6). By the induction hypothesis, we have

$$ \begin{align*}A(m-l-1;X_1,\ldots,\widehat{X_k},\ldots,X_n) = \det_{1 \le i \le n, i \not= k \atop 1 \le j \le n-1} \left( a_{j,m-l-1,n-1}(q,w;X_i) \right). \end{align*} $$

Therefore, the right-hand side of (2.9) is

(2.10) $$ \begin{align} \sum_{k=1}^n (-1)^{k+1} (1-X_k^2) \prod_{1 \le i \le n, i \not= k} q^{-1} & (f(X_i^{-1}) - f(X_k))g(X_k) g(X_i^{-1}) X_i^2 \nonumber\\ & \times \sum_{l=0}^m h(X_k)^{l} \det_{1 \le i \le n, i \not= k \atop 1 \le j \le n-1} \left( h(X_i)^{l+1} a_{j,m-l-1,n-1}(q,w;X_i) \right), \end{align} $$

and we need to show that it is equal to $\det _{1 \le i, j \le n} \left ( a_{j,m,n}(q,w;X_i) \right )$ .

Using (2.1) and (2.3), we note that

$$ \begin{align*}h(X)^{l+1} a_{j,m-l-1,n-1}(q,w;X) = q^{-j} \left( f(X)^{j} g(X)^{n-1} h(X)^l - X^{2n-2} f(X^{-1})^{j} g(X^{-1})^{n-1} h(X)^{m+1} \right), \end{align*} $$

and thus, we can write the determinant in (2.10) as

$$ \begin{align*}\sum_{\sigma, S} (-1)^{I(\sigma)+|S|} \prod_{i \in S} q^{-\sigma(i)} X_i^{2n-2} f(X_i^{-1})^{\sigma(i)} g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \in \overline{S}} q^{-\sigma(i)} f(X_i)^{\sigma(i)} g(X_i)^{n-1} h(X_i)^l, \end{align*} $$

where the sum is over all bijections $\sigma : \{1,2,\ldots ,n \} \setminus \{k\} \to \{1,2,\ldots ,n-1\}$ , all subsets S of $ \{1,2,\ldots ,n \} \setminus \{k\}$ and $I(\sigma )$ is the number of all inversions (i.e., pairs $i,j \in \{1,2,\ldots ,n \} \setminus \{k\} $ with $i<j$ and $\sigma (i)>\sigma (j)$ ). Moreover, $\overline {S}$ denotes the complement of S in $\{1,2,\ldots ,n \} \setminus \{k\}$ . Also note that $(-1)^{I(\sigma )}$ is just the sign of the permutation as it appears in the Leibniz formula of the determinant when expanding it over all permutations. Comparing with (2.10), we multiply by $h(X_k)^{l}$ and take the sum over l. We obtain

$$ \begin{align*} \sum_{\sigma, S} (-1)^{I(\sigma)+|S|} \prod_{i \in S} q^{-\sigma(i)} X_i^{2n-2} f(X_i^{-1})^{\sigma(i)} g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \in \overline{S}} & q^{-\sigma(i)} f(X_i)^{\sigma(i)} g(X_i)^{n-1} \\[4pt] & \times \sum_{l=0}^m h(X_k)^l \prod_{i \in \overline{S}} h(X_i)^l. \end{align*} $$

We evaluate the sum and rearrange some terms:

$$ \begin{align*} \sum_{S} (-1)^{|S|} q^{-n+1} \frac{1- \prod_{i \in \overline{S} \cup \{k\}} h(X_i)^{m+1}}{1-\prod_{i \in \overline{S} \cup \{k\}} h(X_i)} & \prod_{i \in S} X_i^{2n-2} f(X_i^{-1}) g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \in \overline{S}} f(X_i) g(X_i)^{n-1} \\[4pt] & \times \sum_{\sigma} (-1)^{I(\sigma)} \prod_{i \in S} \left[q^{-1} f(X_i^{-1})\right]^{\sigma(i)-1} \prod_{i \in \overline{S}} \left[ q^{-1} f(X_i) \right]^{\sigma(i)-1}. \end{align*} $$

The inner sum is a Vandermonde determinant, which we evaluate. We obtain

$$ \begin{align*} \sum_{S} (-1)^{|S|} q^{-n+1} \frac{1- \prod_{i \in \overline{S} \cup \{k\}} h(X_i)^{m+1}}{1-\prod_{i \in \overline{S} \cup \{k\}} h(X_i)} \prod_{i \in S} X_i^{2n-2} f(X_i^{-1}) & g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \in \overline{S}} f(X_i) g(X_i)^{n-1} \\[4pt] &\quad \times \prod_{1 \le i < j \le n, i,j \not=k} q^{-1} (f(Y_j) - f(Y_i)), \end{align*} $$

with $Y_i=X_i$ if $i \in \overline {S}$ and $Y_i=X_i^{-1}$ if $i \in S$ .

From (2.10), we add the remaining factors and the sum over all k and finally have the full right-hand side of (2.9). We exchange the sum over k and S: now we sum over all proper subsets $S \subseteq [n]$ and all k not in S. If we write $i \notin S$ , then we mean $i \in \{1,2,\ldots ,n\} \setminus S$ . This gives

(2.11) $$ \begin{align} \sum_{S} (-1)^{|S|} & q^{-n+1} \frac{1- \prod_{i \notin S} h(X_i)^{m+1}} {1- \prod_{i \notin S} h(X_i)} \prod_{i \in S} X_i^{2n-2} f(X_i^{-1}) g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \notin S} f(X_i) g(X_i)^{n-1} \nonumber\\[4pt] & \times \sum_{k \notin S} (-1)^{k+1} (1-X_k^2) f(X_k)^{-1} \prod_{1 \le i \le n, i \not= k} q^{-1} (f(X_i^{-1}) - f(X_k)) g(X_i^{-1}) X_i^2 \nonumber\\[4pt] &\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \times \prod_{1 \le i < j \le n, i,j \not=k} q^{-1} (f(Y_j) - f(Y_i)). \end{align} $$

We rearrange (2.11) slightly as follows

$$ \begin{align*} \sum_{S} (-1)^{|S|} q^{-n+1} & \frac{1- \prod_{i \notin S} h(X_i)^{m+1}} {1- \prod_{i \notin S} h(X_i)} \prod_{i \in S} X_i^{2n} f(X_i^{-1}) g(X_i^{-1})^{n} h(X_i)^{m+1} \prod_{i \notin S} g(X_i)^{n-1} \\[4pt] & \qquad\qquad\times \sum_{k \notin S} (-1)^{k+1} (1-X_k^2) \prod_{i \notin S \cup \{k\}} f(X_i) g(X_i^{-1}) X_i^2 \\[4pt] & \qquad\qquad\qquad\qquad\times \prod_{1 \le i \le n \atop i \not=k} q^{-1} \left(f(X_i^{-1}) - f(X_k) \right) \prod_{1 \le i < j \le n, i,j \not=k} q^{-1} (f(Y_j) - f(Y_i)). \end{align*} $$

This is further equal to

(2.12) $$ \begin{align} \sum_{S} (-1)^{|S|} & q^{-n+1} \frac{1- \prod_{i \notin S} h(X_i)^{m+1}} {1- \prod_{i \notin S} h(X_i)} \prod_{i \in S} X_i^{2n} f(X_i^{-1}) g(X_i^{-1})^{n} h(X_i)^{m+1} \prod_{i \notin S} g(X_i)^{n-1} \nonumber\\ & \qquad\quad\times \prod_{1 \le i < j \le n \atop \{i,j\} \cap S \not= \emptyset} q^{-1} (f(Y_j) - f(Y_i)) \prod_{1 \le i < j \le n \atop \{i,j\} \cap S = \emptyset} q^{-1} (f(X_j) - f(X_i)) \nonumber\\ & \qquad\qquad\qquad\times \sum_{k \notin S} (1-X_k^2) \prod_{i \notin S \cup \{k\}} \frac{ f(X_i) g(X_i^{-1}) \left( f(X_k) - f(X_i^{-1}) \right) X_i^2}{f(X_k)-f(X_i)}. \end{align} $$

We divide (2.7) by $q \prod _{1 \le i < j \le n} (f(X_j) - f(X_i))g(X_i) g(X_j)$ and obtain

$$ \begin{align*} q^{-1} \bigg(\prod_{i=1}^{n} f(X_i^{-1}) g(X_i^{-1}) X_i^2 - \prod_{i=1}^{n} & f(X_i) g(X_i) \bigg) \\ &\quad = \sum_{k=1}^n (1-X_k^2) \prod_{1 \le j \le n \atop j \not= k} \frac{f(X_j) g(X_j^{-1})( f(X_k)-f(X_j^{-1})) X_j^2}{f(X_k) - f(X_j)}. \end{align*} $$

By applying this to the variables $(X_i)_{i \in S}$ , we can use this to replace the sum over all $k \notin S$ in (2.12) by something simpler:

$$ \begin{align*} \sum_{S} (-1)^{|S|} q^{-n} & \frac{1- \prod_{i \notin S} h(X_i)^{m+1}} {1- \prod_{i \notin S} h(X_i)} \prod_{i \in S} X_i^{2n} f(X_i^{-1}) g(X_i^{-1})^{n} h(X_i)^{m+1} \prod_{i \notin S} g(X_i)^{n-1} \\ & \qquad\quad \times \prod_{1 \le i < j \le n \atop \{i,j\} \cap S \not= \emptyset} q^{-1} (f(Y_j) - f(Y_i)) \prod_{1 \le i < j \le n \atop \{i,j\} \cap S = \emptyset} q^{-1} (f(X_j) - f(X_i)) \\ &\qquad\qquad\qquad\qquad\qquad\qquad\qquad \times \left(\prod_{i \notin S} f(X_i^{-1}) g(X_i^{-1}) X_i^2 - \prod_{i \notin S} f(X_i) g(X_i) \right). \end{align*} $$

We rearrange terms and take into account that $Y_i=X_i$ if $i \notin S$ (extending the definition slightly by setting $Y_k=X_k$ ). After some cancellation, we obtain

$$ \begin{align*} q^{-n} \sum_{S} (-1)^{|S|}& (1- \prod_{i \notin S} h(X_i)^{m+1}) \prod_{i=1}^n X_i^{n+1} f(X_i^{-1}) g(X_i^{-1}) \\ & \times \prod_{i \in S} X_i^{n-1} g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \notin S} X_i^{-n+1} g(X_i)^{n-1} \prod_{1 \le i < j \le n} q^{-1} (f(Y_j) - f(Y_i)). \end{align*} $$

Using the Vandermonde determinant formula and the fact that $Y_i=X_i^{-1}$ if $i \in S$ and $Y_i=X_i$ if $i \notin S$ , this is equal to

$$ \begin{align*} q^{-\binom{n+1}{2}} \sum_{S, \sigma} & (-1)^{|S|+I(\sigma)} (1- \prod_{i \notin S} h(X_i)^{m+1}) \prod_{i=1}^n X_i^{n+1} f(X_i^{-1}) g(X_i^{-1}), \\ &\qquad \times \prod_{i \in S} X_i^{n-1} f(X_i^{-1})^{\sigma(i)-1} g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \notin S} X_i^{-n+1} f(X_i)^{\sigma(i)-1} g(X_i)^{n-1},\end{align*} $$

which we expand as follows:

(2.13) $$ \begin{align} q^{-\binom{n+1}{2}} & \sum_{S, \sigma} (-1)^{|S|+I(\sigma)} \prod_{i=1}^n X_i^{n+1} f(X_i^{-1}) g(X_i^{-1}) \nonumber\\ & \times \prod_{i \in S} X_i^{n-1} f(X_i^{-1})^{\sigma(i)-1} g(X_i^{-1})^{n-1} h(X_i)^{m+1} \prod_{i \notin S} X_i^{-n+1} f(X_i)^{\sigma(i)-1} g(X_i)^{n-1} \nonumber\\ & \qquad\qquad\quad- q^{-\binom{n+1}{2}} \prod_{i} X_i^{n+1} f(X_i^{-1}) g(X_i^{-1}) h(X_i)^{m+1} \nonumber\\ & \sum_{S, \sigma} (-1)^{|S|+I(\sigma)} \prod_{i \in S} X_i^{n-1} f(X_i^{-1})^{\sigma(i)-1} g(X_i^{-1})^{n-1} \prod_{i \notin S} X_i^{-n+1} f(X_i)^{\sigma(i)-1} g(X_i)^{n-1}, \end{align} $$

for reasons that become clear next. Recall that the sums are over all proper subsets S, but since the sums are equal for $S=\{1,2,\ldots ,n\}$ , we can also sum over all subsets S. The sum in the second term can be written as

$$ \begin{align*} \sum_{S, \sigma} (-1)^{|S|+I(\sigma)} \prod_{i \in S} X_i^{n-1} & f(X_i^{-1})^{\sigma(i)-1} g(X_i^{-1})^{n-1} \prod_{i \notin S} X_i^{-n+1} f(X_i)^{\sigma(i)-1} g(X_i)^{n-1} \\ & = \det_{1 \le i, j \le n}\left( X_i^{-n+1} g(X_i)^{n-1} f(X_i)^{j-1} - X_i^{n-1}g(X_i^{-1})^{n-1} f(X_i^{-1})^{j-1} \right). \end{align*} $$

By the definitions of $f(X)$ and $g(X)$ , this is equal to

$$ \begin{align*}\prod_{i=1}^n X_i^{-n+1} \det_{1 \le i, j \le n}\left( X_i^{j-1} (q+w X_i)^{n-j} (1+q X_i)^{j-1} - X_i^{n-j}(q X_i + w )^{n-j} (X_i+ q )^{j-1} \right). \end{align*} $$

The determinant can be seen to vanish as follows: First, observe that it is a polynomial in $X_1,\ldots ,X_n$ of degree no greater than $2n-2$ in each $X_i$ . For $1 \le i < j \le n$ , the i-th row and the j-th row of the underlying matrix are collinear when setting $X_i=X_j$ or $X_i= X_j^{-1}$ . Moreover, the i-th row vanishes when setting $X_i^2=1$ . It follows that $\prod _{i=1}^{n} (X_i^2-1) \prod _{1 \le i < j \le n} (X_j-X_i)(1-X_i X_j)$ is a divisor of the determinant, but since it is of degree $2n$ in each $X_i$ , the determinant vanishes. The first expression in (2.13) is obviously equal to

$$ \begin{align*}\det_{1 \le i, j \le n} \left( q^{-j} X_i^2 f(X_i^{-1}) g(X_i^{-1}) f(X_i)^{j-1} g(X_i)^{n-1} - q^{-j} X_i^{2n} f(X_i^{-1})^{j} g(X_i^{-1})^n h(X_i)^{m+1} \right), \end{align*} $$

and this is equal to $\det _{1 \le i,j \le n} \left ( a_{j,m,n}(q,w;X_i) \right )$ ; see (2.3) as well as the expression for $h(X)$ in terms of $f(X)$ and $g(X)$ . This concludes the proof of Theorem 1.1.

3.1. Arrowed Gelfand-Tsetlin patterns

To continue the analogy with the ordinary Littlewood identity (1.1) and Macdonald’s bounded version (1.6) of it, both sides of the identities (1.4) and (1.8) will be interpreted combinatorially. For the left-hand side, this was accomplished in another recent paper [Reference Fischer and Schreier-Aigner13], and we will describe the result and adjust to our context next.

In order to motivate the definition for the combinatorial objects, recall the combinatorial interpretation of the left-hand sides of (1.1) and (1.6) in terms of Gelfand-Tsetlin patterns, which is described in Appendix A.3. We need to extend the discussion from there insofar that there is also a sensible extension of the definition of Gelfand-Tsetlin patterns to arbitrary integers sequences $(\lambda _1,\ldots ,\lambda _n)$ . The notion of signed intervals is crucial for this:

$$ \begin{align*} \underline{[a,b]} = \begin{cases} [a,b], & a \le b \\\emptyset, & b=a-1 \\ [b+1,a-1], & b < a-1 \end{cases}. \end{align*} $$

If we are in the last case, then the interval is said to be negative. The condition that defines a Gelfand-Tsetlin pattern can also be written as $a_{i,j} \in [a_{i+1,j},a_{i+1,j+1}]$ . If the bottom row is weakly increasing, we can replace this condition also by $a_{i,j} \in \underline {[a_{i+1,j},a_{i+1,j+1}]}$ (since we then have $a_{i+1,j} \le a_{i+1,j+1}$ as can be seen inductively with respect to n).

We use this now as the definition for arbitrary bottom rows: A (generalized) Gelfand-Tsetlin pattern is a triangular array $A=(a_{i,j})_{1 \le j \le i \le n}$ of integers with $a_{i,j} \in \underline {[a_{i+1,j},a_{i+1,j+1}]}$ for all $i,j$ . Then the sign of a Gelfand-Tsetlin pattern A is

$$ \begin{align*} (-1)^{\# \text{ of negative intervals} \underline{[a_{i+1,j},a_{i+1,j+1}]}}=: \operatorname{\mathrm{sgn}} A. \end{align*} $$

Then

(3.1) $$ \begin{align} s_{(\lambda_1,\ldots,\lambda_n)}(X_1,\ldots,X_n) = \sum_{A=\left( a_{i,j} \right)_{1 \le j \le i \le n}} \operatorname{\mathrm{sgn}} A \prod_{i=1}^n X_i^{\sum_{j=1}^i a_{i,j} - \sum_{j=1}^{i-1} a_{i-1,j}}, \end{align} $$

where the sum is over all Gelfand-Tsetlin patterns $A=(a_{i,j})_{1 \le j \le i \le n}$ with bottom row $(\lambda _n,\lambda _{n-1},\ldots ,\lambda _1)$ and

$$ \begin{align*} s_{(\lambda_1,\ldots,\lambda_n)}(X_1,\ldots,X_n) = \frac{\det_{1 \le i,j \le n} \left( X_i^{\lambda_j+n-j} \right)}{\prod_{1 \le i < j \le n} (X_i-X_j)}.\end{align*} $$

This result is a special case of Theorem 3.4 below that will also cover the combinatorial interpretation of the left-hand side of (1.4) and (1.8). However, this special case appeared essentially also earlier in [Reference Fischer8] (with some details missing).

Definition 3.1. An arrowed Gelfand-Tsetlin pattern (AGTP)Footnote ¹ is a triangular array of the following form:

$$ \begin{align*} \begin{array}{ccccccccccccccccc} & & & & & & & & a_{1,1} & & & & & & & & \\ & & & & & & & a_{2,1} & & a_{2,2} & & & & & & & \\ & & & & & & \dots & & \dots & & \dots & & & & & & \\ & & & & & a_{n-2,1} & & \dots & & \dots & & a_{n-2,n-2} & & & & & \\ & & & & a_{n-1,1} & & a_{n-1,2} & & \dots & & \dots & & a_{n-1,n-1} & & & & \\ & & & a_{n,1} & & a_{n,2} & & a_{n,3} & & \dots & & \dots & & a_{n,n}, & & & \end{array}, \end{align*} $$

where each entry $a_{i,j}$ is an integer decorated with an element from $\{\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow ,\emptyset \}$ and the following is satisfied for each entry a not in the bottom row: Suppose b is the $\swarrow $ -neighbor of a and c is the $\searrow $ -neighbor of a, respectively – that is,

$$ \begin{align*} \begin{array}{ccc} &a& \\ b&&c \end{array}. \end{align*} $$

Depending on the decoration of $b, c$ , denoted by $\operatorname {decor}(b)$ and $\operatorname {decor} ( c )$ , respectively, we need to consider four cases:

• ○ $(\operatorname {decor}(b),\operatorname {decor}( c )) \in \{\nwarrow ,\emptyset \} \times \{\nearrow , \emptyset \}$ : $a \in \underline {[b,c]}$ .

• ○ $(\operatorname {decor}(b),\operatorname {decor}( c )) \in \{\nwarrow ,\emptyset \} \times \{\nwarrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ : $a \in \underline {[b,c-1]}$ .

• ○ $(\operatorname {decor}(b),\operatorname {decor}( c )) \in \{\nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \} \times \{\nearrow ,\emptyset \}$ : $a \in \underline {[b+1,c]}$ .

• ○ $(\operatorname {decor}(b),\operatorname {decor}( c)) \in \{\nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \} \times \{\nwarrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ : $a \in \underline {[b+1,c-1]}$ .

An example is provided next. We write $^\nwarrow e, e^\nearrow , ^\nwarrow e^\nearrow , e$ if the entry e is decorated with $\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow ,\emptyset $ , respectively.

$$ \begin{align*} \begin{array}{ccccccccccccccccc} & & & & & & & & ^\nwarrow2 & & & & & & & & \\ & & & & & & & 2 & & ^\nwarrow3^\nearrow & & & & & & & \\ & & & & & & ^\nwarrow2 & & 2^\nearrow & & 3^\nearrow & & & & & & \\ & & & & & 3 & & ^\nwarrow2 & & ^\nwarrow3^\nearrow & & ^\nwarrow3^\nearrow & & & & & \\ & & & & 2^\nearrow & & 4 & & ^\nwarrow2^\nearrow & & 3^\nearrow & & 2 & & & & \\ & & & ^\nwarrow6 & & ^\nwarrow2^\nearrow & & 5 & & 1^\nearrow & & ^\nwarrow4 & & ^\nwarrow2^\nearrow & & &\end{array}\end{align*} $$

We define the sign of an AGTP $A=(a_{i,j})_{1 \le j \le i \le n}$ as follows: Each negative interval $\underline {[a_{i+1,j}(+1),a_{i+1,j+1}(-1)]}$ with $i \ge 1$ and $j \le i$ contributes a multiplicative $-1$ , choosing $a_{i+1,j}+1$ iff $\operatorname {decor}(a_{i+1,j}) \in \{\nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ and $a_{i+1,j}$ otherwise, and choosing $a_{i+1,j+1} - 1$ iff $\operatorname {decor}(a_{i+1,j+1}) \in \{ \nwarrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ and $a_{i+1,j+1}$ otherwise. There are no negative intervals in rows $1,2,3$ , two in rows $4,5$ and three in row $6$ , so that the sign of the pattern is $-1$ .

We associate the following weight to a given arrowed Gelfand-Tsetlin pattern $A=(a_{i,j})_{1 \le j \le i \le n}$ :

$$ \begin{align*} {\operatorname{W}}(A) = \operatorname{\mathrm{sgn}}(A) t^{\# \emptyset} u^{\# \nearrow} v^{\# \nwarrow} w^{\# \nwarrow \!\!\!\!\!\;\!\! \nearrow} \prod_{i=1}^{n} X_i^{\sum_{j=1}^i a_{i,j} - \sum_{j=1}^{i-1} a_{i-1,j} + \# \nearrow \text{in row} i - \# \nwarrow \text{in row} i}. \end{align*} $$

The weight of our example is

$$ \begin{align*}- t^5 u^5 v^5 w^6 X_1 X_2^3 X_3^3 X_4^3 X_5^4 X_6^6. \end{align*} $$

For this paper, only arrowed Gelfand-Tsetlin patterns with weakly increasing bottom row are relevant, and in this case, the description of the objects can be simplified considerably as follows.

Proof. Suppose $(a_{i,j})_{1 \le j \le i \le n}$ is an AGTP. If $a_{i+1,j} < a_{i+1,j+1}$ for particular $i,j$ , then $a_{i+1,j} \le a_{i,j} \le a_{i+1,j+1}$ . The first inequality has to be strict if the decoration of $a_{i+1,j}$ contains an arrow pointing toward $a_{i,j}$ (i.e., $\operatorname {decor}(a_{i+1,j}) \in \{\nearrow ,\nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ ), while the second inequality has to be strict if $a_{i+1,j+1}$ contains an arrow pointing toward $a_{i,j}$ (i.e., $\operatorname {decor}(a_{i+1,j+1}) \in \{\nwarrow ,\nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ ).

However, if $a_{i+1,j} = a_{i+1,j+1}$ for particular $i,j$ , then $a_{i+1,j}=a_{i,j}=a_{i+1,j+1}$ . In this case,

$$ \begin{align*}(\operatorname{decor}(a_{i+1,j}),\operatorname{decor}(a_{i+1,j+1})) \in \{\emptyset,\nwarrow\} \times \{\emptyset,\nearrow\} \end{align*} $$

or

(3.2) $$ \begin{align} (\operatorname{decor}(a_{i+1,j}),\operatorname{decor}(a_{i+1,j+1})) \in \{\nearrow,\nwarrow \!\!\!\!\!\;\!\! \nearrow\} \times \{\nwarrow,\nwarrow \!\!\!\!\!\;\!\! \nearrow\}, \end{align} $$

where in the second case, there is a contribution of $-1$ to the sign of the object.

These observations imply that if the bottom row is weakly increasing, then the underlying undecorated triangular array is an ordinary Gelfand-Tsetlin pattern and that the properties on the decoration stated in the proposition are satisfied. The only instance when we have a contribution to the sign is in the case of (3.2).

Conversely, a decoration of a given Gelfand-Tsetlin pattern that follows the rule as given in the statement of the proposition is eligible for an arrowed Gelfand-Tsetlin pattern according to Definition 3.1.

The following explicit formula for the generating function of arrowed Gelfand-Tsetlin patterns with fixed bottom row $k_1,k_2,\ldots ,k_n$ is proved in [Reference Fischer and Schreier-Aigner13].

Theorem 3.4. The generating function of arrowed Gelfand-Tsetlin patterns with bottom row $k_1,\ldots ,k_n$ is

$$ \begin{align*}\prod_{i=1}^{n} (t + u X_i + v X_i^{-1} + w) \prod_{1 \le i < j \le n}\left( t + u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) s_{(k_n,k_{n-1},\ldots,k_1)}(X_1,\ldots,X_n), \end{align*} $$

where ${\operatorname {E}}_x$ denotes the shift operator, defined as ${\operatorname {E}}_x p(x) = p(x+1)$ .

The formula has to be applied as follows: First interpret $k_1,\ldots ,k_n$ as variables and apply the operator $\prod _{1 \le i < j \le n} \left ( t + u {\operatorname {E}}_{k_i} + v {\operatorname {E}}_{k_j}^{-1} + w {\operatorname {E}}_{k_i} {\operatorname {E}}_{k_j}^{-1} \right )$ to $s_{(k_n,k_{n-1},\ldots ,k_1)}(X_1,\ldots ,X_n)$ . This will result in a linear combination of expressions of the form $s_{(k_n+i_n,k_{n-1}+i_{n-1},\ldots ,k_1+i_1)}(X_1,\ldots ,X_n)$ for some (varying) integers $i_j$ . The $k_j$ are only specialized to the actual integers after that. Note that we do not necessarily have $k_n+i_n \ge k_{n-1}+i_{n-1} \ge \ldots \ge k_1+i_1$ even if $k_n \ge k_{n-1} \ge \ldots \ge k_1$ , so that the extension of the Schur polynomial in (3.1) is necessary.

Example 3.5. We illustrate the theorem on the example $(k_1,k_2,k_3)=(1,2,3)$ . We list the $8$ Gelfand-Tsetlin pattern with bottom row $1,2,3$ and indicate the possible decorations (one will be listed twice with a disjoint set of decorations), where $L=\{\emptyset , \nwarrow ~\}$ , $R=\{\emptyset , \nearrow \}$ and $LR= \{\emptyset ,\nwarrow ,\nearrow ,\nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ , and on the right, we indicate the generating function restricted to the particular underlying Gelfand-Tsetlin patterns with the indicated decorations, where we use

$$ \begin{align*}L(X)=t+ v X^{-1}, R(X)=t+u X \quad \text{and} \quad LR(X) = t + u X + v X^{-1} + w. \end{align*} $$

$$ \begin{align*} \begin{array}{cl} \begin{array}{ccccc} && {LR \atop 1} && \\ & {L \atop 1} && {LR \atop 2} & \\ {L \atop 1} && {L \atop 2} && {LR \atop 3} \end{array} & X_1 X_2^2 X_3^3 LR(X_1) L(X_2) LR(X_2) L(X_3)^2 LR(X_3) \\ \begin{array}{ccccc} && {LR \atop 2} && \\ & {LR \atop 1} && {R \atop 2} & \\ {L \atop 1} && {L \atop 2} && {LR \atop 3} \end{array} & X_1^2 X_2 X_3^3 LR(X_1) LR(X_2) R(X_2) L(X_3)^2 LR(X_3) \\ \begin{array}{ccccc} && {LR \atop 1} && \\ & {L \atop 1} && {LR \atop 3} & \\ {L \atop 1} && {LR \atop 2} && {R \atop 3} \end{array} & X_1 X_2^3 X_3^2 LR(X_1) L(X_2) LR(X_2) L(X_3) LR(X_3) R(X_3) \\ \begin{array}{ccccc} && {LR \atop 2} && \\ & {LR \atop 1} && {LR \atop 3} & \\ {L \atop 1} && {LR \atop 2} && {R \atop 3} \end{array} & X_1^2 X_2^2 X_3^2 LR(X_1) LR(X_2)^2 L(X_3) LR(X_3) R(X_3) \\ \begin{array}{ccccc} && {LR \atop 3} && \\ & {LR \atop 1} && {R \atop 3} & \\ {L \atop 1} && {LR \atop 2} && {R \atop 3} \end{array} & X_1^3 X_2 X_3^2 LR(X_1) LR(X_2) R(X_2) L(X_3) LR(X_3) R(X_3) \\ \begin{array}{ccccc} && {LR \atop 2} && \\ & {L \atop 2} && {LR \atop 3} & \\ {LR \atop 1} && {R \atop 2} && {R \atop 3} \end{array} & X_1^2 X_2^3 X_3 LR(X_1) L(X_2) LR(X_2) LR(X_3) R(X_3)^2 \\ \begin{array}{ccccc} && {LR \atop 3} && \\ & {LR \atop 2} && {R \atop 3} & \\ {LR \atop 1} && {R \atop 2} && {R \atop 3} \end{array} & X_1^3 X_2^2 X_3 LR(X_1) LR(X_2) R(X_2) LR(X_3) R(X_3)^2 \\ \begin{array}{ccccc} && {LR \atop 2} && \\ & {L \atop 2} && {R \atop 2} & \\ {LR \atop 1} && {\emptyset \atop 2} && {LR \atop 3} \end{array} & X_1^2 X_2^2 X_3^2 LR(X_1) L(X_2) R(X_2) t LR(X_3)^2 \\ \begin{array}{ccccc} && {LR \atop 2} && \\ & {\{ \nearrow, \nwarrow \!\!\!\!\!\;\!\! \nearrow \} \atop 2} && { \{ \nwarrow, \nwarrow \!\!\!\!\!\;\!\! \nearrow \} \atop 2} & \\ {LR \atop 1} && {\emptyset \atop 2} && {LR \atop 3} \end{array} & -X_1^2 X_2^2 X_3^2 LR(X_1) (w+uX_2)(w+v X_2^{-1}) t LR(X_3)^2 \\ \end{array} \end{align*} $$

It is convenient for us to rewrite the formula from Theorem 3.4 as follows.

Corollary 3.6. The generating function of arrowed Gelfand-Tsetlin patterns with bottom row $k_1,\ldots ,k_n$  is

(3.3) $$ \begin{align} \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n}\left(v + w X_i + t X_j + u X_i X_j \right) \prod_{i=1}^{n} X_i^{k_i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)}. \end{align} $$

Proof. Observe that

$$ \begin{align*} & \prod_{i=1}^{n} (t + u X_i + v X_i^{-1}+w) \prod_{1 \le i < j \le n}\left(t+ u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) s_{(k_n,k_{n-1},\ldots,k_1)}(X_1,\ldots,X_n) \notag \\ \quad &= \prod_{i=1}^{n} (t+u X_i + v X_i^{-1}+w) \prod_{1 \le i < j \le n}\left(t + u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)}. \end{align*} $$

This is further equal to

$$ \begin{align*} & \prod_{i=1}^{n} (t + u X_i + v X_i^{-1}+w) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i < j \le n}\left(t + u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \notag \\ \quad &= \prod_{i=1}^{n} (t + u X_i + v X_i^{-1}+w) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i < j \le n}\left(t + u X_i + v X_j^{-1} + w X_i X_j^{-1} \right) \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \notag \\ \quad &= \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n}\left(v + w X_i + t X_j + u X_i X_j \right) \prod_{i=1}^{n} X_i^{k_i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)}, \end{align*} $$

and the assertion follows.

3.2. Generating function with respect to a Schur polynomial weight

We are now ready to obtain our first interpretation. Multiplying (1.4) and (1.8) with $\prod _{i=1}^{n} (X_i^{-1} + 1+w + X_i)$ gives

(3.4) $$ \begin{align} &\frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n} (1+ w X_i + X_j + X_i X_j) \sum_{0 \le k_1 < k_2 < \ldots < k_n} X_1^{k_1-1} X_2^{k_2-1} \cdots X_n^{k_n-1} \right]}{\prod_{1 \le i < j \le n} (X_j-X_i)} \nonumber\\ & \qquad\qquad\qquad\qquad= \prod_{i=1}^{n} (X_i^{-1} + 1+w + X_i) \prod_{i=1}^{n} \frac{1}{1-X_i} \prod_{1 \le i < j \le n} \frac{1+X_i + X_j + w X_i X_j}{1-X_i X_j}, \end{align} $$

and

(3.5) $$ \begin{align} & \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n} (1+w X_i+X_j + X_i X_j) \sum_{0 \le k_1 < k_2 < \ldots < k_n \le m} X_1^{k_1-1} X_2^{k_2-1} \cdots X_n^{k_n-1} \right]}{\prod_{1 \le i < j \le n} (X_j-X_i)}\nonumber \\ & \qquad\qquad\qquad\qquad\qquad\qquad\qquad = \prod_{i=1}^{n} (X_i^{-1} + 1+w + X_i) \nonumber\\ &\qquad \times \frac{\det_{1 \le i, j \le n} \left( X_i^{j-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{m+2n-j} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)}{\prod\limits_{i=1}^n (1-X_i) \prod\limits_{1 \le i < j \le n} (1-X_i X_j)(X_j-X_i)}, \end{align} $$

respectively, and we can now interpret the left-hand sides as the generating function of arrowed Gelfand-Tsetlin patterns with non-negative strictly increasing bottom row, where we need to specialize $t=u=v=1$ in the weight, and in the second case, the entries in the bottom row are less than or equal to m.

In Appendix B, we develop some other (maybe less interesting) combinatorial interpretations of the left-hand sides, which we include for the sake of completeness.

4.1. Right-hand side of (3.4)

For the right-hand side of (3.4), which is

(4.1) $$ \begin{align} \prod_{i=1}^{n} \frac{X_i^{-1}+1+w+X_i}{1-X_i} \prod_{1 \le i < j \le n} \frac{1+X_i + X_j + w X_i X_j}{1-X_i X_j}, \end{align} $$

it is straightforward to give a combinatorial interpretation as a generating function. Recall that, in the ordinary case (1.1), the right-hand side $\prod _{i=1}^{n} \frac {1}{1-X_i} \prod _{1 \le i < j \le n} \frac {1}{1-X_i X_j}$ is interpreted as two-line arrays with entries in $\{1,2,\ldots ,n\}$ , ordered lexicographically, with the top element of each column being greater than or equal to its bottom element. The exponent of $X_i$ in the weight is computed by subtracting from the total number of i’s in the two-line array the number of columns with i as top and bottom element.

To extend this to an interpretation of (4.1), we have one additional column $\binom {j}{i}$ for all pairs $i \le j$ , which are either overlined, underlined, both or neither. An overlined column $\binom {j}{i}$ with $i<j$ contributes an additional multiplicative $X_j$ to the weight, while an underlined column with i as bottom element contributes an additional $X_i$ , and if a column is overlined and underlined, then such a column contributes, in addition to $X_i X_j$ , w. Moreover, an overlined column $\binom {i}{i}$ contributes an additional $X_i$ to the weight, and if it is underlined, then it contributes $X_i^{-1}$ to the weight, and, again, if the column is overlined and underlined, then it contributes also w. In both cases, if the column is neither underlined nor overlined, it contributes nothing in addition.

4.2. Right-hand side of (3.5)

The following theorem provides an interpretation of the right-hand side of (3.5) as a weighted count of (partly non-intersecting) lattice paths. This right-hand side differs from the right-hand side of (1.8) by a simple multiplicative factor. We work as long as possible with general w; however, it will turn out that we need to specialize to $w=0,1$ at some point to obtain a nicer interpretation. We present two different proofs to obtain the result, where the second one is only sketched.

Figure 1 seeks to illustrate the theorem in the case that m is odd.

Figure 1 An example of families of lattice paths in Theorem 4.1.

Theorem 4.1. (1) Assume that $m=2l+1$ . Then the right-hand side of (3.5) has the following interpretation as weighted count of families of n lattice paths.

• ○ The i-th lattice path starts in one point in the set $A_i=\{(-3i+1,-i+1),(-i+1,-3i+1)\}$ , $i=1,2,\ldots ,n$ , and the end points of the paths are $E_j=(n-j+l+1,j-l-2)$ , $j=1,2,\ldots ,n$ .

• ○ Below and on the line $x+y=0$ , the step set is $\{(1,1),(-1,1)\}$ for steps that start in $(-3i+1,-i+1)$ , and it is $\{(1,1),(1,-1)\}$ for steps that start in $(-i+1,-3i+1)$ . Steps of type $(-1,1)$ and $(1,-1)$ with distance $0,2,4,\ldots $ from $x+y=0$ are equipped with the weights $X_1,X_2,X_3,\ldots $ , respectively, while such steps with distance $1,3,5,\ldots $ are equipped with the weights $X_1^{-1},X_2^{-1},X_3^{-1},\ldots $ , respectively.

• ○ Above the line $x+y=0$ , the step set is $\{(1,0),(0,1)\}$ . Above the line $x+y=j-1$ , horizontal steps of the path that ends in $E_j$ are equipped with the weight w.

• ○ The paths can be assumed to be non-intersecting below the line $x+y=0$ . In case $w=1$ , we can also assume them to be non-intersecting above the line $x+y=0$ . In case $w=0$ , $E_j$ can be replaced by $E^{\prime }_j=(n-j+l+1,2j-n-l-2)$ , $j=1,2,\ldots ,n$ , and then we can also assume the paths to be non-intersecting above the line $x+y=0$ .

• ○ The sign of family of paths is the sign of the permutation $\sigma $ with the property that the i-th path connects $A_i$ to $E_{\sigma (i)}$ with an extra contribution of $-1$ if we choose $(-i+1,-3i+1)$ from $A_i$ . Moreover, we have an overall factor of

  $$ \begin{align*}(-1)^{\binom{n+1}{2}} \prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i)(1+X_i). \end{align*} $$

• ○ In case $w=0,1$ , when restricting to non-intersecting paths, let $1 \le i_1 < i_2,\ldots < i_m < n$ be the indices for which we chose $(-3i+1,-i+1)$ from $A_i$ . Then the sign can assumed to be $(-1)^{i_1+\ldots +i_m}$ , and the overall factor is

  $$ \begin{align*}\prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i)(1+X_i). \end{align*} $$

(2) Assume that $m=2l$ . Then, to obtain an interpretation for the right-hand side of (3.5), we only need to replace $E_j$ by a set of two possible endpoints $E_j=\{(n-j+l+1,j-l-2),(n-j+l,j-l-1)\}$ . The overall factor is

$$ \begin{align*}(-1)^{\binom{n+1}{2}} \prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i) \end{align*} $$

in the case when we do not specialize w. The endpoints are replaced by $E^{\prime }_j=\{(n-j+l+1,2j-n-l-2),(n-j+l,2j-n-l-1)\}$ if $w=0$ . In case $w=0,1$ if we restrict to non-intersecting paths and the sign is taken care of as above, then the overall factor is

$$ \begin{align*}\prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i). \end{align*} $$

We discuss the weight and the sign on the example in Figure 1. The weights that come from the individual paths are

$$ \begin{align*}X_1^{-1} \cdot X_1^{-1} \cdot X_2 \cdot X_1 X_3 \cdot X_2 X_3^{-1} \cdot X_5^{-2}, \end{align*} $$

where the factors are arranged in a manner that the i-th factor is the weight of the path that starts in the set $A_i$ . To compute the sign, observe that $\sigma = (6 \, 5 \, 4 \, 3 \, 2 \, 1)$ in one-line notation so that $\operatorname {\mathrm {sgn}} \sigma = -1$ and that we choose the second starting point in $A_i$ except for $i=1$ , so that the total sign is $(-1)\cdot (-1)^5=1$ .

In the case that m is odd, we always need to choose the second lattice point in $A_i$ if $l \ge n-2$ because then all $E_i$ have a non-positive y-coordinate, and this implies that they cannot be reached by any of the first lattice points in $A_i$ since any lattice path starting from the first lattice point in $A_i$ intersects the line $x+y=0$ in a lattice point with positive y-coordinate. This implies that, in the non-intersecting case, the sign is always $1$ . In the case that m is even, the condition is $l \ge n-1$ .

In theses cases and when we have in addition $w=0$ , we can translate the lattice paths easily into pairs of plane partitions. The case $m=2l+1$ is illustrated in Figure 2, while the case $m=2l$ is illustrated in Figure 3. A similar result can in principle be derived for the case $w=1$ , but we omit this here.

Figure 2 Illustration of Corollary 4.2 (1) for $n=7$ and $l=12$ .

Figure 3 Illustration of Corollary 4.2 (1) for $n=7$ and $l=13$ .

Corollary 4.2. Let $w=0$ .

(1) Assume that $m=2l+1$ . In case $l \ge n-2$ , the right-hand side of (3.5) is the generating function of plane partitions $(P,Q)$ of shapes $\lambda , \mu $ , respectively, where $\mu $ is the complement of $\lambda $ in the $n \times l$ -rectangle, P is a column-strict plane partition such that the entries in the i-th row are bounded by $2n+2-2i$ , and Q is a row-strict plane partition of positive integers such that the entries in the i-th row are bounded by $n-i$ . The weight is

$$ \begin{align*} \prod_{i=1}^n X_i^l (X_i^{-1} +1 + X_i)(1+X_i) X_i^{\# \text{ of } 2i-1 \text{ in } P} X_i^{- \, \# \text{ of } 2i \text{ in } P}. \end{align*} $$

(2) Assume that $m=2l$ . In case $l \ge n-1$ , the right-hand side of (3.5) is the generating function of plane partitions $(P,Q)$ of (straight) shape $\lambda $ and skew shape $\mu $ , respectively, such that $\mu $ is the complement of $\lambda $ in the $n \times (l-1)$ -rectangle after possibly deleting the first column of $\mu $ , P is a column strict plane partition such that the entries in the i-th row are bounded by $2n+2-2i$ , and Q is a row-strict plane partition such that the entries in the i-th row are bounded by $n-i$ . The weight is

$$ \begin{align*}\prod_{i=1}^n X_i^l (X_i^{-1} +1 + X_i) X_i^{\# \text{ of } 2i-1 \text{ in } P} X_i^{- \, \# \text{ of } 2i \text{ in } P}. \end{align*} $$

4.3. The cases $n=2$ and $m=2,3$

In this section, we give a list of all objects for the left-hand side and right-hand side of (3.5) in the case $n=2$ and $m=2,3$ . We start with the case that $m=3$ , since this is easier on the right-hand side.

Note that $m=3$ implies $l=1$ . The arrowed monotone triangles are as follows, using the notation from Section 3.

$$ \begin{align*} \begin{array}{ccc} \kern-81pt\begin{array}{ccc} & {LR \atop 0} & \\ {L \atop 0} && {LR \atop 1} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {LR \atop 0} && {R \atop 1} \end{array}, \begin{array}{ccc} & {LR \atop 0} & \\ {L \atop 0} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {LR \atop 0} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 0} && {R \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 0} & \\ {L \atop 0} && {LR \atop 3} \end{array},\\[12pt]\kern-18pt\begin{array}{ccc} & {LR \atop 1} & \\ {LR \atop 0} && {LR \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 0} && {LR \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 3} & \\ {LR \atop 0} && {R \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {L \atop 1} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 1} && {R \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {L \atop 1} && {LR \atop 3} \end{array}, \\[12pt]\kern165pt\begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 1} && {LR \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 3} & \\ {L \atop 1} && {LR \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {L \atop 2} && {LR \atop 3} \end{array}, \begin{array}{ccc} & {LR \atop 3} & \\ {LR \atop 2} && {R \atop 3} \end{array} \end{array} \end{align*} $$

The weights are

(4.2) $$ \begin{align} & X_2(1+X_2^{-1}), X_1(1+X_2), X_2^2(1+X_2^{-1}), X_1 X_2(X_2^{-1}+1+w+X_2), X_1^2(1+X_2), X_2^3(1+X_2^{-1}), \nonumber\\ & X_1 X_2^2(X_2^{-1}+1+w+X_2), X_1^2 X_2(X_2^{-1}+1+w+X_2), X_1^3(1+X_2), X_1 X_2^2(1+X_2^{-1}), X_1^2 X_2(1+X_2), \nonumber\\ & X_1 X_2^3 (1+X_2^{-1}), X_1^2 X_2^2 (X_2^{-1} + 1 + w + X_2), X_1^3 X_2 (1+X_2), X_1^2 X_2^3(1+X_2^{-1}), X_1^3 X_2^2(1+X_2), \end{align} $$

up to the overall factor $LR(X_1) LR(X_2)$ , setting $t=u=v=1$ .

The corresponding paths from Theorem 4.1 are as follows.

The weights are

(4.3) $$ \begin{align} -w,-X_1,-X_1^{-1},-X_2,-X_2^{-1},-X_1^{-1} -X_2, -X_1^{-1} X_2^{-1},-1,-X_1 X_2, -X_1 X_2^{-1}, \end{align} $$

up to the overall factor

$$ \begin{align*}&- X_1 X_2 (1+X_1)(1+X_2)(X_1^{-1} + 1 + w + X_1)(X_2^{-1} +1 +w + X_2)\\[3pt]&\quad= -X_1 X_2 (1+X_1)(1+X_2)LR(X_1) LR(X_2), \end{align*} $$

and, as can easily be seen, the sum of weights agrees with those for the arrowed Gelfand-Tsetlin patterns.

Now we consider the case $m=2$ . We have $l=1$ . The arrowed monotone triangles are as follows:

$$ \begin{align*} \begin{array}{ccc} & {LR \atop 0} & \\ {L \atop 0} && {LR \atop 1} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {LR \atop 0} && {R \atop 1} \end{array}, \begin{array}{ccc} & {LR \atop 0} & \\ {L \atop 0} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {LR \atop 0} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 0} && {R \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 1} & \\ {L \atop 1} && {LR \atop 2} \end{array}, \begin{array}{ccc} & {LR \atop 2} & \\ {LR \atop 1} && {R \atop 2.} \end{array}\end{align*} $$

The weights are

$$ \begin{align*} X_2(1+X_2^{-1}), X_1(1+X_2), X_2^2(1+X_2^{-1}), X_1 X_2(X_2^{-1}+1+w+X_2), X_1^2(1+X_2),\, & X_1 X_2^2(1+X_2^{-1}), \\[3pt] & X_1^2 X_2(1+X_2), \end{align*} $$

up to the overall factor $LR(X_1) LR(X_2)$ , setting $t=u=v=1$ .

As for the lattice paths, the situation is very similar to the case $m=3,l=1$ , only $E_1=(3,-2)$ is replaced by the set $E_1=\{(2,-1),(3,-2)\}$ and $E_2=(2,-1)$ is replaced by the set $E_2=\{(1,0),(2,-1)\}$ . It follows that all the families of paths from the case $m=3$ and $l=1$ also appear here. In addition, we have the following families of lattice paths.

Thus, in addition to the weights in (4.3), these families of lattice paths give

$$ \begin{align*}-1,-w,-X_1,-X_1^{-1},-X_2,-X_2^{-1},-1,w, \end{align*} $$

up to the overall factor

$$ \begin{align*}- X_1 X_2 (X_1^{-1} + 1 + w + X_1)(X_2^{-1} +1 +w + X_2)= -X_1 X_2 LR(X_1) LR(X_2), \end{align*} $$

where the last two weights come from the last picture, first by interpreting the endpoint of the path that starts in $A_1$ as element of $E_2$ and second as element of $E_1$ .

4.4. First proof of Theorem 4.1

The approach of the first proof of Theorem 4.1 is closely related to the approach we used in the proof of Theorem 2.2 in [Reference Fischer and Höngesberg7].

We consider the following bases for Laurent polynomials in X that are invariant under the transformation $X \to X^{-1}$ : let

$$ \begin{align*}q_i(X)=\frac{X^i-X^{-i}}{X-X^{-1}} \qquad \text{and} \qquad b_i(X)=(X+X^{-1})^i. \end{align*} $$

Then $(q_i(X))_{i \ge 0}$ and $(b_i(X))_{i \ge 0}$ are two such bases. It is not hard to verify that

(4.4) $$ \begin{align} q_m(X)= \sum_{r=0}^{(m-1)/2} (-1)^r \binom{m-r-1}{r} b_{m-1-2r}(X). \end{align} $$

In order to derive a combinatorial interpretation of the right-hand side of (3.5), consider

(4.5) $$ \begin{align} \det_{1 \le i, j \le n} \left( X_i^{j-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{m+2n-j} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right). \end{align} $$

We start by considering the case where m is odd: We set $m=2l+1$ , and pull out $\prod _{i=1}^n X_i^{l+n}$ , and obtain

(4.6) $$ \begin{align} \prod_{i=1}^n X_i^{l+n} \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right). \end{align} $$

The entry in the i-th row and j-th column of the matrix underlying the determinant is obtained from

$$ \begin{align*} & \frac{X^{j-l-n-1} (1+X)^{j-1} (1+w X)^{n-j} - X^{-j+l+n+1} (1+X^{-1})^{j-1} (1+w X^{-1})^{n-j}}{X-X^{-1}} \\ &\qquad\qquad\qquad\qquad\qquad\qquad = \sum_{p,q \ge 0} \binom{j-1}{p} \binom{n-j}{q} w^q \frac{X^{j-l-n+p+q-1}-X^{-j+l+n-p-q+1}}{X-X^{-1}} \end{align*} $$

by multiplying with $X-X^{-1}$ and then setting $X=X_i$ . Note that this expression is invariant under replacing X by $X^{-1}$ . From (4.4), it follows that this is further equal to

$$ \begin{align*} \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) & (-1)^r w^q \binom{j-1}{p} \binom{n-j}{q} \\ & \times \binom{|j-l-n+p+q-1|-r-1}{r} b_{|j-l-n+p+q-1|-1-2r}(X). \end{align*} $$

We apply the following lemma. A proof can be found in [Reference Fischer and Schreier-Aigner13, Lemma 7.2]. Note that the lemma also involves complete homogeneous symmetric polynomials $h_k$ with negative k as defined in [Reference Fischer and Schreier-Aigner13, Section 5]. Concretely, we define $h_k(X_1,\ldots ,X_n)=0$ for $k=-1,-2,\ldots ,-n+1$ and

(4.7) $$ \begin{align} h_k(X_1,\ldots,X_n) = (-1)^{n+1} X_1^{-1} \ldots X_n^{-1} h_{-k-n}(X_1^{-1},\ldots,X_n^{-1}) \end{align} $$

for $k \le -n$ . Note that a consequence of this definition is that the latter relation is true for any k.

Lemma 4.4. Let $f_j(Y)$ be formal Laurent series for $1 \le j \le n$ , and define

$$ \begin{align*}f_j[Y_1,\ldots,Y_i]=\sum_{k \in \mathbb{Z}} \langle Y^{k} \rangle f_j(Y) \cdot h_{k-i+1}(Y_1,\ldots,Y_i), \end{align*} $$

where $\langle Y^{k} \rangle f_j(Y)$ denotes the coefficient of $Y^{k}$ in $f_j(Y)$ and $h_{k-i+1}$ denotes the complete homogeneous symmetric polynomial of degree $k-i+1$ . Then

$$ \begin{align*}\frac{\det_{1 \le i, j \le n} \left( f_j(Y_i) \right) }{\prod_{1 \le i < j \le n} (Y_j - Y_i)} = \det_{1 \le i, j \le n} \left( f_j[Y_1,\ldots,Y_i] \right). \end{align*} $$

Noting that a Laurent polynomial in X that is invariant under the replacement $X \to X^{-1}$ can be written as a polynomial in $X+X^{-1}$ , we use the lemma to basically rewrite (4.6) as follows:

$$ \begin{align*} & \frac{ \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)} {\prod_{1 \le i < j \le n} (X_j + X_j^{-1} - X_i - X_i^{-1})} \\& \quad = \prod_{i=1}^n (X_i - X_i^{-1}) \det_{1 \le i, j \le n} \left( \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) (-1)^r w^q \binom{j-1}{p} \binom{n-j}{q} \right. \\& \qquad \left. \times\ \binom{|j-l-n+p+q-1|-r-1}{r} h_{|j-l-n+p+q-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) \right). \end{align*} $$

Now, as $X_j + X_j^{-1} - X_i - X_i^{-1}=(X_i-X_j)(1-X_i X_j)X_i^{-1} X_j^{-1}$ , in order to find a combinatorial interpretation for the right-hand side of (3.5), we need to find a combinatorial interpretation of

$$ \begin{align*} & (-1)^{\binom{n}{2}} \prod_{i=1}^{n} X_i^{l+1}(X_i^{-1}+1+w + X_i) (X_i - X_i^{-1})(1-X_i)^{-1} \\& \qquad \times \det_{1 \le i, j \le n} \left( \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) (-1)^r w^q \binom{j-1}{p} \binom{n-j}{q} \right. \\& \qquad\left. \times\ \binom{|j-l-n+p+q-1|-r-1}{r} h_{|j-l-n+p+q-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) \vphantom{\sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0}}\right)\\& \quad =(-1)^{\binom{n+1}{2}} \prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i)(1+X_i) \\& \qquad\times \det_{1 \le i, j \le n} \left( \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) (-1)^r w^q \binom{j-1}{p} \binom{n-j}{q} \right. \\& \qquad\left. \times\ \binom{|j-l-n+p+q-1|-r-1}{r} h_{|j-l-n+p+q-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) \vphantom{\sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0}}\right). \end{align*} $$

For this purpose, we find a combinatorial interpretation of the entry of the underlying matrix – that is,

$$ \begin{align*} \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-i-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) (-1)^r w^q & \binom{j-1}{p} \binom{n-j}{q} \binom{|j-l-n+p+q-1|-r-1}{r} \\ & \times h_{|j-l-n+p+q-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) \end{align*} $$

in terms of a lattice paths generating function. We simplify the expression using the transformation $q \to n-j-q$ :

(4.8) $$ \begin{align} \sum_{p,q,r \ge 0 \atop |p-q-l-1|-i-2r \ge 0} \operatorname{\mathrm{sgn}}(p-q-l-1) (-1)^r w^{n-j-q} & \binom{j-1}{p} \binom{n-j}{q} \binom{|p-q-l-1|-r-1}{r} \nonumber\\ & \times h_{|p-q-l-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}). \end{align} $$

We simplify the expression further using the following lemma. A combinatorial proof of it using a sign-reversing involution is provided in [Reference Fischer and Höngesberg7, Lemma 7.7].

Lemma 4.5. Let $a,i$ be positive integers with $i \le a$ . Then

$$ \begin{align*}\sum_{r=0}^{(a-i)/2} (-1)^r \binom{a-r-1}{r} h_{a-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) = h_{a-i}(X_1,X_1^{-1},\ldots,X_i,X_i^{-1}). \end{align*} $$

Therefore, the sum in (4.8) is equal to

(4.9) $$ \begin{align} \sum_{p,q} \operatorname{\mathrm{sgn}}(p-q-l-1) w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} h_{|p-q-l-1|-i}(X_1,X_1^{-1},\ldots,X_i,X_i^{-1}). \end{align} $$

We claim the following: If $p-q-l-1 \ge 0$ , then (4.9) is the generating function of lattice paths from $(-3i+1,-i+1)$ to $(n-j+l+1,j-l-2)$ such that the following is satisfied.

• ○ Below and on the line $x+y=0$ , the step set is $\{(1,1),(-1,1)\}$ . Steps of type $(-1,1)$ with distances $0,2,4,\ldots $ from $x+y=n$ are equipped with the weights $X_1,X_2,X_3,\ldots $ , respectively, while steps of type $(-1,1)$ with distances $1,3,5,\ldots $ are equipped with the weights $X_1^{-1},X_2^{-1},X_3^{-1},\ldots $ , respectively.

• ○ Above the line $x+y=0$ , the step set is $\{(1,0),(0,1)\}$ . Above the line $x+y=j-1$ , horizontal steps are equipped with the weight w.

Namely, if we assume that there are q steps of type $(0,1)$ above the line $x+y=j-1$ , and therefore, $n-j-q$ steps of type $(1,0)$ , then the path intersects the line $x+y=j-1$ in the lattice point $(l+1+q,j-l-2-q)$ , assuming that the endpoint of the path is $(n-j+l+1,j-l-2)$ , and there are $\binom {n-j}{q}$ of such paths each of them contributing $w^{n-j-q}$ to the weight. Note that this weight depends on j if $w \not = 0,1$ , and this causes complications when applying the Lindström-Gessel-Viennot lemma.

If we further assume that there are p steps of type $(1,0)$ below the line $x+y=j-1$ , and therefore, $j-p$ steps of type $(0,1)$ , then the last lattice point of such a path on the line $x+y=0$ when traversing the path from $(-3i+1,-i+1)$ to $(n-j+l+1,j-l-2)$ is $(-p+q+l+1,p-q-l-1)$ . Note that by the assumption $p-q-l-1 \ge 0$ , the lattice point $(-p+q+l+1,p-q-l-1)$ is in the second quadrant – that is, $\{(x,y)|x \le 0, y \ge 0 \}$ .

Finally, lattice paths from $(-3i+1,-i+1)$ to $(-p+q+l+1,p-q-l-1)$ with step set $\{(1,1),(-1,1)\}$ have $p-q-l-1-i$ steps of type $(-1,1)$ and $2i-1$ steps of type $(1,1)$ . The generating function of such paths is clearly $h_{p-q-l-1-i}(X_1,X_1^{-1},\ldots ,X_i,X_i^{-1})=h_{|p-q-l-1|-i}(X_1,X_1^{-1},\ldots ,X_i,X_i^{-1})$ .

The situation is very similar if $p-q-l-1 \le 0$ , except that we need to replace the starting point $(-3i+1,-i+1)$ by $(-i+1,-3i+1)$ , and the step set is $\{(1,1),(1,-1)\}$ below the line $x+y=0$ . Again, we can assume that $(-p+q+l+1,p-q-l-1)$ is the last lattice point on the line $x+y=0$ when traversing the path from $(-i+1,-3i+1)$ to $(-p+q+l+1,p-q-l-1)$ . In this case, $(-p+q+l+1,p-q-l-1)$ lies in the fourth quadrant $\{(x,y)|x \le 0, y \le 0 \}$ . We have $-p+q+l+1-i=|p-q-l-1|-i$ steps of type $(1,-1)$ and $2i-1$ steps of type $(1,1)$ ; thus, the generating function in this segment is also $h_{|p-q-l-1|-i}(X_1,X_1^{-1},\ldots ,X_i,X_i^{-1})$ .

Consequently, we can conclude that the right-hand side of (3.5) has the following combinatorial interpretation: We consider families of n lattice paths from $A_i=\{(-3i+1,-i+1),(-i+1,-3i+1)\}$ , $i=1,2,\ldots ,n$ , to $E_j=(n-j+l+1,j-l-2)$ , $j=1,2,\ldots ,n$ , with steps sets and weights as described above. By the Lindström-Gessel-Viennot lemma [Reference Lindström20, Reference Gessel and Viennot15, Reference Gessel and Viennot16], the paths can be assumed to be non-intersecting on and below the line $x+y=0$ .

In case $w=0,1$ , we can also assume them to be non-intersecting. This is clear for $w=1$ . In case $w=0$ , we can assume that there are no steps of type $(1,0)$ above the line $x+y=j-1$ , and, therefore, we can also have $(n-j+l+1,2j-n-l-2)$ on the line $x+y=j-1$ as endpoint since above the line all the $n-j$ steps have to be of type $(0,1)$ . Whenever we choose $(-i+1,-3i+1)$ , this contributes $-1$ to the weight.

In the non-intersecting setting, suppose we choose $(-i+1,-3i+1)$ from $A_i$ for $1 \le i_1 < \ldots < i_m \le n$ , then the sign of the permutation $\sigma $ such that $A_i$ is connected to $E_{\sigma (i)}$ via the paths is $(-1)^{i_1+i_2+\ldots +i_m-m}$ . This gives a total sign of $(-1)^{i_1+i_2+\ldots +i_m}$ . Recall also that we have an additional overall weight of

$$ \begin{align*}(-1)^{\binom{n+1}{2}} \prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i)(1+X_i). \end{align*} $$

Combining the sign from above with $(-1)^{\binom {n+1}{2}}=(-1)^{1+2+\ldots +n}$ , the sign can also be computed as follows: suppose we choose $(-3i+1,-i+1)$ from $A_i$ precisely for $i_1,\ldots ,i_m$ . Then the sign is $(-1)^{i_1+i_2+\ldots +i_m}$ , and in this setting, the overall weight is

$$ \begin{align*}\prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i)(1+X_i). \end{align*} $$

This concludes the proof of the first part of Theorem 4.1.

Now we consider the case where m is even: We set $m=2l$ in (4.5), pull out $\prod _{i=1}^n X_i^{l+n}$ and obtain

$$ \begin{align*}\prod_{i=1}^n X_i^{l+n} \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right). \end{align*} $$

The entry in the i-th row of the j-th column of the matrix underlying the determinant is obtained from

$$ \begin{align*} & \frac{X^{j-l-n-1} (1+X)^{j-1} (1+w X)^{n-j} - X^{-j+l+n} (1+X^{-1})^{j-1} (1+w X^{-1})^{n-j}}{1-X^{-1}} \\ & \qquad\qquad\qquad\qquad\qquad\qquad\qquad= \sum_{p,q \ge 0} \binom{j-1}{p} \binom{n-j}{q} w^q \frac{X^{j-l-n+p+q-1}-X^{-j+l+n-p-q}}{1-X^{-1}}, \end{align*} $$

when multiplying with $1-X^{-1}$ and then setting $X=X_i$ . Now note that

$$ \begin{align*}\frac{X^m-X^{-m-1}}{1-X^{-1}} = q_{m+1}(X) + q_{m}(X) \end{align*} $$

for any integer m, so that we obtain

$$ \begin{align*}\sum_{p,q \ge 0} \binom{j-1}{p} \binom{n-j}{q} w^q \left(q_{j-l-n+p+q}(X) + q_{j-l-n+p+q-1}(X) \right). \end{align*} $$

It follows from (4.4) that this is

$$ \begin{align*} &\sum_{p,q,r \ge 0 \atop |j-l-n+p+q|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q) (-1)^r w^q \binom{j-1}{p} \\& \qquad\qquad\qquad\qquad\times \binom{n-j}{q} \binom{|j-l-n+p+q|-r-1}{r} b_{|j-l-n+p+q|-1-2r}(X) \\&\qquad\qquad + \sum_{p,q,r \ge 0 \atop |j-l-n+p+q-1|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(j-l-n+p+q-1) (-1)^r w^q \binom{j-1}{p} \binom{n-j}{q} \\&\qquad\qquad\qquad\qquad\qquad\qquad\quad \times \binom{|j-l-n+p+q-1|-r-1}{r} b_{|j-l-n+p+q-1|-1-2r}(X). \end{align*} $$

Also, here we simplify the expression using the replacement $q \to n-j-q$ and obtain

$$ \begin{align*} & \sum_{p,q,r \ge 0 \atop |-l+p-q|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(-l+p-q) (-1)^r w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} \binom{|-l+p-q|-r-1}{r} b_{|-l+p-q|-1-2r}(X) \\& \quad + \sum_{p,q,r \ge 0 \atop |-l+p-q-1|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(-l+p-q-1) (-1)^r w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} \binom{|-l+p-q-1|-r-1}{r}\\& \quad \times b_{|-l+p-q-1|-1-2r}(X). \end{align*} $$

This implies the following:

$$ \begin{align*} & \frac{ \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)} {\prod_{1 \le i < j \le n} (X_j + X_j^{-1} - X_i - X_i^{-1})} \\&\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad = \prod_{i=1}^n (1-X_i^{-1}) \det_{1 \le i, j \le n} \left( a_{i,j} \right), \end{align*} $$

with

$$ \begin{align*} a_{i,j} &= \sum_{p,q,r \ge 0 \atop |p-q-l|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(p-q-l) (-1)^r w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} \\&\quad \times \binom{|p-q-l|-r-1}{r} h_{|p-q-l|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}) \\&\quad + \sum_{p,q,r \ge 0 \atop |p-q-l-1|-1-2r \ge 0} \operatorname{\mathrm{sgn}}(p-q-l-1) (-1)^r w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} \\&\quad \times \binom{|p-q-l-1|-r-1}{r} h_{|p-q-l-1|-i-2r}(X_1+X_1^{-1},\ldots,X_i+X_i^{-1}). \end{align*} $$

Using Lemma 4.5, we see that this is equal to

(4.10) $$ \begin{align} b_{i,j} &= \sum_{p,q \ge 0} \operatorname{\mathrm{sgn}}(p-q-l) w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} h_{|p-q-l|-i}(X_1,X_1^{-1},\ldots,X_i,X_i^{-1}) \ \nonumber\\& \quad + \sum_{p,q \ge 0} \operatorname{\mathrm{sgn}}(p-q-l-1) w^{n-j-q} \binom{j-1}{p} \binom{n-j}{q} h_{|p-q-l-1|-i}(X_1,X_1^{-1},\ldots,X_i,X_i^{-1}). \end{align} $$

Here we need to find a combinatorial interpretation of

$$ \begin{align*}(-1)^{\binom{n+1}{2}} \prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w + X_i) \det_{1 \le i, j \le n} \left( b_{i,j} \right). \end{align*} $$

The only modification compared to the odd case is that the endpoints have to be replaced by the following set of two endpoints $E_j=\{(n-j+l+1,j-l-2),(n-j+l,j-l-1)\}$ and that the overall factor is

$$ \begin{align*}\prod_{i=1}^{n} X_i^{l}(X_i^{-1}+1+w+ X_i) , \end{align*} $$

given that the sign is taken care of as above.

This concludes the proof of Theorem 4.1.

4.5. Right-hand side of (3.5), second proof

In this section, we sketch a second proof of Theorem 4.1. It is closely related to the proof of Theorem 2.4 in [Reference Fischer and Höngesberg7]. We only study the case $m=2l+1$ . Again, we need to consider

(4.11) $$ \begin{align} \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right). \end{align} $$

We have the following lemma.

Lemma 4.6. For $n \ge 1$ and $l \in \mathbb {Z}$ , we have

$$ \begin{align*} & \frac{1}{\prod_{1 \le i < j \le n} (X_j-X_i)(X_j^{-1} - X_i^{-1}) \prod_{i,j=1}^n (X_j^{-1} - X_i)} \\&\qquad \times \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} - X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right) \\&\qquad \times \det_{1 \le i, j \le n} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} + X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right) \\&\quad = \frac{(-1)^n}{2} \det_{1 \le i, j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k-i+1}-h_{k+i-1-2n}) \right) \\& \qquad \times \det_{1 \le i,j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-n}+ h_{k-i+1-n}) \right). \end{align*} $$

Proof. We use

$$ \begin{align*}\det\ (A-B) \det(A+B) = \det \left( \begin{array}{c|c} A-B & B \\ \hline 0 & A+B \end{array} \right) = \det \left( \begin{array}{c|c} A-B & B \\ \hline B-A & A \end{array} \right) = \det \left( \begin{array}{c|c} A & B \\ \hline B & A \end{array} \right) \end{align*} $$

to see that the product of determinants on the left-hand side in the assertion of the lemma is equal to

$$ \begin{align*}\det\ \left( \begin{array}{c|c} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} \right)_{1 \le i, j \le n} & \left( X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)_{1 \le i,j \le n} \\ \hline \left( X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)_{1 \le i,j \le n} & \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} \right)_{1 \le i,j \le n} \end{array} \right). \end{align*} $$

Setting $X_{n+i} = X_i^{-1}$ for $i=1,2,\ldots ,n$ , we can also write this is as

$$ \begin{align*}\det\ \left( \begin{array}{c|c} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} \right)_{1 \le i \le 2n \atop 1 \le j \le n} & \left( X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \end{array} \right). \end{align*} $$

We apply Lemma 4.4 to

$$ \begin{align*}\frac{\det\ \left( \begin{array}{c|c} \left( X_i^{j-l-n-1} (1+X_i)^{j-1} (1+ w X_i)^{n-j} \right)_{1 \le i \le 2n \atop 1 \le j \le n} & \left( X_i^{-j+l+n+1} (1+X_i^{-1})^{j-1} (1+w X_i^{-1})^{n-j} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \end{array} \right)}{\prod_{1 \le i < j \le 2n} (X_j-X_i)} \end{align*} $$

and obtain

$$ \begin{align*} & \det\ \left( \left. \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1}(X_1,\ldots,X_i) \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right| \right. \\ &\qquad\qquad\qquad\qquad\qquad\qquad \left. \left( \sum_{k,q} \binom{j-1}{-j-k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1}(X_1,\ldots,X_i) \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right). \end{align*} $$

We multiply from the left with the following matrix:

$$ \begin{align*}(h_{j-i}(X_j,X_{j+1},\ldots,X_{2n}))_{1 \le i,j \le 2n} \end{align*} $$

with determinant $1$ . For this purpose, note that

$$ \begin{align*}\sum_{l=1}^{2n} h_{l-i}(X_l,X_{l+1},\ldots,X_{2n}) h_{k-l+1}(X_1,\ldots,X_l)=h_{k-i+1}(X_1,\ldots,X_{2n}), \end{align*} $$

and therefore, the multiplication results in

$$ \begin{align*} & \det\ \left( \left. \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1}(X_1,\ldots,X_{2n}) \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right| \right. \\ &\qquad\qquad\qquad\qquad\qquad\qquad \left. \left( \sum_{k,q} \binom{j-1}{-j-k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1}(X_1,\ldots,X_{2n}) \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right). \end{align*} $$

We set $X_{i+n}=X_i^{-1}$ for $i=1,2,\ldots ,n$ (so that the arguments of all complete symmetric functions are $(X_1,\ldots ,X_n,X_1^{-1},\ldots ,X_n^{-1})$ ) and omit the $X_i$ ’s now. Also note that, under this specialization, the denominator $\prod _{1 \le i < j \le 2n} (X_j-X_i)$ specializes to the denominator on the left-hand side in the assertion of the lemma. We obtain

$$ \begin{align*}\det\ \left( \begin{array}{c|c} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1} \right)_{1 \le i \le 2n \atop 1 \le j \le n} & \left( \sum_{k,q} \binom{j-1}{-j-k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \end{array} \right). \end{align*} $$

With this specialization, we have $h_k=-h_{-k-2n}$ using (4.7). Therefore, the above is

$$ \begin{align*} &(-1)^n \det\ \left( \begin{array}{c|c} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1} \right)_{1 \le i \le 2n \atop 1 \le j \le n} & \left( \sum_{k,q} \binom{j-1}{-j-k+l+n-q+1} \binom{n-j}{q} w^q h_{-k+i-1-2n} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \end{array} \right) \\& \quad = (-1)^n \det\ \left( \begin{array}{c|c} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k-i+1} \right)_{1 \le i \le 2n \atop 1 \le j \le n} & \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k+i-1-2n} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \end{array} \right). \end{align*} $$

Now, for $j=1,2,\ldots ,n$ , we subtract the $(j+n)$ -th column from the j-th column, and this gives

$$ \begin{align*} & (-1)^n \det\ \left( \left. \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k-i+1}-h_{k+i-1-2n}) \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right| \right. \\&\qquad\qquad\qquad\qquad\qquad\qquad\qquad \left. \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k+i-1-2n} \right)_{1 \le i \le 2n \atop 1 \le j \le n} \right). \end{align*} $$

For $i=n+2,n+3,\ldots ,2n$ , we add the $(2n+2-i)$ -th row to the i-th row. This gives a zero block for $\{(i,j) | n+1 \le i \le 2n, 1 \le j \le n\}$ , since

$$ \begin{align*}\sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k-i+1}-h_{k+i-1-2n}+h_{k-(2n+2-i)+1}-h_{k+(2n+2-i)-1-2n})=0. \end{align*} $$

The lower right block is

$$ \begin{align*} & \det_{n+1 \le i \le 2n \atop 1 \le j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-2n}+ [i\not=n+1]h_{k+2n+2-i-1-2n}) \right) \\&\quad = \det_{n+1 \le i \le 2n \atop 1 \le j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-2n}+ [i\not=n+1]h_{k-i+1}) \right) \\& \quad = \frac{1}{2} \det_{1 \le i,j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-n}+ h_{k-i+1-n}) \right). \end{align*} $$

This concludes the proof of the lemma.

The identity in the lemma involves, up to factors, a product of two determinants on the left-hand side and also a product of two determinants on the right-hand side. This suggests that each of the determinants on the left-hand side equals up to factors a determinant on the right-hand side. This is indeed the case.

More specifically, one can show that (4.11) is up to factors equal to

$$ \begin{align*}(-1)^n \prod_{i=1}^n (1+X_i) X_i^l \det_{1 \le i,j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-n}+ h_{k-i+1-n}) \right). \end{align*} $$

This can be shown by induction with respect to n as suggested in [Reference Fischer and Höngesberg7, Remark 7.4]. Thus, Lemma 4.6 would actually not have been necessary; however, it explains how the expression was obtained much better than the proof by induction.

Using Lemma 7.5 from [Reference Fischer and Höngesberg7], we can conclude further that the expression is equal to

$$ \begin{align*} &(-1)^n \prod_{i=1}^n (1+X_i) X_i^l \\ & \quad \times \det_{1 \le i,j \le n} \left( \sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q h_{k+i-1-n}(X_1,X_1^{-1},\ldots,X_{n-i+1},X_{n-i+1}^{-1}) \right). \end{align*} $$

Also, this formula can be proven directly by induction with respect to n.

Our next goal would be to give a combinatorial interpretation of

$$ \begin{align*}\sum_{k,q} \binom{j-1}{-j+k+l+n-q+1} \binom{n-j}{q} w^q (h_{k+i-1-n}(X_1,X_1^{-1},\ldots,X_{n-i+1},X_{n-i+1}^{-1}). \end{align*} $$

We replace i by $n-i+1$ and q by $n-j-q$ , and then we get rid of k by setting $p=k+l+q+1$ :

$$ \begin{align*}\sum_{p,q} \binom{j-1}{p} \binom{n-j}{q} w^{n-j-q} h_{p-q-l-1-i}(X_1,X_1^{-1},\ldots,X_{i},X_{i}^{-1}). \end{align*} $$

This is equal to (4.9) when taking into account the definition of complete symmetric functions $h_k$ for negative k’s as given in (4.7).