1.1 Quiver representations and stability

In the following, we will mainly be interested in studying the following quiver, which will be called the nested instantons quiver,

(1.1.1)

with relations

$$ \begin{align*} \begin{aligned} &[\alpha_0,\beta_0]+\xi\eta=0,\quad [\alpha_i,\beta_i]=0,\quad \alpha_{i-1}\phi_i-\phi_i\alpha_i=0=\beta_{i-1}\phi_i-\phi_i\beta_i\\ &\gamma_i\alpha_{i-1}-\alpha_i\gamma_i=0=\gamma_i\beta_{i-1}-\beta_i\gamma_i,\quad \phi_i\gamma_i=0,\quad \eta\phi_1=0,\quad \gamma_1\xi=0. \end{aligned} \end{align*} $$

Given a tuple of vector spaces $(W,V_0,\dots ,V_N)$ , one for each node of the quiver (1.1.1) and such that $\dim W=r$ , $\dim V_i=n_i$ , let ${\mathbb X}(r,{\mathbf n})$ be the linear space of representations of the quiver (1.1.1) with fixed dimension vector $(r,{\mathbf n})$ ; namely,

(1.1.2) $$ \begin{align} \begin{aligned} {\mathbb X}(r,{\mathbf n})=&\operatorname{\mathrm{End}} V_0^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\operatorname{\mathrm{End}}(V_1)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_1,V_0)\\ &\oplus\operatorname{\mathrm{Hom}}(V_0,V_1)\oplus\cdots\oplus\operatorname{\mathrm{End}}(V_{N})^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_{N},V_{N-1})\oplus\operatorname{\mathrm{Hom}}(V_{N-1},V_{N}). \end{aligned} \end{align} $$

A representation of numerical type $(r,{\mathbf n})$ of (1.1.1) with relations in the category of vector spaces will be given by the datum of a pair $X=(\mathbb W,h)$ , with $\mathbb W=(W,V_0,\dots ,V_{N})$ such that $\dim W=r$ and $\dim V_i=n_i$ , and

$$ \begin{align*}{\mathbb X}_0(r,{\mathbf n})\ni h=(B_1^0,B_2^0,I,J,B_1^1,B_2^1,F^1,G^1,\dots),\end{align*} $$

where ${\mathbb X}_0(r,{\mathbf n})\subset {\mathbb X}(r,{\mathbf n})$ is the closed subspace of ${\mathbb X}(r,{\mathbf n})$ whose morphisms satisfy the nested ADHM equations (1.1.3)

(1.1.3) $$ \begin{align} \begin{aligned} &[B_1^0,B_2^0]+IJ=0,\quad [B_1^i,B_2^i]=0,\quad B_1^{i-1}F^i-F^iB_1^i=0=B_2^{i-1}F^i-F^iB_2^i\\ &G^iB_1^{i-1}-B_1^iG^i=0=G^iB_2^{i-1}-B_2^iG^i,\quad F^iG^i=0,\quad JF^1=0,\quad G^1I=0. \end{aligned} \end{align} $$

In the following, we need to address the problem of King stability for representations of the nested instantons quiver. The definition of stability we will use follows from considering the moduli space of representations of the framed nested instantons quiver with a moduli space of representations of an auxiliary extended quiver with relations; [Reference King33, Reference Crawley-Boevey12].

In [Reference Bruzzo, Chuang, Diaconescu, Jardim, Pan and Zhang7, Reference von Flach and Jardim54], the two node case (namely, $N=1$ ) was considered. We can here generalize their result to the more general nested instantons quiver (1.1.1).

Proof. $\boldsymbol {(i)\Rightarrow (ii)}$ This is obvious, as a $\Theta $ -stable representation is also $\Theta $ -semistable.

$\boldsymbol {(ii)\Rightarrow (iii)}$ Let us first take a $\Theta $ -semistable representation X having at least one of the $F^i$ not injective. Without loss of generality, let $F^k$ be the only one to be such a map. Then $v_k\in \ker F^k\Rightarrow B_2^{k}v_k\in \ker F^k$ , due to the nested ADHM equations, and $B_2^{k}(\ker F^k)\subset \ker F^k$ (and similarly for $B_1^{k}$ ). Now

$$ \begin{align*}\tilde X=(0,\dots,0,\ker F^k,0,\dots,0,B_1^{k}|_{\ker F^k},B_2^{k}|_{\ker F^k},F^k,0,\dots,0)\end{align*} $$

is a subrepresentation of X of numerical type $(0,\dots ,0,\dim \ker F^k,0,\dots ,0)$ . Thus,

$$ \begin{align*}\tilde{{\mathbf n}}\cdot\boldsymbol\theta+\tilde r\theta_\infty=\theta_k\dim\ker F^k>0,\end{align*} $$

which contradicts the hypothesis of X being $\Theta $ -semistable. Let then X be a $\Theta $ -semistable representation satisfying S1. Let also $S\subseteq V_0$ be a $B^{0}_i$ -invariant subspace of $V_0$ , $i=1,2$ , such that $\operatorname {\mathrm {Im}}(I)\subseteq S$ , and define $\tilde V_0=S$ and $\tilde V_i=(F^1\circ \cdots \circ F^i)^{-1}(S)$ . Then $\tilde X=(W,\tilde V_0,\tilde V_1,\dots ,\tilde V_N)$ , with the morphisms induced by those of X, is a subrepresentation of X of numerical type $(r,\dim S,\dim \tilde V_1,\dots ,\dim \tilde V_N)$ . We have

$$ \begin{align*}\dim\tilde V_i=\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^i)\cap S)=n_i+\dim S - \dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^i)+S), \end{align*} $$

and by semistability,

$$ \begin{align*}{\mathbf n}\cdot\boldsymbol{\theta}=n_0\theta_0+\cdots+n_N\theta_N\ge\dim S\theta_0+\sum_{k=1}^N\left(n_k+\dim S-\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^k)+S)\right)\theta_k=\tilde{{\mathbf n}}\cdot\boldsymbol{\theta}, \end{align*} $$

whence

$$ \begin{align*}n_0(\theta_0+\cdots+\theta_N)\ge n_0\theta_0+\sum_{k=1}^N\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^k)+S)\theta_k\ge\dim S(\theta_0+\cdots+\theta_N). \end{align*} $$

Since $\theta _0+\cdots +\theta _N<0$ , this implies $n_0\le \dim S$ ; thus, $S=V_0$ .

$\boldsymbol {(iii)\Rightarrow (i)}$ If we take a proper subrepresentation $\tilde X$ of numerical type $(\tilde r,\tilde {{\mathbf n}})$ , we just need to check the cases $\tilde r=0$ and $\tilde r=r$ .

• ○ If $\tilde r=r$ , then $\tilde W=W$ , which, in turn, implies that $I\neq 0$ ; otherwise, the ADHM datum $(B_1^0,B_2^0,I,J)$ would not be stable. Since $\tilde X$ is proper, the following diagram commutes:

  (1.1.4)

  
  so that $\tilde n_0>0$ ; otherwise, we would have $I=0$ . Moreover, the following diagram also commutes (and so does the analogous one for $B_2^0$ ):
  (1.1.5)

  
  leading to a contradiction with the stability of $(W,V_0,B_1^0,B_2^0,I,J)$ . Since we are interested in proper subrepresentations of X, at least one $\tilde n_{i>0}$ is not zero, and at least one of these nonzero $\tilde n_k<n_k$ , so that $\boldsymbol \theta \cdot \tilde {{\mathbf n}}+\theta _\infty r<0$ , and X is stable.

• ○ Let now $\tilde r=0$ . Since we are interested in proper subrepresentations, we must choose $\tilde n_0>0$ ; otherwise, $\tilde V_{k>0}=0$ by virtue of the injectivity of $F_k$ . Similarly, one has $\tilde n_i\le \tilde n_{i-1}$ , so that

  $$ \begin{align*}\boldsymbol\theta\cdot\tilde{{\mathbf n}}=\sum_{k=0}^N\theta_k\tilde n_k\le\tilde n_0\sum_{k=0}^N\theta_k<0, \end{align*} $$
  and X is stable.

1.2 The nested instantons moduli space

We want now to discuss the construction of the moduli space of stable representations of the quiver (1.1.1) and its connection to GIT theory and stability. First, recall that we defined the space ${\mathbb X}(r,{\mathbf n})$ of nested ADHM data as in Equation (1.1.2), and an element $X\in {\mathbb X}(r,{\mathbf n})$ is called a nested ADHM datum. On ${\mathbb X}(r,{\mathbf n})$ , we have a natural action of $\mathcal G=\operatorname {\mathrm {GL}}(V_0)\times \cdots \times \operatorname {\mathrm {GL}}(V_{N})$ defined by

(1.2.1) $$ \begin{align} \begin{aligned} \Psi:(g_0,g_1,\dots,g_{N},X)\longmapsto(g_0B_1^0g_0^{-1},g_0B_2^0g_0^{-1},g_0I,Jg_0^{-1},\\ g_1B_1^1g_1^{-1},g_1B_2^1g_1^{-1},g_0F^1g_1^{-1},g_1G^1g_0^{-1},\\ \dots\\ g_{N}B_1^{N}g_{N}^{-1},g_{N}B_2^{N}g_{N}^{-1},g_{N-1}F^{N}g_{N}^{-1},g_{N}G^{N}g_{N-1}^{-1}). \end{aligned} \end{align} $$

Let now ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ be the open locus of $\Theta $ -stable representations in ${\mathbb X}(r,{\mathbf n})$ , where the stability chamber is chosen as in Proposition 1.2. The $\mathcal G$ -action on ${\mathbb X}(r,{\mathbf n})$ is free on the stable locus ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ . Indeed, it is known that the $\Theta $ -stable representations are exactly the simple objects in the category of linear $\Theta $ -semistable representations of a quiver Q, where we let be an algebraically closed field.Footnote ² Then given a $\Theta $ -stable representation X, any $\boldsymbol g\in \mathcal G$ such that $\boldsymbol g\cdot X=X$ defines an endomorphism in of the simple object X. However, is an abelian cateogry, and by Schur’s lemma, is a finite-dimensional division -algebra for any simple object X. Hence, . Thus, we must have , for , and the only such $\boldsymbol g\in \mathcal G$ fixing X corresponds to $\lambda =1$ . Also, it is easy to prove that the $\mathcal G$ -action preserves the space ${\mathbb X}_0(r,{\mathbf n})$ of nested ADHM data satisfying the relations (1.1.3).

Now if $\chi :\mathcal G\to {\mathbb C}^{*}$ is an algebraic character for the algebraic reductive group $\mathcal G$ , we can produce the moduli space of $\chi $ -semistable orbits following a construction due to [Reference King33], ${\mathcal N}^{ss}_\chi (r,{\mathbf n})$ , which is a quasi-projective scheme over ${\mathbb C}$ and is defined as

$$ \begin{align*} {\mathcal N}_\chi^{ss}(r,{\mathbf n})={\mathbb X}_0(r,{\mathbf n})/\!\!/_\chi\mathcal G=\operatorname{\mathrm{Proj}}\left(\bigoplus_{n\ge 0}A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi^n}\right) \end{align*} $$

with $A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi ^n}$ the space of polynomials in the coordinate ring $A({\mathbb X}_0(r,{\mathbf n}))$ satisfying

$$ \begin{align*}A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi^n}=\{f\in A({\mathbb X}_0(r,{\mathbf n}))|f(\boldsymbol h\cdot X)=\chi(\boldsymbol h)^nf(X),\forall\boldsymbol h\in\mathcal G\}.\end{align*} $$

The scheme ${\mathcal N}_\chi ^{ss}(r,{\mathbf n})$ contains an open subscheme ${\mathcal N}^{s}_\chi (r,{\mathbf n})\subset {\mathcal N}^{ss}_\chi (r,{\mathbf n})$ encoding $\chi $ -stable orbits. As in [Reference King33], it turns out that the notions of $\chi $ -stability and of $\Theta $ -stability are closely related.

Proposition 1.4. Let $\Theta =(\theta _0,\theta _1,\dots ,\theta _{N},\theta _\infty )\in \mathbb Z^{N+2}$ and define $\chi _\Theta :\mathcal G\to {\mathbb C}^{*}$ the character

(1.2.2) $$ \begin{align} \chi_\Theta(\boldsymbol h)=\det(h_0)^{-\theta_0}\cdots\det(h_{N})^{-\theta_{N}}. \end{align} $$

A representation X of the nested ADHM quiver (1.1.1) is $\chi _\Theta $ -(semi)stable iff it is $\Theta $ -(semi)stable.

Since the proof for Proposition 1.4 deeply relies on some known results about equivalent characterizations of $\chi $ -stability, we will first recall them. In full generality, let V be a vector space over ${\mathbb C}$ equipped with the action of a connected subgroup G of $U(V)$ , whose complexification is denoted by $G^{{\mathbb C}}$ . Then if $\chi :G\to U(1)$ is a character of G, we can extend it to form its complexification $\chi :G^{{\mathbb C}}\to {\mathbb C}^{*}$ . We then form the trivial line bundle $V\times {\mathbb C}$ , which carries an action of $G^{{\mathbb C}}$ via $\chi $ :

$$ \begin{align*} g\cdot (x,z)=(g\cdot x,\chi(g)^{-1}z),\qquad g\in G,(x,z)\in V\times{\mathbb C}. \end{align*} $$

Given the previous definition, the next lemma due to King [Reference King33] gives an alternative characterization of $\chi $ -(semi)stable points under the $G^{{\mathbb C}}$ -action.

With these notations, if $V^{\mathrm {ss}}(\chi )$ denotes the set of $\chi $ -semistable points of V, $V/\!\!/_\chi G^{{\mathbb C}}$ can be identified with $V^{\mathrm {ss}}(\chi )/\sim $ , where $x\sim y$ in $V^{\mathrm {ss}}(\chi )$ iff $\overline {G^{{\mathbb C}}\cdot x}\cap \overline {G^{{\mathbb C}}\cdot y}\neq \emptyset $ in $V^{\mathrm {ss}}(\chi )$ .

Proof of Prop. 1.4.

Notice first that , as the $\mathcal G$ -action on the open dense ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})\subset {\mathbb X}(r,{\mathbf n})$ is free; thus, the $\mathcal G$ -action on ${\mathbb X}(r,{\mathbf n})$ is effective.

Take a $\theta $ -semistable representation $X\in {\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ and assume it does not satisfy $\chi _\theta $ -semistability. Then there exists a $1$ -parameter subgroup $\lambda (t)$ of $\mathcal G$ such that $\lim _{t\to 0}\lambda (t)\cdot X$ exists and $\chi (\lambda )<0$ . However, each such $1$ -parameter subgroup $\lambda $ determines a filtration $\cdots \supseteq X_n\supseteq X_{n+1}\supseteq \cdots $ of subrepresentations of X ([Reference King33]), and

(1.2.3) $$ \begin{align} \chi_\theta(\lambda)=-\sum_{n\in\mathbb Z}\theta(X_n)\ge 0, \end{align} $$

thus proving one side of the proposition, as the part concerning stability is obvious from the fact that trivial subrepresentations of X correspond to subgroups in $\Delta $ .

Conversely, if X is a $\chi _{\theta }$ -semistable representation, we want to show that it is also a $\theta $ -semistable one. We only need to consider two cases, corresponding to subrepresentations $\tilde X$ of X with $\tilde r=r$ or $\tilde r=0$ . Each vector space in X, say $V_i$ , will have then a direct sum decomposition $V_i=\widetilde V_i\oplus \widehat V_i$ . We will then take a $1$ -parameter subgroup $\lambda (t)$ such that

(1.2.4)

Then one can easily compute

(1.2.5) $$ \begin{align} \begin{aligned} \chi_{\theta}(\lambda(t))\cdot z&=\left[\det(\lambda_0(t))^{-\theta_0}\cdots\det(\lambda_N(t))^{-\theta_N}\right]^{-1}\cdot z\\ &=t^{\tilde{{\mathbf n}}\cdot\boldsymbol\theta}z. \end{aligned} \end{align} $$

It is then a matter of a simple computation to verify that if X was not $\theta $ -semistable, then one would have had $\lim _{t\to 0}\lambda (t)\cdot X\in {\mathbb X}\times \{0\}$ , thus contradicting the $\chi _\theta $ -semistability. A completely analogous computation can be carried over when $\tilde r=r$ , taking

(1.2.6)

and since $(\tilde {{\mathbf n}}-{\mathbf n})\cdot \boldsymbol \theta>0$ if X is supposed not to be $\theta $ -semistable, this would still lead to a contradiction.

Finally, if X was to be $\chi _\theta $ -stable but not $\theta $ -stable, the $1$ -parameter subgroups previously described would have stabilized the pair $(X,z)$ , $z\neq 0$ , in the two different cases $\tilde r=0$ and $\tilde r=r$ , respectively, thus again giving rise to a contradiction.

Because of the previous corollary, in the stability chamber defined by Proposition 1.2, all notions of stability are actually the same, so that a representation satisfying anyone of the conditions in Corollary 1.7 will be called stable, and the corresponding ${\mathcal N}^{ss}_{\chi _\Theta }(r,{\mathbf n})={\mathcal N}(r,{\mathbf n})$ will be referred to as the moduli space of stable representations of quiver (1.1.1) with relations (1.1.3) or, equivalently, as the moduli space of nested instantons. Altogether, the previous considerations prove the following theorem.Footnote ³

Proof. We need to construct a perfect obstruction theory on ${\mathcal N}(r,{\mathbf n})$ . To this end, let $\mathbb Y(r,{\mathbf n})$ be the affine space of representations of the quiver obtained from (1.1.1) by neglecting all the $G^i\in \operatorname {\mathrm {Hom}}(V_{i-1},V_i)$ . Consider $\Theta =(\boldsymbol \theta ,\theta _\infty )\in \mathbb Q^{N+2}$ as in Proposition 1.2, and let $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})\subset \mathbb Y(r,{\mathbf n})$ be the open locus of $\Theta $ -stable points in $\mathbb Y(r,{\mathbf n})$ . Similarly to ${\mathbb X}(r,{\mathbf n})$ , on $\mathbb Y(r,{\mathbf n})$ , there is a natural $\mathcal G$ -action, which is free on the stable locus $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ . Thus, the GIT quotient $\mathbb Y(r,{\mathbf n})/\!/_{\chi _\Theta }\mathcal G$ , with $\chi _\Theta $ defined as in Proposition 1.4, exists as a smooth quasi-projective variety, which we will denote by $\mathcal Y(r,{\mathbf n})$ . The moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is then the closed subscheme of $\mathcal Y(r,{\mathbf n})$ cut by the nested ADHM equations (1.1.3), neglecting $G^i$ . We claim that there exists a vector bundle $\mathcal E$ over a smooth subvariety $\mathcal A$ of $\mathcal Y(r,{\mathbf n})$ , together with a section $\sigma \in H^0(\mathcal A,\mathcal E)$ , such that ${\mathcal N}(r,{\mathbf n})$ is isomorphic to the zero scheme $Z(\sigma )$ :

(1.2.7)

This induces a perfect obstruction theory on ${\mathcal N}(r,{\mathbf n})$ à la Behrend-Fantechi ([Reference Behrend and Fantechi3]) – that is, a perfect complex $\mathbb E\in \mathbf D^{[-1,0]}(Z(\sigma ))$ , together with a morphism in the derived category

where $\mathbb L_{Z(\sigma )}$ is the truncation $\tau ^{\ge -1}L^{\bullet }_{Z(\sigma )}$ of Illusie’s cotangent complex $L^{\bullet }_{Z(\sigma )}\in \mathbf D^{[-\infty ,0]}(Z(\sigma ))$ ([Reference Illusie32]), and ${\mathcal I}\subset \mathcal O_{\mathcal A}$ is the ideal sheaf of the inclusion $Z(\sigma )\hookrightarrow \mathcal A$ . In what follows, we will show how such a vector bundle $\mathcal E\longrightarrow \mathcal A$ is constructed.

On $\mathbb Y(r,{\mathbf n})$ , we introduce the vector bundles $C^i$ , $i=0,\dots ,3$ , defined as

$$ \begin{align*} C^0 &=\bigoplus_{i=0}^N\operatorname{\mathrm{End}}(V_i),\\ C^1 &=\operatorname{\mathrm{End}}(V_0)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\left[\bigoplus_{i=1}^N\left(\operatorname{\mathrm{End}}(V_i)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_i,V_{i-1}\right)\right]\\ & \cong T_{\mathbb Y(r,{\mathbf n})},\\ C^2 &=\operatorname{\mathrm{End}}(V_0)\oplus\operatorname{\mathrm{Hom}}(V_1,W)\oplus\left[\bigoplus_{i=1}^N\left(\operatorname{\mathrm{Hom}}(V_i,V_{i-1})^{\oplus 2}\oplus\operatorname{\mathrm{End}}(V_i)\right)\right],\\ C^3 &=\bigoplus_{i=1}^N\operatorname{\mathrm{Hom}}(V_i,V_{i-1}). \end{align*} $$

Each vector bundle $C^i$ , $i=0,\dots ,3$ carries a natural equivariant structure under the $\mathcal G$ -action on $\mathbb Y(r,{\mathbf n})$ , so they restrict to the stable locus $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ and then descend to vector bundles $\mathcal C^i$ on $\mathcal Y(r,{\mathbf n})$ under the quotient map $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})\xrightarrow {\pi }\mathcal Y(r,{\mathbf n})$ .

On $\mathbb Y(r,{\mathbf n})$ , we also have vector bundle homomorphisms $d^i:C^i\to C^{i+1}$ , $i=0,1,2$ , defined along the fibres as

$$\begin{align*}\begin{aligned} &d^0\begin{pmatrix} h_0\\ h_1\\ \vdots\\ h_N \end{pmatrix} = \begin{pmatrix} [h_0,B_1^0]\\ [h_0,B_2^0]\\ h_0I\\ -Jh_0\\ [h_1,B_1^1]\\ [h_1,B_2^1]\\ h_0F^1-F^1h^1\\ \vdots \end{pmatrix},\qquad d^1\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j\\ b_1^1\\ b_2^1\\ f^1\\ \vdots \end{pmatrix}=\begin{pmatrix} [b_1^0,B_2^0]+[B_1^0,b_2^0]+iJ+Ij\\ jF^1+Jf^1\\ B_1^0f^1+b_1^0F^1-F^1b_1^1-f^1B_1^1\\ B_2^0f^1+b_2^0F^1-F^1b_2^1-f^1B_2^1\\ [b_1^1,B_2^1]+[B_1^1,b_2^1]\\ \vdots\\ B_1^{N-1}f^N+b_1^{N-1}F^N-F^Nb_1^N-f^NB_1^N\\ B_2^{N-1}f^N+b_2^{N-1}F^N-F^Nb_2^N-f^NB_2^N\\ [b_1^N,B_2^N]+[B_1^N,b_2^N] \end{pmatrix},\\ &d^2\begin{pmatrix} c_1\\ c_2\\ c_3\\ \vdots\\ c_{3N+2} \end{pmatrix}=\begin{pmatrix} c_1F^1+B_2^0c_3-c_3B_2^1+c_4B_1^1-B_1^0c_4-Ic_2-F^1c_5\\ c_5F^2+B_2^1c_6-c_6B_2^2+c_7B_1^2-B_1^1c_7-F^2c_8\\ \vdots\\ c_{3i-1}F^i+B_{2}^{i-1}c_{3i}-c_{3i}B_2^i+c_{3i+1}B_1^i-B_1^{i-1}c_{3i+1}-F^ic_{3i+2}\\ \vdots \end{pmatrix}. \end{aligned} \end{align*}$$

Altogether, the datum of the vector bundles $C^i$ , $i=0,\dots ,3$ , and of the bundle homomorphisms $d^j$ , $j=0,1,2$ , descends to the datum of vector bundles $\mathcal C^i$ , $i=0,\dots ,3$ , and bundle homomorphisms $\delta ^j:\mathcal C^i\to \mathcal C^{i+1}$ , $j=0,1,2$ over $\mathcal Y(r,{\mathbf n})$ . Let us point out that when restricted to ${\mathcal N}(r,{\mathbf n})$ , the couple $(\mathcal C^{\bullet },\delta ^{\bullet })$ forms a complex of vector bundles over ${\mathcal N}(r,{\mathbf n})$ , which we claim to have vanishing cohomology in degrees $0$ and $3$ . Let $\mathcal C_A$ , $\mathcal C_B$ and $\mathcal C_{A,B}$ be the following auxiliary complexes of vector bundles over ${\mathcal N}(r,{\mathbf n})$ :Footnote ⁴

with

$$\begin{align*}\delta^0_A(h_0)=\begin{pmatrix} [h_0,B_1^0]\\ [h_0,B_2^0]\\ h_0I\\ -Jh_0 \end{pmatrix},\qquad \delta^1_A\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j \end{pmatrix}=[b_1^0,B_2^0]+[B_1^0,b_2^0]+Ij+iJ; \end{align*}$$

with

$$\begin{align*}\delta^0_B\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=\begin{pmatrix} [h_1,B_1^1]\\ [h_1,B_2^1]\\ \vdots\\ [h_N,B_1^N]\\ [h_N,B_2^N] \end{pmatrix},\qquad \delta^1_B\begin{pmatrix} b_1^1\\ b_2^1\\ \vdots\\ b_1^N\\ b_2^N \end{pmatrix}=\begin{pmatrix} [b_1^1,B_2^1]+[B_1^1,b_2^1]\\ \vdots\\ [b_1^N,B_2^N]+[B_1^N,b_2^N] \end{pmatrix}; \end{align*}$$

with

$$\begin{align*}\delta^0_{A,B}\begin{pmatrix} f^1\\ \vdots\\ f^N \end{pmatrix}=\begin{pmatrix} -B_1^0f^1+f^1B_1^1\\ -B_2^0f^1+f^1B_2^1\\ \vdots\\ -B_1^{N-1}f^N+f^NB_1^N\\ -B_2^{N-1}f^N+f^NB_2^N\\ -Jf^1 \end{pmatrix}, \end{align*}$$

$$\begin{align*}\delta^1_{A,B}\begin{pmatrix} c_3\\ \vdots\\ c_{2N+2}\\ c_2 \end{pmatrix}=\begin{pmatrix} -B_2^0c_3+c_3B_2^1-c_4B_1^1+B_1^0c_4+Ic_2\\ -B_2^1c_5+c_5B_2^2-c_6B_1^2+B_1^1c_6\\ \vdots\\ -B_2^{N-1}c_{2N+1}+c_{2N+1}B_2^N-c_{2N+2}B_1^N+B_1^{N-1}c_{2N+2} \end{pmatrix}. \end{align*}$$

It is then readily verified that there exists a distinguished triangle

(1.2.8)

coming from the fact that $\mathcal C[1]$ is a cone for $\rho =(\rho _0,\rho _1,\rho _2)$ , where

(1.2.9a) $$ \begin{align} \rho_0\begin{pmatrix} h_0\\ \vdots\\ h_N \end{pmatrix}&=\begin{pmatrix} -h_0F^1+F^1h_1\\ \vdots\\ -h_{N-1}F^N+F^Nh_N \end{pmatrix}, \end{align} $$

(1.2.9b) $$ \begin{align} \rho_1\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j\\ b_1^1\\ b_2^1\\ \vdots \end{pmatrix}&=\begin{pmatrix} -b_1^0F^1+F^1b_1^1\\ -b_2^0F^1+F^1b_2^1\\ \vdots\\ -b_1^{N-1}F^N+F^Nb_1^N\\ -b_2^{N-1}F^N+F^Nb_2^N\\ -jF^1 \end{pmatrix}, \end{align} $$

(1.2.9c) $$ \begin{align} \rho_2\begin{pmatrix} c_1\\ c_{2N+3}\\ \vdots\\ c_{3N+3} \end{pmatrix}&=\begin{pmatrix} -c_1F^1+F^1c_{2N+3}\\ \vdots\\ -c_{3N+2}F^N+F^Nc_{3N+3} \end{pmatrix}. \end{align} $$

From the triangle (1.2.8), one gets a long exact sequence in cohomology

and since $\mathcal C_A$ is simply the deformation complex of the standard ADHM quiver, it is well known that $H^0(\mathcal C_A)=H^2(\mathcal C_A)=0$ , [Reference Nakajima and Society43]. Then $H^0(\mathcal C)=0$ by the injectivity of $H^0(\rho ):H^0(\mathcal C_B)\to H^0(\mathcal C_{A,B})$ . In fact, we have

$$\begin{align*}H^0(\rho)\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=0\Rightarrow\begin{pmatrix} F^1h_1\\ \vdots\\ F^Nh_N \end{pmatrix}=0\Rightarrow\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=0 \end{align*}$$

since $F^i$ is injective. Moreover, the map $\mathcal \delta ^1_{A,B}:\mathcal C_{A,B}^1\to \mathcal C_{A,B}^2$ is surjective: this, in turn, means that $H^2(\mathcal C_{A,B})=0$ , which implies $H^3(\mathcal C)=0$ . Indeed, let us consider $(\delta ^1_{A,B})^\vee $ . One has

$$\begin{align*}(\delta^1_{A,B})^\vee(\boldsymbol\phi)=(\delta^1_{A,B})^\vee\begin{pmatrix} \phi_1\\ \vdots\\ \phi_N \end{pmatrix}=\begin{pmatrix} B_2^1\phi_1-\phi_1B_2^0\\ -B_1^1\phi_1+\phi_1B_1^1\\ \vdots\\ B_2^N\phi_N-\phi_NB_2^{N-1}\\ -B_1^N\phi_N+\phi_NB_1^{N-1}\\ \phi_1I \end{pmatrix}, \end{align*}$$

and if $\boldsymbol \phi \in \ker ((\delta ^1_{A,B})^\vee )$ , then $\ker (\phi _1)$ would be a $(B_1^0,B_2^0)$ -invariant subset of $V_0$ containing $\operatorname {\mathrm {Im}}(I)$ , which contradicts the stability of X, by which we conclude that $\ker (\phi _1)=V_0$ . Similar statements also hold true for each other component of $\boldsymbol \phi $ , which we then conclude to be $\boldsymbol \phi =0$ .

Let now s be the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ cutting via its zero locus $Z(s)\subset \mathbb Y(r,{\mathbf n})$ the solutions to the nested ADHM equations (1.1.3) (neglecting $G^i$ ) in $\mathbb Y(r,{\mathbf n})$ – that is,

$$\begin{align*}\begin{aligned} s(X)=\left([B_1^0,B_2^0]+IJ\right)\oplus JF^1\oplus\bigoplus_{i=1}^N\bigg(\left(F^iB_1^i-B_1^{i-1}F^i\right)\oplus\left(F^iB_2^i-B_2^{i-1}F^i\right)\oplus [B_1^i,B_2^i]\bigg). \end{aligned} \end{align*}$$

As s is naturally $\mathcal G$ -equivariant, it also restricts to the stable locus and descends to a section $\sigma \in H^0(\mathcal Y(r,{\mathbf n}),\mathcal C^2)$ . It is easily checked that $d^2\circ s=0$ , so $s(\mathbb Y(r,{\mathbf n}))\subseteq \ker (C^2\xrightarrow {d^2}C^3)$ . Let now $A\subset \mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ be the open subscheme of $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ where $d^2$ is surjective, and whose image under the quotient map is $\mathcal A\subseteq \mathcal Y(r,{\mathbf n})$ . Then $\ker (C^2|_A\xrightarrow {d^2}C^3|_A)$ is a $\mathcal G$ -equivariant vector bundle $E\longrightarrow A$ , which descends to a vector bundle ${\mathcal E}\longrightarrow \mathcal A$ . Also, $s|_A\in H^0(A,E)$ is a $\mathcal G$ -equivariant section, inducing a section $\sigma \in H^0(\mathcal A,{\mathcal E})$ . The moduli space of nested instantons is then constructed as the zero scheme $Z(\sigma )\subset \mathcal A$ . Indeed, as $\delta ^2$ is surjective over ${\mathcal N}(r,{\mathbf n})$ , we have ${\mathcal N}(r,{\mathbf n})\subset \mathcal A$ . Moreover, as the section $\sigma \in H^0(\mathcal A,\mathcal C^2)$ factors through a section of the subbundle ${\mathcal E}\longrightarrow \mathcal A$ , then $Z(s)$ and ${\mathcal N}(r,{\mathbf n})$ define the same closed subscheme of $\mathcal A$ .

Thus, we are precisely in the situation of diagram (1.2.7), and ${\mathcal N}(r,{\mathbf n})\cong Z(\sigma )$ is naturally equipped with a perfect obstruction theory of rank $\operatorname {\mathrm {vd}}=\operatorname {\mathrm {rk}}\mathbb E=(2n_0-n_1)r$ , whose virtual tangent complex is

$$\begin{align*}\mathbb E^\vee=\left[T_{\mathcal A}|_{{\mathcal N}(r,{\mathbf n})}\xrightarrow{{\text d}\sigma}{\mathcal E}|_{{\mathcal N}(r,{\mathbf n})}\right]\in\mathbf D^{[-1,0]}({\mathcal N}(r,{\mathbf n})). \end{align*}$$

In particular, retracing all the steps we have discussed so far, we can see that the K-theory class of the virtual tangent space over a point $X\in {\mathcal N}(r,{\mathbf n})$ attached to the perfect obstruction theory $\mathbb E$ is

$$\begin{align*}\left.T^{\mathrm{vir}}_{{\mathcal N}(r,{\mathbf n})}\right|_X=H^1(\mathcal C^{\bullet}|_X)-H^2(\mathcal C^{\bullet}|_X)\in K^0(\operatorname{\mathrm{pt}}).\end{align*}$$

Remark 1.9. With the notation introduced in the proof of Theorem 1.8, the moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is also equipped with another perfect obstruction theory, induced by the vector bundle $C^2\longrightarrow \mathbb Y(r,{\mathbf n})$ , together with the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ . Indeed, after restriction to the stable locus, the couple $(C^2,s)$ descends to a vector bundle and a section $(\mathcal C^2,\sigma )$ over $\mathcal Y(r,{\mathbf n})$ , and ${\mathcal N}(r,{\mathbf n})\cong Z(\sigma )\subset \mathcal Y(r,{\mathbf n})$ . Then, ${\mathcal N}(r,{\mathbf n})$ is equipped with a second perfect obstruction theory $\tilde {\mathbb E}$ , whose tangent complex is $\tilde {\mathbb E}^\vee =[T_{\mathcal Y(r,{\mathbf n})}|_{{\mathcal N}(r,{\mathbf n})}\xrightarrow {{\text d}\sigma }\mathcal C^2|_{{\mathcal N}(r,{\mathbf n})}]$ . However, the rank of $\tilde {\mathbb E}$ is

$$\begin{align*}\operatorname{\mathrm{rk}}\tilde{\mathbb E}=(2n_0-n_1)r-\sum_{i=1}^Nn_in_{i-1}, \end{align*}$$

which might become negative (e.g., this is the case for $r=1,n_0\ge 3,n_1=n_0-1$ ). This is due to the relations (1.1.3) being overdetermined, as they are not independent. Thus, in Theorem 1.8, we constructed a ‘reduced’ perfect obstruction theory, which automatically takes into account the dependency of the equations cutting the moduli space of nested instantons by exploiting the fact that $s(\mathbb Y(r,{\mathbf n}))\subseteq \ker (C^2\xrightarrow {d^2}C^3)$ .

For future reference, we want now to exhibit some morphisms between different nested instantons moduli spaces and between them and usual moduli spaces of instantons, which are moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ . We obviously have iterative forgetting projections $\eta _i:{\mathcal N}(r,n_0,\dots ,n_i)\to {\mathcal N}(r,n_0,\dots ,n_{i-1})$ . Moreover, we also have other morphisms to underlying moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ , which are summarized by the commutative diagram in Figure 3.

Figure 3 Morphisms between moduli spaces of sheaves.

In order to see that these maps do indeed exist, take a stable representation $[X]$ of the nested instantons quiver. The fact that $[X]$ is stable implies that the morphisms $F^i$ are injective, so that we can construct the stable ADHM datum $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ as follows. Let $\tilde V_i$ be $V_0/\operatorname {\mathrm {Im}}(F^1\cdots F^i)$ and choose a basis of $V_i$ in such a way that

(1.2.10)

whence $V_0=V_i\oplus \tilde V_i$ . Then define the projections $\pi _i=V_0\to V_i$ and $\tilde \pi _i:V_0\to \tilde V_i$ as $\pi _i(v,\tilde v)=v$ and $\tilde \pi _i(v,\tilde v)=\tilde v$ , with $v\in V_i$ , $\tilde v\in \tilde V_i$ . We can then show how $\tilde V_i$ inherits an ADHM structure by its embedding in $V_0$ . Indeed, if we define $\tilde B_1^i=B_1^0|_{\tilde V_i}$ , $\tilde B_2^i=B_2^0|_{\tilde V_i}$ , $\tilde I^i=\tilde \pi _i\circ I$ and $\tilde J^i=J|_{\tilde V_i}$ , the datum $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ satisfies the ADHM Equation (1.2.11):

(1.2.11) $$ \begin{align} [\tilde B_1^i,\tilde B_2^i]+\tilde I^i\tilde J^i=[B_1^0|_{\tilde V_i},B_2^0|_{\tilde V_i}]+\tilde\pi_i\circ I\circ J|_{\tilde V_i}=\bigg(\left[B_1^0,B_2^0\right]+IJ\bigg)\bigg|_{\tilde V_i}=0. \end{align} $$

This new ADHM datum is moreover stable, as if it would exist $0\subset \tilde S_i\subset \tilde V_i$ such that $\tilde B_{1,2}^i(\tilde S_i),\tilde I^i(W)\subset \tilde {S_i}$ it would imply that also the ADHM datum $(W,V_0,B_1^0,B_2^0,I,J)$ would not be stable. In fact, in that case, we could take $0\subset V_i\oplus \tilde S_i\subset V_0$ , and it would be such that $B_1^0(V_i\oplus \tilde S_i),B_2^0(V_i\oplus \tilde S_i),I(W)\subset V_i\oplus \tilde S_i$ . In fact, if we take any $(v,\tilde s)\in V_i\oplus \tilde S_i$ , it happens that $B_1^0(v,s)=(B_1^0|_{V_i}(v),B_1^0|_{\tilde V_i}\tilde s)=(B_1^0|_{V_i},\tilde B_1^i(\tilde s))\in V_i\oplus \tilde S_i$ , $B_2^0(v,s)=(B_2^0|_{V_i}(v),B_2^0|_{\tilde V_i}\tilde s)=(B_2^0|_{V_i},\tilde B_2^i(\tilde s))\in V_i\oplus \tilde S_i$ and $I(W)=I(W)\cap V_i\oplus I(W)\cap \tilde V_i=(\pi _i\circ I)(W)\oplus (\tilde \pi _i\circ I)(W)\subset V_i\oplus \tilde S_i$ . Thus, we constructed a map $\mathfrak p^{(N)}_i:{\mathcal N}(r,n_0,\dots ,n_{N})\to \mathcal M(r,n_0-n_i)$ .

1.3 Nakajima quiver varieties

In this section, we exhibit an embedding of the moduli space of nested instantons into a Nakajima quiver variety, which is a smooth hyperkähler variety; [Reference Nakajima41, Reference Nakajima42, Reference Ginzburg22]. In the vector space ${\mathbb X}(r,{\mathbf n})$ of representations of quiver (1.1.1) of numerical type $(r,{\mathbf n})$ , which, we recall, is defined as

(1.3.1) $$ \begin{align} \begin{aligned} {\mathbb X}(r,{\mathbf n})=\operatorname{\mathrm{End}}(V_0)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\bigoplus_{k=1}^{N}\left[\operatorname{\mathrm{End}}(V_k)^{\oplus 2}\right.\\ \left.\oplus\operatorname{\mathrm{Hom}}(V_{k-1},V_k)\oplus\operatorname{\mathrm{Hom}}(V_k,V_{k-1})\right], \end{aligned} \end{align} $$

we will introduce a family of relations:

(1.3.2) $$ \begin{align} & [B_1^0,B_2^0]+IJ+F^1G^1=0, \end{align} $$

(1.3.3) $$ \begin{align} & [B_1^i,B_2^i]-G^iF^i+F^{i+1}G^{i+1}=0,\quad i=1,\dots,N. \end{align} $$

Then a point $X\in {\mathbb X}(r,{\mathbf n})$ is called stable if it satisfies conditions $\mathbf {S1}$ and $\mathbf {S2}$ in Proposition 1.2. With these conventions, we will define $\mathbb M(r,{\mathbf n})$ to be the space of stable elements of ${\mathbb X}(r,{\mathbf n})$ satisfying the relations (1.3.2)--(1.3.3):

(1.3.4) $$ \begin{align} \mathbb M(r,{\mathbf n})=\{X\in{\mathbb X}(r,{\mathbf n}):X\ \textrm{is stable and satisfies } (1.3.2), (1.3.3)\}. \end{align} $$

Exactly in the same way as we did before, we can easily see that there is a natural action of $\mathcal G=\operatorname {\mathrm {GL}}(V_0)\times \cdots \times \operatorname {\mathrm {GL}}(V_N)$ which is free on $\mathbb M(r,{\mathbf n})$ and preserves the equations 1.3.2--1.3.3: the same is then true for the natural $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ , with $\mathcal U=U(V_0)\times \cdots \times U(V_N)$ . Thus, a moduli space $\mathcal M(r,{\mathbf n})$ of stable $\mathcal U$ -orbits in $\mathbb M(r,{\mathbf n})$ can be defined by means of GIT theory, as was the case for ${\mathcal N}(r,{\mathbf n})$ in the previous sections. Moreover, any stable point of ${\mathbb X}$ satisfying the nested ADHM equations automatically satisfies (1.3.2) and (1.3.3). Indeed, a stable representation of quiver (1.1.1) satisfies, among other relations, the equations

$$\begin{align*}[B_1^0,B_2^0]+IJ=0,\qquad [B_1^{i>0},B_2^{i>0}]=0, \end{align*}$$

while $G^i=0$ , for $i=1,\dots ,N$ , by Corollary 1.3. Thus, any stable representation of quiver (1.1.1) with relations (1.1.3) also satisfies the relations (1.3.2)--(1.3.3) so that ${\mathcal N}(r,{\mathbf n})\hookrightarrow \mathcal M(r,{\mathbf n})$ via the natural inclusion.

Next, let us point out that on each $T\operatorname {\mathrm {Hom}}(V_i,V_k)$ , we can introduce a hermitean metric by defining

(1.3.5) $$ \begin{align} \langle X,Y\rangle=\frac{1}{2}\operatorname{\mathrm{tr}}\left(X\cdot Y^\dagger+X^\dagger\cdot Y\right),\qquad\forall X,Y\in\operatorname{\mathrm{Hom}}(V_i,V_k), \end{align} $$

which, in turn, can be linearly extended to a hermitean metric $\langle -,-\rangle :T\mathbb M(r,{\mathbf n})\times T\mathbb M(r,{\mathbf n})\to {\mathbb C}$ . Finally, we can introduce some complex structures on $T\mathbb M(r,{\mathbf n})$ : given $X\in T\mathbb M(r,{\mathbf n})$ , these are defined as the following $I,J,K\in \operatorname {\mathrm {End}}(T\mathbb M(r,{\mathbf n}))$

(1.3.6) $$ \begin{align} & I(X)=\sqrt{-1}X, \end{align} $$

(1.3.7) $$ \begin{align} & J(X)=(-b_2^{0\dagger},b_1^{0\dagger}, -j^\dagger,i^\dagger,\{-b_2^{i\dagger},b_1^{i\dagger},-g^{i\dagger},f^{i\dagger}\}), \end{align} $$

(1.3.8) $$ \begin{align} & K(X)=I\circ J(X), \end{align} $$

with $X=(b_1^0,b_2^0,i,j,\{b_1^i,b_2^i,f^i,g^i\})$ . These three complex structures make the datum of

$$ \begin{align*}(\mathbb M(r,{\mathbf n}),\langle-,-\rangle,I,J,K)\end{align*} $$

a hyperkähler manifold, as one can readily verify. It is a standard fact that once we fix a particular complex structure, say I, and its respective Kähler form, $\omega _I$ , the linear combination $\omega _{{\mathbb C}}=\omega _J+\sqrt {-1}\omega _K$ is a holomorphic symplectic form for $\mathbb M(r,{\mathbf n})$ . The thing we finally want to prove is that the hyperkähler structure on $\mathbb M(r,{\mathbf n})$ induces a hyperkähler structure on the GIT quotient $\mathcal M(r,{\mathbf n})$ , which will be moreover proven to be smooth. This is made possible by the fact that the natural $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ preserves the hermitean metric and the complex structures we introduced. Then, letting ${\mathfrak u}$ be the Lie algebra of the group $\mathcal U$ , we need to construct a moment map

$$ \begin{align*} \mu:\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}\otimes\mathbb R^3, \end{align*} $$

satisfying

1. 1. $\mathcal G$ -equivariance: $\mu (g\cdot X)=\operatorname {\mathrm {Ad}}_{g^-1}^{*}\mu (X)$ ;

2. 2. $\langle {\text d}\mu _i(X),\xi \rangle =\omega _i(\xi ^{*},X)$ , for any $X\in T\mathbb M(r,{\mathbf n})$ and $\xi \in {\mathfrak u}$ generating the vector field $\xi ^{*}\in T\mathbb M(r,{\mathbf n})$ .

If then $\zeta \in {\mathfrak u}^{*}\otimes \mathbb R^3$ is such that $\operatorname {\mathrm {Ad}}_g^{*}(\zeta _i)=\zeta _i$ for any $g\in \mathcal U$ , $\mu ^{-1}(\zeta )$ is $\mathcal U$ -invariant, and it makes sense to consider the quotient space $\mu ^{-1}(\zeta )/\mathcal U$ . It is known ([Reference Hitchin, Karlhede, Lindström and Roček30]) that if $\mathcal U$ acts freely on $\mu ^{-1}(\zeta )/\mathcal U$ , the latter is a smooth hyperkähler manifold, with complex structures and metric induced by those of $\mathbb M(r,{\mathbf n})$ .

Our task of finding a moment map $\mu :\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}\otimes \mathbb R^3$ then translates into the following. Define $(\mu _1^0,\dots ,\mu _1^N)=\mu _1:\mathbb M(r,{\mathbf n})\to {\mathfrak u}$

(1.3.9) $$ \begin{align} \left\{ \begin{aligned} &\mu_1^0(X)=\frac{\sqrt{-1}}{2}\left([B_1^0,B_1^{0\dagger}]+[B_2^0,B_2^{0\dagger}]+II^\dagger-J^\dagger J+F^1F^{1\dagger}-G^{1\dagger}G^1\right)\\ &\mu_1^1(X)=\frac{\sqrt{-1}}{2}\left([B_1^1,B_1^{1\dagger}]+[B_2^1,B_2^{1\dagger}]-F^{1\dagger}F^1+G^1G^{1\dagger}+F^2F^{2\dagger}-G^{2\dagger}G^2\right)\\ &\qquad\qquad\vdots\\ &\mu_1^N(X)=\frac{\sqrt{-1}}{2}\left([B_1^N,B_1^{N\dagger}]+[B_2^N,B_2^{N\dagger}]-F^{N\dagger}F^N+G^NG^{N\dagger}\right), \end{aligned} \right. \end{align} $$

with $X=(B_1^0,B_2^0,I,J,\{B_1^i,B_2^i,F^i,G^i\})\in \mathbb M(r,{\mathbf n})$ . In addition to $\mu _1$ , we also define a map $\mu _{{\mathbb C}}:\mathbb M(r,{\mathbf n})\to \mathfrak {g}$ , with $\mathfrak g=\mathfrak {gl}(V_0)\times \cdots \times \mathfrak {gl}(V_N)$ :

(1.3.10) $$ \begin{align} \left\{ \begin{aligned} &\mu_{{\mathbb C}}^0(X)=[B_1^0,B_2^0]+IJ+F^1G^1\\ &\mu_{{\mathbb C}}^1(X)=[B_1^1,B_2^1]-G^1F^1+F^2G^2\\ &\qquad\qquad\vdots\\ &\mu_{{\mathbb C}}^N(X)=[B_1^N,B_2^N]-G^NF^N, \end{aligned} \right. \end{align} $$

by means of which we define $\mu _{2,3}:\mathbb M(r,{\mathbf n})\to {\mathfrak u}$ as $\mu _{{\mathbb C}}(X)=(\mu _2+\sqrt {-1}\mu _3)(X)$ . Notice that in absence of $B_i^j$ and $I,J$ the complex moment map we defined would reduce to the Crawley-Boevey moment map in [Reference Crawley-Boevey12]. We then claim that $\mu =(\mu _1,\mu _2,\mu _3)$ is a moment map for the $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ . If this is true and $\chi $ is the algebraic character we introduced in §1.2, the space

(1.3.11) $$ \begin{align} \widetilde{\mathcal M}(r,{\mathbf n})=\frac{\mu_1^{-1}(\sqrt{-1}{\text d}\chi)\cap\mu_{{\mathbb C}}^{-1}(0)\cap\mathbb M(r,{\mathbf n})}{\mathcal U}=\frac{\mu^{-1}(\sqrt{-1}{\text d}\chi,0,0)\cap\mathbb M(r,{\mathbf n})}{\mathcal U} \end{align} $$

is a smooth hyperkähler manifold which, by an analogue of Kempf-Ness theorem, is also isomorphic to $\mathcal M(r,{\mathbf n})$ . In fact, it is known, due to a result of [Reference King33, Reference Nakajima and Society43] and the characterization of $\chi $ -(semi)stable points we gave in the previous sections, that there exists a bijection between $\mu _1^{-1}(\sqrt {-1}{\text d}\chi )$ and the set of $\chi $ -(semi)stable points in $\mu _{{\mathbb C}}^{-1}(0)$ . Then, in order to prove that $\mu $ is actually a moment map, we will first compute the vector field $\xi ^{*}$ generated by a generic $\xi \in {\mathfrak u}$ . Let then $X=(b_1^0,b_2^0,i,j,\{b_1^i,b_2^i,f^i,g^i\})$ be a vector in $T\mathbb M(r,{\mathbf n})$ and $\Psi _X:\mathcal U\to \mathbb M(r,{\mathbf n})$ the action of $\mathcal U$ onto $X\in \mathbb M(r,{\mathbf n})$ : the fundamental vector field generated by $\xi \in {\mathfrak u}$ is

(1.3.12)

where $\gamma $ is a smooth curve $\gamma :(-\epsilon ,\epsilon )\to \mathcal U$ such that and $\dot \gamma (0)=\xi $ . Thus, we can compute

(1.3.13) $$ \begin{align} \begin{aligned} \xi^{*}|_X=\left([\xi_0,b_1^0],[\xi_0,b_2^0],\xi_0i,-j\xi_0,[\xi_1,b_1^1],[\xi_1,b_2^1],\xi_0f^1-f^1\xi_1,\xi_1g^1-g^1\xi_0,\dots\right.\\ \left.\dots,[\xi_N,b_1^N],[\xi_N,b_2^N],\xi_{N-1}f^N-f^N\xi_N,\xi_Ng^N-g^N\xi_{N-1}\right). \end{aligned} \end{align} $$

Then if $\pi _i:\mathbb M(r,{\mathbf n})\to \mathbb M(r,{\mathbf n})$ denotes the projection on the i-th component of the direct sum decomposition induced by (1.3.1) so that i runs over the index set ${\mathcal I}$ , by inspection, one can see that $\omega _1$ is exact, and in particular, $\omega _1={\text d}\lambda _1$ , with

(1.3.14) $$ \begin{align} \lambda_1=\frac{\sqrt{-1}}{2}\operatorname{\mathrm{tr}}\left(\sum_{i\in{\mathcal I}}\pi_i\wedge\pi_{i*}^\dagger\right). \end{align} $$

This implies that

(1.3.15) $$ \begin{align} \langle\mu_1(x),\xi\rangle=\imath_{\xi^{*}}\lambda_1, \end{align} $$

and it is easy to verify that $\mu _1:\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}$ thus defined indeed matches with the definition (1.3.9). Similarly, one can realize that

(1.3.16) $$ \begin{align} &\lambda_2=\Re\left[\operatorname{\mathrm{tr}}\left(\sum_{i\in2\mathbb Z\cap{\mathcal I}}\pi_i\wedge\pi_{1+1*}\right)\right], \end{align} $$

(1.3.17) $$ \begin{align} &\lambda_3=-\sqrt{-1}\Im\left[\operatorname{\mathrm{tr}}\left(\sum_{i\in2\mathbb Z\cap{\mathcal I}}\pi_i\wedge\pi_{1+1*}\right)\right], \end{align} $$

and the moment map components satisfying $\langle \mu _i(x),\xi \rangle =\imath _{\xi ^{*}}\lambda _i$ agree with the combination $\mu _2+\sqrt {-1}\mu _3=\mu _{{\mathbb C}}$ we gave previously in Equation (1.3.10).

2.1 $\operatorname {\mathrm {Hilb}}^{\hat {{\mathbf n}}}({\mathbb C}^2)$ and ${\mathcal N}(1,{\mathbf n})$

Before delving into the analysis of the relation between nested instantons moduli spaces and flags of framed torsion-free sheaves on ${\mathbb P}^2$ , we want to show a special simpler case. In particular, we will prove the existence of an isomorphism between the nested Hilbert scheme of points in ${\mathbb C}^2$ and the nested instantons moduli space ${\mathcal N}(1,n_0,\dots ,n_N)$ . This effectively gives us the ADHM construction of a general nested punctual nested Hilbert scheme on ${\mathbb C}^2$ , which will also be the local model for more general nested Hilbert schemes of points on, say, toric surfaces S. In order to see this, we first recall the definition of a nested Hilbert scheme of points.

Definition 2.1. Let S be a complex (projective) surface and $n_1\ge n_2\ge \cdots \ge n_k$ a sequence of integers. The nested Hilbert scheme of points on S is defined as

(2.1.1) $$ \begin{align} \operatorname{\mathrm{Hilb}}^{(n_1,\dots,n_k)}(S)=S^{[n_1,\dots,n_k]}=\left\{I_1\subseteq I_2\subseteq\cdots\subseteq I_k\subseteq\mathcal O_S:\mathrm{length}\left(\mathcal O_S/I_i\right)=n_i\right\}. \end{align} $$

Alternatively, if X is a quasi-projective scheme over the complex numbers, we can equivalently define the nested Hilbert scheme $X^{[n_1,\dots ,n_k]}=\operatorname {\mathrm {Hilb}}^{(n_1,\dots ,n_k)}(X)$ as

(2.1.2) $$ \begin{align} \operatorname{\mathrm{Hilb}}^{(n_1,\dots,n_k)}(X)=\left\{(Z_1,\dots,Z_k):Z_i\in\operatorname{\mathrm{Hilb}}^{n_i}(X),\ Z_i\ \mathrm{is\ a\ subscheme\ of }\ Z_j\ \mathrm{if }\ i<j\right\}, \end{align} $$

with $Z_i$ being a zero-dimensional scheme, for every $i=1,\dots ,k$ .

Before actually exhibiting the isomorphism we are interested in, we want to prove an auxiliary result, which gives an alternative definition for the nested Hilbert schemes over the affine plane, analogously to the case of Hilbert schemes studied in [Reference Nakajima and Society43].

Proposition 2.2. Let be an algebraically closed field, and ${\mathbf n}$ a sequence of integers $n_0\ge n_1\ge \cdots \ge n_k$ . Define $\hat {{\mathbf n}}$ to be the sequence of integers $\hat {n}_0=n_0\ge \hat {n}_1=n_0-n_k\ge \cdots \ge \hat {n}_k=n_0-n_1$ . Then there exists an isomorphism

(2.1.3)

where , , and . The action of $\mathcal G_{{\mathbf n}}$ is given by

$$ \begin{align*} \boldsymbol g\cdot(b_1^0,b_2^0,i,\dots,b_1^k,b_2^k,f_k)=(g_0b_1^0g_0^{-1},g_0b_2^0g_0^{-1},g_0i,\dots,g_kb_1^kg_k^{-1},g_kb_2^kg_k^{-1},g_{k-1}f_kg_k^{-1}). \end{align*} $$

One can readily notice that the description given by the previous proposition of the nested Hilbert scheme of points does not really coincide with the quiver we were studying throughout this section. However, we can very easily overcome this problem by using the fact that if $(b_1^0,b_2^0,i,j)$ is a stable ADHM datum with $r=1$ , then $j=0$ ; [Reference Nakajima and Society43]. This proves the following proposition.

Proposition 2.3. With the same notations of Proposition 2.2, we have that

All the previous observations, together with corollary 1.7, immediately prove the following theorem.

Theorem 2.4. The moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is isomorphic to the nested Hilbert scheme of points on ${\mathbb C}^2$ when $r=1$ .

$$\begin{align*}{\mathcal N}(1,{\mathbf n})={\mathbb X}_0/\!\!/_\chi\mathcal G\cong\operatorname{\mathrm{Hilb}}^{\hat{{\mathbf n}}}({\mathbb C}^2).\end{align*}$$

2.2 $\mathcal F(r,\boldsymbol \gamma )$ and ${\mathcal N}(r,{\mathbf n})$

A more general result relates the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ to the moduli space of nested instantons. In the case of the two-step quiver, this result was proved in [Reference von Flach and Jardim54]. Here, we give a generalization of their theorem in the case of the moduli space ${\mathcal N}_{r,[r^1],n,\mu }$ represented by a quiver with an arbitrary number of nodes.

By the framing condition, we get that $c_1(E_N)=0$ , while the quotient condition on the subsheaves of $E_N$ naturally implies that the quotients $E_i/E_j$ are $0$ -dimensional sheaves and $c_1(E_j)=0$ , for all $j=0,\dots ,N$ . Then a framed flag of sheaves on ${\mathbb P}^2$ is characterized by the set of integers $(r,\boldsymbol \gamma )$ , where $r=\operatorname {\mathrm {rk}} E_0=\cdots =\operatorname {\mathrm {rk}} E_N$ , $c_2(E_N)=\gamma _N$ , $h^0(E_N/E_j)=\gamma _j+\cdots +\gamma _{N-1}$ so that $c_2(E_j)=\gamma _j+\cdots +\gamma _N$ .

We now define the moduli functor

(2.2.1) $$ \begin{align} \mathsf F_{(r,\boldsymbol\gamma)}:{\mathsf{Sch}}_{{\mathbb C}}^{\mathrm{op}}\to{\mathsf{Sets}} \end{align} $$

by assigning to a ${\mathbb C}$ -scheme S the set

$$ \begin{align*}\mathsf F_{(r,\boldsymbol\gamma)}(S)=\{\mathrm{isomorphism\ classes\ of\ }(2N+2)-\mathrm{ tuples}\ (F_S,\varphi_S,Q^0_S,g^0_S,\dots,Q^{N-1}_S,g^{N-1}_S\}\end{align*} $$

with

• ○ $F_S$ a coherent sheaf over ${\mathbb P}^2\times S$ flat over S and such that $F_S|_{{\mathbb P}^2\times \{s\}}$ is a torsion-free sheaf for any closed point $s\in S$ , $\operatorname {\mathrm {rk}} F_S=r$ , $c_1(F_S)=0$ and $c_2(F_S)=\gamma _N$ ;

• ○ $\varphi _S:F_S|_{\ell _\infty \times S}\xrightarrow {\sim }\mathcal O^{\oplus r}_{\ell _\infty \times S}$ is an isomorphism of $\mathcal O_{\ell _\infty \times S}$ -modules;

• ○ $Q^i_S$ is a coherent sheaf on ${\mathbb P}^2\times S$ , flat over S and supported away from $\ell _\infty \times S$ , such that $h^0(Q_S^i|_{{\mathbb P}^2\times \{s\}})=\gamma _i+\cdots +\gamma _{N-1}$ , for any closed point $s\in S$ ;

• ○ $g^0_S:F_S\to Q^0_S$ and $g^i_S:Q^{i-1}_S\to Q^i_S$ , $i=1,\dots ,N-1$ are surjective morphisms of $\mathcal O_{{\mathbb P}^2\times S}$ -modules.

Two tuples $(F_S,\varphi _S,Q_S^1,g_S^1,\dots ,Q_S^N,g_S^N)$ and $(F_S^{\prime },\varphi _S^{\prime },Q_S^{1,\prime },g_S^{1,\prime },\dots ,Q_S^{{N-1},\prime },g_S^{N-1,\prime })$ are said to be isomorphic if there exist isomorphisms of $\mathcal O_{{\mathbb P}^2\times S}$ -modules $\Theta _S:F_S\to F_S^{\prime }$ and $\Gamma _S^i:Q_S^i\to Q_S^{i\prime }$ such that the following diagrams commute:

If this functor is representable, the variety representing it will be called the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ .

What we want to show next is that the moduli space of flags of torsion-free sheaves on ${\mathbb P}^2$ is a fine moduli space, and that it is indeed isomorphic (as a scheme) to the moduli space of nested instantons we defined previously. First of all, we will focus our attention on proving the following statement.

Proof. Our proof strongly relies on the use of (relative) nested ${\mathsf {Quot}}$ functors, so let us recall their construction and basic properties. First of all, let N be a fixed positive integer and take the universal framed sheaf $(\mathcal F,\psi )$ on ${\mathbb P}^2\times \mathcal M(r,\gamma _N)$ , with $\psi :\mathcal F|_{\ell _\infty \times \mathcal M(r,\gamma _N)}\xrightarrow {\sim }\mathcal O^{\oplus r}_{\ell _\infty \times \mathcal M(r,\gamma _N)}$ an isomorphism of $\mathcal O_{\ell _\infty \times \mathcal M(r,\gamma _N)}$ -modules. Let $\boldsymbol {\gamma }$ an N-tuple of integers $(\gamma _0,\gamma _1,\dots ,\gamma _{N-1})$ . We define the nested ${\mathsf {Quot}}$ functor

(2.2.2) $$ \begin{align} {\mathsf{Quot}}_{(\mathcal F,\boldsymbol\gamma)}:{\mathsf{Sch}}^{\mathrm{op}}_{\mathcal M(r,\gamma_N)}\to{\mathsf{Sets}} \end{align} $$

by

(2.2.3) $$ \begin{align} {\mathsf{Quot}}_{(\mathcal F,\boldsymbol\gamma)}(S)=\{\mathrm{isomorphism\ classes\ of\ }(Q_S^0,q_S^0,\dots,Q_S^{N-1},q_S^{N-1})\}, \end{align} $$

where

• ○ Each $Q^i_S$ is a coherent sheaf on ${\mathbb P}^2\times S$ , flat over S, supported away from $\ell _\infty \times S$ and such that $h^0(Q^i_S|_{\ell _\infty \times \{s\}})=\gamma _i+\cdots +\gamma _{N-1}$ , for any closed point $s\in S$ ;

• ○ $q_S^i:Q_S^{i-1}\to Q_S^i$ is a surjective morphism of $\mathcal O_{{\mathbb P}^2\times S}$ -modules for $i=1,\dots ,N-1$ ;

• ○ $q^0_S:\mathcal F_S\to Q^0_S$ is a surjective morphism of $\mathcal O_{{\mathbb P}^2\times S}$ -modules, where $\mathcal F_S$ is the pullback of $\mathcal F$ to ${\mathbb P}^2\times S$ via

  

By an induction on N along the lines of [Reference Huybrechts and Lehn31, § 2.A.1] one can prove the representability of the nested ${\mathsf {Quot}}$ functor. Note that when $N=1$ , the nested ${\mathsf {Quot}}$ functor reduces to an ordinary ${\mathsf {Quot}}$ functor, which is representable by Grothendieck’s theory. In this case, representability of $\mathsf F_{(r,\gamma _0,\gamma _1)}$ has been established in [Reference von Flach and Jardim54, Prop. 1]. In general, there is a natural forgetting map $\mathsf F_{(r,\boldsymbol \gamma )}\to {\mathsf {Quot}}_{(\mathcal F,\boldsymbol \gamma )}$ , which acts as $(F_S,\varphi _S,Q_S^0,g_S^0,\dots ,Q_S^{N-1},g_S^{N-1})\mapsto (Q_S^0,g_S^0,\dots ,Q_S^{N-1},g_S^{N-1})$ . This map also has an inverse given by setting $F_S$ to be the framed sheaf $F_S=\ker (g^0_S)$ with framing $\varphi _S$ at $\ell _\infty \times S$ induced by the framing $\psi $ of $\mathcal F$ at $\ell _\infty \times \mathcal M(r,\gamma _N)$ . Thus, the functors $\mathsf F_{(r,\boldsymbol \gamma )}$ and ${\mathsf {Quot}}_{(\mathcal F,\boldsymbol \gamma )}$ are naturally isomorphic, and representability of the latter implies representability of the former. The moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ , representing $\mathsf F_{(r,\boldsymbol \gamma )}$ , is then the nested quot scheme $\operatorname {\mathrm {Quot}}^{\boldsymbol \gamma }(\mathcal F)$ relative to $\mathcal M(r,\gamma _N)$ .

Remark 2.8. The previous description of the moduli space of framed flags of sheaves on ${\mathbb P}^2$ suggests we could also take a slightly different perspective on $\mathcal F(r,\boldsymbol \gamma )$ and study its closed subscheme of the moduli of sequences of quotients

$$\begin{align*}Z_0\hookrightarrow\cdots\hookrightarrow Z_N\hookrightarrow F\twoheadrightarrow Q_0\twoheadrightarrow\cdots\twoheadrightarrow Q_N, \end{align*}$$

where F is the trivial vector bundle $F\cong \mathcal O_{{\mathbb P}^2}^{\oplus r}$ . In this sense, $\mathcal F(r,\boldsymbol \gamma )$ seems to be analogous to the Filt-scheme studied by Mochizuki in [Reference Mochizuki38] in the case of curves.

Now that we proved that the definition of moduli space of framed flags of sheaves on ${\mathbb P}^2$ is indeed a good one, we are ready to tackle the problem of showing that there exists an isomorphism between this moduli space and the space of stable representation of the nested instantons quiver we studied in the previous sections. First of all, let us point out that our definition of flags of framed torsion-free sheaves reduces in the rank $1$ case to the nested Hilbert scheme of points on ${\mathbb C}^2$ , and the isomorphism we are interested in was shown to exist in Theorem 2.4 of §2.1. This is, in fact, compatible with the statement of Theorem 2.9.

Before diving into the proof of Theorem 2.9, we need to recall a few facts about the moduli space of framed torsion-free sheaves on ${\mathbb P}^2$ . Our main reference is [Reference Nakajima and Society43, § 2]. Recall then that with any (ordinary) ADHM datum $(W,V,B_1,B_2,I,J)$ is associated a complex of locally free sheaves on ${\mathbb P}^2$ , which we call the ADHM complex of X, of the form

(2.2.4)

where $[x:y:z]$ are homogeneous coordinates on ${\mathbb P}^2$ , $\ell _\infty =\{z=0\}$ is a line at infinity, and

As in [Reference Nakajima and Society43, Lemma 2.1], $\alpha $ is always injective if X satisfies the ADHM equation, while $\beta $ is surjective if and only if X is stable. The cohomology sheaf is a rank r torsion-free sheaf on ${\mathbb P}^2$ , with $c_2(E)=n$ , framed by the induced isomorphism $E|_{\ell _\infty }\xrightarrow {\sim } W\otimes \mathcal O_{\ell _\infty }$ . Conversely, given any framed torsion-free sheaf E on ${\mathbb P}^2$ , there is a stable ADHM datum X such that E is the cohomology of $E_X^{\bullet }$ . This realizes an isomorphism between the moduli space of framed torsion-free sheaves on ${\mathbb P}^2$ with rank r and second Chern class n, and the moduli space $\mathcal M(r,n)$ of stable representations of numerical type $(r,n)$ of the ADHM quiver; cf. [Reference Nakajima and Society43, § 2]. Moreover, if we let $\operatorname {\mathsf {Kom}}({\mathbb P}^2)$ be the category of complexes of sheaves on ${\mathbb P}^2$ and $\operatorname {\mathsf {Rep}}_{\text {ADHM}}$ be the category of representations of the ADHM quiver, we have that the functor

$$\begin{align*}\mathsf K:\operatorname{\mathsf{Rep}}_{\text{ADHM}}\to \operatorname{\mathsf{Kom}}({\mathbb P}^2) \end{align*}$$

defined by $\mathsf K(X)=E_X^{\bullet }$ , with the obvious definition for morphisms, is exact and fully faithful; cf. [Reference von Flach and Jardim54, Prop. 20].

Proof of Theorem 2.9.

We first want to show how, starting from an element of ${\mathcal N}(r,n_0,\dots ,n_N)$ , one can construct a flag of framed torsion-free sheaves on ${\mathbb P}^2$ . As we showed previously, to each $(V_i,B_1^i,B_2^i,F^i)$ in the datum of $X\in {\mathcal N}(r,n_0,\dots ,n_N)$ , we can associate a stable ADHM datum $(W,\widetilde V_i,\widetilde B_1^i,\widetilde B_2^i,\tilde I^i,\tilde J^i)$ , fitting in the diagram (2.2.5)

(2.2.5)

where we suppressed all of the endomorphisms $B_{1,2}^i$ , $\widetilde B_{1,2}^j$ . We will then call $\mathbf Z_i$ , $\mathbf S$ and ${\mathbf Q}_i$ the representations of the ADHM data $(\{0\},V_i,B_1^i,B_2^i)$ , $(W,V_0,B_1^0,B_2^0,I,J)$ and $(W,\widetilde V_i,\widetilde B_1^i,\widetilde B_2^i,\tilde I^i,\tilde J^i)$ , respectively. The the diagram (2.2.5) can be restated in the following form in $\operatorname {\mathsf {Rep}}_{\text {ADHM}}$ :

(2.2.6)

Moreover, if $E^{\bullet }_{\mathbf Z_i}$ , $E^{\bullet }_{\mathbf S}$ and $E^{\bullet }_{{\mathbf Q}_i}$ denotes the ADHM complex corresponding to $\mathbf Z_i$ , $\mathbf S$ and ${\mathbf Q}_i$ , the diagram (2.2.6) induces the following

(2.2.7)

Then, since $\mathbf S$ and ${\mathbf Q}_i$ are stable, by [Reference Nakajima and Society43, Lemma 2.6], one has that ${\mathcal H}^p(E^{\bullet }_{\mathbf S})={\mathcal H}^p(E^{\bullet }_{{\mathbf Q}_i})=0$ , for $p=-1,1$ . Moreover, one can also show that ${\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})=0$ . Indeed, we have

If we let $[x:y:0]=p\in \ell _\infty \subset {\mathbb P}^2$ , we have

and $\ker \beta _{i,p}=0$ , for all $p\in \ell _\infty $ . Thus, ${\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})$ is a zero-dimensional sheaf supported outside of $\ell _\infty $ . In particular, $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i}))=H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})(-1))$ , and the right-hand side vanishes. Indeed, consider the following short exact sequences of sheaves:

(2.2.8a)

(2.2.8b)

From Equation (2.2.8a), we get $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})(-1))\cong H^0(\ker \beta _i(-1))$ , while from Equation (2.2.8b), we get $H^0(\ker \beta _i(-1))=0$ . Thus, $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i}))=0$ , and for each line in (2.2.7), the long exact sequence for the cohomology associated to it reduces to

$$\begin{align*}0\longrightarrow {\mathcal H}^0(E^{\bullet}_{\mathbf S})\longrightarrow {\mathcal H}^0(E^{\bullet}_{{\mathbf Q}_i})\longrightarrow {\mathcal H}^1(E^{\bullet}_{\mathbf Z_i})\longrightarrow 0, \end{align*}$$

and by the ADHM construction, $({\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i}),\varphi )$ is a rank r framed torsion-free sheaf on ${\mathbb P}^2$ , with framing $\varphi :{\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})|_{\ell _\infty }\xrightarrow {\cong }W\otimes \mathcal O_{\ell _\infty }$ . Moreover, ${\mathcal H}^0(E^{\bullet }_{\mathbf S})$ is a subsheaf of ${\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})$ , and ${\mathcal H}^1(E^{\bullet }_{\mathbf Z_i})$ is a quotient sheaf

$$ \begin{align*}{\mathcal H}^1(E^{\bullet}_{\mathbf Z_i})\cong {\mathcal H}^0(E^{\bullet}_{{\mathbf Q}_i})/{\mathcal H}^0(E^{\bullet}_{\mathbf S}), \end{align*} $$

which is $0$ -dimensional and supported away from $\ell _\infty \subset {\mathbb P}^2$ . Finally, one can immediately see from (2.2.7) that ${\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})$ is a subsheaf of ${\mathcal H}^0(E^{\bullet }_{{{\mathbf Q}}_{i+1}})$ . One can moreover check that the numerical invariants classifying flags of sheaves do agree with the statement of the theorem.

Conversely, let $(E_0,\dots ,E_N,\varphi )$ be a flag of framed torsion-free sheaves on ${\mathbb P}^2$ such that $\operatorname {\mathrm {rk}} E_j=r$ , $c_2(E_N)=\gamma _N$ , $h^0(E_N/E_j)=\gamma _j+\cdots +\gamma _{N-1}$ . By definition, each $(E_j,\varphi )$ defines a stable ADHM datum ${\mathbf Q}_j=(\widetilde W_j,\widetilde V_j,\widetilde B_1^j,\widetilde B_2^j,\tilde I^j,\tilde J^j)$ (with the convention of calling $\mathbf S={\mathbf Q}_0$ ), since it can be identified with a framed torsion-free sheaf on ${\mathbb P}^2$ , with $\operatorname {\mathrm {rk}} E_j=r$ , $c_2(E_j)=\gamma _0+\cdots +\gamma _j$ . Moreover, we have the inclusion $E_0\hookrightarrow E_j$ , which induces an epimorphism $\Psi _j:\mathbf S\to {\mathbf Q}_j$ . In fact, we can construct vector spaces $V_0$ , $\widetilde V_j$ , W and $\widetilde W_j$ as in [Reference Nakajima and Society43], so that

$$\begin{align*}V_0\cong H^0(E_0(-1)),\quad \widetilde V_j\cong H^0(E_j(-1)),\quad W\cong H^0\left(E_0|_{\ell_\infty}\right),\quad \widetilde W_j\cong H^0\left(E_j|_{\ell_\infty}\right), \end{align*}$$

and by the fact that the quotient sheaf $E_j/E_0$ is $0$ -dimensional and supported away from $\ell _\infty $ , we can construct an isomorphism

$$\begin{align*}\Psi_{j,2}:H^0\left(E_0|_{\ell_\infty}\right)\xrightarrow{\cong}H^0\left(E_j|_{\ell_\infty}\right). \end{align*}$$

Finally, we have the exact sequence

$$ \begin{align*} 0\longrightarrow E_0\longrightarrow E_j\longrightarrow E_j/E_0\longrightarrow 0, \end{align*} $$

which induces the following exact sequence of cohomology, thanks to the fact that $H^0(E_j(-1))=0$ , being that $E_j$ is a framed torsion-free $\mu $ -semistable sheaf with $c_1(E_j)=0$ (due to the standard ADHM construction), and $H^1(E_j/E_0(-1))=0$ , since the quotient sheaf $E_j/E_0$ is $0$ -dimensional,

$$\begin{align*}0\longrightarrow H^0(E_j/E_0(-1))\longrightarrow H^1(E_0(-1))\xrightarrow{\Psi_{j,1}} H^1(E_j(-1))\longrightarrow 0. \end{align*}$$

The morphism $\Psi _j=(\Psi _{j,1},\Psi _{j,2})$ is then an epimorphism, since both $\Psi _{j,1}$ and $\Psi _{j,2}$ are surjective. Taking into account the flag structure of the datum $(E_0,\dots ,E_N,\varphi )$ , the sequences

give us $(N+1)$ stable ADHM data fitting in the following diagram:

Finally, we need to show that there exists a scheme-theoretic isomorphism between the moduli space of flags of sheaves $\mathcal F(r,\boldsymbol \gamma )$ and the nested ADHM moduli space ${\mathcal N}(r,{\mathbf n})$ . Generalizing the proof of [Reference von Flach and Jardim54, Thm. 18], one may first notice that the complex of sheaves on ${\mathbb P}^2$ in Equation (2.2.4) can be regarded as a family of complexes parametrized by $\mathcal M(r,n_0)$ whose cohomology, by [Reference Nakajima and Society43, § 2] and the relative version of Beilinson’s Theorem (cf. for instance, [Reference Bruzzo, Chuang, Diaconescu, Jardim, Pan and Zhang7, § 3.4]), yields a family of framed torsion-free sheaves on ${\mathbb P}^2$ parametrized by $\mathcal M(r,n_0)$ . Similarly, diagram (2.2.5) enables us to think of (2.2.7) as sequences of complexes parameterized by ${\mathcal N}(r,{\mathbf n})$ (i.e., complexes of sheaves on ${\mathbb P}^2\times {\mathcal N}(r,{\mathbf n})$ ). Thus, passing to cohomology, we get a family of framed torsion-free sheaves also parameterized by ${\mathcal N}(r,{\mathbf n})$ , and the isomorphism we were just describing may be regarded as an element of $\mathsf F_{(r,\boldsymbol \gamma )}({\mathcal N}(r,{\mathbf n}))$ , to which we may associate a unique bijective morphism of schemes ${\mathcal N}(r,{\mathbf n})\to \mathcal F(r,\boldsymbol \gamma )$ by the representability of $\mathsf F_{(r,\boldsymbol \gamma )}$ . Conversely, suppose we are given coherent sheaves $(F_S,Q^0_S,\dots ,Q^{N-1}_S)$ on ${\mathbb P}^2\times S$ , flat over S, defining a family of flags of framed torsion-free sheaves $(F_S,\varphi _S,Q^0_S,g^0_S,\dots ,Q^{N-1}_S,g^{N-1}_S)$ . We can associate to this family of sheaves a family of representations of the nested ADHM quiver parameterized by S (i.e., a morphism $S\to {\mathcal N}(r,{\mathbf n})$ ). Then, corresponding to the universal family, we get a morphism $\mathcal F(r,\boldsymbol \gamma )\to {\mathcal N}(r,{\mathbf n})$ .

3.1 Equivariant torus action and localization

Given an algebraic torus ${\mathbb T}=({\mathbb C}^{\ast })^n$ , any finite-dimensional ${\mathbb T}$ -representation V splits in a direct sum of its weights, which are one-dimensional ${\mathbb T}$ -representations. Each of the weights appearing in the decomposition of the ${\mathbb T}$ -representation V corresponds to a character $\mu \in \widehat {{\mathbb T}}=\operatorname {\mathrm {Hom}}({\mathbb T},{\mathbb C}^{\ast })\cong \mathbb Z^n$ , and thus to a monomial $t^\mu =t_1^{\mu _1}\cdots t_n^{\mu _n}$ in the coordinates $(t_1,\dots ,t_n)$ of ${\mathbb T}$ . We have then a map $\operatorname {\mathrm {tr}}:K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})\to \mathbb Z[t^\mu :\mu \in \widehat {{\mathbb T}}]$ on the representation ring $K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})\cong R({\mathbb T})$ , sending a ${\mathbb T}$ -representation V to its decomposition into weight spaces $\operatorname {\mathrm {tr}}_V=\sum _{\mu }t^\mu $ . Since this map is an isomorphism, in what follows, we will often identify (virtual) ${\mathbb T}$ -modules with their characters.

Consider the tori ${\mathbb T}_1=({\mathbb C}^{\ast })^2$ and ${\mathbb T}_2=({\mathbb C}^{\ast })^r$ . Let $T_1,T_2$ and $R_1,\dots ,R_r$ be the generators of the representation rings $R({\mathbb T}_1)\cong K_{{\mathbb T}_1}^0(\operatorname {\mathrm {pt}})$ and $R({\mathbb T}_2)\cong K_{{\mathbb T}_2}^0(\operatorname {\mathrm {pt}})$ , respectively. On ${\mathbb X}(r,{\mathbf n})$ , there is an action of the algebraic torus ${\mathbb T}={\mathbb T}_1\times {\mathbb T}_2$ . Indeed, given $X\in {\mathbb X}(r,{\mathbf n})$ , a torus element $\mathbf t=(T_1,T_2,R)$ , $R\in ({\mathbb C}^{\ast })^r\subset \operatorname {\mathrm {GL}}_r$ , acts on X as

$$\begin{align*}\mathbf t\cdot X=(T_1B_1^0,T_2B_2^0,IR^{-1},RT_1T_2J,\dots,T_1B_1^N,T_2B_2^N,F^N). \end{align*}$$

The ${\mathbb T}$ -action we just described commutes with the $\mathcal G$ -action of §1.2, so it descends to an action on the GIT quotient ${\mathcal N}(r,{\mathbf n})$ . Similarly, the vector bundle ${\mathcal E}$ in Theorem 1.8 is naturally equivariant; thus, the perfect obstruction theory $\mathbb E\xrightarrow {\phi }{\mathcal N}(r,{\mathbf n})$ lifts to the derived category $\mathbf D^{[-1,0]}_{{\mathbb T}}({\mathcal N}(r,{\mathbf n}))$ of ${\mathbb T}$ -equivariant coherent sheaves on ${\mathcal N}(r,{\mathbf n})$ . Indeed, one can introduce a natural ${\mathbb T}$ -equivariant structure on the vector bundles $C^i$ and the bundle homomorphism $d^i:C^i\to C^{i+1}$ in such a way that the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ is ${\mathbb T}$ -equivariant. These then restrict to the stable locus and descend to ${\mathbb T}$ -equivariant vector bundles $\mathcal C^i$ and bundle maps $\delta ^i:\mathcal C^i\to \mathcal C^{i+1}$ over $\mathcal Y(r,{\mathbf n})$ , together with a ${\mathbb T}$ -equivariant section $\sigma \in H^0(\mathcal Y(r,{\mathbf n}),\mathcal C^2)$ .

In the following, we will denote by $Q=T_1\cdot {\mathbb C}\oplus T_2\cdot {\mathbb C}\in K_0^{{\mathbb T}_1}(\operatorname {\mathrm {pt}})$ the ${\mathbb T}_1$ -representation corresponding to ${\mathbb C}^2$ , and by $\Lambda ^2Q=T_1T_2\cdot {\mathbb C}\in K_0^{{\mathbb T}_1}(\operatorname {\mathrm {pt}})$ its top exterior power.

We begin the analysis of the fixed locus under the ${\mathbb T}$ -action on the moduli space of nested instantons with a brief recall of the results obtained in [Reference von Flach and Jardim54] and show how they enable us to fully characterize the ${\mathbb T}$ -fixed locus of the two-step nested instantons quiver. The main result we want to recall is the following theorem.

Theorem 3.1 (von Flach-Jardim, [Reference von Flach and Jardim54]).

The moduli space ${\mathcal N}(r,n_0,n_1)\cong \mathcal F(r,n_0-n_1,n_1)$ of stable representations of the nested ADHM quiver is a quasi-projective variety equipped with a perfect obstruction theory. Its ${\mathbb T}$ -equivariant deformation-obstruction complex is the following:

with

$$ \begin{align*} \left\{ \begin{aligned} &d_0(h_0,h_1)=\left([h_0,B_1^0],[h_0,B_2^0],h_0I,-Jh_0,[h_1,B_1^1],[h_1,B_2^1],h_0F-Fh_1\right), \\ &d_1(b_1^0,b_2^0,i,j,b_1^1,b_2^1,f)=\left([b_1^0,B_2^1]+[B_1^0,b_2^0]+iJ+Ij,B_1^0f+b_1^0F-Fb_1^1-fB_1^1,\right. \\ &\qquad\qquad\qquad\qquad\qquad\quad \left.B_2^0f+b_2^0F-Fb_2^1-fB_2^1,jF+Jf,[b_1^1,B_2^1]+[B_1^1,b_2^1]\right),\\ &d_2(c_1,c_2,c_3,c_4,c_5)=c_1F+B_2^0c_2-c_2B_2^1+c_3B_1^0-B_1^1c_3-Ic_4-Fc_5. \end{aligned}\right. \end{align*} $$

Thus, the infinitesimal deformation space and the obstruction space at any X will be isomorphic to $H^1[\mathcal C(X)]$ and $H^2[\mathcal C(X)]$ , respectively. ${\mathcal N}(r,n_1,n_2)$ is smooth iff $n_1=1$ ([Reference Cheah10]).

Moreover, it turns out that there exists a surjective morphism $\mathfrak q:(B_1^0,B_2^0,I,J,B_1^1,B_2^1,F)\mapsto (B_1^{\prime },B_2^{\prime },I',J')$ mapping the nested ADHM data of type $(r,n_0,n_1)$ to the ADHM data of numerical type $(r,n_0-n_1)$ ; [Reference von Flach and Jardim54]. Thus, we have two different maps sending the moduli space of stable representations of the nested ADHM quiver to the moduli space of stable representations of ADHM data. Moreover, from Theorem 2.9, one deduces that ${\mathcal N}(r,n_0,n_1)$ can be identified as the incidence variety

$$\begin{align*}{\mathcal N}(r,n_0,n_1)\hookrightarrow\mathcal M(r,n_0)\times\mathcal M(r,n_0-n_1), \end{align*}$$

by means of which one can characterize ${\mathbb T}$ -fixed points of ${\mathcal N}(r,n_0,n_1)$ in terms of the fixed points of $\mathcal M(r,n_0)$ and $\mathcal M(r,n_0-n_1)$ . For the sake of simplicity, let us first focus on the $r=1$ case. We can first take the decomposition $V_0=V\oplus V_1$ , then decompose the vector spaces $V_0$ , V with respect to the action of ${\mathbb T}$ : if $\lambda _0:{\mathbb T}\to \operatorname {\mathrm {End}}(V_0)$ and $\lambda :{\mathbb T}\to \operatorname {\mathrm {End}}(V)$ are morphisms for the toric action on $V_0$ , V, we have

$$\begin{align*}\left\{ \begin{aligned} V & = \bigoplus_{k,l} V(k,l) = \bigoplus_{k,l}\left\{v\in V|\lambda(t)v=t_1^kt_2^lv\right\}\\ V_0 & = \bigoplus_{k,l} V_0(k,l) = \bigoplus_{k,l}\left\{v_0\in V_0|\lambda_0(t)v_0=t_1^kt_2^lv_0\right\}. \end{aligned} \right. \end{align*}$$

Thus, if $X=(W,V,B_1^{\prime },B_2^{\prime },I',J')$ , $X_0=(W,V_0,B_1^0,B_2^1,I,J)$ are ${\mathbb T}$ -fixed points, the very well-known results about the classification of fixed points for ADHM data leads us to the following commutative diagram:

(3.1.1)

The previous propositions give us an easy way of visualizing fixed points of the ${\mathbb T}$ -action on the nested ADHM data. If we suitably normalize each nonzero map to $1$ by the action of $\prod _{k,l}\operatorname {\mathrm {GL}}(V_0(k,l))\times \prod _{k',l'}\operatorname {\mathrm {GL}}(V(k',l'))$ each critical point point can be put into one-to-one correspondence with nested Young diagrams $Y_\mu \subseteq Y_\nu $ . Thus, the fixed points of the original nested ADHM data are classified by couples $(\nu ,\nu \setminus \mu )$ , where $\mu \subset \nu $ and $\nu \setminus \mu $ is the skew Young diagram constructed by taking the complement of $\mu $ in $\nu $ .

If we now take a fixed point $Z=(\nu ,\mu )$ and define $\nu _i=\sum _k\dim V_0(k,1-i)$ , $\nu ^{\prime }_j=\sum _l\dim V_0(1-j,l)$ and similarly $\mu _i=\sum _k\dim V(k,1-i)$ , $\mu ^{\prime }_j=\sum _l\dim V(1-j,l)$ , we can regard $V_0$ and V as ${\mathbb T}$ -modules and write them as

$$\begin{align*}\left\{ \begin{aligned} &V_0=\bigoplus_{k,l}V_0(k,l)=\sum_{i=1}^{M_1}\sum_{j=1}^{\nu^{\prime}_i}T_1^{-i+1}T_2^{-j+1}= \sum_{j=1}^{N_1}\sum_{j=1}^{\nu_j}T_1^{-i+1}T_2^{-j+1}\\ &V=\bigoplus_{k,l}V(k,l)=\sum_{i=1}^{M_2}\sum_{j=1}^{\mu^{\prime}_i}T_1^{-i+1}T_2^{-j+1}= \sum_{j=1}^{N_2}\sum_{j=1}^{\mu_j}T_1^{-i+1}T_2^{-j+1}\\ \end{aligned} \right. \end{align*}$$

with $M_1=\nu _1$ , $M_2=\mu _1$ , $N_1=\nu ^{\prime }_1$ , $N_2=\mu ^{\prime }_1$ . If we now take $V_0=V\oplus V_1$ , we have

$$\begin{align*}V_1=\sum_{(i,j)\in\nu\setminus\mu}T_1^{-i+1}T_2^{-j+1}= \sum_{i=1}^{M_1}\sum_{j=1}^{\nu^{\prime}_i-\mu^{\prime}_i}T_1^{-i+1}T_2^{-\mu_i^{\prime}-j+1}. \end{align*}$$

The virtual tangent space $T^{\textrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z\in K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})$ to ${\mathcal N}(1,n_0,n_1)$ at the ${\mathbb T}$ -fixed point Z can be regarded as a ${\mathbb T}$ -module, so that

$$\begin{align*}\begin{aligned} T_Z^{\textrm{vir}}{\mathcal N}(1,n_0,n_1)&=\operatorname{\mathrm{End}}(V_0)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{End}}(V_1)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(W,V_0)\\ &\quad+\operatorname{\mathrm{Hom}}(V_0,W)\otimes\Lambda^2Q-\operatorname{\mathrm{Hom}}(V_1,W)\otimes\Lambda^2Q+\operatorname{\mathrm{Hom}}(V_1,V_0)(1+\Lambda^2Q-Q)\\ &=(V_1\otimes V_0^{\ast}+V_1\otimes V_1^{\ast}-V_1^{\ast}\otimes V_0)\otimes(Q-1-\Lambda^2Q)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad-V_1^{\ast}\otimes\Lambda^2Q. \end{aligned} \end{align*}$$

In the first place, we might recognize the term $V_0^{\ast }\otimes V_0\otimes (Q-\Lambda ^2Q-1)+V_0+V_0^{\ast }\otimes \Lambda ^2Q$ in the sum as being the tangent space at the moduli space of stable representation of the ADHM quiver $T_{\mathcal M(1,n_0)}|_{\tilde Z}$ , with $\tilde Z=(\nu )$ . Thus, we have

(3.1.2) $$ \begin{align} \begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z&=T_{\mathcal M(1,n_0)}|_{\tilde Z}+(V_1\otimes V_1^{\ast}-V_1^{\ast}\otimes V_0)\otimes(Q-1-\Lambda^2Q)-V_1^{\ast}\otimes\Lambda^2Q. \end{aligned} \end{align} $$

Then

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes(Q-1-\Lambda^2Q)&=(T_1-1)(1-T_2)\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^{i-1}T_2^{\mu_i^{\prime}+j-1}\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\sum_{j=1}^{\nu^{\prime}_i-\mu^{\prime}_i}T_2^{j}\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\left(\frac{1-T_2^{\nu_i^{\prime}-\mu^{\prime}_i+1}}{1-T_2}-1\right)\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\left(\frac{T_2-T_2^{\nu_i^{\prime}-\mu^{\prime}_i+1}}{1-T_2}\right)\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}}), \end{aligned} \end{align*}$$

so that

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes V_1\otimes(Q-1-\Lambda^2Q)&=(T_1-1)\sum_{j=1}^{N_1}\sum_{j'=1}^{\nu_{j}-\mu_{j}}T_1^{-\mu_j-j'+1}T_2^{-j+1}\sum_{i=1}^{M_1}T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}})\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1} T_1^{i-\mu_j}(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\sum_{j'=1}^{\nu_j-\mu_j}T_1^{-j'}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^{i-\mu_j}(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\left(\frac{1-T_1^{-\nu_j+\mu_j-1}}{1-T_1^{-1}}-1\right)\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^{i-\mu_j}-T_1^{i-\nu_j})(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1}), \end{aligned} \end{align*}$$

while we have

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes V_0\otimes(Q-1-\Lambda^2Q)&=(T_1-1)\sum_{j=1}^{N_1}\sum_{j'=1}^{\nu_{j}}T_1^{-j'+1}T_2^{-j+1}\sum_{i=1}^{M_1} T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}})\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^i(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\sum_{j'=1}^{\nu_j}T_1^{-j'}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^i(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\left(\frac{1-T_1^{-\nu_j-1}}{1-T_1^{-1}}-1\right)\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^i-T_1^{i-\nu_j})(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1}), \end{aligned} \end{align*}$$

and

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes\Lambda^2Q&=T_1T_2\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^{i-1}T_2^{\mu_i^{\prime}+j-1}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^iT_2^{\mu_i^{\prime}+j}. \end{aligned} \end{align*}$$

Assembling everything together, we finally get that

(3.1.3) $$ \begin{align} \begin{aligned} T_{{\mathcal N}(1,n_0,n_1)}^{\textrm{vir}}|_Z=&T_{\mathcal M(1,n_0)}|_{\tilde Z}+\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^{i-\mu_j}-T_1^{i})(T_2^{-j+\mu_i^{\prime}+1}-T_2^{-j+\nu_i^{\prime}+1})\\ &-\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^iT_2^{j+\mu_i^{\prime}}. \end{aligned} \end{align} $$

As an immediate generalization of (3.1.3), we can easily see that the ( ${\mathbb T}$ -equivariant) K-theory class of the virtual tangent space to ${\mathcal N}(r,n_0,n_1)$ at a fixed point Z is

$$\begin{align*}\begin{aligned} T_{{\mathcal N}(r,n_0,n_1)}^{\textrm{vir}}|_Z &=T_{\mathcal M(r,n_0)}|_{\tilde Z}+\sum_{a,b=1}^r\sum_{i=1}^{M_1^{(a)}}\sum_{j=1}^{N_1^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_j^{(b)}}-T_1^{i}\right)\\ & \quad \times\left(T_2^{-j+\mu_i^{(a)'}+1}-T_2^{-j+\nu_i^{(a)'}+1}\right)-\sum_{i=1}^{M_1^{(a)}}\sum_{j=1}^{\nu_i^{(a)'}-\mu_i^{(a)'}}T_1^iT_2^{j+\mu_i^{(a)'}}. \end{aligned} \end{align*}$$

Remark 3.3. It turns out that the character representation of the virtual tangent $T^{\mathrm {vir}}_{{\mathcal N}}|_Z$ can be computed by exploiting deformation theory techniques. These techniques may also be employed to compute the virtual fundamental class and ( ${\mathbb T}_1$ -character of) the virtual tangent bundle at fixed points of nested Hilbert schemes on surfaces, as it is done in [Reference Gholampour, Sheshmani and Yau19].

If, in particular, one takes $({\mathbb C}^2)^{[N_0\ge N_1]}$ to be the nested Hilbert scheme of points on ${\mathbb C}^2=\operatorname {\mathrm {Spec}}({\mathbb C}[x_0,x_1])$ , by lifting the natural torus action on ${\mathbb C}^2$ to $\left({\mathbb C}^2\right)^{[N_0\ge N_1]}$ , it is proved in [Reference Gholampour, Sheshmani and Yau19] that the ${\mathbb T}_1$ -fixed locus is isolated and given by the inclusion of monomial ideals $I_0\subseteq I_1$ , which is equivalent to the assignment of couples of nested partitions $\mu \subseteq \nu $ . Then the virtual tangent space at a fixed point is given by

$$ \begin{align*} T_{I_0\subseteq I_1}^{\mathrm{vir}}=-\chi(I_0,I_0)-\chi(I_1,I_1)+\chi(I_0,I_1)+\chi(R,R), \end{align*} $$

with $\chi (-,-)=\sum _{i=0}^2(-1)^i\operatorname {\mathrm {Ext}}_R^i(-,-)$ . Then the ${\mathbb T}_1$ -representation of $T^{\mathrm { vir}}_{I_0\subseteq I_1}$ can be explicitly written in terms of Laurent polynomials in the torus characters $t_1,t_2$ of ${\mathbb T}_1$ . Then in terms of the characters $\mathsf Z_0,\mathsf Z_1$ of the ${\mathbb T}_1$ -fixed $0$ -dimensional subschemes $Z_1\subseteq Z_0\subset {\mathbb C}^2$ corresponding to $I_0\subseteq I_1$ , one has (cf. [Reference Gholampour, Sheshmani and Yau19, Eq. (29)])

$$ \begin{align*} \operatorname{\mathrm{tr}} T^{\mathrm{vir}}_{I_0\subseteq I_1}=\mathsf Z_0+\frac{\overline{\mathsf Z}_1}{t_1t_2}+\left(\overline{\mathsf Z}_0\mathsf Z_1-\overline{\mathsf Z}_0\mathsf Z_0-\overline{\mathsf Z}_1\mathsf Z_1\right)\frac{(1-t_1)(1-t_2)}{t_1t_2}. \end{align*} $$

If we now make the necessary identifications $t_i=T_i^{-1}$ , $\mathsf Z_0=V_0$ and $\mathsf Z_1=V$ , we can see that Equation (29) of [Reference Gholampour, Sheshmani and Yau19] exactly agrees with our prescription for the character representation (3.1.2) of the virtual tangent space $T^{\mathrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z$ , with $n_0=N_0$ and $n_1=N_0-N_1$ .Footnote ⁵

We now move on studying the fixed locus of the more general nested instantons moduli space ${\mathcal N}(r,n_0,\dots ,n_N)$ . However, similarly to the previous case, we first want to show that the moduli space of stable representations of the nested ADHM quiver is equivalently described by the datum of $(N+1)$ moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ – namely, $\mathcal M(r,n_0),\mathcal M(r,n_0-n_1),\dots ,\mathcal M(r,n_0-n_N)$ . To do this, we want to know if it is possible to recover the structure of the nested ADHM quiver given a set of stable ADHM data. First of all, we can notice that as $F^i$ is injective $\forall i$ , we have the sum decomposition $V_0=V_i\oplus \tilde V_i$ , where suitable choices of bases of $V_i$ are made so that Equation (1.2.10) holds. We also have, with analogous choices being made, $V_i=V_{i+1}\oplus \hat V_{i+1}$ , with $\hat {V}_{i+1}=V_i/\operatorname {\mathrm {Im}} F_i$ , so that $V_0=V_i\oplus \hat V_i\oplus \tilde V_{i-1}$ , thus $\tilde V_i=\hat V_i\oplus \tilde V_{i-1}$ .

Let us first focus on the vector spaces $V_0$ and $V_1$ . It can be shown as in [Reference Bruzzo, Chuang, Diaconescu, Jardim, Pan and Zhang7, Reference von Flach and Jardim54] that once we fix a stable ADHM datum $(W,\tilde V_1,\tilde B_1^1,\tilde B_2^1,\tilde I^1,\tilde J^1)$ and the endomorphisms $B_1^1,B_2^1\in \operatorname {\mathrm {End}} V_1$ , it is always possible to reconstruct the stable ADHM datum $(W,V_0,B_1^0,B_2^0,I,J)$ as

(3.1.4) $$ \begin{align} B_1^0=\begin{pmatrix} B_1^1 & B_1^{'1}\\ 0 & \tilde B_1^1 \end{pmatrix},\qquad B_2^0=\begin{pmatrix} B_2^1 & B_2^{'1}\\ 0 & \tilde B_2^1 \end{pmatrix},\qquad I=\begin{pmatrix} I^{'1}\\ \tilde I^1 \end{pmatrix},\qquad J=\begin{pmatrix} 0 & \tilde J^1 \end{pmatrix} \end{align} $$

together with the morphism such that $[B_1^1,B_2^1]=0$ , $B_1^0F^1-F^1B_1^1=B_2^0F^1-F^1B_2^1=0$ and $JF^1=0$ . The same can obviously be done for any of the stable ADHM data $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ we constructed previously, and we would have

(3.1.5) $$ \begin{align} B_1^0=\begin{pmatrix} B_1^i & B_1^{'i}\\ 0 & \tilde B_1^i \end{pmatrix},\qquad B_2^0=\begin{pmatrix} B_2^i & B_2^{'i}\\ 0 & \tilde B_2^i \end{pmatrix},\qquad I=\begin{pmatrix} I^{'i}\\ \tilde I^i \end{pmatrix},\qquad J=\begin{pmatrix} 0 & \tilde J^i \end{pmatrix} \end{align} $$

together with the morphism such that $[B_1^i,B_2^i]=0$ , $B_1^0f^i-f^iA_i=B_2^0f^i-f^iB_2^i=0$ and $Jf^i=0$ . If we now fix

(3.1.6)

which is clearly injective, then obviously $f^i=F^1F^2\cdots F^i$ , where $F^j$ now stands for the linear extension to $V_0$ , and $B_1^0f^i-f^iB_1^i=0$ (resp. $B_2^0f^i-f^iB_2^i=0$ ) is equivalent to $B_1^0F^1F^2\cdots F^{i-1}F^i-F^1F^2\cdots F^iB_1^i=B_1^{i-1}F^i-F^iB_1^i=0$ (resp. $B_2^{i-1}F^i-F^iB_2^i=0$ ), and $Jf^i=JF^1F^2\cdots F^i=0$ . This construction makes it possible to us to classify the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,n_0,\dots ,n_N)$ in terms of the ${\mathbb T}$ -fixed loci of $\mathcal M(r,n_0)$ and $\{\mathcal M(r,n_0-n_i)\}_{i>0}$ . In particular, the ${\mathbb T}$ -fixed locus of $\mathcal M(r,k)$ is into $1-1$ correspondence with coloured partitions $\boldsymbol \mu =(\mu ^1,\dots ,\mu ^r)\in \mathcal P^r$ such that $|\boldsymbol \mu |=|\mu ^1|+\cdots +|\mu ^r|=k$ . This fact and the inclusion relations between the vector spaces $V_i$ prove the following.

As we pointed out in Theorem 1.8, we can read the ( ${\mathbb T}$ -equivariant) K-theory class of the virtual tangent space to ${\mathcal N}(r,{\mathbf n})$ at a fixed point $Z\in {\mathcal N}(r,{\mathbf n})^{{\mathbb T}}$ off the following equivariant complex:

giving us (3.1.7).

(3.1.7) $$ \begin{align} \begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(1,{\mathbf n})}|_{Z}&=\operatorname{\mathrm{End}}(V_0)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(W,V_0)+\operatorname{\mathrm{Hom}}(V_0,W)\otimes\Lambda^2Q\\ &\quad+\operatorname{\mathrm{End}}(V_1)\otimes(Q-1-\Lambda^2Q)-\operatorname{\mathrm{Hom}}(V_1,W)\otimes\Lambda^2Q\\ &\quad+\operatorname{\mathrm{Hom}}(V_1,V_0)\otimes(1+\Lambda^2Q-Q)\\ &\quad+\operatorname{\mathrm{End}}(V_2)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(V_2,V_1)\otimes(1+\Lambda^2Q-Q)+\\ &\quad\cdots\\ &\quad+\operatorname{\mathrm{End}}(V_N)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(V_N,V_{N-1})\otimes(1+\Lambda^2Q-Q) \end{aligned} \end{align} $$

By decomposing the vector spaces $V_i$ in terms of the ${\mathbb T}$ -characters, we can also rewrite the representation of (3.1.7) in $R({\mathbb T})$ as

$$\begin{align*}\begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(r,{\mathbf n})}|_Z & = T_{\mathcal M(r,n_0)}|_{\tilde Z}+\sum_{a,b=1}^r\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{N_0^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_{1,j}^{(b)}}-T_1^{i}\right)\left(T_2^{-j+\mu_{1,i}^{(a)'}+1}-T_2^{-j+\mu_{0,i}^{(a)'}+1}\right)\\ &\quad -\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{\mu_{0,i}^{(a)'}-\mu_{1,i}^{(a)'}}T_1^iT_2^{j+\mu_{1,i}^{(a)'}}\\ &\quad +\sum_{k=2}^{N}\left[\sum_{a,b=1}^r\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{N_0^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_{k,j}^{(b)}}-T_1^{i-\mu_{k-1,j}^{(b)}}\right)\left(T_2^{-j+\mu_{k,i}^{(a)'}+1}-T_2^{-j+\mu_{0,i}^{(a)'}+1}\right)\right], \end{aligned} \end{align*}$$

where the fixed point Z is to be identified with a choice of a sequence of coloured nested partitions $\boldsymbol \mu _1\subseteq \boldsymbol \mu _{2}\subseteq \cdots \subseteq \boldsymbol \mu _{N}\subseteq \boldsymbol \mu _0$ as in Proposition 3.4, $\tilde Z\in \mathcal M(r,n_0)^{{\mathbb T}}$ is the ${\mathbb T}$ -fixed point corresponding to the coloured partition $\boldsymbol \mu _0$ .

Proof. Assuming the generators $R_a$ , $a=1,\dots ,r$ of the the representation ring $R({\mathbb T}_2)$ to be sufficiently generic, we only need to show that $T^{\mathrm {vir}}_{{\mathcal N}(1,{\mathbf n})}|_Z$ is ${\mathbb T}$ -movable. It is moreover sufficient to prove the claim in the case of flags of length $2$ , and the proof immediately generalizes. If $n_0=n_1$ , there is nothing to show, so let $n_0>n_1$ and $Z\in {\mathcal N}(1,n_0,n_1)^{{\mathbb T}}$ a fixed point associated with the nested partitions $\mu _0\supset \mu _1$ . The K-theory class of the virtual tangent space at Z is

$$ \begin{align*} T^{\mathrm{vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z&=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad+\left(V_1\otimes V_1^{\ast}-V_0\otimes V_1^{\ast}\right)\otimes\left(Q-1-\Lambda^2Q\right)-V_1^{\ast}\otimes\Lambda^2Q\\ &=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad-V\otimes V_1^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)-V_1^{\ast}\otimes\Lambda^2Q, \end{align*} $$

where

$$\begin{align*}V_0=\sum_{(i,j)\in Y_{\mu_0}}T_1^{-i+1}T_2^{-j+1},\quad V_1=\sum_{(i,j)\in Y_{\mu_0\setminus\mu_1}}T_1^{-i+1}T_2^{-j+1},\quad V=\sum_{(i,j)\in Y_{\mu_1}}T_1^{-i+1}T_2^{-j+1}, \end{align*}$$

and $Y_{\mu _i}$ denotes the Young diagram associated with $\mu _i$ . By construction, if $\mu _0\supset \mu _1$ , one has $i'>i$ or $j'>j$ , or both, for all $(i,j)\in Y_{\mu _1}$ , $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . We also have that

$$\begin{align*}T_{\mathcal M(1,n_0)}|_{Z_0}=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q \end{align*}$$

contains no constant term, being the tangent space to $\mathcal M(1,n_0)$ at the fixed point $Z_0$ associated with the partition $\mu _0$ . Similarly, $V_1^{\ast }\otimes \Lambda ^2Q$ is manifestly ${\mathbb T}$ -movable. Consider then the remaining term $V\otimes V_1^{\ast }\otimes \left(Q-1-\Lambda ^2Q\right)$ in $T^{\mathrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z$ . The contribution corresponding to $V\otimes V_1^{\ast }$ consists of a sum of monomials of the form $T_1^{-i+i'}T_2^{-j+j'}$ , where $(i,j)\in Y_{\mu _1}$ and $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . The only possibility for a constant term to arise is if $i=i'$ , $j=j'$ , for some $(i,j)\in Y_{\mu _1}$ and $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . This is, however, not possible if $\mu _0\supset \mu _1$ . In a completely analogous way, one can show that no constant term can arise from either $V\otimes V_1^{\ast }\otimes Q$ or $V\otimes V_1^{\ast }\otimes \Lambda ^2Q$ .

3.2 Virtual equivariant holomorphic Euler characteristic

The first virtual invariant we are going to study is the holomorphic virtual equivariant Euler characteristic of the moduli space of nested instantons. The fact that we can decompose the virtual tangent bundle as a direct sum of equivariant line bundles under the torus action we previously described greatly simplifies the computations.

In particular, given a scheme X with a $1$ -perfect obstruction theory $\mathbb E$ , one can define a virtual structure sheaf $\mathcal O_X^{\mathrm {vir}}$ . Moreover, one can choose an explicit resolution of $\mathbb E$ as $[E^{-1}\to E^0]$ a complex of vector bundles. If $[E_0\to E_1]$ denotes the dual complex, then one can also define the virtual tangent bundle $T_X^{\mathrm {vir}}\in K^0(X)$ as the class $T_X^{\mathrm {vir}}=[E_0]-[E_1]$ . With these definitions, the virtual Todd genus of X is defined as $\operatorname {\mathrm {td}}^{\mathrm { vir}}(X)=\operatorname {\mathrm {td}}(T_X^{\mathrm {vir}})$ , and if X is proper, given any $V\in K^0(X)$ , one defines the virtual holomorphic Euler characteristic as

$$\begin{align*}\chi^{\mathrm{vir}}(X,V)=\chi(X,V\otimes O^{\mathrm{vir}}_X), \end{align*}$$

and as a consequence of the virtual Riemann-Roch theorem [Reference Fantechi and Göttsche17], if X is proper and $V\in K^0(X)$ , the virtual holomorphic Euler characteristic admits an equivalent definition as

(3.2.1) $$ \begin{align} \chi^{\mathrm{vir}}(X,V)=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{ch}}(V)\cdot\operatorname{\mathrm{td}}(T_X^{\mathrm{vir}}), \end{align} $$

where $[X]^{\mathrm {vir}}$ is the virtual fundamental class of X, $[X]^{\mathrm {vir}}\in A_{\operatorname {\mathrm {vd}}}(X)$ and $\operatorname {\mathrm {vd}}$ denotes the virtual dimension of X, $\operatorname {\mathrm {vd}}=\operatorname {\mathrm {rk}} E_0-\operatorname {\mathrm {rk}} E_1$ . Clearly, if we are interested in $\chi ^{\mathrm { vir}}(X)$ , then the previous formula reduces to

(3.2.2) $$ \begin{align} \chi^{\mathrm{vir}}(X)=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{td}}(T_X^{\mathrm{vir}}) \end{align} $$

whenever X is proper.

Equations (3.2.1) and (3.2.2) can be made even more explicit. In fact, if we take $n=\operatorname {\mathrm {rk}} E_0$ , $m=\operatorname {\mathrm {rk}} E_1$ , so that $\operatorname {\mathrm {vd}}=n-m$ , and define $x_1,\dots ,x_n$ and $u_1,\dots ,u_m$ to be respectively the Chern roots of $E_0$ and $E_1$ , then (3.2.2) becomes

$$\begin{align*}\chi^{\mathrm{vir}}(X)=\int_{[X]^{\mathrm{vir}}}\prod_{i=1}^n\frac{x_i}{1-{\text e}^{-x_1}}\prod_{j=1}^m\frac{1-{\text e}^{-u_j}}{u_j}, \end{align*}$$

while for (3.2.1), we have

$$\begin{align*}\chi^{\mathrm{vir}}(X,V)=\int_{[X]^{\mathrm{vir}}}\left(\sum_{k=1}^r{\text e}^{v_k}\right)\prod_{i=1}^n\frac{x_i}{1-{\text e}^{-x_1}}\prod_{j=1}^m\frac{1-{\text e}^{-u_j}}{u_j} \end{align*}$$

since we can consider $V\in K^0(X)$ to be a vector bundle on X with Chern roots $v_1,\dots ,v_r$ .

Now, if we have a proper scheme X equipped with an action of a torus $({\mathbb C}^{*})^N$ and an equivariant $1$ -perfect obstruction theory, we can apply virtual equivariant localization in order to compute virtual invariants of X. We will now briefly recall how virtual localization works. First of all, for any equivariant vector bundle B over a proper scheme Z with a $1$ -perfect obstruction theory, which is moreover equipped with a trivial action of $({\mathbb C}^{*})^N$ , we have the decomposition

$$\begin{align*}B=\bigoplus_{{\mathbf k}\in\mathbb Z^N}B^{\mathbf k}, \end{align*}$$

where $B^{\mathbf k}$ denotes the $({\mathbb C}^{*})^N$ -eigenbundles on which the torus acts by $t_1^{k_1}\cdots t_N^{k_N}$ . If we now give a set of variables $\varepsilon _1,\dots ,\varepsilon _N$ , we identify B with $B=\sum _{{\mathbf k}}B^{\mathbf k}{\text e}^{k_1\varepsilon _1}\cdots {\text e}^{k_N\varepsilon _N}\in K^0(Z)[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ . One then defines $B^{\mathrm {fix}}=B^{{\mathbf 0}}$ and $B^{\mathrm { mov}}=\oplus _{{\mathbf k}\neq {\mathbf 0}}B^{\mathbf k}$ . Then the Chern character $\operatorname {\mathrm {ch}}:K^0(Z)\to A^{*}(Z)$ can be extended by $\mathbb Q(\!(\varepsilon _1,\dots ,\varepsilon _N)\!)$ -linearity to

$$ \begin{align*}\operatorname{\mathrm{ch}}:K^0(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!)\to A^{*}(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!).\end{align*} $$

Since the Grothendieck group of equivariant vector bundles $K^0_{({\mathbb C}^{*})^N}(Z)$ is a subring of $K^0(Z)[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ , the restriction of the extension of $\operatorname {\mathrm {ch}}$ to $K^0_{({\mathbb C}^{*})^N}(Z)$ is naturally identified with the equivariant Chern character. Finally, if one denotes by $p_*^{\mathrm {vir}}$ the $\mathbb Q(\!(\varepsilon _1,\dots ,\varepsilon _N)\!)$ -linear extension of $\chi ^{\mathrm {vir}}(Z,-):K^0(Z)\to \mathbb Z$ , and $p_*$ is the equivariant pushforward to a point, one can prove as in [Reference Fantechi and Göttsche17] that

$$\begin{align*}p_*^{\mathrm{vir}}(V)=p_*\left(\operatorname{\mathrm{ch}}(V)\operatorname{\mathrm{td}}(T_Z^{\mathrm{vir}})\cap[Z]^{\mathrm{vir}}\right),\qquad V\in K^0(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!). \end{align*}$$

Then, following [Reference Graber and Pandharipande23], if we have a global equivariant embedding of a scheme X into a nonsingular scheme Y with $({\mathbb C}^{*})^N$ action, we can identify the maximal $({\mathbb C}^{*})^N$ -fixed closed subscheme $X^f$ of X with the scheme-theoretic intersection $X^f=X\cap Y^f$ , where $Y^f$ is the nonsingular set-theoretic fixed point locus. By decomposing $Y^f$ into irreducible components $Y^f=\bigcup _iY_i$ , one can also define $X_i=X\cap Y_i$ , which carry a perfect obstruction theory with virtual fundamental class $[X_i]^{\mathrm {vir}}$ . In this way, if $\tilde V\in K^0_{({\mathbb C}^{*})^N}(X)$ is an equivariant lift of the vector bundle V, $\tilde V_i$ is its restriction to $X_i$ and $p_i:X_i\to \mathrm { pt}$ is the projection, one has that

(3.2.3) $$ \begin{align} \chi^{\mathrm{vir}}(X,\tilde V;\varepsilon_1,\dots,\varepsilon_N)=\sum_i p_{i*}^{\mathrm{vir}}\left(\tilde V_i/\Lambda_{-1}(N_i^{\mathrm{vir}})^\vee\right)=\sum_i p_{i*}^{\mathrm{vir}}\left(\tilde V_i/\Lambda_{-1}(T_X^{\mathrm{vir}}|_{X_i}^{\mathrm{mov}})^\vee\right) \end{align} $$

belongs to $\mathbb Q[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ and the virtual holomorphic Euler characteristic is $\chi ^{\mathrm {vir}}(X,V)=\chi ^{\mathrm {vir}}(X,\tilde V;\boldsymbol 0)$ .

As pointed out at the beginning of §3, we will define invariants in localization, as the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,{\mathbf n})$ is proper, while ${\mathcal N}(r,{\mathbf n})$ is only quasi-projective, so the pushforward in the right-hand side of Equation 3.2.3 is well defined. Computations are now made very easy by the fact that we represented the virtual tangent space to the ${\mathbb T}$ -fixed points to the moduli space of nested instantons in the representation ring $R({\mathbb T})$ . In this way, $T_{X_i}^{\mathrm {vir}}$ is decomposed as a direct sum of line bundles which are moreover eigenbundles of the torus action. Then we can use the properties

$$\begin{align*}\operatorname{\mathrm{ch}}(E\oplus F)=\operatorname{\mathrm{ch}} E+\operatorname{\mathrm{ch}} F,\quad \Lambda_{t}(E\oplus F)=\Lambda_{t}(E)\cdot\Lambda_{t}(F),\quad S_t(E\oplus F)=S_t(E)\cdot S_t(F) \end{align*}$$

and Equation (3.2.3) to compute the equivariant holomorphic Euler characteristic of the moduli space of nested instantons in terms of the fundamental characters $\mathfrak q_{1,2}$ of the torus ${\mathbb T}_1$ . These will be related to the equivariant parameters by $\mathfrak q_i={\text e}^{\beta \varepsilon _i}$ , with $\beta $ being a parameter having a very clear meaning in the physical framework modelling the moduli space of nested instantons as a low energy effective theory. In this framework, it is very easy to explicitly compute the virtual equivariant holomorphic Euler characteristic of the moduli space of nested instantons, as we already described the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,n_0,\dots ,n_N)$ as being zero-dimensional and reduced.Footnote ⁶ As we saw in §3.1, the fixed points of ${\mathcal N}(r,n_0,\dots ,n_N)$ are completely described by r-tuples of nested coloured partitions $\boldsymbol \mu _1\subseteq \cdots \subseteq \boldsymbol \mu _N\subseteq \boldsymbol \mu _0$ , with $\boldsymbol \mu _j\in \mathcal P^r$ , in such a way that $|\boldsymbol \mu _0|=\sum _j|\boldsymbol \mu _0^j|=n_0$ and $|\boldsymbol \mu _0\setminus \boldsymbol \mu _{i>0}|=n_{i>0}$ . In the simplest case of $r=1$ , we get

(3.2.4) $$ \begin{align} \begin{aligned} \chi^{\mathrm{vir}}({\mathcal N}(1,{\mathbf n}),\tilde V;\mathfrak q_1,\mathfrak q_2)=&\sum_{\substack{\mu_1\subseteq\cdots\subseteq\mu_0\\ |\mu_0\setminus\mu_j|=n_j}}\frac{\displaystyle{T_{\mu_0,\mu_1}(\mathfrak q_1,\mathfrak q_2)W_{\mu_0,\dots,\mu_N}(\mathfrak q_1,\mathfrak q_2)}}{\displaystyle{N_{\mu_0}(\mathfrak q_1,\mathfrak q_2)}}\left.\left[\tilde V\right]\right|_{\mu_0,\dots,\mu_N}, \end{aligned} \end{align} $$

where we defined

(3.2.5) $$ \begin{align} N_{\mu_0}(\mathfrak q_1,\mathfrak q_2)&=\prod_{s\in Y_{\mu_0}}\left(1-\mathfrak q_1^{-l(s)-1}\mathfrak q_2^{a(s)}\right)\left(1-\mathfrak q_1^{l(s)}\mathfrak q_2^{-a(s)-1}\right), \end{align} $$

(3.2.6) $$ \begin{align} T_{\mu_0,\mu_1}(\mathfrak q_1,\mathfrak q_2)&=\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\left(1-\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{\prime}}\right), \end{align} $$

(3.2.7) $$ \begin{align} W_{\mu_0,\dots,\mu_N}(\mathfrak q_1,\mathfrak q_2)&=\prod_{k=1}^{N}\prod_{i=1}^{M_0}\prod_{j=1}^{N_0}\frac{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)}}{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)}}, \end{align} $$

with $a(s)$ and $l(s)$ the arm length and the leg length of the box s in the Young diagram $Y_{\mu }$ associated to $\mu $ , respectively. A very interesting and surprising fact can be observed if we rearrange the expression of the holomorphic virtual Euler characteristic of ${\mathcal N}(1,{\mathbf n})$ . In fact, if we perform the summation over the smaller partitions $\mu _1\subseteq \cdots \subseteq \mu _N$ and redefine $q=\mathfrak q_1^{-1}$ , $t=\mathfrak q_2^{-1}$ , we get

(3.2.8) $$ \begin{align} \chi^{\mathrm{vir}}({\mathcal N}(1,{\mathbf n});\mathfrak q_1,\mathfrak q_2)=\sum_{\mu_0}\frac{P_{\mu_0}(q,t)}{N_{\mu_0}(q,t)}, \end{align} $$

and the unexpected fact is that we think $P_{\mu _0}(q,t)$ to be a polynomial in $q,t$ except for a factor $(1-qt)^{-N}$ .

Conjecture 3.6. $P_{\mu _0}(q,t)$ in Equation (3.2.8) is a function of the form

(3.2.9) $$ \begin{align} P_{\mu_0}(q,t)=\frac{Q_{\mu_0}(q,t)}{(1-qt)^N}, \end{align} $$

with $Q_{\mu _0}(q,t)\in \mathbb Z[q,t]$ a polynomial in the $(q,t)$ -variables.

Sometimes the polynomials in (3.2.9) can be given an interpretation in terms of some known symmetric polynomials. Consider the ring $\Lambda (\mathbf x)$ of symmetric functions in the infinite set of variables $\{x_1,x_2,\dots \}$ . It is convenient to sometimes denote by the same symbol X both the formal sum $X=x_1+x_2+\cdots $ and the alphabet $\mathbf x=\{x_1,x_2,\dots \}$ . If $\lambda $ is an integer partition, we can define the monomial functions

$$\begin{align*}m_\lambda=\sum_{\alpha}x_1^{\alpha_1}x_2^{\alpha_2}\cdots, \end{align*}$$

where the sum runs over all the permutations of $\lambda $ . The ring of symmetric functions $\Lambda (\mathbf x)$ will then be the free $\mathbb Z$ -module generated by the monomial functions $m_\lambda $ , for all partitions $\lambda $ . Two other sets of symmetric functions in $\Lambda (\mathbf x)$ are the complete homogeneous symmetric functions $h_\lambda $ and the power functions $p_\lambda $ , defined as

$$ \begin{align*} &h_\lambda=h_{\lambda_1}h_{\lambda_2}\cdots,\qquad h_k=\sum_{\lambda\vdash k}m_\lambda\\ &p_\lambda=p_{\lambda_1}p_{\lambda_2}\cdots,\qquad p_k=\sum_{j>0}x_j^k. \end{align*} $$

The symmetric functions $h_\lambda $ and $p_\lambda $ form other $\mathbb Q$ -bases of $\Lambda (\mathbf x)$ indexed by integer partitions, and we can moreover introduce a symmetric positive-definite bilinear form, the Hall pairing $\langle -,-\rangle $ , such that the two $\mathbb Q$ -bases $m_\lambda $ and $h_\lambda $ are dual to each other (i.e., $\langle h_\lambda ,m_\mu \rangle =\delta _{\lambda \mu }$ , for any choice of partitions $\lambda ,\mu $ ). Consider then the function

$$\begin{align*}\Omega[X]=\exp\left(\sum_{k\ge 1}\frac{p_k[X]}{k}\right), \end{align*}$$

where the plethystic notation $P[X]=P(x_1,x_2,\dots )$ is used, for any $P\in \Lambda (\mathbf x)$ and $X=x_1+x_2+\cdots $ , as before. On the ring $\Lambda (\mathbf x)\otimes _{\mathbb Z}\mathbb Q(q,t)$ , we can introduce the operator

$$\begin{align*}\Delta f=\operatorname{\mathrm{Coeff}}_{z^0}\left(f[X+(1-q)(1-t)/z]\Omega[-zX]\right), \qquad f\in\Lambda(\mathbf x)\otimes_{\mathbb Z}\mathbb Q(q,t), \end{align*}$$

and a set of eigenfunctions for $\Delta $ is given by the modified (or transformed) Macdonald polynomials $\widetilde H_\lambda (\mathbf x;q,t)$ . They form a basis for $\Lambda (\mathbf x)\otimes _{\mathbb Z}\mathbb Q(q,t)$ and are defined as