Liouville-type results for positive solutions of pseudo-relativistic Schrödinger system

Yuxia Guo; Shaolong Peng

doi:10.1017/prm.2021.81

Liouville-type results for positive solutions of pseudo-relativistic Schrödinger system

Part of: Miscellaneous topics - Partial differential equations Partial differential equations

Published online by Cambridge University Press: 13 December 2021

Yuxia Guo and

Shaolong Peng

Show author details

Yuxia Guo: Affiliation:
Department of Mathematics, Tsinghua University, Beijing 100084, P. R. China ([email protected], [email protected])
Shaolong Peng: Affiliation:
Department of Mathematics, Tsinghua University, Beijing 100084, P. R. China ([email protected], [email protected])

Article contents

Abstract
Introduction
Narrow region principle and decay at infinity
Proof of theorem
Proof of theorem
Proof of theorem
Proof of theorems and
Data Availability Statements
References

Rights & Permissions

Abstract

In this paper, we are concerned with the physically engaging pseudo-relativistic Schrödinger system:

\[ \begin{cases} \left(-\Delta+m^{2}\right)^{s}u(x)=f(x,u,v,\nabla u) & \hbox{in } \Omega,\\ \left(-\Delta+m^{2}\right)^{t}v(x)=g(x,u,v,\nabla v) & \hbox{in } \Omega,\\ u>0,v>0 & \hbox{in } \Omega, \\ u=v\equiv 0 & \hbox{in } \mathbb{R}^{N}\setminus\Omega, \end{cases} \]

where $s,t\in (0,1)$

and the mass $m>0.$

By using the direct method of moving plane, we prove the strict monotonicity, symmetry and uniqueness for positive solutions to the above system in a bounded domain, unbounded domain, $\mathbb {R}^{N}$

, $\mathbb {R}^{N}_{+}$

and a coercive epigraph domain $\Omega$

in $\mathbb {R}^{N}$

, respectively.

Keywords

Pseudo-relativistic Schrödinger system direct methods of moving planes Liouville-type theorems monotonicity symmetry

MSC classification

Primary: 35R11: Fractional partial differential equations

Secondary: 35B06: Symmetries, invariants, etc. 35B53: Liouville theorems, Phragmén-Lindelöf theorems

Type: Research Article
Information: Proceedings of the Royal Society of Edinburgh Section A: Mathematics , Volume 153 , Issue 1 , February 2023 , pp. 196 - 228

DOI: https://doi.org/10.1017/prm.2021.81 [Opens in a new window]
Copyright: Copyright © The Author(s), 2021. Published by Cambridge University Press on behalf of The Royal Society of Edinburgh

1. Introduction

We consider the following generalized pseudo-relativistic Schrödinger system:

(1.1)\begin{equation} \begin{cases}\left(-\Delta+m^{2}\right)^{s}u(x)=f(x,u,v,\nabla u) & \hbox{in } \Omega,\\ \left(-\Delta+m^{2}\right)^{t}v(x)=g(x,u,v,\nabla v) & \hbox{in } \Omega,\\ u>0,v>0 & \hbox{in } \Omega, \\ u=v\equiv 0 & \hbox{in } \mathbb{R}^{N}\setminus\Omega, \end{cases}\end{equation}

where $(-\Delta +m^{2})^{s}$ is the generalized pseudo-relativistic Schrödinger operator, $\Omega$ may be bounded domain, unbounded domain, $\mathbb {R}^{N}_+$, $\mathbb {R}^{N}$ or a coercive epigraph domain $\Omega$ in $\mathbb {R}^{N}.$

System (1.1) is closely related to the pseudo-relativistic Schrödinger equations:

(1.2)\begin{equation} \textrm{i}\frac{\partial \Psi}{\partial t}=\left({-}c^{2}\Delta+m^{2}c^{4}\right)^{{1}/{2}}\Psi-mc^{2}\Psi-f(|\Psi(t, x)|^{2})\Psi, \forall (t,x)\in \mathbb{R}_+{\times} \mathbb{R}^{N}, \end{equation}

where $c$ denotes the speed of the light, $m>0$ represents the particle mass, $\Psi (t, x)$ is a complex-valued wave field and $f:[0, 1)\to \mathbb {R}$ is a nonlinear function describing the dynamics of the system consisting of identical spin-$0$ bosons whose motions are relativistic (see [Reference Lieb and Thirring32, Reference Lieb and Yau33]), such as boson stars. If we consider the standing waves solution $\Psi (t, x)=e^{i\omega t}u(x)$ to (1.2), where $\omega \in \mathbb {R}$ is the frequency and $u(x)$ is real-valued, then equation (1.2) becomes a static semi-linear elliptic equation:

(1.3)\begin{equation} \left({-}c^{2}\Delta+m^{2}c^{4}\right)^{{1}/{2}}u(x)-(\omega-mc^{2})u(x)=f(|u(x)|^{2})u(x), \quad \forall x\in \mathbb{R}^{N}. \end{equation}

Recall that the pseudo-relativistic operators $(-\Delta +m^{2})^{s}$ in $\mathbb {R}^{N}$ is defined by (see [Reference Ambrosio1, Reference Carmona, Masters and Simon7])

(1.4)\begin{align} \left(-\Delta+m^{2}\right)^{s}u(x)& :=c_{N,s} m^{{{N}/{2}+s}} P.V. \int_{\mathbb{R}^{N}} \frac{u(x)-u(y)}{|x-y|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}(m|x-y|)\,\textrm{d} y\notag\\ & \quad +m^{2 s}u(x), \end{align}

where $s\in (0, 1)$, $P.V.$ stands for Cauchy principal value, and

(1.5)\begin{equation} c_{N, s}=2^{1-{N}/{2}+s}\pi^{-{N}/{2}}\frac{s(1-s)}{\Gamma(2-s)}, \end{equation}

$K_{\nu }$ denotes the modified Bessel function with order $\nu$, which solves the equation

(1.6)\begin{equation} r^{2} K_{\nu}^{\prime \prime}+r K_{\nu}^{\prime}-\left(r^{2}+\nu^{2}\right) K_{\nu}=0, \end{equation}

and satisfies the following integral representation

\[ K_{\nu}(r)=\displaystyle\int_{0}^{\infty} \textrm{e}^{{-}r \cosh t} \cosh (\nu t) \,\textrm{d}t. \]

It is easy to verify that $K_{\nu }(r)$ is a real and positive function satisfying $K'_{\nu }(r)<0$ for all $r>0$, $K_{\nu }=K_{-\nu }$ for $\nu <0$. Moreover, for $\nu >0$, it holds that (see [Reference Erdélyi, Magnus, Oberhettinger and Tricomi26–Reference Fall and Felli28, Reference Guo and Peng31])

(1.7)\begin{equation} \lim_{r\rightarrow 0}\frac{K_{\nu}(r)}{\frac{\Gamma(\nu)}{2}\left(\frac{r}{2}\right)^{-\nu}}=1. \end{equation}

Hence there exists a small $r_0>0$ and two constants $C_0>c_0>0$ such that

(1.8)\begin{equation} \frac{c_0}{r^{\nu}}\leq K_\nu(r)\leq\frac{C_0}{r^{\nu}},\quad \hbox{for all } r\leq r_0. \end{equation}

Next, for $\nu >0$, we can also derive that

(1.9)\begin{equation} \frac{ K_{\nu}(r)}{\frac{\sqrt{\pi}}{\sqrt{2}} r^{{-}1/2} \textrm{e}^{{-}r}}\to 1, \quad\hbox{as } r \rightarrow\infty. \end{equation}

Hence there exists a large $R_\infty >0$ and two constants $C_\infty >c_\infty >0$ such that

(1.10)\begin{equation} \frac{c_\infty}{r^{1/2}\textrm{e}^{r}}\leq K_\nu(r)\leq\frac{C_\infty}{r^{1/2}\textrm{e}^{r}},\quad \hbox{for all } r\geq R_\infty. \end{equation}

Let

(1.11)\begin{equation} \mathcal{L}_s(\mathbb{R}^{N}):=\left\{u : \mathbb{R}^{N} \rightarrow \mathbb{R} \left|\, \int_{\mathbb{R}^{N}} \frac{\textrm{e}^{-|x|}|u(x)|}{1+|x|^{({N+1})/{2}+s}}\,\textrm{d}x<{+}\infty\right.\right\}. \end{equation}

Then it is easy to verify that for any $u \in \mathcal {L}_s(\mathbb {R}^{N})\cap C_{\text {loc }}^{1,1}(\mathbb {R}^{N})$, the integral on the right-hand side of the definition (1.4) is well-defined. Hence $(-\Delta +m^{2})^{s}u$ makes sense for all functions $u \in \mathcal {L}_s(\mathbb {R}^{N})\cap C_{\text {loc}}^{1,1}(\mathbb {R}^{N})$. We would like to mention that the generalized pseudo-relativistic operators $(-\Delta +m^{2})^{s}$ can also be expressed equivalently via the extension method. For more details, we refer to [Reference Fall and Felli27, Reference Fall and Felli28] and the references therein.

Besides its physical background, we see that the generalized pseudo-relativistic operators $(-\Delta +m^{2})^{s}$ become the fractional Laplacian $(-\Delta )^{s}$ as $m\rightarrow 0+$, which is a nonlocal pseudo-differential operator defined by (see e.g. [Reference Chang and Gonzàlez8, Reference Chen, Li and Li10, Reference Chen, Li and Ma11, Reference Chen, Li and Zhang13, Reference Chen and Qi15, Reference Dai and Qin22, Reference Liu and Ma35, Reference Peng36])

(1.12)\begin{equation} (-\Delta)^{s} u(x) = C_{N, s} \, P.V. \int_{\mathbb{R}^{N}} \frac{u(x)-u(y)}{|x-y|^{N+2s}}\, \textrm{d}y, \end{equation}

where $0< s<1$, $C_{N,s}=(\int _{\mathbb {R}^{N}}(({{1-\cos (2\pi y_{1}))}/{|y|^{N+2s}})\,\textrm {d}y})^{-1}$. Fractional Laplacian $(-\Delta )^{s}$ is well-defined for any $u\in C^{1,1}_{loc}(\mathbb {R}^{N})\cap \dot {L}_{s}(\mathbb {R}^{N})$ with the function spaces

\[ \dot{L}_{s}(\mathbb{R}^{N}):=\left\{ u: \mathbb{R}^{N} \rightarrow \mathbb{R} \left| \,\int_{\mathbb{R}^{N}} \frac{|u(x)|}{1 + |x|^{N+2s}}\,\textrm{d}x <{+}\infty \right.\right\}. \]

It can also be defined equivalently through Caffarelli and Silvestre's extension method (see [Reference Brandle, Colorado, de Pablo and Sanchez3]).

Due to its various applications in fluid mechanics, molecular dynamics, relativistic quantum mechanics of stars (see [Reference Caffarelli and Vasseur6, Reference Constantin18]), in conformal geometry (see [Reference Chen, Li and Ma11]) and in probability and finance (see [Reference Bertoin2, Reference Cabré and Tan4]). Problems involving fractional operators have attracted more and more attention in recent years. Note that the equations with general pseudo-relativistic Schrödinger operator $(-\Delta +m^{2})^{s}$ with $m\geq 0$ are inhomogeneous in general. Therefore scaling invariance does not hold for this kind of problem. To overcome this difficulty, there are two basic approaches in the literature. One approach is to define the operators $(-\Delta +m^{2})^{s}$ via the extension method and reduce the nonlocal problem into a local one in higher dimension space. Another way is to use the integral representation formulae of solutions [Reference Chen, Li and Ma11, Reference Chen, Li and Ou12]. By establishing the equivalence between the fractional order equation and its corresponding integral equation (with Bessel kernel if $m > 0$, Riesz kernel if $m = 0$), one can apply the method of moving planes (spheres) in integral forms, the method of scaling spheres or regularity lifting methods to study various properties of solutions. These two methods have been applied successfully to study the problems involving fractional operators $(-\Delta )^{s} (0< s<1)$, and a series of fruitful results have been achieved (see [Reference Brandle, Colorado, de Pablo and Sanchez3–Reference Caffarelli and Silvestre5, Reference Chen and Li9, Reference Chen, Li and Ou12, Reference Chen and Liu14, Reference Chen and Wu16, Reference Cheng, Huang and Li17, Reference Dai, Fang, Huang, Qin and Wang19–Reference Dai, Liu and Qin21, Reference Dai and Qin24, Reference Dipierro, Soave and Valdinoci25, Reference Frank, Lenzmann and Silvestre29, Reference Guo and Liu30, Reference Liu, Guo and Zhang34, Reference Silvestre37] and the references therein). The disadvantage of the approaches mentioned above is that one needs to impose some additional conditions on the solutions. Recently, Dai et al. [Reference Dai, Qin and Wu23] established a direct method for the general pseudo-relativistic Schrödinger operators.

The present paper aims to extend the direct method of moving planes for a generalized fractional pseudo-relativistic nonlinear Schrödinger system. The nonlinearities depend not only on $u$ but also depend on $\nabla u.$ We first establish maximum principles such as narrow region principle and decay at infinity, then combine these results with the direct method of moving plane. We derive the symmetry and monotonicity of solutions to the pseudo-relativistic Schrödinger system in various domains, including bounded domain, unbounded domain and epigraph $\Omega$ in $\mathbb {R}^{N}.$ And then, we derive Liouville-type results in $\mathbb {R}^{N}$ and $\mathbb {R}^{N}_+$.

In the following, we denote $({\partial u}/{\partial x_{1}},{\partial u}/{\partial x_{2}},\ldots,{\partial u}/{\partial x_{N}})$ by $\mathbf {p}$ and $({\partial v}/{\partial x_{1}},{\partial v}/{\partial x_{2}},\ldots,{\partial v}/{\partial x_{N}})$ by $\mathbf {q}$,

We say a function $f(x, u, v,\mathbf {p}): \, \Omega \times \mathbb {R}\times \mathbb {R}\times \mathbb {R}^{n}\rightarrow \mathbb {R}$ is locally Lipschitz in $u$, uniformly in $\mathbf {p}$ and locally uniformly in $x$: for any $M>0$ and any compact subset $K\subset \bar {\Omega }$, there exists $C_{K,M}>0$ such that, $\forall \, u_1,u_2\in [-M, M]$, $\forall \, x\in K$ and $\forall \, \mathbf {p}\in \mathbb {R}^{N}$,

\[ |f(x, u_1, v, \mathbf{p})-f(x, u_2, v, \mathbf{p})|\leq C_{K,M}|u_1-u_2|. \]

We assume the following conditions on the nonlinearity $f$ and $g$:

\[ (\mathbf{F}_\textbf{{1}}):\quad \frac{\partial f}{\partial v}>0,\quad \frac{\partial g}{\partial u}>0,\quad \frac{\partial f}{\partial u} < m^{2s},\quad \frac{\partial g}{\partial v} < m^{2t}. \]

More precisely, we first concerned the generalized pseudo-relativistic Schrödinger system (1.1) in bounded domain and unbounded domain, the main results are as follows.

Theorem 1.1 Let $\Omega \subset \mathbb {R}^{N}$ be a bounded domain which is convex in $x_1$-direction and symmetric about $\{x\in \mathbb {R}^{N} \mid x_{1}=0\}$. Suppose that $u\in C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })$, $v\in C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })$ solve system (1.19), $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ are Lipschitz continuous with respect to $u,v$. If $f(x,u,v,\mathbf {q})$, $g(x,u,v,\mathbf {q})$ satisfy conditions $\mathbf {F_1}$ and

(1.13)\begin{equation} \begin{cases}f(x_1, x', u, v, p_1, p_2,\dots, p_N)\leq f(\bar{x}_1, x', u, v, -p_1, p_2,\dots, p_N),\\ g(x_1, x', u, v, q_1, q_2,\dots, q_N)\leq g(\bar{x}_1, x', u, v, -q_1, q_2,\dots, q_N),\\ \qquad \forall \ - x_1\geq 0, p_1\geq 0, q_1\geq 0,\ x_1\leq \bar{x}_1\leq{-}x_1. \end{cases}\end{equation}

Then $u(x_1, x')$ and $v(x_1, x')$ are strictly increasing in the left half of $\Omega$ in $x_1$-direction and

\[ u(x_1, x')\leq u({-}x_1, x'),\quad v(x_1, x')\leq v({-}x_1, x'),\quad \forall \ x_1<0, \ (x_1,x')\in \Omega. \]

Moreover, if

(1.14)\begin{equation} \begin{cases}f(x_1, x', u, p_1, p_2,\dots, p_N)=f({-}x_{1}, x', u, -p_1, p_2,\dots, p_N),\\ g(x_1, x', u, q_1, q_2,\dots, q_N)=g({-}x_{1}, x', u, -q_1, q_2,\dots, q_N),\\ \qquad \forall \ x_{1}p_{1}\leq 0,\ x_{1}q_{1}\leq 0, \end{cases}\end{equation}

then we have

\[ u(x_1, x')=u({-}x_1, x'),\quad v(x_1, x')=v({-}x_1, x'), \quad \forall \ x_1<0, \ (x_1,x')\in \Omega. \]

To state our monotonicity and symmetry results for unbounded domain $\Omega$, we need to impose a growth condition on $f(x,u,v,\mathbf {p})$ and $g(x,u,v,\mathbf {q})$: for some $\beta _{1}, \beta _{2}>0$, there exists a positive constant $C$ such that

(1.15)\begin{align} \frac{|f(x, u_1,v, \mathbf{p})-f(x, u_2,v, \mathbf{p})|}{|u_1-u_2|}& \leq C(|u_1|^{\beta_{1}}+|u_2|^{\beta_{1}}), \quad \text{as}\ u_1, u_2\rightarrow 0, \end{align}

(1.16)\begin{align} \frac{|g(x, u, v_1, \mathbf{q})-g(x, u, v_2, \mathbf{q})|}{|v_1-v_2|}& \leq C(|v_1|^{\beta_{2}}+|v_2|^{\beta_{2}}), \quad \text{as}\ v_1, v_2\rightarrow 0. \end{align}

Theorem 1.2 Let $\Omega$ be a unbounded domain in $\mathbb {R}^{N}$ which is convex in $x_1$-direction and symmetric with respect to $\{x\in \mathbb {R}^{N} \mid x_1 = 0\}$. Suppose that $u\in C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })\cap \mathcal {L}_{s}(\mathbb {R}^{N})$, $v\in C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })\cap \mathcal {L}_{t}(\mathbb {R}^{N})$ solve the system (1.19) and $f, g$ are Lipschitz continuous with respect to $u,v$. If $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ satisfy $\mathbf {F}_{\mathbf {1}}$, (1.13), (1.15), (1.16) and $u(x), v(x)$ satisfy the following asymptotic properties:

(1.17)\begin{equation} \limsup_{\substack{|x|\rightarrow\infty \\ x\in\Omega, \, x_1<0}}\left[u(x)\right]^{\beta_{1}}<\frac{m^{2s}}{4C},\quad \limsup_{\substack{|x|\rightarrow\infty \\ x\in\Omega, \, x_1<0}}\left[v(x)\right]^{\beta_{2}}<\frac{m^{2t}}{4C}, \end{equation}

and

(1.18)\begin{equation} \min_{\bar{\Omega}}\left(m^{2s}-2C\left[u(x)\right]^{\beta_{1}}\right)\min_{\bar{\Omega}}\left(m^{2t}-2C\left[v(x)\right]^{\beta_{2}}\right) \geq\max_{\bar{\Omega}}\frac{\partial f}{\partial v}\cdot \max_{\bar{\Omega}} \frac{\partial g}{\partial u}, \end{equation}

where $C$ is the constant in assumption (1.15). Then we have

(i) If there exists a line $\mathcal {L}$ parallel to $x_{1}$-axis satisfying $\mathcal {L}\cap \Omega \neq \emptyset$ such that $\mathcal {L}\cap \Omega ^{c}\neq \emptyset$, then $u(x_1, x')$, $v(x_1, x')$ are strictly increasing in the left half of $\Omega$ in $x_1$-direction and
\[ u(x_1, x')\leq u({-}x_1, x'),\quad v(x_1, x')\leq v({-}x_1, x'), \quad \forall \ x_1<0, \ (x_1,x')\in \Omega. \]
(ii) If any line $\mathcal {L}$ parallel to $x_{1}$-axis such that $\mathcal {L}\cap \Omega \neq \emptyset$ must satisfy $\mathcal {L}\cap \Omega ^{c}=\emptyset$, then there exists $\mu _{0}\leq 0$ such that $u(x_1, x'), v(x_1, x')$ are strictly increasing in $\Omega \cap \{x_{1}<\mu _{0}\}$ in $x_1$-direction and
\begin{align*} \text{if} \ \mu_{0}=0, \quad u(x_1, x')& \leq u({-}x_1, x'),\ v(x_1, x')\leq v({-}x_1, x'),\\ & \qquad \forall \ x_1<0, \ (x_1,x')\in \Omega, \end{align*}

\begin{align*}\text{if} \ \mu_{0}<0, \quad u(x_1, x')& =u(2\mu_{0}-x_1, x'),\ v(x_1, x')=v(2\mu_{0}-x_1, x'),\\ & \qquad \forall \ x_1<\mu_{0}, \ (x_1,x')\in \Omega. \end{align*}

Secondly, we concerned with the following generalized pseudo-relativistic Schrödinger system in $\mathbb {R}^{N}$:

(1.19)\begin{equation} \begin{cases} \left(-\Delta+m^{2}\right)^{s}u(x)=f(x,u,v,\nabla u), & \hbox{ in }\mathbb{R}^{N}, \\ \left(-\Delta+m^{2}\right)^{t}v(x)=g(x,u,v,\nabla v), & \hbox{ in }\mathbb{R}^{N}, \end{cases} \end{equation}

where $s,t\in (0,1)$ and $m>0$. Then we derive the following Liouville-type result in $\mathbb {R}^{N}$.

Theorem 1.3 Liouville theorem in $\mathbb {R}^{N}$

Assume that $u \in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})$, $v \in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})$ be a pair of nonnegative solutions to generalized pseudo-relativistic Schrödinger system (1.19), $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ are Lipschitz continuous about $u,v$ and satisfy assumption $\mathbf {F_1}$ and (1.14). Suppose that $(u,v)$ are bounded and $f,g$ satisfy

\[ (\mathbf{F_{2}}):\quad \min_{\mathbb{R}^{N}}\left(m^{2s}-\frac{\partial f}{\partial u}\right)\min_{\mathbb{R}^{N}}\left(m^{2t}-\frac{\partial g}{\partial v}\right)\geq\max_{\mathbb{R}^{N}}\frac{\partial f}{\partial v}\cdot\max_{\mathbb{R}^{N}}\frac{\partial g}{\partial u}, \]

then we have

\[(u,v)\equiv (C_{1},C_{2}). \]

Thirdly, we consider the generalized pseudo-relativistic Schrödinger system (1.19) in a coercive epigraph $\Omega$. We say a domain $\Omega \subseteq \mathbb {R}^{N}$ is coercive epigraph if there exists a continuous function $\varphi : \mathbb {R}^{N-1}\rightarrow \mathbb {R}$ satisfying

(1.20)\begin{equation} \lim_{|x'|\rightarrow +\infty}\varphi(x')={+}\infty, \end{equation}

such that $\Omega =\{x=(x',x_N)\in \mathbb {R}^{N}|x_N>\varphi (x')\}$. Our main result for generalized pseudo-relativistic Schrödinger system (1.19) in a coercive epigraph $\Omega$ is as follows.

Theorem 1.4 Assume that $u\in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })$, $v\in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C^{1,1}_{loc}(\Omega )\cap C(\bar {\Omega })$ be a pair of nonnegative solution to system (1.1), $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ are Lipschitz continuous with respect to $u,v$. Further suppose that $f,g$ satisfy condition $\mathbf {F_{1}}$ and

(1.21)\begin{equation} \begin{cases}f(x', x_{N}, u, v, p_1, p_2,\dots, p_N)\leq f(x', \bar{x}_{N}, u, v, p_1, p_2,\dots, -p_N),\\ g(x', x_{N}, u, v, q_1, q_2,\dots, p_N)\leq g(x', \bar{x}_{N}, u, v, q_1, q_2,\dots, -q_N),\\ \qquad \forall \ x_N\geq\min_{\mathbb{R}^{N-1}}\varphi, \ p_N\geq 0, \ q_N\geq 0, \ \bar{x}_{N}\geq x_{N}. \end{cases}\end{equation}

Then $(u,v)$ are strictly monotone increasing with respect to the $x_{N}$-axis.

Finally, we shall consider the generalized pseudo-relativistic Schrödinger system (1.1) in $\mathbb {R}^{N}_{+}$. By applying theorem 1.4, we prove the following result.

Theorem 1.5 Liouville theorem in $\mathbb {R}^{N}_{+}$

Assume that $u \in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N}_{+})$, $v \in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N}_{+})$ be a pair of nonnegative solutions of

(1.22)\begin{equation} \begin{cases} \left(-\Delta+m^{2}\right)^{s}u(x)=f(x,u,v,\nabla u), & \forall x\in \mathbb{R}^{N}_{+}, \\ \left(-\Delta+m^{2}\right)^{t}v(x)=g(x,u,v,\nabla v), & \forall x\in \mathbb{R}^{N}_{+}, \\ u=v=0, & \forall x \notin \mathbb{R}^{N}_{+}, \end{cases} \end{equation}

where $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ are Lipschitz continuous about $u,v$ and satisfy condition $\mathbf {F_{1}}$, (1.21) with $x_N\geq \min _{\mathbb {R}^{N-1}}\varphi =0$. Suppose that

\[ \lim_{|x|\rightarrow+\infty}(u(x),v(x))=(0,0),\quad f(x,0,0,0)=g(x,0,0,0)=0. \]

Then we have

\[(u,v)\equiv (0,0).\]

Our paper is organized as follows. In § 2, we establish the narrow region principle for the generalized pseudo-relativistic Schrödinger system (1.1) and give the decay at infinity. The proof of theorem 1.1 and theorem 1.2 will be provided in §§ 3 and 4, respectively. In § 5, we prove the Liouville-type results for the generalized pseudo-relativistic Schrödinger system in $\mathbb {R} ^{N}$. In the last section, we study the strictly monotone and Liouville-type results for the generalized pseudo-relativistic Schrödinger system in a coercive epigraph domain $\Omega$ and the half-space $\mathbb {R}^{N}_+,$ respectively.

2. Narrow region principle and decay at infinity

This section establishes the narrow region principle and study the decay at infinity for a anti-symmetric system. These principles are crucial in applying the direct method of moving planes for the pseudo-relativistic Schrödinger system.

We first introduce the following notations. For arbitrary $\lambda \in \mathbb {R}$, let

\[ T_{\lambda}:=\left\{x \in \mathbb{R}^{N} | x_{1}=\lambda\right\} \]

be the moving planes, and let

(2.1)\begin{equation} \Sigma_{\lambda}:=\left\{x \in \mathbb{R}^{N} | x_{1}<\lambda\right\} \end{equation}

be the region to the left of the plane, and set

\[ x^{\lambda}:=\left(2 \lambda-x_{1}, x_{2}, \dots, x_{N}\right) \]

be the reflection of $x$ about the plane $T_{\lambda }$. Let $(u,v) \in \{\mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})\}\times \{\mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})\}$ be a pair of nonnegative solutions to the pseudo-relativistic Schrödinger system (1.19), we denote the reflected functions by $u_{\lambda }(x):=u(x^{\lambda })$ and $v_{\lambda }(x):=v(x^{\lambda })$. Moreover, we define

(2.2)\begin{equation} \left\{\begin{aligned} U_{\lambda}(x) & :=u\left(x^{\lambda}\right)-u(x) \\ V_{\lambda}(x) & :=v\left(x^{\lambda}\right)-v(x) \end{aligned}\right. \end{equation}

to be the comparison between the values of $u(x), u(x^{\lambda })$ and $v(x), v(x^{\lambda })$. Evidently, $U_{\lambda }$ and $V_{\lambda }$ are anti-symmetric functions. From now on and in the following of the paper, we always use the same $C$ denotes a constant whose value may be different from line to line, and only the relevant dependence is specified.

Proposition 2.1 Narrow region principle

Let $\Omega$ be a bounded open set in $\Sigma _{\lambda }$ which is contained in the region between $T_{\lambda }$ and $T_{\Omega }$, where $T_{\Omega }$ is a hyper-plane that is parallel to $T_{\lambda }$. Let $d(\Omega ):=dist(T_{\lambda },T_{\Omega })$. Suppose that $(U_{\lambda },V_{\lambda })\in \{\mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )\}\times \{\mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )\}$, $U_{\lambda }$, $V_{\lambda }$ are lower semi-continuous on $\bar {\Omega }$, and satisfy

(2.3)\begin{equation} \begin{cases}{(-\Delta+m^{2})^{s} U_{\lambda}(x)+c_{1}(x)U_{\lambda}(x)+c_{2}(x)V_{\lambda}(x)\geq 0} & {\text{in } \Omega}, \\ {(-\Delta+m^{2})^{t} V_{\lambda}(x)+c_{3}(x)V_{\lambda}(x)+c_{4}(x)U_{\lambda}(x)\geq 0} & {\text {in } \Omega}, \\ {U_{\lambda}(x) \geq 0, V_{\lambda}(x) \geq 0} & {\text {in } \Sigma_{\lambda} \backslash \Omega}, \\ {U_{\lambda}(x^{\lambda})={-}U_{\lambda}(x), V_{\lambda}(x^{\lambda})={-}V_{\lambda}(x)} & {\text{in } \Sigma_{\lambda},}\end{cases} \end{equation}

where $c_{1}(x), c_{2}(x), c_{3}(x), c_{4}(x)$ are uniformly bounded from below about $d(\Omega )$ and $c_{2}(x), c_{4}(x)<0$. We assume $\Omega$ is narrow in the sense that, $d(\Omega )\leq {r_{0}}/{4m}$, $r_{0}$ is given by (1.8), and there exists $C_{N,s}>0$ and $C_{N,t}>0$ such that

(2.4)\begin{equation} \inf_{\left\{x\in\Omega\,|\,U_{\lambda}(x)<0\right\}}c_{1}(x)+m^{2s}>{-}\frac{C_{N,s}}{d(\Omega)^{2s}},\quad \inf_{\left\{x\in\Omega\,|\,V_{\lambda}(x)<0\right\}}c_{3}(x)+m^{2t}>{-}\frac{C_{N,t}}{d(\Omega)^{2t}}. \end{equation}

Then we have either

(2.5)\begin{equation} U_{\lambda}(x)> 0,\ V_{\lambda}(x)> 0 \text{ in } \Omega\quad \text{or}\quad U_{\lambda}=V_{\lambda}=0 \quad \text{almost everywhere in} \ \mathbb{R}^{N}. \end{equation}

These conclusions hold for unbounded open set $\Omega$ if we further assume that

\[ \varliminf_{|x| \rightarrow \infty} U_{\lambda}(x)\geq 0,\quad \varliminf_{|x| \rightarrow \infty}V_{\lambda}(x) \geq 0. \]

Proof. Without loss of generalities, we may assume that

\[ T_{\lambda}=\{x\in\mathbb{R}^{N}\,|\,x_{1}=0\} \quad \text{and} \quad \Sigma_{\lambda}=\{x\in\mathbb{R}^{N}\,|\,x_{1}<0\}, \]

and hence $\Omega \subseteq \{x\in \mathbb {R}^{N}\,|-d(\Omega )< x_{1}<0\}$.

If $U_{\lambda }$ or $V_{\lambda }$ is not nonnegative in $\Omega$, we assume there exists a $\bar {x}\in \bar {\Omega }$ such that

\[ U_{\lambda}\left(\bar{x}\right)=\min_{\bar{\Omega}} U_{\lambda}<0. \]

Otherwise, the same arguments as follows can also yield a contradiction for the case that $V_{\lambda }(\bar {x})<0$.

First, we will prove that

(2.6)\begin{equation} U_{\lambda}(x)\geq 0,\quad V_{\lambda}(x)\geq 0 \text{ in } \Omega. \end{equation}

One can further deduce from (2.3) that $\bar {x}$ is in the interior of $\Omega$. It follows that

(2.7)\begin{align} & (-\Delta+m^{2})^{s} U_{\lambda}\left(\bar{x}\right) \nonumber\\ & \quad=c_{N,s}m^{{N}/{2}+s}P.V.\int_{\mathbb{R}^{N}}\frac{U_{\lambda}\left(\bar{x}\right) -U_{\lambda}(y)}{\left|\bar{x}-y\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-y\right|\right)\textrm{d}y+m^{2s}U_{\lambda}\left(\bar{x}\right) \nonumber\\ & \quad =c_{N,s}m^{{N}/{2}+s}P.V.\left\{\int_{\Sigma_\lambda} \frac{U_{\lambda}\left(\bar{x}\right)-U_{\lambda}(y)}{\left|\bar{x}-y\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-y\right|\right) \textrm{d}y\right.\nonumber\\ & \qquad +\left.\int_{\Sigma_\lambda}\frac{U_{\lambda}\left(\bar{x}\right)+U_{\lambda}(y)}{ \left|\bar{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-{y}^{\lambda}\right|\right) \textrm{d}y \right\}+m^{2s}U_{\lambda}\left(\bar{x}\right) \nonumber\\ & \quad \leq c_{N,s}m^{{N}/{2}+s} \int_{\Sigma_\lambda}\left\{\frac{U_{\lambda}\left(\bar{x}\right)-U_{\lambda}(y)}{\left|\bar{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}} +\frac{U_{\lambda}\left(\bar{x}\right)+U_{\lambda}(y)}{\left|\bar{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}}\right\}\notag\\ & \qquad \times K_{{{N}/{2}+s}} \left(m\left|\bar{x}-{y}^{\lambda}\right|\right)\textrm{d}y +m^{2s}U_{\lambda}\left(\bar{x}\right) \nonumber\\ & \quad =c_{N,s}m^{{N}/{2}+s}\int_{\Sigma_\lambda}\frac{2U_{\lambda}\left(\bar{x}\right)}{\left| \bar{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-{y}^{\lambda}\right|\right)\textrm{d}y +m^{2s}U_{\lambda}\left(\bar{x}\right). \end{align}

Let

\[ D:=\left\{y=(y_1,y')\in\mathbb{R}^{N} \mid d(\Omega)< y_{1}-(\bar{x})_{1}<2d(\Omega),\left|y^{\prime}-\left(\bar{x}\right)^{\prime}\right|<\displaystyle\frac{r_{0}}{2m}\right\}, \]

where $(x)^{\prime }=(x_2,x_3,\ldots,x_N)$, then $m|y-\bar {x}|< r_0,$ for all $y\in D$. Denote $t:=y_{1}-(\bar {x})_{1}$, $\tau :=\left |y^{\prime }-(\bar {x})^{\prime }\right |$. Using (1.8), we have

(2.8)\begin{align} & m^{{N}/{2}+s}\int_{\Sigma_\lambda}\frac{1}{\left|\bar{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-\tilde{y}\right|\right)\textrm{d}y \nonumber\\ & \quad \geq m^{{N}/{2}+s}\int_{D}\frac{1}{\left|\bar{x}-y\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\bar{x}-y\right|\right)\textrm{d}y \nonumber\\ & \quad \geq c_{0}\int_{D}\frac{1}{\left|\bar{x}-y\right|^{N+2s}}\,\textrm{d}y \nonumber\\ & \quad =c_{0}\int_{d(\Omega)}^{2d(\Omega)}\int_{0}^{{r_0}/{2m}}\frac{\sigma_{N-1}\tau^{N-2}\,\textrm{d}\tau}{ \left(t^{2}+\tau^{2}\right)^{{{N}/{2}+s}}}\,\textrm{d}t\nonumber\\ & \quad=c_{0}\int_{d(\Omega)}^{2d(\Omega)}\int_{0}^{{r_0}/{2mt}}\frac{\sigma_{N-1}(t\rho)^{N-2}t \,\textrm{d}\rho}{t^{N+2s}\left(1+\rho^{2}\right)^{{{N}/{2}+s}}}\,\textrm{d}t \nonumber\\ & \quad=c_{0}\int_{d(\Omega)}^{2d(\Omega)}\frac{1}{t^{1+2s}}\int_{0}^{{r_0}/{2mt}}\frac{\sigma_{N-1}\rho^{N-2} \,\textrm{d}\rho}{\left(1+\rho^{2}\right)^{{{N}/{2}+s}}}\,\textrm{d}t\nonumber\\ & \quad \geq c_{0}\int_{d(\Omega)}^{2d(\Omega)}\frac{1}{t^{1+2s}}\int_{0}^{1}\frac{\sigma_{N-1}\rho^{N-2} \,\textrm{d}\rho}{\left(1+\rho^{2}\right)^{{{N}/{2}+s}}}\,\textrm{d}t\nonumber\\ & \quad \geq C_{N,s}\int_{d(\Omega)}^{2d(\Omega)}\frac{1}{t^{1+2s}}\,\textrm{d}t=\frac{C_{N,s}}{d(\Omega)^{2s}}, \end{align}

where we have used the substitution $\rho :=\tau /t$ and $\sigma _{N-1}$ denotes the area of the unit sphere in $\mathbb {R}^{N-1}$. Since $c_{1}(x)$ has lower bound, it follows from (2.4), (2.7) and (2.8) that

\begin{align*} & (-\Delta+m^{2})^{s} U_{\lambda}\left(\bar{x}\right)+c_{1}\left(\bar{x}\right)U_{\lambda}\left(\bar{x}\right) \\ & \quad \leq\left[\displaystyle\frac{C_{N,s}}{d(\Omega)^{2s}}+\inf_{\{x\in\Omega\,|\,U_{\lambda}(x)<0\}}c_{1}(x)+m^{2s}\right]U_{\lambda}\left(\bar{x}\right)<0. \end{align*}

By (2.3), we have

(2.9)\begin{equation} c_{2}(\bar{x})V_{\lambda}\left(\bar{x}\right)\geq{-}(-\Delta+m^{2})^{s} U_{\lambda}\left(\bar{x}\right)-c_{1}\left(\bar{x}\right)U_{\lambda}\left(\bar{x}\right)>0. \end{equation}

Since $c_{2}(x)<0$, we deduce from (2.9) that $V_{\lambda }(\bar {x})<0$. We can infer from the lower semi-continuous of $V_{\lambda }$ that there exists some $\tilde {x}$ such that

\[ V_{\lambda}\left(\tilde{x}\right)=\min_{\bar{\Omega}} V_{\lambda}<0. \]

By using the similar arguments as used in (2.7) and (2.8), we can also get

\begin{align*} & (-\Delta+m^{2})^{t} V_{\lambda}\left(\tilde{x}\right)+c_{3}\left(\bar{x}\right)V_{\lambda}\left(\tilde{x}\right) \\ & \quad \leq\left[\displaystyle\frac{C_{N,t}}{d(\Omega)^{2t}}+\inf_{\{x\in\Omega\,|\,V_{\lambda}(x)<0\}}c_{3}(x)+m^{2t}\right]V_{\lambda}\left(\tilde{x}\right)<0. \end{align*}

Note that

(2.10)\begin{equation} \nabla U_{\lambda}\left(\bar{x}\right)=0. \end{equation}

By Taylor expansion, we have

(2.11)\begin{equation} 0=U_{\lambda}\left(x^{0}\right)=U_{\lambda}\left(\bar{x}\right)+\nabla U_{\lambda}\left(\bar{x}\right)\left(x^{0}-\bar{x}\right)+o\left(\left|\bar{x}-x^{0}\right|\right), \end{equation}

where $x^{0}=(0, \bar {x}_{2}, \ldots, \bar {x}_{N}) \in T_{\lambda }$. Hence, we obtain that

(2.12)\begin{equation} U_{\lambda}\left(\bar{x}\right)=o(1) \delta_{\bar{x}}, \end{equation}

for sufficiently small $\delta _{\bar {x}}.$

Thus a combination of (2.11), (2.12), and the assumption (2.4) deduces that

\begin{align*} 0 & \leq(-\Delta+m^{2})^{t} V_{\lambda}\left(\tilde{x}\right)+c_{3}\left(\tilde{x}\right)V_{\lambda}\left(\tilde{x}\right)+c_{4}\left(\tilde{x}\right)U_{\lambda}\left(\tilde{x}\right) \\ & \leq\left[\displaystyle\frac{C_{N,t}}{d(\Omega)^{2t}}+\inf_{\{x\in\Omega\,|\,V_{\lambda}(x)<0\}}c_{3}(x)+m^{2t}\right]V_{\lambda}\left(\tilde{x}\right)+c_{4}\left(\tilde{x}\right)U_{\lambda}\left(\bar{x}\right) \\ & \leq\left[\frac{C_{N,t}}{d(\Omega)^{2t}}+\inf_{\{x\in\Omega\,|\,V_{\lambda}(x)<0\}}c_{3}(x)+m^{2t}\right]V_{\lambda}\left(\tilde{x}\right)+c_{4}\left(\tilde{x}\right)o(1) \delta_{\bar{x}} \\ & <0, \end{align*}

which contradicts (2.3). Thus, equation (2.6) holds.

Subsequently, if there is some point $\bar {x}\in \Omega$ such that

\[ U_{\lambda}\left(\bar{x}\right)=0 \quad \text{or}\quad V_{\lambda}\left(\bar{x}\right)=0. \]

Without loss of generality, we assume

(2.13)\begin{equation} U_{\lambda}\left(\bar{x}\right)=0. \end{equation}

Otherwise, the same arguments as follows can also yield a contradiction for the case that $V_{\lambda }(\bar {x})=0$.

Now we must have

(2.14)\begin{equation} U_{\lambda}=V_{\lambda}=0 \quad \text{almost everywhere in} \ \mathbb{R}^{N}. \end{equation}

If not, we can directly calculate that

(2.15)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U_{\lambda}(\bar{x}) \nonumber\\ & \quad = c_{N,s} m^{{{N}/{2}+s}} P.V. \int_{\mathbb{R}^{N}} \frac{U_{\lambda}(\bar{x})-U_{\lambda}(y)}{|\bar{x}-y|^{{{N}/{2}+s}}} K_{{{N}/{2}+s}}(m|\bar{x}-y|)\,\textrm{d}y+m^{2s}U_{\lambda}(\bar{x}) \nonumber\\ & \quad ={-}c_{N, s} m^{{{N}/{2}+s}} P.V. \int_{\mathbb{R}^{N}} \frac{U_{\lambda}(y)}{|\bar{x}-y|^{{{N}/{2}+s}}} K_{{{N}/{2}+s}}(m|\bar{x}-y|)\,\textrm{d}y \nonumber\\ & \quad = c_{N, s} m^{{{N}/{2}+s}} P.V. \int_{\Sigma_{\lambda}}\left(\frac{K_{{{N}/{2}+s}}(m|\bar{x}-{y}^{\lambda}|)}{|\bar{x}-{y}^{\lambda}|^{{{N}/{2}+s}}}- \frac{K_{{{N}/{2}+s}}(m|\bar{x}-y|)} {|\bar{x}-y|^{{{N}/{2}+s}}}\right)U_{\lambda}(y)\,\textrm{d}y\nonumber\\ & \quad \leq 0 . \end{align}

Combining the above inequality with (2.3) and $c_{2}(x)<0$, we derive that

(2.16)\begin{equation} V_{\lambda}(\bar{x}) \leq 0. \end{equation}

By (2.5), we have $V_{\lambda }(\bar {x}) = 0$.

So, we get

\begin{align*} 0& \leq(-\Delta+m^{2})^{s} U_{\lambda}(\bar{x})+c_{1}(\bar{x})U_{\lambda}(\bar{x})+c_{2}(\bar{x})V_{\lambda}(\bar{x}) \\ & =c_{N, s} m^{{{N}/{2}+s}} P.V. \int_{\Sigma_{\lambda}} \left(\frac{K_{{{N}/{2}+s}}(m|x-{y}^{\lambda}|)}{|x-{y}^{\lambda}|^{{{N}/{2}+s}}}-\frac{K_{{{N}/{2}+s}}(m|x-y|)}{|x-y|^{{{N}/{2}+s}}}\right)U_{\lambda}(y)\,\textrm{d}y \\ & \leq 0, \end{align*}

which implies that $U_{\lambda }=0$ almost everywhere in $\Sigma _{\lambda }$ and hence $U_{\lambda }=0$ almost everywhere in $\mathbb {R}^{N}$.

Similarly, we have

\begin{align*} 0& \leq(-\Delta+m^{2})^{t} V_{\lambda}(\bar{x})+c_{3}(\bar{x})V_{\lambda}(\bar{x})+c_{4}(\bar{x})U_{\lambda}(\bar{x}) \\ & =(-\Delta+m^{2})^{t} V_{\lambda}(\bar{x})\\ & =c_{N, t} m^{{N}/{2}+t} P.V. \int_{\Sigma_{\lambda}} \left(\frac{K_{{N}/{2}+t}(m|x-{y}^{\lambda}|)}{|x-{y}^{ \lambda}|^{{N}/{2}+t}}-\frac{K_{{N}/{2}+t}(m|x-y|)}{|x-y|^{{N}/{2}+t}}\right)V_{\lambda}(y)\,\textrm{d}y\\ & \leq 0. \end{align*}

And it turns out that $V_{\lambda }=0$ almost everywhere in $\mathbb {R}^{N}$.

This finishes the proof of proposition 2.1.

Remark 2.2 It is easy to see from the proof of proposition 2.1, the assumptions ‘$U_{\lambda }$, $V_{\lambda }$ are lower semi-continuous on $\bar {\Omega }$’ and ‘$U_{\lambda }\geq 0$ and $V_{\lambda }\geq 0$, in $\Sigma \setminus \Omega$’ can be weakened into: ‘if $U_{\lambda }<0$ somewhere in $\Sigma _{\lambda }$, then the minimum negative $\inf _{\Sigma _{\lambda }}U_{\lambda }(x)$ is attained in $\Omega$’ and ‘if $V_{\lambda }<0$ somewhere in $\Sigma _{\lambda }$, then the minimum negative $\inf _{\Sigma _{\lambda }}V_{\lambda }(x)$ is attained in $\Omega$’, the same conclusions are still correct.

In order to carry on the direct method of moving planes in $\mathbb {R}^{N}$, we need to the decay at infinity for the solutions to generalized pseudo-relativistic Schrödinger system. We have

Proposition 2.3 Decay at infinity

Let $\Omega$ be an unbounded open set in $\Sigma _{\lambda }$. Assume $U_{\lambda }\in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )$, $V_{\lambda }\in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )$, $(U_{\lambda },V_{\lambda })$ is a pair of solutions to

(2.17)\begin{equation} \begin{cases}{(-\Delta+m^{2})^{s} U_{\lambda}(x)+c_{1}(x)U_{\lambda}(x)+c_{2}(x)V_{\lambda}(x)\geq 0} & {\text{in } \Omega}, \\ {(-\Delta+m^{2})^{t} V_{\lambda}(x)+c_{3}(x)V_{\lambda}(x)+c_{4}(x)U_{\lambda}(x)\geq 0} & {\text{in } \Omega}, \\ {U_{\lambda}(x) \geq 0, V_{\lambda}(x) \geq 0} & {\text {in } \Sigma_{\lambda} \backslash \Omega}, \\ {U_{\lambda}(x^{\lambda})={-}U_{\lambda}(x), V_{\lambda}(x^{\lambda})={-}V_{\lambda}(x)} & {\text{in } \Sigma_{\lambda},} \end{cases} \end{equation}

with

(2.18)\begin{equation} \liminf_{\substack{x\in\Sigma_{\lambda} \\ |x| \rightarrow+\infty}}\left(c_{1}(x)+m^{2s}\right)> 0,\quad \liminf_{\substack{x\in\Sigma_{\lambda} \\ |x| \rightarrow+\infty}} \left(c_{3}(x)+m^{2t}\right)> 0, \end{equation}

where $c_{i}(x)\ (i=1,2,3,4)$ have lower bounds, $c_{2}(x)<0$, $c_{4}(x)<0$ and

\[ \min_{\bar{\Omega}}(c_{1}(x)+m^{2s}) \min_{\bar{\Omega}}(c_{3}(x)+m^{2t})\geq \min_{\bar{\Omega}}c_{2}(x)\min_{\bar{\Omega}}c_{4}(x). \]

If there exists a constant $R_{0}>0$ (depending only on $c_{1}(x)$, $c_{2}(x)$, $c_{3}(x)$, $c_{4}(x)$, $m$, $N$, $s$ and $t$, but independent of $U_{\lambda }$, $V_{\lambda }$ and $\Sigma _{\lambda }$) such that, if $\hat {x}\in \Omega$ satisfying

\[ U_{\lambda}\left(\hat{x}\right)=\min _{\bar{\Omega}} U_{\lambda}(x)<0\quad \text{or}\quad V_{\lambda}\left(\hat{x}\right)=\min _{\bar{\Omega}} V_{\lambda}(x)<0, \]

then $\left |\hat {x}\right |\leq R_{0}$.

Proof. Without loss of generalities, we may assume that, for some $\lambda \leq 0$,

\[ T_{\lambda}=\{x\in\mathbb{R}^{N}\,|\,x_{1}=\lambda\} \quad \text{and} \quad \Sigma_{\lambda}=\{x\in\mathbb{R}^{N}\,|\,x_{1}<\lambda\}. \]

Note that $U_{\lambda }\in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )$, $V_{\lambda }\in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\Omega )$ and $\hat {x}\in \Omega$ satisfying

\[ U_{\lambda}\left(\hat{x}\right)=\min _{\bar{\Omega}} U_{\lambda}(x)<0 \quad \hbox{or}\quad V_{\lambda}\left(\hat{x}\right)=\min _{\bar{\Omega}} V_{\lambda}(x)<0. \]

We assume that

\[ U_{\lambda}\left(\hat{x}\right)=\min _{\bar{\Omega}} U_{\lambda}(x)<0. \]

Otherwise, the same arguments as follows can also yield a contradiction for the case that $V_{\lambda }(\hat {x})=\min _{\bar {\Omega }} V_{\lambda }(x)<0$.

By using similar arguments as in the proof of (2.7), we have

(2.19)\begin{align} (-\Delta+m^{2})^{s} U_{\lambda}\left(\hat{x}\right) & \leq c_{N,s}m^{{N}/{2}+s}\int_{\Sigma_{\lambda}}\frac{2U_{\lambda}\left(\hat{x}\right)}{\left|\hat{x}-{y}^{ \lambda}\right|^{{{N}/{2}+s}}}K_{{{N}/{2}+s}}\left(m\left|\hat{x}-{y}^{\lambda}\right|\right) \textrm{d}y\notag\\ & \quad +m^{2s}U_{\lambda}\left(\hat{x}\right). \end{align}

Note that $\lambda \leq 0$ and $\hat {x}\in \Omega$, it follows that

\[ B_{\left|\hat{x}\right|}\left(\bar{x}\right) \subset \left\{x \in \mathbb{R}^{N} | x_{1}>\lambda\right\}, \]

where $\bar {x}:=(2\left |\hat {x}\right |+(\hat {x})_{1},(\hat {x})^{\prime })$, $(\hat {x})^{\prime }=((\hat {x})_{2},\ldots,(\hat {x})_{N})$.

Thus we derive that, if $|\hat {x}|\geq {R_{\infty }}/{3m}$ ($R_{\infty }$ is same as (1.10)),

(2.20)\begin{align} & m^{{N}/{2}+s}\int_{\Sigma_{\lambda}}\frac{K_{{{N}/{2}+s}}\left(m\left|\hat{x}-{y}^{\lambda}\right|\right)}{\left|\hat{x}-{y}^{\lambda}\right|^{{{N}/{2}+s}}}\,\textrm{d}y\nonumber\\ & \quad \geq m^{{N}/{2}+s}\int_{B_{|\hat{x}|}\left(\bar{x}\right)}\frac{K_{{{N}/{2}+s}}\left(m\left|\hat{x}-y\right|\right)} {\left|\hat{x}-y\right|^{{{N}/{2}+s}}} \,\textrm{d}y\nonumber\\ & \quad \geq m^{{N}/{2}+s}\int_{B_{\left|\hat{x}\right|}\left(\bar{x}\right)}\frac{K_{{{N}/{2}+s}}\left(3m\left|\hat{x}\right|\right)} {3^{{{N}/{2}+s}}\left|\hat{x}\right|^{{{N}/{2}+s}}} \,\textrm{d}y \nonumber\\ & \quad \geq\frac{c_{\infty}m^{({N-1})/{2}+s}\omega_N}{3^{({N+1})/{2}+s}\left|\hat{x}\right|^{s+1/2-{N}/{2}}\textrm{e}^{3m\left|\hat{x}\right|}}, \end{align}

where $\omega _{N}:=|B_{1}(0)|$ denotes the volume of the unit ball in $\mathbb {R}^{N}$. Then we can deduce from (2.17), (2.19), and (2.20) that

(2.21)\begin{align} 0& \leq(-\Delta+m^{2})^{s} U_{\lambda}(\hat{x})+c_{1}(\hat{x})U_{\lambda}(\hat{x})+c_{2}(\hat{x})V_{\lambda}(\hat{x}) \nonumber\\ & \leq\left[\frac{2c_{N,s}c_{\infty}\omega_{N}m^{({N-1})/{2}+s}}{3^{({N+1})/{2}+s} \left|\hat{x}\right|^{s+1/2-{N}/{2}}\textrm{e}^{3m\left|\hat{x}\right|}} +c_{1}(\hat{x})+m^{2s}\right]U_{\lambda}(\hat{x})+c_{2}(\hat{x})V_{\lambda}(\hat{x})\nonumber\\ & =:\left[A_{1}(\hat{x})+c_{1}(\hat{x})+m^{2s}\right]U_{\lambda}(\hat{x})+c_{2}(\hat{x})V_{\lambda}(\hat{x}), \end{align}

where $A_{1}(\hat {x})=({2c_{N,s}c_{\infty }\omega _{N}m^{({N-1})/{2}+s}})/({3^{({N+1})/{2}+s} \left |\hat {x}\right |^{s+1/2-{N}/{2}}\textrm {e}^{3m\left |\hat {x}\right |}})$. From (2.18), we infer that there exists a $R_{1}$ sufficiently large such that, for any $|x|>R_{1}$,

(2.22)\begin{equation} |x|^{s+{1}/{2}-{N}/{2}}\textrm{e}^{3m|x|}\left(c_{1}(x)+m^{2s}\right)>0>{-} \frac{2c_{N,s}c_{\infty}\omega_{N}m^{({N-1})/{2}+s}}{3^{({N+1})/{2}+s}}, \end{equation}

and

(2.23)\begin{equation} |x|^{t+{1}/{2}-{N}/{2}}\textrm{e}^{3m|x|}\left(c_{3}(x)+m^{2t}\right)>0>{-}\frac{2c_{N,t}c_{\infty} \omega_{N}m^{({N-1})/{2}+t}}{3^{({N+1})/{2}+t}}. \end{equation}

Therefore, take $R_{0}:=\max \{{R_{\infty }}/{3m},R_{1}\}$, if $|\hat {x}|>R_{0}$, both (2.21), (2.22), and (2.23) are established. Combine with (2.21), we get

(2.24)\begin{equation} U_{\lambda}(\hat{x})\geq{-}\frac{c_{2}(\hat{x})V_{\lambda}(\hat{x})}{\left[A_{1}+c_{1}(\hat{x})+m^{2s}\right]}. \end{equation}

Next, we will prove that $|\hat {x}|\leq R_{0}$.

If not, then it follows from (2.21), (2.22), and $c_{2}(x)<0$ that

(2.25)\begin{equation} V_{\lambda}\left(\hat{x}\right)<0. \end{equation}

Using the lower semi-continuous of $V_{\lambda }$ that there exists some $\bar {x}$ such that

\[ V_{\lambda}\left(\bar{x}\right)=\min_{\bar{\Omega}} V_{\lambda}<0. \]

Through a computation similar to (2.7), we have

\begin{align*} & (-\Delta+m^{2})^{t} V_{\lambda}\left(\bar{x}\right) \\ & \quad =c_{N,t}m^{{N}/{2}+t}P.V.\int_{\mathbb{R}^{N}}\frac{V_{\lambda}\left(\bar{x}\right) -V_{\lambda}(y)}{\left|\bar{x}-y\right|^{{N}/{2}+t}}K_{{N}/{2}+t}\left(m\left|\bar{x}-y\right|\right) \textrm{d}y+m^{2t}V_{\lambda}\left(\bar{x}\right) \\ & \quad =c_{N,t}m^{{N}/{2}+t}P.V.\left\{\int_{\Sigma_\lambda} \frac{V_{\lambda}\left(\bar{x}\right)- V_{\lambda}(y)}{\left|\bar{x}-y\right|^{{N}/{2}+t}}K_{{N}/{2}+t}\left(m\left|\bar{x}-y\right|\right) \textrm{d}y\right. \\ & \qquad +\left.\int_{\Sigma_\lambda}\frac{V_{\lambda}\left(\bar{x}\right)+V_{\lambda}(y)}{ \left|\bar{x}-{y}^{\lambda}\right|^{{N}/{2}+t}}K_{{N}/{2}+t}\left(m\left|\bar{x}-{y}^{\lambda}\right|\right) \textrm{d}y \right\}+m^{2t}V_{\lambda}\left(\bar{x}\right) \\ & \quad \leq c_{N,t}m^{{N}/{2}+t} \int_{\Sigma_\lambda}\left\{\frac{V_{\lambda}\left(\bar{x}\right)-V_{\lambda}(y)}{\left|\bar{x}-{y}^{\lambda} \right|^{{N}/{2}+t}}+\frac{V_{\lambda}\left(\bar{x}\right)+V_{\lambda}(y)}{\left|\bar{x}-{y}^{\lambda}\right|^{ {N}/{2}+t}}\right\}\\ & \qquad \times K_{{N}/{2}+t}\left(m\left|\bar{x}-{y}^{\lambda}\right|\right)\textrm{d}y +m^{2t}V_{\lambda}\left(\bar{x}\right) \\ & \quad =c_{N,t}m^{{N}/{2}+s}\int_{\Sigma_\lambda}\frac{2V_{\lambda}\left(\bar{x}\right)}{\left|\bar{x}-{y}^{\lambda} \right|^{{N}/{2}+t}}K_{{N}/{2}+t}\left(m\left|\bar{x}-{y}^{\lambda}\right|\right)\textrm{d}y +m^{2t}V_{\lambda}\left(\bar{x}\right)\\ & \quad \leq m^{2t}V_{\lambda}\left(\bar{x}\right). \end{align*}

Similar to (2.20), (2.21), combine with $\min _{\bar {\Omega }}(c_{1}(x)+m^{2s}) \min _{\bar {\Omega }}(c_{3}(x)+m^{2t})\geq \min _{\bar {\Omega }}c_{2}(x)\min _{\bar {\Omega }}c_{4}(x)$. We have, if $|\hat {x}|>R_{0}$,

(2.26)\begin{align} 0& \leq(-\Delta+m^{2})^{t} V_{\lambda}(\bar{x})+c_{3}(\bar{x})V_{\lambda}(\bar{x})+c_{4}(\bar{x})U_{\lambda}(\bar{x})\nonumber\\ & \leq\left[c_{3}(\bar{x})+m^{2t}\right]V_{\lambda}(\bar{x})+c_{4}(\bar{x})U_{\lambda}(\hat{x})\nonumber\\ & \leq \left[c_{3}(\bar{x})+m^{2t}-\frac{c_{2}(\hat{x})c_{4}(\bar{x})}{A_{1}(\bar{x})+c_{1}(\hat{x})+m^{2s}}\right]V_{\lambda}(\bar{x}) \nonumber\\ & \leq \left[\frac{(A_{1}(\bar{x})+c_{1}(\hat{x})+m^{2s})(c_{3}(\bar{x})+m^{2t})-c_{2}(\hat{x})c_{4}(\bar{x})}{A_{1}(\bar{x})+c_{1}(\hat{x})+m^{2s}}\right]V_{\lambda}(\bar{x}) \nonumber\\ & < 0. \end{align}

That is impossible. So we have $|\hat {x}|\leq R_{0}$.

This completes the proof of proposition 2.3.

Remark 2.4 We point out that in both propositions 2.1 and 2.3, we actually only need $(-\Delta +m^{2})^{s} U_{\lambda }(x)+c_{1}(x)U_{\lambda }(x)+c_{2}(x)V_{\lambda }(x)\geq 0$ and $(-\Delta +m^{2})^{t} V_{\lambda }(x)+c_{3}(x)V_{\lambda }(x)+c_{4}(x)U_{\lambda }(x)\geq 0$ holds for $x\in \{x\in \Omega |U_{\lambda }(x)<0\}$ or $x\in \{x\in \Omega |V_{\lambda }(x)<0\}$, not for all $x\in \Omega$; For simplicity and convenience, we assume it hold for all $x\in \Omega$.

3. Proof of theorem 1.1

In this section, we will use the direct method of moving plane to prove the strictly monotonicity and symmetry of nonnegative solutions to generalized pseudo-relativistic Schrödinger system (1.19) in a bounded domain.

Choose an arbitrary direction to be the $x_{1}$-direction and keep the notations $T_{\lambda }$, $\Sigma _{\lambda }$, $x_{\lambda }$, $U_{\lambda }$ and $V_{\lambda }$ defined in § 2.

Proof. Since $\Omega$ is bounded, we may assume that $\Omega \subset \{|x_1|\leq 1\}$ and $\partial \Omega \cap \{x_1=-1\}\neq \emptyset$. Our goal is to show that $U_{\lambda }>0$ and $V_{\lambda }>0$ in $\Sigma _{\lambda }\cap \Omega$ for any $\lambda \in (-1,0)$.

We carry out the direct method of moving planes procedure in two steps.

Step 1. We shall show that there exists $\epsilon >0$ small enough such that, for any $-1<\lambda \leq -1+\epsilon$,

(3.1)\begin{equation} U_\lambda(x)\geq 0,\ V_\lambda(x)\geq 0, \quad \forall \ x\in\Sigma_\lambda\cap\Omega. \end{equation}

Suppose (3.1) is not true, there exists a sequence $\{\lambda _{k}\}\subset (-1,0)$ satisfying $\lambda _k\rightarrow -1$ as $k\rightarrow +\infty$ such that

(3.2)\begin{equation} \inf_{\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}=\inf_{\lambda\in({-}1,\lambda_{k}]}\inf_{x\in\Sigma_\lambda}U_\lambda(x)<0. \end{equation}

Otherwise, the same arguments as follows can also yield a contradiction for the case that

\[ \inf_{\Sigma_{\lambda_{k}}\cap\Omega}V_{\lambda_{k}}=\inf_{\lambda\in({-}1,\lambda_{k}]}\inf_{x\in\Sigma_\lambda}V_\lambda(x)<0. \]

Consequently, there exists $x^{k}\in \Sigma _{\lambda _{k}}\cap \Omega$ such that

(3.3)\begin{equation} w_{\lambda_{k}}(x^{k})=\inf_{\Sigma_{\lambda_{k}}\cap\Omega}w_{\lambda_{k}}=\inf_{\Sigma_{\lambda_{k}}}w_{\lambda_{k}}<0. \end{equation}

It follows directly from (3.2) and (3.3) that ${\partial U_{\lambda }}/{\partial \lambda }|_{\lambda =\lambda _k}(x^{k})\leq 0$, and hence $(\partial _{x_1}u)[(x^{k})^{\lambda _k}]\leq 0$. Note that $x^{k}$ is the interior minimum of $U_{\lambda _k}(x)$, then one has $\nabla _{x}U_{\lambda _k}(x^{k})=0$, i.e.,

(3.4)\begin{equation} (\nabla_{x}u_{\lambda_k})(x^{k})=(\nabla_{x}u)(x^{k}). \end{equation}

By the assumption (1.13) in theorem 1.1, we have

(3.5)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U_{\lambda_{k}}(x^{k})\nonumber\\ & \quad =\left(-\Delta+m^{2}\right)^{s}u_{\lambda_k}(x^{k})-\left(-\Delta+m^{2}\right)^{s}u(x^{k})\nonumber\\ & \quad=f\left((x^{k})^{\lambda_k},u_{\lambda_k}(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_{x}u)((x^{k})^{\lambda_k})\right) =f\left(x^{k},u(x^{k}),v(x^{k}),(\nabla_{x}u)(x^{k})\right)\nonumber\\ & \quad \geq f\left(x^{k},u_{\lambda_k}(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_{x}u)(x^{k})\right)- f\left(x^{k},u(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_{x}u)(x^{k})\right)\nonumber\\ & \qquad+f\left(x^{k},u(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_{x}u)(x^{k})\right)-f \left(x^{k},u(x^{k}),v(x^{k}),(\nabla_{x}u)(x^{k})\right)\nonumber\\ & \quad =:-c_{1}(x^{k})U_{\lambda_k}(x^{k})-c_{2}(x^{k})V_{\lambda_k}(x^{k}), \end{align}

where

(3.6)\begin{equation} c_{1}(x^{k}):={-}\frac{f(x^{k},u_{\lambda_k}(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_xu)(x^{k}))-f(x^{k},u(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_xu)(x^{k}))}{u_{\lambda_k}(x^{k})-u(x^{k})} \end{equation}

and

(3.7)\begin{equation} c_{2}(x^{k}):={-}\frac{f\left(x^{k},u(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_xu)(x^{k})\right)-f\left(x^{k},u(x^{k}),v(x^{k}),(\nabla_xu)(x^{k})\right)}{v_{\lambda_k}(x^{k})-v(x^{k})}. \end{equation}

Similarly, we can get $(-\Delta +m^{2})^{s}U_{\lambda _{k}}(x^{k})=c_{3}(x^{k})V_{\lambda _k}(x^{k})+c_{4}(x^{k})U_{\lambda _k}(x^{k})$, where

(3.8)\begin{equation} c_{3}(x^{k}):={-}\frac{g\left(x^{k},u_{\lambda_k}(x^{k}),v_{\lambda_k}(x^{k}),(\nabla_xv)(x^{k})\right)-g\left(x^{k},u_{\lambda_k}(x^{k}),v(x^{k}),(\nabla_xv)(x^{k})\right)}{v_{\lambda_k}(x^{k})-v(x^{k})} \end{equation}

and

(3.9)\begin{align} c_{4}(x^{k}):={-}\frac{g\left(x^{k},u_{\lambda_k}(x^{k}),v(x^{k}),(\nabla_xv)(x^{k})\right)-g\left(x^{k},u(x^{k}),v(x^{k}),(\nabla_xu)(x^{k})\right)}{u_{\lambda_k}(x^{k})-u(x^{k})}. \end{align}

Since $u,v \in C(\mathbb {R}^{N})$ with compact supports and $f(x, u, v, \mathbf {p})$, $g(x, u, v, \mathbf {q})$ are Lipschitz continuous about $u,v$, so $c_{i}(x) (i=1,2,3,4)$ are uniformly bounded independent of $k$. By using the assumption $\mathbf {F_{1}}$, we know $c_{i}(x)$ satisfy the condition of proposition 2.1.

Note that $\Sigma _{\lambda _{k}}\cap \Omega$ is a narrow region for $k$ large enough and $U_{\lambda },V_{\lambda } \not \equiv 0$. From (3.5)–(3.9) and the Narrow region principle proposition 2.1, one can derive that, for $k$ sufficiently large,

(3.10)\begin{equation} U_{\lambda_{k}}>0,\ V_{\lambda_{k}}>0,\quad \text{in} \ \Sigma_{\lambda_{k}}\cap\Omega, \end{equation}

which yields a contradiction with (3.2). Hence there exists an $\epsilon >0$ small enough such that, (3.1) holds for any $-1<\lambda \leq -1+\epsilon$.

Furthermore, suppose there exist $\hat {\lambda }\in (-1,-1+\epsilon ]$ and $\hat {x}\in \Sigma _{\hat {\lambda }}\cap \Omega$ such that

\[U_{\hat{\lambda}}(\hat{x})=0\quad \text{or}\quad V_{\hat{\lambda}}(\hat{x})=0. \]

For simplify, here we assume that $U_{\hat {\lambda }}(\hat {x})=0$. On the one hand, similar to (3.5), we can infer from the assumption (1.13) in theorem 1.1 that

(3.11)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U_{\hat{\lambda}}(\hat{x})\nonumber\\ & \quad =\left(-\Delta+m^{2}\right)^{s}u_{\hat{\lambda}}(\hat{x})-\left(-\Delta+m^{2}\right)^{s}u(\hat{x})\nonumber\\ & \quad =f\left((\hat{x})^{\hat{\lambda}},u_{\hat{\lambda}}(\hat{x}),v_{\hat{\lambda}}(\hat{x}),(\nabla_{x}u)((\hat{x})^{\hat{\lambda}})\right) -f\left(\hat{x},u(\hat{x}),v(\hat{x}),(\nabla_{x}u)(\hat{x})\right)\nonumber\\ & \quad \geq f\left(\hat{x},u_{\hat{\lambda}}(\hat{x}),v_{\hat{\lambda}}(\hat{x}),(\nabla_{x}u)(\hat{x})\right) -f\left(\hat{x},u(\hat{x}),v(\hat{x}),(\nabla_{x}u)(\hat{x})\right)\nonumber\\ & \quad ={-}c_{2}(\hat{x})V_{\hat{\lambda}}(\hat{x}). \end{align}

On the other hand, we can deduce from (2.15) that

(3.12)\begin{equation} \left(-\Delta+m^{2}\right)^{s}U_{\hat{\lambda}}(\hat{x})\leq 0. \end{equation}

Then it follows from (3.11), (3.12) and $c_{2}(x)>0$ that $V_{\hat {\lambda }}(\hat {x})=0$. Combing (3.11) and the definition of $(-\Delta +m^{2})^{s}$, we can derive that $U_{\lambda }=V_{\lambda }=0$ a.e. in $\mathbb {R}^{N}$. Since $\Omega$ is convex in $x_1$-direction and symmetric about $T_{0}$, $u,v$ satisfy system (1.1), one has $U_{\lambda }\geq 0$, $V_{\lambda }\geq 0$ in $\Sigma _{\lambda }\setminus \Omega$, $U_{\lambda }\not \equiv 0$ or $V_{\lambda }\not \equiv 0$ in $\Sigma _{\lambda }\setminus \Omega$ for any $\lambda \in (-\infty,0)$. That is a contradiction. Therefore, we have, for any $-1<\lambda \leq -1+\epsilon$,

(3.13)\begin{equation} U_\lambda(x)>0,\ V_\lambda(x)>0, \quad \forall \ x\in\Sigma_\lambda\cap\Omega. \end{equation}

Step 2. Step 1 provides a starting point, from which we can now move the plane $T_\lambda$, as long as (3.13) holds, to its limiting position. To this end, let us define

(3.14)\begin{equation} \lambda_{0}:=\sup\left\{\lambda\in\mathbb{R} \mid U_{\mu}> 0\ \textrm{and}\ V_{\mu}> 0,\ \text{in} \ \Sigma_{\mu}\cap\Omega, \ \forall - 1<\mu<\lambda\right\}. \end{equation}

Our goal is to show that $\lambda _{0}=0$.

To prove it, we use the contraction and suppose $\lambda _{0}<0$, then we can infer from (1.1) that $U_{\lambda _{0}}>0$, $U_{\lambda _{0}}>0$ in $(\Omega ^{\lambda _{0}}\setminus \Omega )\cap \Sigma _{\lambda _{0}}$ ($A^{\lambda }$ denotes the reflection of a set $A$ about $T_{\lambda }$), then we must have

(3.15)\begin{equation} U_{\lambda_{0}}(x)>0,\ V_{\lambda_{0}}(x)>0, \quad \forall \ x\in\Sigma_{\lambda_{0}}\cap\Omega. \end{equation}

Otherwise, we assume that there exists $x_{0}\in \Sigma _{\lambda _{0}}\cap \Omega$ such that $U_{\lambda _{0}}(x_{0})=0$ (the same arguments as follows can also yield a contradiction for the case that $V_{\lambda _{0}}(x_{0})=0$). Then from step 1, we have

(3.16)\begin{equation} \left(-\Delta+m^{2}\right)^{s}U_{\lambda_{0}}(x)\geq 0, \quad \forall \ x\in\Sigma_{\lambda_{0}}\cap\Omega. \end{equation}

By the definition of $(-\Delta +m^{2})^{s}$, we have

(3.17)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U_{\lambda_{0}}(x) \nonumber\\ & \quad = c_{N, s} m^{{{N}/{2}+s}} P.V.\int_{\Sigma_{\lambda}}\left(\frac{K_{{{N}/{2}+s}}(m|\bar{x}-{y}^{\lambda}|)}{|\bar{x}-{y}^{\lambda}|^{{{N}/{2}+s}}}-\frac{K_{{{N}/{2}+s}}(m|\bar{x}-y|)} {|\bar{x}-y|^{{{N}/{2}+s}}}\right)\notag\\ & \qquad \times U_{\lambda}(y)\,\textrm{d}y<0. \end{align}

This contradicts (3.16). Hence (3.15) must hold.

Thanks to (3.15), so there exists a compact subset $K\subset \subset \Omega ^{\lambda _{0}}\cap \Sigma _{\lambda _{0}}$ and a constant $c>0$ such that

(3.18)\begin{equation} U_{\lambda_{0}}(x)\geq c>0,\ V_{\lambda_{0}}(x)\geq c>0, \quad \forall \ x\in K\cap\Omega, \end{equation}

and $(\Sigma _{\lambda _{0}}\cap \Omega )\setminus (K\cap \Omega )$ is a narrow region. Since the continuity of $w_{\lambda }$ about $\lambda$, then there exists a sufficiently small $0<\epsilon <\min \{-\lambda _{0},\lambda _{0}+1\}$ such that, for any $\lambda \in [\lambda _{0},\lambda _{0}+\epsilon ]$,

(3.19)\begin{equation} U_{\lambda}(x)>0,\ V_{\lambda}(x)>0, \quad \forall \ x\in K\cap\Omega, \end{equation}

and $(\Sigma _{\lambda _{0}+\epsilon }\cap \Omega )\setminus (K\cap \Omega )$ is also a narrow region.

Next, we will show that

(3.20)\begin{equation} U_{\lambda}(x)\geq 0,\ V_{\lambda}(x)\geq 0, \quad \forall \ x\in(\Sigma_{\lambda}\cap\Omega)\setminus(K\cap\Omega). \end{equation}

If (3.20) is not valid, we may assume that there exists a $\tilde {\lambda }\in (\lambda _{0},\lambda _{0}+\epsilon ]$ (depending on $\epsilon$) such that

(3.21)\begin{equation} \inf_{(\Sigma_{\tilde{\lambda}}\cap\Omega)\setminus(K\cap\Omega)}U_{\tilde{\lambda}}=\inf_{\Sigma_{\tilde{\lambda}}\cap\Omega}U_{\tilde{\lambda}} =\inf_{\lambda\in(\lambda_{0},\tilde{\lambda}]}\inf_{x\in\Sigma_\lambda}U_\lambda(x)<0. \end{equation}

Consequently, there exists $\tilde {x}\in (\Sigma _{\tilde {\lambda }}\cap \Omega )\setminus (K\cap \Omega )$ such that

(3.22)\begin{equation} U_{\tilde{\lambda}}(\tilde{x})=\inf_{(\Sigma_{\tilde{\lambda}}\cap\Omega)\setminus(K\cap\Omega)}U_{\tilde{\lambda}} =\inf_{\Sigma_{\tilde{\lambda}}\cap\Omega}U_{\tilde{\lambda}}=\inf_{\Sigma_{\tilde{\lambda}}}U_{\tilde{\lambda}}<0. \end{equation}

Then we can infer from (3.5)–(3.9) and Narrow region principle proposition 2.1 that

(3.23)\begin{equation} U_{\tilde{\lambda}}>0,\quad V_{\tilde{\lambda}}>0 \quad \text{in} \ (\Sigma_{\tilde{\lambda}}\cap\Omega)\setminus(K\cap\Omega), \end{equation}

which yields a contradiction with (3.21). As a consequence, we have, for any $\lambda \in [\lambda _{0},\lambda _{0}+\epsilon ]$, (3.20) holds. Furthermore, it follows from (3.16) and (3.17) that

(3.24)\begin{equation} U_{\lambda}(x)>0,\ V_{\lambda}(x)>0, \quad \forall \ x\in(\Sigma_{\lambda}\cap\Omega)\setminus(K\cap\Omega), \end{equation}

and hence, for any $\lambda \in [\lambda _{0},\lambda _{0}+\epsilon ]$,

(3.25)\begin{equation} U_{\lambda}(x)>0,\ V_{\lambda}(x)>0, \quad \forall \ x\in\Sigma_{\lambda}\cap\Omega. \end{equation}

This contradicts the definition of $\lambda _{0}$. Thus $\lambda _{0}=0$.

By choosing $(x_1,x')$, $(\bar {x}_1,x')\in \Omega$ with $0>x_1>\bar {x}_1$ and $\lambda =({x_{1}+\bar {x}_{1}})/{2}$, then we get

\[ u(x_{1},x')>u(\bar{x}_{1},x'),\quad v(x_{1},x')>v(\bar{x}_{1},x'), \]

and hence $u(x_{1},x')$ and $v(x_{1},x')$ are strictly increasing in the left half of $\Omega$ in $x_{1}$-direction. Moreover, if $f(x_1, x', u, p_1, p_2,\dots, p_n)= f(-x_{1}, x', u, -p_1, p_2,\dots, p_n)$, then $\hat {u}(x_{1},x'):=u(-x_{1},x')$ and $\hat {v}(x_{1},x'):=v(-x_{1},x')$ also solves system (1.1). Thus we have derived that

(3.26)\begin{equation} \hat{u}(x_1,x')\leq \hat{u}({-}x_1, x'),\ \hat{v}(x_1,x')\leq \hat{v}({-}x_1, x'), \quad \forall \ (x_1,x')\in \Omega, \ x_{1}<0. \end{equation}

Equation (3.26) is equivalent to

(3.27)\begin{equation} u(x_1, x')\geq u({-}x_1, x'),\ v(x_1, x')\geq v({-}x_1, x'), \quad \forall \ (x_1,x')\in \Omega, \ x_{1}<0. \end{equation}

Combining this with $u(x_1, x')\leq u(-x_1, x')$, $u(x_1, x')\leq u(-x_1, x')$ yields that

(3.28)\begin{equation} u(x_1, x')=u({-}x_1, x'),\ v(x_1, x')=v({-}x_1, x'), \quad \forall \ (x_1,x')\in \Omega, \ x_{1}<0, \end{equation}

we see $u$ and $v$ are symmetric in the $x_1$ direction about $\{x\in \mathbb {R}^{N} \mid x_{1}=0\}$. This completes the proof of theorem 1.1.

4. Proof of theorem 1.2

Choose an arbitrary direction to be the $x_{1}$-direction and keep the notations $T_{\lambda }$, $\Sigma _{\lambda }$, $x_{\lambda }$, $U_{\lambda }$ and $V_{\lambda }$ defined in § 2.

We carry out the direct method of moving planes procedure in two steps.

Proof. Step 1. We show that, for sufficiently negative $\lambda$,

(4.1)\begin{equation} U_{\lambda}(x)\geq 0\quad \text{and}\quad V_{\lambda}(x)\geq 0, \quad \forall \ x\in\Sigma_{\lambda}. \end{equation}

If (4.1) does not hold, then

\[ \inf_{\Sigma_{\lambda}}U_{\lambda}<0 \quad \text{or}\quad \inf_{\Sigma_{\lambda}}V_{\lambda}<0. \]

Let's suppose $\inf _{\Sigma _{\lambda }}U_{\lambda }<0$. The same arguments as follows can also yield a contradiction for the case that $\inf _{\Sigma _{\lambda }}V_{\lambda }<0$. Then there exists a sequence $\lambda _k\rightarrow -\infty$ as $k\rightarrow +\infty$ such that

(4.2)\begin{equation} \inf_{x\in\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}(x)=\inf_{\lambda \leq\lambda_k}\inf_{x\in\Sigma_\lambda}U_\lambda(x)<0, \end{equation}

for every $k=1,2,\dots$. Moreover, for every $k=1,2,\dots$, $\inf _{\Sigma _{\lambda _{k}}\cap \Omega }U_{\lambda _{k}}$ is attained at some $x^{k}\in \Sigma _{\lambda _{k}}\cap \Omega$, that is,

(4.3)\begin{equation} U_{\lambda_{k}}(x^{k})=\inf_{\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}=\inf_{\Sigma_{\lambda_{k}}}U_{\lambda_{k}}<0. \end{equation}

Using the assumption (1.13), $F_{1}$, $F_2$, (1.15), (1.16) and the asymptotic property (1.17), we have, for $k$ large enough, at any points $x\in \Sigma _{\lambda _{k}}\cap \Omega$ where $U_{\lambda _{k}}(x)<0$,

(4.4)\begin{equation} |c_{1}(x)|\leq C\left(|u_{\lambda_k}(x)|^{\beta_{1}}+|u(x)|^{\beta_1}\right)\leq 2C\left[u(x)\right]^{\beta_{1}}, \end{equation}

and hence

(4.5)\begin{equation} \liminf_{\substack{x\in\Sigma_{\lambda_{k}}\cap\Omega, U_{\lambda_{k}}<0 \\ |x|\rightarrow+\infty}}c_{1}(x)+m^{2s}\geq{-}2C\cdot \limsup_{\substack{x\in\Sigma_{\lambda_{k}}\cap\Omega, U_{\lambda_{k}}<0 \\ |x|\rightarrow+\infty}}\left[u(x)\right]^{\beta_{1}}+m^{2s}>0. \end{equation}

Similarly, we can infer from the asymptotic property (1.17) that

(4.6)\begin{equation} \liminf_{\substack{x\in\Sigma_{\lambda_{k}}\cap\Omega, V_{\lambda_{k}}<0 \\ |x|\rightarrow+\infty}}c_{4}(x)+m^{2t}\geq{-}2C\cdot \limsup_{\substack{x\in\Sigma_{\lambda_{k}}\cap\Omega, V_{\lambda_{k}}<0 \\ |x|\rightarrow+\infty}}\left[v(x)\right]^{\beta_{2}}+m^{2t}>0. \end{equation}

Combining (4.5), (4.6) and (4.6) the assumption $F_{1}$, $F_2$, we know $c_{i}(x) (i=1,2,3,4)$ satisfy the condition of proposition 2.3. By the Decay at infinity proposition 2.3, it is easy to deduce, and there exists a $R_{0}>0$ such that

(4.7)\begin{equation} |x^{k}|\leq R_{0}, \end{equation}

This will lead to a contradiction provided that $\lambda _{k}\leq -R_{0}$ as $k\rightarrow \infty$. Thus (4.1) holds.

Since $u(x), v(x)\rightarrow 0$ as $|x|\rightarrow +\infty$ and $x\in \Sigma _{0}$, we can actually deduce that $U_{\lambda }\not \equiv 0$ or $V_{\lambda }\not \equiv 0$ in $\Sigma _{\lambda }$ for any $\lambda \leq -R_{0}$ by choosing $R_{0}$ larger. Then, similar to (3.13) in step 1 in the proof of theorem 1.1, we can also get

(4.8)\begin{equation} U_\lambda>0,\ V_\lambda>0, \quad \text{in} \ \Sigma_\lambda\cap\Omega, \quad \forall \ \lambda\leq{-}R_0. \end{equation}

Step 2. Step 1 provides a starting point, from which we can now move the plane $T_\lambda$, as long as (4.8) holds, to its limiting position. To this end, let us define

(4.9)\begin{equation} \lambda_{0}:=\sup\left\{\lambda\in(-\infty,0] \mid U_{\mu}> 0\ \text{and} \ V_{\mu}> 0,\ \text{in} \ \Sigma_{\mu}, \ \forall \ \mu \leq \lambda\right\}. \end{equation}

It follows from step 1 that $-R_{0}\leq \lambda _{0}\leq 0$. One can easily verify that

(4.10)\begin{equation} U_{\lambda_{0}}(x)\geq 0\quad \text{and}\quad V_{\lambda_{0}}(x)\geq 0 \quad \forall \ x\in\Sigma_{\lambda_{0}}. \end{equation}

Case (i). There exists a line $\mathcal {L}$ parallel to $x_{1}$-axis satisfying $\mathcal {L}\cap \Omega \neq \emptyset$ such that $\mathcal {L}\cap \Omega ^{c}\neq \emptyset$. In this case, we will show that

\[\lambda_0=0. \]

Now suppose on the contrary that $\lambda _{0}<0$. Note that $U_{\lambda _{0}}\not \equiv 0$ or $V_{\lambda _{0}}\not \equiv 0$ in $\Sigma _{\lambda _{0}}$, then from the step 1 in the proof of theorem 1.1, we can derive that

(4.11)\begin{equation} U_{\lambda_0}>0,\ U_{\lambda_0}>0, \quad \text{in} \ \Omega\cap\Sigma_{\lambda_0}. \end{equation}

Next, we will show that the plane $T_{\lambda }$ can be moved a little bit further from $T_{\lambda _{0}}$ to the right. More precisely, we will show that there exists a $\varepsilon >0$, such that for any $\lambda \in [\lambda _{0},\lambda _{0}+\varepsilon ]$, it holds

(4.12)\begin{equation} U_\lambda\geq 0,\ V_\lambda\geq 0, \quad \text{in} \ \Sigma_\lambda, \quad \forall \ \lambda_{0}<\lambda\leq\lambda_0+\varepsilon. \end{equation}

To prove (4.12), we shall use the Narrow region principle proposition 2.1 and the Decay at Infinity proposition 2.3.

Suppose (4.12) is not true, for convenience, we assume that there exists a sequence $\{\lambda _{k}\}\subset (\lambda _{0},0)$ satisfying $\lambda _k\rightarrow \lambda _{0}$ as $k\rightarrow +\infty$ such that

(4.13)\begin{equation} \inf_{x\in\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}(x)=\inf_{\lambda \leq\lambda_k}\inf_{x\in\Sigma_\lambda}U_\lambda(x)<0 \end{equation}

Moreover, for every $k=1,2,\dots$, $\inf _{\Sigma _{\lambda _{k}}\cap \Omega }w_{\lambda _{k}}$ can be attained at some $x^{k}\in \Sigma _{\lambda _{k}}\cap \Omega$, that is,

(4.14)\begin{equation} U_{\lambda_{k}}(x^{k})=\inf_{\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}=\inf_{\Sigma_{\lambda_{k}}}U_{\lambda_{k}}<0. \end{equation}

We can deduce the proof of proposition 2.3 that $V_{\lambda _{k}}(x^{k})<0$, and hence there exists some $x^{j}\in \Sigma _{\lambda _{k}}\cap \Omega$ such that $V_{\lambda _{k}}(x^{j})=\inf _{\Sigma _{\lambda _{k}}\cap \Omega }V_{\lambda _{k}}=\inf _{\Sigma _{\lambda _{k}}}V_{\lambda _{k}}<0$. Through direct calculation, we get

(4.15)\begin{equation} \left\{\begin{aligned} (-\Delta+m^{2})^{s} U_{\lambda_{k}}(x^{k})+c_{1}(x)U_{\lambda_{k}}(x)+c_{2}(x)V_{\lambda_{k}}(x)=0,\\ (-\Delta+m^{2})^{t} V_{\lambda_{k}}(x^{k})+c_{3}(x)V_{\lambda_{k}}(x)+c_{4}(x)U_{\lambda_{k}}(x)=0, \end{aligned}\right. \end{equation}

where $c_i(x)$ is the same as (3.6)–(3.9). Now we shall show that $\{x^{k}\}$ is bounded.

If $\{x^{k}\}$ is not bounded, then up to a subsequence (still denote by $\{x^{k}\}$), $|x^{k}|\rightarrow +\infty$ as $k\rightarrow +\infty$. By the assumption (1.15) and (1.16), we have, for $k$ large enough,

(4.16)\begin{equation} |c_{1}(x^{k})|\leq 2C\left[u(x^{k})\right]^{\beta_{1}},\quad|c_{3}(x^{k})|\leq 2C\left[v(x^{k})\right]^{\beta_{2}}. \end{equation}

From (2.21) and (2.26) in Decay at infinity proposition 2.3, we infer that

(4.17)\begin{align} 0& \leq(-\Delta+m^{2})^{t} V_{\lambda_{k}}(x^{j})+c_{3}(x^{j})V_{\lambda_{k}}(x^{j})+c_{4}(x^{j})U_{\lambda_{k}}(x^{j})\nonumber\\ & \leq\left[\frac{(A_{1}(x^{k})+c_{1}(x^{k})+m^{2s})(c_{3}(x^{j})+m^{2t})-c_{2}(x^{k})c_{4}(x^{j})}{A_{1}(x^{k})+c_{1}(x^{k})+m^{2s}}\right]V_{\lambda_{k}}(x^{j})\nonumber\\ & \leq\left[\frac{(A_{1}(x^{k})-2C\left[u(x^{k})\right]^{\beta_{1}}+m^{2s})({-}2C\left[v(x^{j})\right]^{\beta_{2}}+m^{2t})-c_{2}(x^{k})c_{4}(x^{j})}{A_{1}(x^{k})+c_{1}(x^{k})+m^{2s}}\right]V_{\lambda_{k}}(x^{j})\nonumber\\ & <0, \end{align}

where $A_{1}(x)$ is the same to proposition 2.3, the last equation holds because of the assumption (1.18). This leads to a contradiction. Thus there exists $R_{\ast }>0$ such that $|x^{k}|< R_{\ast }$.

In fact, since $\lambda _{0}<0$, one can infer from (1.1) that $U_{\lambda _{0}}>0$, $V_{\lambda _{0}}>0$ in $(\Omega ^{\lambda _{0}}\setminus \Omega )\cap \Sigma _{\lambda _{0}}$ ($A^{\lambda }$ denotes the reflection of a set $A$ about $T_{\lambda }$), and hence (4.11) yields that $U_{\lambda _{0}}>0$, $V_{\lambda _{0}}>0$ in $\Omega ^{\lambda _{0}}\cap \Sigma _{\lambda _{0}}$. So there exists a compact subset $K\subset \subset \Omega ^{\lambda _{0}}\cap \Sigma _{\lambda _{0}}$ and a constant $c>0$ such that

(4.18)\begin{equation} U_{\lambda_{0}}(x)\geq c>0,\ V_{\lambda_{0}}(x)\geq c>0, \quad \forall \ x\in K\cap\Omega\cap B_{R_{{\ast}}}(0), \end{equation}

and $(\Sigma _{\lambda _{0}}\cap \Omega \cap B_{R_{\ast }}(0))\setminus (K\cap \Omega \cap B_{R_{\ast }}(0))$ is a narrow region. Hence we choose sufficiently small $0<\varepsilon <\min \{-\lambda _{0},\lambda _{0}+1\}$ such that, for any $\lambda \in [\lambda _{0},\lambda _{0}+\varepsilon ]$,

(4.19)\begin{equation} U_{\lambda}(x)>0,\ V_{\lambda}(x)>0, \quad \forall \ x\in K\cap\Omega\cap B_{R_{{\ast}}}(0), \end{equation}

and $(\Sigma _{\lambda _{0}+\varepsilon }\cap \Omega \cap B_{R_{\ast }}(0))\setminus (K\cap \Omega \cap B_{R_{\ast }}(0))$ is also a narrow region. Inequality (4.19) imply $x^{k}\in (\Sigma _{\lambda _{0}+\varepsilon }\cap \Omega \cap B_{R_{\ast }}(0))\setminus (K\cap \Omega \cap B_{R_{\ast }}(0))$. From the Lipschitz continuous of $f,g$ with respect to $u,v$, we have $c_{i}(x^{k})$ are uniformly bounded. Since $(\Sigma _{\lambda _{k}}\cap \Omega \cap B_{R_{\ast }}(0))\setminus (K\cap \Omega \cap B_{R_{\ast }}(0))$ is a narrow region. We can infer from the assumption of $c_{i}(x)$ and Narrow region principle proposition 2.1 that

(4.20)\begin{equation} U_{\lambda_{k}}>0,\ V_{\lambda_{k}}>0 \quad \text{in} \ (\Sigma_{\lambda_{k}}\cap\Omega\cap B_{R_{{\ast}}}(0))\setminus(K\cap\Omega\cap B_{R_{{\ast}}}(0))\quad \text{as}\ k\rightarrow\infty, \end{equation}

which yields a contradiction with (4.14). Thus we have derived (4.12).

From step 1 in the proof of 1.1, we have either $U_{\lambda }>0$, $V_{\lambda }>0$, or $U_{\lambda }=V_{\lambda }\equiv 0$ in $\Omega \cap \Sigma _{\lambda }$. Furthermore, since $U_{\lambda _{0}}>0$ in $\Omega \cap \Sigma _{\lambda _{0}}$, by continuity, choosing $\varepsilon >0$ smaller if necessary, we actually have

(4.21)\begin{equation} U_\lambda>0,\ V_{\lambda}>0 \quad \text{in} \ \Omega\cap\Sigma_\lambda, \quad \forall \ \lambda_{0}\leq\lambda\leq\lambda_0+\varepsilon. \end{equation}

This contradicts the definition of $\lambda _{0}$. Thus $\lambda _{0}=0$. The strict monotonicity follows from $U_\lambda >0$ and $V_\lambda >0$ in $\Sigma _\lambda \cap \Omega$ for any $\lambda <\lambda _0$.

Case (ii). Any line $\mathcal {L}$ parallel to $x_{1}$-axis such that $\mathcal {L}\cap \Omega \neq \emptyset$ must satisfy $\mathcal {L}\cap \Omega ^{c}=\emptyset$. We will show that either $\lambda _{0}=0$ or $\lambda _{0}<0$ and $U_{\lambda _{0}}=V_{\lambda _{0}}\equiv 0$ in $\Sigma _{\lambda _{0}}$.

Assume that $\lambda _{0}<0$ but $U_{\lambda _{0}}\not \equiv 0$ or $V_{\lambda _{0}}\not \equiv 0$ in $\Sigma _{\lambda _{0}}$. Then, similar to (4.11), we can derive that

(4.22)\begin{equation} U_{\lambda_0}>0,\ V_{\lambda_0}>0, \quad \text{in} \ \Omega\cap\Sigma_{\lambda_0}. \end{equation}

Next, similar to (4.21), we can show that, there exists $\varepsilon >0$ small enough such that

(4.23)\begin{equation} U_\lambda>0,\ V_\lambda>0, \quad \text{in} \ \Omega\cap\Sigma_\lambda, \quad \forall \ \lambda_{0}\leq\lambda\leq\lambda_0+\varepsilon, \end{equation}

which contradicts the definition of $\lambda _{0}$. As a consequence, $\lambda _{0}=0$ is correct. This completes the proof of theorem 1.2.

5. Proof of theorem 1.3

In this section, we shall prove theorem 1.3. For this purpose, we first establish a maximum principle for the anti-symmetric system in an unbounded domain.

Lemma 5.1 For any function $\phi \in C^{1,1}(\mathbb {R}^{N})$ with $\|\phi \|_{C^{1,1}(\mathbb {R}^{N})}<+\infty$ and any $r>0$, set $\phi _r(x):=\phi (\frac {x}{r})$. Then,

\[ \left|\left[\left(-\Delta+m^{2}\right)^{s}-m^{2s}\right]\phi_r(x)\right|\leq \frac{C}{r^{2s}}, \quad \forall\ x\in\mathbb{R}^{N}, \]

where $C>0$ is a constant depending on $N$, $s$ and $\|\phi \|_{C^{1,1}(\mathbb {R}^{N})}$, but independent of $r$.

The proof of lemma 5.1, please refer to lemma 2.10 in [Reference Dai, Qin and Wu23].

For simplicity reason, in this section, we will use $\Sigma$ to stand for $\Sigma _\lambda$, use $T$ to stand for $T_\lambda$, and use $(U,V)$ to stand for $(U_\lambda,V_\lambda )$, that is

(5.1)\begin{equation} \left\{\begin{aligned} U(x) & :=u\left(x^{\lambda}\right)-u(x) \\ V(x) & :=v\left(x^{\lambda}\right)-v(x). \end{aligned}\right. \end{equation}

Proposition 5.2 Maximum principles for anti-symmetric system in a unbounded domain

Assume that $U \in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})$, $V \in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{loc}^{1,1}(\mathbb {R}^{N})$ are bounded from above, $U({x^{\lambda }})=-U(x)$ and $V({x^{\lambda }})=-V(x)$ in $\Sigma$, where ${x^{\lambda }}$ is the reflection point of $x$ with respect to $T$. Suppose that

(5.2)\begin{align} \begin{cases}{(-\Delta+m^{2})^{s} U(x)+c_{1}(x)U(x)+c_{2}(x)V(x)\leq 0} & x\in\{x\in \Omega| U(x)>0\} \\ {(-\Delta+m^{2})^{t} V(x)+c_{3}(x)V(x)+c_{4}(x)U(x)\leq 0} & x\in\{x\in \Omega| V(x)>0\}, \end{cases} \end{align}

where $c_{2}(x)<0$, $c_{4}(x)<0$, $c_{1}(x)$ and $c_{3}(x)$ satisfy

(5.3)\begin{equation} \inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)>{-}m^{2s}\quad \text{and}\quad \inf_{\{x\in\Sigma \mid V(x)>0\}}\,c_{3}(x)>{-}m^{2t}, \end{equation}

meanwhile

\[ \left(\inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)+m^{2s}\right)\left(\inf_{\{x\in\Sigma \mid V(x)>0\}}\,c_{3}(x)+m^{2t}\right)>\inf_{\Sigma}c_{2}(x) \cdot\inf_{\Sigma}c_{4}(x). \]

Then, we have

(5.4)\begin{equation} U(x) \leq 0,\quad V(x) \leq 0, \quad \forall\ x \in\Sigma. \end{equation}

Proof. Suppose that (5.4) is false, then we have

\[\sup_{\Sigma} U(x)>0 \quad \text{or}\quad \sup_{\Sigma} V(x)>0. \]

We may assume

\[ \sup_{\Sigma} U(x)>0. \]

The same arguments as follows can also yield a contradiction for the case that $\sup _{\Sigma } V(x)>0$.

Since $U$ is bounded from above, we have

\[ M:=\sup_{\Sigma} U(x)>0. \]

Noting that $\Sigma$ is unbounded domain, hence $\sup _{\Sigma } U(x)$ may not be attained. So, there exists sequences $x^{k}\in \Sigma$ and $0<\beta _k<1$ with $\beta _k\rightarrow 1$ as $k\rightarrow \infty,$ such that

(5.5)\begin{equation} U(x^{k})\geq \beta_k M. \end{equation}

Now we denote

\[T=\{x\in\mathbb{R}^{N}| x_1=0\}, \quad \Sigma=\{x\in\mathbb{R}^{N}| x_1<0\}. \]

Let $d_k:=dist(x^{k}, T)$ and

\[ \psi(x)=\begin{cases}\textrm{e}^{{|x|^{2}}/{|x|^{2}-1}}, & |x|<1\\ 0, & |x|\geq 1.\end{cases} \]

Then we see that $\psi \in C_0^{\infty }(\mathbb {R}^{N})$ and $|(-\Delta +m^{2})^{s}\psi (x)|\leq C$ for all $x \in \mathbb {R}^{N}$. Moreover, $(-\Delta +m^{2})^{s}\psi (x)$ is monotone decreasing with respect to $|x|$ and

\[ \displaystyle\frac{\left(-\Delta+m^{2}\right)^{s}\psi(x)}{|x|^{-({N+1})/{2}-s}\textrm{e}^{-|x|}}\to 1,\quad \hbox{as } |x|\rightarrow +\infty. \]

Set

\[\psi_k(x):=\psi\left(\displaystyle\frac{x-{(x^{k})^{\lambda}}}{d_k}\right) \quad \text{and} \quad \tilde{\psi_k}(x)=\psi_k({x}^{\lambda})=\psi\left(\frac{x-x^{k}}{d_k}\right), \]

where ${x}^{\lambda }=(-x_1, x_2,\ldots, x_N)$ is the anti-symmetry point of $x$ with respect to $x_1$ direction.

Obviously $\tilde {\psi _k}-\psi _k$ is anti-symmetric with respect to $T$. Let $\varepsilon _k=(1-\beta _k)M$, we have

\[ U(x^{k})+\varepsilon_k[\tilde{\psi_k}-\psi_k](x^{k})\geq M. \]

We denote

\[U_k(x):=U(x)+\varepsilon_k[\tilde{\psi_k}-\psi_k](x). \]

Then $U_k$ is also anti-symmetric with respect to $T$.

Noting that for any $x\in \Sigma \setminus B_{d_k}(x^{k})$, $U(x)\leq M$ and $\tilde {\psi _k}(x)=\psi _k(x)=0$, we have

\[ U_k(x^{k})\geq U_k(x), \quad \forall \ x\in\Sigma\setminus B_{d_k}(x^{k}). \]

Hence the supremum of $U_k(x)$ in $\Sigma$ is achieved in $B_{d_k}(x^{k})$. Consequently, there exists a point $\overline {x}^{k}\in B_{d_k}(x^{k})$ such that

(5.6)\begin{equation} U_k(\overline{x}^{k})=\sup_{x\in\Sigma} U_k(x)\geq M. \end{equation}

By the choice of $\varepsilon _k$, it is easy to verify that $U(\bar {x}^{k})\geq \beta _k M>0$.

Next, we will evaluate the upper and lower bound of $(-\Delta +m^{2})^{s}U_k(\bar {x}^{k})$. In fact, we can obtain from (5.2) and lemma 5.1 that

(5.7)\begin{equation} \left(-\Delta+m^{2}\right)^{s}U_{k}(\bar{x}^{k})\leq{-}c_{1}(\bar{x}^{k})U(\bar{x}^{k})-c_{2}(\bar{x}^{k})V(\bar{x}^{k})+C\left(\frac{1}{d_k^{2s}}+1\right)\varepsilon_k. \end{equation}

On the other hand, combine with the following facts

\[ \displaystyle\frac{K_{{N}/{2}+s}(m|\bar{x}^{k}-y|)}{|\bar{x}^{k}-y|^{{N}/{2}+s}}> \frac{K_{{N}/{2}+s}(m|\bar{x}^{k}-{y}^{\lambda}|)}{|\bar{x}^{k}-{y}^{\lambda}|^{{n}/{2}+s}}, \quad \forall \ y\in \Sigma, \]

and

\[ U_k(\bar{x}^{k})\geq U_k(y),\quad \forall \ y\in \Sigma. \]

We can derive the following lower bound

(5.8)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U_k(\bar{x}^{k}) \nonumber\\ & \quad =c_{N,s} m^{{N}/{2}+s} P.V.\int_{\mathbb{R}^{N}}\frac{U_k(\bar{x}^{k})-U_k(y)}{|\bar{x}^{k}-y|^{{N}/{2}+s}} K_{{N}/{2}+s}\left(m|\bar{x}^{k}-y|\right)\textrm{d}y+m^{2 s} U_k(\bar{x}^{k}) \nonumber\\ & \quad =c_{N, s}m^{{N}/{2}+s}P.V. \int_{\Sigma}\left[\frac{U_k(\bar{x}^{k})-U_k(y)}{|\bar{x}^{k}-y|^{{N}/{2}+s}} K_{{N}/{2}+s}(m|\bar{x}^{k}-y|)\right. \nonumber\\ & \qquad \left.+\frac{U_k(\bar{x}^{k})+U_k(y)}{|\bar{x}^{k}-{y}^{\lambda}|^{{N}/{2}+s}}K_{{N}/{2}+s} (m|\bar{x}^{k}-{y}^{\lambda}|)\right]dy+m^{2s}U_k(\bar{x}^{k})\nonumber\\ & \quad =c_{N,s} m^{{N}/{2}+s} P.V.\int_{\Sigma}\left(\frac{K_{{N}/{2}+s}(m|\bar{x}^{k}-y|)}{|\bar{x}^{k}-y|^{{N}/{2}+s}} -\frac{K_{{N}/{2}+s}(m|\bar{x}^{k}-{y}^{\lambda}|)}{|\bar{x}^{k}-{y}^{\lambda}|^{{N}/{2}+s}}\right) \notag\\ & \qquad \times (U_k(\bar{x}^{k})-U_k(y))\,\textrm{d}y \nonumber\\ & \qquad +2c_{N,s}m^{{N}/{2}+s} U_k(\bar{x}^{k})\int_{\Sigma} \frac{K_{{N}/{2}+s}(m|\bar{x}^{k}-{y}^{\lambda}|)}{|\bar{x}^{k}-{y}^{\lambda}|^{{N}/{2}+s}}\,\textrm{d}y+m^{2s} U_k(\bar{x}^{k}) \nonumber\\ & \quad >m^{2s}U_k(\bar{x}^{k}). \end{align}

Combining (5.7) and (5.8), we derive

\begin{align*} m^{2 s}U(\bar{x}^{k})& \leq m^{2 s}U_k(\bar{x}^{k})\leq{-}\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}\, c_{1}(x)\right)U(\bar{x}^{k})-c_{2}(\bar{x}^{k})V(\bar{x}^{k})\\ & \quad +C\left(\frac{1}{d_k^{2s}}+1\right)\varepsilon_k. \end{align*}

Now we choose $k$ large enough such that $\beta _{k}>\frac {1}{2}$, then we have

(5.9)\begin{equation} \left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)\beta_{k}+c_{2}(\bar{x}^{k})\frac{V(\bar{x}^{k})}{M}\leq 2C(1-\beta_k), \end{equation}

which implies

(5.10)\begin{equation} \left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)\beta_{k}+c_{2}(\bar{x}^{k})\frac{V(\bar{x}^{k})}{U(\bar{x}^{k})}\leq 2C(1-\beta_k). \end{equation}

If we take the limit $k\rightarrow +\infty$, apply the fact of $c_{2}<0$ and

\[\inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)>{-}m^{2s},\]

we deduce that

\[ V(\bar{x}^{k})>0. \]

Since $V$ is bounded from above, we have

\[ M_{1}:=\sup_{\Sigma} V(x)>0. \]

Hence, there exists sequences $x^{l}\in \Sigma$ and $0<\beta _l<1$ with $\beta _l\rightarrow 1$ as $l\rightarrow \infty$ such that

(5.11)\begin{equation} U(x^{l})\geq \beta_l M_{1}. \end{equation}

By using the similar arguments as above, we can get the following upper bound

(5.12)\begin{equation} \left(-\Delta+m^{2}\right)^{t}V_{l}(\bar{x}^{l})\leq{-}c_{3}(\bar{x}^{l})V(\bar{x}^{l})-c_{4}(\bar{x}^{l})U(\bar{x}^{l})+C\left(\frac{1}{d_l^{2t}}+1\right)\varepsilon_l. \end{equation}

where

\[ d_l:=dist(x^{l}, T)\quad \text{and}\quad \varepsilon_l=(1-\beta_l)M_{1}, \]

\[ V_l(x):=V(x)+\varepsilon_l[\tilde{\psi_l}-\psi_l](x), \]

and

\[ V_l(\overline{x}^{l})=\sup_{x\in\Sigma} V_l(x)\geq M_{1}. \]

Similar to (5.8), we can also derive the following lower bound

(5.13)\begin{equation} \left(-\Delta+m^{2}\right)^{t}V_l(\bar{x}^{l})>m^{2t}V_l(\bar{x}^{l}). \end{equation}

Combine (5.12) and (5.13), we have

(5.14)\begin{equation} \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}V(\bar{x}^{l})+c_{4}(\bar{x}^{l}){U(\bar{x}^{l})}\leq C\left(\frac{1}{d_l^{2t}}+1\right)\varepsilon_l. \end{equation}

If we take the limit $k\rightarrow +\infty$, by (5.10), we have

(5.15)\begin{equation} U(\bar{x}^{k})\leq{-}\frac{c_{2}(\bar{x}^{k})}{\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)}V(\bar{x}^{k}). \end{equation}

Next, let us choose $l$ large enough such that $\beta _{l}>\frac {1}{2}$, then (5.12) becomes

(5.16)\begin{equation} \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}+c_{4}(\bar{x}^{l})\frac{U(\bar{x}^{l})}{M_{1}}\leq 2C(1-\beta_l). \end{equation}

Let $k,l\rightarrow \infty$ in (5.15) and (5.16), we get

\begin{align*} & \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}M_{1}+c_{4}(\bar{x}^{l}){U(\bar{x}^{l})} \\ & \quad \geq \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}{V(\bar{x}^{l})}+c_{4}(\bar{x}^{l}){U(\bar{x}^{l})} \\ & \quad \geq \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}{V(\bar{x}^{l})}+c_{4}(\bar{x}^{l}){U(\bar{x}^{k})} \\ & \quad \geq \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}{V(\bar{x}^{l})}-\displaystyle\frac{c_{2}(\bar{x}^{k})c_{4}(\bar{x}^{l})}{\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)}V(\bar{x}^{k})\\ & \quad \geq \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}{V(\bar{x}^{l})}-\frac{c_{2}(\bar{x}^{k})c_{4}(\bar{x}^{l})}{\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)}V(\bar{x}^{l}). \end{align*}

Hence,

\begin{align*} & \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right)\beta_{l}{V(\bar{x}^{l})}-\displaystyle\frac{c_{2}(\bar{x}^{k})c_{4}(\bar{x}^{l})}{\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)}V(\bar{x}^{l})\\ & \quad \leq 2C(1-\beta_l)M_{1}. \end{align*}

Taking the limit $k,l \rightarrow +\infty$, we have

\[ 0< \left(\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{3}(x)+m^{2t}\right){V(\bar{x}^{l})}-\displaystyle\frac{c_{2}(\bar{x}^{k})c_{4}(\bar{x}^{l})}{\left(\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{1}(x)+m^{2s}\right)}V(\bar{x}^{l})\leq 0. \]

The left inequality used the fact $c_{1}, c_{2}>0$ and

\begin{align*} & \left(\inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)+m^{2s}\right)\left(\inf_{\{x\in\Sigma \mid V(x)>0\}}\,c_{3}(x)+m^{2t}\right)\\ & \quad >\inf_{\{x\in\Sigma \mid U(x)>0\}}c_{2}(x) \cdot\inf_{\{x\in\Sigma \mid V(x)>0\}}c_{4}(x). \end{align*}

That is impossible. So we have

\[ U(x)\leq 0 \quad \text{and}\quad V(x)\leq 0\ \text{in}\ \Sigma. \]

Proof of theorem 1.3. Let $T$ be any hyper-plane, $\Sigma$ be the half space on one side of the plane $T$ and $x_{N}$ be any given direction in $\mathbb {R}^{N}$. Set

\[ U(x)=u({x}^{\lambda})-u(x), \ V(x)=v(x^{\lambda})-v(x),\quad \hbox{ for all } x\in \Sigma,\]

and

\[ T:=\left\{x \in \mathbb{R}^{N} | x_{N}=\lambda \text { for any }\lambda \in \mathbb{R}^{N}\right\}, \]

where $x^{\lambda }$ is the reflection of $x$ with respect to $T$. We shall show that $u$ and $v$ are symmetric with respect to any hyper-plane $T$.

From the assumption in theorem 1.3, we have $U \in \mathcal {L}_{s}(\mathbb {R}^{N})\cap C_{\text {loc}}^{1,1}(\mathbb {R}^{N})$ and $V \in \mathcal {L}_{t}(\mathbb {R}^{N})\cap C_{\text {loc}}^{1,1}(\mathbb {R}^{N})$ are bounded. By a direct calculation, we get

(5.17)\begin{align} & \left(-\Delta+m^{2}\right)^{s}U(x)\nonumber\\ & \quad =f(x^{\lambda},u_{\lambda},v_{\lambda}, \nabla u(x^{\lambda}))-f(x,u,v,\nabla u(x)) \nonumber\\ & \quad \geq f(x,u_{\lambda},v_{\lambda}, \nabla u(x))-f(x,u,v,\nabla u(x)) \nonumber\\ & \quad = f(x,u_{\lambda},v_{\lambda}, \nabla u(x))- f(x,u,v_{\lambda}, \nabla u(x))\notag\\ & \qquad + f(x,u,v_{\lambda}, \nabla u(x))-f(x,u,v,\nabla u(x)) \nonumber\\ & \quad = \frac{\partial f}{\partial u}(x,\xi_{1},v_{\lambda},\nabla u(x))U(x)+\frac{\partial f}{\partial v}(x,u,\eta_{1},\nabla u(x))V(x) \nonumber\\ & \quad ={-}c_{1}(x)U(x)-c_{2}(x)V(x), \end{align}

and

(5.18)\begin{align} & \left(-\Delta+m^{2}\right)^{t}V(x) \nonumber\\ & \quad =g(x^{\lambda},u_{\lambda},v_{\lambda}, \nabla v(x^{\lambda}))-g(x,u,v,\nabla v(x)) \nonumber\\ & \quad \geq g(x,u_{\lambda},v_{\lambda}, \nabla v(x))-g(x,u,v,\nabla v(x)) \nonumber\\ & \quad = g(x,u_{\lambda},v_{\lambda}, \nabla v(x))- g(x,u,v_{\lambda}, \nabla v(x))+ g(x,u,v_{\lambda}, \nabla v(x))-g(x,u,v,\nabla v) \nonumber\\ & \quad =\frac{\partial g}{\partial u}(x,\xi_{2},v_{\lambda},\nabla v(x))U(x)+\frac{\partial g}{\partial v}(x,u,\eta_{2},\nabla u(x))V(x)\nonumber\\ & \quad ={-}c_{3}(x)V(x)-c_{4}(x)U(x), \end{align}

where $\xi _{1}, \xi _{2}$ are the values between $u(x)$ and $u_{\lambda }(x)$, $\eta _{1}, \eta _{2}$ are the values between $v(x)$ and $v_{\lambda }(x)$. Since $f(x, u, v,\mathbf {p}), g(x, u, v,\mathbf {q})$ are Lipschitz continuous with respect to $u,v$, we have $c_i(x)$(i=1,2,3,4) are bounded. By the assumptions of theorem 1.3, we have

\[ \inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)>{-}m^{2s}\quad \text{and}\quad \inf_{\{x\in\Sigma \mid V(x)>0\}}\,c_{3}(x)>{-}m^{2t}, \]

and

\[ \left(\inf_{\{x\in\Sigma \mid U(x)>0\}}\,c_{1}(x)+m^{2s}\right)\left(\inf_{\{x\in\Sigma \mid V(x)>0\}}\,c_{3}(x)+m^{2t}\right)>\inf_{\Sigma} c_{2}(x)\cdot\inf_{\Sigma}c_{4}(x). \]

It follows from proposition 5.2 that

\[ U\leq 0 \quad \hbox{and}\quad V\leq 0\quad \text{in}\ {\Sigma}. \]

In a similar way, we can also prove that $U\leq 0, V\leq 0$ in $\mathbb {R}^{N}\setminus \Sigma$. Hence

(5.19)\begin{equation} U=V \equiv 0\quad \text{in} \ \mathbb{R}^{N}. \end{equation}

This implies that $u$ and $v$ are symmetric with respect to $T$ for any $\lambda$. Since the $x_{N}$-direction can be chosen arbitrarily, (5.19) suggests $u$ and $v$ are radially symmetric about any point. It follows that

\[ (u,v)\equiv (C_{1},C_{2})\quad \text{in}\ \mathbb{R}^{N}. \]

The proof of theorem 1.3 is completed.

6. Proof of theorems 1.4 and 1.5

In this section, by using the direct method of moving plane, we shall first prove the strict monotonicity for system (1.1) in coercive epigraph domain.

Proof of theorem 1.4. Without loss of generality, we assume

\[ \inf_{x\in\Omega}x_N=0. \]

First, let us introduce the following notations

\[ T_{\lambda}:=\left\{x \in \mathbb{R}_{+}^{N} | x_{N}=\lambda, \text{ for some } \lambda \in \mathbb{R}_{+}\right\},\]

\[\Sigma_{\lambda}:=\left\{x \in \mathbb{R}_{+}^{N} | x_{N}<\lambda\right\}. \]

And denoting the reflection of $x$ about the moving plane $T_{\lambda }$ by $x^{\lambda }:=(x_{1}, x_{2}, \ldots, 2 \lambda -x_{N})$. $U_\lambda (x), V_\lambda (x)$ are defined as in § 2.

Next, we carry out the moving planes procedure in two steps.

Step 1. We show that, for $\lambda >0$ sufficiently closing to $0$,

(6.1)\begin{equation} U_{\lambda}(x)>0,\ V_{\lambda}(x) > 0, \quad x \in \Sigma_{\lambda}\cap \Omega. \end{equation}

First, we shall prove that

(6.2)\begin{equation} U_{\lambda}(x)\geq 0,\ V_{\lambda}(x)\geq 0, \quad x \in \Sigma_{\lambda}\cap \Omega. \end{equation}

If (6.2) does not hold, for simplicity, we assume that there exists a sequence $\{\lambda _{k}\}$ and $x^{k}\in \Sigma _{\lambda _{k}}\cap \Omega$ satisfying $\lambda _{k}>0$ and $\lambda _k\rightarrow 0$ as $k\rightarrow +\infty$ such that

(6.3)\begin{equation} U_{\lambda_{k}}(x^{k})=\inf_{\Sigma_{\lambda_{k}}\cap\Omega}U_{\lambda_{k}}=\inf_{\lambda\in(0,\lambda_{k}]}\inf_{x\in\Sigma_\lambda}U_\lambda(x)<0. \end{equation}

It follows directly from (6.3) that ${\partial U_{\lambda }}/{\partial \lambda }|_{\lambda =\lambda _k}(x^{k})\leq 0$, and hence $(\partial _{x_{n}}u) [(x^{k})^{\lambda _k}]\leq 0$. Note that $x^{k}$ is the interior minimum of $U_{\lambda _k}(x)$, then one has $\nabla _{x}U_{\lambda _k}(x^{k})=0$, i.e.,

(6.4)\begin{equation} (\nabla_{x}u_{\lambda_k})(x^{k})=(\nabla_{x}u)(x^{k}). \end{equation}

Through a direct calculation, we get

(6.5)\begin{equation} \left\{\begin{array}{l} (-\Delta+m^{2})^{s} U_{\lambda_{k}}(x)+c_{1}(x)U_{\lambda_{k}}(x)+c_{2}(x)V_{\lambda_{k}}(x)=0,\\ (-\Delta+m^{2})^{t} V_{\lambda_{k}}(x)+c_{3}(x)V_{\lambda_{k}}(x)+c_{4}(x)U_{\lambda_{k}}(x)=0, \end{array}\right. \end{equation}

where $c_i(x)$ is same to (3.6)–(3.9). Due to the Lipschitz continuous of $u,v$, we have $c_{i}(i=1,2,3,4)$ are bounded. We can also infer from $\mathbf {F_{1}}$ that $c_{2}(x)<0, c_{4}(x)<0$ and $-c_{1}(x)+m^{2s}\geq 0$, $-c_{3}(x)+m^{2t}\geq 0$. Since $\Omega$ is a coercive epigraph, $\Sigma _{\lambda }\cap \Omega$ is always is a bounded narrow region for $k$ large enough. In addition, one can easily obtain from system (1.1) that

(6.6)\begin{equation} U_{\lambda}(x)\geq 0,\ V_{\lambda}(x)\geq 0, \quad U_{\lambda}(x)\not\equiv 0 \ \text{or}\ V_{\lambda}(x)\not\equiv 0 \quad \text{in}\ \Sigma_{\lambda}\setminus \Omega. \end{equation}

By the assumption of theorem 1.4 and applying the Narrow region principle proposition 2.1, we can conclude the assertion (6.1).

Step 2. We continue to move the plane $T_{\lambda }$ along the $x_{N}$ -axis until its limiting position as long as (6.1) holds. More precisely, let

(6.7)\begin{equation} \lambda_{0}:=\sup \left\{\lambda>0 | U_{\mu}(x) > 0\ \text{and}\ V_{\mu}(x) > 0, x \in \Sigma_{\mu}\cap \Omega, \mu \leq \lambda\right\}. \end{equation}

We shall show that

(6.8)\begin{equation} \lambda_{0}={+}\infty. \end{equation}

If $\lambda _{0}<+\infty$, we shall show that the plane $T_{\lambda _0}$ can be moved upward a little bit more, that is, there exists an $\varepsilon >0$ small enough such that

(6.9)\begin{equation} U_{\lambda}>0 \quad \text{and}\quad V_{\lambda}>0 \quad \text{in} \ \Sigma_{\lambda}\cap\Omega, \quad \forall \ \lambda_{0}\leq\lambda\leq\lambda_{0}+\varepsilon, \end{equation}

which contradicts the definition (6.7) of $\lambda _{0}$.

Indeed, by the definition of $\lambda _{0}$, we have

\[ U_{\lambda_{0}}\geq 0\quad \text{and}\quad V_{\lambda_{0}}\geq 0 \quad \text{in}\ \Sigma_{\lambda_0}\cap\Omega. \]

Since $u,v>0$ in $\Omega$ and $u=v \equiv 0$ in $\mathbb {R}^{N}\setminus \Omega$, we have $U_{\lambda _{0}}>0, V_{\lambda _{0}}> 0$ for any $x\in \Omega ^{\lambda _{0}}\setminus \Omega$, where the notation $\Omega ^{\lambda }$ denotes the reflection of a given set $\Omega$ with respect to the plane $T_{\lambda }$. Then, we obtain from proposition 2.1 that

\[ U_{\lambda_{0}}>0,\ V_{\lambda_{0}} >0\quad \text{in}\ \Sigma_{\lambda_0}\cap\Omega. \]

Now we choose $\varepsilon _1>0$ sufficiently small such that $(\Sigma _{\lambda _{0}+\varepsilon _1}\setminus \overline {\Sigma _{\lambda _{0}-\varepsilon _1}})\cap \Omega$ is a bounded narrow region. By the fact that $U_{\lambda _{0}}(x)>0, V_{\lambda _{0}}(x)>0$ for $x\in \Omega ^{\lambda _{0}}\cap \Sigma _{\lambda _{0}}$ and the continuity of $U_{\lambda _{0}}$ and $V_{\lambda _{0}}$, there exists $c_0>0$ such that

\[ U_{\lambda_{0}}(x)> c_0, \quad V_{\lambda_{0}}(x)> c_0,\quad \forall \ x\in\overline{\Sigma_{\lambda_0-\varepsilon_1}}\cap \Omega. \]

Therefore, we can choose $0<\varepsilon _2<\varepsilon _1$ sufficiently small such that

(6.10)\begin{equation} U_{\lambda}(x)>\frac{c_0}{2}>0,\quad V_{\lambda}(x)>\frac{c_0}{2}>0,\quad \forall \ x\in\overline{\Sigma_{\lambda_0-\varepsilon_1}}\cap \Omega, \end{equation}

for every $\lambda _0\leq \lambda \leq \lambda _0+\varepsilon _2$. Since $(\Sigma _{\lambda }\setminus \overline {\Sigma _{\lambda _{0}-\varepsilon _1}})\cap \Omega$ is also a bounded narrow region. From the assumption $\mathbf {F}_{\mathbf {1}}$, we know $c_{i}(x)$ satisfy the condition of Narrow region principle proposition 2.1. Thus we can deduce from proposition 2.1 that

\[ U_{\lambda}>0,\ V_{\lambda}>0 \quad \text{in} \ \left(\Sigma_{\lambda}\setminus\overline{\Sigma_{\lambda_{0}-\varepsilon_1}}\right)\cap \Omega, \quad \forall \ \lambda_{0}\leq\lambda\leq\lambda_{0}+\varepsilon_2, \]

and hence

\[ U_{\lambda}>0,\ V_{\lambda}>0 \quad \text{in} \ \Sigma_{\lambda}\cap\Omega, \quad \forall \ \lambda_{0}\leq\lambda\leq\lambda_{0}+\varepsilon_2, \]

which contradicts the definition of $\lambda _{0}$ (6.7). Thus, we have $\lambda _0=+\infty$. Hence, $u$ and $v$ are increasing with respect to the $x_{N}$-axis. This completes the proof of theorem 1.4.

Now we are in the position to prove that the Liouville-type theorem for generalized pseudo-relativistic Schrödinger system (1.22) in $\mathbb {R}^{N}_{+}.$

Proof of theorem 1.5. We first show that if there exists $x_{0} \in \mathbb {R}_{+}^{N}$ such that $u(x_{0})=0$. Then, we have

(6.11)\begin{equation} u(x)\equiv v(x) \equiv 0\quad \text{in } \mathbb{R}_{+}^{N}. \end{equation}

Note that, $u$ is a nonnegative solution of system (1.1) and $u(x_{0})=0$, so we have $\nabla u(x_{0})=0$. On one hand, if $u(x) \not \equiv 0,$ we have

(6.12)\begin{align} & \left(-\Delta+m^{2}\right)^{s}u(x_{0}) \nonumber\\ & \quad = c_{N,s} m^{{{N}/{2}+s}} P.V. \int_{\mathbb{R}^{N}} \frac{u(x_{0})-u(y)}{|x-y|^{{{N}/{2}+s}}} K_{{{N}/{2}+s}}(m|x-y|)\,\textrm{d}y \nonumber\\ & \quad ={-}c_{N, s} m^{{{N}/{2}+s}} P.V. \int_{\mathbb{R}^{N}} \frac{u(y)}{|x-y|^{{{N}/{2}+s}}} K_{{{N}/{2}+s}}(m|x-y|)\,\textrm{d}y < 0. \end{align}

On the other hand, since ${\partial f}/{\partial v}>0$, we have

(6.13)\begin{equation} (-\Delta+m^{2})^{s} u\left(x_{0}\right)=f\left(x,0, v(x_{0}),\quad\nabla u\left(x_{0}\right)\right)> f(x,0,0,0)=0. \end{equation}

This is a contradiction, which implies that $u(x) \equiv 0$ in $\mathbb {R}_{+}^{N}$. Combining this truth with the first equation in (1.1), we must have $f(x, 0, v, \nabla u)=f(x, 0, v, 0)=0$. Hence, if $u(x)\not \equiv 0$,

(6.14)\begin{equation} 0=f(x, 0, v, 0)> f(x, 0, 0, 0)=0. \end{equation}

We can deduce that $v(x) \equiv 0$ in $\mathbb {R}_{+}^{N}$. Thus, we have

\[ u(x)=v(x)=0 \quad \text{in } \mathbb{R}_{+}^{N}. \]

If there exists $x_{0} \in \mathbb {R}_{+}^{N}$ such that $v(x_{0})=0$, we can use the same arguments as above to yield a contradiction by using the fact of ${\partial g}/{\partial u}>0$ and $g(x,0,0,0)=0$.

In the following, we always assume that $u(x)>0$ and $v(x)>0$. Then, we carry out the moving planes procedure in two steps.

Step 1. We shall show that, for $\lambda >0$ sufficiently closing to $0$,

(6.15)\begin{equation} U_{\lambda}(x)\geq 0,\ V_{\lambda}(x) \geq 0, \quad x \in \Sigma_{\lambda}. \end{equation}

Combining the assumption (iii) of theorem 1.5 with $u>0$, $v>0$, we have

\[ \varliminf_{|x| \rightarrow \infty} U_{\lambda}(x) \geq 0 \quad \text{and} \quad \varliminf_{|x| \rightarrow \infty} V_{\lambda}(x) \geq 0. \]

Then similar to the proof of theorem 1.4, by applying proposition 2.1, we get (6.15).

Step 2. We continue to move the plane $T_{\lambda }$ along the $x_{N}$ -axis until its limiting position as long as (6.15) holds. More precisely, let

(6.16)\begin{equation} \lambda_{0}:=\sup \left\{\lambda>0 | U_{\mu}(x) > 0\ \text{and}\ V_{\mu}(x) > 0, x \in \Sigma_{\mu}, \mu \leq \lambda\right\}. \end{equation}

Now we show that

(6.17)\begin{equation} \lambda_{0}={+}\infty. \end{equation}

If (6.17) is not true. Similar to the proof of theorem 1.4, we can derive either

(6.18)\begin{equation} U_{\lambda_{0}}(x)=0,\ V_{\lambda_{0}}(x)=0,\quad x \in {\Sigma}_{\lambda_{0}}, \end{equation}

(6.19)\begin{equation} U_{\lambda_{0}}(x)>0\quad \textrm{and}\quad V_{\lambda_{0}}(x)>0, \quad x \in {\Sigma}_{\lambda_{0}}. \end{equation}

If (6.19) is true, by the similar argument as proof of theorem 1.4, we will show that the plane $T_{\lambda _{0}}$ can be moved upward a little more, which contradicts the definition (6.16) of $\lambda _{0}$. Hence, (6.18) is true. It reveals that

\[ u(x_{1},x_{2},\ldots,x_{N-1},2\lambda_{0})=u(x_{1},x_{2},\ldots,x_{N-1},0)=0 \]

and

\[ v(x_{1},x_{2},\ldots,x_{N-1},2\lambda_{0})=v(x_{1},x_{2},\ldots,x_{N-1},0)=0. \]

That is a contradiction with $u,v>0$. So (6.17) holds.

Therefore, $u$ and $v$ are increasing concerning the $x_{N}$-axis. In terms of the assumption $\lim _{|x|\rightarrow +\infty }(u(x),v(x))=(0,0)$, we know that is impossible. So $u(x)= v(x) \equiv 0 \text { in } \mathbb {R}_+^{N}$.

We complete the proof of theorem 1.5.

Acknowledgments

Yuxia Guo was supported by NSFC (No. 11771235, 12031015). Shaolong Peng is supported by the NSFC (No. 11971049).

Data Availability Statements

All data generated or analysed during this study are included in this published article.

References

Ambrosio, V.. Ground states solutions for a non-linear equation involving a pseudo-relativistic Schrödinger operator. J. Math. Phys. 57 (2016), 051502.CrossRef Google Scholar

Bertoin, J.. Lévy processes. Cambridge Tracts in Mathematics, vol. 121 (Cambridge: Cambridge University Press, 1996).Google Scholar

Brandle, C., Colorado, E., de Pablo, A. and Sanchez, U.. A concave-convex elliptic problem involving the fractional Laplacian. Proc. R. Soc. Edinburgh-A: Math. 143 (2013), 39–71.CrossRef Google Scholar

Cabré, X. and Tan, J.. Positive solutions of nonlinear problems involving the square root of the Laplacian. Adv. Math. 224 (2010), 2052–2093.CrossRef Google Scholar

Caffarelli, L. and Silvestre, L.. An extension problem related to the fractional Laplacian. Commun. PDEs 32 (2007), 1245–1260.CrossRef Google Scholar

Caffarelli, L. and Vasseur, L.. Drift diffusion equations with fractional diffusion and the quasi-geostrophic equation. Ann. Math. 171 (2010), 1903–1930.CrossRef Google Scholar

Carmona, R., Masters, W. C. and Simon, B.. Relativistic Schrödinger operators: asymptotic behavior of the eigenfunctions. J. Funct. Anal. 91 (1990), 117–142.CrossRef Google Scholar

Chang, S.-Y. A. and Gonzàlez, M. d. M.. Fractional Laplacian in conformal geometry. Adv. Math. 226 (2011), 1410–1432.CrossRef Google Scholar

Chen, W. and Li, C.. Moving planes, moving spheres, and a priori estimates. J. Differ. Equ. 195 (2003), 1–13.CrossRef Google Scholar

Chen, W., Li, C. and Li, Y.. A direct method of moving planes for the fractional Laplacian. Adv. Math. 308 (2017), 404–437.CrossRef Google Scholar

Chen, W., Li, Y. and Ma, P.. The fractional Laplacian (Hackensack, NJ: World Scientific Publishing Co. Pte. Ltd., 2020), 331.CrossRef Google Scholar

Chen, W., Li, C. and Ou, B.. Classification of solutions for an integral equation. Commun. Pure Appl. Math. 59 (2006), 330–343.CrossRef Google Scholar

Chen, W., Li, Y. and Zhang, R.. A direct method of moving spheres on fractional order equations. J. Funct. Anal. 272 (2017), 4131–4157.CrossRef Google Scholar

Chen, Y. and Liu, B.. Symmetry and non-existence of positive solutions for fractional p-Laplacian systems. Nonlinear Anal. 183 (2019), 303–322.CrossRef Google Scholar

Chen, W. and Qi, S.. Direct methods on fractional equations. Disc. Cont. Dyn. Syst. – A 39 (2019), 1269–1310.CrossRef Google Scholar

Chen, W. and Wu, L.. The sliding methods for the fractional $p$

-Laplacian. Adv. Math. 361 (2020), 106933.Google Scholar

Cheng, T., Huang, G. and Li, C.. The maximum principles for fractional Laplacian equations and their applications. Commun. Contemp. Math. 19 (2017), 1750018.CrossRef Google Scholar

Constantin, P.. Euler equations, Navier–Stokes equations and turbulence. In Mathematical foundation of turbulent viscous flows. Lecture Notes in Mathematics, vol. 1871, pp. 1–43 (Berlin: Springer, 2006).CrossRef Google Scholar

Dai, W., Fang, Y., Huang, J., Qin, Y. and Wang, B.. Regularity and classification of solutions to static Hartree equations involving fractional Laplacians. Disc. Contin. Dyn. Syst. – A 39 (2019), 1389–1403.CrossRef Google Scholar

Dai, W. and Liu, Z.. Classification of nonnegative solutions to static Schrödinger-Hartree and Schrödinger-Maxwell equations with combined nonlinearities. Calc. Var. & PDEs 58 (2019), 156.CrossRef Google Scholar

Dai, W., Liu, Z. and Qin, G.. Classification of nonnegative solutions to static Schrödinger-Hartree-Maxwell type equations. SIAM J. Math. Anal. 53 (2021), 1379–1410.CrossRef Google Scholar

Dai, W. and Qin, G.. Classification of nonnegative classical solutions to third-order equations. Adv. Math. 328 (2018), 822–857.CrossRef Google Scholar

Dai, W., Qin, G. and Wu, D.. Direct methods for pseudo-relativistic Schrödinger operators. J. Geom. Anal. 31 (2021), 5555–5618.CrossRef Google Scholar

Dai, W. and Qin, G.. Liouville type theorems for fractional and higher order Hénon-Hardy equations via the method of scaling spheres, preprint, submitted for publication, arXiv:1810.02752.Google Scholar

Dipierro, S., Soave, N. and Valdinoci, E.. On fractional elliptic equations in Lipschitz sets and epigraphs: regularity, monotonicity and rigidity results. Math. Ann. 369 (2017), 1283–1326.CrossRef Google Scholar

Erdélyi, A., Magnus, W., Oberhettinger, F. and Tricomi, F. G.. Higher transcendental functions, vol. II (New York: McGraw-Hill, 1953).Google Scholar

Fall, M. M. and Felli, V.. Sharp essential self-adjointness of relativistic Schrödinger operators with a singular potential. J. Funct. Anal. 267 (2014), 1851–1877.CrossRef Google Scholar

Fall, M. M. and Felli, V.. Unique continuation properties for relativistic Schrödinger operators with a singular potential. Disc. Contin. Dyn. Syst. – A 35 (2015), 5827–5867.CrossRef Google Scholar

Frank, R. L., Lenzmann, E. and Silvestre, L.. Uniqueness of radial solutions for the fractional Laplacian. Commun. Pure Appl. Math. 69 (2013), 1671–1726.CrossRef Google Scholar

Guo, Y. and Liu, J.. Liouville type theorems for positive solutions of elliptic system in $R^{N}$

. Commun. Partial Differ. Equ. 33 (2008), 263–284.CrossRef Google Scholar

Guo, Y. and Peng, S.. Symmetry and monotonicity of nonnegative solutions to pseudo-relativistic Choquard equations. Z. Angew. Math. Phys. 72 (2021), 20, Paper No. 120.CrossRef Google Scholar

Lieb, E. H. and Thirring, W. E.. Gravitational collapse in quantum mechanics with relativistic kinetic energy. Ann. Phys. 155 (1984), 494–512.CrossRef Google Scholar

Lieb, E. H. and Yau, H. T.. The Chandrasekhar theory of stellar collapse as the limit of quantum mechanics. Commun. Math. Phys., 112 (1987), 147–174.CrossRef Google Scholar

Liu, J., Guo, Y. and Zhang, Y.. Liouville-type theorems for polyharmonic systems in $R^{N}$

. J. Differ. Equ. 225 (2006), 685–709.CrossRef Google Scholar

Liu, B. and Ma, L.. Radial symmetry results for fractional Laplacian systems. Nonlinear Anal. 146 (2016), 120–135.CrossRef Google Scholar

Peng, S.. Liouville theorems for fractional and higher order Hénon-Hardy systems on $\mathbb {R}^{n}$

. Complex Variables Elliptic Equ. 66 (2021), 1839–1863.CrossRef Google Scholar

Silvestre, L.. Regularity of the obstacle problem for a fractional power of the Laplace operator. Commun. Pure Appl. Math. 60 (2007), 67–112.CrossRef Google Scholar

Article contents

Liouville-type results for positive solutions of pseudo-relativistic Schrödinger system

Abstract

Keywords

MSC classification

1. Introduction

Theorem 1.3 Liouville theorem in $\mathbb {R}^{N}$

Theorem 1.5 Liouville theorem in $\mathbb {R}^{N}_{+}$

2. Narrow region principle and decay at infinity

Proposition 2.1 Narrow region principle

Proposition 2.3 Decay at infinity

3. Proof of theorem 1.1

4. Proof of theorem 1.2

5. Proof of theorem 1.3

Proposition 5.2 Maximum principles for anti-symmetric system in a unbounded domain

6. Proof of theorems 1.4 and 1.5

Acknowledgments

Data Availability Statements

References

Save article to Kindle

Save article to Dropbox

Save article to Google Drive

Reply to: Submit a response

Your details

You have entered the maximum number of contributors

Conflicting interests