Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2024-12-23T19:36:48.309Z Has data issue: false hasContentIssue false
  • English
  • Français

Hardy–Littlewood–Sobolev inequality and existence of the extremal functions with extended kernel

Part of: Nonlinear integral equations Partial differential equations

Published online by Cambridge University Press:  31 October 2022

Zhao Liu
Zhao Liu*
Affiliation:
School of Mathematics and Computer Science, Jiangxi Science and Technology Normal University, Nanchang 330038, P. R. China ([email protected])
Rights & Permissions [Opens in a new window]

Abstract

In this paper, we consider the following Hardy–Littlewood–Sobolev inequality with extended kernel(0.1)

\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta}}{|x-y|^{n-\alpha}}f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}, \end{equation}
for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n})$, $g\in L^{q'}(\mathbb {R}_+^{n})$ and $p,\,\ q'\in (1,\,\infty )$, $\beta \geq 0$, $\alpha +\beta >1$ such that $\frac {n-1}{n}\frac {1}{p}+\frac {1}{q'}-\frac {\alpha +\beta -1}{n}=1$.

We prove the existence of all extremal functions for (0.1). We show that if $f$ and $g$ are extremal functions for (0.1) then both of $f$ and $g$ are radially decreasing. Moreover, we apply the regularity lifting method to obtain the smoothness of extremal functions. Finally, we derive the sufficient and necessary condition of the existence of any nonnegative nontrivial solutions for the Euler–Lagrange equations by using Pohozaev identity.

Keywords

Existence of extremal functionsEuler–Lagrange equationsPohozaev identityHardy–Littlewood–Sobolev inequality

MSC classification

Primary: 45G15: Systems of nonlinear integral equations
Secondary: 35B40: Asymptotic behavior of solutions
Type
Research Article
Information
Proceedings of the Royal Society of Edinburgh Section A: Mathematics , Volume 153 , Issue 5 , October 2023 , pp. 1683 - 1705
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of The Royal Society of Edinburgh

1. Introduction

The classical Hardy–Littlewood–Sobolev inequality that was obtained by Hardy and Littlewood [Reference Hardy and Littlewood36] for $n = 1$ and by Sobolev [Reference Sobolev50] for general $n$ states that

(1.1)\begin{equation} \int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}}|x-y|^{-(n-\alpha)} f(x)g(y){\rm d}x{\rm d}y\leq C_{\alpha,n,p}\|f\|_{L^{p}(\mathbb{R}^{n})}\|g\|_{L^{q'}(\mathbb{R}^{n})} \end{equation}

with $1< p,\,q'<\infty,\, 0<\alpha < n$ and $\frac {1}{p}+\frac {1}{q'}+\frac {n-\alpha }{n}=2$.

Lieb [Reference Lieb39] employed the rearrangement inequalities to obtain the existence of the extremal functions of inequality (1.1). Furthermore, they also classified extremals of the inequality (1.1) and computed the sharp constant $C_{\alpha,n,p}$ only when one of $p$ and $q'$ is equal to $2$ or $p=q'$.

Through the inequality (1.1), we can deduce many important geometrical inequalities such as the Gross logarithmic Sobolev inequality [Reference Gross31] and the Moser–Onofri–Beckner inequality [Reference Beckner1]. It is also well-known that if we pick $\alpha =2,\, p = q' = {2n}/{(n+2)}$, then the Hardy–Littlewood– Sobolev inequality is in fact equivalent to the Sobolev inequality by Green's representation formula. By using the competing symmetry method, Carlen and Loss [Reference Carleman8] provided a different proof from Lieb's of the sharp constants and extremal functions in the diagonal case $p = q' = {2n}/{(n+\alpha )}$ and Frank and Lieb [Reference Frank and Lieb25] offered a new proof using the reflection positivity of inversions in spheres in the special diagonal case. Frank and Lieb [Reference Frank and Lieb26] further employed a rearrangement-free technique developed in [Reference Frank and Lieb27] to recapture the best constant of inequality (1.1). Folland and Stein [Reference Folland and Stein24] extended the inequality (1.1) to the Heisenberg group and established the Hardy–Littlewood–Sobolev inequality on Heisenberg group. Frank and Lieb [Reference Frank and Lieb27] classify the extremals of this inequality in the diagonal case. This extends the earlier work of Jerison and Lee [Reference Jerison and Lee38] for sharp constants and extremals for the Sobolev inequality on the Heisenberg group in the conformal case in their study of CR Yamabe problem. Furthermore, Han et al. [Reference Han, Lu and Zhu34] established the double-weighted Hardy–Littlewood–Sobolev inequality (namely, Stein–Weiss inequality) on the Heisenberg group and discussed the regularity and asymptotic behaviour of the extremal functions. Recently, Chen et al. [Reference Chen, Lu and Tao13] used the concentration-compactness principle to obtain existence of extremals of the Stein–Weiss inequality on the Heisenberg group for all indices. We also mention that when $p = q' = {2n}/{(n+\alpha )}$, Euler–Lagrange equation of the extremals to the Hardy–Littlewood–Sobolev inequality in the Euclidean space is a conformal invariant integral equation. The inequality (1.1) and its extensions have many applications in partial differential equations. Some remarkable extensions have already been obtained on the upper half space by Dou and Zhu [Reference Dou and Zhu22], on compact Riemannian manifolds by Han and Zhu [Reference Han and Zhu35] and the reversed (weighted) Hardy–Littlewood–Sobolev inequality in [Reference Chen, Liu, Lu and Tao10, Reference Dou and Zhu23, Reference Ngô and Nguyen48, Reference Ngô and Nguyen49]. For more results about the (weighted) Hardy–Littlewood–Sobolev inequality, the general weighted inequalities and their corresponding Euler–Lagrange equations, refer to e.g. [Reference Beckner2, Reference Brascamp and Lieb3, Reference Chen, Liu and Lu9, Reference Chen and Li15–Reference Dai and Liu20, Reference Gao, Liu, Moroz and Yang28, Reference Han and Lin32, Reference Hu and Liu37, Reference Lieb and Loss42–Reference Lu and Zhu45, Reference Moroz and Van Schaftingen47, Reference Stein and Weiss51] and the references therein.

Recently, Gluck [Reference Gluck30] proved the following sharp Hardy–Littlewood–Sobolev inequality with extended kernel in the conformal invariant case ($p=\tfrac {2(n-1)}{n+\alpha -2}$, $q'=\tfrac {2n}{n+\alpha +2\beta }$)

(1.2)\begin{equation} \Big|\int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} K(x'-y,x_n) f(y)g(x) {\rm d}y{\rm d}x\Big|\leq C_{n,\alpha,\beta,p} \|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

where $K$ is a kernel of the form

\[ K(x',x_n)=K_{\alpha,\beta}(x',x_n)=\frac{x_n^{\beta}}{(|x'|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}, \quad x=(x',x_n)\in \mathbb{R}^{n-1}\times (0,\infty), \]

and $\alpha$, $\beta$ satisfy $\beta \geq 0$, $0<\alpha +\beta < n-\beta$,

(1.3)\begin{equation} \frac{n-\alpha-2\beta}{2n}+\frac{n-\alpha}{2(n-1)}<1. \end{equation}

In fact, for $\alpha =0$, $\beta =1$, the kernel $K_{\alpha,\beta }$ is the classical Poisson kernel. Hang et al. [Reference Hang, Wang and Yan33] derived the Hardy–Littlewood–Sobolev inequality with the Poisson kernel and proved the existence of extremals for this inequality by the concentration-compactness principle [Reference Lions40, Reference Lions41]. For the conformal invariant case, they classified the extremal functions of the inequality, and computed the sharp constant. Integral inequality with the Poisson kernel is highly related to Carleman's proof of isoperimetric inequality in the plane (see [Reference Carleman7]). For $\alpha \in (0,\,1)$, $\beta =1-\alpha$, the kernel $K_{\alpha,\beta }$ is related to the divergence form operator $u\mapsto {\rm div}(x_n^{\alpha } \nabla u)$ (the poly-harmonic extension operator) on the half space. Chen [Reference Chen14] established sharp Hardy–Littlewood–Sobolev inequality (1.2). He also generalized Carleman's inequality for harmonic functions in the plane to poly-harmonic functions in higher dimensions. Dou and Zhu [Reference Dou and Zhu22] studied the sharp Hardy–Littlewood–Sobolev inequality on the upper half space and the existences of extremal functions for $\beta =0$. Dou et al. [Reference Dou, Guo and Zhu21] investigated the integral inequality (1.2) in the special index through the methods based on conformal transformation for $\beta =1$. Different from Dou et al. [Reference Dou, Guo and Zhu21], Chen et al. [Reference Chen, Lu and Tao12] derived the Hardy–Littlewood–Sobolev inequality to all critical index for $\beta =1$. Furthermore, Chen et al. [Reference Chen, Liu, Lu and Tao11] extended it to the weighted Hardy–Littlewood–Sobolev inequality.

In this paper, we extended the Hardy–Littlewood–Sobolev inequality with extended kernel in the conformal invariant case to all critical index. That is,

Theorem 1.1 Let $n\geq 2,$ $1< p,\, q'<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q'$ satisfy

\[ \frac{n-1}{n}\frac{1}{p}+\frac{1}{q'}-\frac{\alpha+\beta-1}{n}=1. \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n}),$ $g\in L^{q'}(\mathbb {R}_+^{n}),$

(1.4)\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

We remark that the constant $C_{n,\alpha,\beta,p}$ above can be considered as the least one such that the above inequality holds for all nonnegative functions $f\in L^{p}(\partial \mathbb {R}^{n}_+)$, $g\in L^{q'}(\mathbb {R}^{n}_+)$. This constant $C_{n,\alpha,\beta,p}$ is often referred as the best constant for the Hardy–Littlewood–Sobolev inequality with extended kernel.

Define

\[ Tf(x)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}f(y){\rm d}y,\quad T'g(y)=\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}g(x){\rm d}x. \]

Throughout this paper, we always assume that $q$ and $q'$ are conjugate numbers. That is, $q$ and $q'$ satisfy $\frac {1}{q}+\frac {1}{q'}=1$. By duality, it is easy to verify that the inequality (1.4) is equivalent to the following two corollaries.

Corollary 1.2 Assume that $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta >1,$ $1< p<\frac {n-1}{\alpha +\beta -1},$ and

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that

(1.5)\begin{equation} \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

Corollary 1.3 Assume that $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta >1,$ $1< q'<\frac {n}{\alpha +\beta },$ and

\[ \frac{1}{p'}=\frac{n}{n-1}\left(\frac{1}{q'}-\frac{\alpha+\beta}{n}\right). \]

Then there is a constant $C_{n,\alpha,\beta,q'}>0$ such that

(1.6)\begin{equation} \|T'g\|_{L^{p'}(\partial\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|g\|_{L^{q'}(\mathbb{R}^{n}_+)}. \end{equation}

Once we establish the Hardy–Littlewood–Sobolev inequality with extended kernel, it is natural to ask whether the extremal functions for inequality (1.4) actually exist. To answer this question, we turn to consider the following maximizing problem

(1.7)\begin{equation} C_{n,\alpha,\beta,p}:= \sup \{\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)} \mid \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1, f\geq 0\}, \end{equation}

where $p$, $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

It is not hard to verify that the extremals of inequality (1.5) are those solving the maximizing problem (1.7). We use the rearrangement inequality to prove the attainability of maximizers for the maximizing problem (1.7).

Theorem 1.4 Let $n\geq 2,$ $1< p,\, q<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1,$ and suppose that $\alpha,$ $\beta,$ $p,$ $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there exists some function $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ such that $f\geq 0,$ $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1,$ and $\|Tf\|_{L^{q}( \mathbb {R}^{n}_+)}=C_{n,\alpha,\beta,p}$. Moreover, all extremal functions are radially symmetric and strictly decreasing about some point $y_0\in \partial \mathbb {R}^{n}_+$.

We now turn our attention to study the regularity of the extremal functions for inequality (1.5), the Euler–Lagrange equation for extremal functions, up to a constant multiplier, is given by

(1.8)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

We prove

Theorem 1.5 Let $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1$ and $1 < p< \frac {n-1}{\alpha +\beta -1}$. Suppose that $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$ is nonnegative solution to (1.8) with $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. Then $f\in C^{\infty }(\partial \mathbb {R}^{n}_+)$.

Assume that

\[ u(y)=f^{p-1}(y), \quad v(x)=Tf(x). \]

Denote

\[ \theta=\frac{1}{p-1},\quad \kappa=q-1. \]

Euler–Lagrange equation (1.8) can be rewritten as the following integral system

(1.9)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}}u^{\theta}(y) {\rm d}y, & x\in\mathbb{R}^{n}_+. \end{cases} \end{equation}

We use the Pohozaev identity to prove the following theorem.

Theorem 1.6 For $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1,$ $\theta >0,$ $\kappa >0,$ assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial $C^{1}$ solutions of (1.9), then a necessary condition for $\theta$ and $\kappa$ is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

Obviously, extremals $(f,\,g)$ of inequality (1.4) satisfies the integral system (1.9). In light of theorems Reference Chen, Lu and Tao3.1, Reference Chen4.1 and Reference Christ, Liu and Zhang5.1, we obtain the sufficient and necessary condition for existence of positive solutions to the integral system (1.9).

Theorem 1.7 For $\theta >0,$ $\kappa >0,$ let $n,$ $\alpha,$ $\beta,$ $p,$ $q$ satisfy all the hypotheses of theorems Reference Chen, Lu and Tao3.1, Reference Chen4.1 and Reference Christ, Liu and Zhang5.1, then the sufficient and necessary condition for the existence of a pair of nonnegative nontrivial solutions $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ to system (1.9) is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

The following Liouville type theorem was proved by Gluck.

Theorem 1.8 (see [Reference Gluck30])

Let $n\geq 2$ and suppose $\alpha,$ $\beta$ satisfy $\beta \geq 0,$ $0<\alpha +\beta < n-\beta$ and (1.3). If $u\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}(\mathbb {R}^{n}_+)$ are positive solutions of (1.9) with $\theta =\frac {n+\alpha -2}{n-\alpha }$ and $\kappa =\frac {n+\alpha +2\beta }{n-\alpha -2\beta }$. Then there exists $c_1>0,$ $d>0$ and $y_0\in \partial \mathbb {R}^{n}_+$ such that

\[ u(y)=\frac{c_1}{\big(d^{2}+|y-y_0|^{2}\big)^{{(n-\alpha)}/{2}}} \ {\rm for\ all\ } \ y\in \partial\mathbb{R}^{n}_+. \]

With the help of theorem 1.7, we use weaker assumption (1.10) to obtain theorem 1.9 instead of the conformal invariant case.

Theorem 1.9 Let $n\geq 2$ and suppose $\alpha,$ $\beta$ satisfy $\beta \geq 0,$ $0<\alpha +\beta < n-\beta$. If $u\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}(\mathbb {R}^{n}_+)$ are nonnegative nontrivial solutions of (1.9) with

(1.10)\begin{equation} 0<\theta\leq\frac{n+\alpha-2}{n-\alpha},\quad 0<\kappa\leq\frac{n+\alpha+2\beta}{n-\alpha-2\beta}. \end{equation}

Then

\[ \theta=\frac{n+\alpha-2}{n-\alpha},\quad \kappa=\frac{n+\alpha+2\beta}{n-\alpha-2\beta}. \]

Moreover, there exists $c_1>0$, $d>0$ and $y_0\in \partial \mathbb {R}^{n}_+$ such that

\[ u(y)=\frac{c_1}{\big(d^{2}+|y-y_0|^{2}\big)^{{(n-\alpha)}/{2}}} \ {\rm for\ all\ } \ y\in \partial\mathbb{R}^{n}_+. \]

From theorem 1.7, we must have $\theta =\frac {n+\alpha -2}{n-\alpha }$ and $\kappa =\frac {n+\alpha +2\beta }{n-\alpha -2\beta }$. Then, the proof is completely similar to the proof by Gluck in [Reference Gluck30], so we omit the details.

This paper is organized as follows. In § 2, we prove the Hardy–Littlewood–Sobolev inequality with the extended kernel. In § 3, by the rearrangement inequality, we obtain the existence of extremals of the inequality. Section 4 is devoted to the regularity estimate of the extremal functions of the Hardy–Littlewood–Sobolev inequality with the extended kernel. In § 5, using the Pohozaev identity in integral forms, we give sufficient and necessary conditions for the existence of nonnegative nontrivial solutions.

2. The proof of theorem 2.1

In this section, we use the Marcinkiewicz interpolation theorem and weak type estimate to establish the Hardy–Littlewood–Sobolev inequality with the extended kernel.

Theorem 2.1 Let $n\geq 2,$ $1< p,\, q'<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q'$ satisfy

\[ \frac{n-1}{n}\frac{1}{p}+\frac{1}{q'}-\frac{\alpha+\beta-1}{n}=1. \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n})$, $g\in L^{q'}(\mathbb {R}_+^{n}),$

(2.1)\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

Proof. For $t>0$ and $x'\in \mathbb {R}^{n-1}$, define

\[ K_t(x')=\frac{t^{\beta}}{(|x'|^{2}+t^{2})^{{(n-\alpha)}/{2}}}. \]

Then, for $x=(x',\,x_n)\in \mathbb {R}^{n}_+$, $y\in \partial \mathbb {R}^{n}_+$, we have

\[ K(x'-y,x_n)=K_{x_n}(x'-y),\quad Tf(x)=(K_{x_n}*f)(x'). \]

We are ready to prove theorem 2.1 via proving inequality (1.5). For $p\in (1,\,\frac {n-1}{\alpha +\beta -1})$ and $q$ given by $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. By the Marcinkiewicz interpolation theorem (see [Reference Stein and Weiss52]), we only need to prove the following weak-type estimate:

(2.2)\begin{equation} \|Tf\|_{L_w^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

That is, we need to show that there is a constant $C_{n,\alpha,\beta,p}>0$ such that

\[ \lambda|\{x\in\mathbb{R}^{n}_+{\mid} |Tf(x)|>\lambda\}|^{{1}/{q}}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}, \quad \forall f\in L^{p}(\partial\mathbb{R}^{n}_+), \ \forall \lambda>0. \]

Without the loss of generality, we may assume that $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1$. Assume that $r,\,s$ satisfy

(2.3)\begin{equation} r\in \left(\frac{(n-1)p}{(1-\alpha)p+n-1}, \frac{np}{(1-\alpha-\beta)p+n-1}\right),\quad \frac{1}{r}+1=\frac{1}{p}+\frac{1}{s},\ s\geq 1. \end{equation}

It follows from the Young equality that

\begin{align*} & \int_{\substack{x\in\mathbb{R}^{n}_+ \\ 0< x_n< a}}|Tf(x)|^{r}{\rm d}x\\ & \quad=\int_0^{a}\int_{\mathbb{R}^{n-1}}|(K_{x_n}*f)(x')|^{r}{\rm d}x'{\rm d}x_n\\ & \quad\leq\|f\|_{L^{p}(\mathbb{R}^{n-1})}\int_0^{a}\|K_{x_n}\|^{r}_{L^{s}(\mathbb{R}^{n-1})}{\rm d}x_n\\ & \quad=\int_0^{a}\left(\int_{\mathbb{R}^{n-1}}\frac{x_n^{\beta s}}{(|x'|^{2}+x_n^{2})^{{((n-\alpha)s)}/{2}}}{\rm d}x'\right)^{{r}/{s}}{\rm d}x_n\\ & \quad\leq\int_0^{a}x_n^{{((n-1)r)}/{s}+(\alpha+\beta-n)r}{\rm d}x_n\left(\int_{\mathbb{R}^{n-1}}\frac{1}{(|x'|^{2}+1)^{{((n-\alpha)s)}/{2}}}{\rm d}x'\right)^{{r}/{s}}. \end{align*}

One can deduce from (2.3) that

\[ \frac{(n-1)r}{s}+(\alpha+\beta-n)r>{-}1,\quad (n-\alpha)s>n-1. \]

Then, we have

\[ \int_{\substack{x\in\mathbb{R}^{n}_+ \\ 0< x_n< a}}|Tf(x)|^{r}{\rm d}x\leq C_1 a^{{((n-1)r)}/{s}+(\alpha+\beta-n)r+1}. \]

In view of the Hölder inequality and the integration of the extended kernel, we can see that

\[ \|K_{x_n}*f(x')\|_{L^{\infty}(\mathbb{R}^{n-1})}\leq C x_n^{{(n-1)}/{p'}+(\alpha+\beta-n)}. \]

Since $p\in (1,\,\frac {n-1}{\alpha +\beta -1})$, we know that $\frac {n-1}{p'}+(\alpha +\beta -n)<0$. Then, we derive that

\begin{align*} & |\{x\in\mathbb{R}^{n}_+{\mid} |Tf(x)|>\lambda\}|\\ & \quad=\Big|\Big\{x\in\mathbb{R}^{n}_+{\mid} 0< x_n< C \lambda^{{p'}/{(n-1+p'(\alpha+\beta-n))}},\quad |Tf(x)|>\lambda\Big\}\Big|\\ & \quad\leq \frac{1}{\lambda^{r}}\int_{x\in\mathbb{R}^{n}_+,\quad 0< x_n< C \lambda^{{p'}/{(n-1+p'(\alpha+\beta-n))}}}|Tf(x)|^{r}{\rm d}x\\ & \quad\leq C' \lambda^{{np}/{((\alpha+\beta-1)p-n+1)}}\\ & \quad\leq C' \lambda^{{-}q}, \end{align*}

which implies that

(2.4)\begin{equation} \|Tf\|_{L_w^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

Note that inequality (2.4) implies, via the Marcinkiewicz interpolation [Reference Stein and Weiss52], that

\[ \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \]

or even slight stronger inequality

(2.5)\begin{equation} \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p,q}(\partial\mathbb{R}^{n}_+)}. \end{equation}

where Lorentz norm $\|\cdot \|_{L^{p,q}}$ is defined by

\[ \|u\|_{L^{p,q}}=p^{{1}/{q}}\left(\int_0^{\infty} t^{q} \mid |u|>t|^{{q}/{p}}\frac{{\rm d}t}{t}\right)^{{1}/{q}}. \]

3. The proof of theorem 3.1

In the following, we will employ rearrangement inequality to investigate the existence of maximizers for the maximizing problem

(3.1)\begin{equation} C_{n,\alpha,\beta,p}:= \sup \{\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)} \mid \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1, f\geq 0\}. \end{equation}

We prove

Theorem 3.1 Let $n\geq 2,$ $1< p,\, q<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there exists some function $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ such that $f\geq 0,$ $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1,$ and $\|Tf\|_{L^{q}( \mathbb {R}^{n}_+)}=C_{n,\alpha,\beta,p}$. Moreover, all extremal functions are radially symmetric and strictly decreasing about some point $y_0\in \partial \mathbb {R}^{n}_+$.

Proof. Using symmetrization argument, we first show that the supremum of (3.1) is attained by radially symmetric functions. Now, we recall the important Riesz rearrangement inequality. Let $u$ be a measurable function on $\mathbb {R}^{n}$, the symmetric rearrangement of $u$ is the nonnegative lower semi-continuous radial decreasing function $u^{*}$ that has the same distribution as $u$. Then, we have

\[ \int_{\mathbb{R}^{n}}{\rm d}x\int_{\mathbb{R}^{n}}u(x)v(y-x)w(y){\rm d}y\leq \int_{\mathbb{R}^{n}}{\rm d}x\int_{\mathbb{R}^{n}}u^{*}(x)v^{*}(y-x)w^{*}(y){\rm d}y. \]

Using the fact $\|w\|_{L^{p}(\mathbb {R}^{n})}=\|w^{*}\|_{L^{p}(\mathbb {R}^{n})}$ for $p>0$ and the standard duality argument, we see, for $1\leq p\leq \infty$,

\[ \|u*v\|_{L^{p}(\mathbb{R}^{n})}\leq\|u^{*}*v^{*}\|_{L^{p}(\mathbb{R}^{n})}. \]

Moreover, if $u$ is nonnegative radially symmetric and strictly decreasing in the radial direction, $v$ is nonnegative, $1< p<\infty$ and

\[ \|u*v\|_{L^{p}(\mathbb{R}^{n})}=\|u^{*}*v^{*}\|_{L^{p}(\mathbb{R}^{n})}<\infty, \]

then from Brascamp et al. [Reference Brascamp, Lieb and Luttinger4], we have,

(3.2)\begin{equation} v(x)=v^{*}(x-x_0), \end{equation}

for some $x_0\in \mathbb {R}^{n}$.

Now, assume $f_i$ is a maximizing sequence in (3.1). Since

\[ \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=\|f^{*}\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1 \]

and

\begin{align*} \|Tf_i\|_{L^{q}(\mathbb{R}^{n}_+)}^{q}& =\int_0^{\infty}\|K_{x_n}*f_i\|_{L^{q}(\mathbb{R}^{n-1})}^{q}{\rm d}x_n\\ & \leq\int_0^{\infty}\|K_{x_n}*f_i^{*}\|_{L^{q}(\mathbb{R}^{n-1})}^{q}{\rm d}x_n\\ & =\|Tf_i^{*}\|_{L^{q}(\mathbb{R}^{n}_+)}^{q}. \end{align*}

We know that $f_i^{*}$ is also a maximizing sequence. Hence, we may assume $f_i$ is a nonnegative radial decreasing function.

For any $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ and any $\lambda >0$, we let $f^{\lambda }(y)=\lambda ^{-({(n-1)}/{p})}f(\frac {y}{\lambda })$, then it is easy to check that

\[ \|f^{\lambda}\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)},\quad \|Tf^{\lambda}\|_{L^{q}(\mathbb{R}^{n}_+)}=\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}. \]

For convenience, denote $e_1^{'}=(1,\,0,\,\ldots,\,0)\in \mathbb {R}^{n-1}$ and

\[ a_i=\sup_{\lambda>0}f_i^{\lambda}(e_1^{'})=\sup_{\lambda>0}\lambda^{-({(n-1)}/{p})}f_i\left(\frac{e_1^{'}}{\lambda}\right). \]

It follows that

\[ 0\leq f_i(y)\leq a_i|y|^{{-({(n-1)}/{p})}} \]

and hence

\[ \|f_i\|_{L^{p,\infty}(\partial\mathbb{R}^{n}_+)}\leq \omega_{n-1}^{{1}/{p}}a_i. \]

Thus, by (2.5), we have

\begin{align*} \|Tf_i\|_{L^{q}(\mathbb{R}^{n}_+)}& \leq C_{n,\alpha,\beta,p}\|f_i\|_{L^{p,q}(\partial\mathbb{R}^{n}_+)}\\ & \leq C_{n,\alpha,\beta,p}\|f_i\|^{1-{p}/{q}}_{L^{p,\infty}(\partial\mathbb{R}^{n}_+)} \|f_i\|^{{p}/{q}}_{L^{p}(\partial\mathbb{R}^{n}_+)}\\ & \leq C_{n,\alpha,\beta,p}a_i^{1-{p}/{q}}, \end{align*}

which implies $a_i\geq c(n,\,\alpha,\,\beta,\, p)>0$. We may choose $\lambda _i>0$ such that $f_i^{\lambda _i}(e_1^{'})\geq c(n,\,\alpha,\,\beta,\, p)>0$. Replacing $f_i$ by $f_i^{\lambda _i}$, we may assume $f_i(e_1^{'})\geq c(n,\,\alpha,\,\beta,\, p)>0$. On the other hand, since $f_i$ is nonnegative radially decreasing and $f_i\in L^{p}(\partial \mathbb {R}^{n}_+)=1$, it is obvious that

\[ f_i(y)\leq \omega_{n-1}^{{1}/{p}}|y|^{{-({(n-1)}/{p})}}. \]

Hence after passing to a subsequence, we may find a nonnegative radially decreasing function $f$ such that $f_i\rightarrow f$ a.e. It follows that $f(y)\geq c(n,\,\alpha,\,\beta,\, p) > 0$ for $|y|\leq 1$, and $\|f\|_{ L^{p}(\partial \mathbb {R}^{n}_+)}\leq 1$. From Brezis and Lieb's Lemma [Reference Brezis and Lieb6], we see

\[ \int_{\partial\mathbb{R}^{n}_+}\big||f_i(y)|^{p}-|f(y)|^{p}-|f_i(y)-f(y)|^{p}\big|{\rm d}y\rightarrow 0,\quad \text{as}\ \ i\rightarrow\infty. \]

It follows that

(3.3)\begin{equation} \begin{aligned} \|f_i-f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)} & =\|f_i\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1)\\ & =1-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{aligned} \end{equation}

On the other hand, since $Tf_i(x)\rightarrow Tf(x)$ for $x\in \mathbb {R}^{n}_+$ and $\|Tf_i\|_{L^{q}(\mathbb {R}^{n}_+)}\leq C_{n,\alpha,\beta,p}$, we see

\begin{align*} \|Tf_i\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}& =\|Tf\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}-\|Tf_i-Tf\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}+o(1)\\ & \leq C^{q}_{n,\alpha,\beta, p}\|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ C^{q}_{n,\alpha,\beta, p}\|f_i-f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{align*}

Hence,

(3.4)\begin{equation} 1\leq \|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ \|f_i-f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{equation}

By (3.3) and (3.4) and letting $i\rightarrow \infty$, we derive

\[ 1\leq \|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ (1-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)})^{{q}/{p}}. \]

Since $q>p$ and $f\neq 0$, we deduce that $\|f\|_{ L^{p}(\partial \mathbb {R}^{n}_+)}=1$. Hence, $f_i\rightarrow f$ in $L^{p}(\partial \mathbb {R}^{n}_+)$ and $f$ is a maximizer. This implies the existence of an extremal function.

Assume $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ is a maximizer, then so is $|f|$. Hence $\|Tf\|_{L^{q}(\mathbb {R}^{n}_+)}=\|T|f|\|_{L^{q}(\mathbb {R}^{n}_+)}$, which implies either $f\geq 0$ or $f\leq 0$. Without loss of generality, we only consider the case of $f\geq 0$, then the Euler–Lagrange equation after scaling by a positive constant is given by equation (3.1)

(3.5)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{(|x'-y|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

On the other hand, for $x_n>0$,

\[ \|K_{x_n}*f\|_{L^{q}(\mathbb{R}^{n}_+)}=\|K_{x_n}*f^{*}\|_{L^{q}(\mathbb{R}^{n}_+)}. \]

By (3.2), we deduce that

\[ f(y)=f^{*}(y-y_0)=f^{*}(|y-y_0|), \]

for some $y_0\in \partial \mathbb {R}^{n}_+$. It follows from the Euler–Lagrange equation (3.5) and lemma 2.2 of Lieb [Reference Lieb39] that $f$ must be strictly decreasing along the radial direction.

4. The proof of theorem 4.1

In this section, we establish the regularity properties of solutions to the following Euler–Lagrange equation:

(4.1)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{(|x'-y|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

We prove

Theorem 4.1 Let $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1$ and $1 < p< \frac {n-1}{\alpha +\beta -1}$. Suppose that $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$ is nonnegative solution to (4.1) with $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. Then $f\in C^{\infty }(\partial \mathbb {R}^{n}_+)$.

Let $u(y)=f^{p-1}(y)$, $v(x)=Tf(x)$, $\theta =\frac {1}{p-1}$ and $\kappa =q-1$. Then Euler–Lagrange equation (4.1) can be rewritten as the following integral system

(4.2)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y){\rm d}y, & x\in\mathbb{R}^{n}_+, \end{cases} \end{equation}

with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$. If $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$, then $u\in L_{loc}^{\theta +1}(\partial \mathbb {R}^{n}_+)$. Therefore, to prove theorem 4.1, it is sufficient to prove the following lemma.

Lemma 4.2 Assume that $\beta \geq 0,$ $\alpha +\beta > 1$ and $\frac {\alpha +\beta -1}{n-1}< \theta < \infty,$ and $0<\kappa <\infty$ given by

\[ \frac{1}{\kappa+1}=\frac{n-1}{n}\left(\frac{n-\alpha-\beta}{n-1}-\frac{1}{\theta+1}\right). \]

Suppose that $(u,\,v)$ is a pair of nonnegative solutions of (4.2) with $u\in L_{loc}^{\theta +1}(\partial \mathbb {R}^{n}_+)$. Then $u\in C^{\infty }(\partial \mathbb {R}^{n}_+)$ and $v\in C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

To prove lemma 4.2, we first establish two local regularity results, which are spirited by Brezis and Kato's lemma A.1 in [Reference Brezis and Kato5], Hang et al.'s propositions 5.2 and 5.3 in [Reference Hang, Wang and Yan33], Li's theorem 1.3 in [Reference Li Remark on some conformally invariant integral equations44], Dou and Zhu's propositions 4.3 and 4.4 in [Reference Dou and Zhu22].

For $R>0$, define

\begin{align*} & B_R(x)=\{y\in \mathbb{R}^{n} \mid |y-x|< R,\ x\in \mathbb{R}^{n}\},\\ & B_R^{n-1}(x)=\{y\in \partial\mathbb{R}^{n}_+{\mid} |y-x|< R,\ x\in \partial\mathbb{R}^{n}_+\},\\ & B_R^{+}(x)=\{y=(y_1,y_2,\ldots,y_n)\in B_R(x) \mid y_n>0,\ x\in \partial\mathbb{R}^{n}_+\}. \end{align*}

For $x = 0$, we write

\[ B_R=B_R(0),\quad B_R^{n-1}=B_R^{n-1}(0),\ B_R^{+}=B_R^{+}(0). \]

Lemma 4.3 Assume that $\alpha +\beta >1,$ $1< a,\,b\leq \infty,$ $1\leq r<\infty,$ and $\frac {n}{n-\alpha -\beta }< p< q<\infty$ satisfy

(4.3)\begin{equation} \frac{\alpha+\beta}{n}<\frac{r}{q}+\frac{1}{a}<\frac{r}{p}+\frac{1}{a}<1, \frac{n}{ar}+\frac{n-1}{b}=\frac{\alpha+\beta}{r}+(\alpha+\beta-1). \end{equation}

Suppose that $v,\,h\in L^{p}(B^{+}_R)$, $V\in L^{a}(B_R^{+})$, and $U\in L^{b}(B_R^{n-1})$ are all nonnegative functions with $h|_{B^{+}_{R/2}}\in L^{q}(B^{+}_{R/2})$, and

\[ v(x)\leq\int_{B_R^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\Big[ \int_{B_R^{+}}\frac{z_n^{\beta} V(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\Big]^{{1}/{r}}{\rm d}y+h(x),\ \forall x\in B_R^{+}. \]

There is a $\epsilon =\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)>0$, and $C=C(n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b,\,\epsilon )>0$ such that if

\[ \|U\|_{L^{b}(B_R^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B_R^{+})}\leq \epsilon(n,\alpha,\beta,p,q,r,a,b), \]

then,

\[ \|v\|_{L^{q}(B^{+}_{R/4})}\leq C(n,\alpha,\beta,p,q,r,a,b,\epsilon)\left(R^{{n}/{q}-{n}/{p}} \|v\|_{L^{p}(B^{+}_{R})}+\|h\|_{L^{q}(B^{+}_{R/2})}\right). \]

Proof. By scaling, we may assume $R = 1$. Assume that $v,\, h\in L^{q}(B^{+}_1)$. For $y\in B_1^{n-1}$, denote

\[ u(y)=\int_{B_1^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x. \]

Let $p_1$ and $q_1$ be the numbers defined by

(4.4)\begin{equation} \frac{1}{p_1}=\frac{n}{n-1}\left(\frac{r}{p}+\frac{1}{a}-\frac{\alpha+\beta}{n}\right),\quad \frac{1}{q_1}=\frac{n}{n-1}\left(\frac{r}{q}+\frac{1}{a}-\frac{\alpha+\beta}{n}\right). \end{equation}

Then, it follows from inequality (1.6) that

(4.5)\begin{align} & \|u\|_{L^{p_1}(B^{n-1}_1)}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}, \end{align}
(4.6)\begin{align} & \|u\|_{L^{q_1}(B^{n-1}_1)}\leq C(n,\alpha,\beta,q,r,a,b,\epsilon)\|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{q}(B^{+}_{1})}. \end{align}

Given $0<\delta _1<\delta _2\leq \frac {1}{2}$, for $x\in B^{+}_{\delta _2}$, we have

\begin{align*} v(x)& \leq\int_{B_{{(\delta_1+\delta_2)}/{2}}^{n-1}}\frac{ x_n^{\beta} U(y)u^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y +\int_{B_1^{n-1}\setminus{B_{{(\delta_1+\delta_2)}/{2}}^{n-1}}}\frac{ x_n^{\beta} U(y)u^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y+h(x)\\ & :=I_1(x)+I_2(x)+h(x). \end{align*}

By (4.3) and (4.4), we deduce that

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{b}+\frac{1}{q_1r}-\frac{\alpha+\beta-1}{n-1}\right), \]

which combines with (1.5) and the Hölder inequality, it yields that

\[ \|I_1\|_{L^{q}(B_{\delta_1}^{+})}\leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})} \|u\|^{{1}/{r}}_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})}. \]

Since $p>\frac {n}{n-\alpha -\beta }$, it follows from the Hölder inequality and (4.5) that

\begin{align*} I_2(x)& \leq \frac{C(n,\alpha,\beta)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|u\|^{{1}/{r}}_{L^{p_1}(B^{n-1}_1)}\\ & \leq \frac{C(n,\alpha,\beta,p,r,a,b)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|v\|_{L^{p}(B_1^{+})}. \end{align*}

Then, we have

(4.7)\begin{align} \begin{aligned} \|v\|_{L^{q}(B_{\delta_1}^{+})} & \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|u\|^{{1}/{r}}_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})}\\ & \quad+ \frac{C(n,\alpha,\beta,p,r,a,b)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|v\|_{L^{p}(B_1^{+})} +\|h\|_{L^{q}(B^{+}_{{1}/{2}})}. \end{aligned} \end{align}

On the other hand, for $y\in B_{{(\delta _1+\delta _2)}/{2}}^{n-1}$, we derive

\begin{align*} u(y)& = \int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\int_{B_1^{+}\setminus{B_{\delta_2}^{+}}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x\\ & \leq\int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\frac{C(n,\alpha,\beta)}{(\delta_2-\delta_1)^{n-\alpha-\beta}}\int_{B_1^{+}\setminus{B_{\delta_2}^{+}}}V(x)v^{r}(x){\rm d}x\\ & \leq\int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\frac{C(n,\alpha,\beta,a,b,p,r)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}. \end{align*}

Combining this and inequality (4.6), we obtain

(4.8)\begin{equation} \begin{aligned}\|u\|_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})} & \leq C(n,\alpha,\beta,a,b,p,r) \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{q}(B^{+}_{1})}\\ & \quad+ \frac{C(n,\alpha,\beta,a,p,r)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}. \end{aligned} \end{equation}

By (4.7) and (4.8), we see

\begin{align*} \|v\|_{L^{q}(B_{\delta_1}^{+})}& \leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\left(\frac{1}{(\delta_2-\delta_1)^{n-\alpha-\beta}} +\frac{1}{(\delta_2-\delta_1)^{n-\alpha-\beta}}\right)\|v\|_{L^{p}(B_1^{+})}\\ & \quad+ \frac{1}{2}\|v\|_{L^{q}(B_{\delta_2}^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}, \end{align*}

if $\epsilon$ is small enough. One can employ the usual iteration procedure (see [Reference Han and Lin32]) to obtain

(4.9)\begin{equation} \|v\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon) \big(\|v\|_{L^{p}(B_1^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}\big). \end{equation}

For $v,\, h\in L^{p}(B^{+}_1)$, we will show inequality (4.9) still holds. Let $0\leq \eta (x)\leq 1$ be the measurable function such that

\[ v(x)\leq\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\Big[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\Big]^{{1}/{r}}{\rm d}y+\eta(x)h(x),\quad \forall x\in B_1^{+}. \]

Define a map $T_1$ by

\[ T_1(\varphi)(x)\leq\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\left[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)|\varphi(z)|^{r}}{|z-y|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}y. \]

Choosing small enough $\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)$, in view of the integral inequality (1.5), we have

\begin{align*} & \|T_1(\varphi)\|_{L^{p}(B_{1}^{+})}\\& \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|\varphi\|_{L^{p}(B_1^{+})}\leq\frac{1}{2}\|\varphi\|_{L^{p}(B_1^{+})},\\ & \|T_1(\varphi)\|_{L^{q}(B_{1}^{+})}\\& \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|\varphi\|_{L^{q}(B_1^{+})}\leq\frac{1}{2}\|\varphi\|_{L^{q}(B_1^{+})}. \end{align*}

Furthermore, one can utilize the Minkowski inequality to obtain that for $\varphi,\, \psi \in L^{p}(B^{+}_1)$,

\[ |T_1(\varphi)(x)-T_1(\psi)(x)|\leq T_1(|\varphi-\psi|)(x),\ \ x\in B^{+}_1, \]

which implies

\[ \|T_1(\varphi)-T_1(\psi)\|_{L^{p}(B_{1}^{+})}\leq \|T_1(|\varphi-\psi|)\|_{L^{p}(B_{1}^{+})}\leq \frac{1}{2}\|\varphi-\psi\|_{L^{p}(B_{1}^{+})}. \]

Similarly, we also obtain

\[ \|T_1(\varphi)-T_1(\psi)\|_{L^{q}(B_{1}^{+})}\leq \frac{1}{2}\|\varphi-\psi\|_{L^{q}(B_{1}^{+})}. \]

for any $\varphi,\, \psi \in L^{q}(B^{+}_1)$.

Set $h_j(x)=\min \{v(x),\, j\}$, using the regular lifting theorem with contracting operators which can be seen in [Reference Chen and Li16, Reference Ma, Chen and Li46], we may find a unique $u_j\in L^{q}(B^{+}_1)$ such that

\begin{align*} v_j(x)& =T_1(v_j)(x)+\eta(x)h_j(x)\\ & =\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\left[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)v_j^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}y+\eta(x)h_j(x),\quad \forall x\in B_1^{+}. \end{align*}

Applying a priori estimate to $v_j$, we obtain

(4.10)\begin{equation} \|v_j\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon) \big(\|v_j\|_{L^{p}(B_1^{+})}+\|h_j\|_{L^{q}(B^{+}_{{1}/{2}})}\big). \end{equation}

Observing that

\[ v(x)=T_1(v)(x)+\eta(x)h(x), \]

then we see that

\begin{align*} \|v_j-v\|_{L^{p}(B_{1}^{+})}& \leq\|T_1(v_j)-T_1(v)\|_{L^{p}(B_{1}^{+})}+\|h_j-h\|_{L^{p}(B_{1}^{+})}\\ & \leq\frac{1}{2}\|v_j-v\|_{L^{p}(B_{1}^{+})}+\|h_j-h\|_{L^{p}(B_{1}^{+})}. \end{align*}

Hence,

\[ \|v_j-v\|_{L^{p}(B_{1}^{+})}\leq 2\|h_j-h\|_{L^{p}(B_{1}^{+})}\rightarrow 0, \text{ as } j\rightarrow\infty. \]

Taking a limit process in inequality (4.10), we conclude that

\[ \|v\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\left(\|v\|_{L^{p}(B_1^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}\right). \]

This completes the proof of lemma 4.3.

Similarly, we also can obtain the following local regularity lemma.

Lemma 4.4 Assume that $\alpha +\beta >1,$ $1< a,\,b\leq \infty,$ $1\leq r<\infty,$ and $\tfrac {n-1}{n-\alpha -\beta }< p< q<\infty$ satisfy

(4.11)\begin{equation} \frac{\alpha+\beta-1}{n-1}<\frac{r}{q}+\frac{1}{a}<\frac{r}{p}+\frac{1}{a}<1,\quad \frac{n-1}{ar}+\frac{n}{b}=\frac{\alpha+\beta-1}{r}+(\alpha+\beta). \end{equation}

Suppose that $u,\,g\in L^{p}(B^{n-1}_R),$ $V\in L^{b}(B_R^{+})$ and $U\in L^{a}(B_R^{n-1})$ are all nonnegative functions with $g|_{B^{n-1}_{R/2}}\in L^{q}(B^{n-1}_{R/2}),$ and

\[ u(y)\leq\int_{B_R^{+}}\frac{x_n^{\beta} V(x)}{|x-y|^{n-\alpha}}\left[ \int_{B_R^{n-1}}\frac{x_n^{\beta} U(z)u^{r}(z)}{|z-x|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}x+g(y),\quad\forall y\in B_R^{n-1}. \]

There is a $\epsilon =\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)>0$, and $C=C(n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b,\,\epsilon )>0$ such that if

\[ \|U\|^{{1}/{r}}_{L^{b}(B_R^{n-1})}\|V\|_{L^{a}(B_R^{+})}\leq \epsilon(n,\alpha,\beta,p,q,r,a,b), \]

then,

\begin{align*} & \|u\|_{L^{q}(B^{n-1}_{R/4})}\\& \quad \leq C(n,\alpha,\beta,p,q,r,a,b,\epsilon)\big(R^{{(n-1)}/{q}-{(n-1)}/{p}} \|u\|_{L^{p}(B^{n-1}_{R})}+\|g\|_{L^{q}(B^{n-1}_{R/2})}\big). \end{align*}

Based on lemmas Reference Chen and Li4.3 and Reference Christ, Liu and Zhang4.4, we prove lemma 4.2. For $R>0$, define

\[ u_R(y)=\int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x,\quad v_R(x)=\int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y. \]

By (4.2), we have

\[ u(y)=\int_{B_R^{+}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x+u_R(y),\quad v(x)=\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y+v_R(x). \]

We first verify that if $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, then

\[ v\in L^{\kappa+1}_{loc}(\overline{\mathbb{R}^{n}_+}),\quad v_R\in L^{\infty}_{loc} (B_R^{+}\cup B_R^{n-1}). \]

Indeed, since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, we see $u<\infty$, a.e. on $\partial \mathbb {R}^{n}_+$. This implies $v<\infty$, a.e. on $\mathbb {R}^{n}_+$. Hence there exists an $x^{0}=(x_1^{0},\,x_2^{0},\,\ldots,\,x_n^{0})\in B_R^{+}$ and $x_n^{0}>\frac {R}{4}$ such that $v(x^{0})<\infty$. It follows that

\begin{align*} \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y & \leq c\int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{(x^{0}_n)^{\beta} u^{\theta}(y)}{|x^{0}-y|^{n-\alpha}}{\rm d}y\\ & \leq c v(x^{0})<\infty. \end{align*}

For $0 <\delta < 1$, $x \in B^{+}_{\delta R}$, it holds,

\[ v_R(x)\leq\frac{c R^{\beta}}{(1-\delta)^{n-\alpha}} \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y, \]

which implies that

\[ v_R\in L^{\infty}_{loc} (B_R^{+}\cup B_R^{n-1}). \]

Thanks to the integral inequality (1.5) with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we derive that

\[ \left[\int_{\mathbb{R}^{n}_+}\left( \int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y \right)^{\kappa+1}{\rm d}x\right]^{{1}/{(\kappa+1)}} \leq \|u\|^{\theta}_{L^{\theta+1}(B^{n-1}_R)}<\infty. \]

Hence,

\[ v\in L^{\kappa+1}_{loc}(B_R^{+}\cup B_R^{n-1}). \]

Since $R$ is arbitrary, we deduce that

\[ v\in L^{\kappa+1}_{loc}(\overline{\mathbb{R}^{n}_+}). \]

We now turn to verify that $u_R\in L^{\infty }_{loc}(B^{n-1}_R).$ Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, there is a $y_0\in B^{n-1}_{{R}/{2}}$ such that $u(y_0)<\infty$. Thus,

\begin{align*} \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x|^{n-\alpha}}{\rm d}x & \leq c\int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y_0|^{n-\alpha}}{\rm d}x\\ & \leq c u(y_0)<\infty. \end{align*}

For $0 <\delta < 1$, $x \in B^{n-1}_{\delta R}$, one can calculate that

\[ u_R(y)=\frac{c }{(1-\delta)^{n-\alpha}} \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x|^{n-\alpha}}{\rm d}x<\infty, \]

which leads to $u_R\in L^{\infty }_{loc}(B^{n-1}_R).$

To prove the regularity of $u$, we discuss two cases.

Case 1. $\frac {\alpha +\beta -1}{n-\alpha -\beta }<\theta <\frac {n+\alpha +\beta -2}{n-\alpha -\beta }$.

Since $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$ and $\theta <\frac {n+\alpha +\beta -2}{n-\alpha -\beta }$, we have $\kappa >\frac {n+\alpha +\beta }{n-\alpha -\beta }$. Then one can deduce that

\[ \kappa-\frac{\alpha+\beta}{n}(\kappa+1)>\frac{1}{\theta}, \quad \text{and } \kappa-\frac{\alpha+\beta}{n}(\kappa+1)>1. \]

Hence, we choose a fixed number $r$ such that

\[ 1<\kappa-\frac{\alpha+\beta}{n}(\kappa+1)\leq r\leq\kappa, \text{ and } r>\frac{1}{\theta}, \]

then it follows that

\[ u^{{1}/{r}}(y)\leq \left(\int_{B_R^{+}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x\right)^{{1}/{r}} +u^{{1}/{r}}_R(y). \]

Then,

\[ v(x)\leq\int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)}{|x-y|^{n-\alpha}} \left(\int_{B_R^{+}}\frac{z_n^{\beta} v^{\kappa-r}(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\right)^{{1}/{r}}{\rm d}y +h_R(x), \]

where

\[ h_R(x)=\int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)u_R^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y +v_R(x). \]

Since $u\in L^{\infty }_{loc}(\partial \mathbb {R}^{n}_+)$, for any $x\in B^{+}_R$, it holds,

\[ \int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)u_R^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y \leq \|u_R\|_{L^{\infty}(B^{n-1}_R)} \int_{B^{n-1}_R} \frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y. \]

It follows from inequality (1.5) and $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ that

\[ h_R\in L^{q_0}(B_R^{+}\cup B_R^{n-1}), \]

where $\frac {1}{q_0}=\frac {1}{\kappa +1}-\frac {n-1}{n}\frac {1}{r(\theta +1)}.$ For $\epsilon > 0$ small enough, one can choose $\kappa -\frac {\alpha +\beta }{n}(\kappa +1)+\epsilon >1+\epsilon$ such that

\[ q_0=\frac{rn(\kappa+1)}{rn-(k+1)(n-1)({1}/{(\theta+1)})}=\frac{rn(\kappa+1)}{n\epsilon}>\frac{\kappa+1}{\epsilon} \]

can be any large number when we choose $\epsilon$ small enough. Hence, it follows that $h_R\in L^{q}(B_R^{+}\cup B_R^{n-1})$ for any $q<\infty$.

Let

\[ a=\frac{\kappa+1}{\kappa-r},\quad b=\frac{\theta+1}{\theta-{1}/{r}}, \ p=\kappa+1>\frac{n}{n-\alpha-\beta}, \]

which combines with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we obtain

\[ \frac{n}{ar}+\frac{n-1}{b}=\frac{\alpha+\beta}{r}+(\alpha+\beta-1),\quad \frac{r}{p}+\frac{1}{a}=\frac{\kappa}{\kappa+1}<1. \]

Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}_{loc}(\overline {\mathbb {R}^{n}_+})$, one can choose $q$ such that $q\in (\kappa +1,\,\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)})$, then it is easy to check that $\tfrac {r}{q}+\tfrac {1}{a}>\tfrac {\alpha +\beta }{n}$. It follows from lemma 4.3 that $v|_{B_{{R}/{4}}^{+}}\in L^{q}({B^{+}_{{R}/{4}}})$. Notice that $\frac {n\kappa }{\alpha +\beta }<\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)}$. For $q\in (\tfrac {n\kappa }{\alpha +\beta },\,\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)})$, we have

\begin{align*} u(y)& \leq R^{\beta}\left(\int_{B^{+}_{{R}/{4}}}|x-y|^{{((\alpha-n)q)}/{(q-\kappa)}}{\rm d}x\right)^{{(q-\kappa)}/{q}} \|v\|^{\kappa}_{L^{q}(B^{+}_{{R}/{4}})}+u_{{R}/{4}}(y)\\ & \leq cR^{\alpha+\beta-n+({(n(q-k))}/{q})}\|v\|^{\kappa}_{L^{q}(B^{+}_{{R}/{4}})}+u_{{R}/{4}}(y)<\infty, \end{align*}

which implies that

\[ u|_{B^{n-1}_{{R}/{8}}}\in L^{\infty}(B^{n-1}_{{R}/{8}}). \]

Since every point may be viewed as a centre, we see $u\in L^{\infty }_{loc}(\partial \mathbb {R}^{n}_+)$, and hence $v\in L^{\infty }_{loc}(\overline {\mathbb {R}^{n}_+})$.

For any $R>0$, one can apply

\[ \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y<\infty, \text{ and } \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y_0|^{n-\alpha}}{\rm d}x<\infty \]

to obtain $v_R \in C^{\infty }(B^{+}_R\cup B_R^{n-1})$ and $u_R \in C^{\infty }(B_R^{n-1})$ which yields that $u \in C^{\gamma }_{loc}(\partial \mathbb {R}^{n}_+)$ for $0 <\gamma < 1$. By the standard potential theory (see [Reference Gilbarg and Trudinger29], chap. 4) and bootstrap method, we see that $(u,\, v) \in C^{\infty }(\partial \mathbb {R}^{n}_+)\times C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

Case 2. $\frac {n+\alpha +\beta -2}{n-\alpha -\beta }\leq \theta <\infty$.

Since $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, it is easy to check that

\[ \theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)>\frac{1}{\kappa},\text{ and } \theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)\geq 1. \]

Choosing a fixed number $r$ satisfying

\[ 1\leq\theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)\leq r\leq\theta, \text{ and } r>\frac{1}{\kappa}, \]

then it follows that

\[ v^{{1}/{r}}(x)\leq \left(\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y\right)^{{1}/{r}} +v^{{1}/{r}}_R(x). \]

Hence,

\[ u(y)\leq\int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)}{|x-y|^{n-\alpha}} \left(\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(z)}{|x-z|^{n-\alpha}}{\rm d}z\right)^{{1}/{r}}{\rm d}x +g_R(y), \]

where

\[ g_R(y)=\int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)v_R^{{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x +u_R(y). \]

For any $y\in B^{n-1}_R$, it holds,

\[ \int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)v_R^{{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x \leq \|v_R\|_{L^{\infty}(B^{+}_R)} \int_{B^{+}_R} \frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x. \]

It follows from inequality (1.6) that $g_R\in L^{q_1}(B_R^{n-1})$ with $q_1$ given by

\[ \frac{1}{q_1}=\frac{1}{\theta+1}-\frac{n}{n-1}\frac{1}{r(\kappa+1)}. \]

Let

\[ a=\frac{\theta+1}{\theta-r},\quad b=\frac{\kappa+1}{\kappa-{1}/{r}}, \quad p=\theta+1>\frac{n-1}{n-\alpha-\beta}, \]

which combines with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we obtain

\[ \frac{n-1}{ar}+\frac{n}{b}=\frac{\alpha+\beta-1}{r}+(\alpha+\beta),\quad \frac{r}{p}+\frac{1}{a}=\frac{\theta}{\theta+1}<1. \]

Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}_{loc}(\overline {\mathbb {R}^{n}_+})$, one can choose $q$ such that

\[ q\in \left(\theta+1,\frac{r(n-1)(\theta+1)}{(\alpha+\beta-1)(\theta+1)-(n-1)(\theta-r)}\right), \]

then it is easy to check that $\frac {r}{q}+\frac {1}{a}>\frac {\alpha +\beta -1}{n-1}$. It follows from lemma 4.4 that $u|_{B^{n-1}_{{R}/{4}}}\in L^{q}({B^{n-1}_{{R}/{4}}})$. Arguing this as we did in case 1, and by the standard bootstrap method, we conclude that $(u,\, v) \in C^{\infty }(\partial \mathbb {R}^{n}_+)\times C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

5. The proof of theorem 1.7

In this section, we investigate the necessary and sufficient condition for the existence of nonnegative nontrivial solutions to the following integral system:

(5.1)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}}u^{\theta}(y) {\rm d}y, & x\in\mathbb{R}^{n}_+. \end{cases} \end{equation}

From theorems Reference Chen, Lu and Tao3.1 and Reference Chen4.1, to obtain the proof of theorem 1.7, it is sufficient to prove the following theorem.

Theorem 5.1 For $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1,$ $\theta >0,$ $\kappa >0,$ assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial $C^{1}$ solutions of (5.1), then a necessary condition for $\theta$ and $\kappa$ is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

Proof. Assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial solutions of the integral system (5.1). One can apply the integration by parts to obtain

\begin{align*} & \int_{B_R^{n-1}} u^{\theta}(y)(y\nabla u(y)){\rm d}y\\ & \quad= \frac{1}{\theta+1}\int_{B_R^{n-1}} y\nabla (u^{\theta+1}(y)){\rm d}y\\ & \quad= \frac{R}{\theta+1}\int_{\partial B_R^{n-1}}u^{\theta+1}(y){\rm d}\sigma-\frac{n-1}{\theta+1}\int_{B_R^{n-1}} u^{\theta+1}(x){\rm d}x. \end{align*}

Similarly, one can also derive that

\begin{align*} & \int_{B_R^{+}} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad = \frac{R}{\kappa+1}\int_{\{\partial B_R^{+}\cap x_n>0\}}v^{\kappa+1}(x){\rm d}\sigma-\frac{n}{\kappa+1}\int_{B_R^{+}} v^{\kappa+1}(x){\rm d}x.\\ \end{align*}

It follows from $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ that there exists $R=R_j\rightarrow +\infty$ such that

\[ R_j\int_{\partial B_{R_j}^{n-1}} u^{\theta+1}(y){\rm d}\sigma\rightarrow 0,\quad R_j\int_{\{\partial B_{R_j}^{+}\cap x_n>0\}} v^{\kappa+1}(x){\rm d}\sigma\rightarrow 0. \]

Therefore, we get

(5.2)\begin{equation} \begin{aligned} & \int_{\partial\mathbb{R}^{n}_+} u^{\theta}(y)(y\nabla u(y)){\rm d}y+\int_{\mathbb{R}^{n}_+} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad={-}\frac{n-1}{1+\theta}\int_{\partial\mathbb{R}^{n}_+} u^{1+\theta}(x){\rm d}x-\frac{n}{1+\kappa}\int_{\mathbb{R}^{n}_+} v^{1+\kappa}(x){\rm d}x. \end{aligned} \end{equation}

On the other hand, one can calculate that

\begin{align*} \nabla u(y) y& =\frac{{\rm d}[u(\rho y)]}{{\rm d}\rho}\Big|_{\rho=1}\\ & ={-}(n-\alpha)\int_{\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n+2-\alpha}} [(y-x) y]v^{\kappa}(x){\rm d}x, \end{align*}

and

\begin{align*} \nabla v(x) x& =\frac{{\rm d}[v(\rho x)]}{{\rm d}\rho}\Bigg|_{\rho=1}\\ & ={-}(n-\alpha)\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n+2-\alpha}}[(y-x) x]u^{\theta}(y){\rm d}y\\ & \quad+ \beta\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y){\rm d}y. \end{align*}

It follows from Fubini's theorem that

\begin{align*} & \int_{\partial\mathbb{R}^{n}_+} u^{\theta}(y)(y\nabla u(y)){\rm d}y+\int_{\mathbb{R}^{n}_+} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad= (\alpha+\beta-n)\int_{\mathbb{R}^{n}_+}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y)v^{\kappa}(x){\rm d}y{\rm d}x\\ & \quad= (\alpha+\beta-n)\int_{\partial\mathbb{R}^{n}_+}u^{\theta+1}(y){\rm d}y\\ & \quad= (\alpha+\beta-n)\int_{\mathbb{R}^{n}_+}v^{\kappa+1}(x){\rm d}x. \end{align*}

This together with (5.2) implies that $\frac {n-1}{\theta +1}+\frac {n}{\kappa +1}=n-\alpha -\beta$.

Acknowledgements

The author is supported by the NNSF of China (No. 12261041), the Natural Foundation of Jiangxi Province (No. 20202BABL211001), the Educational Committee of Jiangxi Province (No. GJJ211101) and the Fundamental Research Funds for the Central Universities (No. 2020QNBJRC005).

References

Beckner, W.. Sharp Sobolev inequality on the sphere and the Moser–Trudinger inequality. Ann. Math. 138 (1993), 213–242.CrossRefGoogle Scholar
Beckner, W.. Weighted inequalities and Stein–Weiss potentials. Forum Math. 20 (2008), 587–606.CrossRefGoogle Scholar
Brascamp, H. J. and Lieb, E. H.. Best constants in Young's inequality, its converse and its generalization to more than three functions. Adv. Math. 20 (1976), 151–173.CrossRefGoogle Scholar
Brascamp, H. J., Lieb, E. H. and Luttinger, J. M.. A general rearrangement inequality for multiple integrals. J. Funct. Anal. 17 (1974), 227–237.CrossRefGoogle Scholar
Brezis, H. and Kato, T.. Remarks on the Schrödinger operator with singular complex potentials. J. Math. Pures Appl. 58 (1979), 137–151.Google Scholar
Brezis, H. and Lieb, E. H.. A relation between pointwise convergence of functions and convergence of functionals. Proc. Am. Math. Soc. 88 (1983), 486–490.CrossRefGoogle Scholar
Carleman, T.. Zur theorie de minimalflächen. Math. Z. 9 (1921), 154–160.CrossRefGoogle Scholar
Carlen, E. and Loss, M.. Extremals of functionals with competing symmetries. J. Funct. Anal. 88 (1990), 437–456.CrossRefGoogle Scholar
Chen, L., Liu, Z. and Lu, G.. Symmetry and regularity of solutions to the weighted Hardy–Sobolev type system. Adv. Nonlinear Stud. 16 (2016), 1–13.CrossRefGoogle Scholar
Chen, L., Liu, Z., Lu, G. and Tao, C.. Reverse Stein–Weiss inequalities and existence of their extremal functions. Trans. Am. Math. Soc. 370 (2018), 8429–8450.CrossRefGoogle Scholar
Chen, L., Liu, Z., Lu, G. and Tao, C.. Stein–Weiss inequalities with the fractional Poisson kernel. Rev. Mat. Iberoam. 36 (2020), 1289–1308.CrossRefGoogle Scholar
Chen, L., Lu, G. and Tao, C.. Hardy–Littlewood–Sobolev inequalities with the fractional Poisson kernel and their applications in PDEs. Acta Math. Sin. (Engl. Ser.) 35 (2019), 853–875.CrossRefGoogle Scholar
Chen, L., Lu, G. and Tao, C.. Existence of extremal functions for the Stein–Weiss inequalities on the Heisenberg group. J. Funct. Anal. 277 (2019), 1112–1138.CrossRefGoogle Scholar
Chen, S.. A new family of sharp conformally invariant integral inequalities. Int. Math. Res. Not. IMRN 5 (2014), 1205–1220.CrossRefGoogle Scholar
Chen, W. and Li, C.. The best constant in a weighted Hardy–Littlewood–Sobolev inequality. Proc. Am. Math. Soc. 136 (2008), 955–962.CrossRefGoogle Scholar
Chen, W. and Li, C.. Methods on Nonlinear Elliptic Equations, AIMS Book Series on Diff. Equ. and Dyn. Sys., Vol. 4 (USA: Springfield, 2010).Google Scholar
Chen, W., Jin, C., Li, C. and Lim, J.. Weighted Hardy–Littlewood–Sobolev inequalities and systems of integral equations. Discrete Contin. Dyn. Syst. Suppl. 35 (2019), 853–875.Google Scholar
Christ, M., Liu, H. and Zhang, A.. An Sharp Hardy–Littlewood–Sobolev inequalities on the octonionic Heisenberg group. Calc. Var. Partial Differ. Equ. 55 (2016), 11.CrossRefGoogle Scholar
Christ, M., Liu, H. and Zhang, A.. An Sharp Hardy–Littlewood–Sobolev inequalities on quaternionic Heisenberg groups. Nonlinear Anal. 130 (2016), 361–395.CrossRefGoogle Scholar
Dai, W. and Liu, Z.. Classification of positive solutions to a system of Hardy–Sobolev type equations. Acta Math. Sci. Ser. B (Engl. Ed.) 37 (2017), 1415–1436.Google Scholar
Dou, J., Guo, Q. and Zhu, M.. Subcritical approach to sharp Hardy–Littlewood–Sobolev type inequalities on the upper half space. Adv. Math. 312 (2017), 1–45.CrossRefGoogle Scholar
Dou, J. and Zhu, M.. Sharp Hardy–Littlewood–Sobolev inequality on the upper half space. Int. Math. Res. Not. IMRN 3 (2015), 651–687.CrossRefGoogle Scholar
Dou, J. and Zhu, M.. Reversed Hardy–Littlewood–Sobolev Inequality. Int. Math. Res. Not. IMRN 19 (2015), 9696–9726.CrossRefGoogle Scholar
Folland, G. B. and Stein, E. M.. Estimates for the $\partial H$ complex and analysis on the Heisenberg group. Commun. Pure Appl. Math. 27 (1974), 429–522.CrossRefGoogle Scholar
Frank, R. L. and Lieb, E. H.. Inversion positivity and the sharp Hardy–Littlewood–Sobolev inequality. Calc. Var. Partial Differ. Equ. 39 (2010), 85–99.CrossRefGoogle Scholar
Frank, R. L. and Lieb, E. H., A new rearrangement-free proof of the sharp Hardy–Littlewood–Sobolev inequality, Spectral Theory, Function Spaces and Inequalities (B. M. E. A Brown, ed.), Oper. Theory Adv. Appl. Vol. 219 (Basel: Birkh auser, 2012), 55–67.CrossRefGoogle Scholar
Frank, R. L. and Lieb, E. H.. Sharp constants in several inequalities on the Heisenberg group. Ann. Math. 176 (2012), 349–381.Google Scholar
Gao, F., Liu, H., Moroz, V. and Yang, M.. High energy positive solutions for a coupled Hartree system with Hardy–Littlewood–Sobolev critical exponents. J. Differ. Equ. 287 (2021), 329–375.CrossRefGoogle Scholar
Gilbarg, D. and Trudinger, N. S.. Elliptic Partial Differential Equations of Second Order, 2nd ed. Fundamental Principles of Mathematical Science, Vol. 224 (Berlin: Springer, 1983).Google Scholar
Gluck, M.. Subcritical approach to conformally invariant extension operators on the upper half space. J. Funct. Anal. 278 (2020), 108082.CrossRefGoogle Scholar
Gross, L.. Logarithmic Sobolev inequalities. Am. J. Math. 97 (1976), 1061–1083.CrossRefGoogle Scholar
Han, Q. and Lin, F. H., Elliptic Partial Differential Equations, Courant Lecture Notes in Mathematics, Vol. 1. New York University, Courant Institute of Mathematical Sciences (New York; American Mathematical Society, Providence, RI, 1997).Google Scholar
Hang, F., Wang, X. and Yan, X.. Sharp integral inequalities for harmonic functions. Commun. Pure Appl. Math. 61 (2008), 54–95.CrossRefGoogle Scholar
Han, X., Lu, G. and Zhu, J.. Hardy–Littlewood–Sobolev and Stein–Weiss inequalities and integral systems on the Heisenberg group. Nonlinear Anal. 75 (2012), 4296–4314.CrossRefGoogle Scholar
Han, Y. and Zhu, M.. Hardy–Littlewood–Sobolev inequalities on compact Riemannian manifolds and applications. J. Differ. Equ. 260 (2016), 1–25.CrossRefGoogle Scholar
Hardy, G. H. and Littlewood, J. E.. Some properties of fractional integrals. Math. Z. 27 (1928), 565–606.CrossRefGoogle Scholar
Hu, Y. and Liu, Z.. Classification of positive solutions for an integral system on the half space. Nonlinear Anal. 199 (2020), 1–18.CrossRefGoogle Scholar
Jerison, D. and Lee, J.. Extremals for the Sobolev inequality on the Heisenberg group and the CR Yamabe problem. J. Am. Math. Soc. 1 (1988), 1–13.CrossRefGoogle Scholar
Lieb, E. H.. Sharp constants in the Hardy–Littlewood–Sobolev and related inequalities. Ann. Math. 118 (1983), 349–374.CrossRefGoogle Scholar
Lions, P.. The concentration-compactness principle in the calculus of variations. The limit case. I. Rev. Mat. Iberoamericana 1 (1985), 145–201.CrossRefGoogle Scholar
Lions, P.. The concentration-compactness principle in the calculus of variations. The limit case. II. Rev. Mat. Iberoamericana 1 (1985), 45–121.CrossRefGoogle Scholar
Lieb, E. H. and Loss, M.. Analysis, 2nd ed, Graduate studies in Mathematics, Vol. 14 (Providence, RI: American Mathematical Society, 2001).Google Scholar
Liu, Z.. Symmetry and monotonicity of positive solutions for an integral system with negative exponents. Pacific J. Math. 300 (2019), 419–430.CrossRefGoogle Scholar
Li Remark on some conformally invariant integral equations, Y. Y.. The method of moving spheres. J. Eur. Math. Soc. 6 (2004), 153–180.Google Scholar
Lu, G. and Zhu, J.. Symmetry and regularity of extremals of an integral equation related to the Hardy–Sobolev inequality. Calc. Var. Partial Differ. Equ. 42 (2011), 563–577.CrossRefGoogle Scholar
Ma, C., Chen, W. and Li, C.. Regularity of solutions for an integral system of Wolff type. Adv. Math. 226 (2011), 2676–2699.CrossRefGoogle Scholar
Moroz, V. and Van Schaftingen, J.. Groundstates of nonlinear Choquard equations: Hardy–Littlewood–Sobolev critical exponent. Commun. Contemp. Math. 17 (2015), 1550005.CrossRefGoogle Scholar
Ngô, Q. A. and Nguyen, V. H.. Sharp reversed Hardy–Littlewood–Sobolev inequality on $\mathbb {R}^{n}$. Israel J. Math. 220 (2017), 189–223.CrossRefGoogle Scholar
Ngô, Q. A. and Nguyen, V. H.. Sharp reversed Hardy–Littlewood–Sobolev inequality on the half space $\mathbb {R}_+^{n}$. Int. Math. Res. Not. IMRN 20 (2017), 6187–6230.Google Scholar
Sobolev, S. L.. On a theorem in functional analysis (in Russian). Mat. Sb 4 (1938), 471–497.Google Scholar
Stein, E. M. and Weiss, G.. Fractional integrals on n-dimensional Euclidean space. J. Math. Mech. 7 (1958), 503–514.Google Scholar
Stein, E. M. and Weiss, G.. Introduction to Fourier Analysis on Euclidean Spaces, Princeton Mathematical Series, Vol. 32 (Princeton, NJ: Princeton University Press, 1971).Google Scholar