II. RELATED WORK

Several methods for inverse tone mapping have been proposed in recent years. These can be grouped according to how the dynamic range expansion problem is tackled [Reference Banterle, Artusi, Debattista and Chalmers8]:

• • Global operators. Where the same global expansion function is applied to each pixel of the LDR image. A monotonic function is usually adopted and its shape is adjusted according to the characteristics of the scene present in the image. Typical characteristics are the scene type (dark, average, bright), contrast, amount of saturated pixel, mean and median luminance, and the perceived amount of brightness in the scene [Reference Luzardo, Aelterman, Luong, Philips, Ochoa and Rousseaux7, Reference Bist, Cozot, Madec and Ducloux9–Reference Masia, Serrano and Gutierrez11].

• • Local operators. Where each region of the image is expanded in a different way, depending on a given criterion. Commonly, these methods classify different parts of the image as a region with a diffuse or a specular highlight in order to expand each one using a different expansion function. Other methods seek to identify salient objects in the image in order to expand them in a different way than in the rest of the image [Reference Wang6, Reference Huo, Yang and Li12, Reference Masia and Gutierrez13].

• • Expansion maps. Where an expansion map is used to direct the expansion of the LDR content. This expansion map is a non-binary mask that represents the weights to be used for the expansion of each pixel in the LDR image. The main difference of each method is the way how this map is created [Reference Banterle, Ledda, Debattista and Chalmers14–Reference Kovaleski and Oliveira17].

• • User-guided techniques. Where the user intervention is required to address the expansion. In these methods, the user helps to add detailed information lost in over/under-exposed areas on the original LDR image prior to being expanded [Reference Wang, Wei, Zhou, Guo and Shum18].

• • Deep Learning techniques. Where Deep Convolutional Neural Networks (CNNs) are used to automatically expand an input LDR image. Not all, but most of these methods address the dynamic range expansion in to reconstruct data that have been lost from the original signal due to clipping, quantization, tone mapping, or gamma correction [Reference Eilertsen, Kronander, Denes, Mantiuk and Unger19–Reference Zhang and Lalonde24].

In the remainder of this section, we discuss some very specific inverse tone mapping techniques. Kovaleski and Oliveira [Reference Kovaleski and Oliveira17] proposed a reverse tone mapping operator for images and videos which can deal with images with a wide range of exposure conditions. This technique is based on the automatic computation of a brightness enhancement function, also called expand-map, that determines areas where image information may have been lost and fills these regions using a smooth function. Likewise, Huo et al. [Reference Huo, Yang, Dong and Brost15] proposed a method that considers the perceptual properties of the Human Visual System (HVS). Their approach is based on the fact that the perceived brightness of each point in an image is not determined by its absolute luminance, but by a complex sequence of steps that happens in the HVS. According to the authors, due to this and the fact that for applications of rendering a believable HDR impression (entertainment/TV) rather than a scientifically HDR recovery, it is not necessary to create a perfect reconstruction of the original HDR image. Rather, an approximation that does not produce a significant change in the visual sensation experienced by the observers should be enough, which can be achieved by imitating the local retina response. Their algorithm first extracts the luminance and chrominance channels from the LDR input image. Then, the luminance channel is used to compute the local surrounding luminance using an iterative bilateral filter. Afterward, the new expanded HDR luminance is obtained by combining the input luminance channel and the local surrounding using the local response of retina as the mathematical model, which was estimated previously from the global one. Finally, the HDR image is generated by combining the computed HDR luminance and the chrominance channel from the input LDR image. According to the authors, this method is capable to enhance the local contrast and preserve details in the resulting expanded HDR image.

Masia et al. [Reference Masia, Serrano and Gutierrez11] proposed a dynamic inverse tone mapping algorithm based on image statistics. The main assumption of this method is that input LDR images are not always correctly exposed. In this way, the authors propose to use a simple gamma curve as inverse tone mapping operator in which the gamma value is estimated from a multi-linear model that incorporates the key value [Reference Akyuz and Reinhard25], the number of overexposed pixels, and the geometric mean luminance as input parameters. Bist et al. [Reference Bist, Cozot, Madec and Ducloux9] proposed a method based on the conservation of the lighting style aesthetics. As with previous approaches, this method uses a simple gamma correction curve as an expansion operator; however, in this method, the gamma value is computed based on the median of the luminance in the input LDR image. Despite these methods produce good results across a wide range of exposure conditions; because they are based on preserving the aspect of the original LDR image, they might not always reflect the artistic intentions intrinsic to the HDR domain.

In recent studies, Endo et al. [Reference Endo, Kanamori and Mitani21] proposed an inverse tone mapping method based on Deep Learning. This is one of the first approaches that explore the use of deep CNNs. The key idea is to synthesize LDR images taken with different exposures based on supervised learning, and then reconstruct an HDR image by merging them. In a so-called learning phase, authors trained two deep neural network models by using 2D convolutional and 3D deconvolutional networks to learn the changes in exposures of the synthesized LDR images created from an HDR dataset. The first model was trained to output N up-exposure images, while the second one to output N down-exposed images. Both models were built using the same architecture but trained separately. The synthesized images used to train the models were created by simulating cameras with different camera response functions and exposures from each image in the HDR dataset. For the inverse tone mapping of a single input LDR image; first, a set of synthesized LDR images is generated using the built models, in a so-called inference phase. Then, the input LDR image and k synthesized LDR images selected systematically, s.t. $k\leq 2N$, are used to compute the final HDR image by merging them using the method proposed by Debevec and Malik [Reference Debevec and Malik26]. According to the authors, their method can reproduce not only natural tones without introducing visible noise but also the colors of saturated pixels. Likewise, Eilertsen et al. [Reference Eilertsen, Kronander, Denes, Mantiuk and Unger19] proposed a technique for HDR image reconstruction from a single exposure using deep CNNs. This method is focused on reconstructing the information that has been lost in saturated image areas, such as highlights lost due to saturation of the camera sensor, by using deep CNNs. The authors first trained the model using a dataset of HDR images with their corresponding LDR images. This dataset was generated by simulating sensor saturation for a range of cameras using random values for exposure, white balance, noise level, and camera curve. The model proposed by the authors for the HDR reconstruction is a fully convolutional deep hybrid dynamic range autoencoder network. It is defined as “hybrid” because it mixes the behavior of a classic autoencoder/decoder that transforms and reconstructs dimensional data, and a denoising autoencoder/decoder that is trained to reconstruct the original uncorrupted data. The encoder operates on the LDR input image, which contains corrupted data in saturated regions, to convert it into a latent feature representation. Then, the decoder uses this representation to generate the final HDR image by restoring corrupted data. Authors claim that their method can reconstruct a high-resolution visually convincing HDR image using only an arbitrary single exposed LDR image as an input.

In this paper, we propose a fully-automatic inverse tone mapping algorithm based on mid-level mapping. Our approach allows expanding LDR images into HDR domain with peak brightness over 1000 nits. For this, a computationally simple non-linear expansion function that takes only one parameter is used. This parameter is automatically estimated using simple first-order image statistics. Unlike [Reference Bist, Cozot, Madec and Ducloux9, Reference Masia, Serrano and Gutierrez11], our proposed expansion function offers better capabilities to increase the perceived image quality of the resulting HDR image than a gamma expansion curve. Furthermore, it can expand an LDR image into an HDR image with a peak brightness of over 1000 nits. The proposed algorithm can reach the same performance of more computationally complex methods, with the advantage that it is simple enough to be used for practical purposes such as real-time processing in embedded systems.

III. PROPOSED EXPANSION FUNCTION

To expand the dynamic range of an LDR image ($I_{LDR}$) and obtain its inverse tone-mapped HDR image counterpart ($I_{HDR}$), a method based on mid-level mapping is proposed. The tone mapper function proposed in [Reference Lottes27] is adopted. In this function, two parameters to control the shape of the curve (a and d) and two parameters to set the anchor point are defined (b and c), as follows:

(1)\begin{equation} f(L)=L_w=\frac{{L}^a}{{L}^{(ad)} b+c} \quad {\rm s.t.}\quad{L\in[0,1]}\end{equation}

Where L is the display luminance of $I_{LDR}$, $L_w$ is the expanded luminance of $I_{HDR}$ in the real-world domain, and $L_w\in [0,L_{w,\max }]$. The parameter $L_{w,\max }$ is the maximum luminance in which $I_{LDR}$ wants to be expanded, and its value usually depends on the peak brightness of the HDR display where $I_{HDR}$ will be displayed. $L_{w,\max }$ can also be considered as the relative luminance to the maximum luminance produced by the HDR display. Then, a value of 1 means that an output HDR-image that reaches the peak luminance of the HDR display is required.

As mentioned by the authors, this function was designed to result in believably real images and to allow adaptation to a large range of viewing conditions. In our case, this formulation offers sufficient freedom to change the overall brightness impression of the scene and preserve its artistic intent, without resulting in an “unbelievable” image.

Figure 1 shows a graph of the proposed function. The parameters a and d allow controlling the shape of the so-called toe (contrast) and shoulder (expansion speed) of the curve, respectively. Our function offers the possibility to fine-tune the resulting inverse tone-mapped HDR image to improve its perceived image quality, for example by increasing the contrast [Reference Rempel28]. As can be seen in Fig. 1, the parameters $m_{i}$ and $m_{o}$ act as an anchor point for the curve and represent the middle-gray value defined for the input $I_{LDR}$ and the expected middle-gray value in the output $I_{HDR}$, respectively. The middle-gray value is a tone that is perceptually about halfway between black and white on a lightness scale. These parameters enable us to control the overall luminance of $I_{HDR}$. For this, the parameter $m_{i}$ is set to the middle-gray predefined for $I_{LDR}$ (e.g. 0.214 for sRGB linear LDR images), and $m_{o}$ is adjusted to the middle-gray value desired in $I_{HDR}$. In practice, this operation can be seen as mid-level mapping. A low value for the $m_{o}$ causes $I_{HDR}$ to become darker, and a higher value brighter.

Fig. 1. The shape of the proposed inverse tone mapping function. LDR input values are normalized between 0 and 1. The maximum output HDR value depends on the peak luminance in nits of the HDR display e.g. 6000, or 1 for expressing that we intend to achieve the maximum luminance supported by the display.

Considering the following constraints: $f(m_{i})=m_{o}$ and $f(1)=L_{w,\max }$. The anchor points b and c can be computed as follows:

(2)\begin{align} b& =\frac{{m_{i}}^a L_{w,\max} - m_{o}}{m_o({m_i}^{ad} - 1)L_{w,\max}}\end{align}

(3)\begin{align} c& =\frac{{m_{i}}^{ad}m_o-{m_i}^a L_{w,\max}}{m_o({m_i}^{ad}-1) L_{w,\max}} \\ & \quad{\rm s.t.}\quad{m_i, m_o \in ]0,1[} \quad{ L_{w,\max}, a, d>0}\end{align}

Note that practically these conditions imply that $m_i$ should be different from 0 (a completely black frame) and 1 (a completely white frame), both are border cases that can easily be accounted for.

As can be seen, the proposed function in equation (1) is applied only to the luminance channel in order to leave the chromatic channels of $I_{LDR}$ unaffected. Despite the existence of other methods that allow reconstructing the color image while preserving its saturation better [Reference Artusi, Pouli, Banterle and Akyüz29], we decided to use the method proposed by Mantiuk et al. [Reference Mantiuk, Tomaszewska and Heidrich30]. In this, the color image $I_{HDR}$ is reconstructed by preserving the ratio between the red, green, and blue channels as follows:

(4)\begin{equation} C_{HDR}=\left (\left (\frac{C_{LDR}}{L}-1\right )s+1\right ){L_w}\quad {\rm s.t.}\quad {s} \geqslant 1 \end{equation}

Where $C_{LDR}$ and $C_{HDR}$ denote one of the color channel values (red, green, blue) of the $I_{LDR}$ and $I_{HDR}$, respectively. The parameter s is a color saturation parameter and allows to compensate for the loss of saturation during the luminance expansion operation. By fixing the saturation (s), contrast (a), and expansion speed (d), the expand operation of $I_{LDR}$ can be performed easily using equations (1)–(4). In addition, the proposed function allows changing the peak brightness of the resulting HDR image without affecting its middle tones, which is controlled by the parameter $m_o$.

IV. FULLY-AUTOMATIC INVERSE TONE MAPPING

The proposed solution exploits the premise that an inverse tone mapping procedure should take into account the context in which the scene unfolds [Reference Akyüz, Fleming, Riecke, Reinhard and Bülthoff31]. In this sense, the context could be objectively expressed as features of the scene that are used to lead the inverse tone mapping process.

Our algorithm is based on a mid-level mapping approach. Image features are used to estimate the middle-gray level of the HDR output. For the proposed expansion function, parameters s, a, and d are fixed manually; then, the input $m_o$, the value required for the expansion operation, is automatically estimated through a multi-linear function. Our function uses simple image-statistics from the LDR image as input parameters to estimate $m_o$. Section (A) describes the scene image dataset used for training, Section (B) explains how the corresponding human-chosen inverse tone mapping parameters were obtained through psycho-visual experiment, and Section (C) describes the machine learning method (multi-linear regression) used to estimate the inverse tone mapping parameters from scene features.

A) HDR-LDR Image Dataset

The following HDR video datasets were used to create our HDR-LDR Image Dataset: Tears of Steel [32] (1 video sequence), Stuttgart HDR image dataset [Reference Froehlich, Grandinetti, Eberhardt, Walter, Schilling and Brendel33] (15 video sequences), and a short film (1 video sequence) created by the Flemish Radio and Television Broadcasting Organization (VRT). These video datasets were selected because they include video sequences that contain scenes with a wide range of contrast and lighting conditions. Additionally, they were professionally graded by experts from VRT in HDR and LDR, using a SIM2-HDR47E HDR screen [1] and a Barco type RHDM-2301 LDR screen as reference displays, respectively. Professionally graded video content refers to video sequences that have been obtained by altering and/or enhancing a master (or a raw input video) in order to be properly displayed on a specific display. This process involves an artistic step, where the “grader” manipulates the input to better express the director's artistic intentions on the final graded content, which is extremely important to offer a reliable base to the discussion on how the content is perceived by the viewers.

From each video sequence in the HDR video datasets, we obtained two professionally graded video sequences, one in LDR and another one in HDR. HDR video sequences obtained are considered as ground truth, with most of the artistic intentions that content creators consider when they work in the HDR domain. The HDR-LDR Image Dataset was created by selecting one frame per second from each professionally graded video sequence. As can be seen in Table 1, the entire dataset includes 1631 pairs of images, an LDR image and its HDR counterpart. More details about the image dataset created for this study can be seen in Table 1. Figure 2 shows examples of LDR images from the dataset.

Fig. 2. Example of LDR images obtained from the HDR-LDR image dataset. As can be seen, the dataset contains images with a wide range of contrast and lighting conditions.

Table 1. Details about the size and number of pairs included in the HDR-LDR Image Dataset.

For each LDR image in the dataset, simple first-order image statistics, most of them computed on the luminance (L), were extracted. Image statistics include the average (${L_{avg}}$), variance (${L_{var}}$), and median (${L_{med}}$), and those included in Table 2. Images in the dataset were normalized into the $[0,1]$ range and linearized by using a gamma correction curve with a power coefficient value $\alpha =2.2$, hence L was computed as follows:

(5)\begin{equation} \begin{aligned} L& =0.213R+0.715G+0.072B \\ & \qquad{\rm s.t.}\quad R,G,B\in[0,1] \end{aligned} \end{equation}

Where R, G, and B are, respectively, the red, green, and blue channel of the linearized input image. In Table 2, ${L_{\min }}$ and ${L_{\max }}$ refer to the minimum and maximum luminance value, respectively. ${N}$ is the total number of pixels in the image without outliers, and ${N_{ov}}$ is the total number of overexposed pixels. Overexposed pixels refers to those pixels where at least one color channel is greater than or equal to $(254/255)$. The contrast ${C}$ was computed using only the luminance channel in logarithmic scale.

Table 2. First-order image statistics computed.

B) Subjective study

To carry out our subjective study, we selected a subset of 180 pairs of images (an LDR and its HDR image counterpart) from the HDR-LDR Image Dataset, which corresponds to 11% of the total number of pairs in the dataset. These pairs were selectively chosen in such a way the final subset contains images with equally distributed contrast and lighting conditions.

Six different middle-gray values ($m_o$) to inverse tone map each LDR image from the subset were obtained. Six non-experts participants, between 25 and 40 years old with normal eye-sight, took part in this study. They were asked to tweak $m_o$ until they found the subjective best match between the inverse tone-mapped HDR image and its corresponding professionally graded HDR image (ground truth).

To facilitate the matching task and prevent that the differences in saturation between the two images affect our results, both, the luminance channel of the inverse tone-mapped image ($L_{EDR}$) and the luminance channel of HDR ground truth ($L_{HDR}$), were displayed on the same display, a SIM2 HDR screen, in a four split-screen pattern. Figure 3 depicts how the subjective study to obtain $m_o$ was carried out.

Fig. 3. Scheme of the subjective experiment carried out to manually obtain middle-gray out values that best match the luminance channel of the inverse tone-mapped HDR image (EDR) and its corresponding professionally graded HDR image (HDR) in the pair. The middle-gray value ($m_o$) is used by the inverse tone mapping algorithm (iTMO) to compute EDR.

As can be seen in Fig. 3, $L_{EDR}$ and $L_{HDR}$ were divided into four regions. The top-left and bottom-right regions of $L_{HDR}$ ($L_{HDR,1}$ and $L_{HDR,4}$) and the top-right and bottom-left regions of $L_{EDR}$ ($L_{EDR,2}$ and $L_{EDR,3}$) were selected and displayed on the HDR screen. A desktop application was implemented to collect the middle-gray values from the participants. In this, the participant sits 2 m from the screen changed the $m_o$ value using a slider-like control. This value was used as an input parameter to compute $L_{EDR}$ in real-time using the $m_o$ value selected by the participant. For this study, the contrast (a) and expansion speed (d) were set manually in 1.25 and 4.0, respectively, in order to increase the contrast and minimize that diffuse whites in the input produce artifacts in the output. Similarly, the saturation (s) was manually fixed in 1.25 as suggested in [Reference Bist, Cozot, Madec and Ducloux9], and ${L_{w,\max }}$ in 0.67 to specify that we want to reach only 67% of the maximum luminance of the SIM2 display (around 4000 nits). A total of 1080 $m_o$ values (180 values per participant) were collected. To avoid any light interference, this study was conducted in a light-controlled room with dim lights.

C) Multi-linear regression

As is [Reference Masia, Serrano and Gutierrez11], a multi-linear regression approach was used. We seek to model the relationship between the middle-gray value in the output ($m_o$, dependent variable) from first-order image statistics (independent variables). This allowed us to keep the model simple enough in order to have the possibility of running the proposed inverse tone mapping algorithm in real-time.

First of all, an outlier analysis was performed over the data collected in the subjective study. In this analysis, all observations that were deemed as extreme values were removed. Given the IQR (Interquartile Range) of the samples in a group (values of $m_o$ obtained for the same pair of images), an extreme value is a data point that is below $Q_1-1.5IQR$ or above $Q_3+1.5IQR$. Then, a multivariate outliers detection using Mahalanobis' distance was performed [Reference Filzmoser, Garrett and Reimann34]. Mahalanobis was used as a metric for estimating how far each case is from the center of all the variables' distributions. At the end of the outlier analysis, 47 input values were removed from the dataset. Consequently, a multi-linear regression was carried out using a step-wise regression method for variable selection. In this method, a variable is entered into the model whether the significance level of its F-value is less than a so-called entry value, and it is removed if the significance level is greater than the removal value. For this, values of 0.05 and 0.10 were used as entry and removal value. After we applied this method, the following multi-linear model was obtained:

(6)\begin{equation} m_o = 0.017+ 0.097 {L_h}+0.008{C}-0.028{P_{ov}} \end{equation}

The P-value obtained for the F-test of overall significance test was much lower than our significance level ($F(3,126.673)$, p<0.01). This test is a specific form of the F-test that compares a model with no predictors to the model obtained in the regression. It also contains the null hypothesis that the model explains zero variance in the dependent variable [Reference Draper and Smith35]. Then the lower P-value obtained means that the model explains a significant amount of the variance in the middle-gray output value ($m_o$). In other words, we can conclude that the linear combination of the geometric mean ($L_h$), contrast ($c_o$), and the percentage of overexposed pixels ($P_{ov}$) were significantly related to the middle-gray output value ($m_o$).

We also found that the $R^2$ of our model was 0.702 ($R^2=0.702$). $R^2$ is a statistical measure of how much of the variability in the outcome is explained by the independent variables [Reference Draper and Smith35]. This means that the 70% of the variation in the linear combination of the geometric mean ($L_h$), contrast ($C$), and the percentage of overexposed pixels ($P_{ov}$), is explained by the equation (6).

As can be seen in equation (6) and as was expected, the geometric mean (${L_h}$), which is the parameter that represents the perceptual overall brightness of the input image, has the highest positive relation with the middle-gray value in the output ($m_o$), followed by the contrast (${C}$). This means that LDR images with a high perceptual brightness value tend to expand into HDR images with high brightness. This is also controlled by the contrast, in the sense that LDR images with high perceptual brightness but with lower contrast are not expanded to very bright HDR images. Another interesting finding was that the negative relation between the percentage of overexposed pixels (${P_{ov}}$) and $m_o$ acts as a control parameter that avoids that an overexposed image to be expanded to a very bright HDR image. In fact, in the presence of a highly overexposed image, ${C}$ has a value close to zero, and ${L_h}$ and ${L_h}$ close to one. This results in a dimmed positive middle-gray output value ($m_o$) which is a more pleasant inverse tone-mapped HDR image.

It should be noted that equation (6) produces acceptable results for natural scenes. During an extensive testing and after a detailed analysis, it was verified that equation (6) only degenerates in very rare cases: artificial (non-natural) frames that are made up nearly completely (>98%) of monochromatic blue pixels that are overexposed.

SUPPLEMENTARY MATERIAL

Supplementary material is available at http://telin.ugent.be/~gluzardo/midlevel-itmo