2.1. Image translation using convolutional neural network

In silico labeling (ISL) was one of the first published deep-learning approaches that could determine the correlation between transmitted-light microscopy images and fluorescent images⁽Reference Brent and Boucheron ^{28 )}. It succeeded in predicting the location and intensity of nuclear labeling on bright-field images with good accuracy. The fluorescent images generated by the ISL model could be used in quantifying the dimensions of cellular structure and classifying the physiological state of cells⁽Reference Christiansen, Yang and Ando ^{11 )}. Specifically, the ISL model performed the task of pixel-to-pixel classification and Christiansen et al. used a z-stack image, consisting of z slices of bright-field images, as the input for multi-channel fluorescent image prediction. The model was inspired by the popular U-Net⁽Reference Ronneberger, Fischer and Brox ^{29 )} and inception networks⁽Reference Szegedy, Liu and Jia ^{30 )}. For each translation, five image patches were cropped out of the input bright-field image stack with different side lengths at the same centroid location and simultaneously fed into five independent input computational heads of the model, respectively; the outputs from the five computational heads were concatenated together and transmitted to the final prediction computational head of model to synthesize the fluorescent images. As each input computational head processed one side-length area, it allowed the model to learn spatial information from multiple spatial extents. The other special point of this network was that it was composed of repeated modules; the mutually independent modules had the same architecture but different parameters that controlled the number of hidden convolutional kernels. The ISL model implemented the translation piece by piece. A stack of bright-field image patches with $ 250\times 250\;{\mathrm{px}}^2 $ was fed into the model. The ISL model predicted an 8-bit gray-scale image, where each unique gray-scale value had a probability. For each pixel, these probabilities were represented by a 256-element vector. Such a process ensured accuracy; however, it sacrificed computational efficiency, as the model was large and the parameters for each repeated module were hard to tune⁽Reference Christiansen, Yang and Ando ^{11 )}.

Soon after the first trial for fluorescent image translation, Ounkomol et al. ⁽Reference Ounkomol, Seshamani, Maleckar, Collman and Johnson ^{22 )} proposed their work for label-free prediction of multi-channel immuno-fluorescence (IF) images from 3D transmitted-light or 2D electron microscopy (EM) images. As EM images contain plenty of sub-cellular structure information, this model was capable of presenting more intracellular details. The model can also automatically register EM images on large target IF images. The model for label-free prediction was a CNN based on the U-Net architecture. It contained down- and up-sampling substructures with shape concatenation processes. Compared to an ISL model, its architecture was much more intuitional and close to the initial U-Net architecture⁽Reference Ounkomol, Seshamani, Maleckar, Collman and Johnson ^{22 )}.

2.2. Image translation based on generative adversarial neural network

Image translation approaches with deep learning have gone through a breathtaking evolution in recent years. The initial attempt using CNNs was an impressive achievement, but its results tended to be blurry. This was because the loss function was minimized by averaging all plausible outputs⁽Reference Isola, Zhu, Zhou and Efros ^{20 )}. To overcome the disadvantages of the blurring issue, the generative adversarial neural network (GAN) was proposed⁽Reference Goodfellow, Pouget-Abadie and Mirza ^{16 ,} Reference Denton, Chintala, Fergus, Cortes and Lawrence ^{31 –}Reference Salimans, Goodfellow, Zaremba, Cheung, Radford, Chen, Lee and Sugiyama ^{33 )}. The basic concept for GAN was adding another CNN to the generative model for discriminating whether the output is real or fake. This adversary system was trained together and encouraged the generative network to produce more convincing results and prevent the outputs from being blurry.

The original GANs generated outputs from random vectors⁽Reference Gui, Sun, Wen, Tao and Ye ^{34 ,} Reference Yi, Walia and Babyn ^{35 )}; however, subsequent variants have since been derived. For example, Li et al. employed CycleGAN⁽Reference Zhu, Park, Isola and Efros ^{18 )} for unsupervised content-preserving transformation for optical microscopy (UTOM)⁽Reference Li, Zhang and Qiao ^{21 )}. CycleGAN employed two generators to transform images between two domains back and forth, respectively, and two discriminators were used for penalizing whether the generated images belong to each domain; images transformed forward and then transformed backward should be the same. One speciality of CycleGAN was that images from the two domains did not need to be corresponding, so it allowed for unsupervised translation⁽Reference Zhu, Park, Isola and Efros ^{18 )}. Meanwhile, the UTOM model used a saliency constraint to maintain the saliency within the images unchanged during transformation to avoid distortion of content. Generators in UTOM had the same number of input and output channels to simplify the transactions. It was shown that UTOM achieved comparable performance to the supervised CNN model without having any paired training datasets. However, the training dataset for UTOM was larger than the supervised model, otherwise, the accuracy of translation would be limited⁽Reference Li, Zhang and Qiao ^{21 )}.

UTOM achieved a great performance for image translation, but the limitation of the dataset could not be neglected. Therefore, a model that combined the advantages of supervised learning and GANs could be another solution. cGAN was a promising approach suitable for tasks with limited but paired datasets⁽Reference Mirza and Osindero ^{36 )}. The discriminator in cGANs not only judged whether the outputs were real or fake but also determined whether the inputs and outputs were consistent. As a supervised learning process, the loss of the generator was a combination of the discriminating result and the distance between the outputs and ground truth. Isola et al. ⁽Reference Isola, Zhu, Zhou and Efros ^{20 )} proposed an image-to-image translation model with cGAN. The model succeeded in image generation from sparse annotations, future frame prediction, or photo reconstruction. Their generator for cGANs was inspired by U-Net⁽Reference Ronneberger, Fischer and Brox ^{29 )}, and the discriminator used a specially designed “PatchGAN” classifier to enhance the spatial penalization⁽Reference Isola, Zhu, Zhou and Efros ^{20 ,} Reference Li and Wand ^{37 )}. This approach was widely used as the benchmark for style transfer of cellular images⁽Reference Hollandi, Szkalisity and Toth ^{38 )}.

Based on the concept of cGAN, Han et al. ⁽Reference Han and Yin ^{39 )} produced a model for cellular image translation. The model was established for the DIC to PC transition. The only difference in the architecture from the original cGAN model was the model took the predefined cell masks as additional inputs and the system consisted of two discriminators, one for DIC–PC pairs and one for mask–PC pairs. This model showed that the generator performed better compared to only DIC images for inputs.

After the success of applying cGANs in microscopy image translation, Nguyen et al. ⁽Reference Nguyen, Bui and Thai ^{23 )} designed a complex system with multiple cGANs applied on microscopy images taken from breast-cancer and bone-osteosarcoma cell lines. The system implemented not only the transformation of images from fluorescent images to PC images but also the interpretation between different fluorescent channels. The architectures for each translation model were basic pixel-to-pixel image translation networks introduced in the work of Isola et al. ⁽Reference Isola, Zhu, Zhou and Efros ^{20 )}, and they implemented a novel algorithm for evaluating the generated result. Their work achieved great success in the prediction of subcellular localization of proteins with other proteins on well-registered images. It proved that a cellular image translator based on cGAN was an effective tool for studies about relationships between proteins and organelles and to visualize the subcellular structure⁽Reference Nguyen, Bui and Thai ^{23 )}.

Lee et al. provided the DeepHCS++ model for the fluorescent image translation task⁽Reference Lee, Oh, Her and Jeong ^{40 )}. The architecture of the DeepHCS++ model consisted of two parts: transformation and refinement networks. The transformation network was similar to the label-free model of Ounkomol et al.; however, instead of using one up-sampling decoder for multi-task learning, three independent decoders were applied to produce three channels of fluorescent images, respectively. The intermediately produced fluorescent images were then fed into three U-Net shape refinement networks for the final predictions. Conditional adversarial loss calculations were only applied to the final output images rather than intermediate products. Results showed that the performance for translation-refinement networks was better than single translation networks⁽Reference Lee, Oh, Her and Jeong ^{40 )}.

2.3. Attention neural network

Images generated by GANs derived from lower-resolution feature maps and higher-level details are calculated with spatially local points. This process has a significant limitation as it only takes spatially local information for calculations⁽Reference Zhang, Goodfellow, Metaxas and Odena ^{27 ,} Reference Cordonnier ^{41 )}. However, the ISL model showed that the translation required information from multiple spatial extents of correlated bright-field images, which is why it took five scales of inputs. Fortunately, self-attention GANs (SAGANs) provided an approach for long-range dependency modeling for image generation.

Self-attention mechanisms allowed for computing dependencies within a single sequence without the restriction of position⁽Reference Vaswani, Shazeer, Parmar, Guyon, Von Luxburg and Bengio ^{26 ,} Reference Cheng, Dong and Lapata ^{42 ,} Reference Parmar, Vaswani and Uszkoreit ^{43 )}. It had been successfully used in sequence-to-sequence processing such as reading comprehension, textual analysis and language translation. This approach can be transferred to spatial analysis in image processing⁽Reference Guo, Xu and Liu ^{44 )}. Zhang et al. proposed the SAGAN, which aimed to calculate the dependencies of one position on all others through feature maps. The result showed that the SAGAN was an effective model for the calculation of long-range dependencies and upgraded the qualities of output images⁽Reference Zhang, Goodfellow, Metaxas and Odena ^{27 )}. Besides, SAGANs applied spectral normalization⁽Reference Miyato, Kataoka, Koyama and Yoshida ^{45 –}Reference Odena, Buckman and Olsson ^{47 )} to the generator and discriminator to enhance the stability of GAN training. The mechanisms of SAGAN were introduced in our model, further details will be elaborated in the next section.

3.1. Generator

The architecture of the generator is inspired by the U-Net⁽Reference Ronneberger, Fischer and Brox ^{29 )} and ResNet⁽Reference He, Zhang, Ren and Sun ^{48 )} networks. U-Net was the first network designed for biological image segmentation. It contains an encoder–decoder system linked by skip connections. The encoder consists of a down-sampling process to transform the images to feature maps with more channels but smaller sizes; on the contrary, the decoder is an up-sampling process that transforms the intermediate feature maps to the output size. Intermediate outputs from the encoder are concatenated to the decoder which enhances spatial accuracy. One difference in our model is its two independent generation paths for image translation and nuclei semantic segmentation. Each channel of the image translation output represents one health state of the nuclei, meanwhile, the nuclei segmentation path provides binary masks for each state or background. The generator is constituted by a series of sub-modules, and each sub-module corresponds to one down-sampling or up-sampling process. Additionally, there is one bottleneck module to connect the encoder and decoders. Figure 2 shows the structure of the sub-modules. Skip connections for residual learning are employed in the sub-modules to enhance the robustness of the model. Each sub-module contains three parts, a shape-invariant convolutional layer, a sampling convolutional layer, and a residual connection⁽Reference He, Zhang, Ren and Sun ^{48 )}. The final activation function layer for the image generation decoder is Tanh to restrict the range of image values between −1 and 1. The output of the mask generation process is sent into a SoftMax function to calculate the probability of pixels belonging to each category. The output of the mask generation path has one more channel than the image generation path, which is for the background probability prediction. Finally, the image generation outputs are multiplied by the probability results from the segmentation path to produce the translation result. Details of the architecture of the generator are shown in Figure 3.

Figure 2. Network architectures of sub-modules for generator. (a) Down-sampling sub-module. (b) Bottleneck sub-module. (c) Up-sampling sub-module.

Figure 3. Generator architecture. The numbers in each sub-module indicate the numbers of input and output channels, respectively. Architectures of sub-modules are presented in Figure 2, and attention modules are shown in Figure 4.

3.2. Attention-based module

Attention algorithm-based modules are utilized to learn long-distance spatial dependencies. They are inserted between the decoders to exchange spatial information during image generation. The initial attention algorithm module is self-attention which calculates the spatial dependencies within the feature maps themselves. However, we want the attention module to also share the spatial dependencies between two generation paths. As our attention module derives from the self-attention module, we will describe the structure of the self-attention module first, and then compare the improvement of our attention-based module. The self-attention network is presented in Figure 4a, and the processing steps are listed below.

1. 1. Send the input feature map stack into three 1 × 1 convolutional layers and reshape the outputs into 1D sequences, $ q $ , $ k $ , and $ v $ .

2. 2. Calculate the attention of feature $ q $ and $ k $ .

3. 3. Feed the attention output into a SoftMax activation function.

4. 4. Calculate the attention of the SoftMax output with feature $ v $ .

5. 5. Reshape the 1D sequence back to the original shape.

6. 6. Send the feature into another 1 × 1 convolutional layer to keep the self-attention module output the same shape as the input.

7. 7. Add the self-attention output back to the feature map stack and multiply with a learnable parameter.

Figure 4. Attention module architectures. (a) Architecture of the self-attention module. (b) Architecture of the cross-attention module.

In our model, our aim is to share the spatial dependency information from both generation paths. Therefore, we extract hidden features from both generation paths using in Step 1, ( $ {q}_{image} $ , $ {k}_{image} $ , and $ {v}_{image} $ from image feature domain; $ {q}_{mask} $ , $ {k}_{mask} $ , and $ {v}_{mask} $ from mask feature domain). Our attention module contains two self-attention calculation paths which remain the same as the self-attention module; at the same time, hidden feature $ {q}_{image} $ (from the image path), $ {k}_{mask} $ and $ {v}_{mask} $ (from the mask path) are sent to the algorithm from Step 2 to Step 6 to measure the correlation between decoders. This process is similar to the work in by Hou et al. ⁽Reference Hou ^{49 )}. These are concatenated with the image self-attention outputs and sent to the next layer in the image generation path. As previous work proves that a U-Net with one encoder–decoder pair is enough for biomedical image semantic segmentation⁽Reference Ronneberger, Fischer and Brox ^{29 )}, we decided not to make the mask generation process as complex as image generation. Therefore, the correlated attention algorithm is not applied to the mask generation path. The module is named cross-attention as it takes inputs from both paths and feeds them back. The cross-attention modules are inserted between feature map layers of the same size from different generation decoders. The architecture of the cross-attention module is presented in Figure 4b.

3.3. Discriminator and spectral normalization

The discriminator does not produce translation results but instead guides the training destination of the generator and self-upgrades simultaneously. Normally, the discriminator is a down-sample process which assesses whether the input is good or not. Isola et al. introduced a discriminator named PatchGAN for their image-to-image translation network⁽Reference Isola, Zhu, Zhou and Efros ^{20 )}. PatchGAN outputs N × N patches and penalizes the patches independently, where the size of patches can be much smaller than the full size of the image. They found that the PatchGAN in the discriminator performed well in enforcing correctness at high frequencies and that low-frequency correctness could be achieved using conventional methods, such as the L1 distance restrictor. The discriminator takes the concatenation of bright-field image stacks and generated translated images as the input and runs a series of convolutional calculations. The architecture of the discriminator is shown in Figure 5; it is also made up of down-sampling sub-modules which have a similar structure as the one used in the encoders of the generator. Self-attention modules are applied in our discriminator; they are inserted after the down-sampling sub-modules at higher levels.

Figure 5. Discriminator architecture. The numbers in each sub-module indicate the numbers of input and output channels, respectively. Numbers in attention boxes represent the size of input feature maps.

To improve the performance of the discriminator, Miyato et al. created a new normalization method called spectral normalization and applied it to the discriminator of their GAN⁽Reference Miyato, Kataoka, Koyama and Yoshida ^{45 )}. They discovered that spectral normalization can stabilize the training of the discriminator, as it tuned fewer extra hyper-parameters and had lower computational costs compared to other regularization techniques. In the work of Zhang et al., spectral normalization was applied to both the generator and discriminator. This achieved better performance compared to applying it only to the discriminator. Hence, we also apply spectral normalization to our discriminator and generator⁽Reference Zhang, Goodfellow, Metaxas and Odena ^{27 )}.

3.4. Loss function and evaluation metrics

The fundamental working mechanism of GANs can be understood as a competition between the generator and discriminator. The generator aims to produce outputs that fool the discriminator into being unable to distinguish whether its products are real or fake. On the contrary, the discriminator aims to train itself to not be fooled by the generator. Mathematically, this process can be regarded as finding the solution to minimum and maximum objective functions. In cGAN, the generation process is under given conditions, in the translation task the output fluorescent images need to be correlated to the input bright-field image stacks. Therefore the discriminator needs to take the input bright-field image stacks as the condition during the penalization. The objective function of cGAN is shown in Equation (1):

(1) $$ {\mathcal{L}}_{cGAN}\left(G,D\right)={\unicode{x1D53C}}_{x,y\sim {P}_{data}\left(x,y\right)}\left[\log D\left(x,y\right)\right]+{\unicode{x1D53C}}_{x\sim {P}_{data}(x)}\left[1-\log D\left(x,G(x)\right)\right], $$

where $ {P}_{data} $ is the ground truth dataset, $ x $ is the input bright-field image, and $ y $ is the target fluorescent image; $ G $ and $ D $ represent the generator and the discriminator, respectively. The generator is trained to minimize the objective function and the discriminator is trained to maximize it; the generator is described in Equation (2):

(2) $$ {G}^{\ast }=\arg \underset{G}{\min}\underset{D}{\max}\left({\mathcal{L}}_{cGAN}\left(G,D\right)+\lambda {\mathcal{L}}_{Image}(G)\right), $$

where $ {\mathcal{L}}_{Image} $ is the conventional loss between the generated outputs and the ground truth targets, such as L1 loss (details to follow). $ \lambda $ is the weight that balances the two types of losses.

The total loss for the system is a combination of adversarial loss and conventional losses. Typical conventional losses include mean absolute error loss (MAE) or mean squared error (MSE) loss, also known as L1 and L2 distances, respectively. The conventional loss function does well in penalizing the low-frequency errors which are supplementary to adversarial loss. In previously published approaches⁽Reference Isola, Zhu, Zhou and Efros ^{20 ,} Reference Lee, Oh, Her and Jeong ^{40 )}, the weight of the conventional loss is normally one or two orders of magnitude higher than the adversarial loss in cGAN, we will follow this precedent. Meanwhile, MAE losses are used rather than MSE losses in image generation as research showed that MAE-based methods cause less blurring in practice⁽Reference Isola, Zhu, Zhou and Efros ^{20 )}. Moreover, based on the experience from Hore et al., we introduce a structural similarity index measure (SSIM) distance for the conventional loss calculation⁽Reference Lee, Oh, Her and Jeong ^{40 )}. SSIM is a method for measuring the similarity between two images; it is a perception-based model that considers the degradation in structural information, which is expressed in Equation (3):

(3) $$ SSIM\left(x,y\right)=\frac{\left(2{\mu}_x\hskip0.35em {\mu}_y+{c}_1\right)\left(2{\delta}_{xy}+{c}_2\right)}{\left({\mu}_x^2+{\mu}_y^2+{c}_1\right)\left({\delta}_x^2+{\delta}_y^2+{c}_2\right)}, $$

where $ {\mu}_{\left(\cdot \right)} $ and $ {\delta}_{\left(\cdot \right)} $ are the mean and variance of $ x $ and $ y $ , $ {\delta}_{xy} $ is the covariance of $ x $ and $ y $ ; $ {c}_1 $ and $ {c}_2 $ are constant values based on the intensity range of $ x $ and $ y $ . The output value of the SSIM function is from 0 to 1, the more similar they are the higher the value is, and hence, the SSIM distance of $ x $ and $ y $ can be calculated as $ {\mathcal{L}}_{SSIM}=1- SSIM\left(y,G(x)\right) $ . Thus, the conventional loss for image generation is presented in Equation (4):

(4) $$ {\mathcal{L}}_{Image}=\left(1-\alpha \right){\mathcal{L}}_{L1}\left(y,G(x)\right)+\alpha {\mathcal{L}}_{SSIM}\left(y,G(x)\right), $$

where $ \alpha $ is the weight to balance L1 loss and the SSIM distance, $ \alpha \hskip0.35em \in \hskip0.35em \left[0,1\right] $ .

Other than losses from the image generation path, the output of the mask generation path predicts the possibilities of pixels belonging to each category. The loss of segmentation from the mask generation is defined as the classical cross-entropy. The final loss for the generator is a weighted sum of image generation losses, including conventional loss and adversarial, and mask loss, total loss calculation is shown in Equation (5):

(5) $$ {\mathcal{L}}_{Total}={\mathcal{L}}_{cGAN}+{\mu}_1{\mathcal{L}}_{Image}+{\mu}_2{\mathcal{L}}_{Mask}, $$

where, $ {\mu}_1 $ and $ {\mu}_2 $ are weights for each loss. In practice, we chose a dynamic weight for $ {\mu}_2 $ , the loss for mask generation. A higher value for $ {\mu}_2 $ was applied to ensure the mask generation reaches the convergence point first so that the mask generation path provides highly credible spatial information to the cross-attention module, and gradually lowers the weight of the mask to enhance the image generation.

In addition, the performance of the model is evaluated using three metrics: the L1 distance (i.e., the MAE), the structural similarity index (SSIM), and the standard peak signal-to-noise ratio (PSNR)⁽Reference Hou, Chang, Ma, Shan, Chen, Wallach and Larochelle ^{50 )}.

4.1. Biological experiment and microscopy operation

In this section, we further describe the methodological approach from a biological point of view, providing further motivation for the work and presenting the datasets employed.

The cells used in this study are Chinese hamster ovary (CHO)-K1 cells [ATCC-CCL-61], which are induced to undergo apoptosis, to analyze and predict the state of their nuclei. These are cultured in Ham’s F-12K (Kaighn’s) Medium (Life Technologies Limited, Gibco, Paisley, Scotland, United Kingdom) supplemented with 10% fetal bovine serum (FBS). They are grown in T-75 flasks (10 ml of culture medium) and incubated at 37°C and 5% CO₂.

1. 1. When the cells have reached minimum confluency of 60%, ~100,000 cells are seeded onto glass coverslips (13 mm diameter) in 500 μL of medium and left to adjust overnight. All rinsing steps are done with Phosphate buffered saline (PBS).

2. 2. To induce apoptosis, the cells are exposed to 500 μL of 1 μM of staurosporine (in medium, Apexbio Technology LLC, Houston, Texas, United States) and incubated for 0, 1, 3, 6, and 8 hr.

3. 3. Once the respective time has been reached, the cells are fixed in 4% Paraformaldehyde (PFA) in PBS for 30 min.

4. 4. The nuclei are stained with Hoechst solution (1:10,000 dilution in medium, Thermofisher Scientific, Pierce Biotechnology, Rockford, Illinois, United States) for 10 min.

5. 5. Finally, the coverslips are mounted onto glass slides and left to curate overnight.

The cells are imaged on a Leica SP5 confocal laser scanning microscope using an HC PL APO CS2 63×, 1.4 NA oil objective. Hoechst-stained nuclei are excited with a 405 (nm) diode laser (laser power: 7%). Single-plane bright-field images and fluorescent image stacks are taken of the nuclei. The fluorescent image stacks have a z-axis range of 7.2 μm and consist of 24 slices spaced 0.3 μm apart.

4.2. Dataset preparation

Our method is designed to produce fluorescent images together with semantic segmentation. The most important part of the image pre-processing pipeline was segmenting the fluorescent images based on the health state of each nucleus. The states of the nuclei change over time, that is, after 3 hr nuclei start to split. In the pre-processing phase, fragmented nuclei were merged using Gaussian and median filters. Subsequently, a marker-controlled watershed segmentation process was applied to separate adherent nuclei. All steps in the pre-processing pipeline are listed below and shown in Figure 6.

1. 1. Resize raw images to an appropriate size that maintains adequate information. In this work, we resize the images from $ 512\times 512\;{\mathrm{px}}^2 $ to $ 256\times 256\;{\mathrm{px}}^2 $ .

2. 2. Apply min–max normalization for each raw fluorescent image and rescale to 0 to 255 (8-bit intensity range).

3. 3. Concatenate fluorescent images along z-stack. Choose the maximum value along the z-stack for each pixel and generate the maximum image.

4. 4. Apply contrast limited adaptive histogram equalization (CLAHE)⁽Reference Pizer, Johnston, Ericksen, Yankaskas and Muller ^{51 )} to each image. Set the contrast limiting value to be 2, and the size of tiles to be $ 32\times 32\;{\mathrm{px}}^2 $ .

5. 5. Apply Gaussian smoothing to images using a $ 3\times 3\;{\mathrm{px}}^2 $ kernel. In practice, we apply it four times so that fragmented spots from the same nuclei are connected.

6. 6. Apply median filtering with a $ 5\times 5\;{\mathrm{px}}^2 $ kernel. In practice, we apply it 12 times to remove salt-and-pepper noise from the images.

7. 7. Calculate the Otsu threshold for the images and generate the mask for nuclei.

8. 8. Calculate the distance for pixels within the mask to the mask boundaries. Then select the local maximum points on the distance maps and label them as the kernel for individuals.

9. 9. Apply the watershed algorithm⁽Reference Beucher and Lantuejoul ^{52 ,} Reference Beucher ^{53 )} to segment the masks.

10. 10. For nuclei incubated in staurosporine for less than 8 hr, the number of apoptotic cells is much less than healthy cells. The split nuclei can be easily filtered out manually.

11. 11. For nuclei at 8 hr. Apply individual segmented nucleus masks to the maximum fluorescent images. Calculate the standard deviation of the intensity value within each segmented nucleus image. Use the Otsu thresholding method⁽Reference Otsu ^{54 )} to separate these nuclei into two groups. The group which has a higher value of standard deviation corresponds to fragmented nuclei and to intact nuclei. Inspect the output of the segmentation process based on standard deviation, and manually correct the splitting results.

12. 12. Individual masks belonging to the same group are added together and used to generate the masks for healthy nuclei and apoptotic nuclei.

13. 13. Apply both nuclei masks on the center layer of the z-stack fluorescent images after CLAHE. Each fluorescent image is divided into two channels. The two-channel fluorescent images are used as the ground-truth targets for training.

14. 14. For bright-field images, apply min–max normalization for rescaling and concatenation along z-stack.

Figure 6. Dataset preparation process. (a) Maximum intensity along z-stack of fluorescent images, (b) threshold and watershed segmentation output, (c) automatic classification result, (d) manually revised result, the revised individual in the yellow circle, (e) example of image dataset for training, contains bright-field images, split fluorescent images, and masks.

4.3. Dataset augmentation

Due to the limited number of images in our dataset, data augmentation was applied for the training of the model. Random flips and rotations were applied, followed by a random crop of $ 128\times 128\;{\mathrm{px}}^2 $ out of $ 256\times 256\;{\mathrm{px}}^2 $ . Random flips and rotations encourage the model to not be restricted by the orientation of the input images. A random crop of images helps individual nuclei maintain the same sizes and ratios (width to height) compared to the original images, but the locations of nuclei on images vary at each iteration. Data augmentation ensures the inputs to the model are different for every epoch and prevents over-fitting of the model.

4.4. Training details

The code was implemented using Python and PyTorch. This work was carried out using the BlueCrystal Phase 4 facility of the Advanced Computing Research Centre, University of Bristol (http://www.bristol.ac.uk/acrc/). The system is equipped with Nvidia P100 GPU with 16 GB of RAM. The optimizer for the generator and discriminator used here was the Adam optimizer with beta values of 0.5 to 0.999⁽Reference Kingma and Ba ^{55 )}, while the learning rates were $ {10}^{-5} $ and $ {10}^{-4} $ , respectively⁽Reference Heusel, Ramsauer, Unterthiner, Nessler, Hochreiter, Guyon and Von LuxBurg ^{56 )}. The batch size for training was 8. The total number of learning epochs was 4,500. Cross-validation was applied to the training dataset, with the number of images for training, validation, and testing being 66:12:8. The training and validation datasets were rearranged every 50 epochs. As mentioned above, the weight for the mask generation loss varies during each iteration; the initial weight was set to 250 and decayed 10% every 1,500 epochs and no lower than 2.5.