FireflyAxolotl

During the collection of tomography data (e.g. Electron Micrographs, X-ray CTs, etc.) multiple projections of an imaged sample are obtained for angles covering a 180 or 360 degree span. Once the projections are collected, it is possible to attain a construct a 3D image using an inverse Radon Transformation (see Figure 1). However, in specific cases either the tomogram’s projections cannot be collected for specific ranges of angles (therefore creating Missing-Wedges) or too few projections are collected to adequately sample the 180/360 degree span (Projection Undersampling). These missing projections can have adverse effects on the tomogram’s reconstruction, reducing its signal-to-noise ratio and also introducing undesirable visual artifacts. To correct for these adverse effects, we propose the use of Sinogram Inpainting Deep Neural Networks (DNNs) to inpaint (in other words, to predict and reintroduce) the uncollected projections in the tomogram’s sinograms (marked as red in Figure 2).

Figure 1. If for height position (in the above figure for Y=537) all the rows from all projections across the different angles get stacked vertically, the sinogram for the corresponding height position can be formed. The reconstruction of a single sinogram, is equivalent to the reconstruction of a XZ-slice of the overall tomogram reconstruction.

Figure 2. As seen both the Missing-Wedge and Projection Undersampling problems lead to certain areas of the sinogram not being collected (marked as red), which results in the introduction of undesirable artifacts in the reconstructions.

To train the Sinogram Inpainting DNNs, sinograms that are already affected by the Missing-Wedge or the Projection Undersampling problem are being used. The idea behind it, is to train tomogram-exclusive DNNs, that will learn how to inpaint a specific tomogram’s sinograms using the same Missing-Wedge or Projection Undersampling affected sinograms. This approach is chosen in an effort to prevent the introduction of hallucination artifacts (see Figure 3), that may occur by inpainting with a DNN trained on a different tomogram’s sinograms.

Figure 3. In this hallucination artifact example case a DNN is trained using a tomogram that contains multiple spherical particles and a metallic pin in its rotation axis. Then it is used to inpaint the sinograms of projection undersampled tomogram that contains only multiple spherical particles. After the sinogram inpaint, the metallic pin feature may be introduced by the DNN even if it is not originally present in the inference dataset. This will result in the appearance of metallic pin artifact in the reconstruction, making it an introduced hallucination artifact.

To train while using sinograms affected by the Missing-Wedge or Projection Undersampling problems, the sinograms are first split to multiple sinogram-patches. This splitting increases the number of training instances. Within these sinogram-patches, the areas which correspond to collected projections are split into two groups; the first group is given as input to the DNN, whilst the second group are artificially obscured (see Figure 4). The DNN’s goal during training is to offer inpainting predictions for the artificially obscured projections, and since for them the ground truth is available, a prediction error can be measured and the DNN can be trained. Once trained, it can be used to inpaint the uncollected projections of the sinograms.

Figure 4. From the parts of the sinograms that correspond to collected projections, some are artificially obscured before they are given as input to Sinogram Inpainting DNN during training (marked as transparent orange). During training, the DNN will then offer inpainting predictions for these areas and since the ground truth for them is known, a prediction error for them can be measured and the DNN can be trained. After training and during the inference, the predictions of the DNN for the uncollected areas of the sinograms (marked as red) can be used to inpaint them.

Our motivation for our proposed Sinogram Inpainting DNNs is for the tomogram reconstructions made by the inpainted sinograms to not replace the reconstructions made by the original sinograms. Instead, the resulting tomogram reconstructions will act as auxiliary modalities.

Namely, the resulting tomogram reconstructions from the inpainted sinograms can be used to aid human annotators and/or particle picking software to locate a high number of the particles present in the tomograms and to accurately estimate their locations and orientations. Then using the tomogram reconstructions made by the original sinograms, subtomogram averaging may offer more accurate results by utilising the high number of particles located and the accurate knowledge of their locations and orientations.

Alternatively, in tomograms where subtomogram averaging is not applicable, the tomogram reconstructions made by the inpainted sinograms may help the analysis of these tomograms by offering a denoised and de-artifacted version of their reconstructions, devoid of externally introduced hallucination artifacts.

Below in Figures 5 and 6 some early sinogram inpainting results are being presented.

Figure 5. Example of Sinogram Inpainting of Missing-Wedge(s) using a DNN. As seen by the green marks, after the reconstruction the Missing-Wedge related artifacts are significantly suppressed.

Figure 6. Example of Sinogram Inpainting of Projection Undersampled sinograms using a DNN, as seen by the green marks, after the reconstruction the Projection Undersampling related artifacts are significantly suppressed.

Project Development

Dr Dimitrios Bellos – Research Software Engineer in the Artificial Intelligence and Informatics Theme

Project Supervision

Dr Mark Basham – Head of the Artificial Intelligence and Informatics Theme

Professor Angus Kirkland – Science Director of the Correlated Imaging Theme

Dr Judy Kim – Deputy Theme Director of Correlated Imaging Theme

Dr Chen Huang – Electron Microscopy Development Scientist in the Correlated Imaging Theme

Project Collaborators

Dr James Parkhurst – Postdoctoral Research Associate in the Structural Biology Theme

Dr Maud Dumoux – Technology Lead for Cryo Imaging in the Structural Biology Theme

Dr Neville Yee – Research Software Engineer in the Artificial Intelligence and Informatics Theme

Funded by:

This project is being funded by the Rosalind Franklin Institute and Department of Materials, University of Oxford.