ImPACT: Innovative Visualization Technology to Lead to Creation of a New Growth Industry

With the arrival of super-aging society, there is increasing demand for technical support to enable people to continue working while preserving their health and beauty. NII has participated in ImPACT to realize an early diagnosis of disease, and inspection of the internal structure, with advances in photo-acoustic imaging, which performs real-time 3D visualization of changes in properties and functions inside human bodies and substances, non-invasively and non-destructively. The photo-acoustic system is a promising new technology that integrates state-of-the-art laser and ultrasound technologies, where 3D structures of objects can be reconstructed by sensing emitted ultrasound from the objects that absorb near-infrared irradiation. It enables to image the state of the human body and objects whose insides are not visible, non-invasively and non-destructively. In this research, we develop computer-vision technologies to obtain clear images and extract bio-image features to support a diagnosis. For example, we proposed a registration method to generate high-quality 3D volumes in which vessels become clearly visible by aligning shot-volumes that are misaligned by body motions. We are also developing a technology that automatically models vascular structures, which helps in understanding blood vessel conditions strongly related to illnesses.

Virtual Blood Vessels using Stereo X-ray Images

We propose a fully automatic system to reconstruct and visualize 3D blood vessels in Augmented Reality (AR) system from stereo X-ray images with bones and body fat. Currently, typical 3D imaging technologies are expensive and carrying the risk of irradiation exposure. To reduce the potential harm, we only need to take two X-ray images before visualizing the vessels. Our system can effectively reconstruct and visualize vessels in following steps. We first conduct initial segmentation using Markov Random Field and then refine segmentation in an entropy based post-process. We parse the segmented vessels by extracting their centerlines and generating trees. We propose a coarse-to-fine scheme for stereo matching, including initial matching using affine transform and dense matching using Hungarian algorithm guided by Gaussian regression. Finally, we render and visualize the reconstructed model in a HoloLens based AR system, which can essentially change the way of visualizing medical data. We have evaluated its performance by using synthetic and real stereo X-ray images, and achieved satisfactory quantitative and qualitative results.

Blood Vessel Extraction by Semi-Supervised Learning

In this paper, we focus on semi-supervised learning for biomedical image segmentation, so as to take advantage of huge unlabelled data. We observe that there usually exist some homogeneous connected areas of low confidence in biomedical images, which tend to confuse the classifier trained with limited labelled samples. To cope with this difficulty, we propose to construct forest oriented super pixels(voxels) to augment the standard random forest classifier, in which super pixels(voxels) are built upon the forest based code. Compared to the state-of-the-art, our proposed method shows superior segmentation performance on challenging 2D/3D biomedical images. The full implementation (based on Matlab) is available at

Separation of Transmitted Light and Scattering Components in Transmitted Microscopy

In transmitted light microscopy, a specimen tends to be observed as unclear. This is caused by a phenomenon that an image sensor captures the sum of these scattered light rays traveled from different paths due to scattering. To cope with this problem, we propose a novel computational photography approach for separating directly transmitted light from the scattering light in a transmitted light microscope by using high-frequency lighting. We first investigated light paths and clarified what types of light overlap in transmitted light microscopy. The scattered light can be simply represented and removed by using the difference in observations between focused and unfocused conditions, where the high-frequency illumination becomes homogeneous. Our method makes a novel spatial multiple-spectral absorption analysis possible, which requires absorption coefficients to be measured in each spectrum at each position. Experiments on real biological tissues demonstrated the effectiveness of our method.

Vascular Registration in Photoacosutic Imaging by Low-Rank Alignment

Photoacoustic (PA) imaging has been gaining attention as a new imaging modality that can non-invasively visualize blood vessels inside biological tissues. In the process of imaging large body parts through multi-scan fusion, alignment turns out to be an important issue, since body motion degrades image quality. In this paper, we carefully examine the characteristics of PA images and propose a novel registration method that achieves better alignment while effectively decomposing the shot volumes into low-rank foreground (blood vessels), dense background (noise), and sparse complement (corruption) components on the basis of the PA characteristics. The results of experiments using a challenging real data-set demonstrate the efficacy of the proposed method, which significantly improved image quality, and had the best alignment accuracy among the state-of-the-art methods tested.

3D Structure Modeling of Dense Capillaries by Tracking

A newly developed imaging technique called light-sheet laser microscopy imaging can visualize the detailed 3D structures of capillaries. Capillaries form complicated network structures in the obtained data, and this makes it difficult to model vessel structures by existing methods that implicitly assume simple tree structures for blood vessels. To cope with such dense capillaries with network structures, we propose to track the flow of blood vessels along a base-axis using a multiple-object tracking framework. We first track multiple blood vessels in cross-sectional images along a single axis to make the trajectories of blood vessels, and then connect these blood vessels to reveal their entire structures. This framework is efficient to track densely distributed vessels since it uses only a single cross-sectional plane. The network structure is then generated in the post-processing by connecting blood vessels on the basis of orientations of the trajectories. The results of experiments using a challenging real data-set demonstrate the efficacy of the proposed method, which are capable of modeling dense capillaries.

Cell Image Analysis

Advances in optics and imaging systems have enabled biologists to visualize a living specimen's dynamic pro-cesses by time-lapse microscopy images. However, the image data recorded during even a single experiment may consist of hundreds of objects over thousands of images, which makes manual inspection a tedious and inaccurate option. We addresses the need for computational tools and community resources that can enable the quantitative system-level biology automatically. The domain of our current interests are cell image analysis algorithms including microscopy image restoration, cell mitotic event detection and cell tracking by data association, which are integrated into an online automated system.

Cell Detection

Cell detection in microscopy images is essential for automated cell behavior analysis including cell shape analysis and cell tracking. Robust cell detection in high-density and lowcontrast images is still challenging since cells often touch and partially overlap, forming a cell cluster with blurry intercellular boundaries. In such cases, current methods tend to detect multiple cells as a cluster. If the control parameters are adjusted to separate the touching cells, other problems often occur: a single cell may be segmented into several regions, and cells in lowintensity regions may not be detected. To solve these problems, we first detect redundant candidate regions, which include many false positives but in turn very few false negatives, by allowing candidate regions to overlap with each other. Next, the score for how likely the candidate region contains the main part of a single cell is computed for each cell candidate using supervised learning. Then we select an optimal set of cell regions from the redundant regions under non-overlapping constraints, where each selected region looks like a single cell and the selected regions do not overlap. We formulate this problem of optimal region selection as a binary linear programming problem under non-overlapping constraints. We demonstrated the effectiveness of our method for several types of cells in microscopy images. Our method performed better than five representative methods, achieving an F-measure of over 0.9 for all data sets. Experimental application of the proposed method to 3D images demonstrated that also works well for 3D cell detection.

Mitosis Detection

We propose an effective approach for automated mitosis detection using phase-contrast time-lapse microscopy, which is a non-destructive imaging modality, thereby allowing continuous monitoring of cells in culture. In our approach, we present a probabilistic model for event detection, which can simultaneously 1) identify spatio-temporal patch sequences that contain a mitotic event and 2) localize a birth event, defined as the time and location at which cell division is completed and two daughter cells are born. Our approach significantly outperforms previous approaches in terms of both detection accuracy and computational efficiency, when applied to multipotent C3H10T1/2 mesenchymal and C2C12 myoblastic stem cell populations.

Cell Tracking based on Frame by Frame Data Association

The image segmentation algorithm segments blobs from input images that can be individual cells or cell clusters (overlapped cells), and the mitosis detection algorithm locates birth events where and when one cell divides into two cells. Based on the outputs of these two algorithms, we developed a cell-blob correspondence algorithm performing data association between the cells in the previous frame and the blobs segmented in the current frame. In detail, the association algorithm makes the following hypotheses with corresponding likelihood for all possible cell actions.

Cell Tracking based on Global Data Association

Automated cell tracking in populations is important for research and discovery in biology and medicine. We propose a cell tracking method based on global spatiotemporal data association which considers hypotheses of initialization, termination, translation, division and false positive in an integrated formulation. Firstly, reliable tracklets (i.e., short trajectories) are generated by linking detection responses based on frame-by-frame association. Next, these tracklets are globally associated over time to obtain final cell trajectories and lineage trees. During global association, tracklets form tree structures where a mother cell divides into two daughter cells. We formulate the global association for tree structures as a maximum-a-posteriori (MAP) problem and solve it by linear programming. This approach is quantitatively evaluated on sequences with thousands of cells captured over several days.

Wound Healing Assay

The wound healing assay in vitro is widely used for research and discovery in biology and medicine. This assay allows for observing the healing process in vitro in which the cells on the edges of the artificial wound migrate toward the wound area. The influence of different culture conditions can be measured by observing the change in the size of the wound area. For further investigation, more detailed measurements of the cell behaviors are required. We present an application of automatic cell tracking in phase-contrast microscopy images to wound healing assay. The cell behaviors under three different culture conditions have been analyzed. Our cell tracking system can track individual cells during the healing process and provide detailed spatio-temporal measurements of cell behaviors including cell density, cell migration speed and direction, and the statistics of cell mitosis events. The application demonstrates the effectiveness of automatic cell tracking for quantitative and detailed analysis of the cell behaviors in wound healing assay in vitro.