Reading: Baig JVICU’17 — Multiple Hypothesis Colorization and Its Application to Image Compression (JPEG)

Outperforms JPEG By a Large Margin

In this story, “Multiple Hypothesis Colorization and Its Application to Image Compression” (Baig JVICU’17), is briefly presented. I read this because I work on video coding research. In this paper:

Bit Savings by Chroma Colorization

• First, while colorization for artistic purposes simply involves predicting plausible chroma, colorization for compression requires generating output colors that are as close as possible to the ground truth with only grayscale image as input.
• Second, many objects in the real world exhibit multiple possible colors. Thus, in order to disambiguate the colorization problem some additional information must be stored to reproduce the true colors with good accuracy.
• Finally, the proposed approach outperforms JPEG color coding by a large margin as shown above.

This is a paper in 2019 Elsevier JVICU with impact factor of 2.645. (Sik-Ho Tsang @ Medium)

Outline

1. Proposed Framework
2. Experimental Results

1. Proposed Framework

Proposed Framework

• The proposed network architecture predicts K color hypotheses per pixel by inputting the grayscale image.
• This is achieved by means of a CNN whose single trunk splits at a certain depth into K distinct branches, each outputting a 2-channel color output per pixel. All layers, both in the trunk as well as in the K branches, are fully convolutional.

K-Branch Variants on CIFAR100

• K=5 has the best performance.
• The network is based on ResNet, which is shown in the appendix of the paper.

1.1. Branch Index

• In order to be decodable, a simple solution is to store an index of the branch that best approximates the ground truth color for every pixel.
• Then at the decoding time, the grayscale version of the image goes through the network to produce the K color outputs and use the stored branch indices to select the appropriate branch for every pixel.

1.2. Superpixels for Branch Index

Superpixels for Branch Index

• Oracle one is treated as the ideal case.
• Two ways are considered to define regions for compression:
• (1) a subdivision of the image into a grid of fixed-size square cells, and
• (2) a segmentation of the grayscale photo into superpixels using traditional bottom-up segmentation methods.

1.3. Global Correction

• The global correction corrects for a similarity transform (scale and translation) between the ground truth color and the estimate.
• In a compression scenario, These 4 global parameters (2 for each color channel) are simply stored to produce more accurate color reconstructions.

2. Experimental Results

2.1. RD Curves

RD Curves

• The proposed one using superpixels (Red) outperforms JPEG (Blue).

2.2. Visualizations

Visualizations

• With low cost superpixel encoding, the reconstructed colors are close to the original one.

SOTA Comparison

• From the above figure, we can see that the bytes used by the proposed approach are much fewer than [12] and [13], but without any obvious quality degradation.

This is the 35th story in this month!

Reference

[2017 JVICU] [Baig JVICU’17]
Multiple Hypothesis Colorization and Its Application to Image Compression

Codec Intra Prediction

JPEG [MS-ROI] [Baig JVICU’17]
HEVC [Xu VCIP’17] [Song VCIP’17] [Li VCIP’17] [Puri EUSIPCO’17] [IPCNN] [IPFCN] [HybridNN, Li ICIP’18] [Liu MMM’18] [CNNAC] [Li TCSVT’18] [Spatial RNN] [PS-RNN] [AP-CNN] [MIP] [Wang VCIP’19] [IntraNN] [CNNAC TCSVT’19] [CNN-CR] [CNNMC Yokoyama ICCE’20] [PNNS] [CNNCP]
VVC [CNNIF & CNNMC] [Brand PCS’19]

My Other Previous Reviews