\documentclass[a4paper,11pt]{jctvcdoc}

\usepackage[section]{placeins}
\usepackage{amsmath}

%\usepackage{hyperref}
%\hypersetup{colorlinks=true}
\usepackage{soul}

\usepackage[margin=10pt,font=small,format=hang,labelsep=endash]{caption}
\usepackage[position=bottom]{subfig}
\captionsetup[subfloat]{position=top}
\usepackage{multirow}
\usepackage{dcolumn}
\newcolumntype{.}{D{.}{.}{-1}}
\usepackage{colortbl}
\usepackage{makecell}

\usepackage[strings]{underscore}
\usepackage{csquotes}
\MakeOuterQuote{"}
\EnableQuotes

\title{Decoding a 10-bit sequence using an 8-bit decoder}
\author{%
	David Flynn
	\email{dflynn@blackberry.com}
	\and
	Gaëlle Martin-Cocher
	\email{gmartincocher@blackberry.com}
	\and
	Dake He
	\email{dhe@blackberry.com}
}

\jctvcmeeting{13th Meeting: Incheon, KR, 18 -- 26 April 2013}
\jctvcdocnum{M0255}
\jctvcdocstatus{Input Document to JCT-VC}
\jctvcdocpurpose{Proposal}
\jctvcdocsource{Research In Motion Ltd.}

\begin{document}
\maketitle
\begin{abstract}
HEVC, like other well-known codecs, contains profiles that accommodate
the decoding of non-8-bit video sequences.  In particular, HEVC contains
a 10-bit profile that permits the carriage of 10-bit video sequences.
However, for some implementations, the extra resources required for
conformance to a higher bit-depth operating point can be prohibitive, yet
there exist circumstances where a decoder may wish to offer best-effort
decoding of a bitstream.  Two techniques are presented that facilitate
the decoding of 10-bit video bitstreams using an 8-bit decoder.  The
first uses the normal 8-bit reconstruction process, while the second
involves addition of a rounding process during the reconstruction
process.  PSNR losses using low-delay main-10 bitstreams average at a
6~dB and 2.5~dB respectively.  Further results are presented and
specification text is provided for a possible annex.
\end{abstract}

\section{Background}
One such class of decoder that is particularly sensitive to the issue of
decoding bitstreams at a bit-depth greater than 8-bits is the software
decoder.  Many general purpose processors can only perform operations at
certain word sizes, eg 8-, 16-, 32-, or 64-bit, and the memory layout
for video data used by software is strongly linked to the word size.
For 8-bit video, it is obvious that it is very efficient to store
individual samples in memory.  However, for 10-bit video, rather than
increasing the storage requirement by 25\%, the next available word size
(16-bit) doubles the storage and memory bandwidth requirements.  While
it is possible to use novel data packing formats to reduce the impact,
the burden of continually packing and unpacking the data can be
considerable for general purpose microprocessors\footnote{This technique
is certainly used in many hardware implementations}

\begin{figure}[h]
\footnotesize
\caption{Abstract decoder signal processing path in HEVC}
\label{fig:hevc-decoder}
\centering
\includegraphics[page=1]{figures/blockdiags}
\end{figure}

The abstract signal processing path of an HEVC decoder operating at a
nominal bit-depth $d$ is shown in Figure~\ref{fig:hevc-decoder}.
Considering the case when $d=10$, the intra and inter prediction
processes, P, produce 10-bit unsigned predicted samples that are combined
with an (inverse quantised and inverse transformed) 11-bit signed
residual, resulting in after clipping, 10-bit unsigned reconstructed
samples.  The in-loop filters for deblocking and SAO do not alter the
bit-depth of the reconstructed samples (although internal bit-widths may
differ).  Finally the reconstructed samples are stored in a buffer
available for future prediction and output.

\section{Method--1: 8-bit decoding by adjusting inverse transform scaling}
\label{sec:method1}
In general, the magnitude of samples produced by the reconstruction
process is governed by the magnitude of the coefficients, the QP
for inverse quantisation\footnote{The quantisation process itself is
bit-depth independent}, the scaling factor in the last stage of the
inverse transform, the default values used for prediction when reference
samples are not available, and the clipping limits.

Many of these aspects are controlled via the derived BitDepthX,
adjusting the effective value of this variable will from the same
bitstream produce a different output bit-depth.  An initial modification
to the HM decoder to provide 8-bit decoding in this manner was performed
using the following alterations:
\begin{itemize}
\item The values RestrictedBitDepthY and RestrictedBitDepthC are set to 8.
\item Unpacking operations that use the bit-depth as a binarization
      process parameter, in particular SAO, are performed using the
      appropriate bitstream native bit-depth.
\item Unpacked SAO offsets are adjusted by a factor of
      $2^\text{RestrictedBitDepthX} \div 2^\text{BitDepthX}$.
\item All other reconstruction processes are be performed with
      RestrictedBitDepthX substituted for BitDepthX, where X=\{Y,C\}.
      ie, as if the bitstream was signalled as 8-bit.
\end{itemize}

To examine the modified decoder behaviour in unfavourable conditions,
the HM-10 low-delay-B main-10 sequences \cite{jctvc-l1100} were decoded
and the PSNR measured against the original input sequences.
Table~\ref{tab:method1-psnr} illustrates the per-sequence losses
when compared against the PSNR of the normal decoder.
Figure~\ref{fig:psnr-evolution} shows how the PSNR loss evolves as the
sequence progresses.  Visual inspection of the decoded video shows the
following traits:
\begin{itemize}
\item Intra pictures experience DC drift increasing towards the bottom
      right.
\item Drift (most noticeable in colour or saturation) evolves over
      the sequence.
\item The distortion introduced can be significant (in excess of 11dB)
      and can be clearly discernible on a video sequence in a single
      stimulus test.
\item The distortion becomes more significant as QP decreases.
\end{itemize}

\begin{table}[h!]
\footnotesize
\caption{PSNR loss associated with the method of
section~\protect\ref{sec:method1} averaged over 10s sequnces at QPs 22/27/32/37
using low-delay main-10 configuration}
\label{tab:method1-psnr}
\centering
\begin{tabular}{l c c}
\hline
Sequence & Luma PSNR & Chroma PSNR\\
\hline
BQMall & −8.3/−7.2/−2.7/−2.3 & −8.8/−6.0/−2.4/−4.5\\
BQSquare & −4.5/−1.9/−1.3/−0.3 & −4.9/−2.8/−1.1/−1.2\\
BQTerrace & −12.3/−6.3/−4.0/−2.0 & −10.0/−3.9/−1.7/−2.2\\
BasketballDrillText & −10.8/−9.7/−3.2/−2.2 & −6.9/−5.1/−1.4/−1.1\\
BasketballDrill & −9.5/−10.3/−3.3/−3.2 & −7.6/−5.8/−1.7/−1.1\\
BasketballDrive & −10.8/−8.2/−5.9/−3.3 & −11.4/−8.3/−6.0/−4.5\\
BasketballPass & −6.0/−3.3/−0.8/−0.3 & −4.0/−1.6/−0.5/−0.8\\
BlowingBubbles & −6.7/−2.1/−2.0/−0.3 & −7.6/−4.7/−1.9/−1.5\\
Cactus & −13.6/−9.6/−7.5/−5.3 & −12.1/−8.3/−4.4/−4.9\\
ChinaSpeed & −10.9/−8.3/−5.0/−1.4 & −10.3/−9.1/−4.9/−3.7\\
FourPeople & −15.1/−12.7/−7.2/−3.9 & −12.2/−8.0/−5.0/−2.0\\
Johnny & −9.4/−9.8/−8.5/−1.7 & −15.8/−6.1/−4.0/−3.3\\
Kimono1 & −12.5/−8.7/−3.5/−8.5 & −10.8/−6.9/−4.7/−6.2\\
KristenAndSara & −9.3/−7.7/−14.7/−2.9 & −15.0/−5.2/−5.0/−3.2\\
ParkScene & −15.2/−8.2/−8.0/−6.2 & −12.3/−5.4/−4.4/−2.7\\
PartyScene & −7.9/−7.7/−3.8/−1.5 & −8.2/−5.5/−1.4/−2.0\\
PeopleOnStreet & −17.3/−13.2/−8.6/−4.4 & −16.3/−11.6/−7.7/−6.9\\
RaceHorses & −6.2/−5.7/−2.0/−0.2 & −4.7/−4.0/−0.5/−1.2\\
RaceHorsesC & −10.7/−7.3/−6.8/−1.7 & −7.6/−8.5/−1.8/−2.5\\
SlideEditing & −7.4/−7.3/−2.8/−0.5 & −6.3/−1.7/−1.0/−0.6\\
SlideShow & −5.2/−3.1/−1.0/−0.3 & −6.6/−2.6/−1.4/−1.1\\
Traffic & −17.2/−15.0/−9.6/−8.4 & −12.3/−10.7/−6.7/−7.0\\
\hline
\end{tabular}
\end{table}

In a more complete implementation, the derivation of QpBdOffsetY would
require the use of BitDepthY not RestrictedBitDepthY, otherwise there is
the potential for the derivation of QP to wrap incorrectly.  This effect
cannot be observed using the constant QP anchor sequences. 


\section{Method--2: Hybrid 8-bit--10-bit decoding using rounding in picture construction process}
\label{sec:method2}

For an 8-bit decoder, many calculations are already computed using
16-bit or larger words.  For example, the inverse transformed residual
is a 16-bit signed value, and that the combination of the residual and
prediction must occur at a minimum of 10-bits due to the signed nature
of the residual (assuming a reasonable residual and that addition with
saturation is unavailable).  On software systems, this must be
implemented using the next available word size, typically 16-bit.
Therefore there may exist a better trade-off as to the position of
conversion from a 10-bit system to an 8-bit system, especially if the
reconstruction quality can be improved.

\begin{figure}[h!]
\footnotesize
\caption{Illustrative signal coding path of hybrid 8-bit--10-bit decoder}
\label{fig:method2}
\centering
\includegraphics[page=2]{figures/blockdiags}
\end{figure}

An opportune position to perform such a conversion is immediately
following the addition of the prediction and residual, prior to
clipping.  The proposed system is illustrated in
figure~\ref{fig:method2}.

If the reconstruction loop now performs what is an effective 10- to
8-bit conversion, it will also be necessary to perform a corresponding
8- to 10-bit conversion.  We propose that this is performed on the
samples used for intra prediction without significant impacting the cost
of intra prediction, and in the case of inter prediction it is performed
after motion compensation so as not to increase the cost of sub-pel
filtering.  The method for increasing the bit-width is a left shift
operation.

Since significant drift was observed in the intra pictures produced by
the method of section~\ref{sec:method1}, it is worth noting that the
method essentially implements a truncation of the residual.  The effect
of this is to introduce a systematic less-than-unity gain into the
reconstruction loop, through consistent under-correction of the
prediction, an error that accumulates due to the nature of intra
prediction.  The choice of rounding method is essential to reducing the
drift in intra pictures.  The obvious choice for implementing rounding,
rounding half values towards positive infinity ($\lfloor (x + 2^{n-1})
\div 2^n \rfloor$), as favoured in other parts of the HEVC specification
also contains a systematic bias resulting in a loop gain greater than
one.  An alternative rounding technique, as used in IEEE~754 is to round
half values towards the nearest even value\footnote{also known as
bankers' rounding}.  For sufficiently well distributed values, this
method is unbiased for both positive and negative numbers.  Such a
method can be implemented using \verb|(x + 1 + ((x>>n)&1)) >> n|.

A modification to the HM decoder that provides 8-bit decoding using this
method implemented the following changes to the decoding process:
\begin{itemize}
\item The values RestrictedBitDepthY and RestrictedBitDepthC are set to 8.
      \\
      Note: the value of BitDepthX as used below is as normally derived
      from the bitstream.
\item Unpacking operations that use the bit-depth as a binarization
      process parameter, in particular SAO, are performed using the
      appropriate bitstream native bit-depth.
\item Unpacked SAO offsets are adjusted to by a factor of
      $2^\text{RestrictedBitDepthX} \div 2^\text{BitDepthX}$.
\item Inverse quantisation, including the setting of QpBdOffset, and the
      inverse transformation of residuals are performed using BitDepthX.
\item The process for generating intra prediction samples involves
      multiplying the samples used for intra prediction by~
      $2^{\text{BitDepthX}-\text{RestrictedBitDepthX}}$.  The intra
      prediction process is then performed using BitDepthX.
\item The process for generating inter prediction samples involves
      multiplying the resulting samples by~
      $2^{\text{BitDepthX}-\text{RestrictedBitDepthX}}$.  The inter
      prediction process is performed using RestrictedBitDepthX.
\item The picture reconstruction process involves adding the prediction
      samples to the residual samples and then reducing their magnitude
      using~\verb|(x + 1 + ((x>>2)&1)) >> 2| prior to clipping, with
      clipping limits set to~$[0,2^\text{RestrictedBitDepthX}-1]$.
\item All other reconstruction processes are performed with
      RestrictedBitDepthX substituted for BitDepthX, where X=\{Y,C\}.
\end{itemize}

\begin{table}[hb]
\caption{PSNR loss associated with method of section~\protect\ref{sec:method2}
averaged over 10s sequences at QPs 22/27/32/37 using low-delay main-10
configuration}
\centering
\footnotesize
\begin{tabular}{l c c}
\hline
Sequence & Luma PSNR & Chroma PSNR \\
\hline
BQMall & −7.1/−3.5/−1.1/−0.6 & −3.7/−2.7/−1.5/−1.2\\
BQSquare & −3.9/−1.7/−0.3/−0.2 & −4.7/−2.0/−0.8/−0.6\\
BQTerrace & −6.3/−2.6/−0.8/−1.3 & −4.2/−3.2/−1.7/−1.0\\
BasketballDrillText & −7.2/−2.8/−0.6/−0.4 & −5.4/−2.9/−1.0/−0.4\\
BasketballDrill & −6.5/−3.1/−0.5/−0.5 & −5.3/−4.1/−1.5/−0.3\\
BasketballDrive & −3.9/−2.3/−1.7/−0.7 & −5.8/−2.6/−2.3/−0.9\\
BasketballPass & −4.0/−1.3/−0.3/−0.2 & −2.8/−1.0/−0.4/−0.3\\
BlowingBubbles & −3.5/−1.3/−0.5/−0.3 & −4.6/−1.7/−1.4/−0.8\\
Cactus & −6.2/−2.3/−1.5/−1.5 & −5.1/−4.1/−1.5/−0.8\\
ChinaSpeed & −7.7/−3.0/−2.8/−0.6 & −8.5/−4.0/−2.7/−1.2\\
FourPeople & −8.9/−4.2/−1.5/−3.5 & −4.3/−4.3/−2.3/−0.8\\
Johnny & −7.2/−2.4/−2.1/−1.4 & −7.8/−4.1/−2.3/−1.2\\
Kimono1 & −5.6/−1.7/−1.4/−1.1 & −2.7/−1.8/−2.6/−1.0\\
KristenAndSara & −5.4/−2.6/−0.9/−2.3 & −7.8/−3.4/−4.0/−1.4\\
ParkScene & −5.6/−2.0/−1.0/−0.5 & −4.6/−2.8/−0.8/−0.7\\
PartyScene & −3.3/−2.9/−0.6/−0.4 & −3.4/−1.7/−1.0/−0.9\\
PeopleOnStreet & −3.6/−1.2/−1.2/−0.7 & −4.4/−3.2/−3.5/−1.3\\
RaceHorses & −3.9/−1.4/−0.8/−0.3 & −5.1/−0.9/−0.5/−0.7\\
RaceHorsesC & −2.4/−1.3/−0.8/−0.4 & −4.2/−2.9/−1.9/−0.7\\
SlideEditing & −6.1/−2.0/−1.5/−0.3 & −1.9/−0.7/−0.3/−0.2\\
SlideShow & −6.5/−1.9/−0.6/−0.3 & −4.4/−2.4/−1.1/−0.5\\
Traffic & −6.2/−5.1/−2.3/−1.1 & −4.5/−3.0/−2.4/−0.8\\
\hline
\end{tabular}
\end{table}

After performing the same analysis as per section~\ref{sec:method1},
objective results show a significant reduction in losses compared to the
first method.  Visual inspection shows that the gross DC drift observed
using the first method is avoided when using this method, to the extent
that with a single stimulus test it can be difficult observe a
degradation.

\section{Further results}
The enclosed spreadsheet includes results for all sequences and all QPs.
For each test point it shows PSNR figures averaged for the entire
sequence, and two additional results averaged over the first and last
ten frames respectively.

Figure~\ref{fig:method-comparison} illustrates the typical drift effect
observed when using the two methods compared to the vanilla HM-10
decode after decoding the ten-second 500-frame sequence Cactus encoded
using the low-delay main-10 configuration at QP=27.  The figure shows
the first intra and the last inter frame of the sequence.
Video sequences using both methods are available at the meeting for
viewing.

Figure~\ref{fig:psnr-evolution} illustrates the temporal performance
of the two methods with plots of the per-frame PSNR (subsequent to a
five tap median filter to remove variation due to the GOP structure).
The Cactus sequence used in figure~\ref{fig:method-comparison} is
marked in black. 
Considering all of the sequences, in most cases the performance of the
second method is superior to the first method.  In circumstances where
this doesn't appear to be the case, such as for SlideShow at around
frames 115--160 for QP=22, careful inspection shows a peculiar
uncorrected large magnitude intra prediction error that is only evident
when performing a dual stimulus test.

\begin{figure}[h!]
\footnotesize
\caption{Typical distortion introduced by 8-bit decoding of 10-bit
sequences.  Sequence cactus QP=27, first frame (left), last
frame (right). (a,b) Method--1, (c,d) Method--2, (e,f) HM-10.}
\label{fig:method-comparison}
\centering
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f0-gcf6e84e.png.jpeg}
}%
\qquad
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f499-gcf6e84e.png.jpeg}
}%
\\
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f0-ga7cc31e.png.jpeg}
}%
\qquad
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f499-ga7cc31e.png.jpeg}
}%
%\\
%\subfloat[]{
%  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f0-gf5a6305.png.jpeg}
%}%
%\qquad
%\subfloat[]{
%  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f499-gf5a6305.png.jpeg}
%}%
\\
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f0-g538e6fe.png.jpeg}
}%
\qquad
\subfloat[]{
  \includegraphics[width=0.44\textwidth]{figures/cactus-ldmain10-qp27-f499-g538e6fe.png.jpeg}
}%
\end{figure}

\begin{figure}
\footnotesize
\caption{Plot of frame-number versus $\Delta\text{PSNR}$ for all sequences
at particular QPs using Method--1 (left) and Method--2 (right).}
\label{fig:psnr-evolution}
\centering
\subfloat[QP 22]{
  \includegraphics[page=1,width=0.40\textwidth]{figures/graph.pdf}
}%
\qquad
\subfloat[QP 22]{
  \includegraphics[page=5,width=0.40\textwidth]{figures/graph.pdf}
}%
%\qquad
%\subfloat[QP 22]{
%  \includegraphics[page=9,width=0.25\textwidth]{figures/graph.pdf}
%}%
\\
\subfloat[QP 27]{
  \includegraphics[page=2,width=0.40\textwidth]{figures/graph.pdf}
}%
\qquad
\subfloat[QP 27]{
  \includegraphics[page=6,width=0.40\textwidth]{figures/graph.pdf}
}%
%\qquad
%\subfloat[QP 27]{
%  \includegraphics[page=10,width=0.25\textwidth]{figures/graph.pdf}
%}%
\\
\subfloat[QP 32]{
  \includegraphics[page=3,width=0.40\textwidth]{figures/graph.pdf}
}%
\qquad
\subfloat[QP 32]{
  \includegraphics[page=7,width=0.40\textwidth]{figures/graph.pdf}
}%
%\qquad
%\subfloat[QP 32]{
%  \includegraphics[page=11,width=0.25\textwidth]{figures/graph.pdf}
%}%
\\
\subfloat[QP 37]{
  \includegraphics[page=4,width=0.40\textwidth]{figures/graph.pdf}
}%
\qquad
\subfloat[QP 37]{
  \includegraphics[page=8,width=0.40\textwidth]{figures/graph.pdf}
}%
%\qquad
%\subfloat[QP 37]{
%  \includegraphics[page=12,width=0.25\textwidth]{figures/graph.pdf}
%}%
\end{figure}

\section{Proposed specification text}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.9,page=1]{spec-annex-f-crop.pdf}
\end{figure}

\begin{figure}[h!]
\centering
\includegraphics[scale=0.9,page=2]{spec-annex-f-crop.pdf}
\end{figure}

\section*{}
\textbf{Research in Motion Limited may have current or pending patent rights
relating to the technology described in this contribution and,
conditioned on reciprocity, is prepared to grant licenses under
reasonable and non-discriminatory terms as necessary for implementation
of the resulting ITU-T Recommendation \textbar{} ISO/IEC International
Standard (per box 2 of the ITU-T/ITU-R/ISO/IEC patent statement and
licensing declaration form).}

\bibliographystyle{IEEEtran}
\bibliography{JCTVC-M0255}

\end{document}
