dash-3d-tmp/phd
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

More writing, citations, fix references

Browse Source
This commit is contained in:
Thomas Forgione
2019-09-13 12:01:43 +02:00
parent e198861812
commit 4641129216
15 changed files with 281 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
src/system-bookmarks/bookmark.tex
View File

@@ -1,7 +1,7 @@
\copied{}
\section{Adding bookmarks into DASH NVE framework}\label{sec:bookmarks}
\section{Adding bookmarks into DASH NVE framework\label{sb:bookmarks}}
In this section, we explain how to include a new interaction in the system described in Section~\ref{sec:dash3d}.
In this section, we explain how to include a new interaction in the system described in the previous chapter.
\subsection{Interaction --- Visual}
@@ -22,12 +22,12 @@ In order to avoid users to lose context, clicking on a bookmark triggers an auto
We implement an additional interaction that displays a preview of the bookmark's viewpoint while it is hovered by the user's mouse.
A small thumbnail of the viewport is displayed below the bookmark.
\subsection{Segments utility at bookmarked viewpoint}\label{sec:utility}
\subsection{Segments utility at bookmarked viewpoint\label{sb:utility}}
Introducing bookmarks is a way to make users navigation more predictable.
Indeed, since they are emphasized and, in a way, recommended viewpoints, bookmarks are more likely to be visited by a significant portion of users than any other viewpoint on the scene.
As such, bookmarks can be used as a way to optimize streaming by downloading segments in an optimal, pre-computed order.
More specifically, segment utility as introduced in Section~\ref{sec:dash3d} is only an approximation of the segment's true contribution to the current viewpoint rendering.
More specifically, segment utility as introduced in Section~\ref{d3:utility} is only an approximation of the segment's true contribution to the current viewpoint rendering.
When bookmarks are defined, it is possible to obtain a perfect measure of segment utility by performing an offline rendering at each bookmark's viewpoint.
Then, by simply counting the number of pixels that are rendered using each segment, we can rank the segments by order of importance in the rendering.
@@ -56,22 +56,22 @@ This utility definition is the same for geometry and texture segments, which all
\end{axis}
\end{tikzpicture}
\caption{Impact of using the precomputed information of bookmarks to select segments to download\label{fig:precomputation}}
\caption{Impact of using the precomputed information of bookmarks to select segments to download\label{sb:precomputation}}
\end{figure}
\begin{figure}[th]
\includegraphics[width=0.49\columnwidth]{assets/system-bookmarks/bookmark/ground-truth.png}
\includegraphics[width=0.49\columnwidth]{assets/system-bookmarks/bookmark/geometry.png}
\caption{A bookmarked viewpoint (left), and a pixel to geometry segment map (right)}\label{fig:bookmarks-utility}
\caption{A bookmarked viewpoint (left), and a pixel to geometry segment map (right)\label{sb:bookmarks-utility}}
\end{figure}
Figure~\ref{fig:bookmarks-utility} depicts a ``pixel to geometry segment'' map: all pixels of the same color in the right image display an element of the same geometry segment.
Figure~\ref{sb:bookmarks-utility} depicts a ``pixel to geometry segment'' map: all pixels of the same color in the right image display an element of the same geometry segment.
We render such maps offline, for each bookmark, and use it to compute the true utility $\mathcal{U}^*(s)$ of segment $s$.
\subsection{MPD modification}
We now present how to introduce bookmarks information in the Media Presentation Description (MPD) file, to be used in a DASH framework.
Bookmarks are fully defined by a viewport description, and the additional content needed to properly render and use a bookmark in a system consists in three images: a thumbnail of the point of view at the bookmark, along with two ``pixel to segment'' maps (see Figure~\ref{fig:bookmarks-utility}, right image).
Bookmarks are fully defined by a viewport description, and the additional content needed to properly render and use a bookmark in a system consists in three images: a thumbnail of the point of view at the bookmark, along with two ``pixel to segment'' maps (see Figure~\ref{sb:bookmarks-utility}, right image).
For this reason, we create a separate adaptation set in the MPD\@.
The bookmarked viewport information is stored as a supplemental property.
Bookmarks adaptation set only contain one representation, composed of three segments corresponding to the three images described earlier.
@@ -80,7 +80,7 @@ Bookmarks adaptation set only contain one representation, composed of three segm
\lstinputlisting[%
language=XML,
caption={MPD description of a geometry adaptation set, and a texture adaptation set.},
label=listing:bookmark-as,
label=sb:bookmark-as,
emph={%
MPD,
Period,
@@ -98,7 +98,7 @@ Bookmarks adaptation set only contain one representation, composed of three segm
]{assets/system-bookmarks/bookmark-as.xml}
\end{figure}
An example of a bookmark adaptation set is depicted on Listing~\ref{listing:bookmark-as}.
An example of a bookmark adaptation set is depicted on Listing~\ref{sb:bookmark-as}.
The three first values in the supplemental property are the camera position coordinates, and the three last values are the target point coordinates.
\subsection{System-aware bookmarks}
@@ -107,7 +107,7 @@ The information we include in the MPD to optimize streaming at bookmarked viewpo
Indeed, displaying a thumbnail of what can be seen from a bookmark might fool users into thinking that all necessary segments visible from the bookmarked viewpoint have been downloaded.
In case this would be not true, users' Quality of Experience would be unsatisfactory.
In order to give users a sense of the amount of information readily available at a given bookmarked viewpoint, we use the pixel to segment maps described in Section~\ref{sec:utility} to create a mask of segment availability.
In order to give users a sense of the amount of information readily available at a given bookmarked viewpoint, we use the pixel to segment maps described in Section~\ref{sb:utility} to create a mask of segment availability.
Since we know which segments have been downloaded at any given time, we know which pixels in the thumbnail accurately depict what the user will see when clicking on the bookmark.
We thus render the thumbnail with the mask of already downloaded segments superimposed over it.
@@ -116,8 +116,8 @@ We thus render the thumbnail with the mask of already downloaded segments superi
\subsection{Loader modifications}
We build on the loader introduced in~\cite{forgione2018dash} (Algorithm 1) to implement a client adaptation logic.
We include a bookmark adaptation logic such that (i) when a bookmark is hovered for the first time, the corresponding images (see Listing~\ref{listing:bookmark-as}) are downloaded, and (ii) when a bookmark is clicked, we switch from utility $\mathcal{U}$ to true utility $\mathcal{U}^*$ to determine which segments to download next.
We build on the loader introduced in Algorithm~\ref{d3:next-segment} to implement a client adaptation logic.
We include a bookmark adaptation logic such that (i) when a bookmark is hovered for the first time, the corresponding images (see Listing~\ref{sb:bookmark-as}) are downloaded, and (ii) when a bookmark is clicked, we switch from utility $\mathcal{U}$ to true utility $\mathcal{U}^*$ to determine which segments to download next.
\begin{algorithm}[th]
\SetKwInOut{Input}{input}
@@ -134,14 +134,14 @@ We include a bookmark adaptation logic such that (i) when a bookmark is hovered
{- Optimize a criterion $\Omega$ based on $\mathcal{U}$ values and well chosen viewpoint $v(t_i)$ to select the next segment to query }
{\begin{equation*}
s^{*}_i= \argmax{s \in \mathcal{S} \backslash \mathcal{B}_i \cap \mathcal{FC}} \Omega_{\theta_i} \Big(\mathcal{U}(s,v(t_i))\Big) \label{eq1}
s^{*}_i= \argmax{s \in \mathcal{S} \backslash \mathcal{B}_i \cap \mathcal{FC}} \Omega_{\theta_i} \Big(\mathcal{U}(s,v(t_i))\Big) \label{sb:eq1}
\end{equation*} \\
given parameters $\theta_i$ that gathers both online parameters $(i,t_i,v(t_i),\widehat{BW_i}, \widehat{\tau_i}, \mathcal{B}_i)$ and offline metadata;}
{- Update the buffer $\mathcal{B}_{i+1}$ for the next decision: $s^{*}_i$ and lowest \texttt{representations} of $s^{*}_i$ are considered downloaded\;}
{- \Return{segment $s^{*}_i$, buffer $\mathcal{B}_{i+1}$}\;}
{\caption{Algorithm to identify the next segment to query\label{algorithm:nextsegment}}}
{\caption{Algorithm to identify the next segment to query\label{sb:next-segment}}}
\end{algorithm}
\todo[inline]{to be modified to include bookmarks}