From a9b8c36c9688a74c8e3fdf6ee50795b5c4c401e7 Mon Sep 17 00:00:00 2001 From: Thomas Forgione Date: Mon, 7 Oct 2019 15:16:01 +0200 Subject: [PATCH] Some updates --- assets/system-bookmarks/bookmark-as.xml | 3 +- src/plan.tex | 50 ++++++++++----------- src/system-bookmarks/bookmark.tex | 60 +++++++++++-------------- 3 files changed, 53 insertions(+), 60 deletions(-) diff --git a/assets/system-bookmarks/bookmark-as.xml b/assets/system-bookmarks/bookmark-as.xml index 0265202..7f4933a 100644 --- a/assets/system-bookmarks/bookmark-as.xml +++ b/assets/system-bookmarks/bookmark-as.xml @@ -6,8 +6,7 @@ repr1/ - - + diff --git a/src/plan.tex b/src/plan.tex index 826ce84..dfcc323 100644 --- a/src/plan.tex +++ b/src/plan.tex @@ -1,26 +1,26 @@ -\frontmatter{} - -\input{introduction/main} -\resetstyle{} - -\mainmatter{} - -\input{foreword/main} -\resetstyle{} - -\input{state-of-the-art/main} -\resetstyle{} - -\input{preliminary-work/main} -\resetstyle{} - -\input{dash-3d/main} -\resetstyle{} - +% \frontmatter{} +% +% \input{introduction/main} +% \resetstyle{} +% +% \mainmatter{} +% +% \input{foreword/main} +% \resetstyle{} +% +% \input{state-of-the-art/main} +% \resetstyle{} +% +% \input{preliminary-work/main} +% \resetstyle{} +% +% \input{dash-3d/main} +% \resetstyle{} +% \input{system-bookmarks/main} -\resetstyle{} - -\backmatter{} - -\input{conclusion/main} - +% \resetstyle{} +% +% \backmatter{} +% +% \input{conclusion/main} +% diff --git a/src/system-bookmarks/bookmark.tex b/src/system-bookmarks/bookmark.tex index a038af4..605178a 100644 --- a/src/system-bookmarks/bookmark.tex +++ b/src/system-bookmarks/bookmark.tex @@ -4,7 +4,7 @@ \subsection{Desktop interaction} -Regardind desktop interaction, we keep the interaction we described in Section~\ref{bi:our-nve}, namely: +Regarding desktop interaction, we keep the interaction we described in Section~\ref{bi:our-nve}, namely: \begin{itemize} \item W, A, S and D keys to translate the camera; \item mouse motions to rotate the camera. @@ -25,7 +25,7 @@ For this reason, we display a small joystick on the bottom left corner of the sc \begin{itemize} \item moving the joystick up makes the camera move forward; \item moving the joystick down makes the camera move backwards; - \item moving the joystick sideways makes the camera move sidewars. + \item moving the joystick sideways makes the camera move sideways. \end{itemize} A screenshot of this interface is displayed in Figure~\ref{sb:mobile}. @@ -44,11 +44,11 @@ In this work, we chose a slightly different way of representing bookmarks due to \item viewport bookmarks are simple, but people who are not in the field of vision are not familiar with this representation; \item arrow bookmarks are complex, and need to be refreshed when the camera moves, which can harm the framerate of the rendering. \end{itemize} -For these reasons, we changed the display to a vertical bar with a 2D sprite of a pictural representation of an eye. +For these reasons, we changed the display to a vertical bar with a 2D sprite of a pictorial representation of an eye. This 2D sprite is always facing the camera to prevent it from being invisible when the camera would be on the side of it. Screenshots of user interfaces with bookmarks are available in Figures~\ref{sb:desktop} and~\ref{sb:mobile}. -The size of the sprite changes when time goes by following a sine to help the user differenciate what is part of the scene and what is extra widgets. +The size of the sprite changes when time goes by following a sine to help the user distinguish what is part of the scene and what is extra widgets. Since our scene is static, a user knows when seeing a bookmark that it is not part of the scene. The other parameters of the bookmarks remain unchanged since Chapter~\ref{bi}: in order to avoid users to lose context, clicking on a bookmark triggers an automatic, smooth, camera displacement that ends up at the bookmark. @@ -81,12 +81,13 @@ When bookmarks are defined, it is possible to obtain a perfect measure of segmen We define $\mathcal{U}^{*} (s,B_i)$ as being the true utility of a segment $s$ in a viewpoint defined at bookmark $B_i$. \fresh{} -Algorithm~\ref{sb:algo-optimal-order} computes the best order of segments from a viewpoint. -It starts with an empty model, and it tries all the segments from a set of candidates, and computes the PSNR between the corresponding render and the ground truth render. +Algorithm~\ref{sb:algo-optimal-order} sorts segments according to their true utility. +It takes as input the considered viewpoint, the ground truth from this viewpoint and the set of segments to sort. +It starts with an empty model, and tries all the segments from the set of candidates, and computes the PSNR between the corresponding render and the ground truth render. With all those PSNRs, it is able to determine which segment is the one that brings the best $\Delta\text{PSNR} / s$, $s$ being the size of the segment in bytes. Once it found the best segment, it registers it, and starts again. That way, it is able to generate an order of segments sorted by $\Delta\text{PSNR} / s$. -This order is then saved as a JSON file and a client will be able to download it to know which segments are more appropriate from a certain viewpoint. +This order is then saved as a JSON file that a client can download it to know which segments are more appropriate from a certain viewpoint. \begin{algorithm}[th] \SetKwInOut{Input}{input} @@ -144,10 +145,22 @@ This order is then saved as a JSON file and a client will be able to download it \caption{Computation of the optimal order of segments from a bookmark\label{sb:algo-optimal-order}} \end{algorithm} -% This utility is simply the ratio between the number of pixels displaying that segment on screen, and the total screen area (in pixels). -% This utility definition is the same for geometry and texture segments, which allows all segments to be ranked by order of importance, i.e.\ of decreasing utility. +Sorting all the segments from the model would be an excessively time consuming computations. +To speed up the algorithm, we only sort the 200 best segments, and we choose these segments among a filtered set of candidates. +To find those candidates, we reuse the ideas developed in Chapter~\ref{bi}. +We render the ``pixel to geometry segment'' and ``pixel to texture'' maps, as shown in Figure~\ref{sb:bookmarks-utility}. +These renderings allow us to know what geometry segment and what texture correspond to each pixel, and filter out useless candidates. -\copied{} +\begin{figure}[th] + \centering + \includegraphics[width=0.32\columnwidth]{assets/system-bookmarks/bookmark/ground-truth.png} + \includegraphics[width=0.32\columnwidth]{assets/system-bookmarks/bookmark/geometry.png} + \includegraphics[width=0.32\columnwidth]{assets/system-bookmarks/bookmark/texture.png} + \caption{A bookmarked viewpoint (left), a pixel to geometry segment map (center), and a pixel to texture map (right)\label{sb:bookmarks-utility}} +\end{figure} + +Figure~\ref{sb:precomputation} shows how this precomputation improves the quality of rendering. +Each curve represents the PSNR one can obtain by downloading a certain amount of data. \begin{figure}[th] \centering @@ -173,22 +186,16 @@ This order is then saved as a JSON file and a client will be able to download it \caption{Impact of using the precomputed information of bookmarks to select segments to download\label{sb:precomputation}} \end{figure} -\begin{figure}[th] - \includegraphics[width=0.49\columnwidth]{assets/system-bookmarks/bookmark/ground-truth.png} - \includegraphics[width=0.49\columnwidth]{assets/system-bookmarks/bookmark/geometry.png} - \caption{A bookmarked viewpoint (left), and a pixel to geometry segment map (right)\label{sb:bookmarks-utility}} -\end{figure} - -Figure~\ref{sb:bookmarks-utility} depicts a ``pixel to geometry segment'' map: all pixels of the same color in the right image display an element of the same geometry segment. -We render such maps offline, for each bookmark, and use it to compute the true utility $\mathcal{U}^*(s)$ of segment $s$. \subsection{MPD modification} +\copied{} + We now present how to introduce bookmarks information in the Media Presentation Description (MPD) file, to be used in a DASH framework. -Bookmarks are fully defined by a viewport description, and the additional content needed to properly render and use a bookmark in a system consists in three images: a thumbnail of the point of view at the bookmark, along with two ``pixel to segment'' maps (see Figure~\ref{sb:bookmarks-utility}, right image). +Bookmarks are fully defined by a viewport description, and the additional content needed to properly render and use a bookmark in a system consists in three images: a thumbnail of the point of view at the bookmark, along with the JSON file giving the optimal segment order for this viewpoint. For this reason, we create a separate adaptation set in the MPD\@. The bookmarked viewport information is stored as a supplemental property. -Bookmarks adaptation set only contain one representation, composed of three segments corresponding to the three images described earlier. +Bookmarks adaptation set only contain one representation, composed of two segments: the thumbnail used for the desktop and the JSON file. \begin{figure}[th] \lstinputlisting[% @@ -215,19 +222,6 @@ Bookmarks adaptation set only contain one representation, composed of three segm An example of a bookmark adaptation set is depicted on Listing~\ref{sb:bookmark-as}. The three first values in the supplemental property are the camera position coordinates, and the three last values are the target point coordinates. -\subsection{System-aware bookmarks} - -The information we include in the MPD to optimize streaming at bookmarked viewpoints can also be used to give a sense of the system state to the user. -Indeed, displaying a thumbnail of what can be seen from a bookmark might fool users into thinking that all necessary segments visible from the bookmarked viewpoint have been downloaded. -In case this would be not true, users' Quality of Experience would be unsatisfactory. - -In order to give users a sense of the amount of information readily available at a given bookmarked viewpoint, we use the pixel to segment maps described in Section~\ref{sb:utility} to create a mask of segment availability. -Since we know which segments have been downloaded at any given time, we know which pixels in the thumbnail accurately depict what the user will see when clicking on the bookmark. -We thus render the thumbnail with the mask of already downloaded segments superimposed over it. - -\todo[inline]{Figure of altered thumbnail} - - \subsection{Loader modifications} We build on the loader introduced in Algorithm~\ref{d3:next-segment} to implement a client adaptation logic.