System bookmarks

src/config.sty

@@ -8,7 +8,7 @@ anchorcolor = blue]{hyperref}
 \usepackage{xspace}
 \usepackage{url}
 \usepackage{lmodern}
-\usepackage{algorithm2e}
+\usepackage[ruled]{algorithm2e}
 \usepackage{datatool}
 \usepackage{pgfplots}
 \usepackage{graphicx}

src/introduction/video-vs-3d.tex

@@ -25,7 +25,7 @@ It can be chosen directly by the user or automatically determined by analysing t
     \caption{The different resolutions available for a Youtube video}
 \end{figure}
 
-In the same way, the recent works in 3D streaming have proposed many ways to progressively streaming 3D models, allowing the user to have a low resolution without having to wait, and being able to interact with the model while the details are being downloaded.
+In the same way, the recent work in 3D streaming have proposed many ways to progressively streaming 3D models, allowing the user to have a low resolution without having to wait, and being able to interact with the model while the details are being downloaded.
 
 \subsection{Media types}
 

src/main.tex

@@ -11,6 +11,8 @@ chapterprefix=true% like in standard class "report"
 \usepackage{listing-config}
 
 \begin{document}
+\let\rawref\ref%
+\renewcommand{\ref}[1]{\rawref{#1} (page~\pageref{#1})}
 
 \frontmatter
 

src/preliminary-work/bookmarks-impact.tex

@@ -6,7 +6,7 @@ We now describe an experiment that we conducted on 51 participants, with two goa
 First, we want to measure the impact of 3D bookmarks on navigation within an NVE\@.
 Second, we want to collect traces from the users so that we can replay them for reproducible experiments for comparing streaming strategies in Section 4.
 
-\subsection{Our NVE\ref{bi:our-nve}}
+\subsection{Our NVE\label{bi:our-nve}}
 To ease the deployment of our experiments to users in distributed locations on a crowdsourcing platform, we implement a simple Web-based NVE client using THREE.js\footnote{http://threejs.org}.
 The NVE server is implemented with node.js\footnote{http://nodejs.org}.
 The NVE server streams a 3D scene to the client; the client renders the scene as the 3D content are received.
@@ -45,7 +45,7 @@ The interface also includes a button to reset the camera back to the starting po
 \end{figure}
 
 
-\subsection{3D Bookmarks}
+\subsection{3D Bookmarks\label{bi:3d-bookmarks}}
 Our NVE supports 3D bookmarks.
 A 3D bookmark, or bookmark for short, is simply a fixed camera location (in 3D space), a view direction, and a focal.
 Bookmarks visible from the user's current viewpoint are shown as 3D objects in the scene.

src/system-bookmarks/bookmark.tex

@@ -9,6 +9,7 @@ Regardind desktop interaction, we keep the interaction we described in Section~\
     \item W, A, S and D keys to translate the camera;
     \item mouse motions to rotate the camera.
 \end{itemize}
+A screenshot of this interface is displayed in Figure~\ref{sb:desktop}.
 
 \subsection{Mobile interaction}
 
@@ -16,8 +17,8 @@ Regardind desktop interaction, we keep the interaction we described in Section~\
 Mobile interactions are more complex because the user does not have neither keyboard nor mouse to interact with.
 However, there are some other sensors on most mobile devices that can help interaction.
 The most useful sensor for 3D interaction on mobile devices is definitely the gyroscope.
-We use the gyroscope to enable a user to turn his device to turn the virtual camera.
+We use the gyroscope to enable a user to rotate his device to rotate the virtual camera.
-We also add the possibility to turn the camera by drag and dropping the scene.
+We also add the possibility to rotate the camera by drag and dropping the scene.
 This way, the user is not forced to perform a real-world half-turn to be able to look behind or to keep its device pointing to the sky (which can quickly become tiring) to look up.
 These interactions, however, do not allow the user to move the camera: he can rotate it but not translate it.
 For this reason, we display a small joystick on the bottom left corner of the screen that mimics the first person video games interactions and  allow the user translating the camera:
@@ -26,43 +27,127 @@ For this reason, we display a small joystick on the bottom left corner of the sc
     \item moving the joystick down makes the camera move backwards;
     \item moving the joystick sideways makes the camera move sidewars.
 \end{itemize}
+A screenshot of this interface is displayed in Figure~\ref{sb:mobile}.
 
-\copied{}
 \section{Adding bookmarks into DASH NVE framework\label{sb:bookmarks}}
 
+\copied{}
 In this section, we explain how to include a new interaction in the system described in the previous chapter.
+\fresh{}
 
 \subsection{Interaction --- Visual}
 
-We decide to add bookmarks of recommended viewpoints in the 3D scene.
+In Chapter~\ref{bi} Section~\ref{bi:3d-bookmarks}, we described two 3D widgets that we used to display bookmarks to users.
-A bookmark is defined by a camera position, orientation, and intrinsic parameters, and offers a particular view of the 3D virtual environment.
+One of the conclusions of the user-study, described in Section~\ref{bi:user-study}, was that the impact of the way we display bookmark was not significant.
-As such, it should be represented using a widget that (i) is attached to a particular position, therefore appearing small (respectively big) when it is far (respectively close), and (ii) points at a particular direction, allowing the user to predict what can be seen from this viewpoint.
+In this work, we chose a slightly different way of representing bookmarks due to some concerns with our original representations:
-Bookmarks have been already introduced in the literature, with various appearances \todo[inline]{figure of possible appearances}.
+\begin{itemize}
-Since, no particular preeminence of one design on the others has been demonstrated in previous work, we arbitrarily choose to use \todo[inline]{type of bookmark} in this work.
+    \item viewport bookmarks are simple, but people who are not in the field of vision are not familiar with this representation;
+    \item arrow bookmarks are complex, and need to be refreshed when the camera moves, which can harm the framerate of the rendering.
+\end{itemize}
+For these reasons, we changed the display to a vertical bar with a 2D sprite of a pictural representation of an eye.
+This 2D sprite is always facing the camera to prevent it from being invisible when the camera would be on the side of it.
+Screenshots of user interfaces with bookmarks are available in Figures~\ref{sb:desktop} and~\ref{sb:mobile}.
 
-Bookmarks can be created either automatically, or manually defined by an expert user (e.g.\ the 3D model designer, or administrator).
+The size of the sprite changes when time goes by following a sine to help the user differenciate what is part of the scene and what is extra widgets.
-Bookmarks could even be derived from the observation of user behavior, by focusing on the most visited areas of the models.
+Since our scene is static, a user knows when seeing a bookmark that it is not part of the scene.
-Automated ways of defining bookmarks or adapting them to user behavior is beyond the scope of this paper; methods have been proposed in \todo[inline]{References}
 
-We choose to implement two interactions with bookmarks.
+The other parameters of the bookmarks remain unchanged since Chapter~\ref{bi}: in order to avoid users to lose context, clicking on a bookmark triggers an automatic, smooth, camera displacement that ends up at the bookmark.
-The first, most obvious one, is to position the user camera on the bookmark's viewpoint when the user clicks on the bookmark.
+We also display a thumbnail of the bookmark's viewpoint when the mouse hovers over a bookmark.
-In order to avoid users to lose context, clicking on a bookmark triggers an automatic, smooth, camera displacement that ends up at the bookmark.
+Such thumbnail is displayed in Figure~\ref{sb:desktop}.
-% We use Hermite's polynomials to compute this displacement, as proposed in MMSYS16. Lol we don't :'(
+Note that since on mobile, there is no mouse and thus no pointer, thumbnails are never downloaded nor displayed.
-We implement an additional interaction that displays a preview of the bookmark's viewpoint while it is hovered by the user's mouse.
+
-A small thumbnail of the viewport is displayed below the bookmark.
+\begin{figure}[ht]
+    \centering
+    \includegraphics[width=\textwidth]{assets/system-bookmarks/screenshots/desktop.png}
+    \caption{Screenshot of the desktop version, with a bookmark and its thumbnail on the bottom left corner\label{sb:desktop}}
+\end{figure}
+
+\begin{figure}[ht]
+    \centering
+    \includegraphics[width=\textwidth]{assets/system-bookmarks/screenshots/mobile.png}
+    \caption{Screenshot of the mobile version, with its joystick on the bottom left corner\label{sb:mobile}}
+\end{figure}
 
 \subsection{Segments utility at bookmarked viewpoint\label{sb:utility}}
+\copied{}
 Introducing bookmarks is a way to make users navigation more predictable.
 Indeed, since they are emphasized and, in a way, recommended viewpoints, bookmarks are more likely to be visited by a significant portion of users than any other viewpoint on the scene.
 As such, bookmarks can be used as a way to optimize streaming by downloading segments in an optimal, pre-computed order.
 
 More specifically, segment utility as introduced in Section~\ref{d3:utility} is only an approximation of the segment's true contribution to the current viewpoint rendering.
 When bookmarks are defined, it is possible to obtain a perfect measure of segment utility by performing an offline rendering at each bookmark's viewpoint.
-Then, by simply counting the number of pixels that are rendered using each segment, we can rank the segments by order of importance in the rendering.
+% Then, by simply counting the number of pixels that are rendered using each segment, we can rank the segments by order of importance in the rendering.
 
 We define $\mathcal{U}^{*} (s,B_i)$ as being the true utility of a segment $s$ in a viewpoint defined at bookmark $B_i$.
-This utility is simply the ratio between the number of pixels displaying that segment on screen, and the total screen area (in pixels).
+
-This utility definition is the same for geometry and texture segments, which allows all segments to be ranked by order of importance, i.e.\ of decreasing utility.
+\fresh{}
+Algorithm~\ref{sb:algo-optimal-order} computes the best order of segments from a viewpoint.
+It starts with an empty model, and it tries all the segments from a set of candidates, and computes the PSNR between the corresponding render and the ground truth render.
+With all those PSNRs, it is able to determine which segment is the one that brings the best $\Delta\text{PSNR} / s$, $s$ being the size of the segment in bytes.
+Once it found the best segment, it registers it, and starts again.
+That way, it is able to generate an order of segments sorted by $\Delta\text{PSNR} / s$.
+This order is then saved as a JSON file and a client will be able to download it to know which segments are more appropriate from a certain viewpoint.
+
+\begin{algorithm}[th]
+    \SetKwInOut{Input}{input}
+    \SetKwInOut{Output}{output}
+
+    \SetKwData{BookmarkViewpoint}{bookmark\_viewpoint}
+    \SetKwData{OptimalOrder}{optimal\_order}
+    \SetKwData{TotalModel}{total\_model}
+    \SetKwData{EmptyModel}{empty\_model}
+    \SetKwData{I}{i}
+    \SetKwData{BestSegment}{best\_segment}
+    \SetKwData{BestPsnr}{best\_delta\_psnr}
+    \SetKwData{PreviousPsnr}{previous\_psnr}
+    \SetKwData{None}{none}
+    \SetKwData{CurrentModel}{current\_model}
+    \SetKwData{EmptyRender}{empty\_render}
+    \SetKwData{CurrentRender}{current\_render}
+    \SetKwData{GroundTruthRender}{ground\_truth\_render}
+    \SetKwData{CurrentPsnr}{current\_psnr}
+    \SetKwData{CurrentDeltaPsnr}{current\_delta\_psnr}
+    \SetKwData{Segment}{segment}
+    \SetKwData{Candidates}{candidates}
+    \SetKwFunction{Render}{render}
+    \SetKwFunction{Psnr}{psnr}
+
+    \Input{The bookmark viewpoint, the ground truth render at this viewpoint, the candidate segments}
+    \Output{The optimal order of the segments}
+
+    \OptimalOrder\leftarrow{} []\;
+    \EmptyRender\leftarrow\Render{\EmptyModel,\BookmarkViewpoint}\;
+    \PreviousPsnr\leftarrow\Psnr(\EmptyRender,\GroundTruthRender)\;
+
+    \TotalModel\leftarrow\EmptyModel\;
+
+    \For{\I\in0\ldots200}{%
+
+        \BestSegment\leftarrow\None\;
+        \BestPsnr\leftarrow0\;
+
+        \For{\Segment\in\Candidates}{%
+
+            \CurrentModel\leftarrow\TotalModel\cup\Segment\;
+            \CurrentRender\leftarrow\Render{\CurrentModel,\BookmarkViewpoint}\;
+            \CurrentPsnr\leftarrow\Psnr{\CurrentRender,\GroundTruthRender}\;
+            \CurrentDeltaPsnr\leftarrow\CurrentPsnr$-$\PreviousPsnr\;
+
+            \If{\CurrentDeltaPsnr$>$\BestPsnr}{%
+                \BestSegment\leftarrow\Segment\;
+                \BestPsnr\leftarrow\CurrentDeltaPsnr\;
+            }
+        }
+        \OptimalOrder\leftarrow\OptimalOrder+\BestSegment\;
+        \TotalModel\leftarrow\TotalModel\cup\BestSegment\;
+    }
+    \caption{Computation of the optimal order of segments from a bookmark\label{sb:algo-optimal-order}}
+\end{algorithm}
+
+% This utility is simply the ratio between the number of pixels displaying that segment on screen, and the total screen area (in pixels).
+% This utility definition is the same for geometry and texture segments, which allows all segments to be ranked by order of importance, i.e.\ of decreasing utility.
+
+\copied{}
 
 \begin{figure}[th]
     \centering