System bookmarks
This commit is contained in:
parent
9f73fde350
commit
b24c4ba92b
Binary file not shown.
After Width: | Height: | Size: 2.5 MiB |
Binary file not shown.
After Width: | Height: | Size: 2.1 MiB |
|
@ -8,7 +8,7 @@ anchorcolor = blue]{hyperref}
|
|||
\usepackage{xspace}
|
||||
\usepackage{url}
|
||||
\usepackage{lmodern}
|
||||
\usepackage{algorithm2e}
|
||||
\usepackage[ruled]{algorithm2e}
|
||||
\usepackage{datatool}
|
||||
\usepackage{pgfplots}
|
||||
\usepackage{graphicx}
|
||||
|
|
|
@ -25,7 +25,7 @@ It can be chosen directly by the user or automatically determined by analysing t
|
|||
\caption{The different resolutions available for a Youtube video}
|
||||
\end{figure}
|
||||
|
||||
In the same way, the recent works in 3D streaming have proposed many ways to progressively streaming 3D models, allowing the user to have a low resolution without having to wait, and being able to interact with the model while the details are being downloaded.
|
||||
In the same way, the recent work in 3D streaming have proposed many ways to progressively streaming 3D models, allowing the user to have a low resolution without having to wait, and being able to interact with the model while the details are being downloaded.
|
||||
|
||||
\subsection{Media types}
|
||||
|
||||
|
|
|
@ -11,6 +11,8 @@ chapterprefix=true% like in standard class "report"
|
|||
\usepackage{listing-config}
|
||||
|
||||
\begin{document}
|
||||
\let\rawref\ref%
|
||||
\renewcommand{\ref}[1]{\rawref{#1} (page~\pageref{#1})}
|
||||
|
||||
\frontmatter
|
||||
|
||||
|
|
|
@ -6,7 +6,7 @@ We now describe an experiment that we conducted on 51 participants, with two goa
|
|||
First, we want to measure the impact of 3D bookmarks on navigation within an NVE\@.
|
||||
Second, we want to collect traces from the users so that we can replay them for reproducible experiments for comparing streaming strategies in Section 4.
|
||||
|
||||
\subsection{Our NVE\ref{bi:our-nve}}
|
||||
\subsection{Our NVE\label{bi:our-nve}}
|
||||
To ease the deployment of our experiments to users in distributed locations on a crowdsourcing platform, we implement a simple Web-based NVE client using THREE.js\footnote{http://threejs.org}.
|
||||
The NVE server is implemented with node.js\footnote{http://nodejs.org}.
|
||||
The NVE server streams a 3D scene to the client; the client renders the scene as the 3D content are received.
|
||||
|
@ -45,7 +45,7 @@ The interface also includes a button to reset the camera back to the starting po
|
|||
\end{figure}
|
||||
|
||||
|
||||
\subsection{3D Bookmarks}
|
||||
\subsection{3D Bookmarks\label{bi:3d-bookmarks}}
|
||||
Our NVE supports 3D bookmarks.
|
||||
A 3D bookmark, or bookmark for short, is simply a fixed camera location (in 3D space), a view direction, and a focal.
|
||||
Bookmarks visible from the user's current viewpoint are shown as 3D objects in the scene.
|
||||
|
|
|
@ -9,6 +9,7 @@ Regardind desktop interaction, we keep the interaction we described in Section~\
|
|||
\item W, A, S and D keys to translate the camera;
|
||||
\item mouse motions to rotate the camera.
|
||||
\end{itemize}
|
||||
A screenshot of this interface is displayed in Figure~\ref{sb:desktop}.
|
||||
|
||||
\subsection{Mobile interaction}
|
||||
|
||||
|
@ -16,8 +17,8 @@ Regardind desktop interaction, we keep the interaction we described in Section~\
|
|||
Mobile interactions are more complex because the user does not have neither keyboard nor mouse to interact with.
|
||||
However, there are some other sensors on most mobile devices that can help interaction.
|
||||
The most useful sensor for 3D interaction on mobile devices is definitely the gyroscope.
|
||||
We use the gyroscope to enable a user to turn his device to turn the virtual camera.
|
||||
We also add the possibility to turn the camera by drag and dropping the scene.
|
||||
We use the gyroscope to enable a user to rotate his device to rotate the virtual camera.
|
||||
We also add the possibility to rotate the camera by drag and dropping the scene.
|
||||
This way, the user is not forced to perform a real-world half-turn to be able to look behind or to keep its device pointing to the sky (which can quickly become tiring) to look up.
|
||||
These interactions, however, do not allow the user to move the camera: he can rotate it but not translate it.
|
||||
For this reason, we display a small joystick on the bottom left corner of the screen that mimics the first person video games interactions and allow the user translating the camera:
|
||||
|
@ -26,43 +27,127 @@ For this reason, we display a small joystick on the bottom left corner of the sc
|
|||
\item moving the joystick down makes the camera move backwards;
|
||||
\item moving the joystick sideways makes the camera move sidewars.
|
||||
\end{itemize}
|
||||
A screenshot of this interface is displayed in Figure~\ref{sb:mobile}.
|
||||
|
||||
\copied{}
|
||||
\section{Adding bookmarks into DASH NVE framework\label{sb:bookmarks}}
|
||||
|
||||
\copied{}
|
||||
In this section, we explain how to include a new interaction in the system described in the previous chapter.
|
||||
\fresh{}
|
||||
|
||||
\subsection{Interaction --- Visual}
|
||||
|
||||
We decide to add bookmarks of recommended viewpoints in the 3D scene.
|
||||
A bookmark is defined by a camera position, orientation, and intrinsic parameters, and offers a particular view of the 3D virtual environment.
|
||||
As such, it should be represented using a widget that (i) is attached to a particular position, therefore appearing small (respectively big) when it is far (respectively close), and (ii) points at a particular direction, allowing the user to predict what can be seen from this viewpoint.
|
||||
Bookmarks have been already introduced in the literature, with various appearances \todo[inline]{figure of possible appearances}.
|
||||
Since, no particular preeminence of one design on the others has been demonstrated in previous work, we arbitrarily choose to use \todo[inline]{type of bookmark} in this work.
|
||||
In Chapter~\ref{bi} Section~\ref{bi:3d-bookmarks}, we described two 3D widgets that we used to display bookmarks to users.
|
||||
One of the conclusions of the user-study, described in Section~\ref{bi:user-study}, was that the impact of the way we display bookmark was not significant.
|
||||
In this work, we chose a slightly different way of representing bookmarks due to some concerns with our original representations:
|
||||
\begin{itemize}
|
||||
\item viewport bookmarks are simple, but people who are not in the field of vision are not familiar with this representation;
|
||||
\item arrow bookmarks are complex, and need to be refreshed when the camera moves, which can harm the framerate of the rendering.
|
||||
\end{itemize}
|
||||
For these reasons, we changed the display to a vertical bar with a 2D sprite of a pictural representation of an eye.
|
||||
This 2D sprite is always facing the camera to prevent it from being invisible when the camera would be on the side of it.
|
||||
Screenshots of user interfaces with bookmarks are available in Figures~\ref{sb:desktop} and~\ref{sb:mobile}.
|
||||
|
||||
Bookmarks can be created either automatically, or manually defined by an expert user (e.g.\ the 3D model designer, or administrator).
|
||||
Bookmarks could even be derived from the observation of user behavior, by focusing on the most visited areas of the models.
|
||||
Automated ways of defining bookmarks or adapting them to user behavior is beyond the scope of this paper; methods have been proposed in \todo[inline]{References}
|
||||
The size of the sprite changes when time goes by following a sine to help the user differenciate what is part of the scene and what is extra widgets.
|
||||
Since our scene is static, a user knows when seeing a bookmark that it is not part of the scene.
|
||||
|
||||
We choose to implement two interactions with bookmarks.
|
||||
The first, most obvious one, is to position the user camera on the bookmark's viewpoint when the user clicks on the bookmark.
|
||||
In order to avoid users to lose context, clicking on a bookmark triggers an automatic, smooth, camera displacement that ends up at the bookmark.
|
||||
% We use Hermite's polynomials to compute this displacement, as proposed in MMSYS16. Lol we don't :'(
|
||||
We implement an additional interaction that displays a preview of the bookmark's viewpoint while it is hovered by the user's mouse.
|
||||
A small thumbnail of the viewport is displayed below the bookmark.
|
||||
The other parameters of the bookmarks remain unchanged since Chapter~\ref{bi}: in order to avoid users to lose context, clicking on a bookmark triggers an automatic, smooth, camera displacement that ends up at the bookmark.
|
||||
We also display a thumbnail of the bookmark's viewpoint when the mouse hovers over a bookmark.
|
||||
Such thumbnail is displayed in Figure~\ref{sb:desktop}.
|
||||
Note that since on mobile, there is no mouse and thus no pointer, thumbnails are never downloaded nor displayed.
|
||||
|
||||
\begin{figure}[ht]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{assets/system-bookmarks/screenshots/desktop.png}
|
||||
\caption{Screenshot of the desktop version, with a bookmark and its thumbnail on the bottom left corner\label{sb:desktop}}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}[ht]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{assets/system-bookmarks/screenshots/mobile.png}
|
||||
\caption{Screenshot of the mobile version, with its joystick on the bottom left corner\label{sb:mobile}}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Segments utility at bookmarked viewpoint\label{sb:utility}}
|
||||
\copied{}
|
||||
Introducing bookmarks is a way to make users navigation more predictable.
|
||||
Indeed, since they are emphasized and, in a way, recommended viewpoints, bookmarks are more likely to be visited by a significant portion of users than any other viewpoint on the scene.
|
||||
As such, bookmarks can be used as a way to optimize streaming by downloading segments in an optimal, pre-computed order.
|
||||
|
||||
More specifically, segment utility as introduced in Section~\ref{d3:utility} is only an approximation of the segment's true contribution to the current viewpoint rendering.
|
||||
When bookmarks are defined, it is possible to obtain a perfect measure of segment utility by performing an offline rendering at each bookmark's viewpoint.
|
||||
Then, by simply counting the number of pixels that are rendered using each segment, we can rank the segments by order of importance in the rendering.
|
||||
% Then, by simply counting the number of pixels that are rendered using each segment, we can rank the segments by order of importance in the rendering.
|
||||
|
||||
We define $\mathcal{U}^{*} (s,B_i)$ as being the true utility of a segment $s$ in a viewpoint defined at bookmark $B_i$.
|
||||
This utility is simply the ratio between the number of pixels displaying that segment on screen, and the total screen area (in pixels).
|
||||
This utility definition is the same for geometry and texture segments, which allows all segments to be ranked by order of importance, i.e.\ of decreasing utility.
|
||||
|
||||
\fresh{}
|
||||
Algorithm~\ref{sb:algo-optimal-order} computes the best order of segments from a viewpoint.
|
||||
It starts with an empty model, and it tries all the segments from a set of candidates, and computes the PSNR between the corresponding render and the ground truth render.
|
||||
With all those PSNRs, it is able to determine which segment is the one that brings the best $\Delta\text{PSNR} / s$, $s$ being the size of the segment in bytes.
|
||||
Once it found the best segment, it registers it, and starts again.
|
||||
That way, it is able to generate an order of segments sorted by $\Delta\text{PSNR} / s$.
|
||||
This order is then saved as a JSON file and a client will be able to download it to know which segments are more appropriate from a certain viewpoint.
|
||||
|
||||
\begin{algorithm}[th]
|
||||
\SetKwInOut{Input}{input}
|
||||
\SetKwInOut{Output}{output}
|
||||
|
||||
\SetKwData{BookmarkViewpoint}{bookmark\_viewpoint}
|
||||
\SetKwData{OptimalOrder}{optimal\_order}
|
||||
\SetKwData{TotalModel}{total\_model}
|
||||
\SetKwData{EmptyModel}{empty\_model}
|
||||
\SetKwData{I}{i}
|
||||
\SetKwData{BestSegment}{best\_segment}
|
||||
\SetKwData{BestPsnr}{best\_delta\_psnr}
|
||||
\SetKwData{PreviousPsnr}{previous\_psnr}
|
||||
\SetKwData{None}{none}
|
||||
\SetKwData{CurrentModel}{current\_model}
|
||||
\SetKwData{EmptyRender}{empty\_render}
|
||||
\SetKwData{CurrentRender}{current\_render}
|
||||
\SetKwData{GroundTruthRender}{ground\_truth\_render}
|
||||
\SetKwData{CurrentPsnr}{current\_psnr}
|
||||
\SetKwData{CurrentDeltaPsnr}{current\_delta\_psnr}
|
||||
\SetKwData{Segment}{segment}
|
||||
\SetKwData{Candidates}{candidates}
|
||||
\SetKwFunction{Render}{render}
|
||||
\SetKwFunction{Psnr}{psnr}
|
||||
|
||||
\Input{The bookmark viewpoint, the ground truth render at this viewpoint, the candidate segments}
|
||||
\Output{The optimal order of the segments}
|
||||
|
||||
\OptimalOrder\leftarrow{} []\;
|
||||
\EmptyRender\leftarrow\Render{\EmptyModel,\BookmarkViewpoint}\;
|
||||
\PreviousPsnr\leftarrow\Psnr(\EmptyRender,\GroundTruthRender)\;
|
||||
|
||||
\TotalModel\leftarrow\EmptyModel\;
|
||||
|
||||
\For{\I\in0\ldots200}{%
|
||||
|
||||
\BestSegment\leftarrow\None\;
|
||||
\BestPsnr\leftarrow0\;
|
||||
|
||||
\For{\Segment\in\Candidates}{%
|
||||
|
||||
\CurrentModel\leftarrow\TotalModel\cup\Segment\;
|
||||
\CurrentRender\leftarrow\Render{\CurrentModel,\BookmarkViewpoint}\;
|
||||
\CurrentPsnr\leftarrow\Psnr{\CurrentRender,\GroundTruthRender}\;
|
||||
\CurrentDeltaPsnr\leftarrow\CurrentPsnr$-$\PreviousPsnr\;
|
||||
|
||||
\If{\CurrentDeltaPsnr$>$\BestPsnr}{%
|
||||
\BestSegment\leftarrow\Segment\;
|
||||
\BestPsnr\leftarrow\CurrentDeltaPsnr\;
|
||||
}
|
||||
}
|
||||
\OptimalOrder\leftarrow\OptimalOrder+\BestSegment\;
|
||||
\TotalModel\leftarrow\TotalModel\cup\BestSegment\;
|
||||
}
|
||||
\caption{Computation of the optimal order of segments from a bookmark\label{sb:algo-optimal-order}}
|
||||
\end{algorithm}
|
||||
|
||||
% This utility is simply the ratio between the number of pixels displaying that segment on screen, and the total screen area (in pixels).
|
||||
% This utility definition is the same for geometry and texture segments, which allows all segments to be ranked by order of importance, i.e.\ of decreasing utility.
|
||||
|
||||
\copied{}
|
||||
|
||||
\begin{figure}[th]
|
||||
\centering
|
||||
|
|
Loading…
Reference in New Issue