260 lines
13 KiB
TeX
260 lines
13 KiB
TeX
\fresh{}
|
|
\section{Evaluation}\label{sb:evaluation}
|
|
|
|
\subsection{Preliminary user study}
|
|
|
|
Before conducting the user study on mobile devices, we designed a user study for desktop devices.
|
|
This experiment was conducted on a little more than a dozen of people, with the model described in the previous chapter.
|
|
Bookmarks were positioned from user-generated panoramic picture available on Google Maps, and the task consisted in retrieving spots on the 3D model from a picture: users were presented with an image coming from Google Street View and user had to find the corresponding spot in the 3D model.
|
|
|
|
Due to the fact the task was hard, and that our users were familiar with 3D navigation, they preferred navigating slowly in the scene, and did not use bookmarks as much as they did during the experiment we ran in Chapter~\ref{bi}.
|
|
|
|
For these reasons, we decided to setup a new experiment, on a model a little larger, with a less complex task, and we decided to conduct this experiment on mobile device exclusively, to see how bookmarks help people navigate in a scene when controls are more cumbersome.
|
|
|
|
\subsection{Mobile navigation user study}
|
|
|
|
\subsubsection{Models}
|
|
|
|
In this user study, we use two models.
|
|
\begin{itemize}
|
|
\item For the tutorial, we use a first model from a video game, representing a small scene, to maintain a good framerate and to prevent users from getting lost in the scene.
|
|
\item For all the other parts of the experiment, we used an extended version of the model used in the previous chapter.
|
|
\end{itemize}
|
|
|
|
\subsubsection{Experiment}
|
|
|
|
The experiment consists in 4 phases: a tutorial, a comparison between interfaces with and without bookmarks, a comparison between two streaming policies, and a final navigation during which the user is looking for objects in the scene.
|
|
|
|
\paragraph{Tutorial}
|
|
|
|
The experiment starts with a tutorial, to get the users accustomed to our interface.
|
|
This tutorial shows the different types of interactions available and explains how to use them.
|
|
It then presents bookmarks to the users.
|
|
|
|
\paragraph{Bookmark}
|
|
|
|
This part of the experiment consists in two 1 minute long sessions: the first one has a naked interface where the only available interactions are translations and rotations of the camera, and the second one augments the interface with bookmarks.
|
|
There are no special tasks other than to take a walk around the model.
|
|
This part ends with a small questionnaire where users are asked whether they prefer navigating with bookmarks, and they can use a text field to describe their reasons.
|
|
|
|
The main objective of this part of the experiment is not really to know whether people like using the bookmarks or not: we already know from our previous work and from the other parts of this experiment that they do like using the bookmarks.
|
|
This part most importantly acts as an extended tutorial: the first half trains the users with the controls, and the second half trains them with the bookmarks, and this is why we decided not to randomize those two halves.
|
|
|
|
\paragraph{Streaming}
|
|
|
|
This part of the experiment also consists in two 1 minute long sessions that use different streaming policies.
|
|
One of those experiment has the default greedy policy described in~\ref{d3:dash-adaptation}, and the other one has the enhanced policy for bookmarks.
|
|
The order of those two sessions is randomized to avoid biases.
|
|
|
|
% Since we know that the difference between our streaming policies is subtle, we designed a task a little more complex in order to highlight the differences so that the user can see it.
|
|
Since the behaviours of our streaming policy only differ when the user clicks a bookmark, we designed a task where the users have to perform a guided tour of the scene, where each bookmark is a step of the tour.
|
|
The user starts in the scene, and one of the bookmarks is blinking.
|
|
The user has to click the bookmark, and wait a little when he arrives at the destination.
|
|
Once some data has been downloaded, and the user is satisfied with the data downloaded, they can look for the next blinking bookmarks.
|
|
This setup is repeated for each streaming policy, and after the two sessions, the users have to answer a questionnaire asking the question \emph{In what session did you find the streaming the smoothest?}
|
|
The questionnaire also has a text field for users to explain their answer if they wish.
|
|
|
|
\paragraph{Free navigation}
|
|
|
|
The last part of the experiment is a free navigation.
|
|
Diamonds are hidden in the scene, and are invisible until the user is close enough.
|
|
The users have to find the diamonds, and they can navigate by using indifferently the controls and the bookmarks.
|
|
The loading policy is the default greedy policy for half of the users, and the enhanced policy for bookmarks for the other half, and this order has been randomized.
|
|
With this part of the experiment, we hope to see differences in terms of PSNR for the two policies, when users are not forced to click on bookmarks.
|
|
|
|
\subsubsection{Apparatus\todo{lol, i like this title but im not sure}}
|
|
|
|
During these experiments, we need a server and a client.
|
|
The server is hosted on an Acer Aspire V3 with an Intel Core i7 3632QM processor.
|
|
The user is given a One Plus 5 that is connected to the server via Wi-fi.
|
|
There is no artificial bandwidth limitation due to the fact that the bandwidth is already limited by the Wi-fi network and by the performances of the mobile device.
|
|
|
|
\subsection{Results}
|
|
|
|
18 users participated in this user-study, 15 males and 3 females, average age is 20.7 and standard deviation is 0.53.
|
|
We only proposed this user study to relatively young people to ensure they are used to mobile devices.
|
|
|
|
\subsubsection{Qualitative results --- Interaction}
|
|
|
|
People use and enjoy using the bookmarks.
|
|
It helps them navigating in the scene, and the few people that do not like bookmarks most often have the following reasons:
|
|
|
|
\begin{itemize}
|
|
\item they are already really comfortable with using the virtual joystick
|
|
\item they find using the virtual joystick funnier to use
|
|
\end{itemize}
|
|
|
|
We could argue that they do not like the bookmarks because they make the task too easy, and thus, less fun.
|
|
|
|
\subsubsection{Qualitative results --- Streaming}
|
|
|
|
Among the 18 participants of this user study, 10 confirmed that they preferred the optimized policy, 4 preferred the greedy policy, and 4 did not perceive the difference.
|
|
Another interesting fact is that on the last part of the experiment (the free navigation) the average number of clicks on bookmarks is 3 for users having the greedy policy and 5.3 for users having the optimized policy.
|
|
Even though statistical significance is not reached, this result seems to indicate that a policy optimized for bookmarks could lead users to click more on bookmarks.
|
|
|
|
\subsubsection{Quantitative results}
|
|
|
|
By collecting all the traces during the experiments, we are able to replay the rendering and evaluate the PSNR that users got during their experiment.
|
|
Figure~\ref{sb:psnr-second-experiment} shows the average PSNR that user got while navigating during the second experiment (bookmark path).
|
|
Below the PSNR curve is a curve that shows how many users were moving to or staying at a bookmark position.
|
|
As we can see, the two policies perform in the same way in the beginning when few users are moving to a bookmarks.
|
|
However, when they start clicking on bookmarks, the gap grows and our optimized policy perform better.
|
|
Figure~\ref{sb:psnr-second-experiment-after-click} shows the PSNR after a click on a bookmark.
|
|
To compute these curves, we isolated the ten seconds after each click on a bookmark that occurs and we averaged them all.
|
|
These curves isolate the effect of our optimized policy, and shows the difference a user can feel when clicking on a bookmark.
|
|
|
|
Figures~\ref{sb:psnr-third-experiment} and~\ref{sb:psnr-third-experiment-after-click} represent the same curves on the third experiment (free navigation).
|
|
On average, the difference in terms of PSNR is less obvious, and both strategies seem to perform the same way.
|
|
This may be due to the lower number of users clicking on bookmarks.
|
|
However, Figure~\ref{sb:psnr-third-experiment-after-click} is clear: the optimized policy performs way better after a click on a bookmark.
|
|
|
|
\begin{figure}[th]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
ylabel=PSNR,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=60,
|
|
ymin=0,
|
|
name=first plot,
|
|
xmajorticks=false,
|
|
]
|
|
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-0.dat};
|
|
\addlegendentry{Greedy}
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-1.dat};
|
|
\addlegendentry{Greedy optimized for bookmarks}
|
|
|
|
\end{axis}
|
|
|
|
\begin{axis}[
|
|
xlabel=Time (in s),
|
|
ylabel=Ratio of clicks,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzhalfheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=60,
|
|
ymin=0,
|
|
ymax=1,
|
|
at=(first plot.south),
|
|
anchor=north,
|
|
yshift=-0.5cm,
|
|
]
|
|
|
|
\addplot[smooth, color=DarkGreen] table [y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-2.dat};
|
|
\addplot[smooth, color=blue] table [dashed, y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-3.dat};
|
|
|
|
\end{axis}
|
|
|
|
\end{tikzpicture}
|
|
\caption{Comparison of the PSNR during the second experiment: above, PSNR for greedy and greedy optimized for bookmarks; below, ratio of people clicking on a bookmark.\label{sb:psnr-second-experiment}}
|
|
\end{figure}
|
|
|
|
\begin{figure}[th]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
xlabel=Time (in s),
|
|
ylabel=PSNR,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=10,
|
|
]
|
|
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-after-clicks-0.dat};
|
|
\addlegendentry{Greedy}
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/second-experiment-after-clicks-1.dat};
|
|
\addlegendentry{Greedy optimized for bookmarks}
|
|
|
|
\end{axis}
|
|
\end{tikzpicture}
|
|
\caption{Comparison of the PSNR after a click on a bookmark during the second experiment\label{sb:psnr-second-experiment-after-click}}
|
|
\end{figure}
|
|
|
|
\begin{figure}[th]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
ylabel=PSNR,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=60,
|
|
ymin=0,
|
|
name=first plot,
|
|
xmajorticks=false,
|
|
]
|
|
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-0.dat};
|
|
\addlegendentry{Greedy}
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-1.dat};
|
|
\addlegendentry{Greedy optimized for bookmarks}
|
|
|
|
\end{axis}
|
|
|
|
\begin{axis}[
|
|
xlabel=Time (in s),
|
|
ylabel=Ratio of clicks,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzhalfheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=60,
|
|
ymin=0,
|
|
ymax=1,
|
|
at=(first plot.south),
|
|
anchor=north,
|
|
yshift=-0.5cm,
|
|
]
|
|
|
|
\addplot[smooth, color=DarkGreen] table [y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-2.dat};
|
|
\addplot[smooth, color=blue] table [dashed, y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-3.dat};
|
|
|
|
\end{axis}
|
|
|
|
\end{tikzpicture}
|
|
\caption{Comparison of the PSNR during the third experiment: above, PSNR for greedy and greedy optimized for bookmarks; below, ratio of people clicking on a bookmark.\label{sb:psnr-third-experiment}}
|
|
\end{figure}
|
|
|
|
\begin{figure}[th]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
xlabel=Time (in s),
|
|
ylabel=PSNR,
|
|
no markers,
|
|
width=\tikzwidth,
|
|
height=\tikzheight,
|
|
cycle list name=mystyle,
|
|
legend pos=south east,
|
|
xmin=0,
|
|
xmax=10,
|
|
]
|
|
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-after-clicks-0.dat};
|
|
\addlegendentry{Greedy}
|
|
\addplot table [y=y, x=x]{assets/system-bookmarks/final-results/third-experiment-after-clicks-1.dat};
|
|
\addlegendentry{Greedy optimized for bookmarks}
|
|
|
|
\end{axis}
|
|
\end{tikzpicture}
|
|
\caption{Comparison of the PSNR after a click on a bookmark during the third experiment\label{sb:psnr-third-experiment-after-click}}
|
|
\end{figure}
|