dash-3d-tmp
/
phd
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

- experimental setup updated

This commit is contained in:
acarlier 2019-10-18 16:08:51 +02:00
parent 5782e6e709
commit 2219c2afab
1 changed files with 23 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
src/system-bookmarks/user-study.tex
View File

@ -3,65 +3,66 @@
\subsection{Preliminary user study} \subsection{Preliminary user study}
Before conducting the user study on mobile devices, we designed a user study for desktop devices. Before conducting the user study on mobile devices, we designed a preliminary user study for desktop devices.
This experiment was conducted on a little more than a dozen of people, with the model described in the previous chapter. This experiment was conducted on twelve users, using the 3D model described in the previous chapter (i.e. the Marina Bay district in Singapore).
Bookmarks were positioned from user-generated panoramic picture available on Google Maps, and the task consisted in retrieving spots on the 3D model from a picture: users were presented with an image coming from Google Street View and they had to find the corresponding spot in the 3D model. Bookmarks were sampled from the set of locations of user-uploaded panoramic pictures available on Google Maps, and the task consisted in matching real-world pictures to their virtual location on the 3D model: users were presented with an image coming from Google Street View and they were asked to find the exact same location in the 3D model.
Due to the fact the task was hard, and that our users were familiar with 3D navigation, they preferred navigating slowly in the scene, and did not use bookmarks as much as they did during the experiment we ran in Chapter~\ref{bi}. Due to the great difficulty of the task was, as well as the relative familiarity of the users with 3D navigation, the user behaviour was biased towards navigating slowly in the scene. The users almost never clicked the bookmarks, much less as they did during the experiment we ran in Chapter~\ref{bi}.
For these reasons, we decided to setup a new experiment, on a model a little larger, with a less complex task, and we decided to conduct this experiment on mobile device exclusively, to see how bookmarks help people navigate in a scene when controls are more cumbersome. For these reasons, we decided to setup a new experiment, with a less complex task, and we decided to conduct this experiment on mobile device exclusively, to see how bookmarks help people navigate in a scene when controls are more cumbersome.
\subsection{Mobile navigation user study} \subsection{Mobile navigation user study}
\subsubsection{Models} \subsubsection{Models}
In this user study, we use two models. In this user study, we display two successive 3D models to the users:
\textbf{A FIGURE WOULD BE NICE !!!}
\begin{itemize} \begin{itemize}
\item For the tutorial, we use a first model from a video game, representing a small scene, to maintain a good framerate and to prevent users from getting lost in the scene. \item For the tutorial phase, we use a model derived from a video game, representing a small scene, in order to prevent users from getting lost in the scene.
\item For all the other parts of the experiment, we used an extended version of the model used in the previous chapter. \item For all the other parts of the experiment, we use a larger version of the Singaporean district 3D model, that include neighbouring districts such as Central Business District.
\end{itemize} \end{itemize}
\subsubsection{Experiment} \subsubsection{Experiment}
The experiment consists in 4 phases: a tutorial, a comparison between interfaces with and without bookmarks, a comparison between two streaming policies, and a final navigation during which the user is looking for objects in the scene. The experiment is articulated into four phases: a tutorial, a comparison between interfaces with and without bookmarks, a comparison between two streaming policies, and a final navigation during which the user is looking for objects in the scene.
\paragraph{Tutorial} \paragraph{Tutorial}
The experiment starts with a tutorial, to get the users accustomed to our interface. The experiment starts with a tutorial, to get the users accustomed to the controls and the interface.
This tutorial shows the different types of interactions available and explains how to use them. This tutorial showcases the different types of interactions available, including bookmarks, and explains how to use them.
It then presents bookmarks to the users.
\paragraph{Bookmark} \paragraph{Bookmark}
This part of the experiment consists in two 1 minute long sessions: the first one has a naked interface where the only available interactions are translations and rotations of the camera, and the second one augments the interface with bookmarks. The second part of the experiment consists in two 1 minute long sessions: the first session displays a bookmarks-free interface where the only available interactions are translations and rotations of the camera, and the second one augments the interface with bookmarks.
There are no special tasks other than to take a walk around the model. There are no special tasks other than to navigate around the model.
This part ends with a small questionnaire where users are asked whether they prefer navigating with bookmarks, and they can use a text field to describe their reasons. The part ends with a small questionnaire where users are asked whether they prefer navigating with or without bookmarks, along with a text field to explain their answer.
The main objective of this part of the experiment is not really to know whether people like using the bookmarks or not: we already know from our previous work and from the other parts of this experiment that they do like using the bookmarks. The main objective of this part of the experiment is not really to know whether people like using the bookmarks or not: we already know from our previous work and from the other parts of this experiment that they do like using the bookmarks.
This part most importantly acts as an extended tutorial: the first half trains the users with the controls, and the second half trains them with the bookmarks, and this is why we decided not to randomize those two halves. This part most importantly acts as an extended tutorial: the first half trains the users with the basic controls, and the second half trains them to specifically use the bookmarks. This is why we decided not to randomize those two steps at this point.
\paragraph{Streaming} \paragraph{Streaming}
This part of the experiment also consists in two 1 minute long sessions that use different streaming policies. This part of the experiment also consists in two 1 minute long sessions that use different streaming policies.
One of those experiment has the default greedy policy described in~\ref{d3:dash-adaptation}, and the other one has the enhanced policy for bookmarks. One of those experiment has the default greedy policy described in~\ref{d3:dash-adaptation}, and the other one has the enhanced policy for bookmarks described in the previous section.
The order of those two sessions is randomized to avoid biases. The order of those two sessions is randomized to avoid biases.
% Since we know that the difference between our streaming policies is subtle, we designed a task a little more complex in order to highlight the differences so that the user can see it. % Since we know that the difference between our streaming policies is subtle, we designed a task a little more complex in order to highlight the differences so that the user can see it.
Since the behaviours of our streaming policy only differ when the user clicks a bookmark, we design a task where the users have to perform a guided tour of the scene, where each bookmark is a step of the tour. Since the behaviours of our streaming policy only differ when the user clicks a bookmark, we design a task where the users have to perform a guided tour of the scene, where each bookmark is a step of the tour.
The user starts in the scene, and one of the bookmarks is blinking. The user starts from anywhere in the scene, and one of the bookmarks is blinking.
The user has to touch the bookmark, and wait a little when he arrives at the destination. The user has to touch the bookmark, and observe the recommended viewpoint for a while when arriving at destination.
Once some data has been downloaded, and the user is satisfied with the data downloaded, they can look for the next blinking bookmarks. Once some data has been downloaded and the user could get a feeling about the quality of the streaming, another bookmark starts blinking to move one with the tour.
This setup is repeated for each streaming policy, and after the two sessions, the users have to answer a questionnaire asking the question \emph{In what session did you find the streaming the smoothest?} This setup is repeated for each streaming policy, and after the two sessions, the users have to answer a questionnaire asking the question \emph{In what session did you find the streaming the smoothest?}
The questionnaire also has a text field for users to explain their answer if they wish. The questionnaire also has a text field for users to explain their answer if they wish.
\paragraph{Free navigation} \paragraph{Free navigation}
The last part of the experiment is a free navigation. The last part of the experiment is a free navigation with an object-finding task.
Diamonds are hidden in the scene, and are invisible until the user is close enough. Diamonds are hidden in the scene, and are invisible until the user is close enough.
The users have to find the diamonds, and they can navigate by using indifferently the controls and the bookmarks. The users have to find the diamonds, and they can navigate by using indifferently the controls and the bookmarks.
The loading policy is the default greedy policy for half of the users, and the enhanced policy for bookmarks for the other half, and this order has been randomized. The loading policy is the default greedy policy for half of the users, and the enhanced policy for bookmarks for the other half, and this order has been randomized.
With this part of the experiment, we hope to see differences in terms of PSNR for the two policies, when users are not forced to click on bookmarks.
This is the most important part of the study, as we aim at observing several aspects. First, we hope that users navigate using the bookmarks. Since no guideline has been given to them as to how to interact, we want to observe whether they naturally use the bookmarks or not.
In addition, we want to prove the superiority of our bookmark-optimized streaming policy by observing that users tend to perceive a better visual quality (as measured by the PSNR).
\subsubsection{Setup} \subsubsection{Setup}
During these experiments, we need a server and a client. During these experiments, we need a server and a client.