\fresh{}
\section{Experiments}
\subsection{Setup}

\subsubsection{Tutorial}

The experiment starts with a tutorial, so the users can get accustomed to our interface.
This tutorial shows the different types of interactions available and explains how to use them.

\subsubsection{Bookmarks}

This part of the experiment consists in two 1 minute long sessions: the first one has a naked interface where the only available intarctions are translations and rotations of the camera, and the second one enhances the interface with bookmarks.
There are no special tasks other than to take a walk around the model.
This part ends with a small questionnaire where users are asked whether they prefer navigating with bookmarks, and they can use a text field to describe their reasons.

The main objective of this part of the experiment is not really to know whether people like using the bookmarks or not: we already know from our previous work and from the other parts of this experiment that they do like using the bookmarks.
This part most importantly acts as an extended tutorial: the first half trains the user with the controls, and the second half trains them with the bookmarks, and this is why we decided not to randomize those two halves.

\subsubsection{Streaming}

This part of the experiment also consists in two 1 minute long sessions that use different streaming policies.
One of those experiment has the default greedy policy described in \todo{add ref}, and the other one has the enhanced policy for bookmarks.
The order of those two sessions is randomized to avoid biases.

Since we know that the difference between our streaming policies are subtle, we designed a task a little more complex in order to highlight the differences so that the user can see it.
The user starts in the scene, and one of the bookmarks is blinking.
The user has to click the bookmark, and wait a little when he arrives at the destination.
Once some data has been downloaded, and the user is satisfied with the data downloaded, they can look for the next blinking bookmarks.
This setup is repeated for each streaming policy, and after the two sessions, the users have to answer a questionnaire asking the question \emph{In what session did you find the streaming the smoothest?}
The questionnaire also has a text field for users to explain their answer if they wish.

\subsubsection{Free navigation}

The last part of the experiment is a free navigation.
Diamonds are hidden in the scene, and are invisible until the user is close enough.
The users have to find the diamonds, and they can navigate by using the controls and the bookmarks.
The loading policy is the default greedy policy for half of the users, and the enhanced policy for bookmarks for the other half.

\subsection{Results}

\subsubsection{Qualitative results --- Interaction}

People use and enjoy using the bookmarks.
It helps them navigating in the scene, and the few people that do not like bookmarks most often have the following reasons:

\begin{itemize}
    \item they are already really confortable with using the virtual joystick
    \item they find using the virtual joystick funnier to use
\end{itemize}

We could argue that they do not like the bookmarks because they make the task too easy, and thus, less fun.

\subsubsection{Qualitative results --- Streaming}

The experiment confirms that the differences between the loading policies are subtle, and, unlike we expected, they seem to prefer the default greedy policy to the enhanced one.
Another thing is that people do not like waiting after clicking a bookmark, and making them wait in the second part of the experiment is difficult, which reduces even more the differences between the two policies, since the enhanced policy fallbacks to greedy when the camera moves.\todo{this sentence is ugly af}

\subsubsection{Quantitative results}

\begin{figure}[th]
    \centering
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=Time (in s),
            ylabel=PSNR,
            no markers,
            width=\tikzwidth,
            height=\tikzheight,
            cycle list name=mystyle,
            legend pos=south east,
            xmin=0,
            xmax=60,
            ymin=0,
        ]

            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/second-experiment-0.dat};
            \addlegendentry{Greedy}
            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/second-experiment-1.dat};
            \addlegendentry{Greedy optimized for bookmarks}

        \end{axis}
    \end{tikzpicture}
    \caption{Comparison of the PSNR during the second experiment}
\end{figure}

\begin{figure}[th]
    \centering
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=Time (in s),
            ylabel=PSNR,
            no markers,
            width=\tikzwidth,
            height=\tikzheight,
            cycle list name=mystyle,
            legend pos=south east,
            xmin=0,
            xmax=10,
        ]

            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/second-experiment-after-clicks-0.dat};
            \addlegendentry{Greedy}
            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/second-experiment-after-clicks-1.dat};
            \addlegendentry{Greedy optimized for bookmarks}

        \end{axis}
    \end{tikzpicture}
    \caption{Comparison of the PSNR after a click on a bookmark during the second experiment}
\end{figure}

\begin{figure}[th]
    \centering
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=Time (in s),
            ylabel=PSNR,
            no markers,
            width=\tikzwidth,
            height=\tikzheight,
            cycle list name=mystyle,
            legend pos=south east,
            xmin=0,
            xmax=60,
            ymin=0,
        ]

            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/third-experiment-0.dat};
            \addlegendentry{Greedy}
            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/third-experiment-1.dat};
            \addlegendentry{Greedy optimized for bookmarks}

        \end{axis}
    \end{tikzpicture}
    \caption{Comparison of the PSNR during the third experiment}
\end{figure}

\begin{figure}[th]
    \centering
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=Time (in s),
            ylabel=PSNR,
            no markers,
            width=\tikzwidth,
            height=\tikzheight,
            cycle list name=mystyle,
            legend pos=south east,
            xmin=0,
            xmax=10,
        ]

            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/third-experiment-after-clicks-0.dat};
            \addlegendentry{Greedy}
            \addplot table [y=y, x=x]{assets/system-bookmarks/simulation-results/third-experiment-after-clicks-1.dat};
            \addlegendentry{Greedy optimized for bookmarks}

        \end{axis}
    \end{tikzpicture}
    \caption{Comparison of the PSNR after a click on a bookmark during the third experiment}
\end{figure}
