We now describe an experiment that we conducted on 51 participants, with two goals in mind.
First, we want to measure the impact of 3D bookmarks on navigation within an NVE\@.
Second, we want to collect traces from the users so that we can replay them for reproducible experiments for comparing streaming strategies in Section~\ref{bi:system}.
To ease the deployment of our experiments to users in distributed locations on a crowdsourcing platform, we implement a simple web-based NVE client using THREE.js// \footnote{http://threejs.org}.
The NVE server is implemented with node.js. // \footnote{http://nodejs.org}.
The NVE server streams a 3D scene to the client; the client renders the scene as the 3D content is received.
The user can navigate within the NVE in the following way; the camera can be translated using the arrow keys along four directions: forward, backward, to the left, and to the right.
Alternatively, the keys W, A, S and D can also be used for the same actions.
This choice was inspired by 3D video games, which often use these keys in conjunction with the mouse to move an avatar.
The virtual camera can rotate in four different directions using the keys I, K, J and L.
The user can also rotate the camera by dragging the mouse in the desired direction.
Finally, following the UI of popular 3D games, we also give users the possibility to lock their pointer and use their mouse as a virtual camera.
The mouse movement controls the camera rotation.
The user can always choose to lock the pointer, or unlock it using the escape key.
The interface also includes a button to reset the camera back to the starting position in the scene.
A 3D bookmark, or bookmark for short, is simply a fixed camera location (in 3D space), a view direction, and a focal.
Bookmarks visible from the user's current viewpoint are shown as 3D objects in the scene.
Figure X depicts some bookmarks from our NVE.
The user can click on a bookmark object to automatically move and align its viewpoint to that of the bookmark.
The movement follows a Hermite curve joining the current viewpoint to the viewpoint of the bookmark.
The tangent of the curve is the view direction.
The user can hover the mouse pointer over a bookmark object to see a thumbnail view of the 3D scene as seen from the bookmark. (Figure X, bottom left).
In our work, we consider two different possibilities for displaying bookmarks: viewports (Figure X top left) and arrows (Figure X top right).
A viewport is displayed as a pyramid where the top corresponds to the optical center of its viewpoint and the base corresponds to its image plane.
The arrows are view dependent.
The bottom of the arrow turns towards the current position, to better visualize the relative position of the bookmark.
Bookmarks allow the user to achieve a large movement within the 3D environment using a single action (a mouse click).
Since bookmarks are part of the scene, they are visible only when not hidden by other objects from the scene.
We chose size and colors that are salient enough to be easily seen, but not too large to limit the occlusion of regions within the scene.
When reaching the bookmark, the corresponding arrow or viewport is not visible anymore, and subsequently will appear in a different color, to indicate that it has been clicked (similar to web links).
We use four 3D scenes (one for the tutorial and three for the actual experiments) which represent recreated scenes from a famous video game.
Those models are light (a few thousand of triangles per model) and are sent before the experiment starts.
We keep the models small so that users can perform the task with acceptable latency from any country using a decent internet connection.
Our NVE does not stream the 3D content for these experiments, in order to avoid unreliable conditions caused by the network bandwidth variation, which might affect how the users interact.
Since we are interested in studying how efficiently users navigate in the 3D scene, we ask our participants to complete a task which forces them to visit, at least partially, various regions in the scene.
To this end, we hide a set of 8 coins on the scene: participants are asked to collect the coins by clicking on them.
In order to avoid any bias due to the coins position, we predefined 50 possible coin locations per scene, and randomly select 8 out of these 50 positions each time a new participant starts the experiment.
Participants are first presented with an initial screen to collect some preliminary information: age, gender, the last time they played 3D video games, and self-rated 3D gaming skills. We ask those questions because we believe that someone who is used to playing 3D video games should browse the scene more easily, and thus, may not need to use our bookmarks.
Then, the participants go through a tutorial to learn how the UI works, and how to complete the task.
The different interactions (keyboard navigation, mouse navigation, bookmarks interaction) are progressively introduced to participants, and the tutorial ends once the participant completes an easy version of the task.
The tutorial is always performed on the same scene.
Then, each participant has to complete the task three times.
Each task is performed on a different scene, with a different interface.
A NoReco interface lets the participant navigates without any bookmarks.
The other two interfaces allow a participant to move using bookmarks displayed as viewports (denoted as Viewports) and arrows (denoted as Arrows) respectively.
The coins are chosen randomly, based on the coin configurations that were used by previous participants: if another participant has done an experiment with a certain set of coins, on a certain scene, with a certain type of bookmarks, the current participant will do the experiment with the same set of coins, on the same scene, but with a different type of bookmarks.
This policy allows us to limit the bias that could be caused by coin locations.
Once a participant has found all coins, a button is shown on the interface to let the participant move to the next step.
Alternatively, this button may appear one minute after the sixth coin was found.
This means that a user is authorized to move on without completing the task, in order to avoid potential frustration caused by not finding the remaining two coins.
After completing the three tasks, the participants have to answer a set of questions about their experience with the bookmarks (we refer to the bookmarks as _recommendations_ in the experiments).
@bi:questions shows the list of questions.
#figure(
table(
columns: (auto, auto, auto),
align: left,
[], [*Questions*], [*Answers*],
[1], [What was the difficulty level WITHOUT recommendation?], [3.04 / 5 $plus.minus 0.31$],
[2], [What was the difficulty level WITH recommendation?], [2.15 / 5, $plus.minus 0.30$],
[3], [Did the recommendations help you to find the coins?], [42 Yes, 5 No],
[4], [Did the recommendations help you to browse the scene?], [49 Yes, 2 No],
[5], [Do you think recommendations can be helpful?], [49 Yes, 2 No],
The answers are summarized in Table~\ref{bi:questions}.
Note that not all questions were answered by all participants.
The participants seem to find the task to be of average difficulty (3.04/5) when they have no bookmarks to help their navigation.
They judge the task to be easier in average (2.15/5) with bookmarks, which indicates that bookmarks ease the completion of the task.
Almost all users (49 out of 51) think the bookmarks are useful for browsing the scene, and most users (42 out of 51) think bookmarks are also useful to complete the given task.
This is slightly in contradiction with our setup; even if coins may appear in some bookmarked viewpoints (which is normal since the viewpoints have been chosen to get the most complete coverage of the scene), most of the time no coin is visible in a given bookmark, and there are always coins that are invisible from all bookmarks.
The strongest result is that almost all users (49 out of 51) find bookmarks to be helpful.
Interestingly, and regardless of the bookmark type, users who have bookmarks complete the task more than half of the time, and spend in average significantly less time to complete the task: 2 minutes and 16 seconds using Viewports and 2 minutes and 33 seconds using Arrows.
Although Viewports seem to help users a little bit more in completing the task than Arrows, the performance difference between both types of bookmarks is not significant enough to conclude on which type of bookmarks is best.
We found that the difference in mean number of coins collected with and without bookmarks is not high enough to be statistically significant: we would need more experiments to reach the significance.
The mean time spent on the task however is statistically significant.
#figure(
table(
columns: (auto, auto, auto, auto),
[*BM type*], [*Total distance*], [*Distance to a bookmark*], [*Ratio*],
The table shows that this higher speed is due to the bookmarks, as more than 60% of the distance traveled by users with bookmarks happens when users click on bookmarks and fly to the destination.
In the previous paragraphs, we have shown how bookmarks are well perceived by users (looking at the questionnaire answers).
We also showed that users tend to be more efficient in completing the task when they have bookmarks than when they do not.
We can say that bookmarks have a positive impact on navigation within the 3D scene, but since users move, on average, twice as fast, it might have a negative impact on the streaming of objects to the client.
// \caption{Comparison of the triangles queried after a certain time}\label{bi:triangles-curve}
// \end{figure}
Figure X shows a CDF of the percentage of 3D mesh triangles in the scene that have been queried by users after a certain time. We plotted this same curve for users with and without bookmarks.
As expected, the fact that the users can browse the scene significantly quicker with bookmarks reflects on the demand on the 3D content.
Users need more triangles more quickly, which either leads to more demand on network bandwidth, or if the bandwidth is kept constant, leads to fewer objects being displayed.
In the next section, we introduce experiments based on our user study traces that show how the rendering is affected by the presence of bookmarks and how to improve it.
