We now describe our setup and the data we use in our experiments. We present an evaluation of our system and a comparison of the impact of the design choices we introduced in the previous sections.
\subsection{Experimental Setup}
\subsubsection{Model}
We use a city model of the Marina Bay area in Singapore in our experiments.
We partition the geometry into a k-$d$ tree until the leafs have less than 10000 faces, which gives us 64 adaptation sets, plus one containing the large faces.
To evaluate our system, we collected realistic user navigation traces that we can replay in our experiments.
We presented six users with a web interface, on which the model was loaded progressively as the user could interact with it.
The available interactions were inspired by traditional first-person interactions in video games, i.e., W, A, S, and D keys to translate the camera, and mouse to rotate the camera.
We asked users to browse and explore the scene until they felt they had visited all important regions.
We then asked them to produce camera navigation paths that would best present the 3D scene to a user that would discover it.
To record a path, the users first place their camera to their preferred starting point, then click on a button to start recording.
Every 100ms, the position, viewing angle of the camera and look-at point are saved into an array that will then be exported into JSON format.
The recorded camera trace allows us to replay each camera path to perform our simulations and evaluate our system.
We tested our implementation under three network bandwidth of 2.5 Mbps, 5 Mbps, and 10 Mbps with an RTT of 38 ms, following the settings from DASH-IF~\citep{dash-network-profiles}.
In our experiments, we set up a virtual camera that moves along a navigation path, and our access engine downloads segments in real time according to Algorithm~\ref{d3:next-segment}.
We log in a JSON file the time when a segment is requested and when it is received.
By doing so, we avoid wasting time and resources to evaluate our system while downloading segments and store all the information necessary to plot the figures introduced in the subsequent sections.
\subsubsection{Hardware and Software}
The experiments were run on an Acer Aspire V3 with an Intel Core i7 3632QM processor and an NVIDIA GeForce GT 740M graphics card.
The DASH client is written in Rust\footnote{\url{https://www.rust-lang.org/}}, using Glium\footnote{\url{https://github.com/glium/glium}} for rendering, and reqwest\footnote{\url{https://github.com/seanmonstar/reqwest/}} to load the segments.
\subsubsection{Metrics}
To objectively evaluate the quality of the resulting rendering, we use PSNR\@.
The scene as rendered offline using the same camera path with all the geometry and texture data available is used as ground truth.
Note that a pixel error can occur in our case only in two situations: (i) when a face is missing, in which case the color of the background object is shown, and (ii) when a texture is either missing or downsampled.
We do not have pixel error due to compression.
\subsubsection{Experiments}
We present experiments to validate our implementation choices at every step of our system.
We replay the user-generated camera paths with various bandwidth conditions while varying key components of our system.
We also try several utility metrics for geometry segments: an offline one, which assigns to each geometry segment $s^G$ the cumulated 3D area of its belonging faces $\mathcal{A}_{3D}(s^G)$; an online one, which assigns to each geometry segment the inverse of its distance to the camera position; and finally our proposed method, as described in Section~\ref{d3:utility} ($\mathcal{A}_{3D}(s^G)/\mathcal{D}{(v{(t_i)},AS^G)}^2$).
We consider two streaming policies to be applied by the client, proposed in Section~\ref{d3:dash-client}.
The greedy strategy determines, at each decision time, the segment that maximizes its predicted utility at arrival divided by its predicted delivery delay, which corresponds to equation (\ref{d3:greedy}).
The second streaming policy that we run is the one we proposed in equation (\ref{d3:smart}).
Figure~\ref{d3:preparation} shows how the space partition can affect the rendering quality.
We use our proposed utility metrics (see Section~\ref{d3:utility}) and streaming policy from Equation (\ref{d3:smart}), on content divided into adaptation sets obtained either using a $k$-d tree or an Octree and run experiments on all camera paths at 5 Mbps.
The octree partitions content into non-homogeneous adaptation sets; as a result, some adaptation sets may contain smaller segments, which contain both important (large) and non-important polygons. For the $k$-d tree, we create cells containing the same number of faces $N_a$ (here, we take $N_a=10k$).
Figure~\ref{d3:preparation} shows that the system seems to be slightly less efficient with an Octree than with a $k$-d tree based partition, but this result is not significant.
An online-only utility improves the results, as it takes the user viewing frustum into consideration, but still, the proposed utility (in Section~\ref{d3:utility}) performs better.
Clearly, the PSNR significantly improves when the 3D area of faces is considered for creating the segments. Since all segments are of the same size, sorting the faces by area before grouping them into segments leads to a skew distribution of how useful the segments are. This skewness means that the decision that the client makes (to download those with the largest utility first) can make a bigger difference in the quality.
In the first 30 sec, since there are relatively few 3D contents downloaded, making a better decision at what to download matters more: we observe during that time that the proposed method leads to 1 --- 1.9 dB better in quality terms of PSNR compared to Greedy.
Table~\ref{d3:percentages} shows the distribution of texture resolutions that are downloaded by greedy and our Proposed scheme, at different bandwidths.
The table clearly shows a weakness of the greedy policy: as the bandwidth increases, the distribution of downloaded textures resolution stays more or less the same.
In contrast, our proposed streaming policy adapts to an increasing bandwidth by downloading higher resolution textures (13.9\% at 10 Mbps, vs. 0.3\% at 2.5 Mbps).
In fact, an interesting feature of our proposed streaming policy is that it adapts the geometry-texture compromise to the bandwidth. The textures represent 57.3\% of the total amount of downloaded bytes at 2.5 Mbps, and 70.2\% at 10 Mbps.
In other words, our system tends to favor geometry segments when the bandwidth is low, and favor texture segments when the bandwidth increases.
\caption{Percentages of downloaded bytes for textures from each resolution, for the greedy streaming policy (left) and for our proposed scheme (right)\label{d3:percentages}}
