dash-3d-tmp
/
phd
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Work

This commit is contained in:
Thomas Forgione 2019-10-03 12:03:42 +02:00
parent aeb78691db
commit a71b0be28b
No known key found for this signature in database
GPG Key ID: 203DAEA747F48F41
15 changed files with 485 additions and 520 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/conclusion/future-work.tex
View File

@ -15,7 +15,7 @@ Geometry data is transmitted as OBJ files which is the worst possible format for
Being able to support multi resolution geometry would be even better, and even if performing multi resolution on a large and heterogeneous scene just like ours is difficult, we have no doubt that semantic information can help this task. Being able to support multi resolution geometry would be even better, and even if performing multi resolution on a large and heterogeneous scene just like ours is difficult, we have no doubt that semantic information can help this task.
\subsection{Performance optimization} \subsection{Performance optimization}
Performance has already been discussed in Chapter~\ref{d3i}. Performance has already been discussed in Chapter~\ref{d3}.
However, in this work, we never tried to remove data from the media engine when it is no longer useful. However, in this work, we never tried to remove data from the media engine when it is no longer useful.
This means that on a really large scene, performance is bound to become damaged due to the growing amount of data to render, and the saturation of memory. This means that on a really large scene, performance is bound to become damaged due to the growing amount of data to render, and the saturation of memory.
For our system to be able to support such scenes, it needs to unload data. For our system to be able to support such scenes, it needs to unload data.

72
src/dash-3d-implementation/clients.tex
View File

@ -1,72 +0,0 @@
\fresh{}
\section{Clients}
All DASH clients are built from the same basic bricks, as shown in Figure~\ref{d3i:dash-scheme}:
\begin{itemize}
\item the \emph{access client}, which is the module that deals with making requests and receiving responses;
\item the \emph{segment parsers}, which decodes the data downloaded by the access client, whether it be materials, geometry or textures;
\item the \emph{control engine}, which analyses the bandwidth to dynamically adapt to it;
\item the \emph{media engine}, which renders the multimedia content and the user interface to the screen.
\end{itemize}
\begin{figure}[ht]
\centering
\begin{tikzpicture}
% Server
\draw[rounded corners=5pt,fill=Pink] (-10, 0) rectangle (-3, 7.5);
\node at (-9, 7) {Server};
% Segments
\begin{scope}[shift={(0.5,0.5)}]
\foreach \x in {0,...,3}
{
\draw [fill=Bisque](\x/2-7.5, 1.5-\x/2) rectangle (\x/2-5.5, 6-\x/2);
\node at (\x/2-6.5, 5.5-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
\node at (\x/2-6.5, 4.75-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
\draw [fill=LightBlue] (\x/2-6.5, 3.825-\x/2) circle (2pt) {};
\draw [fill=LightBlue] (\x/2-6.5, 3.325 -\x/2) circle (2pt) {};
\draw [fill=LightBlue] (\x/2-6.5, 2.825 -\x/2) circle (2pt) {};
\node at (\x/2-6.5, 2-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
}
\end{scope}
% MPD
\draw[fill=LightBlue] (-9.5, 6.5) rectangle (-7.5, 0.5);
\node at(-8.5, 3.5) {MPD};
% Client
\draw[rounded corners=5pt, fill=LemonChiffon] (-2, 0) rectangle (3, 7.5);
\node at (-0.5, 7) {DASH client};
% Access client
\draw[fill=PaleGreen] (-1.5, 0.5) rectangle (2.5, 1.5);
\node at (0.5, 1) {Access Client};
% Media engine
\draw[fill=PaleGreen] (-1.5, 5.5) rectangle (2.5, 6.5);
\node at (0.5, 6) {Media Engine};
% Control engine
\draw[fill=PaleGreen] (-1.5, 2) rectangle (0.25, 5);
\node[align=center] at (-0.625, 3.5) {Control \\ Engine};
% Segment parser
\draw[fill=PaleGreen] (0.75, 2) rectangle (2.5, 5);
\node[align=center] at (1.625, 3.5) {Segment \\ Parser};
% Access client to server
\draw[double arrow=5pt colored by RoyalBlue and white] (-3.25, 1.0) -- (-1.0, 1.0);
% Access client to control engine
\draw[double ended double arrow=5pt colored by RoyalBlue and white] (-0.625, 1.25) -- (-0.625, 2.5);
% Acces client to segment parser
\draw[double arrow=5pt colored by RoyalBlue and white] (1.625, 1.25) -- (1.625, 2.5);
% Segment parser to media engine
\draw[double arrow=5pt colored by RoyalBlue and white] (1.625, 4.5) -- (1.625, 5.75);
\end{tikzpicture}
\caption{DASH client-server architecture\label{d3i:dash-scheme}}
\end{figure}

93
src/dash-3d-implementation/introduction.tex
View File

@ -1,93 +0,0 @@
\fresh{}
\section{Introduction}
In the previous chapter, we discussed the theoritical aspects of DASH based 3D streaming\@.
We showed different ways of structuring and downloading content, and we evaluated the parameters.
Implementation was noticeably absent from the previous chapter, and this is why, in this chapter, we detail every aspect of the implementation of the DASH-3D client, from the way segments are built to how they are rendered through how they are downloaded.
All in all, we want three pieces of software.
\begin{itemize}
\item \textbf{We need a preprocessor}, that can take as input our models (we use the Wavefront OBJ format) and that prepares and structures the content and generates an MPD\@. This program is written in Rust.
\item \textbf{We need a client available on the web}, so we can easily have demos and conduct user-studies with real users trying the real interface on desktop or mobile devices (this implementation is detailed in Section~\ref{d3i:js-implementation}). This client is written in JavaScript.
\item \textbf{We need a client that compiles to native code}, so we can easily run simulations with maximum performance to be able to compare different setups or parameters with more precision (this implementation is detailed in Section~\ref{d3i:rust-implementation}). This client is written in Rust.
\end{itemize}
\section{A note about Rust language}
During our work, we implemented the preprocessor as well as a client in Rust.
In this section, we explain the specifities of Rust and why it is a great language for those tasks.
\subsection{Specifity of Rust}
Rust is a system programming language focused on safety.
It is made to be performant (and effectively has performances comparable to C or C++) but with some nice features.
Developpers coming from C++ (as myself) might see it as a language like C++ but that forbids undefined behaviours.\footnote{in Rust, when you need to execute code that might lead to undefined behaviours, you need to put it inside an \texttt{unsafe} block. Many operations will not be available outside an \texttt{unsafe} block (e.g., dereferencing a pointer, or mutating a static variable). The idea is that you can use \texttt{unsafe} blocks when you require it, but you should avoid it as much as possible and when you do it, you must be particularly careful.}
The most powerful concept from Rust is \emph{ownership}.
Basically, every value has a variable that we call its \emph{owner}.
To be able to use a value, you must either be its owner or borrow it.
There are two types of borrow, the immutable borrow and the mutable borrow (people from C++ can see them as having a const reference or a reference to a variable).
The compiler comes with the \emph{borrow checker} which makes sure you only use variables that you are allowed to.
For example, the owner can only use the value if it is not being borrowed, and it is only possible to either borrow mutably a value once, or immutably borrow a value as many times as you want.
At first, the borrow checker seems particularly efficient to detect bugs in concurrent software, but in fact, it is also decisive in non concurrent code.
Consider the piece of C++ code in Listings~\ref{d3i:undefined-behaviour-cpp} and~\ref{d3i:undefined-behaviour-cpp-it}.
\begin{figure}[ht]
\centering
\begin{minipage}[c]{0.85\textwidth}
\lstinputlisting[
language=c++,
label={d3i:undefined-behaviour-cpp},
caption={Undefined behaviour: with for each syntax}
]{assets/dash-3d-implementation/undefined-behaviour.cpp}
\lstinputlisting[
language=c++,
label={d3i:undefined-behaviour-cpp-it},
caption={Undefined beheaviour with iterator syntax}
]{assets/dash-3d-implementation/undefined-behaviour-it.cpp}
\end{minipage}
\end{figure}
Of course, this loop should go undefintely because the vector grows in size as we add elements in the loop.
But the most important thing here is that since we add elements to the vector, it will eventually need to be reallocated, and that reallocation will invalidate the iterator, meaning that the following iterator will provoke an undefined behaviour.
The equivalent code in Rust is in Listings~\ref{d3i:undefined-behaviour-rs} and~\ref{d3i:undefined-behaviour-rs-it}.
\begin{figure}[ht]
\centering
\begin{minipage}[c]{0.45\textwidth}
\lstinputlisting[
language=rust,
caption={Rust version of Listing~\rawref{d3i:undefined-behaviour-cpp}},
label={d3i:undefined-behaviour-rs}
]{assets/dash-3d-implementation/undefined-behaviour.rs}
\end{minipage}
\quad\quad\quad
\begin{minipage}[c]{0.45\textwidth}
\lstinputlisting[
language=rust,
caption={Rust version of Listing~\rawref{d3i:undefined-behaviour-cpp-it}},
label={d3i:undefined-behaviour-rs-it}
]{assets/dash-3d-implementation/undefined-behaviour-it.rs}
\end{minipage}
\end{figure}
What happens is that the iterator needs to borrow of the vector.
Since it is borrowed, it can no longer be borrowed as mutable since mutating it could invalidate the other borrowers.
And effectively, the borrow checker will crash the compiler with the error in Listing~\ref{d3i:undefined-behaviour-rs-error}.
\begin{figure}[ht]
\lstinputlisting[
language=XML,
caption={Error given by the compiler on Listing~\ref{d3i:undefined-behaviour-rs}},
label={d3i:undefined-behaviour-rs-error}
]{assets/dash-3d-implementation/undefined-behaviour-error.txt}
\end{figure}
This example is one of the many examples of how powerful the borrow checker is: in Rust code, there can be no dangling reference, and all the segmentation faults coming from them are detected by the compiler.
The borrow checker may seem like an enemy to newcomers because it often rejects code that seem correct, but once you get used to it, you understand what is the problem with the code and either fix the problem easily, or realise that the whole architecture is wrong and understand why.
\subsection{Tooling}
Even better, Rust comes with great tooling.
\begin{itemize}
\item \href{https://github.com/rust-lang/rust}{\textbf{\texttt{rustc}}} is the Rust compiler. It is very confortable due to the nice error messages it displays.
\item \href{https://github.com/rust-lang/cargo}{\textbf{\texttt{cargo}}} is the official Rust's project and package manager. It manages compilation, dependencies, documentation, tests, etc\ldots
\item \href{https://github.com/racer-rust/racer}{\textbf{\texttt{racer}}}, \href{https://github.com/rust-lang/rls}{\textbf{\texttt{rls}} (Rust Language Server)} and \href{https://github.com/rust-analyzer/rust-analyzer}{\textbf{\texttt{rust-analyzer}}} are software that manage automatic compilation to display errors in code editors as well as providing semantic code completion.
\item \href{https://github.com/rust-lang/rustfmt}{\textbf{\texttt{rustfmt}}} auto formats code.
\item \href{https://github.com/rust-lang/rust-clippy}{\textbf{\texttt{clippy}}} is a linter that detects unidiomatic code and suggests modifications.
\end{itemize}
It is probably for those reasons that Rust is the \emph{most loved programming language} according to the Stack Overflow Developper Survey in~\citeyear{so-survey-2016,so-survey-2017,so-survey-2018} and~\citeyear{so-survey-2019}.

280
src/dash-3d-implementation/js-implementation.tex
View File

@ -1,280 +0,0 @@
\fresh{}
\subsection{JavaScript client\label{d3i:js-implementation}}
\subsubsection{Media engine}
\paragraph{THREE.js.}
On the web browser, the best way to perform 3D rendering is to use WebGL\@.
However, WebGL is very low level and it can be really painful to write code, even to render a simple triangle.
For example, \href{https://www.tutorialspoint.com/webgl/webgl_drawing_a_triangle.htm}{this tutorial}'s code contains 121 lines of javascript, 46 being code (not comments or empty lines) to render a simple, non-textured triangle.
For this reason, it seems unreasonable to build a system like the one we are describing in raw WebGL\@.
There are many libraires that wrap WebGL code and that help people building 3D interfaces, and \href{https://threejs.org}{THREE.js} is probably one of the most popular.
THREE.js acts as a 3D engine built on WebGL\@.
It provides classes to deal with everything we need:
\begin{itemize}
\item the \textbf{Renderer} class contains all the WebGL code needed to render a scene on the web page;
\item the \textbf{Object} class contain all the boilerplate needed to manage the tree structure of the content, it contains a transform and it can have children that are other objects;
\item the \textbf{Scene} class is the root object, it contains all of the objects we want to render and it is passed as argument to the render function;
\item the \textbf{Geometry} and \textbf{BufferGeometry} classes are the classes that hold the vertices buffers, we will discuss that more in Section~\ref{d3i:geometries};
\item the \textbf{Material} class is the class that holds the properties used to render geometry (the most important information being the texture), there are many classes derived from Material, and the developper can choose what material he wants for its objects;
\item the \textbf{Mesh} class is the class that links the geometry and the material, it derives the Object class and can thus be added to a scene and renderer.
\end{itemize}
A snippet of the basic usage of these classes is given in Listing~\ref{d3i:three-hello-world}.
\begin{figure}[th]
\lstinputlisting[%
language=javascript,
caption={A THREE.js \emph{hello world}},
label=d3i:three-hello-world,
]{assets/dash-3d-implementation/base.js}
\end{figure}
\paragraph{Geometries.\label{d3i:geometries}}
Geometries are the classes that hold the vertices, texture coordinates, normals and faces.
There are two most important geometry classes in THREE.js:
\begin{itemize}
\item the \textbf{Geometry} class, which is made to be developper friendly and allows easy editing but can suffer issues of performance;
\item the \textbf{BufferGeometry} class, which is harder to use for a developper, but allows better performance since the developper controls and data is transmitted to the GPU\@.
\end{itemize}
Of course, in this work, we are concerned about performance of our system, and we will not be able to use normal geometries.
However, the way our system works, the way changes happen to the 3D content is always the same: we only add faces and textures to the model.
Therefore, we made a class that derives BufferGeometry, and that makes it more convenient for us.
\begin{itemize}
\item It has a constructor that takes as parameter the number of faces: it allocates all the memory needed for our buffers so we do not have to reallocate it which would be inefficient.
\item It keeps track of the number of faces it is currently holding: it can then avoid rendering faces that have not been filled and knows where to put new faces.
\item It provides a method that adds a face to the geometry.
\item It also keeps track of what part of the buffers has been transmitted to the GPU\@: THREE.js allows us to set the range of the buffer that we want to update and we are able to update only what is necessary.
\end{itemize}
\paragraph{Our 3D model class.\label{d3i:model-class}}
As said in the previous subsections, a geometry and a material a bound together in a mesh.
This means that we are forced to have has many meshes as there are materials in our model.
To make this easy to manage, we made a \textbf{Model} class, that holds everything we need.
We can add vertices, faces, and materials to this model, and it will internally deal with the right geometries, materials and meshes.
In order to avoid having many models that have the same material (which would harm performance since it would increase the number of \texttt{glDrawArray} calls), it automatically merges faces that share the same material in the same buffer geometry, as shown in Figure~\ref{d3i:render-structure}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}
\node[align=center] at(1.5, -1) {DASH-3D\\Structure};
\node at(-1.5, -3) {\texttt{seg0.obj}};
\draw[fill=Pink] (0, -2) rectangle (3, -3);
\node at(1.5, -2.5) {Material 1};
\draw[fill=PaleGreen] (0, -3) rectangle (3, -4);
\node at(1.5, -3.5) {Material 2};
\node at(-1.5, -6) {\texttt{seg1.obj}};
\draw[fill=Pink] (0, -5) rectangle (3, -7);
\node at(1.5, -6) {Material 1};
\node at(-1.5, -9) {\texttt{seg2.obj}};
\draw[fill=PaleGreen] (0, -8) rectangle (3, -9);
\node at(1.5, -8.5) {Material 2};
\draw[fill=LightBlue] (0, -9) rectangle (3, -10);
\node at(1.5, -9.5) {Material 3};
\node[align=center] at (7.5, -1) {Renderer\\Structure};
\node at(10.5, -3.5) {Object 1};
\draw[fill=Pink] (6, -2) rectangle (9, -5);
\node at(7.5, -3.5) {Material 1};
\node at(10.5, -7) {Object 2};
\draw[fill=PaleGreen] (6, -6) rectangle (9, -8);
\node at(7.5, -7) {Material 2};
\node at(10.5, -9.5) {Object 3};
\draw[fill=LightBlue] (6, -9) rectangle (9, -10);
\node at(7.5, -9.5) {Material 3};
\node[minimum width=0.5cm,minimum height=2cm] (O1) at (6.25, -3.5) {};
\draw[-{Latex[length=3mm]}, color=FireBrick] (3, -2.5) -- (O1);
\draw[-{Latex[length=3mm]}, color=FireBrick] (3, -6) -- (O1);
\node[minimum width=0.5cm,minimum height=2cm] (O2) at (6.25, -7) {};
\draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -3.5) -- (O2);
\draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -8.5) -- (O2);
\node[minimum width=3cm,minimum height=2cm] (O3) at (7.5, -9.5) {};
\draw[-{Latex[length=3mm]}, color=RoyalBlue] (3, -9.5) -- (O3);
\node at (1.5, -10.75) {$\vdots$};
\node at (7.5, -10.75) {$\vdots$};
\end{tikzpicture}
\caption{Restructuration of the content on the renderer\label{d3i:render-structure}}
\end{figure}
\subsubsection{Access client}
In order to be able to implement our DASH-3D client, we need to implement the access client, which is responsible for deciding what to download and download it.
To do so, we use the strategy pattern, as shown in Figure~\ref{d3i:dash-loader}.
We have a base class named \texttt{LoadingPolicy} that contain some attributes and functions to keep data about what has been downloaded that a derived class can use to make smart decisions, and exposes a function named \texttt{nextSegment} that takes two arguments:
\begin{itemize}
\item the MPD, so that a strategy can know all the metadata of the segments before making its decision;
\item the camera, because the best segment depends on the position of the camera.
\end{itemize}
The greedy, greedy predictive and proposed policies from the previous chapter are all classes that derive from \texttt{LoadingPolicy}.
Then, the main class responsible for the loading of segments is the \texttt{DashLoader} class.
It uses \texttt{XMLHttpRequest}s, which are the usual way of making HTTP requests in JavaScript, and it calls the corresponding parser on the results of those requests.
The \texttt{DashLoader} class accepts as parameter a function that will be called each time some data has been downloaded and parsed: this data can contain vertices, texture coordinates, normals, materials or textures, and they can all be added to the \texttt{Model} class that we described in Section~\ref{d3i:model-class}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}[scale=0.65]
\draw (0, 0) rectangle (5, -4);
\draw (0, -1) -- (5, -1);
\node at (2.5, -0.5) {DashClient};
\node[right] at (0, -1.5) {\scriptsize loadNextSegment()};
\draw (5, -2) -- (8, -2);
\draw (8, 0) rectangle (14, -4);
\draw (8, -1) -- (14, -1);
\node at (11, -0.5) {LoadingPolicy};
\node[right] at (8, -1.5) {\scriptsize nextSegment(mpd, camera)};
\draw (1, -6) rectangle (7, -10);
\draw (1, -7) -- (7, -7);
\node at (4, -6.5) {Greedy};
\node[right] at (1, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw (8, -6) rectangle (14, -10);
\draw (8, -7) -- (14, -7);
\node at (11, -6.5) {GreedyPredictive};
\node[right] at (8, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw (15, -6) rectangle (21, -10);
\draw (15, -7) -- (21, -7);
\node at (18, -6.5) {Proposed};
\node[right] at (15, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw[-{Triangle[open, length=3mm, width=3mm]}] (4, -6) -- (4, -5) -- (11, -5) -- (11, -4);
\draw (11, -6) -- (11, -5) -- (8, -5);
\draw (18, -6) -- (18, -5) -- (8, -5);
\end{tikzpicture}
\caption{Class diagram of our DASH client\label{d3i:dash-loader}}
\end{figure}
\subsubsection{Performance}
In JavaScript, there is no way of doing parallel computing without using \emph{web workers}.
A web worker is a script in JavaScript that runs in the background, on a separate thread and that can communicate with the main script by sending and receiving messages.
Since our system has many tasks to do, it seems natural to use workers to manage the streaming without impacting the framerate of the renderer.
However, what a worker can do is very limited, since it cannot access the variables of the main script.
Because of this, we are forced to run the renderer on the main script, where it can access the HTML page, and we move all the other tasks to the worker (the access client, the control engine and the segment parsers), and since the main script is the one communicating with the GPU, it will still have to update the model with the parsed content it receives from the worker.
Using the worker does not so much improve the framerate of the system, but it reduces the latency that occurs when receiving a new segment, which can be very frustrating since in a single thread scenario, each time a segment is received, the interface freezes for around half a second.
A sequence diagram of what happens when downloading, parsing and rendering content is shown in Figure~\ref{d3i:sequence}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}
\node at(0, 1) {Main script};
\draw[->, color=LightGray] (0, 0.5) -- (0, -17.5);
\node at(2.5, 1) {Worker};
\draw[->, color=LightGray] (2.5, 0.5) -- (2.5, -17.5);
\node at(10, 1) {Server};
\draw[->, color=LightGray] (10, 0.5) -- (10, -17.5);
% MPD
\draw[color=blue] (0, 0) -- (2.5, -0.1) -- (10, -0.5);
\draw[color=blue, fill=PaleLightBlue] (10, -0.5) -- (10, -1.5) -- (2.5, -2) -- (2.5, -1) -- cycle;
\node[color=blue, rotate=5] at (6.25, -1.25) {Download};
\node[color=blue, above] at(1.25, 0.0) {Ask MPD};
\draw[color=blue, fill=LightBlue] (2.375, -2) rectangle (2.625, -2.9);
\node[color=blue, right=0.2cm] at(2.5, -2.45) {Parse MPD};
\draw[color=blue] (2.5, -2.9) -- (0, -3);
\draw[color=blue, fill=LightBlue] (-0.125, -3.0) rectangle(0.125, -3.5);
\node[color=blue, left=0.2cm] at (0.0, -3.25) {Update model};
% Ask segments
\begin{scope}[shift={(0, -3.5)}]
\draw[color=red] (0, 0) -- (2.5, -0.1);
\node[color=red, above] at(1.25, 0.0) {Ask segment};
\draw[color=red, fill=Pink] (2.375, -0.1) rectangle (2.625, -1);
\node[color=red, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=red] (2.5, -1) -- (10, -1.5);
\draw[color=red, fill=PalePink] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=red, rotate=5] at (6.25, -2.25) {Download};
\draw[color=red, fill=Pink] (2.375, -3) rectangle (2.625, -3.9);
\node[color=red, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=red] (2.5, -3.9) -- (0, -4);
\draw[color=red, fill=Pink] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=red, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
% Ask more segments
\begin{scope}[shift={(0, -8)}]
\draw[color=DarkGreen] (0, 0) -- (2.5, -0.1);
\node[color=DarkGreen, above] at(1.25, 0.0) {Ask segment};
\draw[color=DarkGreen, fill=PaleGreen] (2.375, -0.1) rectangle (2.625, -1);
\node[color=DarkGreen, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=DarkGreen] (2.5, -1) -- (10, -1.5);
\draw[color=DarkGreen, fill=PalePaleGreen] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=DarkGreen, rotate=5] at (6.25, -2.25) {Download};
\draw[color=DarkGreen, fill=PaleGreen] (2.375, -3) rectangle (2.625, -3.9);
\node[color=DarkGreen, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=DarkGreen] (2.5, -3.9) -- (0, -4);
\draw[color=DarkGreen, fill=PaleGreen] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=DarkGreen, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
% Ask even more segments
\begin{scope}[shift={(0, -12.5)}]
\draw[color=purple] (0, 0) -- (2.5, -0.1);
\node[color=purple, above] at(1.25, 0.0) {Ask segment};
\draw[color=purple, fill=Plum] (2.375, -0.1) rectangle (2.625, -1);
\node[color=purple, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=purple] (2.5, -1) -- (10, -1.5);
\draw[color=purple, fill=PalePlum] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=purple, rotate=5] at (6.25, -2.25) {Download};
\draw[color=purple, fill=Plum] (2.375, -3) rectangle (2.625, -3.9);
\node[color=purple, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=purple] (2.5, -3.9) -- (0, -4);
\draw[color=purple, fill=Plum] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=purple, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
\foreach \x in {0,...,5}
{
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2) rectangle (0.125, -\x/2-0.5);
}
\node[color=Goldenrod, left=0.2cm] at (0.0, -1.5) {Render};
\foreach \x in {0,...,7}
{
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-3.5) rectangle (0.125, -\x/2-4);
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-8) rectangle (0.125, -\x/2-8.5);
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-12.5) rectangle (0.125, -\x/2-13);
}
\node[color=Goldenrod, left=0.2cm] at (0.0, -5.5) {Render};
\node[color=Goldenrod, left=0.2cm] at (0.0, -10) {Render};
\node[color=Goldenrod, left=0.2cm] at (0.0, -14.5) {Render};
\end{tikzpicture}
\caption{Repartition of the tasks on the main script and the worker\label{d3i:sequence}}
\end{figure}

20
src/dash-3d-implementation/main.tex
View File

@ -1,20 +0,0 @@
\chapter{DASH-3D implementation\label{d3i}}
\minitoc{}
\newpage
\input{dash-3d-implementation/introduction}
\resetstyle{}
\input{dash-3d-implementation/rust-exporter}
\resetstyle{}
\input{dash-3d-implementation/clients}
\resetstyle{}
\input{dash-3d-implementation/js-implementation}
\resetstyle{}
\input{dash-3d-implementation/rust-implementation}
\resetstyle{}

28
src/dash-3d-implementation/rust-exporter.tex
View File

@ -1,28 +0,0 @@
\fresh{}
\section{Content preparation\label{d3i:rust-exporter}}
The piece of software that does the preprocessing of the model mostly consists in file manipulation and is written is Rust as well.
It successively preprocess the geometry and then the textures.
The MPD is generated by a library named \href{https://github.com/netvl/xml-rs}{xml-rs}, and that works like a stack:
\begin{itemize}
\item a structure is created on the root of the MPD file;
\item the \texttt{start\_element} method creates a new child in the XML file;
\item the \texttt{end\_element} method ends the current child and pops the stack.
\end{itemize}
This structure is passed along with our geometry and texture preprocessors that can add elements to the XML file as they are generating the corresponding files.
The other details of the preprocessing is available in Section~\ref{d3:dash-3d}.
\subsection{Geometry preparation}
The content preparation starts with the geometry.
It structures the faces of the model into a $k$-d tree, and then shrink the bounding boxes to avoid having empty space.
The cells of our $k$-d tree structure are our adaptation sets, so we traverse them and sort the faces by area and make segments, recording all the metadata in the MPD\@.
\subsection{Materials and texture preparation}
Then, we prepare the materials and the textures.
Exporting the materials files is quite straightforward: we simply add an attribute to each material specifying the number of faces painted with this materials in the model so we know the size of the buffers we need to allocate.
We then prepare the textures.
In OpenGL, texture sizes must be power of two.
So the first thing we do is resize the input texture to the next power of two.
We save it, add its metadata to the MPD, divide the size by two and keep doing this process until one of the dimension is less than 32 pixels.

17
src/dash-3d-implementation/rust-implementation.tex
View File

@ -1,17 +0,0 @@
\fresh{}
\subsection{Rust client\label{d3i:rust-implementation}}
In order to run our simulation, we want to have a native implementation of our DASH client that is performant and reliable.
We chose to use Rust to accomplish this.
Our requirements are quite different that the ones we had to deal with in our JavaScript implementation.
In this setup, we want to build a system that is the closest to our theoritical concepts.
Therefore, we do not have a full client in Rust (meaning an application to which you would give the URL to an MPD file and that would allow you to navigate in the scene while it is being downloaded).
In order to be able to run simulations, we develop the bricks of the DASH client separately: the access client and the media engine are totally separated:
\begin{itemize}
\item the \textbf{simulator} takes a user trace as a parameter, it then replays the trace using specific parameters of the access client and outputs a file containing the history of the simulation (what files have been downloaded, and when);
\item the \textbf{renderer} takes the user trace as well as the history generated by the simulator as parameters, and renders images that correspond to what would have been seen.
\end{itemize}
When simulating experiments, we will run the simulator on many traces that we collected during user-studies, and we will then run the renderer program on it to generate images corresponding to the simulation.
We are then able to compute PSNR between those frames and the ground truth frames.
Doing so guarantees us that our simulator is not affected by the performances of our renderer.

339
src/dash-3d/client.tex
View File

@ -6,6 +6,78 @@ In this section, we specify a DASH NVE client that exploits the preparation of t
The generated MPD file describes the content organization so that the client gets all the necessary information to make educated decisions and query the 3D content it needs according to the available resources and current viewpoint. The generated MPD file describes the content organization so that the client gets all the necessary information to make educated decisions and query the 3D content it needs according to the available resources and current viewpoint.
A camera path generated by a particular user is a set of viewpoint $v(t_i)$ indexed by a continuous time interval $t_i \in [t_1,t_{end}]$. A camera path generated by a particular user is a set of viewpoint $v(t_i)$ indexed by a continuous time interval $t_i \in [t_1,t_{end}]$.
\fresh{}
All DASH clients are built from the same basic bricks, as shown in Figure~\ref{d3:dash-scheme}:
\begin{itemize}
\item the \emph{access client}, which is the module that deals with making requests and receiving responses;
\item the \emph{segment parsers}, which decodes the data downloaded by the access client, whether it be materials, geometry or textures;
\item the \emph{control engine}, which analyses the bandwidth to dynamically adapt to it;
\item the \emph{media engine}, which renders the multimedia content and the user interface to the screen.
\end{itemize}
\begin{figure}[ht]
\centering
\begin{tikzpicture}
% Server
\draw[rounded corners=5pt,fill=Pink] (-10, 0) rectangle (-3, 7.5);
\node at (-9, 7) {Server};
% Segments
\begin{scope}[shift={(0.5,0.5)}]
\foreach \x in {0,...,3}
{
\draw [fill=Bisque](\x/2-7.5, 1.5-\x/2) rectangle (\x/2-5.5, 6-\x/2);
\node at (\x/2-6.5, 5.5-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
\node at (\x/2-6.5, 4.75-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
\draw [fill=LightBlue] (\x/2-6.5, 3.825-\x/2) circle (2pt) {};
\draw [fill=LightBlue] (\x/2-6.5, 3.325 -\x/2) circle (2pt) {};
\draw [fill=LightBlue] (\x/2-6.5, 2.825 -\x/2) circle (2pt) {};
\node at (\x/2-6.5, 2-\x/2) {\fcolorbox{black}{LightBlue}{Segment}};
}
\end{scope}
% MPD
\draw[fill=LightBlue] (-9.5, 6.5) rectangle (-7.5, 0.5);
\node at(-8.5, 3.5) {MPD};
% Client
\draw[rounded corners=5pt, fill=LemonChiffon] (-2, 0) rectangle (3, 7.5);
\node at (-0.5, 7) {DASH client};
% Access client
\draw[fill=PaleGreen] (-1.5, 0.5) rectangle (2.5, 1.5);
\node at (0.5, 1) {Access Client};
% Media engine
\draw[fill=PaleGreen] (-1.5, 5.5) rectangle (2.5, 6.5);
\node at (0.5, 6) {Media Engine};
% Control engine
\draw[fill=PaleGreen] (-1.5, 2) rectangle (0.25, 5);
\node[align=center] at (-0.625, 3.5) {Control \\ Engine};
% Segment parser
\draw[fill=PaleGreen] (0.75, 2) rectangle (2.5, 5);
\node[align=center] at (1.625, 3.5) {Segment \\ Parser};
% Access client to server
\draw[double arrow=5pt colored by RoyalBlue and white] (-3.25, 1.0) -- (-1.0, 1.0);
% Access client to control engine
\draw[double ended double arrow=5pt colored by RoyalBlue and white] (-0.625, 1.25) -- (-0.625, 2.5);
% Acces client to segment parser
\draw[double arrow=5pt colored by RoyalBlue and white] (1.625, 1.25) -- (1.625, 2.5);
% Segment parser to media engine
\draw[double arrow=5pt colored by RoyalBlue and white] (1.625, 4.5) -- (1.625, 5.75);
\end{tikzpicture}
\caption{DASH client-server architecture\label{d3:dash-scheme}}
\end{figure}
\copied{}
The DASH client first downloads the MPD file to get the material (.mtl) file containing information about all the geometry and textures available for the entire 3D model. The DASH client first downloads the MPD file to get the material (.mtl) file containing information about all the geometry and textures available for the entire 3D model.
At time instance $t_i$, the DASH client decides to download the appropriate segments containing the geometry and the texture to generate the viewpoint $v(t_{i+1})$ for the time instance $t_{i+1}$. At time instance $t_i$, the DASH client decides to download the appropriate segments containing the geometry and the texture to generate the viewpoint $v(t_{i+1})$ for the time instance $t_{i+1}$.
@ -131,3 +203,270 @@ s^{\texttt{GREEDY}}_i= \argmax{s \in \mathcal{S} \backslash \mathcal{B}_i \cap \
\label{d3:greedy} \label{d3:greedy}
\end{equation} \end{equation}
\fresh{}
\subsection{JavaScript client\label{d3:js-implementation}}
In order to be able to evaluate our system, we need to collect traces and perform analyses on them.
Since our scene is large, and since the system we are describing allows navigating in a streaming scene, we developed a web client that implements our utility metrics and policies.
\subsubsection{Media engine}
Of course, in this work, we are concerned about performance of our system, and we will not be able to use the normal geometries described in Section~\ref{f:geometries}.
However, the way our system works, the way changes happen to the 3D content is always the same: we only add faces and textures to the model.
Therefore, we made a class that derives BufferGeometry, and that makes it more convenient for us.
\begin{itemize}
\item It has a constructor that takes as parameter the number of faces: it allocates all the memory needed for our buffers so we do not have to reallocate it which would be inefficient.
\item It keeps track of the number of faces it is currently holding: it can then avoid rendering faces that have not been filled and knows where to put new faces.
\item It provides a method that adds a face to the geometry.
\item It also keeps track of what part of the buffers has been transmitted to the GPU\@: THREE.js allows us to set the range of the buffer that we want to update and we are able to update only what is necessary.
\end{itemize}
\paragraph{Our 3D model class.\label{d3:model-class}}
As said in the previous subsections, a geometry and a material a bound together in a mesh.
This means that we are forced to have has many meshes as there are materials in our model.
To make this easy to manage, we made a \textbf{Model} class, that holds everything we need.
We can add vertices, faces, and materials to this model, and it will internally deal with the right geometries, materials and meshes.
In order to avoid having many models that have the same material which would harm performance, it automatically merges faces that share the same material in the same buffer geometry, as shown in Figure~\ref{d3:render-structure}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}
\node[align=center] at(1.5, -1) {DASH-3D\\Structure};
\node at(-1.5, -3) {\texttt{seg0.obj}};
\draw[fill=Pink] (0, -2) rectangle (3, -3);
\node at(1.5, -2.5) {Material 1};
\draw[fill=PaleGreen] (0, -3) rectangle (3, -4);
\node at(1.5, -3.5) {Material 2};
\node at(-1.5, -6) {\texttt{seg1.obj}};
\draw[fill=Pink] (0, -5) rectangle (3, -7);
\node at(1.5, -6) {Material 1};
\node at(-1.5, -9) {\texttt{seg2.obj}};
\draw[fill=PaleGreen] (0, -8) rectangle (3, -9);
\node at(1.5, -8.5) {Material 2};
\draw[fill=LightBlue] (0, -9) rectangle (3, -10);
\node at(1.5, -9.5) {Material 3};
\node[align=center] at (7.5, -1) {Renderer\\Structure};
\node at(10.5, -3.5) {Object 1};
\draw[fill=Pink] (6, -2) rectangle (9, -5);
\node at(7.5, -3.5) {Material 1};
\node at(10.5, -7) {Object 2};
\draw[fill=PaleGreen] (6, -6) rectangle (9, -8);
\node at(7.5, -7) {Material 2};
\node at(10.5, -9.5) {Object 3};
\draw[fill=LightBlue] (6, -9) rectangle (9, -10);
\node at(7.5, -9.5) {Material 3};
\node[minimum width=0.5cm,minimum height=2cm] (O1) at (6.25, -3.5) {};
\draw[-{Latex[length=3mm]}, color=FireBrick] (3, -2.5) -- (O1);
\draw[-{Latex[length=3mm]}, color=FireBrick] (3, -6) -- (O1);
\node[minimum width=0.5cm,minimum height=2cm] (O2) at (6.25, -7) {};
\draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -3.5) -- (O2);
\draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -8.5) -- (O2);
\node[minimum width=3cm,minimum height=2cm] (O3) at (7.5, -9.5) {};
\draw[-{Latex[length=3mm]}, color=RoyalBlue] (3, -9.5) -- (O3);
\node at (1.5, -10.75) {$\vdots$};
\node at (7.5, -10.75) {$\vdots$};
\end{tikzpicture}
\caption{Reordering of the content on the renderer\label{d3:render-structure}}
\end{figure}
\subsubsection{Access client}
In order to be able to implement our DASH-3D client, we need to implement the access client, which is responsible for deciding what to download and download it.
To do so, we use the strategy pattern, as shown in Figure~\ref{d3:dash-loader}.
We have a base class named \texttt{LoadingPolicy} that contain some attributes and functions to keep data about what has been downloaded that a derived class can use to make smart decisions, and exposes a function named \texttt{nextSegment} that takes two arguments:
\begin{itemize}
\item the MPD, so that a strategy can know all the metadata of the segments before making its decision;
\item the camera, because the best segment depends on the position of the camera.
\end{itemize}
The greedy, greedy predictive and proposed policies from the previous chapter are all classes that derive from \texttt{LoadingPolicy}.
Then, the main class responsible for the loading of segments is the \texttt{DashLoader} class.
It uses \texttt{XMLHttpRequest}s, which are the usual way of making HTTP requests in JavaScript, and it calls the corresponding parser on the results of those requests.
The \texttt{DashLoader} class accepts as parameter a function that will be called each time some data has been downloaded and parsed: this data can contain vertices, texture coordinates, normals, materials or textures, and they can all be added to the \texttt{Model} class that we described in Section~\ref{d3:model-class}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}[scale=0.65]
\draw (0, 0) rectangle (5, -4);
\draw (0, -1) -- (5, -1);
\node at (2.5, -0.5) {DashClient};
\node[right] at (0, -1.5) {\scriptsize loadNextSegment()};
\draw (5, -2) -- (8, -2);
\draw (8, 0) rectangle (14, -4);
\draw (8, -1) -- (14, -1);
\node at (11, -0.5) {LoadingPolicy};
\node[right] at (8, -1.5) {\scriptsize nextSegment(mpd, camera)};
\draw (1, -6) rectangle (7, -10);
\draw (1, -7) -- (7, -7);
\node at (4, -6.5) {Greedy};
\node[right] at (1, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw (8, -6) rectangle (14, -10);
\draw (8, -7) -- (14, -7);
\node at (11, -6.5) {GreedyPredictive};
\node[right] at (8, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw (15, -6) rectangle (21, -10);
\draw (15, -7) -- (21, -7);
\node at (18, -6.5) {Proposed};
\node[right] at (15, -7.5) {\scriptsize nextSegment(mpd, camera)};
\draw[-{Triangle[open, length=3mm, width=3mm]}] (4, -6) -- (4, -5) -- (11, -5) -- (11, -4);
\draw (11, -6) -- (11, -5) -- (8, -5);
\draw (18, -6) -- (18, -5) -- (8, -5);
\end{tikzpicture}
\caption{Class diagram of our DASH client\label{d3:dash-loader}}
\end{figure}
\subsubsection{Performance}
In JavaScript, there is no way of doing parallel computing without using \emph{web workers}.
A web worker is a script in JavaScript that runs in the background, on a separate thread and that can communicate with the main script by sending and receiving messages.
Since our system has many tasks to do, it seems natural to use workers to manage the streaming without impacting the framerate of the renderer.
However, what a worker can do is very limited, since it cannot access the variables of the main script.
Because of this, we are forced to run the renderer on the main script, where it can access the HTML page, and we move all the other tasks to the worker (the access client, the control engine and the segment parsers), and since the main script is the one communicating with the GPU, it will still have to update the model with the parsed content it receives from the worker.
Using the worker does not so much improve the framerate of the system, but it reduces the latency that occurs when receiving a new segment, which can be very frustrating since in a single thread scenario, each time a segment is received, the interface freezes for around half a second.
A sequence diagram of what happens when downloading, parsing and rendering content is shown in Figure~\ref{d3:sequence}.
\begin{figure}[ht]
\centering
\begin{tikzpicture}
\node at(0, 1) {Main script};
\draw[->, color=LightGray] (0, 0.5) -- (0, -17.5);
\node at(2.5, 1) {Worker};
\draw[->, color=LightGray] (2.5, 0.5) -- (2.5, -17.5);
\node at(10, 1) {Server};
\draw[->, color=LightGray] (10, 0.5) -- (10, -17.5);
% MPD
\draw[color=blue] (0, 0) -- (2.5, -0.1) -- (10, -0.5);
\draw[color=blue, fill=PaleLightBlue] (10, -0.5) -- (10, -1.5) -- (2.5, -2) -- (2.5, -1) -- cycle;
\node[color=blue, rotate=5] at (6.25, -1.25) {Download};
\node[color=blue, above] at(1.25, 0.0) {Ask MPD};
\draw[color=blue, fill=LightBlue] (2.375, -2) rectangle (2.625, -2.9);
\node[color=blue, right=0.2cm] at(2.5, -2.45) {Parse MPD};
\draw[color=blue] (2.5, -2.9) -- (0, -3);
\draw[color=blue, fill=LightBlue] (-0.125, -3.0) rectangle(0.125, -3.5);
\node[color=blue, left=0.2cm] at (0.0, -3.25) {Update model};
% Ask segments
\begin{scope}[shift={(0, -3.5)}]
\draw[color=red] (0, 0) -- (2.5, -0.1);
\node[color=red, above] at(1.25, 0.0) {Ask segment};
\draw[color=red, fill=Pink] (2.375, -0.1) rectangle (2.625, -1);
\node[color=red, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=red] (2.5, -1) -- (10, -1.5);
\draw[color=red, fill=PalePink] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=red, rotate=5] at (6.25, -2.25) {Download};
\draw[color=red, fill=Pink] (2.375, -3) rectangle (2.625, -3.9);
\node[color=red, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=red] (2.5, -3.9) -- (0, -4);
\draw[color=red, fill=Pink] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=red, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
% Ask more segments
\begin{scope}[shift={(0, -8)}]
\draw[color=DarkGreen] (0, 0) -- (2.5, -0.1);
\node[color=DarkGreen, above] at(1.25, 0.0) {Ask segment};
\draw[color=DarkGreen, fill=PaleGreen] (2.375, -0.1) rectangle (2.625, -1);
\node[color=DarkGreen, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=DarkGreen] (2.5, -1) -- (10, -1.5);
\draw[color=DarkGreen, fill=PalePaleGreen] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=DarkGreen, rotate=5] at (6.25, -2.25) {Download};
\draw[color=DarkGreen, fill=PaleGreen] (2.375, -3) rectangle (2.625, -3.9);
\node[color=DarkGreen, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=DarkGreen] (2.5, -3.9) -- (0, -4);
\draw[color=DarkGreen, fill=PaleGreen] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=DarkGreen, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
% Ask even more segments
\begin{scope}[shift={(0, -12.5)}]
\draw[color=purple] (0, 0) -- (2.5, -0.1);
\node[color=purple, above] at(1.25, 0.0) {Ask segment};
\draw[color=purple, fill=Plum] (2.375, -0.1) rectangle (2.625, -1);
\node[color=purple, right=0.2cm] at (2.5, -0.55) {Compute utilities};
\draw[color=purple] (2.5, -1) -- (10, -1.5);
\draw[color=purple, fill=PalePlum] (10, -1.5) -- (10, -2.5) -- (2.5, -3) -- (2.5, -2) -- cycle;
\node[color=purple, rotate=5] at (6.25, -2.25) {Download};
\draw[color=purple, fill=Plum] (2.375, -3) rectangle (2.625, -3.9);
\node[color=purple, right=0.2cm] at(2.5, -3.45) {Parse segment};
\draw[color=purple] (2.5, -3.9) -- (0, -4);
\draw[color=purple, fill=Plum] (-0.125, -4.0) rectangle(0.125, -4.5);
\node[color=purple, left=0.2cm] at (0.0, -4.25) {Update model};
\end{scope}
\foreach \x in {0,...,5}
{
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2) rectangle (0.125, -\x/2-0.5);
}
\node[color=Goldenrod, left=0.2cm] at (0.0, -1.5) {Render};
\foreach \x in {0,...,7}
{
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-3.5) rectangle (0.125, -\x/2-4);
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-8) rectangle (0.125, -\x/2-8.5);
\draw[color=Goldenrod, fill=LemonChiffon] (-0.125, -\x/2-12.5) rectangle (0.125, -\x/2-13);
}
\node[color=Goldenrod, left=0.2cm] at (0.0, -5.5) {Render};
\node[color=Goldenrod, left=0.2cm] at (0.0, -10) {Render};
\node[color=Goldenrod, left=0.2cm] at (0.0, -14.5) {Render};
\end{tikzpicture}
\caption{Repartition of the tasks on the main script and the worker\label{d3:sequence}}
\end{figure}
\subsection{Rust client\label{d3i:rust-implementation}}
However, a web client is not sufficient to analyse our streaming policies: many tasks are performed (such as rendering, and managing the interaction) and all this overhead pollutes the analysis of our policies.
This is why we also implemented a client in Rust, for simulation, so we can gather precise simulated data.
Our requirements are quite different that the ones we had to deal with in our JavaScript implementation.
In this setup, we want to build a system that is the closest to our theoretical concepts.
Therefore, we do not have a full client in Rust (meaning an application to which you would give the URL to an MPD file and that would allow you to navigate in the scene while it is being downloaded).
In order to be able to run simulations, we develop the bricks of the DASH client separately: the access client and the media engine are totally separated:
\begin{itemize}
\item the \textbf{simulator} takes a user trace as a parameter, it then replays the trace using specific parameters of the access client and outputs a file containing the history of the simulation (what files have been downloaded, and when);
\item the \textbf{renderer} takes the user trace as well as the history generated by the simulator as parameters, and renders images that correspond to what would have been seen.
\end{itemize}
When simulating experiments, we will run the simulator on many traces that we collected during user-studies, and we will then run the renderer program on it to generate images corresponding to the simulation.
We are then able to compute PSNR between those frames and the ground truth frames.
Doing so guarantees us that our simulator is not affected by the performances of our renderer.

2
src/dash-3d/conclusion.tex
View File

@ -12,7 +12,7 @@ This way, our system answers, at least partially, all the open problems we menti
\item \textbf{It prepares and structures the content in a way that enables streaming}: all this preparation is precomputed, and all the content is structured, even materials and textures. Furthermore, textures are prepared in a multi-resolution manner, and even though multi-resolution geometry is not discussed here, the difficulty of integrating it in this system seem moderated: we could encode levels of detail in different representations and define a utility metric for each representation and the system should adapt naturally. \item \textbf{It prepares and structures the content in a way that enables streaming}: all this preparation is precomputed, and all the content is structured, even materials and textures. Furthermore, textures are prepared in a multi-resolution manner, and even though multi-resolution geometry is not discussed here, the difficulty of integrating it in this system seem moderated: we could encode levels of detail in different representations and define a utility metric for each representation and the system should adapt naturally.
\item \textbf{We are able to estimate the utility of each segment} by exploiting all the metadata given in the MPD and by analysing the camera parameters of the user. \item \textbf{We are able to estimate the utility of each segment} by exploiting all the metadata given in the MPD and by analysing the camera parameters of the user.
\item \textbf{We proposed a few streaming policies}, from the easiest to implement to the more complex, that are able to exploit the utility metrics we defined in order to guess the best decision. \item \textbf{We proposed a few streaming policies}, from the easiest to implement to the more complex, that are able to exploit the utility metrics we defined in order to guess the best decision.
\item \textbf{The implementation is efficient}: the content preparation allows a client to get all the information it needs from metadata and the server has nothing else to do than to serve files. Special attention has been granted to the client's performance, and explanations are given in Chapter~\ref{d3i}. \item \textbf{The implementation is efficient}: the content preparation allows a client to get all the information it needs from metadata and the server has nothing else to do than to serve files. Special attention has been granted to the client's performance.
\end{itemize} \end{itemize}
However, the work described in this chapter does not take any quality of experience metrics into account. However, the work described in this chapter does not take any quality of experience metrics into account.

15
src/dash-3d/content-preparation.tex
View File

@ -1,7 +1,7 @@
\copied{} \copied{}
\section{Content preparation\label{d3:dash-3d}} \section{Content preparation\label{d3:dash-3d}}
In this section, we describe how we preprocess and store the 3D data of the NVE, consisting of a polygon soup, textures, and material information into a DASH-compliant Media Presentation Description (MPD) file. In this section, we describe how we pre-process and store the 3D data of the NVE, consisting of a polygon soup, textures, and material information into a DASH-compliant Media Presentation Description (MPD) file.
In our work, we use the \texttt{obj} file format for the polygons, \texttt{png} for textures, and \texttt{mtl} format for material information. In our work, we use the \texttt{obj} file format for the polygons, \texttt{png} for textures, and \texttt{mtl} format for material information.
The process, however, applies to other formats as well. The process, however, applies to other formats as well.
@ -14,6 +14,19 @@ This element does not apply to NVE, and we use a single \texttt{period} for the
Each \texttt{period} element contains one or more adaptation sets, which describe the alternate versions, formats, and types of media. Each \texttt{period} element contains one or more adaptation sets, which describe the alternate versions, formats, and types of media.
We utilize adaptation sets to organize a 3D scene's material, geometry, and texture. We utilize adaptation sets to organize a 3D scene's material, geometry, and texture.
\fresh{}
The piece of software that does the preprocessing of the model mostly consists in file manipulation and is written is Rust as well.
It successively preprocess the geometry and then the textures.
The MPD is generated by a library named \href{https://github.com/netvl/xml-rs}{xml-rs}, and that works like a stack:
\begin{itemize}
\item a structure is created on the root of the MPD file;
\item the \texttt{start\_element} method creates a new child in the XML file;
\item the \texttt{end\_element} method ends the current child and pops the stack.
\end{itemize}
This structure is passed along with our geometry and texture preprocessors that can add elements to the XML file as they are generating the corresponding files.
\copied{}
\subsection{Adaptation Sets} \subsection{Adaptation Sets}
When the user navigates freely within an NVE, the frustum at given time almost always contains a limited part of the 3D scene. When the user navigates freely within an NVE, the frustum at given time almost always contains a limited part of the 3D scene.
Similar to how DASH for video streaming partitions a video clip into temporal chunks, we segment the polygons into spatial chunks, such that the DASH client can request only the relevant chunks. Similar to how DASH for video streaming partitions a video clip into temporal chunks, we segment the polygons into spatial chunks, such that the DASH client can request only the relevant chunks.

126
src/foreword/implementation.tex Normal file
View File

@ -0,0 +1,126 @@
\fresh{}
\section{Implementation details}
During this thesis, a lot of software has be written, and for this software to be successful and efficient, we took care of choosing the right languages.
When it comes to 3D streaming systems, there are two kind of software that we need.
\begin{itemize}
\item \textbf{Interactive applications} that can run on as many devices as possible whether it be desktop or mobile in order to try and to conduct user studies. For this context, we chose the \textbf{JavaScript language}, since it can run on many devices and it has great support for WebGL\@.
\item \textbf{Native applications} that can run fast on desktop devices, in order to run simulations and evaluate our ideas. For this context, we chose the \textbf{Rust} language, which is a somewhat recent language that provides both the efficiency of C and C++ and the safety of functional languages.
\end{itemize}
\subsection{JavaScript}
\paragraph{THREE.js.}
On the web browser, the best way to perform 3D rendering is to use WebGL\@.
However, WebGL is very low level and it can be really painful to write code, even to render a simple triangle.
For example, \href{https://www.tutorialspoint.com/webgl/webgl_drawing_a_triangle.htm}{this tutorial}'s code contains 121 lines of javascript, 46 being code (not comments or empty lines) to render a simple, non-textured triangle.
For this reason, it seems unreasonable to build a system like the one we are describing in raw WebGL\@.
There are many libraires that wrap WebGL code and that help people building 3D interfaces, and \href{https://threejs.org}{THREE.js} is probably one of the most popular.
THREE.js acts as a 3D engine built on WebGL\@.
It provides classes to deal with everything we need:
\begin{itemize}
\item the \textbf{Renderer} class contains all the WebGL code needed to render a scene on the web page;
\item the \textbf{Object} class contain all the boilerplate needed to manage the tree structure of the content, it contains a transform and it can have children that are other objects;
\item the \textbf{Scene} class is the root object, it contains all of the objects we want to render and it is passed as argument to the render function;
\item the \textbf{Geometry} and \textbf{BufferGeometry} classes are the classes that hold the vertices buffers, we will discuss that more in Section~\ref{f:geometries};
\item the \textbf{Material} class is the class that holds the properties used to render geometry (the most important information being the texture), there are many classes derived from Material, and the developer can choose what material he wants for its objects;
\item the \textbf{Mesh} class is the class that links the geometry and the material, it derives the Object class and can thus be added to a scene and renderer.
\end{itemize}
A snippet of the basic usage of these classes is given in Listing~\ref{f:three-hello-world}.
\begin{figure}[th]
\lstinputlisting[%
language=javascript,
caption={A THREE.js \emph{hello world}},
label=f:three-hello-world,
]{assets/dash-3d-implementation/base.js}
\end{figure}
\paragraph{Geometries.\label{f:geometries}}
Geometries are the classes that hold the vertices, texture coordinates, normals and faces.
There are two most important geometry classes in THREE.js:
\begin{itemize}
\item the \textbf{Geometry} class, which is made to be developer friendly and allows easy editing but can suffer issues of performance;
\item the \textbf{BufferGeometry} class, which is harder to use for a developer, but allows better performance since the developer controls and data is transmitted to the GPU\@.
\end{itemize}
\subsection{Rust}
In this section, we explain the specificities of Rust and why it is a great language for those tasks.
\subsubsection{Specificity of Rust}
Rust is a system programming language focused on safety.
It is made to be efficient (and effectively has performances comparable to C or C++) but with some nice features.
Developers coming from C++ (as myself) might see it as a language like C++ but that forbids undefined behaviours.\footnote{in Rust, when you need to execute code that might lead to undefined behaviours, you need to put it inside an \texttt{unsafe} block. Many operations will not be available outside an \texttt{unsafe} block (e.g., dereferencing a pointer, or mutating a static variable). The idea is that you can use \texttt{unsafe} blocks when you require it, but you should avoid it as much as possible and when you do it, you must be particularly careful.}
The most powerful concept from Rust is \emph{ownership}.
Basically, every value has a variable that we call its \emph{owner}.
To be able to use a value, you must either be its owner or borrow it.
There are two types of borrow, the immutable borrow and the mutable borrow (people from C++ can see them as having a const reference or a reference to a variable).
The compiler comes with the \emph{borrow checker} which makes sure you only use variables that you are allowed to.
For example, the owner can only use the value if it is not being borrowed, and it is only possible to either borrow mutably a value once, or immutably borrow a value as many times as you want.
At first, the borrow checker seems particularly efficient to detect bugs in concurrent software, but in fact, it is also decisive in non concurrent code.
Consider the piece of C++ code in Listings~\ref{f:undefined-behaviour-cpp} and~\ref{f:undefined-behaviour-cpp-it}.
\begin{figure}[ht]
\centering
\begin{minipage}[c]{0.85\textwidth}
\lstinputlisting[
language=c++,
label={f:undefined-behaviour-cpp},
caption={Undefined behaviour: with for each syntax}
]{assets/dash-3d-implementation/undefined-behaviour.cpp}
\lstinputlisting[
language=c++,
label={f:undefined-behaviour-cpp-it},
caption={Undefined beheaviour with iterator syntax}
]{assets/dash-3d-implementation/undefined-behaviour-it.cpp}
\end{minipage}
\end{figure}
Of course, this loop should go endlessly because the vector grows in size as we add elements in the loop.
But the most important thing here is that since we add elements to the vector, it will eventually need to be reallocated, and that reallocation will invalidate the iterator, meaning that the following iterator will provoke an undefined behaviour.
The equivalent code in Rust is in Listings~\ref{f:undefined-behaviour-rs} and~\ref{f:undefined-behaviour-rs-it}.
\begin{figure}[ht]
\centering
\begin{minipage}[c]{0.45\textwidth}
\lstinputlisting[
language=rust,
caption={Rust version of Listing~\rawref{f:undefined-behaviour-cpp}},
label={f:undefined-behaviour-rs}
]{assets/dash-3d-implementation/undefined-behaviour.rs}
\end{minipage}
\quad\quad\quad
\begin{minipage}[c]{0.45\textwidth}
\lstinputlisting[
language=rust,
caption={Rust version of Listing~\rawref{f:undefined-behaviour-cpp-it}},
label={f:undefined-behaviour-rs-it}
]{assets/dash-3d-implementation/undefined-behaviour-it.rs}
\end{minipage}
\end{figure}
What happens is that the iterator needs to borrow of the vector.
Since it is borrowed, it can no longer be borrowed as mutable since mutating it could invalidate the other borrowers.
And effectively, the borrow checker will crash the compiler with the error in Listing~\ref{f:undefined-behaviour-rs-error}.
\begin{figure}[ht]
\lstinputlisting[
language=XML,
caption={Error given by the compiler on Listing~\ref{f:undefined-behaviour-rs}},
label={f:undefined-behaviour-rs-error}
]{assets/dash-3d-implementation/undefined-behaviour-error.txt}
\end{figure}
This example is one of the many examples of how powerful the borrow checker is: in Rust code, there can be no dangling reference, and all the segmentation faults coming from them are detected by the compiler.
The borrow checker may seem like an enemy to newcomers because it often rejects code that seem correct, but once you get used to it, you understand what is the problem with the code and either fix the problem easily, or realise that the whole architecture is wrong and understand why.
\subsubsection{Tooling}
Even better, Rust comes with great tooling.
\begin{itemize}
\item \href{https://github.com/rust-lang/rust}{\textbf{\texttt{rustc}}} is the Rust compiler. It is very confortable due to the nice error messages it displays.
\item \href{https://github.com/rust-lang/cargo}{\textbf{\texttt{cargo}}} is the official Rust's project and package manager. It manages compilation, dependencies, documentation, tests, etc\ldots
\item \href{https://github.com/racer-rust/racer}{\textbf{\texttt{racer}}}, \href{https://github.com/rust-lang/rls}{\textbf{\texttt{rls}} (Rust Language Server)} and \href{https://github.com/rust-analyzer/rust-analyzer}{\textbf{\texttt{rust-analyzer}}} are software that manage automatic compilation to display errors in code editors as well as providing semantic code completion.
\item \href{https://github.com/rust-lang/rustfmt}{\textbf{\texttt{rustfmt}}} auto formats code.
\item \href{https://github.com/rust-lang/rust-clippy}{\textbf{\texttt{clippy}}} is a linter that detects unidiomatic code and suggests modifications.
\end{itemize}
It is probably for those reasons that Rust is the \emph{most loved programming language} according to the Stack Overflow Developper Survey in~\citeyear{so-survey-2016,so-survey-2017,so-survey-2018} and~\citeyear{so-survey-2019}.

3
src/foreword/main.tex
View File

@ -5,3 +5,6 @@
\input{foreword/video-vs-3d} \input{foreword/video-vs-3d}
\resetstyle{} \resetstyle{}
\input{foreword/implementation}
\resetstyle{}

3
src/introduction/outline.tex
View File

@ -21,9 +21,6 @@ We then define utility metrics that associates score to each chunk depending on
Then, we present a client and various streaming policies based on our utilities that can benefit from the DASH format. Then, we present a client and various streaming policies based on our utilities that can benefit from the DASH format.
We finally evaluate the different parameters of our client. We finally evaluate the different parameters of our client.
In Chapter~\ref{d3i}, we explain the whole implementation of DASH-3D.\todo{going to change}
Implementing DASH-3D required a lot of effort, and since both user studies and simulations are required, we describe the two clients we implemented: one client using web technologies to enable easy user studies and one client that is compiled to native code and that allows us to run efficient simulations and precisely compare the impact of the parameters of our system.
In Chapter~\ref{sb}, we present our last contribution: the integration of the interaction ideas that we developed in Chapter~\ref{bi} into DASH-3D. In Chapter~\ref{sb}, we present our last contribution: the integration of the interaction ideas that we developed in Chapter~\ref{bi} into DASH-3D.
We first develop an interface that allows desktop as well as mobile devices to navigate in a 3D scene being streamed, and that introduces a new style of bookmarks. We first develop an interface that allows desktop as well as mobile devices to navigate in a 3D scene being streamed, and that introduces a new style of bookmarks.
We then explain why simply applying the ideas developed in Chapter~\ref{bi} is not sufficient and we propose more efficient pre-computations that can enhance the streaming. We then explain why simply applying the ideas developed in Chapter~\ref{bi} is not sufficient and we propose more efficient pre-computations that can enhance the streaming.

3
src/plan.tex
View File

@ -17,9 +17,6 @@
\input{dash-3d/main} \input{dash-3d/main}
\resetstyle{} \resetstyle{}
\input{dash-3d-implementation/main}
\resetstyle{}
\input{system-bookmarks/main} \input{system-bookmarks/main}
\resetstyle{} \resetstyle{}

2
src/state-of-the-art/3d-interaction.tex
View File

@ -15,7 +15,7 @@ In Unicam~\cite{two-pointer-input}, the so-called click-to-focus strategy automa
The recent Drag'n Go interaction~\cite{drag-n-go} also hits a destination point while offering control on speed and position along the camera path. The recent Drag'n Go interaction~\cite{drag-n-go} also hits a destination point while offering control on speed and position along the camera path.
This 3D interaction is designed in the screen space (it is typically a mouse-based camera control), where cursor's movements are mapped to camera movements following the same direction as the on-screen optical-flow. This 3D interaction is designed in the screen space (it is typically a mouse-based camera control), where cursor's movements are mapped to camera movements following the same direction as the on-screen optical-flow.
Some 3D browsers provide a viewpoint menu offering a choice of viewpoints~\cite{visual-preception-3d},~\cite{showmotion}. Some 3D browsers provide a viewpoint menu offering a choice of viewpoints~\cite{visual-perception-3d},~\cite{showmotion}.
Authors of 3D scenes can place several viewpoints (typically for each POI) in order to allow easy navigation for users, who can then easily navigate from viewpoint to viewpoint just by selecting a menu item. Authors of 3D scenes can place several viewpoints (typically for each POI) in order to allow easy navigation for users, who can then easily navigate from viewpoint to viewpoint just by selecting a menu item.
Such viewpoints can be either static, or dynamically adapted: the authors from~\cite{dual-mode-ui} report that users clearly prefer navigating in 3D using a menu with animated viewpoints than with static ones. Such viewpoints can be either static, or dynamically adapted: the authors from~\cite{dual-mode-ui} report that users clearly prefer navigating in 3D using a menu with animated viewpoints than with static ones.