Work

assets/introduction/cube.obj

@@ -0,0 +1,25 @@
+mtllib cube.mtl
+
+usemtl cubemtl
+
+v -0.5 -0.5 -0.5
+v -0.5 -0.5  0.5
+v -0.5  0.5 -0.5
+v -0.5  0.5  0.5
+v  0.5 -0.5 -0.5
+v  0.5 -0.5  0.5
+v  0.5  0.5 -0.5
+v  0.5  0.5  0.5
+
+vt 0.0 0.0
+vt 0.0 1.0
+vt 1.0 0.0
+vt 1.0 1.0
+
+f 1/1 2/3 4/4 3/2
+f 2/1 6/3 8/4 4/2
+f 6/1 5/3 7/4 8/2
+f 5/1 1/3 3/4 7/2
+f 4/1 8/3 7/4 3/2
+f 2/1 1/3 5/4 6/2
+

assets/introduction/materials.mtl

@@ -0,0 +1,5 @@
+newmtl cubemtl
+Ka 1.0 1.0 1.0
+Kd 1.0 1.0 1.0
+Ks 1.0 1.0 1.0
+map_Kd cube.png

src/dash-3d/client.tex

@@ -131,5 +131,68 @@ s^{\texttt{GREEDY}}_i= \argmax{s \in \mathcal{S} \backslash \mathcal{B}_i \cap \
 \label{d3:greedy}
 \end{equation}
 
+\fresh{}
+\subsection{Performance}
 
+Another important aspect of our client is performance.
+To sum up the bottlenecks of performance described in Section~\ref{i:rendering}, we need to keep the least amount of objects possible, to do the least amount of \texttt{glDrawArray}, but we want to have separated objects so that frustum culling is efficient.
+In our NVE, there are more than a thousand materials and they are reused all over the scene.
+Creating local objects to benefit from frustum culling is not an option in our case, so we merge all faces that share material and draw them in a single \texttt{glDrawArray} call.
 
+In order to do so, the content that was previously structured in a way to optimize streaming is restructured live on the client to optimize rendering, as show in Figure~\ref{d3:render-structure}.
+That way, when geometry is downloaded for a new material, we know how much space we should allocate for the buffers, and when geometry is downloaded for a material already allocated, we just edit the buffer and transmit it to the GPU\@.
+
+\begin{figure}[ht]
+    \centering
+
+    \begin{tikzpicture}
+
+        \node[align=center] at(1.5, -1) {DASH-3D\\Structure};
+
+        \node at(-1.5, -3) {\texttt{seg0.obj}};
+        \draw[fill=LightCoral] (0, -2) rectangle (3, -3);
+        \node at(1.5, -2.5) {Material 1};
+        \draw[fill=LightGreen] (0, -3) rectangle (3, -4);
+        \node at(1.5, -3.5) {Material 2};
+
+        \node at(-1.5, -6) {\texttt{seg1.obj}};
+        \draw[fill=LightCoral] (0, -5) rectangle (3, -7);
+        \node at(1.5, -6) {Material 1};
+
+        \node at(-1.5, -9) {\texttt{seg2.obj}};
+        \draw[fill=LightGreen] (0, -8) rectangle (3, -9);
+        \node at(1.5, -8.5) {Material 2};
+        \draw[fill=Lavender] (0, -9) rectangle (3, -10);
+        \node at(1.5, -9.5) {Material 3};
+
+        \node[align=center] at (7.5, -1) {Renderer\\Structure};
+
+        \node at(10.5, -3.5) {Object 1};
+        \draw[fill=LightCoral] (6, -2) rectangle (9, -5);
+        \node at(7.5, -3.5) {Material 1};
+
+        \node at(10.5, -7) {Object 2};
+        \draw[fill=LightGreen] (6, -6) rectangle (9, -8);
+        \node at(7.5, -7) {Material 2};
+
+        \node at(10.5, -9.5) {Object 3};
+        \draw[fill=Lavender] (6, -9) rectangle (9, -10);
+        \node at(7.5, -9.5) {Material 3};
+
+        \node[minimum width=3cm,minimum height=2cm] (O1) at (7.5, -3.5) {};
+        \draw[-{Latex[length=3mm]}, color=FireBrick] (3, -2.5) -- (O1);
+        \draw[-{Latex[length=3mm]}, color=FireBrick] (3, -6) -- (O1);
+
+        \node[minimum width=3cm,minimum height=2cm] (O2) at (7.5, -7) {};
+        \draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -3.5) -- (O2);
+        \draw[-{Latex[length=3mm]}, color=DarkGreen] (3, -8.5) -- (O2);
+
+        \node[minimum width=3cm,minimum height=2cm] (O3) at (7.5, -9.5) {};
+        \draw[-{Latex[length=3mm]}, color=RoyalBlue] (3, -9.5) -- (O3);
+
+        \node at (1.5, -10.75) {$\vdots$};
+        \node at (7.5, -10.75) {$\vdots$};
+    \end{tikzpicture}
+
+    \caption{Restructuration of the content on the renderer\label{d3:render-structure}}
+\end{figure}

src/dash-3d/conclusion.tex

@@ -12,7 +12,7 @@ Exploiting DASH's concepts to design 3D streaming systems allow us to tackle som
     \item \textbf{It has built-in support for materials and textures}: we use a DASH adaptation set for the materials, and the average color of textures are given in the MPD, meaning that a client is not forced to render everything in white while not having the texture for the materials.
     \item \textbf{It doesn't require any computation on the server side}: the only computation required is preprocessing the model and creating metadata to allow a client make smart decisions, once those precomputations are done, the artifacts can be deployed to a static server like Apache or nginx and all the computation lod is deported to the client, making this solution scalable.
     \item \textbf{It has support for multi-resolution}: in our implementation, we use multi-resolution textures, and even though multi-resolution geometry is not implemented yet, the challenge here lies more on the compression side than on the streaming side. Once a portion of geometry is encoded into different levels of details, we just have to create representations and segments for those levels and define their corresponding utility.
-    \item \todo{we didn't talk about performance, even though we have a few things to say about this in the client and right here}
+    \item \textbf{Performance has been taken into consideration}, and even though the many textures and the heterogeneity of our model prevents us from reaching optimal performances, we still manage to restructure the content on the client side to have a decent framerate.
 \end{itemize}
 
 However, the work described in this chapter does not take any quality of experience aspects into account.

src/introduction/3d-model.tex

@@ -0,0 +1,80 @@
+\section{What is a 3D model?}
+
+Before talking about 3D streaming, we need to define what is a 3D model and how it is rendered.
+
+\subsection{Content of a 3D model}
+
+A 3D model consists in 3D points (that are called \emph{vertices}), texture coordinates, nomals, faces, materials and textures.
+The Wavefront OBJ is probably the best to give an introduction to 3D models since it describes all these elements.
+A 3D model encoded in the OBJ format typically consists in two files: the materials file (\texttt{.mtl}) and the object file (\texttt{.obj}).
+
+\paragraph{}
+The materials file declare all the materials that the object file will reference.
+Each material has a name, ambient, diffuse and specular colors, as well as texture maps.
+A simple material file is visible on Listing~\ref{i:mtl}.
+
+\paragraph{}
+The object file declare the 3D content of the objects.
+It declares vertices, texture coordinates and normals from coordinates (e.g.\ \texttt{v 1.0 2.0 3.0} for a vertex, \texttt{vt 1.0 2.0} for a texture coordinate, \texttt{vn 1.0 2.0 3.0} for a normal).
+These elements are numbered starting from 1.
+Faces are declared by using the indices of these elements. A face is a polygon with any number of vertices and can be declared in multiple manners:
+
+\begin{itemize}
+    \item \texttt{f 1 2 3} defines a triangle face that joins the first, the second and the third vertex declared;
+    \item \texttt{f 1/1 2/3 3/4} defines a triangle similar but with texture coordinates, the first texture coordinate is associated to the first vertex, the third texture coordinate is associated to the second vertex, and the fourth texture coordinate is associated with the third vertex;
+    \item \texttt{f 1//1 2//3 3//4} defines a triangle similar but using normal instead of texture coordinates;
+    \item \texttt{f 1/1/1 2/3/3 3/4/4} defines a triangle with both texture coordinates and normals.
+\end{itemize}
+
+It can include materials from a material file (\texttt{mtllib path.mtl}) and use apply it to faces.
+A material is applied by using the \texttt{usemtl} keyword, followed by the name of the material to use.
+The faces declared after a \texttt{usemtl} are painted using the material in question.
+An example of object file is visible on Listing~\ref{i:obj}.
+
+\begin{figure}[th]
+    \centering
+    \begin{subfigure}[b]{0.4\textwidth}
+        \lstinputlisting[
+            language=XML,
+            caption={An object file describing a cube},
+            label=i:obj,
+        ]{assets/introduction/cube.obj}
+    \end{subfigure}\quad%
+    \begin{subfigure}[b]{0.4\textwidth}
+        \lstinputlisting[
+            language=XML,
+            caption={A material file describing a material},
+            label=i:mtl,
+        ]{assets/introduction/materials.mtl}
+        \vspace{0.2cm}
+        \includegraphics[width=\textwidth]{assets/introduction/cube.png}
+        \caption*{A rendering of the cube}
+    \end{subfigure}
+    \caption{The OBJ representation of a cube and its render\label{i:cube}}
+\end{figure}
+
+\subsection{Rendering a 3D model\label{i:rendering}}
+
+To be able to render a model, it is first required to send the data (vertices, textures coordinates, normals, faces and textures) to the GPU, and then only the rendering can be done.
+Then, to render a 3D model, the objects from the model are traversed, the materials and textures are bound to the target on which the rendering will be done, and then, \texttt{glDrawArray} or \texttt{glDrawElements} function is called.
+To understand how performance is impacted by the structure of the model, we need to realize two things:
+
+\begin{itemize}
+    \item calling many times \texttt{glDrawArray} on small arrays is considerably slower than calling it once on a big array;
+    \item calling \texttt{glDrawArray} on a small array is faster than calling it on a big array.
+\end{itemize}
+
+However, due to the way the materials and textures work, we are forced to call \texttt{glDrawArray} at least as many times as there are materials in the model.
+Minimizing the numbers of materials used in a 3D model is thus critical for rendering performances.
+
+However, frustum culling can be used to improve the performance of rendering.
+Frustum culling is a technique that consists in avoiding drawing objects that are not in the field of view of the user's camera.
+Frustum culling is efficient when they are many objects in a scene since it gives potential for skips.
+
+These two aspects are somehow contradictory, and to have greatest performance for 3D rendering, one must ensure that:
+\begin{itemize}
+    \item the least amount of materials are used, and most objects that share materials are drawn together in a single \texttt{glDrawArray} call;
+    \item objects are not all drawn together and regrouped by location to keep the frustum culling efficient.
+\end{itemize}
+
+

src/introduction/<

@@ -0,0 +1,62 @@
+\section{What is a 3D model?}
+
+Before talking about 3D streaming, we need to define what is a 3D model and how it is rendered.
+
+\subsection{Content of a 3D model}
+
+A 3D model consists in 3D points (that are called \emph{vertices}), texture coordinates, nomals, faces, materials and textures.
+The Wavefront OBJ is probably the best to give an introduction to 3D models since it describes all these elements.
+A 3D model encoded in the OBJ format typically consists in two files: the materials file (\texttt{.mtl}) and the object file (\texttt{.obj}).
+
+\paragraph{}
+The materials file declare all the materials that the object file will reference.
+Each material has a name, ambient, diffuse and specular colors, as well as texture maps.
+A simple material file is visible on Listing~\ref{i:mtl}.
+
+\paragraph{}
+The object file declare the 3D content of the objects.
+It declares vertices, texture coordinates and normals from coordinates (e.g.\ \texttt{v 1.0 2.0 3.0} for a vertex, \texttt{vt 1.0 2.0} for a texture coordinate, \texttt{vn 1.0 2.0 3.0} for a normal).
+These elements are numbered starting from 1.
+Faces are declared by using the indices of these elements. A face is a polygon with any number of vertices and can be declared in multiple manners:
+
+\begin{itemize}
+    \item \texttt{f 1 2 3} defines a triangle face that joins the first, the second and the third vertex declared;
+    \item \texttt{f 1/1 2/3 3/4} defines a triangle similar but with texture coordinates, the first texture coordinate is associated to the first vertex, the third texture coordinate is associated to the second vertex, and the fourth texture coordinate is associated with the third vertex;
+    \item \texttt{f 1//1 2//3 3//4} defines a triangle similar but using normal instead of texture coordinates;
+    \item \texttt{f 1/1/1 2/3/3 3/4/4} defines a triangle with both texture coordinates and normals.
+\end{itemize}
+
+It can include materials from a material file (\texttt{mtllib path.mtl}) and use apply it to faces.
+A material is applied by using the \texttt{usemtl} keyword, followed by the name of the material to use.
+The faces declared after a \texttt{usemtl} are painted using the material in question.
+An example of object file is visible on Listing~\ref{i:obj}.
+
+\begin{figure}[th]
+    \centering
+    \begin{subfigure}[b]{0.4\textwidth}
+        \lstinputlisting[
+            language=XML,
+            caption={An object file describing a cube},
+            label=i:obj,
+        ]{assets/introduction/cube.obj}
+    \end{subfigure}\quad%
+    \begin{subfigure}[b]{0.4\textwidth}
+        \vspace{-0.5cm}
+        \lstinputlisting[
+            language=XML,
+            caption={A material file describing a material},
+            label=i:mtl,
+            emph={%
+                newmtl,
+                Ka,
+                Kd,
+                Ks,
+                map_Ka
+            }
+        ]{assets/introduction/materials.mtl}
+        \vspace{0.2cm}
+        \includegraphics[width=\textwidth]{assets/introduction/cube.png}
+        \caption*{A render of the cube}
+    \end{subfigure}
+    \caption{The OBJ representation of a cube and its render\label{i:cube}}
+\end{figure}

src/introduction/challenges.tex

@@ -0,0 +1,47 @@
+\fresh{}
+
+\section{Challenges}
+
+The objective of our work is to design a system that allows a user to access remote 3D content that guarantees good quality of service and quality of experience.
+A 3D streaming client has lots of tasks to accomplish:
+
+\begin{itemize}
+    \item render a scene;
+    \item find out what to download and download it;
+    \item parse the result of the download;
+    \item add the parsed result to the scene;
+    \item manage the interaction with the user.
+\end{itemize}
+
+This brings multiple issues that we need to take care of.
+
+To add to the challenge, we hope to have a client that would be portable between devices, and that would work easily on both desktop and mobile setups.
+For this reason, we decided to implement a client in JavaScript, that uses THREE.js for rendering (which is based on WebGL) and even with the recent progress of browser based rendering, the fact that the client needs to be written in JavaScript already harms its performance.
+
+\subsection{Streaming aspects}
+
+Of course, such a system can exist only if it does streaming.
+3D models grow in size as the 3D reconstruction tools advance, and even though network systems also advance, it is still unacceptable for a user to have to wait until all the model is downloaded to be able to visualise and interact with it.
+Therefore, we have to face the standard issues that come naturally when talking about streaming, we need to find out what data to download and when to download it. This raises many questions, especially in 3D, since we have to deal with two main types of data: geometry and texture. A model can have a high resolution by having a high resolution geometry, with many small faces, or by having a high resolution texture, mapping a high resolution image on basic geometry. A client has to decide whether to download geometry or texture, and in which proportions.
+
+\paragraph{}
+We have to answer those questions while respecting constraints required for performant software:
+
+\begin{itemize}
+    \item \textbf{our system must keep the computational load on the server low}, since a server must serve many clients, and a solution that requires even low computation on the server will scale difficultly;
+    \item \textbf{our system must keep the computational load on the client low}, since the end user will use his own device, whether it be an old computer or a mobile device, and has many other things to do than simply downloading content.
+\end{itemize}
+
+\subsection{Rendering aspects}
+
+When rendering a 3D scene, the first step is to build vertex buffers and textures that are sent to the GPU\@.
+Communicating between the CPU and the GPU is usually slow, but in most cases, it is not that big of a deal since once the data has been sent to the GPU, the rendering part is fast, and no more communication is needed: in old 3D video games (and even in some recent ones), we had a loading screen while the CPU was transmitting the information to the GPU\@.
+
+In the streaming context however, this is an important concern.
+Each time a portion of the model is downloaded, we need to send it to the GPU while at the same time rendering the scene.
+
+
+\subsection{Interaction aspects}
+
+We also have to provide the user with an interface that allows him to navigate in the scene.
+This brings some extra work the client: it needs to manage input (for a desktop interface, this typically consist in keyboard and mouse inputs, for a mobile device, the touchscreen and the gyroscope), and add elements to the display that need to be updated and renderer.

src/introduction/main.tex

@@ -1,4 +1,10 @@
 \chapter{Introduction\label{i}}
 
+\input{introduction/3d-model}
+\resetstyle{}
+
 \input{introduction/video-vs-3d}
 \resetstyle{}
+
+\input{introduction/challenges}
+\resetstyle{}

src/plan.tex

@@ -1,8 +1,8 @@
+\mainmatter{}
+
 \input{introduction/main}
 \resetstyle{}
 
-\mainmatter{}
-
 \input{state-of-the-art/main}
 \resetstyle{}