dash-3d-tmp
/
phd
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Intro

This commit is contained in:
Thomas Forgione 2019-10-02 14:42:51 +02:00
parent a1676138c9
commit c8ca88e51b
No known key found for this signature in database
GPG Key ID: 203DAEA747F48F41
1 changed files with 23 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
src/foreword/video-vs-3d.tex
View File

@ -3,13 +3,20 @@
\section{Similarities and differences between video and 3D\label{i:video-vs-3d}} \section{Similarities and differences between video and 3D\label{i:video-vs-3d}}
Contrary to what one might think, the video streaming scenario and the 3D streaming one share many similarities: at a higher level of abstraction, they are both systems that allow a user to access remote content without having to wait until everything is loaded. Contrary to what one might think, the video streaming scenario and the 3D streaming one share many similarities: at a higher level of abstraction, they are both systems that allow a user to access remote content without having to wait until everything is loaded.
Analyzing the similarities and the differences between the video and the 3D scenarios as well as having knowledge about video streaming litterature is~\todo{is key or are key?} key to developing an efficient 3D streaming system. Analyzing the similarities and the differences between the video and the 3D scenarios as well as having knowledge about video streaming literature is~\todo{is key or are key?} key to developing an efficient 3D streaming system.
\subsection{Chunks of data}
In order to be able to perform streaming, data needs to be segmented so that a client can request chunks of data and display it to the user while requesting another chunk.
In video streaming, data chunks typically consist in a few seconds of video.
In mesh streaming, some progressive mesh approaches encode a a base mesh that contains low resolution geometry and textures and different chunks that increase the resolution of the base mesh.
Otherwise, a mesh can also be segmented by separating geometry and textures, creating chunks that contain some faces of the model, or some textures.
\subsection{Data persistence} \subsection{Data persistence}
One of the main differences between video and 3D streaming is the persistence of data. One of the main differences between video and 3D streaming is the persistence of data.
In video streaming, only one second of video is required at a time. In video streaming, only one second of video is required at a time.
Of course, most of video streaming services prefetch some future chunks, and keep in cache some previous ones, but a minimal system could work without latency and keep in memory only two chunks: the current one and the next one. Of course, most video streaming services prefetch some future chunks, and keep in cache some previous ones, but a minimal system could work without latency and keep in memory only two chunks: the current one and the next one.
In 3D streaming, each chunk is part of a scene, and already a few problems appear here: In 3D streaming, each chunk is part of a scene, and already a few problems appear here:
\begin{itemize} \begin{itemize}
@ -17,11 +24,11 @@ In 3D streaming, each chunk is part of a scene, and already a few problems appea
\item chunks do not become obsolete the way they do in video, a user navigating in a 3D scene may come back to a same spot after some time, or see the same objects but from elsewhere in the scene. \item chunks do not become obsolete the way they do in video, a user navigating in a 3D scene may come back to a same spot after some time, or see the same objects but from elsewhere in the scene.
\end{itemize} \end{itemize}
\subsection{Multiresolution} \subsection{Multi-resolution}
All the major video streaming platforms support multiresolution streaming. All major video streaming platforms support multi-resolution streaming.
This means that a client can choose the resolution at which it requests the content. This means that a client can choose the resolution at which it requests the content.
It can be chosen directly by the user or automatically determined by analysing the available resources (size of the screen, downoading bandwidth, device performances, etc\ldots) It can be chosen directly by the user or automatically determined by analysing the available resources (size of the screen, downloading bandwidth, device performances, etc\ldots)
\begin{figure}[th] \begin{figure}[th]
\centering \centering
@ -29,7 +36,7 @@ It can be chosen directly by the user or automatically determined by analysing t
\caption{The different resolutions available for a Youtube video} \caption{The different resolutions available for a Youtube video}
\end{figure} \end{figure}
In the same way, the recent work in 3D streaming have proposed many ways to progressively streaming 3D models, allowing the user to have a low resolution without having to wait, and being able to interact with the model while the details are being downloaded. In the same way, recent work in 3D streaming have proposed many ways to progressively streaming 3D models, displaying a low resolution to the user without latency, and supporting interaction with the model while the details are being downloaded.
\subsection{Media types} \subsection{Media types}
@ -42,12 +49,6 @@ Thus, the most important thing a video streaming system should do is optimize th
That's why, on a video on Youtube for example, there may be 6 resolutions for images (144p, 240p, 320p, 480p, 720p and 1080p) but only 2 resolutions for sound. That's why, on a video on Youtube for example, there may be 6 resolutions for images (144p, 240p, 320p, 480p, 720p and 1080p) but only 2 resolutions for sound.
This is one of the main differences between video and 3D streaming: in a 3D scene, geometry and texture sizes are approximately the same, and leveraging between those two types of content is a key problem. This is one of the main differences between video and 3D streaming: in a 3D scene, geometry and texture sizes are approximately the same, and leveraging between those two types of content is a key problem.
\subsection{Chunks of data}
In order to be able to perform streaming, data needs to be segmented so that a client can request chunks of data and display it to the user while requesting another chunk.
In video streaming, data chunks typically consist in a few seconds of video.
In mesh streaming, it can either by segmenting faces in chunks, with a certain number of faces per chunk, or, in the case of progressive meshes, it can be segmented in a chunk containing the base mesh and different chunks encoding the data needed to increase the resolution of the previous level of detail.
\subsection{Interaction} \subsection{Interaction}
The ways of interacting with the content is probably the most important difference between video and 3D. The ways of interacting with the content is probably the most important difference between video and 3D.
@ -71,13 +72,13 @@ Even though these interactions seem easy to handle, giving the best possible exp
\item press the right arrow key to move 5 seconds forwards; \item press the right arrow key to move 5 seconds forwards;
\item press the \texttt{J} key to move 10 seconds backwards; \item press the \texttt{J} key to move 10 seconds backwards;
\item press the \texttt{L} key to move 10 seconds forwards; \item press the \texttt{L} key to move 10 seconds forwards;
\item press one of the number key (on the first row of the keyoard, below the function keys) to move the corresponding decile of the video. \item press one of the number key (on the first row of the keyboard, below the function keys) to move the corresponding decile of the video.
\end{itemize} \end{itemize}
\end{itemize} \end{itemize}
There are even ways of controlling the other options, for example, \texttt{F} puts the player in fullscreen mode, up and down arrows changes the sound volume, \texttt{M} mutes the sound and \texttt{C} activates the subtitles. There are even ways of controlling the other options, for example, \texttt{F} puts the player in fullscreen mode, up and down arrows changes the sound volume, \texttt{M} mutes the sound and \texttt{C} activates the subtitles.
All the interactions are summmed up in Figure~\ref{i:youtube-keyboard}. All the interactions are summed up in Figure~\ref{i:youtube-keyboard}.
\newcommand{\relativeseekcontrol}{LightBlue} \newcommand{\relativeseekcontrol}{LightBlue}
\newcommand{\absoluteseekcontrol}{LemonChiffon} \newcommand{\absoluteseekcontrol}{LemonChiffon}
@ -279,14 +280,18 @@ Some interfaces mimic the video scenario, where the only variable is the time an
These interfaces are not interactive, and can be frustrating to the user who might feel constrained. These interfaces are not interactive, and can be frustrating to the user who might feel constrained.
Some other interfaces add 2 degrees of freedom to the previous one: the user does not control the position of the camera but they can control the angle. This mimics the scenario of the 360 video. Some other interfaces add 2 degrees of freedom to the previous one: the user does not control the position of the camera but they can control the angle. This mimics the scenario of the 360 video.
This is typically the case of the video game \emph{nolimits 2: roller coaster simulator} which works with VR devices (oculus rift, HTC vive, etc\ldots) where the only interaction the user has is turning the head.
Finally, most of the other interfaces give at least 5 degrees of freedom to the user: 3 being the coordinates of the position of the camera, and 2 being the angle (assuming the up vector is unchangeable, some interfaces might allow that giving a sixth degree of freedom). Finally, most of the other interfaces give at least 5 degrees of freedom to the user: 3 being the coordinates of the position of the camera, and 2 being the angle (assuming the up vector is unchangeable, some interfaces might allow that giving a sixth degree of freedom).
The most common controls are the trackball controls where the user rotate the object like a ball \href{https://threejs.org/examples/?q=controls#misc_controls_trackball}{(live example here)} and the orbit controls, which behave like the trackball controls but preserving the up vector \href{https://threejs.org/examples/?q=controls#misc_controls_orbit}{(live example here)}.
Another popular way of controlling a free camera in a virtual environment is the first person controls \href{https://threejs.org/examples/?q=controls#misc_controls_pointerlock}{(live example here)}.
These controls are typically used in shooting video games, the mouse rotates the camera and the keyboard is used to translate it.
\subsection{Relationship between interface, interaction and streaming} \subsection{Relationship between interface, interaction and streaming}
In both video and 3D systems, streaming affects the interaction. In both video and 3D systems, streaming affects the interaction.
For example, in a video streaming scenario, if a user sees that the video is fully loaded, they might start moving around on the timeline, but if they sees that the streaming is just enough to not stall, they might prefer staying peaceful and just watch the video. For example, in a video streaming scenario, if a user sees that the video is fully loaded, they might start moving around on the timeline, but if they sees that the streaming is just enough to not stall, they might prefer staying peaceful and just watch the video.
If the streaming stalls for too long, the user migth seek somewhere else hoping for the video to resume, or get frustrated and leave the video. If the streaming stalls for too long, the user might seek somewhere else hoping for the video to resume, or get frustrated and leave the video.
The same types of behaviour occur in 3D streaming: if a user is somewhere in a scene, and sees more data appearing, they might wait until enough data has arrived, but if they sees nothing happens, they might leave to look for data somewhere else. The same types of behaviour occur in 3D streaming: if a user is somewhere in a scene, and sees more data appearing, they might wait until enough data has arrived, but if they sees nothing happens, they might leave to look for data somewhere else.
Those examples show how streaming can affect the interaction, but the interaction also affects the streaming. Those examples show how streaming can affect the interaction, but the interaction also affects the streaming.
@ -295,9 +300,9 @@ However, if a user starts seeking at a different time of the streaming, the stre
Just like in the video setup, the way a user navigates in a networked virtual environment affects the streaming. Just like in the video setup, the way a user navigates in a networked virtual environment affects the streaming.
Moving slowly allows the system to collect and display data to the user, whereas moving frenetically puts more pressure on the streaming: the data that the system requested may be obsolete when the response arrives. Moving slowly allows the system to collect and display data to the user, whereas moving frenetically puts more pressure on the streaming: the data that the system requested may be obsolete when the response arrives.
Morevoer, the interface and the way elements are displayed to the user also impacts his behaviour. Moreover, the interface and the way elements are displayed to the user also impacts his behaviour.
A streaming system can use this effect to its users benefit by providing feedback on the streaming to the user via the interace. A streaming system can use this effect to its users benefit by providing feedback on the streaming to the user via the interface.
For example, on YouTube, the buffered portion of the video is displayed in light grey on the timeline, whereas the portion that remains to be downloaded is displayed in dark grey. For example, on Youtube, the buffered portion of the video is displayed in light grey on the timeline, whereas the portion that remains to be downloaded is displayed in dark grey.
A user is more likely to click on the light grey part of the timeline that on the dark grey part, preventing the streaming from stalling. A user is more likely to click on the light grey part of the timeline that on the dark grey part, preventing the streaming from stalling.
\begin{figure}[th] \begin{figure}[th]