|
|
|
|
@@ -7,8 +7,19 @@ In the next sections, we review the 3D streaming related work, from 3D compressi
|
|
|
|
|
|
|
|
|
|
\subsection{Compression and structuring}
|
|
|
|
|
|
|
|
|
|
The most popular compression model for 3D is progressive meshes: they were introduced in~\citep{progressive-meshes} and allow a progressive transmission of a mesh by sending a low resolution mesh first, called \emph{base mesh}, and then transmitting detail information that a client can use to increase the resolution.
|
|
|
|
|
To do so, an algorithm, called \emph{decimation algorithm}, starts from the original mesh and iteratively removes vertices and faces by merging vertices through the so called \emph{edge collapse} operation (Figure~\ref{sote:progressive-scheme}).
|
|
|
|
|
According to \citep{maglo20153d}, mesh compression can be divided in four categories:
|
|
|
|
|
\begin{itemize}
|
|
|
|
|
\item single-rate mesh compression, seeking to reduce the size of a mesh;
|
|
|
|
|
\item progressive mesh compression, encoding meshes in many levels of resolution that can be downloaded and rendered one after the other;
|
|
|
|
|
\item random accessible mesh compression, where parts of the models can be decoded in any order;
|
|
|
|
|
\item mesh sequence compression, compressing mesh animations.
|
|
|
|
|
\end{itemize}
|
|
|
|
|
|
|
|
|
|
Since our objective is to stream 3D static scenes, single-rate mesh and mesh sequence compressions are less interesting for us.
|
|
|
|
|
This section thus focuses on progressive meshes and random accessible mesh compression.
|
|
|
|
|
|
|
|
|
|
Progressive meshes were introduced in~\citep{progressive-meshes} and allow a progressive transmission of a mesh by sending a low resolution mesh first, called \emph{base mesh}, and then transmitting detail information that a client can use to increase the resolution.
|
|
|
|
|
To do so, an algorithm, called \emph{decimation algorithm}, starts from the original full resolution mesh and iteratively removes vertices and faces by merging vertices through the so called \emph{edge collapse} operation (Figure~\ref{sote:progressive-scheme}).
|
|
|
|
|
|
|
|
|
|
\begin{figure}[ht]
|
|
|
|
|
\centering
|
|
|
|
|
@@ -65,10 +76,10 @@ To do so, an algorithm, called \emph{decimation algorithm}, starts from the orig
|
|
|
|
|
\caption{Vertex split and edge collapse\label{sote:progressive-scheme}}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
Every time two vertices are merged, vertices and faces are removed from the original mesh, and the resolution of the model decreases a little.
|
|
|
|
|
Every time two vertices are merged, a vertex and two faces are removed from the original mesh, decreasing the resolution of the model.
|
|
|
|
|
After content preparation, the mesh consists in a base mesh and a sequence of partially ordered edge split operations.
|
|
|
|
|
Thus, a client can start by downloading the base mesh, display it to the user, and keep downloading and displaying details as time goes by.
|
|
|
|
|
This process reduces the time a user has to wait before seeing something, and increases the quality of experience.
|
|
|
|
|
Thus, a client can start by downloading the base mesh, display it to the user, and keep downloading refinement operations (vertex splits) and display details as time goes by.
|
|
|
|
|
This process reduces the time a user has to wait before seeing something, thus increases the quality of experience.
|
|
|
|
|
|
|
|
|
|
These methods have been vastly researched \citep{bayazit20093,mamou2010shape}, but very few of these methods can handle meshes with attributes, such as texture coordinates.
|
|
|
|
|
|
|
|
|
|
@@ -76,7 +87,7 @@ These methods have been vastly researched \citep{bayazit20093,mamou2010shape}, b
|
|
|
|
|
With the same objective, \citep{pop-buffer} proposes pop buffer, a progressive compression method based on quantization that allows efficient decoding.
|
|
|
|
|
|
|
|
|
|
Following this, many approaches use multi triangulation, which creates mesh fragments at different levels of resolution and encodes the dependencies between fragments in a directed acyclic graph.
|
|
|
|
|
\citep{batched-multi-triangulation} proposes Nexus: a GPU optimized version of multi triangulation that pushes its performances to real time.
|
|
|
|
|
In \citep{batched-multi-triangulation}, the authors propose Nexus: a GPU optimized version of multi triangulation that pushes its performances to make real time rendering possible.
|
|
|
|
|
It is notably used in 3DHOP (3D Heritage Online Presenter, \citep{3dhop}), a framework to easily build web interfaces to present 3D models to users in the context of cultural heritage.
|
|
|
|
|
|
|
|
|
|
Each of these approaches define its own compression and coding for a single mesh.
|
|
|
|
|
@@ -84,12 +95,12 @@ However, users are frequently interested in scenes that contain many meshes, and
|
|
|
|
|
|
|
|
|
|
To answer those issues, the Khronos group proposed a generic format called glTF (GL Transmission Format,~\citep{gltf}) to handle all types of 3D content representations: point clouds, meshes, animated model, etc.\
|
|
|
|
|
glTF is based on a JSON file, which encodes the structure of a scene of 3D objects.
|
|
|
|
|
It contains a scene tree with cameras, meshes, buffers, materials, textures, animations an skinning information.
|
|
|
|
|
Although relevant for compression, transmission and in particular streaming, this standard does not yet consider view-dependent streaming which is required for large scene remote visualisation.
|
|
|
|
|
It contains a scene graph with cameras, meshes, buffers, materials, textures, animations an skinning information.
|
|
|
|
|
Although relevant for compression, transmission and in particular streaming, this standard does not yet consider view-dependent streaming which is required for large scene remote visualisation and that we address in our work.
|
|
|
|
|
|
|
|
|
|
% Zampoglou
|
|
|
|
|
|
|
|
|
|
\citep{zampoglou} are the first to propose to use DASH to stream 3D content.
|
|
|
|
|
\citep{zampoglou} is the first paper that proposes to use DASH to stream 3D content.
|
|
|
|
|
In their work, the authors describe a system that allows users to access 3D content at multiple resolutions.
|
|
|
|
|
They organize the content, following DASH terminology, into periods, adaptation sets, representations.
|
|
|
|
|
Their first adaptation set codes the tree structure of the scene graph.
|
|
|
|
|
@@ -98,7 +109,7 @@ To avoid requests that would take too long and thus introduce latency, the repre
|
|
|
|
|
The authors discuss the optimal number of polygons that should be stored in a single segment.
|
|
|
|
|
On the one hand, using segments containing very few faces will induce many HTTP requests from the client, and will lead to poor streaming efficiency.
|
|
|
|
|
On the other hand, if segments contain too many faces, the time to load the segment will be long and the system loses adaptability.
|
|
|
|
|
This approach works well for several objects, but does not handle view-dependent streaming, which is desirable in the use case of large NVEs\@.
|
|
|
|
|
Their approach works well for several objects, but does not handle view-dependent streaming, which is desirable in the use case of large NVEs\@.
|
|
|
|
|
|
|
|
|
|
\subsection{Viewpoint dependency}
|
|
|
|
|
|
|
|
|
|
@@ -108,7 +119,7 @@ This means that the progressive compression must allow a decoder to choose what
|
|
|
|
|
This is typically called \emph{random accessible mesh compression}.
|
|
|
|
|
\citep{maglo2013pomar} is such an example of random accessible progressive mesh compression.
|
|
|
|
|
|
|
|
|
|
In the case of large scene 3D streaming, viewpoint dependent streaming is a must-have: a user will only be seeing one small portion of the scene at each time, and a system that does not adapt its streaming to the user's point of view is bound to have poor quality of experience.
|
|
|
|
|
In the case of streaming a large 3D scene, viewpoint dependent streaming is a must-have: a user will only be seeing one small portion of the scene at each time, and a system that does not adapt its streaming to the user's point of view is bound to have poor quality of experience.
|
|
|
|
|
|
|
|
|
|
A simple way to implement viewpoint dependency is to access the content near the user's camera.
|
|
|
|
|
This approach, implemented in Second Life and several other NVEs (e.g.,~\citep{peer-texture-streaming}), only depends on the location of the avatar, not on its viewing direction.
|
|
|
|
|
@@ -118,28 +129,28 @@ A simple approach is to retrieve the objects based on distance: the spatial dist
|
|
|
|
|
|
|
|
|
|
More recently, Google integrated Google Earth 3D module into Google Maps.
|
|
|
|
|
Users are now able to go to Google Maps, and click the 3D button which shifts the camera from the top-down view.
|
|
|
|
|
Even though there are no associated publications, it seems that the interface does view dependent streaming: low resolution from the center of the point of view gets downloaded right away, and then, data farther away or higher resolution data gets downloaded.
|
|
|
|
|
|
|
|
|
|
In the same vein, \citep{3d-tiles} developed 3D Tiles, is a specification for visualizing massive 3D geospatial data developed by Cesium and built on top of glTF\@.
|
|
|
|
|
Even though there are no associated publications, it seems that the interface does view dependent streaming: low resolution from the center of the point of view gets downloaded right away, and then, data farther away or higher resolution data gets downloaded since it appears in a second time.
|
|
|
|
|
The choice of the nearby can be based based on an a priori, discretized, partitioned version of the environment; for example, \citep{3d-tiles} developed 3D Tiles, is a specification for visualizing massive 3D geospatial data developed by Cesium and built on top of glTF\@.
|
|
|
|
|
Their main goal is to display 3D objects on top of regular maps.
|
|
|
|
|
|
|
|
|
|
\subsection{Geometry and textures}
|
|
|
|
|
|
|
|
|
|
As discussed in Chapter~\ref{f:3d}, meshes consists in two main types of data: geometry and textures.
|
|
|
|
|
When addressing 3D streaming, one must find a compromise between geometry and textures, and a system needs to solve this compromise.
|
|
|
|
|
|
|
|
|
|
Balancing between streaming of geometry and texture data are considered by~\citep{batex3},~\citep{visual-quality-assessment}, and~\citep{mesh-texture-multiplexing}.
|
|
|
|
|
All three work considered a single, manifold textured mesh model with progressive meshes.
|
|
|
|
|
Their approach is to combine the distortion caused by having lower resolution meshes and textures into a single view independent metric.
|
|
|
|
|
As discussed in Chapter~\ref{f:3d}, most 3D scenes consists in two main types of data: geometry and textures.
|
|
|
|
|
When addressing 3D streaming, one must handle the concurrency between geometry and textures, and a system needs to solve this compromise.
|
|
|
|
|
|
|
|
|
|
Balancing between streaming of geometry and texture data is addressed by~\citep{batex3},~\citep{visual-quality-assessment}, and~\citep{mesh-texture-multiplexing}.
|
|
|
|
|
Their approaches combine the distortion caused by having lower resolution meshes and textures into a single view independent metric.
|
|
|
|
|
\citep{progressive-compression-textured-meshes} also deals with the geometry / texture compromise.
|
|
|
|
|
This work designs a cost driven framework for 3D data compression, both in terms of geometry and textures.
|
|
|
|
|
This framework generates an atlas for textures that enables efficient compression and multiresolution scheme.
|
|
|
|
|
All four works considered a single, manifold textured mesh model with progressive meshes, and is not applicable in our work since we deal with large scenes with autointersections and T vertices.
|
|
|
|
|
|
|
|
|
|
\citep{simon2019streaming} propose a way to stream a set of textures by encoding the textures into a video.
|
|
|
|
|
Regarding texture streaming, \citep{simon2019streaming} propose a way to stream a set of textures by encoding the textures into a video.
|
|
|
|
|
Each texture is segmented into tiles of a fixed size.
|
|
|
|
|
Those tiles are then ordered to minimise dissimilarities between consecutive tiles, and encoded as a video.
|
|
|
|
|
By benefiting from the video compression techniques, they are able to reach a better rate-distortion ratio than webp, which is the new standard for texture transmission, and jpeg.
|
|
|
|
|
However, the geometry / texture compromise is not the point of that paper.
|
|
|
|
|
|
|
|
|
|
% \copied{}
|
|
|
|
|
% \subsection{Prefetching in NVE}
|
|
|
|
|
% The general prefetching problem can be described as follows: what are the data most likely to be accessed by the user in the near future, and in what order do we download the data?
|
|
|
|
|
|