In this thesis, we focus on the objective of delivering large, massive 3D scenes over the network.
While 3D streaming is not the most popular research field, there has been a special attention around 3D content compression, in particular progressive compression which can be considered a premise for 3D streaming.
In the next sections, we review the 3D streaming related work, from 3D compression and structuring to 3D interaction.
Since our objective is to stream 3D static scenes, single-rate mesh and mesh sequence compressions are less interesting for us.
This section thus focuses on progressive meshes and random accessible mesh compression.
Progressive meshes were introduced in #cite("progressive-meshes") and allow a progressive transmission of a mesh by sending a low resolution mesh first, called _base mesh_, and then transmitting detail information that a client can use to increase the resolution.
To do so, an algorithm, called _decimation algorithm_, starts from the original full resolution mesh and iteratively removes vertices and faces by merging vertices through the so-called _edge collapse_ operation (Figure X).
// \node at (G2) [circle,fill=red,inner sep=2pt]{};
//
// \draw[-{Latex[length=3mm]}] (Top1) to [out=30, in=150] (Top2);
// \draw[-{Latex[length=3mm]}] (Bottom2) to [out=-150, in=-30] (Bottom1);
//
// \node at (2, 1.75) {Edge collapse};
// \node at (2, -1.75) {Vertex split};
//
//
// \end{tikzpicture}
// \caption{Vertex split and edge collapse\label{sote:progressive-scheme}}
// \end{figure}
Every time two vertices are merged, a vertex and two faces are removed from the original mesh, decreasing the model resolution.
At the end of this content preparation phase, the mesh has been reorganized into a base mesh and a sequence of partially ordered edge split operations.
Thus, a client can start by downloading the base mesh, display it to the user, and keep downloading refinement operations (vertex splits) and display details as time goes by.
This process reduces the time a user has to wait before seeing a downloaded 3D object, thus increases the quality of experience.
#cite("streaming-compressed-webgl") develop a dedicated progressive compression algorithm based on iterative decimation, for efficient decoding, in order to be usable on web clients.
With the same objective, #cite("pop-buffer") proposes pop buffer, a progressive compression method based on quantization that allows efficient decoding.
Following these, many approaches use multi triangulation, which creates mesh fragments at different levels of resolution and encodes the dependencies between fragments in a directed acyclic graph.
In #cite("batched-multi-triangulation"), the authors propose Nexus: a GPU optimized version of multi triangulation that pushes its performances to make real time rendering possible.
It is notably used in 3DHOP (3D Heritage Online Presenter, #cite("3dhop")), a framework to easily build web interfaces to present 3D objects to users in the context of cultural heritage.
Each of these approaches define its own compression and coding for a single mesh.
However, users are often interested in scenes that contain multiple meshes, and the need to structure content emerged.
To answer those issues, the Khronos group proposed a generic format called glTF (GL Transmission Format, #cite("gltf")) to handle all types of 3D content representations: point clouds, meshes, animated models, etc.
glTF is based on a JSON file, which encodes the structure of a scene of 3D objects.
It contains a scene graph with cameras, meshes, buffers, materials, textures and animations.
Although relevant for compression, transmission and in particular streaming, this standard does not yet consider view-dependent streaming, which is required for large scene remote visualization and which we address in our work.
3D streaming means that content is downloaded while the user is interacting with the 3D object.
In terms of quality of experience, it is desirable that the downloaded content falls into the user's field of view.
This means that the progressive compression must encode a spatial information in order to allow the decoder to determine content adapted to its viewpoint.
This is typically called _random accessible mesh compression_.
#cite("maglo2013pomar") is such an example of random accessible progressive mesh compression.
#cite("cheng2008receiver") proposes a receiver driven way of achieving viewpoint dependency with progressive mesh: the client starts by downloading the base mesh, and from then is able to estimate the importance of the different vertex splits, in order to choose which ones to download.
Doing so drastically reduces the server computational load, since it only has to send data, and improves the scalability of this framework.
In the case of streaming a large 3D scene, view-dependent streaming is fundamental: a user will only be seeing one small portion of the scene at each time, and a system that does not adapt its streaming to the user's point of view is bound to induce a low quality of experience.
A simple way to implement viewpoint dependency is to request the content that is spatially close to the user's camera.
This approach, implemented in Second Life and several other NVEs (e.g., #cite("peer-texture-streaming")), only depends on the location of the avatar, not on its viewing direction.
It exploits spatial coherence and works well for any continuous movement of the user, including turning.
Once the set of objects that are likely to be accessed by the user is determined, the next question is in what order should these objects be retrieved.
A simple approach is to retrieve the objects based on distance: the spatial distance from the user's virtual location and rotational distance from the user's view.
More recently, Google integrated Google Earth 3D module into Google Maps (@rw:google-maps).
Users are now able to go to Google Maps, and click the 3D button which shifts the camera from the aerial view.
Even though there are no associated publications to support this assertion, it seems clear that the streaming is view-dependent: low resolution from the center of the point of view gets downloaded first, and higher resolution data gets downloaded for closer objects than for distant ones.
caption: [Screeshot of the 3D interface of Google Maps]
)<rw:google-maps>
Other approaches use level of details.
Level of details have been initially used for efficient 3D rendering #cite("lod").
When the change from one level of detail to another is direct, it can create visual discomfort to the user.
This is called the _popping effect_ and level of details have the advantage of enabling techniques, such as geomorhping #cite("hoppe-lod"), to transition smoothly from one level of detail to another.
Level of details have then been used for 3D streaming.
For example, #cite("streaming-hlod") propose an out-of-core viewer for remote model visualization based by adapting hierarchical level of details #cite("hlod") to the context of 3D streaming.
Level of details can also be used to perform viewpoint dependant streaming, such as #cite("view-dependent-lod").
In order to increase the texture rendering speed, a common technique is the _mipmapping_ technique.
It consists in generating progressively lower resolutions of an initial texture.
Lower resolutions of the textures are used for polygons which are far away from the camera, and higher resolutions for polygons closer to the camera.
Not only this reduces the time needed to render the polygons, but it can also reduce the aliasing effect.
Using these lower resolutions can be especially interesting for streaming.
#cite("mipmap-streaming") proposes the PTM format which encode the mipmap levels of a texture that can be downloaded progressively, so that a lower resolution can be shown to the user while the higher resolutions are being downloaded.
Since 3D data can contain many textures, #cite("simon2019streaming") propose a way to stream a set of textures by encoding them into a video.
Each texture is segmented into tiles of a fixed size.
Those tiles are then ordered to minimize dissimilarities between consecutive tiles, and encoded as a video.
By benefiting from the video compression techniques, the authors are able to reach a better rate-distortion ratio than webp, which is the new standard for texture transmission, and jpeg.
When addressing 3D streaming, one must handle the concurrency between geometry and textures, and the system needs to address this compromise.
Balancing between streaming of geometry and texture data is addressed by #cite("batex3"), #cite("visual-quality-assessment"), and #cite("mesh-texture-multiplexing").
Their approaches combine the distortion caused by having lower resolution meshes and textures into a single view independent metric.
#cite("progressive-compression-textured-meshes") also deals with the geometry / texture compromise.
This work designs a cost driven framework for 3D data compression, both in terms of geometry and textures.
The authors generate an atlas for textures that enables efficient compression and multi-resolution scheme.
All four works considered a single mesh, and have constraints on the types of meshes that they are able to compress.
Since the 3D scenes we are interested in in our work consist in soups of textured polygons, those constraints are not satisfied and we cannot use those techniques.
In traditional video games, including online games, there is no requirement for 3D data streaming.
Video games either come with a physical support (CD, DVD, Blu-Ray) or they require the downloading of the game itself, which includes the 3D data, before letting the user play.
However, transferring data from the disk to the memory is already a form of streaming.
This is why optimized engines for video games use techniques that are reused for streaming such as level of details, to reduce the details of objects far away for the point of view and save the resources to enhance the level of detail of closer objects.
Some other online games, such as #link("https://secondlife.com")[Second Life], rely on user generated data, and thus are forced to send data from users to others.
In such scenarios, 3D streaming is appropriate and this is why the idea of streaming 3D content for video games has been investigated.
For example, #cite("game-on-demand") proposes an online game engine based on geometry streaming, that addresses the challenge of streaming 3D content at the same time as synchronization of the different players.
An example of NVE streaming framework is 3D Tiles #cite("3d-tiles"), which is a specification for visualizing massive 3D geospatial data developed by Cesium and built on top of glTF.
Their main goal is to display 3D objects on top of regular maps, and their visualization consists in a top-down view, whereas we seek to let users freely navigate in our scenes, whether it be flying over the scene or moving along the roads.
In~\citeyear{3d-tiles-10x}, 3D Tiles streaming system was improved by preloading the data at the camera's next position when known in advance (with ideas that are similar to those we discuss and implement in Chapter~\ref{bi}, published in~\citeyear{bookmarks-impact}) and by ordering tile requests depending on the user's position (with ideas that are similar to those we discuss and implement in Chapter~\ref{d3}, published in~\citeyear{dash-3d}).
#cite("zampoglou") is another example of a streaming framework: it is the first paper that proposes to use DASH to stream 3D content.
In their work, the authors describe a system that allows users to access 3D content at multiple resolutions.
They organize the content, following DASH terminology, into periods, adaptation sets, representations and segments.
Their first adaptation set codes the tree structure of the scene graph.
Each further adaptation set contains both geometry and texture information and is available at different resolutions defined in a corresponding representation.
To avoid requests that would take too long and thus introduce latency, the representations are split into segments.
The authors discuss the optimal number of polygons that should be stored in a single segment.
On the one hand, using segments containing very few faces will induce many HTTP requests from the client, and will lead to poor streaming efficiency.
On the other hand, if segments contain too many faces, the time to load the segment is long and the system loses adaptability.
Their approach works well for several objects, but does not handle view-dependent streaming, which is desirable in the use case of large NVEs.
