Multi-Resolution 3D Rendering for High-Performance Web AR: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Camila Xu and Version 1 by Argyro Maria Boutsi.

In the context of web augmented reality (AR), 3D rendering that maintains visual quality and frame rate requirements remains a challenge. The lack of a dedicated and efficient 3D format often results in the degraded visual quality of the original data and compromises the user experience. 

  • augmented reality
  • multi-resolution
  • AR.js
  • 3D visualization
  • web

1. Introduction

Web AR advances along with high-end mobile computing and alters how experiences are interpreted. Common digital overlays vary from text and animations to 3D graphics and low-poly models. However, there is a demand for more intuitive, realistic and efficient types of media in order to increase user engagement and retention. Such examples are the detailed and high-resolution 3D models that are derived from laser scanning, photogrammetry or fast surface modeling. Their geometry is represented as a mesh of triangles, quadrilaterals or other polygons that includes the shape, size and position of the object’s vertices, edges and faces. These mesh data often exhibit spatial redundancies, such as repeated vertex positions or similar attribute values, non-manifold geometry, outliers and surface discontinuities. Besides the inner structure, the accompanied texture maps, material properties, lighting information, animation data, metadata, etc., add to their overall complexity. The process of streaming these assets on the web is subject to a set of physical and logistical constraints: storage, bandwidth, latency, memory and computational power [1]. WebGL graphics API and rendering engines typically expect well-formed meshes with consistent topology. Since there is a lack of 3D web standards, a combination of proprietary formats, third-party libraries and custom workflows tailored to the requirements of the current project is usually adopted. Besides hardware acceleration, common rendering practices include adaptive streaming, progressive loading and optimized compression.
Meanwhile, AR is a motion-tracking issue that requires stable poses and accurate localization. The recovery of 3D poses from a frame sequence captured by a live camera involves many steps of intense processing. Feature detection and matching, feature tracking, robust estimation techniques such as RANSAC and relocalization algorithms can place pressure on the computing capabilities of mobile devices. Besides a high CPU load, the real-time 3D rendering proves to be a resource-intensive, long process, even for 3D models larger than 4 or 5 MB [2]. The absence of a lightweight 3D format or a standardized method on a graphics pipeline can result in a user experience degradation and performance limitations. This may manifest as reduced frame rates, increased latency or the inefficient use of device resources, ultimately impacting the responsiveness and smoothness of AR interactions. Moreover, the meshes used as AR overlays may undergo modifications or simplifications as a trade-off for quick loading times and low bandwidth, resulting in visual degradation, reduced detail or lower fidelity compared to the original ones. Therefore, the AR overlays may not fully capture the intricate details and realism of the original 3D content, affecting the overall experience.

2. Optimized 3D Rendering

Numerous studies have addressed the optimized web 3D visualization of large and complex geometry. In this section, only view-dependent approaches, able to adapt to the concurrent operation with the already arduous task of AR, and the tile-based rendering of mobile devices are presented and discussed. For instance, progressive mesh streaming prevents geometric attributes but is computationally intensive since every part of the model is rendered if certain criteria or other techniques are not set [9][3]. Multi-triangulation is constrained by the high CPU load required to traverse the direct acyclic graphs (DAGs) and by the fact that is not possible to fully exploit the GPU processing power [10][4].

2.1. Compression

Compression significantly reduces the file size of 3D models, making them more efficient to store, transfer and render in real-time applications. General-purpose compression libraries, such as Zstd [11][5] or Oodle [12][6], target real-time visualization cases where a fast decompression speed is a crucial factor. However, they are not designed to exploit redundancies in vertex and index data and they may result in low compression ratios. Instead of basic mesh decimation and simplification algorithms, which try to reduce the vertex and face count with minimal shape changes [13][7], dedicated arbitrary geometry algorithms utilize one or a combination of the following techniques: quantization, delta encoding, vertex cache optimization or octree-based encoding. The Draco algorithm primarily adopts quantization and delta encoding to reduce the precision of vertex attributes and encode the differences between the consecutive ones. While it is highly effective at compressing complex meshes with high vertex and triangle counts, it does not prioritize the UV texture mapping quality. Additional compression techniques specific to textures may need to be considered alongside Draco for optimal results, such as Khronos TeXture (KTX2) [14][8]. Other algorithms convert the connectivity, geometry and vertex attributes into a stream of symbols and bits to reduce the data volume and allow for progressive transmission during decompression [15,16,17][9][10][11]. The decompression time on the client side is of greater importance than the compression rate. In the case of single-rate compression, WebGL-Loader of Google provides a better decompression ratio than the Draco system but they both do not support progressive encoding/decoding [18][12]. The Corto library compresses large point clouds and meshes with per vertex attributes, aiming to support streaming and adaptive rendering [3][13]. Regarding decoding, edges are connected in such a way that allows for the efficient traversal and fast reconstruction of the mesh topology, especially when viewing distance or angle-based criteria are employed.

2.2. View-Dependent Techniques

A model compression technology can only shorten the download time; the rendering operations on mobile devices still consume a large amount of CPU, memory and battery resources. A natural approach to reducing the model data is a simplification method that represents the models with selective levels of detail depending on their properties or user-defined criteria [19][14]. Level-of-detail (LOD) techniques can be broadly categorized into discrete and continuous LOD based on the transition between levels, as well as into hierarchical, screen space and temporal LOD if the respective optimization factor is scene hierarchy, screen visibility or temporal coherence. View-dependent LOD considers the viewer’s position and orientation to dynamically select the appropriate LODs, optimizing the rendering for the areas visible to the viewer. Since the display of an overlay is dynamically adjusted based on the viewer’s perspective and the direction of the camera, view-dependent LOD could be a valuable feature for AR. Unlike the view-dependent approaches, the meshoptimizer library reorders triangles to minimize the overdraw from all directions and not only from the point of view [7][15]. It primarily focuses on mesh optimization, such as vertex cache optimization, index buffer optimization and vertex quantization, rather than a specific loading strategy. These processes help to decrease memory consumption and enhance the rendering performance by minimizing the time required for index fetching. However, quantizing vertex attributes may cause a minimal loss of visual quality. A hybrid approach that includes the aforementioned techniques should be engaged. Thus, encoding, quantization and compression come as a natural extension to the family of multi-resolution algorithms. Multi-resolution structures for the web have been well studied since 2000 and typically involve representing a 3D model using a hierarchy of levels of detail with varying degrees of geometric complexity [20][16]. Any type of multi-resolution representation can be naturally extended to a view-dependent streaming geometry due to its intrinsic characteristics [21][17]. For example, in multi-resolution strip masking, tiles can be rendered at various resolutions, according to their global and local properties, using a set of coarse to fine masks, which are precomputed triangle strip indices [22][18]. While it effectively offloads a significant portion of the CPU load to the GPU, it is less geometrically optimal than local LOD algorithms. On the other hand, several algorithms have been introduced for cluster-based partition driven by several criteria, such as partition, hierarchy, kernel, quantum theory, etc. [23,24][19][20]. Gobbetti and Marton transformed point clouds into two-manifold meshes and encoded them into compact multi-resolution structures consisting of variable-resolution quad patches [25][21]. The patches store both geometry and texture information in a lightweight, tightly packed texture atlas. These 3D structures have the potential to be employed as AR overlays since the GPU-accelerated adaptive tessellation algorithm of the method minimizes the CPU overhead. On the other hand, the Nexus algorithm adopts a patch-based data structure, from which view-dependent conforming mesh representations can be efficiently extracted by combining precomputed patches [3][13]. Since each patch is itself a mesh composed of a few thousand triangles, the multi-resolution extraction cost is amortized over many graphics primitives, and CPU/GPU communication can be optimized to fully exploit the complex memory hierarchy of modern graphics platforms. It defines a publishing format (NXS) for multi-resolution meshes that stores the sequence of the increasingly coarse partitions. More recent multi-resolution schemes elaborate their parallel compression algorithm to the GPU for hardware acceleration [26,27][22][23]. They achieve significant performance improvements in cases of multi-core computing, currently excluding consumer mobile devices. A hardware tessellation-based patch rendering with continuous LOD is a promising approach for AR, offering a stable frame rate for high-fidelity outputs with optimum memory and computational usage, but it currently addresses triangular models [28][24].

3. Web AR Approaches for 3D Overlays

In general, the synergy of open-source JavaScript AR software such as AR.js with the Three.js library has a variety of applications in interior design [29][25], cultural heritage [30][26], synthetic visualizations [31][27] and education [32,33][28][29]. Nitika et al. developed a WebRTC marker-based application with AR.js and Three.js to analyze web AR performance, examine potential rendering optimizations and proposing a cloud-based solution [34][30]. Rendering can be offloaded to cloud servers, minimizing the overhead, but data transfer suffers from E2E latency and the risk of security and privacy threats. MEC-oriented web AR solutions compensate for this latency, as well as the weak computational efficiency of browsers, through the operation of a client–server model typically located close to the user’s device. However, both technologies are prone to single-point failures and network loads while huge or multiple chunks of 3D data are transferred. From a rendering performance standpoint, there is not a lightweight 3D format file suited to overlays, while the emergence of dedicated mobile web AR browsers [35,36][31][32] has not resolved the compatibility issues that arise from the lack of a standard for web 3D objects. The Universal Scene Description (USDZ) schema, developed by Apple and Pixar, is a compact format that incorporates AR functionalities into 3D content [37][33]. Apart from the typical storage of geometry, materials, textures, animations, lighting and camera properties, it defines the state, the tracking mode and the location data of the AR experience. Being primarily designed for the iOS operating system, it is tightly integrated with Apple’s ARKit and has limited authoring tools. This conflict between having a uniform cross-platform SDK or API and leveraging the unique characteristics of each platform is still an open problem [38][34]. glTF and its binary type glb is a royalty-free specification of the Khronos Group for the efficient loading of 3D scenes on the web [39][35]. A remaining issue is that the various glTF renderers do not always appear to produce visually identical results [40][36]. Apart from the format, large-sized and detailed source data of arbitrary geometry require pre-processing or further optimization steps at runtime by the rendering engine or the framework. gltfpack of the meshoptimizer suite [41][37] applies a set of optimization steps, including encoding, reordering, quantization and memory management, to the main GPU procedures (vertex processing, rasterization and fragment shading), while the Gltf pipeline by CesiumJS [42][38] supports quantization and Draco compression. Only a few approaches can be considered complete regarding remote 3D rendering during an AR session. A dynamic adaptive streaming technique represents AR assets at different LODs and only the LODs of high fidelity are selected by a heuristic to be visualized each time. It manages to decrease the start-up latency by up to 90% with respect to a download-and-play baseline, as well as the amount of data needed to deliver the AR experience by up to 79%, without sacrificing the visual quality [43][39]. Another work dedicated to the rendering phase of augmentation, developed for Mobile AR (MAR), exploits a meta-file during the progressive HTTP streaming to determine the next requested chunk of data and enhance the visual quality of the view-dependent display [44][40]. An HTTP adaptive AR streaming system that aims to provide high-quality AR streaming services with low latency over time-varying wireless networks has recently been introduced [45][41]. The system utilizes progressive mesh techniques coupled with a metafile structure to render, display and schedule the streaming of chunks of data according to the wireless network load. Unlike the view-dependent rendering of Nexus.js, it adapts to different network conditions, and it takes into account the displayed scale values of the AR overlays on the screen, ensuring that the requested chunks of content are optimized to enhance human visual perception.

References

  1. Ponchio, F. Streaming 3D Content. In Proceedings of the 2nd Workshop on Flexible Resource and Application Management on the Edge, Minneapolis, MN, USA, 27 June–1 July 2022; pp. 1–2.
  2. Qiao, X.; Ren, P.; Dustdar, S.; Liu, L.; Ma, H.; Chen, J. Web AR: A Promising Future for Mobile Augmented Reality—State of the Art. Challenges. and Insights. Proc. IEEE 2019, 107, 651–666.
  3. Evans, A.; Agenjo, J.; Blat, J. A Pipeline for the Creation of Progressively Rendered Web 3D Scenes. Multimed. Tools Appl. 2018, 77, 20355–20383.
  4. Ponchio, F.; Hormann, K. Interactive Rendering of Dynamic Geometry. IEEE Trans. Vis. Comput. Graph. 2008, 14, 914–925.
  5. Zstandard. Available online: https://facebook.github.io/zstd/ (accessed on 23 June 2023).
  6. Oodle Compression. Available online: http://www.radgametools.com/oodle.htm (accessed on 23 June 2023).
  7. Xian, Y.; Fan, Y.; Huang, Y.; Wang, G.; Tu, C.; Meng, X.; Peng, J. Mesh Simplification With Appearance-Driven Optimizations. IEEE Access 2020, 8, 165769–165778.
  8. KTX File Format Specification. Available online: https://registry.khronos.org/KTX/specs/2.0/ktxspec.v2.html (accessed on 23 June 2023).
  9. Maglo, A.; Lavoué, G. Florent Dupont. and Céline Hudelot. 3D Mesh Compression: Survey. Comparisons. and Emerging Trends. ACM Comput. Surv. 2015, 7, 1–41.
  10. Yang, B.; Jiang, Z.; Shangguan, J.; Li, F.W.B.; Song, C.; Guo, Y.; Xu, M. Compressed Dynamic Mesh Sequence for Progressive Streaming. Comput. Anim. Virtual Worlds 2019, 30, e1847.
  11. Portaneri, C.; Alliez, P.; Hemmer, M.; Birklein, L.; Schoemer, E. Cost-Driven Framework for Progressive Compression of Textured Meshes. In Proceedings of the 10th ACM Multimedia Systems Conference, Amherst, MA, USA, 18–21 June 2019; pp. 175–188.
  12. Introducing Draco: Compression for 3D Graphics. Available online: https://opensource.googleblog.com/2017/01/introducing-Draco-compression-for-3d.html (accessed on 23 June 2023).
  13. Ponchio, F.; Dellepiane, M. Fast Decompression for Web-Based View-Dependent 3D Rendering. In Proceedings of the 20th International Conference on 3D Web Technology—Web3D ’15, Heraklion, Greece, 18–21 June 2015; pp. 199–207.
  14. Meng, F.; Zha, H. Streaming Transmission of Point-Sampled Geometry Based on View-Dependent Level-of-Detail. In Proceedings of the Fourth International Conference on 3-D Digital Imaging and Modeling, 3DIM 2003, Banff, AB, Canada, 6–10 October 2003; IEEE: Banff, AB, Canada, 2003; pp. 466–473.
  15. Meshoptimizer. Available online: https://github.com/zeux/meshoptimizer/tree/master (accessed on 23 June 2023).
  16. Rusinkiewicz, S.; Levoy, M. QSplat: A Multi-resolution Point Rendering System for Large Meshes. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques—SIGGRAPH ’00, New Orleans, LA, USA, 23–28 July 2000; pp. 343–352.
  17. Kim, J.; Lee, S.; Kobbelt, L. View-Dependent Streaming of Progressive Meshes. In Proceedings of the Shape Modeling Applications, Washington, DC, USA, 7–9 June 2004; pp. 209–220.
  18. Pouderoux, J.; Marvie, J.-E. Adaptive Streaming and Rendering of Large Terrains Using Strip Masks. In Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, Dunedin, New Zealand, 29 November 2005; pp. 299–306.
  19. Xu, D.; Tian, Y. A Comprehensive Survey of Clustering Algorithms. Ann. Data Sci. 2015, 2, 165–193.
  20. Henriques, R.; Madeira, S.C. Triclustering Algorithms for Three-Dimensional Data Analysis: A Comprehensive Survey. ACM Comput. Surv. 2018, 51, 1–43.
  21. Gobbetti, E.; Marton, F. Adaptive quad patches: An adaptive regular structure for web distribution and adaptive rendering of 3D models. In Proceedings of the 17th International Conference on 3D Web Technology, Los Angeles, CA, USA, 4–5 August 2012; pp. 9–16.
  22. Dworak, D.; Pietruszka, M. Three-Dimensional Multi-Resolution and Hardware Accelerated Compression for the Efficient Interactive Web Applications. In Multimedia and Network Information Systems. Advances in Intelligent Systems and Computing; Choroś, K., Kopel, M., Kukla, E., Siemiński, A., Eds.; Springer: Berlin/Heidelberg, Germany, 2019; Volume 833.
  23. Herrero-Pérez, D.; Castejón, P.J.M. Multi-GPU Acceleration of Large-Scale Density-Based Topology Optimization. Adv. Eng. Softw. 2021, 103006, 157–158.
  24. Masood, Z.; Jiangbin, Z.; Ahmad, I.; Irfan, M.; Ahmad, N. A Novel Method for Adaptive Terrain Rendering Using Memory-Efficient Tessellation Codes for Virtual Globes. J. King Saud Univ.—Comput. Inf. Sci. 2022, 34910, 9393–9408.
  25. Nagashree, R.R.; Rakshitha, M.; Bharadwaj, R.; Sultana, S.; Nadig, N.M. Markerless Augmented Reality Application for Interior Designing. In Proceedings of the 2022 Second International Conference on Advanced Technologies in Intelligent Control, Environment, Computing & Communication Engineering (ICATIECE), Karnataka, India, 16–17 December 2022; pp. 1–5.
  26. Hussein, H.A.; Maher, S.M.; Shuwandy, M.L.; Armaneesa Naaman Hasoon, M.A.A.; Salih, M.M. Proposed Website Architecture with Embedded Augmented Reality to Discover Cultural Heritage. In Proceedings of the 2nd International Conference on Emerging Technologies and Intelligent Systems; Springer International Publishing: Cham, Switzerland, 2023; Volume 584, pp. 559–569.
  27. Ritsos, P.; Mearman, J.W.; Jackson, J.R.; Roberts, J.C. Synthetic Visualizations in Web-Based Mixed Reality. In Proceedings of the Immersive Analytics: Exploring Future Visualization and Interaction Technologies for Data Analytics: Workshop, IEEE Conference on Visualization (VIS), Phoenix, AZ, USA, 1 October 2017.
  28. Coma-Tatay, I.; Casas-Yrurzum, S.; Casanova-Salas, P.; Fernández-Marín, M. FI-AR Learning: A Web-Based Platform for Augmented Reality Educational Content. Multimed. Tools Appl. 2019, 78, 6093–6118.
  29. Rodrigues, B.; André, D.R.C.D.; Martins, V.F.; Bressan, P.A.; Paiva Guimarães, M. WebAR: A Web-Augmented Reality-Based Authoring Tool with Experience API Support for Educational Applications. In Universal Access in Human–Computer Interaction. Designing Novel Interactions; Antona, M., Stephanidis, C., Eds.; Springer International Publishing: Cham, Switzerland, 2017; Volume 10278, pp. 118–128.
  30. Nitika, N.; Sharma, T.K.; Rajvanshi, S.; Kishore, K. A Study of Augmented Reality Performance in Web Browsers (WebAR). In Proceedings of the 2021 2nd International Conference on Computational Methods in Science & Technology (ICCMST), Mohali, India, 17–18 December 2021; pp. 281–286.
  31. MacIntyre, B.; Hill, A.; Rouzati, H.; Gandy, M.; Davidson, B. The Argon AR Web Browser and Standards-Based AR Application Environment. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, 26–29 October 2011; pp. 65–74.
  32. Qiao, X.; Ren, P.; Nan, G.; Liu, L.; Dustdar, S.; Chen, J. Mobile Web Augmented Reality in 5G and beyond: Challenges. Opportunities. and Future Directions. China Commun. 2019, 16, 141–154.
  33. USDZ Schemas for AR. Available online: https://developer.apple.com/documentation/realitykit/usdz-schemas-for-ar (accessed on 23 June 2023).
  34. Maclntyre, B.; Smith, T.F. Thoughts on the Future of WebXR and the Immersive Web. In Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Munich, Germany, 16–20 October 2018; pp. 338–342.
  35. gLTF Runtime 3D Asset Delivery. Available online: https://www.khronos.org/gltf/ (accessed on 23 June 2023).
  36. William Lentz, R.; Paul Brutzman, D.; Kamburelis, M. X3D and GlTF Model Differencing for Conversions. Comparison and Conformance Testing. In Proceedings of the 26th International Conference on 3D Web Technology, Pisa, Italy, 8 November 2021; pp. 1–7.
  37. gltfpack. Available online: https://github.com/zeux/meshoptimizer/tree/master/gltf (accessed on 23 June 2023).
  38. glTF Pipeline. Available online: https://github.com/CesiumGS/gltf-pipeline (accessed on 23 June 2023).
  39. Petrangeli, S.; Simon, G.; Haoliang, W.; Vishy, S. Dynamic Adaptive Streaming for Augmented Reality Applications. In Proceedings of the 2019 IEEE International Symposium on Multimedia (ISM), San Diego, CA, USA, 9–11 December 2019; pp. 56–567.
  40. Hyunmin, N.; Park, G.S.; Hwangjun, S. Progressive Mesh-Based HTTP Adaptive Augmented Reality Streaming System. In Proceedings of the 2021 IEEE 46th Conference on Local Computer Networks (LCN), Edmonton, AB, Canada, 4–7 October 2021; pp. 257–264.
  41. Noh, H.; Song, H. HA 2 RS: HTTP Adaptive Augmented Reality Streaming System. IEEE Trans. Mob. Comput. 2023, 22, 2741–2755.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
Video Production Service
Feedback