In mid 2023, Kerbl & Kopanas et al. introduced a new 3D reconstruction algorithm that creates 3D scenes from photos, representing them via gaussian splats [1]. Since then, Gaussian Splats have led to a huge hype in the visual computing community since their visual quality is superior to triangle-based photogrammetry approaches, but they are also fairly fast to render and easily editable, unlike NERF-based approaches.

Although Gaussian Splatting was not part of the original proposal - they were published roughly at the time of the proposal's submission - their importance to 3D-scans of real-world environments turned them into a high-priority task afterwards. In recent months we spent time learning about splatting algorithms and building our own desktop renderers. In the following months, we will then gradually port them to Potree-Next.

The following figures shows gaussian splat models from our desktop renderer (model is from Kerbl & Kopanas et al.), and to the right the 2D-projected gaussians it consists of. Each gaussian can be thought of as an ellipsoid comprising a color, opacity, position, scale and rotation value. During rendering, the 3D ellipsoid is approximated by a screen-facing 2D ellipsoid (as shown), which is not a perfect representation, but one that is very fast to render. Of note are the larger splats in the right picture that seem to be missing in the left one. This is because the reconstruction algorithm tries to optimize splats so that they represent the original images as good as possible, and such large splats often imitate subtle illumination behaviours, including dust/mist/atmospheric scattering, or they can simply be undesirable artifacts like floaters. They still affect the image, but due to their low opacity in a more subtle way. 

Initially, we will integrate gaussians plats without levels-of-detail, i.e., the size of the models is somewhat restricted since loading larger models will take time. However, there are compression and progressive-streaming approaches that still work well for models with up to around 10 million splats, as seen by several dedicated splat viewers that were published in 2024. 

[1] https://github.com/graphdeco-inria/gaussian-splatting