Meet the SPIRIT Partners: a combined interview with Cornelius Hellge and Anna Hilsmann from the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut, HHI.
Dr.-Ing. Cornelius Hellge
heads the Multimedia Communications Group at Fraunhofer HHI since 2015. He received the Dipl.-Ing. degree in Media Technology from the Ilmenau University of Technology in 2006 and the Dr.-Ing. degree with distinction (summa cum laude) from the Berlin University of Technology in 2013. His current research interest is in the development of new formats for volumetric video for mixed reality, system integration of the new video codec VVC, and the further evolution of the 5G standard.
Dr.-Ing. Anna Hilsmann
heads the Vision & Imaging Technologies Department at Fraunhofer HHI where she also leads the Computer Vision & Graphics group. Her research focuses on Visual Computing and Computer Vision, and she is especially interested in model-based deep learning, incorporating prior knowledge into AI systems. She obtained her PhD in Computer Science from Humboldt University in 2014, with highest honors and a Diploma in Electrical Engineering and Information Technology from RWTH Aachen in 2007.
Q: Tell us more about the Fraunhofer HHI and its contribution to the SPIRIT project.
Cornelius: The Fraunhofer HHI in Berlin is a research institute working on a wide range of topics related to telecommunications, from mobile and optical communication networks to the processing and coding of video signals. The departments involved in the SPIRIT project is the Video Communication and Applications department as well as the Vision & Imaging Technologies department Together with my colleague Anna, we bring strong expertise in computer graphics, video coding and standardisation.
For SPIRIT, we are providing an innovative and complete streaming platform for Extended Reality (XR) content, where we have volumetric video representations rendered on a server and stream it over a cellular network using Web Real-Time Communication (WebRTC) to the end client. This is a platform that we have been developing over the last year. We will be developing it further over the course of SPIRIT and putting it into the project Open calls so that other companies can use it. This platform allows the streaming of highly realistic and photorealistic representations to basically any device on the market today because the hardware requirements on the end device are not very low. All you need is a hardware video recorder and there’s not a lot of computation on the device. The main computation is done on the server in the cloud, which is connected to low-latency cellular networks.
Anna: I’m heading the Vision and Imaging Technologies department and we have a long history of creating interactive virtual humans from volumetric video data. We have a volumetric studio where we capture human performance, which can then be viewed in 360 degrees of freedom, for example in virtual reality (VR) environments. Even if highly realistic and detailed, these are still videos, meaning that we play back the captured performance. However, in recent years we have developed methods to generate or learn animatable representations from this volumetric video data and put it into a format that can later be animated from, for example, speech or text or any other input specifying a pose or facial animation sequence. We also add neural rendering enhancing the rendering to make it more realistic. We then create interactive avatars or digital representations of people that can be used in hybrid formats or communication devices.
Q: What made you join SPIRIT’s journey?
Anna: It perfectly fits with our experience. Especially in the last three or four years, we have been working on technology for hybrid meetings and communication scenarios and technologies around the metaverse and virtual reality. Therefore, the SPIRIT project perfectly fits with the experiences and research topics that we’re currently dealing within Fraunhofer HHI.
Cornelius: SPIRIT is interesting because, from my perspective, it targets the most attractive use case for XR, meaning communication and multi-party communication in XR.
What also brought us to the project were the partners. SPIRIT has an excellent consortium, with all the most innovative partners in Europe working on the core parts of advancing communications networks. Ericsson, for example, comes from the mobile operator side, and Deutsche Telekom from the manufacturing and operator side. And we at Fraunhofer HHI are proud to contribute with our technology to this great consortium.
Q: At the end of its lifetime, what do you think SPIRIT’s contribution to the European XR environment will be?
Cornelius: I think in the end we will be able to provide a platform for small companies to distribute or develop applications that can bring highly realistic content to XR. Because the SPIRIT platform is standards-based, what is developed within the project can be applied to other platforms, both the streaming part and the encoding and avatar part. So, all the components that the SPIRIT partners are developing and that the participants in the Open calls are going to improve can also be used in other international formats and platforms that might emerge in the future. I think we will give the European partners a pretty good playground to develop future immersive technologies thanks to the SPIRIT platform.