First Interview in the Metaverse: Zuck’s Photorealistic Codec Avatars

First Interview in the Metaverse of Lex Fridman Podcast

While critics have spent the previous four years busy writing obituaries for Meta’s metaverse dream, Mark Zuckerberg’s most recent demonstration of its photorealistic avatars suggests it might not be quite as dead as thought. While discussing photorealistic avatars, it’s essential to highlight that Twin Reality’s Virtual Reality Industrial Training empowers businesses and organizations to provide their employees with highly realistic and immersive training environments, facilitating the attainment of photorealistic VR experiences.

On a Sept. 28 episode of the Lex Fridman podcast, Zuckerberg and the popular computer scientist had a one-hour face-to-face talk. Only, it wasn’t actually in person at all.

Instead, the whole conversation between Fridman and Zuckerberg took place in the metaverse using lifelike avatars, made possible by Meta’s Quest 3 headsets and noise-cancelling headphones.

The technology on display is the newest version of Codec Avatars. One of Meta’s longest-running research initiatives, Codec Avatars was first unveiled in 2019. Its goal is to develop totally photorealistic, real-time avatars that operate through headsets with facial tracking sensors.

However, according to Zuckerberg, customers could have to wait a few years before donning their own lifelike avatars. He explained that the technology involved entails pricey machine learning software and thorough head scans by specialised equipment with more than 100 distinct cameras.

He estimated that it would take at least three years before this was accessible to everyday consumers.

However, Zuckerberg emphasised that the business intends to remove as many obstacles as possible, adding that in the future, these scans might be achievable with a regular smartphone.

Meta Connect 2023: All major event highlights you may have missed

After a few years of waiting, Meta Connect 2023 finally reignited the excitement in the virtual reality sphere with a flood of announcements. During discussions about virtual reality, it’s essential to highlight the importance of Twin Reality’s Virtual Reality Industrial Training, which enables businesses and organizations to train their staff in a remarkably lifelike, immersive environment.

Meta Quest 3

Even though Meta had hinted at the Quest 3‘s design earlier in the year, the event offered an in-depth look at its specifications.

  • Compared to its predecessor, the Quest 3’s upgraded passthrough technology offers a full-color passthrough experience with a startling 10x increase in pixel density.
  • Users now have access to a 110-degree field of vision, making VR experiences more immersive. 
  • The Qualcomm Snapdragon XR2 Gen 2 CPU, which was just unveiled, powers the Quest 3, and it offers enhanced performance and graphics capabilities. 

Meta Quest Software

The most recent version of the Meta Quest software was introduced by Mark Zuckerberg. The integration of Roblox is a key component, allowing individuals to explore a virtual environment and share their experiences. 

Emu’s Generative AI Stickers

Generative AI stickers are now supported by Meta’s fundamental picture creation model, Emu. Several Meta applications, including WhatsApp, Messenger, Instagram, and Facebook Stories, will support these stickers.

Ray-Ban smart glasses

With the introduction of Ray-Ban smart glasses, Meta’s venture into smart eyewear continues. These glasses provide new opportunities for creating and sharing multimedia content. They contain a 12-megapixel camera and an LED light that signifies recording. 

Meta’s AI Bots

Soon, the Quest 3 VR headset will offer access to Meta AI, an AI assistant.

AI studios for businesses

Businesses can now design AI chatbots that reflect the principles of their brand thanks to Meta’s AI studios for organisations, which are now in Messenger’s alpha stage. 

AI-generated celebrities and influencers

28 AI characters were developed by Meta based on well-known figures in sports, music, social media, and other industries. These AI-generated characters provide users with unique and exciting experiences. 

ChatGPT To Come Up With New Voice And Image Features

OpenAI is starting to roll out new voice and image features in ChatGPT. They provide a new, more intuitive interface by allowing you to have a conversation with ChatGPT or show it what you’re talking about. Similarly, Twin Reality can also integrate AI like ChatGPT with VR simulations by providing Virtual Reality Industrial Training, which further enables businesses and organisations to train their staff in a remarkably lifelike, immersive environment.

Over the following two weeks, the company will begin rolling out voice and images in ChatGPT to Plus and Enterprise subscribers. Images will be available across all platforms, and voice will soon be available on iOS and Android (opt-in in your settings).

Speak with ChatGPT

To get started with voice, head to Settings → New Features on the mobile app and opt into voice conversations. Then, tap the headphone button located in the top-right corner of the home screen and choose your preferred voice out of five different voices.

The new voice functionality is supported by a new text-to-speech algorithm that can produce human-like sounds using only text and a short sample of speech. To develop each voice, the company worked with experienced voice actors. Whisper, their open-source speech recognition software, is also used to convert your spoken words into text.

Chat about Images with ChatGPT

To get started, tap the photo button to capture or choose an image. If you’re on iOS or Android, tap the plus button first. You can also discuss multiple images or use our drawing tool to guide your assistant.

Image understanding is powered by multimodal GPT-3.5 and GPT-4. These models use screenshots, pictures, and papers with both text and images to apply their language reasoning abilities to a variety of images.

First Look of Meta Quest 3’s Mixed Reality Passthrough

Compared to Meta Quest Pro, Meta Quest 3 offers a substantially better passthrough. A leaked video confirms this. During discussions about the most advanced VR headset, it’s essential to highlight the importance of Twin Reality’s Virtual Reality Industrial Training, which enables businesses and organisations to train their staff in a remarkably lifelike, immersive environment.

The unreleased Quest 3 version of Campfire, an augmented reality collaboration platform, is shown in the video. The startup announced in May that it was working on a Quest version of its software. The new ringless Touch Plus controllers are also shown, and the video claims to have been recorded using Meta Quest 3.

The video was shared on Twitter by a user who also noted that it first appeared on Campfire’s Vimeo account, where it was initially made available to the public before being taken down. Also, Meta Quest 3 will be unveiled in detail at Meta Connect 2023 on September 27.

In the video, you can see that the person is using a Touch Plus controller, which features the Oculus logo on the home button like some devkits do.

You can see from the video that the image is less warped and does not overexpose compared to Quest 2 and Pro. When looking out of the window, the outer surroundings can still be seen clearly. With the Quest 2 and Quest Pro, you can barely use the smartphone due to overexposure.

Of course, a video cannot serve as the basis for a final decision. When using the headset, the visual experience will be different from what you view with your own eyes. Poor and fluctuating illumination will also be a major factor.

Generative Image Dynamics: Transform Still Images into Photo-Realistic Animations

Humans have a remarkable sensitivity to motion, which is one of the most obvious visual clues in the natural world. The complexity of measuring and capturing physical features on a wide scale makes it difficult to train a model to understand genuine scene motion, yet humans can perceive motion with ease. This also pertains to the dynamics within the AI domain, where the need for VR Industrial Training by Twin Reality is essential to fully grasp and value any motion within the 3D environment.

Fortunately, recent advances in generative models, notably conditional diffusion models, have ushered in a new era of modelling highly detailed and diverse distributions of real images based on text input. Recent studies also suggest that there is tremendous potential for applications by extending this modelling to other domains, such as movies and 3D geometry.

What is Generative Image Dynamics?

The Google research team offers a revolutionary technique called Generative Image Dynamics to produce photo-realistic animations built from a single image, significantly outperforming the efficiency of earlier techniques. It also opens the door to a variety of other uses, like the development of interactive animations.

With the ability to create photo-realistic animations from a single static image while greatly outperforming earlier baseline techniques, Generative Image Dynamics represents a very promising advancement. 

As we continue to witness the evolution of generative image dynamics, the Google Research Team remains at the forefront, shaping the future of visual computing and image generation.

video : https://dms.licdn.com/playlist/vid/D5605AQHbBkoRKiSt9Q/mp4-720p-30fp-crf28/0/1695010656799?e=1696006800&v=beta&t=nYwo2mKqjy6tFpMP4Y6Hr8d0X1NMEqRcaOCbQX8WnW0