vr on Calliope Davies

Meta Reality Labs VR/AR CAVE

Sat, 10 Feb 2024 00:00:00 +0000

In 2024 and 2025, I worked at Meta Reality Labs as a contingent worker under Pixela as a software engineer developing and maintaining their CAVE setup. CAVEs are essentially stages with displays on the floor and walls and motion capture for the user that can be used for a wide range of VR applications. This CAVE in particular was designed to be bleeding edge and capable of simulating arbitrary hardware for prototyping consumer VR/AR HMDs, allowing for rapid, low cost design iteration and user feedback.

On the operations team for this CAVE, we largely worked with researchers and engineers from other teams who were developing hardware VR/AR prototypes. A Unity or Unreal simulation of the hardware could be loaded on the CAVE, allowing quick and easy demonstration of the hardware characteristics of these projects, along with the ability to adjust them on demand. For example, this could include testing different HMD FOVs, optical behavior of lens geometry, performance in ambient light in different scenarios, or mixed reality performance with physical props and furniture. This obviates the need for producing expensive intermediary hardware prototypes.

My role specifically was fairly spread between helping other teams realize their projects in the CAVE ecosystem, maintaining the render cluster and corresponding infrastructure, and giving demonstrations to groups and individuals coming through to see the technology. I helped dial in parameters for testing/display based on user requirements and feedback, as well as consult on adapting new projects to the platform.

As a complex cluster system with lots of moving parts running with very tight tolerances, maintaining and operating it was non-trivial. To this end, I worked on cluster management tools in Python and PowerShell, standardized system setup procedures, and validating updates to middleware and drivers across machines, in addition to a wide variety of hardware and software IT work.

Some broad (hopefully non-NDA violating) specs of the CAVE in question, for those curious about its performance or complexity:

Capable of 240Hz mono or 120Hz stereo operation with active shutter glasses
Stage with high density, high contrast LED panels on the floor and three walls, several meters on a side
Compute cluster consisting of ten render nodes and one control node, all with high performance Xeon CPUs and Quadro GPUs, networked with 100Gb InfiniBand
Fifteen 240Hz motion capture cameras for full tracking of HMD and props
Capable of simulating both dark shadows and bright daytime sun at the same time

First Contact - Oculus Quest Port

Wed, 01 Jan 2020 00:00:00 +0000

This was our second contract working with the Oculus team - this time, to port the First Contact Oculus Rift demo to the upcoming Oculus Quest. This was quite a big task, as, in addition to the vast difference in power between a PC capable of running Oculus Rift and the upcoming mobile based Quest platform, the original Rift version (from our understanding) hadn’t even been initially intended as a full product, so there was a lot that needed to be optimized.

My job was to coordinate the graphics side of the port. This involved working with our artists to help optimize assets wherever possible and reducing material complexity as much as we could. We managed to bring both the triangle count and number of draw calls in the scene by orders of magnitude.

The original game relied almost entirely on dynamic lighting, which obviously wasn’t tenable on the Quest platform, but the changes in lighting were vital to the experience of the game. What I ended up doing was going into the engine to add a new shading model that would let us “fake” dynamic lighting through dynamic adjustment of the ambient section of the shading pipeline. As a result, almost every material in the final game is unlit, allowing for a vast improvement in performance. This also allowed us to fake global lighting effects like glow colors during certain points of the game.

In addition to these improvements, I cut down the overall complexity of all materials wherever possible. Our biggest bottleneck ended up being number of texture samples, so I was able to cut unnecessary texture maps, combine others, and replaced many with purely procedurally generated logic.

Oculus Touch NUX Localization

Thu, 10 Oct 2019 00:00:00 +0000

In early 2018, Fun Bits was contracted to add support for a large list of languages to a few of its demos and tutorial applications. The main one I worked on of these was the Touch Basics (or Touch NUX) application, which was what would run upon first setup of the Oculus Touch controllers.

This was a little bit of a challenge, as the project had been built on, at the time, a pretty old version of UE4, which predated many later features for both localization and font support in VR. Additionally, we were adding over 20 new languages, so getting an efficient pipeline to get the newly localized content into the game and tested was vital.

I built several tools for this project, mostly in Python, to optimize this flow. Translated data had to be checked for invalid characters, signs of incorrect translations, and missing keys. The data had to be quickly brought into the engine without stomping any data we didn’t want it to. We also had to isolate all characters used by all text to make sure it would be properly included in the displayed font for all languages.

By the end of this project, we had a pretty polished process - the tools I had been working on allowed us to reliably handle the data we recieved from the localization team and to quickly identify any errors at several steps in the pipeline.

Orbital Loop

Sun, 10 Jan 2016 00:00:00 +0000

Orbital Loop is a music “game” for Google Daydream, as one of two projects (with reGrow) we were working on for the then upcoming VR platform. It’s essentially a relaxing music composition application, intended to take advantage of VR to give a more engrossing experience than would necessarily be possible with a conventional platform.

I was moved over to working primarily on this project after reGrow was cancelled. My main tasks were to assist with gameplay systems (especially to get logic more performant on the Daydream platform) and to set up and maintain a backend system for managing user data and uploading and downloading user created songs, for which we used a combination of Google App Engine and PlayFab.

reGrow

Sat, 09 Jan 2016 00:00:00 +0000

reGrow was one of the two projects Fun Bits was working on (with Orbital Loop) for the then upcoming Google Daydream mobile VR platform, and the first project I worked on at Fun Bits that was really our own. The premise changed some over its development, but always revolved around the idea of growing and breeding gardens of flowers.

I was involved with two parts of this project. My primary contribution was to the graphical systems of the game, where I set up a system for spawning and animating large numbers of instanced meshes with minimal material overhead. This was important because, especially on the devices we were developing on (which were appreciably weaker than the intended production devices), draw calls were at a premium, and naively rendering potentially dozens of flowers, each assembled from half a dozen or more meshes was not feasible.

I also built several gameplay systems to allow designers to quickly iterate and not worry about system performance issues, and put together a system for modeling plant genetics and breeding.

Sadly, the project ended up getting cancelled, and all effort was consolidated into Orbital Loop until its release.