The Holodeck is an empty space known from Star Trek that can create 3D environments used by the crew of a spaceship to prepare for missions and for entertainment. This sci-fi space simulated everything from lush jungles to Sherlock Holmes’ London. The fully interactive holodeck environments were infinitely customizable using only language – the team only had to ask the computer to create an environment and the space appeared on the holodeck.
Today, virtual interactive environments are also used to train robots before deployment in the real world. According to doctoral student Yue Yang, until now artists created these environments by hand. “Artists could spend a whole week building one environment,” Yang added.
A space that adapts interactive 3D environments was created in collaboration between the University of Washington and the Allen Institute. Named after its Star Trek ancestor, the Holodeck is capable of creating a virtually unlimited number of environments using artificial intelligence to interpret requests. “We can use language to control it,” says Yang. “You can easily describe any environments you want.”
Holodeck uses insights embedded in Large Language Models (LLM), the systems underlying ChatGPT and other chatbots. “Language is a very short representation of the whole world,” Yang said. LLMs have a surprisingly high level of knowledge about spatial design due to the large amount of text they absorb during their training. Essentially, Holodeck works by engaging the LLM in conversation using a carefully structured series of hidden queries to break down user requests into specific parameters.
For example, a modern Holodeck might be asked to create a “1b1b apartment for a scientist with a cat”. The system fulfills this request by dividing it into several stages: first the floor and walls are created, then the doorway and windows. Next, Holodeck searches the Objaverse, a large library of pre-made digital objects, for what furnishings you might expect in such a space: a coffee table, a cat tower, and so on. Finally, the Holodeck prompts the placement module, which the researchers designed to constrain the placement of objects.
The researchers also tested the Holodeck’s ability to create scenes that are less typical in robotics research and more difficult to create by hand than apartment interiors, such as shops, public spaces and offices. Comparing Holodeck’s outputs to ProcTHOR’s, which were generated using human-generated rules rather than AI-generated text, the researchers again found that human raters preferred the Holodeck-generated scenes. This preference applied to many indoor spaces, from science laboratories to art studios, from dressing rooms to wine cellars.
Source: Science Daily