In new research, experiments showed that robots trained with this dataset, called RoboSpatial, outperformed those trained with baseline models at the same robotic task, demonstrating a complex understanding of both spatial relationships and physical object manipulation. 

For humans, visual perception shapes how we interact with the environment, from recognizing different people to maintaining an awareness of our body’s movements and position. Despite previous attempts to imbue robots with these skills, efforts have fallen short as most are trained on data that lacks sophisticated spatial understanding.  

Because deep spatial comprehension is necessary for intuitive interactions, if left unaddressed, these spatial reasoning challenges could hinder future AI systems’ ability to comprehend complex instructions and operate in dynamic environments, said Luke Song, lead author of the study and a current PhD student in engineering at The Ohio State University.

“To have true general-purpose foundation models, a robot needs to understand the 3D world around it," he said. “So spatial understanding is one of the most crucial capabilities for it.”

The study was recently given as an oral presentation at the Conference on Computer Vision and Pattern Recognition. 

To teach robots how to better interpret perspective, RoboSpatial includes more than a million real-world indoor and tabletop images, thousands of detailed 3D scans, and 3 million labels describing rich spatial information relevant to robotics. Using these vast resources, the framework pairs 2D egocentric images with full 3D scans of the same scene so the model learns to pinpoint objects using either flat-image recognition or 3D geometry. 

According to the study, it’s a process that closely mimics visual cues in the real world. 

For instance, while current training datasets might allow a robot to accurately describe a “bowl on the table,” the model would lack the ability to discern where on the table it actually is, where it should be placed to remain accessible, or how it might fit in with other objects. In contrast, RoboSpatial could rigorously test these spatial reasoning skills in practical robotic tasks, first by demonstrating object rearrangement and then by examining the models’ capacity to generalize to new spatial reasoning scenarios beyond their original training data.

“Not only does this mean improvements on individual actions like picking up and placing things, but also leads to robots interacting more naturally with humans,” said Song. 

One of the systems the team tested this framework on was a Kinova Jaco robot, an assistive arm that helps people with disabilities connect with their environment. 

During training, it was able to answer simple close-ended spatial questions like “can the chair be placed in front of the table?” or “is the mug to the left of the laptop?” correctly. 

These promising results reveal that normalizing spatial context by improving robotic perception could lead to safer and more reliable AI systems, said Song. 

While there are still many unanswered questions about AI development and training, the work concludes that RoboSpatial has the potential to serve as a foundation for broader applications in robotics, noting that more exciting spatial advancements will likely branch from it. 

“I think we will see a lot of big improvements and cool capabilities for robots in the next five to ten years,” said Song. 

Co-authors include Yu Su from Ohio State and Valts Blukis, Jonathan Tremblay, Stephen Tyree and Stan Birchfield from NVIDIA. This work was supported by the Ohio Supercomputer Center.

In the first work to introduce the concept, researchers have outlined the role an AI concierge, a technologically advanced assistant, may play in various areas of the service sector as well as the different forms such a helper might embody. 

Their paper envisions a virtual caretaker that, by combining natural language processing, behavioral data and predictive analytics, would anticipate a customer’s needs, suggest certain actions, and automate routine tasks without having to be explicitly commanded to do so. 

Though such a skilled assistant is still years away, Stephanie Liu, lead author of the paper and an associate professor of hospitality management at The Ohio State University, and her colleagues drew insight from several contemporary fields, including service management, psychology, human-computer interaction and ethics research, to detail what opportunities and challenges might arise from having an AI concierge manage human encounters. 

“The traditional service industry uses concierges for high-end clients, meaning that only a few people have access to them,” Liu said. “Now with the assistance of AI technology, everybody can have access to a concierge providing superior experiences.”

On that premise, the benefits of incorporating AI into customer service are twofold: It would allow companies to offer around-the-clock availability and consistency in their operations as well as improve how individuals engage with professional service organizations, she said. 

Moreover, as the younger workforce gravitates to more tech-oriented jobs and global travel becomes more common, generative AI could be an apt solution to deal with the escalating demands of evolving hospitality trends, said Liu. 

“The development of AI technology for hotels, restaurants, health care, retail and tourism has a lot of potential,” she said. 

The paper was published recently in the Journal of Service Management. 

Despite the social and economic benefits associated with implementing such machines, how effective AI concierges may be at completing a task is dependent on both the specific situation and the type of interface consumers use, said Liu. 

There are four primary forms a smart aide might take, each with distinctive attributes that would provide consumers with different levels of convenience, according to Liu. 

The first type is a dialogue interface that uses only text or speech to communicate, such as ChatGPT, a conversational agent often used to make inquiries and garner real-time assistance. Many of these interactive devices are already used in hotels and medical buildings for contactless booking or to connect consumers with other services and resources. 

The second is a virtual avatar that employs a vivid digital appearance and a fully formed persona to foster a deeper emotional connection with the consumer. This method is often utilized for telehealth consultations and online learning programs.  

The third iteration is a holographic projection wherein a simulated 3D image is brought into the physical world. According to the paper, this is ideally suited for scenarios where the visual impact is desired, but physical assistance itself is not necessary. 

The paper rounds out the list by suggesting an AI concierge that would present as a tangible, or touchable robot. This form would offer the most human-like sensory experiences and would likely be able to execute multiple physical tasks, like transporting heavy luggage. 

Some international companies have already developed these cutting-edge tools for use in a limited capacity. One robotic concierge, known as Sam, was designed to aid those in senior living communities by helping them check in, make fall risk assessments and support staff with non-medical tasks. Another deployed at South Korea’s Incheon International Airport helped consumers navigate paths to their destination and offered premier shopping and dining recommendations. 

Yet as advanced computing algorithms become more intertwined in our daily lives, industry experts will likely have to consider consumer privacy concerns when deciding when and where to implement these AI systems. One way to deal with these issues would be to create the AI concierge with limited memory or other safewalls to protect stored personal data, such as identity and financial information, said Liu.  

“Different companies are at different stages with this technology,” said Liu. “Some have robots that can detect customers’ emotions or take biometric inputs and others have really basic ones. It opens up a totally different level of service that we have to think critically about.”

What’s more, the paper notes that having a diversity of concierge options available for consumers to choose from is also advantageous from a mental health standpoint.

Because AI is viewed as having less agency than their human counterparts, it might help mitigate psychologically uncomfortable service situations that could arise because of how consumers feel they might be perceived by a human concierge. This reduced apprehension regarding the opinion of a machine may encourage heightened comfort levels and result in more favorable responses about the success of the AI concierge, said Liu. 

Ultimately, there’s still much multidisciplinary testing to be done to ensure these technologies can be applied in a widespread and equitable manner. Liu adds that future research should seek to determine how certain design elements, such as the perceived gender, ethnicity or voice of these robotic assistants, would impact overall consumer satisfaction. 

William Ausich, a professor of earth sciences at The Ohio State University who has studied paleontology for over five decades, believes so. 

In a letter published today in the Proceedings of the National Academy of Sciences, Ausich offered commentary on a study published in the same issue on a new soft robot that may help researchers in the field test their hypotheses regarding how long-extinct creatures might have maneuvered through their environments. 

“Being able to test how these organisms once operated is really important for understanding paleoecology and the history of life on Earth,” said Ausich. “Making soft robots is a very innovative new approach for us to accomplish that goal.”

Ausich discussed a study led by Richard Desatnik and Carmel Majidi at Carnegie Mellon University, and Zach J. Patterson from Carnegie Mellon University and Massachusetts Institute of Technology. The researchers developed a soft robot prototype, named “Rhombot,” which is a biomimetic – meaning it uses concepts from nature to solve complex problems.

The Rhombot was modeled after a genus of echinoderms called Pleurocystites. Echinoderms generally refer to invertebrates like starfish, sea cucumbers and sand dollars that possess distinct internal skeletons, but the Pleurocystites are extinct organisms that have a flattened body with two large feeding appendages that move along the seafloor.

Unfortunately, there are no known pleurocystitid fossils whose imprints could reveal more about the mechanics of their movement, so interpreting these animals’ life habits has been restricted to investigating their skeletal morphology, said Ausich. 

Now, with the rise of paleobionics – a field that combines advances in robotics with tried-and-true paleontological principles – researchers are beginning to fill in those deep gaps within the fossil record.

According to Ausich, the creators of Rhombot began by mimicking the special connective tissue echinoderms have and setting up various theoretical and physical simulations for the Rhombot to successfully move across a contact-rich surface meant to represent an ancient, hard seafloor. 

The letter then notes that the authors found their experiments confirmed one of researchers’ previous predictions about the organisms’ motions, as the robot moved in an anterior direction with its feeding appendages first as a result of the movement of its stem, or the wide tail that extends from its back. The team also learned that Rhombot’s speed was maximized by the sweeping gait of its tail and that real-life specimens with certain body-to-tail ratios might have developed an evolutionary trend for increased speed.

These findings are particularly informative for modeling extinct organisms for which current scientists have no modern analog to compare them to, said Ausich. It’s so innovative that the letter suggests that Rhombot and other soft robotic technologies like it could also be utilized to analyze behavioral data from animals in other taxa, as well as evaluate evolutionary changes from one ancient form to the next.

Because scientists are working on a timescale of millions of years, having a clearer picture of the evolutionary puzzle could provide clues into why some species lived and others died off. 

“Extinction is a big issue on Earth today, and there’s only so much biologists can do to mitigate it,” said Ausich. “But by studying what came before and after extinction and understanding the lifestyles of the organisms that worked or didn’t work back then, we can actually provide  a perspective on survival that nobody else can.” 

And it’s possible that these new perspectives and theories could shine a light on what kinds of species will survive the next great extinction event. 

Although the robot is still a long while away from mass development, it could one day serve as an educational tool, animating long-gone prehistoric beings before researchers’ very eyes. Something like this would be especially useful for getting younger generations to fall in love with a lesser-known side of paleontology – that is, the side without dinosaurs, Ausich said. 

“To be able to show a child or student how a specimen lying in rock might have actually moved – it just excites the imagination,” said Ausich.

Led by a team of researchers at The Ohio State University, the study published recently in the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022 describes a framework for testing and characterizing the safety of legged robots, machines that, unlike their wheeled counterparts, rely on mechanical limbs for movement. The study found that many current legged robotic models don’t always act predictably in response to real-life situations, meaning it’s hard to predict whether they’ll fail – or succeed – at any given task that requires movement.

“Our work reveals that these robotic systems are complex and, more importantly, anti-intuitive,” said Bowen Weng, a PhD student in electrical and computer engineering at Ohio State. “It means you can’t rely on the robot’s ability to know how to react in certain situations, so the completeness of the testing becomes even more important.”

As mobile robots evolve to carry out more diverse and sophisticated tasks, many in the scientific community also note that the industry needs a set of universal safety testing regulations, especially as robots and other artificial intelligence have gradually begun to flow into our everyday lives. Legged robots especially, which are often made of metal and can run as fast as 20 mph, could quickly become safety hazards when expected to operate alongside humans in real and often unpredictable environments, said Weng.

“Testing is really about assessing risk, and our aim is to investigate how much risk robotics currently presents to users or customers while in a working condition,” he said. 

While there are currently some safety specifications in place for the deployment of legged robots, Weng noted that there isn’t yet any common agreement on how to test them in the field. 

This study develops the first data-driven, scenario-based safety testing framework of its kind for legged robots, said Weng. 

“In the future, these robots might have the chance to live with human beings side-by-side, and will most likely be collaboratively produced by multiple international parties,” he said. “So having safety and testing regulations in place is extremely important for the success of this kind of product.” 

The research, which was partly inspired by Weng’s work as a vehicle safety researcher at the Transportation Research Center, which partners with the National Highway Traffic Safety Administration, takes advantage of sample-based algorithms to discern how simulated robots  partially driven by machine learning techniques would fail during real-world testing. 

Though various factors can be used to characterize a robot’s overall safety performance, this study analyzed a set of conditions under which the robot would not fall over while actively navigating a new environment. And because many of the algorithms the team used stemmed from previous robotics experiments, they were able to design multiple scenarios for the simulations to run. 

One trial focused on studying the robot’s ability to move while performing tasks at different gaits, such as walking backward or stepping in place. In another, researchers tested whether the robot would take a tumble if it was periodically pushed with enough force to alter its direction.

The study showed that while one robot failed to stay upright for 3 out of 10 trials when asked to slightly speed up its gait, another could remain upright over 100 trials when pushed from its left side, but fell over during 5 out of 10 trials when the same force was applied to its right side. 

Eventually, the researchers’ framework could help certify the commercial deployment of legged robots and help establish a safety benchmark for robots created with different structures and properties, though Weng noted it’ll be a while before it can be implemented. 

“We believe this data-driven approach will help create an unbiased, more efficient way to make observations of robots in the conditions of a test environment,” he said. “What we’re working towards isn’t immediate, but for researchers down the line.”

Co-authors were Guillermo Castillo and Ayonga Hereid of Ohio State, and Wei Zhang of the Southern University of Science and Technology in Shenzhen, China. This work was supported by the National Science Foundation and the National Natural Science Foundation of China.