Penn State Developing Worker-centered Human Robot Partnerships

"Once we capture workers' cognitive load, we try to transfer this information into the robot so that the collaborative robot can monitor workers' cognitive load," Jebelli added.

Whenever the cognitive load is recognized to be higher than a specific threshold, the robot will reduce its pace to provide a safer environment for the workers, said Jebelli. This response could help design a collaborative robotic system that understands the human partner’s mental state and hopefully improve workers’ safety and productivity in the long term. The team published their results in two papers in Automation in Construction.

They also proposed a BCI-based system to operate a robot remotely.

“The ability to control a robot by merely imagining the commands can open new avenues to designing hands-free robotic systems in hazardous environments where humans require their hands to retain their balance and perform an action,” said Mahmoud Habibnezhad, a postdoctoral fellow conducting research with Jebelli.

The researchers capture workers’ brainwave signals with a wearable electroencephalogram (EEG) device and convert these signals into robotic commands.

“In our research, first we trained the subjects with a multiple imagery experiment,” said Yizhi Liu, doctoral student of architectural engineering. “The signal is then collected through EEG sensors and a spatial feature extraction technique called a common spatial pattern,” said Liu.

He explained that participants view images of specific actions, such as workers grabbing bricks with their right hands, and then imagine these actions. For example, when a subject imagines their right hand grabbing something, the right cortex of their brain generates a higher EEG signal than their left-brain area. The researchers employed machine learning to train the robots, using participants’ thoughts when imagining the actions. Subsequently, these translated signals will be transferred as digital commands to the robots through ROS, or robotic operating system, Liu added.

For the BCI system to continuously interpret brainwave signals from workers in near real-time, the researchers used three key elements — a wearable EEG device, a signal-interpretation application program interface (API), and a cloud server. The wearable EEG device captures the brainwave signals and sends them to the cloud server, and then the API begins generating commands.

The researchers created a network of channels between workers’ wearable biosensors and robots using ROS that acts as the middleware connecting different systems. Through these channels, commands such as the right-hand movement, left-hand movement and stop signal, can be easily sent to the robot. However, more nuanced commands require more data and improve the performance of the system and teleoperation of the robot, according to Jebelli.

“We developed a brain-computer interface system, which we can think of as a person trying to learn a new language that doesn’t know how to generate commands,” he said. “We try to connect different commands with some predefined patterns of their brainwaves.”

With more commands, the researchers can train and improve the performance of the system, according to Jebelli. These different commands include tasks such as controlling the robot, stopping the robot, or designing some predesigned work plan, such as delivering material from point A to point B by thinking about some specific tasks in the dictionary.

“This is a framework that we tested out for one robot, that is a proof-of-concept that the framework is working,” said Habibnezhad. “We can improve the framework by using different robots or drones or different systems. We can improve the accuracy of the control by using more commands and trying to extract more patterns and defining different controls.”