HMI Dojo initiative: a digital twin development in Unreal Engine 5

Imagine tracking work progress on a construction site not only with your eyes but remotely, whilst being far from the area. How much more convenient would it be to follow the machine’s surroundings not only via windows but with an extended view on a tablet inside the cockpit? Or try to estimate the savings that virtual simulations can offer instead of trying new approaches in real life.

This is the promise of the digital twin concept, a key area of the new HMI initiative run in Spyrosoft’s R&D programme. The project seeks to develop DigiTwin, a digital counterpart to a mini excavator, that replicates the physical machine’s functionality in the virtual realm using Unreal Engine 5 technology. In this article, we walk you through the journey of creating the DigiTwin, its current features and potential enhancements, keeping in mind how such a concept could shape the future of heavy machinery or agriculture industries.

What is a digital twin development?

A digital twin development allows the creation of a digital counterpart to a real-world device like a vehicle, construction machinery, or even an entire entity such as a factory, shop, or field. This digital replica mirrors the object’s physical state and allows monitoring of the functions, simulating scenarios or conducting analysis without interfering with reality.

Digital twin supports decision-making for changes in physical facilities. For example, having a virtual copy of a supermarket, you can run scenarios for various cash registers and self-checkout arrangements and then introduce the one with the highest capacity.

Digital twin concept development in the DigiTwin R&D project

The DigiTwin is a specific project undertaken as a Dojo initiative – Spyrosoft’s internal research and development programme. It represents a digital twin of a particular machine, in this case, a mini excavator (hence the name – digger twin). The goal is to track the device’s operation in real-time and interact with it via a dedicated application.

The idea behind the initiative was to provide support for heavy or agricultural machinery as the concept could further be transferred to other devices. In the case of an excavator’s digital twin development, the project fulfils two roles:

• DigiTwin supports the operator to control the machine’s position better with visualisation on a tablet inside the cockpit,
• It enables the construction manager or supervisor to monitor and access real-time information about the device, even in a different geographical area.

Although DigiTwin is still in progress (that is a constant state for R&D projects), several feature enhancements and upgrade ideas are still in the pipeline. Plans include controlling the machine from a tablet, tracking the excavator’s surroundings or supporting the operator, i.e., by transferring construction work parameters to the application and monitoring the level of reaching a target such as the depth or shape of an excavated trench.

DigiTwin as a Dojo Initiative

The initiative aims not merely to create a product but to apply new solutions and pave the way in laboratory conditions before realising a potential project in this area. That’s why the project has been launched as part of our internal R&D, Dojo HMI.

The objectives we set when launching the initiative include the following:

• expanding the HMI offering with digital twin services,
• enhancing Spyrosoft’s visibility in the agriculture and heavy machinery market,
• scaling technical competencies in Unreal Engine 5,
• scaling 3D design capabilities,
• deepening the knowledge of Microsoft Azure IoT services.

How does DigiTwin work? Technologies used

DigiTwin operates through connectivity with a physical excavator model. The machine is steered via a controller and has mounted sensors that collect data about its movement and position.

Then, data is transmitted to the IoT Hub in the Microsoft Azure cloud. The cloud serves as a connection point between the physical model and its digital twin, represented in the application running on Unreal Engine 5.

The Unreal Engine 5 application receives data from the cloud and mirrors the operation of the excavator in real-time, creating its dynamic 3D model. The person supervising the work may therefore have the physical machine out of sight but see its movements on the digital replica.

Hardware and physical excavator model

As none of us had a regular-sized machine in their garage, we bought a metal electric control digger EC160E Volvo Double-E-Hobby which is an exact copy of a 16-tonne excavator scaled in 1:14. As an operator’s tablet, we chose the iiyama ProLite T2736MSC-B1 27″ touch monitor.

Sensors and controllers

As the main microcontroller, we’ve picked the ESP32-S3 running FreeRTOS. It is responsible for sensor data gathering and communication with the Microsoft Azure cloud. The measurements themselves are done with four LSM6DSO IMUs from STMicroelectronics – three as accelerometers for the digger joint (movable arm segment) absolute angles and one as the gyro for the cabin rotation. All the MCU-sensors communication is based on the I²C protocol.

3D modelling in Blender and 3ds Max

The 3D model of the excavator for the DigiTwin project was created using Blender and 3ds Max. Both these platforms are well adapted for work with Unreal Engine. Although we faced some difficulties while importing data into Unreal Engine 5, eventually we came up with a refined method. You can read more about the case in the article Skeletal animations in Unreal Engine 5. How to import a 3D model?.

The role of Unreal Engine 5

Unreal Engine is a game engine that enables the creation and rendering of advanced, realistic 3D objects and environments. Although primarily known for its usage in computer game production, it also finds application in other areas such as film, animation, architecture, engineering, and medicine.

We chose Unreal Engine 5 for the DigiTwin project for several reasons:

• this game engine already has mechanisms that simplify the creation of solutions like digital twins. It allows for the rendering of advanced 3D graphics in real-time, which is crucial for dynamically replicating the machine in the application;
• with the growing role of 3D graphics across various industries, game engine manufacturers started paying attention to other markets. We decided to test if Unreal Engine 5 is a good fit for the construction and agriculture sectors;
• the choice of Unreal Engine was also motivated by a desire for team competence development. We wanted to use this project to enhance our skills, showcase the technology’s capabilities, and offer clients solutions based on the engine.

Data processing in Microsoft Azure’s IoT Hub

As public clouds are one of the most convenient and available hosting and computing solutions, we turned to Microsoft Azure for that project. In the previous Dojo initiative, we got closer to Amazon Web Services, so this time, we agreed that we would get to know the second largest cloud service provider and eventually compare the two platforms.

Also, Azure cloud offers a wide range of services and tools related to the Internet of Things – IoT Hub – which is crucial for a project that involves collecting and transmitting data from sensors. We wanted to test how this module would work with Unreal Engine and how it would operate in practice.

Future development plans

As mentioned earlier, the DigiTwin project aims to deepen competencies rather than releasing a market-ready product. The MVP version, created over several weeks, includes “just” a replication of the physical device’s movement in Unreal Engine.

However, we still have several feature enhancements and ideas in our backlog, including:

• working on the surroundings. We plan to refine the environment in which the machine operates to make it more realistic and mimic the actual surroundings, such as the construction area with blockades or terrain inclines;
• operator’s POV. The current animation covers the outside and we intend to add the inside perspective so the operator can have a supporting view on the tablet;
• expansion to augmented and virtual reality: it is possible to extend the product to augmented and virtual reality capabilities – adding the option of using AR/VR glasses to interact with the machine;
• production process development. We aim to develop the infrastructure for the entire software production process, including CI/CD systems and Unreal Engine tests;
• direct device connection bypassing the cloud. Among our plans is the concept of connecting the excavator to the Unreal application via micro-ROS;
• attendance at industry events and shows. After our debut at the local Learn and Chat meeting in Szczecin and Car HMI Europe in Berlin, we plan to showcase DigiTwin at subsequent events.

Outcomes to date

Thanks to the work on DigiTwin, our HMI team was able to deepen its knowledge and experience in areas such as Unreal Engine 5, Azure, IoT, and the creation of digital twins. Now we will utilise the knowledge to develop the initiative further and launch commercial projects. What’s more, our stand at Car HMI received an extremely positive reception and encouraging feedback from the event’s participants. And still, more is yet to come.

Are you interested in digital twin software? Learn more about this solution here.

Meet the DigiTwin and experts standing behind during the Discovery Day 2023 by PRN

On November 16, 2023, we will be present at the Discovery Day 2023 event held by the Pittsburgh Robotics Network at David L. Lawrence Convention Center. If you would like to check the DigiTwin in action, talk about project implementation or learn about the HMI opportunities for your company, reach us at the event or leave a message via the form below.