(Standing from left) Tan Kian Ann, Business Development Manager, HelloHolo; A/Prof Jeannie Lee, Deputy Director, Centre for Immersification, SIT; Kee Cheng Heng, Managing Director, HelloHolo; Samuel Lee, Deputy CTO, HOPE Technik.

(Seated from left) A/Prof Alfred Tan, Programme Leader, SIT-University of Glasgow Mechanical Engineering; Asst Prof Kyrin Jo Liong, SIT.

(Team members not pictured) Asst Prof Vivek Balachandran, Programme Leader, ICT (Information Security), SIT; Emran Poh, Research Engineer, SIT. Photo credit: A/Prof Jeannie Lee

During a casual chat in the faculty lunchroom, the conversation drifted to the long working hours of design engineers. When engineers create and assemble large, heavy prototypes on the workshop floor, the work is often laborious, time-consuming, and dangerous.

The topic sparked an idea in Asst Professor Kyrin Liong’s mind. What if the design engineers could use holographic prototypes to optimise the designs before any physical prototyping and assembly takes place? Aside from being faster and safer, it would also be more cost effective.

Assembling an Interdisciplinary Team

From left: Tan Kian Ann, Business Development Manager, HelloHolo; Asst Prof Kyrin Liong, SIT; Emran Poh, Research Engineer, SIT. Photo credit: Asst Prof Kyrin Liong

To get her idea off the ground, Asst Prof Liong needed a dream team, which she assembled from the Engineering and ICT academic clusters. Collaborations among SIT faculty are de rigueur at the university, so the team had already worked together previously on research projects and co-teaching arrangements.

A/Prof Jeannie Lee, who is the Deputy Director of SIT’s Centre for Immersification, led the overall development and software framework for the mixed reality application. She also led the research investigation on the best user interface methods.

“The combination of knowledge and expertise from different domains allows for different perspectives and creative solutions and applications. It is important to have an interdisciplinary team working together,” said A/Prof Lee.

In addition to faculty staff, Integrative Team Project (ITP) student teams as well as a bevy of student assistants from SIT contributed to different aspects of the project at various stages.

SIT-Industry Collaboration

Also on board were industry collaborators from HOPE Technik Pte Ltd and HelloHolo Pte Ltd. While HOPE Technik is a long-time collaborator of SIT, HelloHolo was recently approached to co-teach the Engineering Design Graphics module for the SIT-UofG Mechanical Engineering programme.

HOPE Technik’s design engineers served as the project’s domain experts. They advised SIT faculty on possible use cases and offered access to their workshop floor and garage to perform user testing.

Meanwhile, HelloHolo co-developed the app, lending their knowledge and expertise in creating robust and efficient mixed reality applications.

Asst Prof Liong said, “Working with industry partners enables faculty to have first-hand experience of industry perspective, which complements our theoretical knowledge and background. Together, we can synergistically build a new trajectory that benefits both industry and academic partners.”

How the Mixed Reality App Works

The 4-tonne ordnance loader prototype created by HOPE Technik. Photo credit: Asst Prof Kyrin Liong

The test case created by HOPE Technik was a 4-tonne ordnance loader (OL) sized 3.5 metres long and 1.5 metres wide, with an expandable height of 2 metres. It is easy to imagine how physical assembly testing of a prototype this size would pose significant challenges.

Using the MR application, engineers can load the digital models within the application and test the feasibility of the assembly process of the OL deployed via Microsoft HoloLens 2. Design engineers can safely navigate assembly issues prior to handling the physical prototypes.

Samuel Lee, Deputy CTO, HOPE Technik demonstrating how the mixed reality app works with Microsoft HoloLens 2. Photo credit: A/Prof Jeannie Lee

Once the design reaches a more stable stage, engineers can start building the physical prototype. By this time, the assembly would have been through multiple iterations on a 1:1 scale. Any degree of design refinement necessary at this stage would be less onerous.

Developing the Application

Commented SIT research engineer Emran Poh, who led the technical development of the MR software based on the identified software features, “Due to the multidisciplinary nature of this project, we needed to understand the problems faced by the design engineers.”

First, the team created ‘user stories’ or features to be developed before further dividing them into specific tasks. Next, the team identified the most suitable mixed reality hardware devices and software development toolkits plug-ins. Finally, the team developed the app using Microsoft’s Mixed Reality Toolkit (MRTK) on Microsoft HoloLens 2.

The 3D model of the prototype had to be converted graphically using engineering software SolidWorks and then rendered within a mixed reality environment. Then, the team developed the features and user interface for design engineers. 

Testing on the Workshop Floor

Testing the OL assembly on the workshop floor using the MR app.

The team conducted playtests together with design engineers on the workshop floor to further evaluate the usability and reception of the application.

Users were able to view the components from various angles as well as ‘grasp’ and ‘move’ the various mechanical parts, and the display also included selection options within the mixed reality app.

A/Prof Lee said, “We completed the first software prototype within 6 to 9 months. However, we iterated on the existing software base, so we can create subsequent versions with additional features and user interface methods e.g., visualising virtual mechanical assemblies, display of Finite Element Analysis (FEA) graphics, and optimisations.”

“The main challenge was the conversion and optimisation of the 3D model from engineering design software into mixed reality. We had to achieve a balance between a smooth frame rate (to avoid motion sickness) and a high resolution, so the visualisations serve the user’s needs,” she explained. 

Asst Prof Vivek Balachandran said, “Visualisation of mechanical parts in virtual reality provides a safe environment for training new engineers who need to manage these heavy devices. The app also provides visual access to otherwise inaccessible areas, giving the user a deeper understanding of the mechanics of these parts.”

Potential Real-world Applications

Any industry that requires design engineering will be able to benefit from the MR application. Engineering design firms that need to test assembly process feasibility can use this app to save resources.

A/Prof Alfred Tan said that a basic digital platform can be developed so that various industries can customise it to serve more specific and unique needs.

While it is still early days, the app can be refined to further integrate with the laws of physics for realistic real-world response. That way, the app will serve a greater purpose than simply fitting pieces together. Instead, it can be used to test the weight-bearing properties of structures or identify stress points in the various components, thereby enhancing accuracy and the safety of the users.

“The application can provide visualisation and training in different engineering domains that require structured steps for performing a task. In the future, we could include remote collaboration and training features to further improve training and knowledge transfer,” said A/Prof Lee.

A/Prof Tan also elaborated on the potential use of the MR app as an educational tool. He said, “It will benefit students who need extra help in visualising how components come together. The tricky part is how to devise a way to allow images seen by the user to be viewed by a larger audience group simultaneously.”

The challenge is to create a steady image that can be viewed comfortably by an audience. Currently, the visualisations produced via the Hololens while the user is moving around can be disorienting when viewed by an external party, and even cause motion sickness. A potential solution is to create a suitable tracking rail or slider for a camera that can capture multiple angles.

Growing Applied Research at SIT

This project was funded through SIT’s quarterly Joint Ignition Grant, which provides a quantum of up to S$250,000. The team started work in October 2019 and the project will be finalised in July 2022.

The team from HOPE Technik said, "Working on this grant with SIT was a pleasure. It was organised, well-paced, and the team delivered according to plan.”

Kee Cheng Heng, Managing Director of HelloHolo added, “The partnership between SIT and HOPE has created a platform that allows us to explore how immersive media technologies can be applied in new and innovative ways. The research done on Finite Element Analysis (FEA), together with the ability of HoloLens 2 to visualise point cloud data, allows us to revolutionise and streamline the way engineering and manufacturing can be conducted.”