Introduction
In a world where technology is constantly striving for innovation, the integration of mobile applications and sensor hardware has taken a big step forward. This blog presents the fascinating journey of capturing omnidirectional motion using sensors, especially those connected to wrist – worn devices, and seamlessly combining them with mobile experiences created using Flutter.
We believe that new advances in technology should be available to everyone equally, regardless of their personal or physiological ability. Although existing human-computer interaction models are effective, they often lack the intuitive and natural processes that lead to good communication.
Persons with disabilities may face challenges in keeping pace with technological advancements due to the lack of accessibility features in current electronic devices. This problem is particularly prevalent within the speech-impaired and hard-of-hearing communities, which is deeply concerning. It’s imperative that we all have the capability to express ourselves in a manner that ensures universal understanding.
Typically, sign language is practiced as a way around this problem, but it has certain limitations of its own. There are multiple sign languages practiced around the world, and are generally not mutually intelligible. Moreover, most of the population is not familiar with any sign language.
Enter Glovatrix, a startup that firmly believes in their responsibility to use their IoT expertise for the betterment of society. Their goal was to create a system that captures the hand movements in sign language, and turns them into a universal language that goes beyond traditional communication.
This led to the birth of ‘Fifth Sense’, an innovative system designed to seamlessly bridge this gap in ability. We at CoReCo Technologies proudly joined hands with Glovatrix to extend the necessary technical support to bring the system to reality through digital transformation.
Our efforts are not to create new technology, but to redefine the way we interact with technology to make communication more efficient and useful.
Typical Challenges
Before we could bring Glovatrix’s vision to fruition, we had to deal with or mitigate some challenges fundamental to this use-case:
- Real time or near real-time functionality
- Accurate and precise capture of subtle hand movements
- Recognition of minute variations in gestures due to users’ physical abilities, behaviour, cultural differences, adapting to various contexts and languages
- Privacy and security are paramount. Robust data protection is necessary to safeguard users’ personal information once captured.
The Fifth Sense System
The Hardware:
Sign language is a combination of macro hand movements and a series of palm and finger formations. To detect these, 6 sensors are used for each hand – one sensor at each of the 5 fingertips, and the last one to capture palm movements.
These sensors all link up to a device on the wrist, worn like a watch. This sends the gesture data to a mobile app via Bluetooth. This is the core of the Fifth Sense System.
The sensors comprise of a set of accelerometers, gyroscopes, and magnetometers to create a dynamic 3D representation of the user’s movements, speed, and energy. The system measures a body’s specific force, angular rate and the orientation of the body and presents three-dimensional vector parameters in a digitized analogue format.
The whole sensor system is integrated into a glove, one on each hand. This makes it an ultra-convenient and accurate accessory.
The Software:
The next step is the mobile application. The app takes the gesture data from the sensor-infused gloves and sends this data to the cloud for further processing.
We used Flutter to build the app, offering cross platform support. Flutter’s ability to create visually appealing and interactive responses enhances the user experience, making interactions not only functional, but beautiful.
With every new gesture that Fifth Sense users perform, the app retains those gesture sequences as a library. These sequences are then appropriately labelled as per the user’s personalised interpretations. This ensures a fast response, important for efficient use. The data sets are sent to the cloud server via a secure POST request.
Complex algorithms are used on the server side to process the data trails of the movement parameters received from the app. The gestures so received are converted into meaningful text or words. The algorithm is constantly improved through machine learning, adapting to the user’s specific movements with improving accuracy over time.
The server’s response is then sent back to the mobile application, where it shows up as a transcription on the screen and can be played as audio feedback. The entire process, from gesture to translation, takes mere milliseconds and creates an almost real-time interaction.
The IoT ecosystem, along with embedded system design and development techniques has played a vital role in overcoming the critical challenges:
- Accelerating the process to get Real Time response was a challenge. It is handled by applying data compression techniques, optimised data packaging, and transfer protocols with efficient data processing algorithms.
- Facilitating the device-app, app-cloud, and cloud-app data transfers with a high level of accuracy and reliability was another challenge. There was the possibility of data losses and data overload, which could result in delayed or incorrect responses and system crashes. This too was handled by the application of data optimisations and having adequate integrity checks at appropriate stages.
- The requirement to handle a wide range of user scenarios is catered to by strong machine learning techniques for personalised user experiences. This depends on the users’ physiological and behavioural parameters.
Conclusion
The integration of gesture sensing devices with mobile applications effectively represents the evolution of human-computer interaction.
Our journey has taken us from the challenge of connecting communication to creating a communication system that not only facilitates communication but does it the right way – with intelligent learning, engagement, and safety.
The Fifth Sense System and other solutions like it could have a variety of applications besides accessibility. From exciting gaming experiences to gesture controlled medical equipment for surgeons, the possibilities are endless.
As we stand at the crossroads of innovation in hardware and elegance in software, the impact of wearable devices for gesture capture, and the use of apps for interpretation continues beyond the world of technology. It supports the lives of people with communication issues and opens the door to a world of connection and greater understanding.
At CoReCo Technologies, our focus lies in utilizing technology to solve real-world issues and add value to end-users. Throughout the solutioning phase, our primary focus remains on problem-solving rather than the technology itself. For us, technology is a means to an end, not the final goal. Additionally, we go the extra mile to find optimal solutions within the given constraints such as cost and time.
As of January 2024, we have served 60+ global customers with 100+ digital transformation projects successfully executed. For more details, please visit us at www.corecotechnologies.com or write to us at [email protected].
Saifuddin Shaikh – Software Engineer
and
Suhas Patil – Director, Embedded Systems
CoReCo Technologies Private Limited
