Embedded UI Testing Framework

Computer Vision and Machine Learning based Test Automation

Orion Embedded UI Testing Framework, a Computer Vision and Machine Learning based Test Automation Framework, helps automate user interface testing for embedded devices. Embedded devices are typically either tested manually or require a custom-built automatic test framework created from scratch. This is because embedded devices have different interfaces and operation systems. Even if automatic testing is possible, it usually requires test-specific modifications for the firmware, which slightly lowers validity of the test. Our solution collects information from devices using a camera and microphone. It can be used with any device, and no firmware modification is required. Orion Embedded UI Testing Framework:

• Allows automated testing for any embedded device with a user interface
• Removes the need for test-specific firmware changes
• Enables tests to be written using standard frameworks instead of creating solutions from scratch
• Optimizes factory production

Features

• Icon Recognition. Based on OpenCV algorithms, it can find any of the pictures that the test admin submitted to the “library.” Any picture could actually be treated as an icon: buttons, toolbars, LEDs, etc.
• Text Recognition. Responsible for finding and recognizing the texts present in the picture and based on 2 pre-trained neural networks. The first stage network performs picture segmentations to find and separate text areas in the picture, and the second stage network resolves those areas into lines of text.
• Sound Recognition. Uses audio fingerprinting to create a “fingerprint” of an audio track that testers expect to find. Then listens to the microphone to identify these “fingerprints.”
• API. Simple interface that encapsulates all the logic and allows tester to use recognition in the tests.
• Auto-test Framework. Testers can use to write and run tests. In our development, we use Robot Framework, as it’s most commonly used for embedded device testing. But, in reality, an API could be adapted for any framework (e.g. Appium, Selenium, etc).
• Setup Helper Application. A cross-platform PC app that helps the tester establish the testing setup and configure the camera for best recognition quality. This app helps correct camera angles, sharpness, brightness, etc. Its job is to correct any perspective distortions and ensure best possible recognition.

Benefits

• Universality. It does not matter what OS runs on the device, what interfaces it has or whether it has any unused memory – the camera and microphone can take a picture of virtually any device.
• Non-invasiveness. No changes of the firmware required. The testing will be 100% comparable to the manual testing.
• Tester-friendliness. The system provides a simple but powerful API that can be integrated with any popular auto-test framework. It’s not required that testers learn specific framework or language or create a custom solution of their own. And our system includes a helper application that enables the tester to create a testing environment and configure the camera. 

Possible applications

• Automate manual testing. If you develop a new embedded product or if you have an existing one that you test manually, our framework can be used to automate the testing. 
• Enhance your CI/CD setup. The framework can be easily integrated into your CI/CD setup to speed up the development process and improve quality and time to market.
• Automated smoke testing. Automating most basic user scenarios to create smoke tests is usually the best way to start automating your testing. Our framework makes it easy to create a setup that will be running these scenarios automatically on a daily basis.
• Stability testing. One thing that is challenging to check manually is overall system stability, as no tester would want to run the same scenario over and over hundreds and thousands of times. Our framework makes it possible to automate this process and ensure your device works stable and does not have memory leaks.
• Factory Testing Automation. Since our solution is not invasive, it can check the devices that come from the factory belt. The system ensures that the screen shows the correct colors and doesn’t have broken pixels, LEDs work as expected, and sounds are played with no distortions, speeding up production, reducing costs, and improving product quality.

Additional Services

• Integration. Our team can help you create a test automation setup for your product from scratch and incorporate it into your CI/CD pipeline. If you already have a test automation setup, we can enhance it with new tests to reduce manual testing. 
• Customization. Every embedded product is different, making automating testing a unique process. We can modify our framework and enhance it with new features to adapt to your products and their specific needs. 
• Maintenance. We will help you maintain your testing setup, fix issues, and enhance it with new tests.