Computer Vision and Machine Learning based Test Automation

Orion Embedded UI Testing Framework, a Computer Vision and Machine Learning based Test Automation Framework, helps automate user interface testing for embedded devices. Embedded devices are typically either tested manually or require a custom-built automatic test framework created from scratch. This is because embedded devices have different interfaces and operation systems. Even if automatic testing is possible, it usually requires test-specific modifications for the firmware, which slightly lowers validity of the test. Our solution collects information from devices using a camera and microphone. It can be used with any device, and no firmware modification is required. Orion Embedded UI Testing Framework:

  • Allows automated testing for any embedded device with a user interface
  • Removes the need for test-specific firmware changes
  • Enables tests to be written using standard frameworks instead of creating solutions from scratch
  • Optimizes factory production


  • Icon Recognition. Based on OpenCV algorithms, it can find any of the pictures that the test admin submitted to the “library.” Any picture could actually be treated as an icon: buttons, toolbars, LEDs, etc.
  • Text Recognition. Responsible for finding and recognizing the texts present in the picture and based on 2 pre-trained neural networks. The first stage network performs picture segmentations to find and separate text areas in the picture, and the second stage network resolves those areas into lines of text.
  • Sound Recognition. Uses audio fingerprinting to create a “fingerprint” of an audio track that testers expect to find. Then listens to the microphone to identify these “fingerprints.”
  • API. Simple interface that encapsulates all the logic and allows tester to use recognition in the tests.
  • Auto-test Framework. Testers can use to write and run tests. In our development, we use Robot Framework, as it’s most commonly used for embedded device testing. But, in reality, an API could be adapted for any framework (e.g. Appium, Selenium, etc).
  • Setup Helper Application. A cross-platform PC app that helps the tester establish the testing setup and configure the camera for best recognition quality. This app helps correct camera angles, sharpness, brightness, etc. Its job is to correct any perspective distortions and ensure best possible recognition.


  • Universality. It does not matter what OS runs on the device, what interfaces it has or whether it has any unused memory – the camera and microphone can take a picture of virtually any device.
  • Non-invasiveness. No changes of the firmware required. The testing will be 100% comparable to the manual testing.
  • Tester-friendliness. The system provides a simple but powerful API that can be integrated with any popular auto-test framework. It’s not required that testers learn specific framework or language or create a custom solution of their own. And our system includes a helper application that enables the tester to create a testing environment and configure the camera. 
  • Factory Testing Automation. Since our solution is not invasive, it can check the devices that come from the factory belt. The system can ensure that the screen shows the correct colors, doesn’t have broken pixels, LEDs work as expected and plays sounds with no distortions, speeding up the production, reducing costs and improving production quality.

Additional Services

  • Integration. Deployment and integration in your environment.
  • Customization. Development of new or adaptation of existing features to fit your needs.
  • Maintenance. Support of deployed solution in case of found defects.