To make sure that Autónomo works as expected, we tested the Alpha release under specific conditions and environments using the Unity Test Runner. There are two different testing environments for our software - the Unity Editor and the Oculus Quest headset. Most of our profiling and unit tests, however, were carried out on the Unity Editor. There are a couple reasons for this decision: first, the process of loading the game onto the Quest headset is very time-consuming and tough to automate. Secondly, the headset does not have an in-built Unity debugger, unlike the editor, so when an error occurs, one can not see the error message or any other details about the error, unless we display the message on a game object (like a Canvas).
We used profiling techniques to gain insight into how Autónomo is performing, then isolated and eliminated problems and bottlenecks. Due to time constraints, this could only be done at a rudimentary level. However, engineers at NTT Data will be further imporving and optimising Autónomo after the handover.
Our Performance Targets:
We used the Unity Profiler to understand per-frame CPU and GPU performance metrics and to help identify bottlenecks. Similarly, we used the Unity Editor to see rendering statistics (such as FPS, Draw Calls, etc.) for our most important scenes. Alongside this, we also used the Unity Frame Debugger how Unity to debug pipeline problems. Ultimately, our Beta release met the performance targets for most of our scenes.
We wrote a wide range of unit tests using the Unity Test Framework to find and diagnose errors in (but not limited to) individual Unity prefabs,
scripts (Traffic Light Control, for example), and methods (Navigator.FindShortestRoute() - our pathfinding algorithm, for example).
Tests were created iteratively — as more functionalities were added — and code coverage (although tough to exactly calculate for a Unity application) was perceived to be high by our project partners.
Although Autónomo's journey is presented as a (traditional) waterfall model, its development model was in fact an iterative one. After an initial planning, resource estimation, and risk analysis phase, we repeated each development stage multiple times, with each subsequent completion of the cycle incrementally improving the software. This helped us quickly recognise and implement enhancements — giving us the ability to rapidly adapt to the needs of the project and our partners — while providing us a swifter turnaround after each iteration.