Signature-based detection of behavioural deviations in flight simulators - Experiments on FlightGear and JSBSim

Flight simulators are systems composed of numerous off-the-shelf components that allow pilots and maintenance crew to prepare for common and emergency flight procedures for a given aircraft model. A simulator must follow severe safety specifications to guarantee correct behaviour and requires an extensive series of prolonged manual tests to identify bugs or safety issues. In order to reduce the time required to test a new simulator version, this paper presents rule-based models able to automatically identify unexpected behaviour (deviations). The models represent signature trends in the behaviour of a successful simulator version that are compared to the behaviour of a new simulator version. Empirical analysis on nine types of injected faults in the popular FlightGear and JSBSim open source simulators shows that our approach does not miss any deviating behaviour considering faults which change the flight environment, and that we are able to find all the injected deviations in 4 out 7 functional faults and 75% of the deviations in 2 other faults.