QtWS15- Are you testing enough - Qt application Quality Assurance, Harri Porten, froglogic
Qt World Summit 2015, OCTOBER 5-7, BERLIN, GERMANY
Are you testing enough? Qt application Quality Assurance:
Have we fixed all bugs in our software? How many bugs are present in the first place? Even the latter question can be very hard to answer.
Every non-trivial Qt application includes various UI controls and internal elements that can all have different states. Taking into account the asynchronous nature of events, threads, and even slot invocations, the number of possible combinations explodes quickly.
Often it is neither economically acceptable nor technically feasible to test all combinations. Which ones to focus on? How many bugs may we miss by limiting our efforts?
A popular metric for measuring the quality of our testing is code coverage. The most commonly used levels will be depicted and compared. Special attention will be paid to Qt-specifics like code generated by the Meta Object Compiler moc and related macros. And what about a user interface declared in QML? We will evaluate the idea of UI coverage.
We will also look at approaches like "monkey testing" and "fuzzing" that promise to achieve a deeper test penetration without much extra effort. Or methodologies like Pairwise Testing that help to reduce the number of combinations to test. Again, we will examine how well those techniques apply to a Qt GUI.
For security critical applications, the quality of Qt itself needs to be taken into account. An overview of the development process including change reviews, unit tests and Continuous Integration (CI) will be given.