We recently released Dashman 1.0.0 beta 3 and the reception was, well, embarrassing. There were a lot of issues that people run into that made Dashman unusable for them. We use Sentry to collect error reports and we got plenty of reports. That was very useful and we managed to tackle most of the in a short time. Once we fixed almost every single error we encountered, we decided to create a comprehensive manual test suite (Dashman testing is particularly hard to automate) so as not to have any further regressions in the future. To achieve this we are using Test Management for Jira. The result was surprising and also embarrassing.
These are our initial set of tests, which we know is not absolutely complete but it touches almost every single feature of Dashman:
The ones with a green dot are the tests that were successful. The ones with red or blue are the ones that failed. Ouch! These tests have an average of 12 steps each and red means one step failed, even if it was trivial to workaround it. It’s not a complete disaster but it’s much worse than we expected. Out of this meticulous testing we created 14 different issues that we are going to fix in the upcoming weeks and they will be part of the next planned release, Dashman 1.0.0 beta 4. Fear not! The new version will be much better.