Lately I got the feeling I am finding more issues while ‘just tooling around’ then through following my test scripts to the letter. The used test scripts do not offer much new insights besides the ones for regression purposes. Of course scripts for new functionality will lead to new findings but after fixing those, not so much. Still, writing down extensive scripts for my tests during our sprint seems like overkill to me. I know my app, I know what it should and shouldn’t do, I don’t need scripted tests for everything. So, I am sort of freewheeling a lot.
Since exploratory testing is sometimes thought of as ‘just clicking around’ and used as an excuse not to have test documentation (somewhat guilty on both charges), I wanted to try out something I read about in an article on exploratory testing from the Dojo (Ministry of Test): the PQIP approach. In the article Simon Tomes talks about how he uses this approach to document his exploratory test sessions.
When using the PQIP approach you basically group what you find during a testing session into one of four categories: Problems, Questions, Ideas, or Praise.
Write down what you found, with reproduction path. This problem can be discussed to see if it needs to be fixed right away.
Ask and write down questions about how things work, or how to break something. This helps you understand what has been build and why things work in a certain way. Also, a question might lead to a found problem, or a solved problem.
When you are exploring your app, you might encounter things that do not feel right with respect to for example user experience. Write this down and discuss to help make your app better.
When you find things you like in the app, write it down and let the developers know. It feels nice to give compliments.
I liked this idea so a few weeks back I asked a developer to join me in an exploratory testing session, to try and ‘lose’ a fingerprint we need in a new feature (creating situations in which the device fingerprint is not or no longer recognised by our app). I told him I wanted to document the session using the PQIP approach. He responded enthusiastically, so we went at it the following morning at 10.
During our one hour session we found several problems and improvements, answered many questions, and saw some great solutions the developer had built on the specific feature. We used old fashioned fountain pen and paper to document during the session. We lost the app’s fingerprint once, when we added an Android system fingerprint (yes, this sounds weird) so that goal was achieved as well. All in all the vibe and energy during the session was great. The developer saw things he didn’t immediately think about and I learned things about how to better break stuff. Together we thought of things to improve the tested functionality as well as current functionality.
In conclusion we can speak of a great session result! The PQIP approach is a good way of documenting exploratory test sessions. I think I spend an hour and a half afterwards on documenting the findings in Jira, this was including reproducing the found problems and ideas. Next time I have to write down reproduction paths better during the session; my notes were OK for me, but did not include a complete and clear reproduction path for the developer.
(Exploratory) testing sessions with other team members can lead to interesting things that will improve the quality of your app. New, or different, insights in how software works can make a difference for the user’s experience. With the entire team (including the PO) you will have to decide which improvements will be implemented.
For exploratory testing it is important to have a goal, e.g. try to find circumstances in which we can lose the fingerprint used for login in our app. Agree upon a suitable time-box to try and reach this goal, e.g. one hour. Try to not get distracted too much by what you find along the way, and write things down that are not directly a necessity to achieve your goal. These can be things that need more investigation or are related to another piece of functionality. Furthermore, try to think out of the box and have some fun while doing the session.