Introduction to Exploratory Testing

Hi, everyone. My name is Brittany Stewart. I'm a senior QA specialist from Konabos. Today, I'm excited to share with you guys about introduction to exploratory testing. Now, this is a 30 minute session or just about. But this is an extraction from a bigger presentation on a full day workshop that was done. So my aim here is to share with you as much as I can today about the world of exploratory testing. And so on that note, let's begin. Before we begin these presentations, I just like to encourage everyone to listen, and to relax and enjoy the session. But the major goals for this session is that by the end of this learning session, you should understand the concepts of exploratory testing, be familiar with some of the key principles. So that's like understanding what, where, when, and how and why to use exploratory testing, you'll get an understanding of exploratory testing process, and be exposed to some of the techniques and strategies. So this is something that I've been doing for the last six years. exploratory testing is something that all of us do in some shape or form, but I don't think everybody knows it yet. So let me go through and share with you guys what to have. And hopefully you'll learn a lot from this session too. So for section one, we're gonna go through the fundamentals of exploratory testing. So what exactly is exploratory testing? Well, exploratory testing is an approach to software testing that is described as the simultaneous learning test design and test execution. But I do like the definition by Elizabeth Hendrickson, which are quote here. So she says, It's simultaneously designing, executing tests, and learning about the system using your insights about the last experiment to inform the next. So that's just a high level definition. In the testing space, there's a lot of discussion about which definitions are right. And you know, the best way to explore it, or sorry, in the testing community, there are discussions about the best way to explain it. And so this definition is the one that I like to reference when we start talking about exploratory testing. Now, like other testing techniques, exploratory testing has its pros and its cons. So we're gonna get into a bit of that. But before I do, for those of you who are probably not that familiar with testing, I want to explain some differences between exploratory testing, as well as the other types of testing that are out there that we normally use. So in this slide, I have three testing types. On one end, we have our scripted tests, which most persons are used to. So this scripted tests are the test ones are where we write our test scripts, whether they're automated or nots, we write test cases, and they're designed ahead of time, they're usually reviewed by the team. And then feedback is given at the end of executing these test cases. So with that type of testing, feedback is usually slow execution for this one, as also usually follows like a step by step instruction approach. And we're normally clear about what the expected results are. I also have a note here that this can be manual or automated, and that is so for expert for the scripted test, on the total opposite end of scripted testing is ad hoc tests. And I think a lot of people do confuse ad hoc tests with exploratory testing. So this is just to clear that up here are no ad hoc tests do not have any test cases, there's no strategy. There is no sharing of findings, no documentation, and it cannot be automated. Now, from my experience, exploratory testing falls right in the middle of these two approaches. It's the sweet spot where the process of creating and running tests is simultaneous as we said in the previous slide, but with this type of process, it allows us to give rapid and valuable feedback to the team. Test results are unpredictable which is expect Tip on predictability. And then it enhances our personal freedom as a tester to be able to execute as we see fit. And then this may be automated, but that's something that we will talk about a little later on in the session as well. Now, there are many different types of exploratory testing, but I've included these top three. So you have the freestyle approach, which I think a lot of people do. It doesn't include any rules nor approaches nor specifications, it's just like at the beginning of a project, you're just going through, and you don't know anything, it's more of a learning situation. And then you have the scenario based approach, which is based on real user stories most times, or we use end to end scenarios. But the main thing here is to base or testing on a specific user, user persona, or per user scenario, to gather testing. And then the last one is strategy based testing. So this one is more performed by an experienced tester, as I have noted here, because this one includes different types of testing methodology, methodologies, and strategies that we use to enhance or exploratory testing experience.

So with the three definitions that are really the three types of exploratory testing that I just shared, I like to ask the question, you know, do you think you've already been doing exploratory testing? And I would like it, if you think about it, are you doing more of an ad hoc style right now, based on the information I gave? Or are you doing more scripted? And there's nothing in between? can pick up about that, and let me know as well. So the next part is, when do you actually use exploratory testing, I know a lot of people are confused about this one. So I've designed a slide to highlight the areas where I believe it's most important to use it. Firstly, in an agile environment where our Sprint's are short, and we have releases that are very frequent, it's important to, or you can use exploratory testing to help you in these instances. So Agile projects definitely do benefit from exploratory testing, because we're able to provide fast feedback. Secondly, when you need to learn about a product functionality or customer requirements, so whenever you start a new project, or corner boss, you know, it's a lot of information to take it and you need to start somewhere. So exploratory testing is usually my go to approach to start. It helps me to just learn about product functionality. I do look at requirements, meet my assumptions, question things, because I'm very curious. So exploratory testing allows us to do that you learn about the product, you can review the customer requirements as well. Thirdly, when requirements are unclear, or non existent, and, you know, that happens a lot. In this time, we have clients sometimes that are not clear on what exactly they want. So exploratory testing is one of those approaches that you can pull on to help to start somewhere and start getting clarity on requirements if there are none in place. And then, as I said, before you can provide it allows you to provide faster feedback. So in the scenarios where I get a new build on stage, for example, or the testing environment, I will make sure to do an exploratory tests before I ever go into my sanity or smoke tests. Just to be able to do a quick run through as I say, Here, you have Quicken a lot of how things are looking on the build, is it stable enough, and then I will go into the detail testing. And then lastly, when you want to diversify the testing process, and this is after you've completed your scripted testing, so we can only do so much with our scripted tests, they stop and usually when they stopped, that's where you have to stop. But exploratory testing gives us more diversity in terms of our testing. And this is one of the reasons I love it, especially when our team has automated regression suites in place. And so automation is there to support testing, it leaves more time on the testers QA times to be able to fully explore the system. So these are just some key moments that you can pull on using exploratory testing. So during my years of doing this There have been some challenges that I've faced. And I've also read about. And so in this next slide, I have proposed some solutions to exploratory testing. So let's take a look. First challenge that I have experienced and heard of is lack of traceability. And that's just due to the nature of it. exploratory testing doesn't require much planning. Because you know, there's no predefined test plan or strategy or test scripts. But to overcome this challenge, what I found helpful is to log the test as as and when it's executed, and I share my progress with the team. I'm going to talk about this a little bit more further down, you'll see the process. But that is a solution to the traceability, it can be tracked. Secondly, lack of product knowledge. So when people start out with exploratory testing, they're like, you know, I don't know, I'm not really sure what to do. And that's okay, you're not really supposed to know exploratory testing does allow us opportunity to learn, while we're tests while we're designing our test, and while we're executing. So the solution for this part is to include another skill tester with you in the exploratory testing session, or leverage pair testing. And that's just pulling on somebody else on the team who may know more about the product, and you get another call with them, and do it a session together. And then lastly, not knowing when or how to adapt exploratory testing. So to do this, it's best to leverage a test method to find or find bugs in the software when time is limited. As I said, in an agile environment, when we're trying to move fast, we can pull on exploratory testing. So these are some challenges. But we have solutions. And I'm going to show you exactly how I do this shortly. I included this slide. Because I love the image. It basically summarizes all that I've said, just know about why do we do exploratory testing. So firstly, it ensures faster testing. It does allow us to deliver feedback at a faster pace. And so with that, it supports our agile environments. It allows us to discover unexpected defects. And it provides real time thinking, which is lovely. And then enables us to do real time learning, which is also lovely for somebody who is very curious about what you're testing. It requires less preparation. So that's also a big plus, we can get started faster. And so with all of that it saves us time.

Find more books, discover the unknown unknowns. So I don't know if anybody's familiar with this quote, but I love it because it in testing, it helps to understand why exploratory testing is important. So I'm gonna read it, it says there are known knowns, there are things we know, we know. We also know that there are known unknowns, and that is to say, we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know, we don't know. And so in testing, basically, why exploratory testing is important is it helps us to understand each of these areas. I'm going to proceed and let me tell you why. Because I fall in this very cool quadrant to explain this. When we're testing risk. Thinking about risk risk classification is a very important part of what we're doing. There are always issues that we have to be able to say, Okay, this one is needs to get fixed first, or this one needs to be pushed to the next release and things like that. So this quadrant kind of breaks down the known unknowns, unknowns on those details. But I won't go into too much about it. What I wanted to highlight here is the other types of scripted tests allow us to find the known unknowns, the unknown knowns, what we're exploratory testing comes in, and is most beneficial is when there are unknown unknowns, and those are the things that it is impossible for us to, to know or to predict. You know, what I found is that some of the nastiest bugs hide in these areas in the unknown, unknown areas, and usually customers find those before QA does, or, you know, like a big visited thing happens and it's just because we weren't looking in that area you wouldn't have known to look so. The question is, you know, how can we put I simply catch the bugs that live in this unknown unknown area. And, from my experience exploratory testing is one of the best ways to do that, because it gives us the freedom to look without judgment, you know. So let me move on to the next one. In the testing world, and this is something I experienced a lot myself is when we're doing our scripted tests, and we have to run the same tests every day. If we don't have automation, for example, and you're doing meaning manual testing, it's really easy for us to miss things. And this is not our fault on any one person. It's just human nature. It's called inattentional blindness or observational blindness. I've also seen it listed us. And so it's easy for us to miss things. Just because we get fixated on a particular target, or we're so fixated on one thing that we overlook information in another place. And exploratory testing can help us to overcome that I have included this video because it brings awareness to the fact that, you know, what I'm saying is that we are only human, and we can only notice so much. So I'm gonna play this video. Hopefully it can work. And we'll see demonstration of you know how easy it is for us to miss things?

Okay, I'm not able to play the video. But that's okay. Essentially, whenever we're looking at something for too long, it's only natural for us to miss certain things. And so I just want anybody who is out there watching, you know, if you've ever missed a bug that got lost in production, or something of that nature, it's okay. It's completely normal, we're looking at a lot. But exploratory testing can help us to break out of this, and to overcome this challenge as testers. So the main purpose of this was, I wanted to let everyone know that you can do exploratory testing, people might find it hard to start or you're not really sure where to start. So this is going to be a quickstart guide to get you up and running in your exploratory testing sessions.

First and foremost, it's important to have a brain in the loop. And so the QA or whoever is doing it is very important. And to illustrate that, I've included a screenshot here of a bug that was found in Google Map some time ago, a long time ago, were a couple they were looking for directions from Sweden, Stockholm, Sweden yet and they were trying to get to Oslo, Norway. And this is the route that Google gave them, took them all around to London or the United Kingdom, and then brought them back to to Norway. Now I had to share this because I don't any scripted tests would have caught this or something like this in the beginning. Maybe visual regression, I'm not sure. But I really feel like only a human could pick up something like so like this could pick up a book like this. So if you've ever had to use Google Maps to lead you anywhere, and it gave you a new, weird routes. This is this is an example of that. But you see here that our exploratory testing, with the exploratory testing concept, you do need to have a human being involved in it. And so on that note, I want to highlight that with the brain in the loop mindset and how you frame your thoughts is very important before you start. It's important to be hover, like be very curious about what it is you are working on. So embrace that experimentation, mindset, and at Konabos. We like to say keep exploring I love that because that's the mindset that's the mindset that you need to have. So be curious you must want to find out you know, where's the source of truth for example, you It's important not to let the requirements will hold you ransom because sometimes they don't capture everything. So it's good to question it's good to be willing to explore. The other thing is leverage the Main knowledge. So I come from a design background. And I do pull on my, my web design knowledge to help me with testing. So I do focus on like the content. There are other people I know who have backgrounds in law and engineering, it's important for QA to understand that you can bring some of your knowledge and even if it's from a user perspective, it doesn't matter. But domain knowledge can really help you. With testing, especially exploratory testing. So make sure you didn't know domain knowledge is a part of it toolkits when it comes to this one. Personal freedom, my favorite part, because the scripted tests are so rigid, sometimes you have to try and stick to the exact steps. Being curious, you're I always want to click out and try different things. And so exploratory testing does offer you the personal freedom to be able to try some things and to see what the results are. And then lastly, you know, exploratory testers are usually very creative. Like I said, you think outside the box, you critically think you even think about the user's perspective, and you use that to guide your testing sessions. So mindset is very important. And without a test done in this process, it wouldn't be successful. So remember, your goals. If we're going on any, like anything that we're doing, we have to have a plan, we have to have a goal. And so I like to remind myself of these things before I start and exploratory testing. And so you can choose one of these, it doesn't have to be all three. But your goal could be to gain an understanding of how the application works, one, what interface looks like, what are the functionality that it has implemented. And this could be one of the goals. The other one is to force the software to exhibit all its capabilities. So you're really working to, you know, answer some hard questions and probably break it as people would say you want to try to break the software, that's fine, that could be a goal to. And then the last one is to find more bugs, you'd be exploring those edge cases, or as I said, the unknown unknowns, to see what is actually happening. Are there potential spots where bugs are being missed? The goal is very important, because it helps to prevent you from going into an ad hoc testing type of situation where you kind of aimlessly going through. But with this, you have a plan, you have a goal, you know where you're going, and you can stay focused with purpose and intent during the exploratory testing session. So I love to reference this diagram as a quickstart. Guide. It breaks it down pretty simple to me. So first thing you got to do when you do an exploratory testing is you have to have a plan as with any, like if you were going hiking or on on a trip, you would need to plan Additionally, you need to track then you need to capture some, you know, whatever your findings are. And then it's very important to share feedback, because it's.

So I'm gonna go into each of these one by one. First of all, when it comes to us starting in terms of the planning, in the software testing world, we use test charters to help us to plan our exploration. So what exactly is that test charter? Well, test charters are just statements that guide our testing. As I said before, you know, you want to make sure that you have a purpose, you have an intent for what you're doing. And so the first step is test charters. Questions that I like to ask myself at the beginning of the sessions are what am I exploring? If you are going on a hiking trip, for example, you'd also need to figure out what it is you need to bring with you do you need water in a flashlight similarly with this, what sources do I need? I'm going to give some examples shortly. And then I like to ask what kind of information am I hoping to find that's just to make a goal a little bit clearer as well. So test charters do help with our effectiveness and efficiency when we're doing this. But they can be very difficult to create or some people don't really know where to start and so I love to reference a little bit a little bit Elizabeth Hendrickson test charter formats here, where she breaks it down for us your a lot it says you explore the target and targets can be features or requirements or modules, for example, with the particular resource resource examples could be a tool for testing to a data set a particular technique, if certain configurations need to be on and off in a particular environment. So things like that, what do you need to help you to get the testing done? And then to discover information? What is it that you're looking for? Here are some examples, you could be looking at a feature, discover more about that it could be checking into the security of the application, you could be wanting to find out what's the performance like for this website? What was the usability for it? Is there any accessibility being consistent being considered in this. So these are some examples of information that you could be looking to discover just high level ideas to get you started. But feel free to deep dive into this. And then, as I said, like good. Writing Test charters is something that I'm still working on every day. But that did find some good examples of test charters. So I wanted to share with you good test starters usually offer direction without over specifying the test actions. Because when you get into the details like that, that's when you're going back more to descriptive tests, as opposed to being exploratory. So a good example of a test charter is one that's not too broad, and it's not too specific. Here's a good example, explore the checkout page, with each product type available to a new user to discover any issues with the checkout flow. So it's not too specific, it's not too broad. A bad example of a test charter would be something like this one where you have test the checkout process for an e commerce website x. That's definitely too broad, you're not really sure where you're going with that one. And then here's another one testing, test clicking a gift card to add it to the cart, and view what's on the confirmation page. So that one note sounds more like a test case, it's telling you exactly what to click where and the expected results. So this is an example of a bad charter. As you gain experience in writing charters, it does become easier to specify intention and to write these charters. But as I said, use the template, the Explorer with to discover template to start it does help to just get you in the right direction. Alright, the next point is tracking or exploratory testing. How do you do that? In the industry, best practice is to use session based testing. I'm not going to go into all the details here I have five parts that you could consider, but I'm going to focus on the time boxing, the review and the debrief part of this. So in terms of the time in terms of the time boxing, it's recommended right now that you spend 45 minutes or to 90 minutes max doing your exploratory testing sessions. So that's just setting out a timer, and making sure that you stop at the end. Because otherwise you end up two things can happen you end up going down rabbit holes, because when you testing is always see things things that are not within your goal or within your plan will pop up and it will catch you off guard and can you lose track of what it is you're actually trying to do. So the timer's session based approach helps you to stay focused. I mean, also it helps you to add this to your timesheets. This is something that I had a problem with in the past and people weren't sure. You know, what, how much time are you spending on this exploratory testing? Why is it taking so long things like that, when you track your time, this way, it makes it easier. Also, if you find yourself going over 90 minutes, you do run the chance of meeting upon observational blindness that I showed before where you can easily miss things. So it's always good to take a break. And then to come back and restart your testing sessions within this recommended time. Okay, so the next point is, how do you capture so when we talk about the review and the debriefing? How do you actually capture the exploratory testing notes? It's not as document sorry, it's not as detailed as the scripted test, but you can capture certain things and so I like to use these tips to guide me. When you're making notes for your exploratory testing sessions, you can capture things like comments, anything pause A tip that you see happening, positive feedback is always good. Any concerns that you had while you were doing the session, and it cleared problems that you saw, if there were any questions that came up, you should note those as well. If any suggestions and enhancements come to main, you note those and same thing for the ideas. To me, it's best to note everything. And then you come back to it in a debrief session to figure out what's most important to share with the team. Additionally, I love to use visuals to help. So if I'm using a mind map, or I'm using a regular note taking up, I do use these icons to represent the status of certain areas that I'm testing or how I feel about it. So this is just a guide to show you how you can possibly.

And then here's an example of an exploratory testing session that I did some time ago, this is just a regular notion note, board. But you see, I have my charter outline. I had details on the build, I made a lot of notes, and I even put it in a table. So the format of how you document doesn't really matter. But what matters is that you do try to capture as much information as you can with the guide of what I said before. Okay, so how do you know share your findings from exploratory testing with your team. This is a very important part of the process, otherwise, you don't really grow. And so to do this, this is my recommendation, you can share your findings file, this report, which is called that P Qi P report. Basically, though, it's just a report that highlights the praises the positives, the questions that you have any problems that you see, and if any ideas so you can narrow it down to just those four areas, capture those, share those points on it, maybe you have a blog or something, you can share it there any problem that you found. I've also found it helpful, share it in refinements or even or retrospectives, if there was anything that went well for from my praises section, you can include those if there are any problems, which are areas for improvement, you highlight those as well. So it's nothing too formal. It's just enough for auditing purposes and tracking purposes. And then it's just lean documentation, I believe this is a good way to keep track of your exploratory testing sessions, and to share the findings. Okay, so session three, Section three is about generating ideas. At some point, when you're doing an exploratory testing session, it becomes hard for you to think of new scenarios. And so there are a number of ways that you can actually generate more exploratory testing ideas. This gets into a little bit more of the advanced side of things. But I just want to introduce you to it. Some ways that you can generate new ideas for further exploration, you can use user personas. So by placing yourself in the user shoes, it's very important to understand who the users are. And you've put yourself in that frame of mind and run through the application in that way. Another thing that you can use are huge heuristics and mnemonics, which we have a lot of them in the testing community, too many to list here. But mental short, these are basically mental shortcuts that allow you to solve problems really quickly. So you can just pull on it while you're doing the session, and it will help you to guide you. Another thing is using variations. So if you switch up your navigation, for example, instead of going to the home to the checkout page via the the icon on the top, you can find another way in the football. So switching up navigation, thinking of different combinations of inputs, for example, will help to diversify the test and give you more ideas for exploring. And then lastly, pair testing. So that's just working closely with another person on your team. It doesn't have to be another QA person, it could be anybody on the team who has knowledge. During those sessions, you ask questions, and you get guidance, you get ideas from those sessions on how to probably expand your testing and to look in other areas.

On this site, I wanted to share with you some of my top heuristics the how for ones that have I've been using for a while now to help me, um, I won't go into detail, but there are resources available that will talk about each of these. Each of them will help you to do more with your exploratory testing. And as I said, you know, using the pair testing approach, where two people test the same thing on application at the same time, it doesn't matter who you're with, I gave an example us expiratory tests here, but you can call on another team member, our product owner and developer, RBA and YouTube in a session, it allows you to ask the questions, get more information that can help to expand your testing efforts. Okay, now, let's talk about the traceability and coverage part. So, as I said, you know that one of the issues that we had was, how do you keep track of this, and one of the best ways I found to do this was using the Mind Map. Here's an example of how I tracked my findings. So you'll see here, I capture exactly what it is I'm testing on. I included the date, the time for the session, I included my test charts. And then you see I'm using the PQ IP format that I mentioned here. So All praises are highlighted. Any questions also using the icons that I mentioned, and ideas, lots of ideas were were generated during that. And then I was able to highlight problems, which turned into bugs or other user stories for action with the team. And then just general comments and notes that would have been helpful for testing in the future. So this is just an example of how you can use our mind map to help you to track and see the coverage of your testing. Okay, Session Four has to do with automation and exploratory testing. There's been a lot of discussions around this one. And so I like to ask, you know, can exploratory tests be automated? I do want more information on this. But in my experience, my answer is, yes, exploratory testing can be automated to a certain extent, I've seen in the space where people are actually, you know, getting into the code and simulating certain behaviors that are more of an exploratory nature than just doing our regression regression testing. So this is something that can be explored some more. But automation in exploratory testing is possible. Here, I wanted to share some passive exploratory testing tools. So passive meaning you know, there are tools that you use for documenting to capture if you don't have anything right now to capture your test. If you don't use our mind map, you can also use this extension, it's available free on Chrome exporter testing extension, it allows you to take notes, the box, the questions, the ideas, so gives you the PQ IP format, reporting that I mentioned, this is a good one. When it comes to tracking your time, you can use any time tracker, but there's this ultra time available on Chrome, which I use. And then as you're going through exploratory testing sessions, when you want to come up with edge cases, bug magnet is a nice extension to use because it gives you lots of input variables to include. So you don't have to go searching on the web for like zip codes or invalid emails or invalid addresses for example, it has all of that in there in different languages. And then all in one tools I used to use X ray in the past to do some exploratory testing that does help us to capture in the details in our tickets allows you to create the bug tickets directly from that still exploring as your test plans that's what we're exploring now. So when I get more information on that I will share but real entually any other note taking app actually use can work.

So just to summarize, everything that we just spoke about exploratory testing is a dynamic approach that complements scripted tests and it offers its own benefits. He principles in this exploratory testing is that you know, you lean on your domain knowledge, your expertise is very important. Being curious and embracing your critical thinking skills and then being creative is theirs are also very important in exporter testing and test Think process for exploratory testing is designed execute document. And you use your test charters to help you to design and use sessions to help you with execution. And then you document your findings. But everything happens at once, test strategies, so you should be putting on risk based testing, as well as time boxing for efficiency, and effectiveness. And then lastly, collaboration and communication. The most important part here is that you feed the information back to your team. It's essential to test us it's essential to developers, it's essential to everybody on the team. And so that concludes this session. I hope you learned a lot about exploratory testing, at least just to get started. If you have more questions about it. There's references to some of the books I've read that have helped me here. Those will be helpful. There are also additional links to heuristics to help you to expand your exploratory testing, and some general resources and links available. Thank you everyone. Yay, that was good.