How to defuse a bomb… Wait, I mean a bug

Helpful testing methods to get a quality product out faster

When you have a product release happening very soon and there are major features going out, you may feel a bit of pressure trying to make sure there are no bugs going out with those features. Testing the product under a strict deadline may even feel like you are defusing a bomb. Okay, it may be a little less pressure than that, but it is a good analogy for what life is like as a tester. When a bug (bomb) doesn’t get defused in time, it can explode right in the end user’s face. Such an explosion is definitely not what the product team wants, so let’s talk about some testing methods that can help to catch that bug before it gets to the user, plus what you can do even if it does. Follow this advice, and you can feel like the guy below when you defuse a bug.

Prevent it

Prevention is always the best way to keep terrible things from happening. That’s why you are “supposed” to go to the doctor every single year and get that check-up. It is also why we are implementing testing methods such as pair testing, solid testing environments, and a strong automated testing suite on my team. Pair testing is where I get to sit with a developer on code they just barely wrote and find any obvious problems before they merge it to our main code base. The developer and I are both sitting at the same computer, and we discuss the feature that is being tested. We find bugs together before they are merged, saving us time in the long run waiting on builds. This exercise has already caught some pretty crazy things before they became issues for people testing on our main code base.  The best part is that a pair test only takes around 10-15 minutes. They can go longer, but if you are testing the code early and often, they usually don’t need to. Pair testing is not just limited to a developer and tester. It can be used at any time in the testing process and include anyone on the product team. I particularly like to grab our UX designer to do pair testing with me on new features. He can typically find issues with the finer details of a feature before I can because he already designed every small detail. While I am fast at finding usability issues, he quickly identifies when a button is 2px off or when text is using the wrong font. By leveraging the unique abilities of my team like those of our UX designer, I save time in the long run and can get more testing done faster. As we have started our pair testing on different teams at Lucid, we have noticed quite a few benefits. For instance, the developers have started learning about the way a tester thinks. They understand that if a change is going to happen to a shape, I am going to play with all the options to edit a shape and make sure they all work. Testers also benefit because it is a chance to talk about all the different areas of the product that could be affected by the change. If a piece of code changes in the downloading feature, a developer can let a tester know that publishing could also be impacted. It is a win-win for all parties involved. Besides pair testing, there are two more critical things to keep in mind when trying to prevent bugs. First, having multiple testing environments helps greatly. All of our testers can be testing different code at the same time before any of it gets to the main code base. Creating and maintaining these environments (one for every three to five developers) has been a tremendous amount of work. However, it is paying off because it is saving time trying to make sure our 12 testers can test new things in the separate environments simultaneously instead of waiting on each other. Another critical means of prevention is automated tests that work. Could you imagine sending in a bomb bot with a broken arm to defuse something? Neither can I. That is why it is critical to not only have automated tests but to have automation you can trust. Otherwise, you could end up experiencing something similar to what happened to these guys who stumbled upon a bomb defusal robot having some issues.

You can also check out this article by one of my team’s developers to get a better idea of the value of having working automated tests. FLAKY TESTS: THE TESTER’S F WORD

Defuse it

So the bomb—I mean bug—made it past some automation and pair tests. Now what? This is where testers shine. I have a lot of respect for the testers I work with. They are diligent, focused, and detail oriented. We scour the product constantly. We know every piece of it like the back of our hand. We do sanity tests, functional tests, session-based tests, integration tests, regression tests, and others— all before the user even sees the latest updates. To top it off, we do all of this pretty fast. The key to getting everything done quickly is making sure your testing team works together. If you notice that your team is struggling with something, speak up about it. A good tester can’t be afraid to dissent from the popular opinion if they know something is not working. Along with expressing your concern, try presenting some solutions to solve the problem. Getting the conversation started about a problem is usually the hardest part. Let’s talk about one of the testing methods I just mentioned—session-based tests (SBTs). An SBT is a form of testing that lets you dive into a specific part of your product. Pair testing is great for catching things right off the bat, but an SBT lets you find the edge cases. Compared to normal testing or just writing out individual test cases, SBTs provide an organized structure to think about a feature as a whole.  The following steps explain the basics of how to perform an SBT:

1. Make a goal for your SBT. This step is as simple as writing one line about why you are doing the SBT. Your goal could be to learn more about an area of the product you are unfamiliar with. It could also be to test a new feature and get a solid idea of how to do so.
2. Make a charter.  A charter should be a list of the areas of a product you want to hit with your testing to help guide you during the SBT. This list is based off of your goal. Think about all things you want to accomplish with the test and write them down. This can be a very detailed list that goes into every single thing you will touch or an overall list that generally outlines where you anticipate going with your testing. The idea is to fit the charter to your needs.
3. Set aside between 30 minutes to 2 hours for your testing. The amount of time you spend on an SBT really depends on what you are testing. If you are testing a large area of the product, you will probably need to spend more time on it versus if you are testing something small.
4. Time yourself at 15-minute intervals. Every 15 minutes, ask yourself if you are following the charter you made. If you are not, are you going in a good direction? Should you get back on charter or maintain your current course? There is no right answer. This check is just to make sure you are getting the most of of this testing time.
5. Take notes during your SBT. These notes can be bugs you find to report. It can be things in the flow that don’t seem intuitive, or it can be anything you are curious about and want to keep diving into. The point of the notes is to keep track of what you are doing.
6. Report on your SBT. At the end of the SBT, go to another member of your testing team and talk them through your SBT. They may have more insight into things you can do and what things you may have missed.

Though it may feel easy to skip once you get into the swing of an SBT, it is critical that you always do step six. No matter what, different members of a team can always provide valuable insight into how a person’s testing can improve—different people all have different insights. There are members of my team who are amazing at finding every tiny edge case. Another member is great at understanding how the end user thinks. Similar to a bomb “squad,” you have to work as a team to get things done.

Contain the explosion

You have finished your pair tests and SBTs. The automated tests are all passing. You have tested every case you could think of. It is finally time to ship to the end user. Unfortunately,  a bug still made it out despite all the effort. The good news is there are a few testing methods you can still implement in this situation. We call them bug scrubs and bug triages. A bug scrub is when multiple people from your team get together and “play” with a specific area of the product. This can be a group of testers or a group of various people from the product team including product managers, developers, UX designers, and support team members. Similar to an SBT, you will want to start off a good scrub with a goal. For example, the goal could be to test an area of the product that is flaky and bound to have bugs. We also like to do scrubs in particularly complicated areas of the product to help the whole team understand the area. As a testing group, we have a scrub at least every other week. These provide great value because it lets you find issues before a user does and has the added benefit of teaching your team how to use the area you are testing that day. In large products, it is essential that you keep your entire QA team up to date on the features so that anyone can test anything that comes up. A bug triage is a very simple meeting that usually includes a product manager, a developer, and a QA specialist. It is a daily five-minute meeting to go through the latest issues and prioritize them. Some days there may be no issues to triage, but keeping the meeting on a daily basis is important for those days where everything seems to be breaking down. At Lucid, this meeting consists of the head PM, QA specialist, and developer on our Chart and Press projects. Having all three in attendance is critical for a successful triage. The PM gets an understanding of the backlog and what issues arise frequently and need to be fixed. The developer gets an opportunity to explain how hard an issue would be to fix. The QA specialist gets an understanding of areas that need to be focused on for future testing and can also give a better explanation of the issue at hand relative to the rest of the product. No matter what step you are in when trying to defuse a bug, it is always important to work as a team. Whether that is the product team or the testing team, you will benefit from getting each person’s unique insights into the way the product should work. Testing methods like pair tests, SBTs, bug scrubs, and bug triages are there to help both you and your team work more effectively towards accomplishing the goal of getting the best product to your end user as quickly as possible.