How QA Saved the Day: Navigating Risky Pre-Holiday Releases with Confidence

The week before Thanksgiving can be a challenging time in software development. Product leads are often eager to meet their monthly targets, pushing for feature releases even when the timing may not align with QA best practices. This case study recounts a real-world scenario where QA successfully navigated a difficult political landscape to delay the release of a high-risk feature.

The Situation

A product team was eager to push a feature update in the pre-Thanksgiving release. However, a critical issue emerged: the feature's code would not be ready by the established code freeze date. The Tech Lead assured the team they would perform a rigorous code review to ensure quality, but this promise came with a significant caveat: the Tech Lead would be unavailable for much of the Thanksgiving week.

This posed two major risks:

1. Customer Impact: If a unique issue arose, there would be limited support to troubleshoot and resolve it during the holiday week.
   
2. End-of-Month Reporting: Thanksgiving coincides with a busy reporting period for many customers. A failure in reporting functionality would disrupt critical business operations for users who rely on accurate, timely data.

The Decision

As QA, our role was not to simply test and approve code but to assess the broader implications of shipping an incomplete feature. We conducted a release risk assessment and outlined the risks of pushing the feature:

1. The incomplete code would likely introduce instability, given the reduced availability of team members to support post-release fixes.
   
2. The potential for reporting issues posed a reputational risk to the company.
   
3. Customers had no immediate expectation for the feature, reducing the urgency to release it.

Despite pressure from the Tech Lead and Product Leads, we escalated the concerns to the VP of Engineering.

The Outcome

The VP of Engineering supported QA's recommendation to delay merging the new feature. This decision ensured the following:

• The feature shipped in the post-Thanksgiving release, with ample time for thorough testing and code review.
  
• The delay had no measurable impact on the overall project schedule.
  
• Customers experienced no interruptions or issues during the Thanksgiving week.

The Tech Lead later acknowledged that delaying the release was the right call, given the complexity of the feature and the risks involved.

Key Takeaways

1. Use Risk Assessments to Advocate for Quality: Clearly articulate the risks of rushing a release, especially when they affect critical customer workflows or coincide with challenging timelines like holiday periods.

2. Balance Business and Technical Priorities: While Product Leads may push for releases to meet goals, it's essential to weigh those goals against the potential impact on customers and the company's reputation.

3. Escalate When Necessary: When a decision involves significant risk, involve leadership to ensure all perspectives are considered and that the final decision aligns with the company's values and priorities.

4. Delays Aren't Always Bad: In this case, the delayed feature had no negative impact on the project schedule or customer satisfaction. Taking the time to get it right paid off in the long run.

This case underscores the importance of QA's role as a gatekeeper, not just for software quality but for the overall success of the product and customer experience. By staying firm and focused on the bigger picture, QA can navigate even the trickiest political landscapes to ensure the best outcomes for all stakeholders.

Does your QA team have a plan for handling high-pressure release situations? Share your strategies or lessons learned in the comments!

Mastering Risk-Based Testing: How QA Teams Can Prioritize for Quality and Efficiency

When it comes to Quality Assurance testing, understanding risk is key to delivering effective, efficient, and focused testing. Testing is about more than running checks on the entire product; it's about knowing where to focus your energy to ensure that any changes work as intended and don't disrupt critical areas of functionality.

Why Risk-Based Testing Matters

In QA, risk-based testing helps us prioritize. Instead of "testing everything," we zero in on areas with the highest potential for issues. Prioritizing testing effort based on the likelihood and impact of changes allows QA to focus on high-risk areas, ensuring more targeted, effective coverage.

When change occurs in software, it often ripples through several parts of the application, and each area of impact needs to be tested. Knowing the high-risk areas allows QA to test smarter, not just harder, delivering results that matter most to both users and stakeholders.

Story Time: "Test Everything" vs. Focused Testing

I'll never forget the numerous times I was instructed by my VP of Engineering to "test everything" before a big release. As a QA engineer, "test everything" sounds straightforward but rarely provides the best strategy for actual risk mitigation. Broadly testing every feature leaves little time to focus on edge cases or specific areas impacted by changes. With each feature tested only at the surface level, there's a higher chance that unexpected bugs, especially edge cases, slip through.

But when you're empowered to dive into high-risk areas, to dissect each change, you're often able to uncover issues that no amount of shallow regression testing would catch. Focusing on high-risk areas enables QA to surface issues that may seem minor but could disrupt critical functionality if left unchecked.

Identifying Risky Areas in Testing

So, how do we get to a point where we can intelligently identify and prioritize risks? Here are some of the techniques I've found valuable:

1. Analyze the Code Changes

When you get a code change, don't look only at what's new. Look at which existing parts of the system interact with these changes. Are there shared functions or dependencies? Minor alterations to shared code can have far-reaching effects.

Pro Tip: Build a habit of analyzing commit messages, reviewing PRs, and talking to developers about areas they feel are sensitive to change.

2. Understand Business Impact and Critical Paths

Not every feature carries the same weight in terms of business impact. Some features are directly tied to revenue, while others are key to user experience. Work with product managers to identify which features are most critical to the business, and make sure these paths are covered in testing.

Example: For an e-commerce platform, a change affecting the checkout flow demands high-priority testing since issues here could directly impact sales.

3. Ask Questions and Look for Assumptions

Testing assumptions developers or product owners have can reveal risks no one initially considered. Ask questions like: - Are there alternative ways users might interact with this feature? - What data assumptions are built into the feature?

Real-World Insight: Once during a testing cycle, I identified a critical bug by questioning the developer's assumption about what users would input into a form. It turns out that edge cases in user data broke the feature in production.

4. Use the Impacted Module Approach

When major changes impact a module, list out each dependent component. Take the extra time to test these areas in-depth; they're prime candidates for breakage, especially if they share interfaces with the modified code.

For Example: A change in the database schema might not seem like it impacts front-end functionality, but if your module pulls data from affected tables, it could disrupt page display.

5. Prioritize Test Cases Based on Risk Severity

Not all test cases carry equal weight. Evaluate which ones will deliver the best insights for time invested. Prioritize based on the potential damage a bug might cause in production. Work with stakeholders to determine what they'd consider "critical" failures.

Story: I once was told to "test everything," but by focusing on high-impact test cases first, I managed to discover a major issue in time for the developers to fix it, preventing a potentially costly post-release hotfix.

How to Approach "Test Everything" Requests

When you're asked to "test everything," it's usually because stakeholders want to ensure nothing is missed. But "test everything" often leads to overstretching the QA team's efforts. Here's how to handle this:

1. Clarify Priorities: Politely ask what areas are the highest priorities.
2. Suggest a Risk-Based Approach: Explain the benefits of focusing on high-risk areas and offer to create a prioritized test plan.
3. Educate on the Cost of Regression Testing: Share data on the time it would take to test everything versus targeted testing to help decision-makers see the value of a focused approach.

Embracing Risk-Based Testing in Your QA Strategy

To wrap it up, effective risk-based testing is about asking the right questions, using your knowledge of the application's structure, and constantly refining your approach to prioritize high-risk areas. By emphasizing risk assessment in your QA strategy, you move beyond just "covering" your application to understanding where it's most vulnerable.

QA isn't about testing everything; it's about testing the right things. When we focus our efforts strategically, we ensure quality where it matters most.

Effortlessly Access Jira Tickets with a Custom Bookmarklet

In my previous company, I often needed to access Jira tickets to review or comment on them. The usual process was a bit tedious: I'd have to open Jira, then manually type the ticket number into the search bar to locate it. This multi-step routine took up valuable time, and I knew there had to be a faster, more efficient way.

The Solution: A Bookmarklet for Jira

I created a simple bookmarklet to streamline this process. With this bookmarklet, I can simply enter the ticket number, and it instantly opens the Jira ticket. It's an efficient shortcut that removes the extra steps of navigating through Jira manually.

Since I only work in one Jira project, the bookmarklet is set up for my specific project. However, if I were working across multiple projects, I could modify the code to prompt me to enter the project ID each time or simply omit the project number from the URL.

Bookmarklet Example

To use this bookmarklet, create a new browser bookmark and paste the following code into the URL field. Be sure to update the Jira domain (e.g., company.atlassian.net) and project ID (e.g., PP24) to match your Jira environment.

javascript:(function(){ 
  var ticketNumber = prompt("Enter Jira Ticket Number"); 
  if (ticketNumber) {
    window.location.href = "https://company.atlassian.net/browse/PP24-" + ticketNumber; 
  } 
})();

This little tool saves time and effort, especially for anyone working regularly within Jira. Just click the bookmark, enter the ticket number, and you're immediately taken to the ticket-no need for manual searches.

Boo! Common Frightful Phrases in Software QA

As the leaves turn and the nights grow longer in October, we're not just preparing for Halloween; in the tech world, we're also bracing for the spooky season of Software Quality Assurance (QA) testing. Here's a light-hearted look at some of the most terrifying phrases you might hear echoing through the halls of software development, perfect for this time of year.

1. The Build is Broken

Imagine this: You're walking through a dimly lit corridor, and suddenly, from the shadows, a voice whispers, "The build is broken." It's not just a phrase; it's a curse that threatens to derail your entire project. It's like finding out your pumpkin has rotted from the inside out.

2. Regression Testing Found New Bugs

This is akin to opening your trick-or-treat bag only to find that all your sweets have turned into bugs. It's not just the new features that are haunted; the old ones have now decided to join the ghost party.

3. We've Hit a Showstopper

Hearing this is like discovering a haunted house in your software. This isn't just any old specter; it's the kind that stops the entire production line, making everyone question if they've angered some tech deity.

4. It Works on My Machine

This phrase is the tech equivalent of a ghost story that ends with, "But when I got home, it was still there?" It's the mystery that keeps developers up at night, trying to replicate an issue that seems to vanish like a wisp of smoke in daylight.

5. "There's a Memory Leak"

If software had a ghost story, this would be it. It's as if your code has been possessed by a poltergeist that keeps eating away at your resources until your application collapses into a spectral heap.

6. We Didn't Test for That Scenario

Imagine setting out to trick-or-treat only to find your costume has a hidden flaw that everyone notices. This phrase is the realization that your testing didn't cover the 'vampire at the door' scenario.

7. The Database is Corrupted

This is like finding out that all your Halloween candy has turned into a pile of dust. Data corruption is the nightmare where all your hard work vanishes, leaving you with nothing but echoes in an empty digital tomb.

8. The Third-Party Service Just Changed Their API

In our spooky software tale, this is the moment the witch decides to rewrite her spellbook without telling anyone. Suddenly, your integrations are as outdated as a Victorian ghost story.

9. We're Out of Memory

This phrase brings to mind a haunted house party where the guests keep arriving until the house literally can't hold anymore. Your software is the house, and memory is the space, creaking under the weight of too many guests.

10. We Need to Rollback to the Previous Version

Ever had to undo your Halloween decorations because the kids were too scared? This is the software equivalent. It's admitting that the new version is more of a fright than a delight, and we need to go back to the "safe" version.

This Halloween, as you carve your pumpkins, remember these eerie phrases. They might just give you a chill, but they also remind us of the critical, albeit slightly terrifying, role QA plays in our digital world. Here's to hoping your projects are more treat than trick this season!

Happy Coding, and don't let the bugs bite!

QA Clichés

Some common Football cliches.

In the many years of my software testing, there are several clichés or commonly repeated phrases that testers, developers, and project managers might say during meetings:

• "It works on my machine." - Often said by developers when a bug can't be reproduced in their local environment.

• "That's not a bug, it's a feature." - A humorous or sometimes serious claim that unintended behavior might actually provide some value or was intended all along.

• "Have you tried testing using incognito mode?" - Often highlights issues related to session management, caching, or initialization.

• "Works as designed." - This can be a genuine clarification or a way to push back on a bug report when the software is behaving according to the specifications, even if those specifications might now seem flawed.

• "It's not reproducible." - When testers or users report an issue that can't be consistently replicated, leading to challenges in debugging.

• "We need more test cases." - Often said when unexpected issues arise, suggesting that the existing test suite might not be comprehensive enough.

• "Let's take this offline." - Not unique to QA but commonly used when a bug or issue leads to a discussion that's too detailed or tangential for the current meeting.

• "Did we test for this scenario?" - A question that arises when an unforeseen issue comes up, questioning the coverage of the test cases.

• "The user would never do that." - A sometimes risky assumption about how the software will be used, which might overlook edge cases or unexpected user behavior.

• "How quickly can you test this?" - Suggesting that QA engineers can speed up testing without impacting the quality of the test.

• "This should be an easy fix." - Often underestimated, what seems simple might involve complex underlying code changes.

• "We'll fix it in the next sprint/release." - When time runs out, or when a bug is deemed not critical enough for immediate action.

• "Automate all the things!" - While automation in QA is crucial, this phrase humorously points to the sometimes overly enthusiastic push for automation without considering the ROI.

• "It passed in staging, why is it failing in production?" - Highlighting environment-specific issues or differences in data sets.

• "QA found another 'corner case'." - Recognizing that QA teams often find bugs in the most unexpected or rarely used functionalities.

These clichés reflect the ongoing dialogue between intention, design, implementation, and real-world use in software development. They encapsulate the challenges, humor, and sometimes the frustrations inherent in the QA process.

Next Week's blog post is about scary sayings heard in QA.

Playing Politics at Work

Welcome to the world of Quality Assurance, where finding bugs is just one part of the job. Beyond the technical challenges, there's a whole other layer that can be even more complicated: navigating the political landscape of corporate America. You might think the office politics are confined to boardrooms, but the reality is, they can affect everything - your projects, your career, and even the quality of your work.

The Reality: A Tight-Knit Core Group

In many businesses, there's a core group of people who pull the strings behind the scenes. They're often managers or senior leaders who've known each other for years and wield significant influence. These people tend to hang out together, have lunch offsite, and make key decisions that ripple throughout the organization. They may not be the ones in the trenches, but they control what happens on the battlefield.

For newcomers - especially those with fresh ideas and an eagerness to improve processes - this can feel like a brick wall. You might bring a brilliant suggestion for streamlining the release process or optimizing the testing workflow, but getting buy-in from the core group can feel impossible. They may view outsiders, especially those with different perspectives, as a threat to their status quo.

My Experience: The Manager with the Sales Hat

I've been in that position before. My former manager at a very large ecommerce site was so focused on impressing this core group that our one-on-one meetings became a rarity. Instead, he'd be off attending every meeting he could with the key decision-makers, trying to climb the corporate ladder. He wasn't a bad person, but he made it clear where his priorities lay - being seen by the right people, rather than supporting the people under him.

It was frustrating, to say the least. There I was, trying to push for improvements in our testing processes and advocating for better quality, but it often felt like my efforts were invisible. If I wanted to make progress, I needed to understand the lay of the land and find ways to navigate around it.

Finding Your Path: Focus, Over-Deliver, and Skill Up

Here's the reality check: as a QA Engineer, you may not be part of the "in" crowd right away, but you don't have to be sidelined. There are strategies, that I have learned, which you can use to build your reputation and make your mark:

1. Focus on Your Goals: Don't get distracted by the politics, but don't ignore them either. Focus on delivering high-quality work consistently. Make sure your bug reports are thorough, your test plans are well-documented, and you're always looking for ways to improve the product.

2. Over-Deliver When It Matters: This is your ticket to building credibility. Sometimes, it means burning the midnight oil or working through a weekend. But if you can pull off a critical testing phase or help recover a project that's about to miss a deadline, people notice. It shows you're dedicated and reliable, and it might even catch the eye of those key decision-makers.

3. Be Ready with Your 'Sales Hat': When you do have ideas for improvements, be prepared to sell them like a pro. This means doing your research, building a solid business case, and framing your suggestions in a way that appeals to their priorities. It's not just about being right - it's about being persuasive.

4. Invest in Your Skills: The political climate might not change overnight, but you can keep improving yourself. Learn Python, JavaScript, or whatever new tool is in demand. These skills not only help you automate tedious testing tasks, but they also make you more valuable, both inside and outside of your current organization.

Why It's Worth It

It can be tempting to give up when it feels like you're shouting into a void. But navigating the politics is a skill in itself, and one that can pay off in the long run. You don't have to become best friends with the core group, but if you can show that you're competent, dependable, and skilled, you might just find yourself being invited to that offsite lunch one day.

And if you don't? Well, the skills and reputation you build along the way will follow you wherever you go. You'll be prepared for whatever comes next, whether it's a new opportunity within the company or a fresh start somewhere else.

After all, Quality Assurance isn't just about making sure the software works - it's about finding a way to thrive, even when the landscape around you isn't as polished as the code you test.

The Silent QA Detective

Throughout this month, I am concentrating on QA Stories, highlighting remarkable events from various companies I've worked with. This week, I'm reminded of a reserved QA engineer whom we'll call Steve. He was exceptional at his job - uncovering even the most stubborn bugs. His ability to pinpoint the trickiest problems in web applications made him legendary among developers, despite his perpetually low-key demeanor. Here are a few stories from Steve's bug-hunting escapades that are still circulated within the Dev communities.

The Phantom Dropdown Issue

Steve's first victory was what developers called the "Phantom Dropdown Issue." It was a classic case of "it works on my machine" that had stumped everyone for weeks. The dropdown worked perfectly in all environments except for a few random occurrences on production. Most of the team chalked it up to user error, assuming that the customers must have been doing something wrong.

Steve, however, decided to dig deeper. He observed that the bug occurred only during very specific conditions- when a user with a regional setting of "en-UK" tried to access the dropdown on a Monday morning. It turned out that the JavaScript controlling the dropdown couldn't handle a particular time parsing during this transition. Steve quietly filed the bug report, detailing the obscure conditions that caused it. Developers were stunned - how had he even thought to test that?

When asked how he found it, he shrugged and said, "I just listen to WRKO for the talk, so I had time to think." That left everyone scratching their heads. How Steve's radio habits connected to the bug was beyond them, but they knew better than to question the man who could find a needle in a haystack.

The One-Pixel Invisible Button

Another time, Steve uncovered what the team later dubbed "The One-Pixel Invisible Button" issue. Users reported that their session would randomly refresh, losing all their progress. As usual, no one could reproduce it. Frustration was high, and Steve, as always, remained calm.

One afternoon, he took his laptop, sat in the break room, and pulled up the site. A few minutes later, he came back to the team, holding a stack of printed screenshots with highlighted regions. The culprit? A single, misplaced invisible button- just one pixel wide- hidden within the footer.

Apparently, whenever a user's mouse hovered over that pixel while scrolling, it triggered a refresh event. Steve discovered that it only affected users on certain screen resolutions - an anomaly that most testers would have ignored. He summed it up with his characteristic nonchalance: "Think outside the box, but don't forget to close the box once you're done."

The Mysterious Cache Flaw

Then there was the time he identified a cache issue that occurred only during high-traffic events. Users would get logged out unexpectedly, right during peak periods such as quarter end. The developers went back and forth with the server logs but couldn't pinpoint the root cause.

Steve, though, noticed a pattern. It only happened when users were refreshing their pages at precisely midnight. Most saw it as a coincidence, but Steve wasn't one for coincidences. He discovered that the session cookie's expiration time overlapped with a server-side cache refresh cycle, which triggered unexpected logouts. It was such a specific edge case that the fix involved changing a single line of code, but it saved the team's reputation.

When asked how he managed to spot the issue, Steve just said, "Don't assume, verify." Again, no one understood how that was relevant, but they were just relieved he found the problem.

Conclusion: The Value of Quiet Observation

Steve's bug-hunting prowess became the stuff of legend, not because he was loud or boisterous, but because he knew how to pay attention to the little details. While others might focus on the obvious, he had an uncanny ability to consider the strange and unexpected scenarios that others dismissed. To Steve, finding bugs was like slowly chiseling away at Stone Mountain with a hand tool- meticulous, precise, and never rushed.

He never sought the spotlight, but in the world of QA, the results spoke for themselves. Developers might have laughed at his quirky remarks or puzzled over his non-sequiturs, but they also knew one thing for sure: if there was a bug hiding somewhere in the code, Steve would find it.

The Importance of Time Management in QA

In the dynamic world of software development, effective time management is crucial for Quality Assurance (QA) teams. As a QA manager, it's essential to ensure that your team balances their time between tracking bugs and seeking additional help when solutions are not immediately apparent. This balance not only improves productivity but also ensures that new issues do not pile up while existing ones are being investigated.

The Story of Peter

Let me share an example from my own experience. Years ago, I managed an employee named Peter (not his real name). Peter was exceptional at identifying and investigating why customers were encountering errors. He had a knack for finding those weird, one-off cases that often slipped through the cracks. However, Peter's dedication to solving these issues sometimes led to a significant problem: he would spend so much time investigating a single issue that new issues would start to accumulate. Peter had the habit of trying to solve really complex issues and would work late into the night to do so. While he was highly skilled and well-loved by the engineers, he simply spent too much time on one issue.

Setting Time Limits and Seeking Developer Input

To address this, it's important for managers to set time limits on bug fixes. If an issue isn't resolved within the allocated time, it's crucial to collaborate with developers to see if they can provide additional input. This collaborative approach not only speeds up the resolution process but also brings fresh perspectives to the problem.

The Role of Logging

One effective strategy that often helps is adding additional logging to the system. Enhanced logging can provide valuable insights into why things are not working correctly, making it easier to identify and fix issues. However, it's equally important to remember to remove the logging once the issue has been resolved. There's nothing worse than having to debug future issues with an overwhelming amount of logging data to sift through.

Conclusion

So, fellow QA testers, remember:

1. Set time limits for bug investigations.
2. Seek help if you can't solve it in time.
3. Log smart, but clean up afterwards.

Time management isn't just about working faster; it's about working smarter. Let's keep hunting those bugs efficiently, so we can all go home at a reasonable time. Happy testing!

The Great Spell-Checker Saga

Let me take you back to one of the more entertaining moments of my software testing career. Picture this: I'm on the Electronic Services team, and we're rolling out a spell-checking tool for a project. Seems simple, right? Wrong.

Spell-Check Shenanigans Story

This rollercoaster of a bug hunt taught me a few things:

1. Quick fixes might be fun, but they're usually useless: Adding words to the dictionary every time something breaks is like trying to fix a flat tire by painting the car. It looks productive but does nothing.

2. There's always a second opinion for a reason: Developer #2 might as well have been wearing a superhero cape. Sometimes, it takes a fresh brain to clean up the chaos left behind by "creative" problem-solving.

In the end, we got the spell-checker to work, but not without a few laughs and some comically clumsy developer moments along the way. And hey, sometimes in QA, that's just how it goes.

Backwards Law

Have you ever felt like the most elusive bugs always seem to surface at the most inopportune times? Perhaps you've spent hours poring over test cases, only to find that the most critical defects were discovered by a seemingly random user.

This phenomenon isn't uncommon. In fact, it's a well-known principle in software quality assurance called the Backwards Law.

What is the Backwards Law?

The Backwards Law suggests that the most significant problems in a system are often the ones we least expect or anticipate. In other words, the bugs that cause the most disruption or embarrassment are typically not the ones we spend the most time testing.

Why Does the Backwards Law Exist?

There are several reasons why the Backwards Law holds true:

• Overconfidence: When we believe a system is thoroughly tested, we may become complacent and overlook potential issues.
• Cognitive biases: Our brains are wired to seek patterns and confirmation, which can lead us to ignore contradictory evidence or unexpected outcomes.
• Unforeseen circumstances: Real-world usage can expose vulnerabilities that are difficult to simulate in a controlled testing environment. How to Apply the Backwards Law in Quality Assurance While the Backwards Law might seem counterintuitive, it can be leveraged to improve your testing strategies.

Here are some tips:

1. Embrace uncertainty: Recognize that even the most meticulously planned tests cannot account for every possible scenario.
2. Prioritize risk: Identify the areas of your system that are most critical to the user experience or business objectives. Focus your testing efforts on these high-risk areas.
3. Encourage exploratory testing: Allow testers to explore the system freely, without strict adherence to predefined test cases. This can help uncover unexpected issues.
4. Leverage user feedback: Gather feedback from real users to identify problems that may have been missed during testing.
5. Conduct stress testing: Simulate heavy loads and extreme conditions to uncover performance bottlenecks or unexpected failures.

By understanding and applying the Backwards Law, you can develop a more effective and comprehensive quality assurance strategy. Remember, the most significant bugs are often the ones we least expect.

Use it All The Time

My testing has consistently shown that the Backwards Law is particularly effective at uncovering unique bugs, especially during the post-regression phase. It's during these final QA checks that unexpected issues often arise. The challenge lies in rapidly reproducing these bugs so engineering can address them promptly. They frequently inquire about the specific testing activity that led to the discovery.

Do you have any experiences with the Backwards Law in your quality assurance work? Share your thoughts in the comments below.