Archive for February, 2016

Regression Testing and Scope

Posted on: February 19th, 2016 by admin No Comments

Regression Testing and Scope

A brief exchange on twitter on the topic of regression testing leads me to this post.  Twitter, is not the place to really discuss this topic as there are too few characters and there are many factors that go into determining the scope of regression test.  Regression testing is the testing of portions of the product that were built previous to this iteration.  The testing is to ensure that in the addition of the new features to the product we have not inadvertently broken the product.  A good definition of regression testing can be found from ISTQB.


Risk Aversion 

One of the things we will consider in determining the scope of our regression testing will be the level at which our company is comfortable with the risk associated with the product in the field.  If our company makes medical or perhaps automotive or aerospace products, we may generally be risk averse since the consequences of our mistakes can be catastrophic to our customers. If this is the case we may adopt a policy of 100% regression testing.


Historical Record 

Our approach to regression testing may be based on our organization’s historical record with product updates.  For example, perhaps in the past, we have delivered product updates to the customer that have had defects, not only in the new release content but in the prior built portion of the product.  Consider a parameter that is accidentally altered or an error in the build process that allows a particular outdated software module to be built into the new product.  It could be a simple “fat finger” mistake that corrupts the old portion of the software.  If we have experienced many of these types of events, we may opt for 100% regression testing or as close to it as we can get.


Level of Automation

It might seem strange that level of automation may play a role in determining our approach to regression testing; it is not that strange in fact.  There are often two project-level constraints upon our test that require us to consider what level of regression testing will be performed. Those two constraints are; 1) the amount of time available for regression testing, and 2) the money (often labor) available for regression testing that does not ruin our business case.

For the first situation, we are up against the delivery deadline and must cram some 50 pounds of testing in a 10-pound amount of time to mix metaphors.  If we only have manual testing we have to decide what is most likely to cause serious problems to the customer and conduct that testing only so we can launch on time.  The second situation is the amount of money available for spending on regression testing that will not destroy the business case for the software update or change.

In both cases, automation can make this decision easier. Effective automation means people are not as heavily involved in the regression test execution.  There is human involvement in the creation of automation, setting up the specific test and interpretation of the test results.  However, if we have been building our automated regression testing suite all along as we developed the product, the investment in time has already been made.  Also, machines do not get tired, do not mind working late at night or even many hours or days in a row.  If we have significant automation we may opt to conduct a larger scope of regression testing as the impact to our team and the business case may not be significant. This is a safe approach.


Consequences of the Changes

This is really where the heart of the matter resides.  Regression testing is performed to confirm that what once worked is continuing to perform and that our updates have not caused any errors elsewhere in the product or system.  Using this as the approach, we consider the updates to the software and what features or functions would possibly be impacted by that alteration.  If the nature of our change is only one software module minimally connected to or depending upon other parts of the software the scope of regression testing may be small. If the change has a larger potential for impact, for example,  an operating system change, we will consider the consequences of that change on the software as a system and the scope of the regression testing will be larger. 

In addition to prioritizing by what was changed in the product, we consider the consequences of a failure upon the customer.  Using this approach, those grievous potential failures will be the features upon which we focus our regression test.  Our limiting the scope of the testing saves time and money, but as we are making a decision about what gets tested based upon the severity of impact .  Less severe issues will likely go undetected which means a defect can make it to the customer, at which time corrective actions must be administered to the field.



Personally, I have seen that defects can creep into the development work at any time and from any place, even into the old portions of the code. After all, we are updating the entire software release even when our intention is to add one software module or add to an existing software module.  Errors can creep in all along the delivery cycle and can impact anywhere in the product.  With the cost of automation coming down and the ability to quickly test with this automation, regression testing can be easily accommodated and cover a great range of the scope of the product with little damage to the test personnel via long hours.  The problem with this approach is you cannot decide today to automate regression testing and have it available tomorrow. The organization must have an understanding at the strategic level for developing the regression testing and pursue with a conscious and deliberate set of actions.

Organization Management and the Pendulum

Posted on: February 15th, 2016 by admin No Comments



I have been re-reading my copy of Introduction to Quality Control by Kaoru Ishikawa.  If you have been following our blog, you will know we frequently write on quality items and it is not strange for us to read these types of books. However, there is a reference in this book that takes me back to some of the companies at which I have worked in the past.  The specific paragraph is noted below[1]:

Presidential QC diagnosis should not be carried out on the premise that everything is bad, using top management muscle to expose malpractice n a deal shortcomings.  Like the doctor who examines a patient in order to diagnose an illness and commence treatment and promptly so that the illness gets no worse, the presidential QC diagnoses aims for action.  Its purpose is to enlist everyone’s cooperation to pinpoint weaknesses and systematically improve the situation.  This means that CEOs must never become angry, even when their company’s flaws and shameful weaknesses are exposed, and those being diagnosed must also describe their shortcomings clearly an honestly, exactly as patients would explain their symptoms to a doctor.

I do not believe I have ever seen an executive have a melt-down or respond angrily regarding some poor performance, lucky me.  I have however, seen these issues covered up, downplayed and flat out ignoring for what appears to be the sake of organization politics. Then we wonder why the organization’s  operating costs and project schedule balloon out of control.

There can be little doubt that the olden day’s executive’s angry responses were harmful.  Experience suggests there is a new poor way of responding, and that is to not worry about the patient.  Shortcomings are not discussed and explored out in the open but covered up and hidden for the sake of creating the illusion of a healthy patient.  We are less concerned with making the patient healthy, but use makeup to camouflage what could be a larger systematic issue.   As a Vice President of Quality once told me, it is a cancer that threatens to take down the host company.  I personally believe reliance on political rhetoric to the extent of covering up operational and personnel issues is equally as bad as yelling. Neither are the way to continuous improvement or a very satisfying work environment.  As equally bad as yelling and screaming is neglecting or taking no action that cures the patient or  eases the distress.  Opting for doing nothing, or makeup and superficial contrivances to make the patient look like they are well or getting better is not a solution. We must remember that even inaction is an action with equal if not greater effects as taking the wrong actions.

[1] Ishikawa, K. (1990). Introduction to Quality Control. Tokyo: 3A. page 8

Decision Matrix Applied to Test Strategy

Posted on: February 11th, 2016 by admin No Comments

We can use a decision matrix to help determine  the best test strategy.  In this instance, the decision matrix is comparing what we believe to be vehicle testing success criterion (such as the fidelity of the test results and ability to duplicate, the speed at which we can test and meeting critical dependencies) against a number of possible solutions.  The highest scoring approach represents the best approach given the constraints.



Optimism is a Kind of Blindness

Posted on: February 9th, 2016 by admin No Comments

 Optimism is a Kind of Blindness

Perhaps it is because I have seen the word optimism abused that I say optimism is a kind of blindness.  Optimism like hope is used to justify our limited planning and reduced talent.  Through optimism, we encourage troops to charge across a field that will surely end in their death and little else (Pickett’s charge).  It is optimism that responds to valid constraints or risks cavalierly or worse yet shooting the nay-saying messenger.  Consider the Allied Operation Market Garden, dropping paratroopers on top of tanks. The messenger that brought this to the attention of “management” was (-Major Brian Urquhart) and he would be asked to take sick leave because he pointed out that dropping paratroopers on top of tanks would be a poor idea.  He rocked the boat.

I am not comparing the seriousness of troops and their lives with business, but this approach is not limited to generals and wartime.  This approach happens in business every day I imagine.  This same do not rock the boat, optimism in spite of facts or evidence is precisely when optimism becomes fantasy as Learn. Lead. Repeat. put it on twitter.

Optimism and fantasy

Optimism and fantasy


Objectivity, Projects and Product Test

This is not a trivial thing. Any objective of significance will come with challenges and pitfalls.  A bad or negative attitude is just as bad as blind adherence through optimism.  Finding the “truth” can be difficult, something’s are not readily knowable, and sometimes we do not know the right question or have access to measurements that can help us find out if we are deluding ourselves with optimism or hope.

  1. Think critically
  2. Seek other perspectives than our own (think critically on these perspectives)
  3. Identify metrics that will help us understand what is possible
  4. Take accurate and appropriate measurements
  5. Respond quickly to what you learn

If you have been involved in many projects, you will know that optimism can damage starting with our project plan, through project resources (understaffed) and through progress tracking, all in the name of optimism.  If you have been a testing person, you will have seen late deliveries, buggy iterations and the desire to quickly move the last iteration to production in spite of what the previous iterations of the product suggest the next iteration quality will be.  We may even decide to launch because we have not seen any failures neglecting the fact that we have not conducted any tests.

There is a place for Optimism

To be sure there is a place for optimism, if we believe we are doomed then we likely are doomed.  On the other hand, we cannot let rampant optimism lead us to believe what is impossible to be very possible but perhaps just out of reach.  As Learn.Lead.Repeat suggested on Twitter, the challenge is to help leaders understand that optimism is much different than fantasy, and hope without wisdom is fantasy.


Optimism is a Kind of Blindness

Poor Excuse for Not Automating Testing

Posted on: February 5th, 2016 by admin 1 Comment

Poor Excuse for Not Automating Testing

Recently I came across and participated in a social media exchange that proposed that automating product testing (software) was really not helpful.  Their assertion was backed with comments about personnel new to testing will be unable to learn how to test.

Testing and System Complexity

System and software complexity, the number of interactions, permutations (configurations) and feature content, and failure mode consequences can make comprehensive testing via solely manual techniques very improbable.  The only time solely manual methods may work, is for very simple and small scale products.  The number of test cases for a moderately complex system can be in the thousands.

Testing and Repeatability

Done correctly, automating testing helps ensure repeatability of testing.  This is important when we find a defect and we wish to trace the root cause of the problem, the exact set of steps of the test that evoked the defect.  Of course, configuration management plays a significant role here also, but testing that is not automated has variation, including the dwell time between steps that can be part of the reason we see the defect.

Testing and Time to Market

Business product time to market pressure requires a balance between engaging and growing the talent along with getting the right quality product into the customer’s hands.  Experience indicates testing time to be one of those perceived obstacles to launch and moving to an entirely or even predominantly manual testing method will only make this situation worse.

Product Growth, Testing, and Learning

Unless a product is a one-time creation, there are plenty of opportunities for developing testing skills of the talent in the subsequent iterations.  Each new product or new feature will need to be tested (not just regression tested).  This thinking through the testing, creating the test plan and test cases, even executing these new test cases manually a time or two to make sure we understand how the automated test case should perform are all mechanisms and opportunities for learning.

Learning Testing and Senior Personnel

Our more seasoned or senior test personnel can also help our team become more knowledgeable about testing and critical thinking regarding the product.  Pairing a more seasoned test talent with a person new to product testing is an opportunity for on the job training.  It is also possible for your new team members to undergo self-development by reading and studying other works and applying where they are able.

Testing Organizations

Our new team members can learn also from organizations that promote understanding of product testing.  The list below are some interesting software testing organizations but there are many more.  As the manager of a test department, I had my entire software test personnel go through the ASTQB certifications process.  This helped establish the fundamentals of testing the product along with methods for improving the throughput.  We made heavy use of automated testing and were always increasing that as we conducted manual testing.


Automated testing does not get tired, it does not want to be with the family for dinner, and has no problem working over the weekend or late at night.  It is possible to test nearly around the clock without the damage that happens when we make people work over Thanksgiving or Christmas holidays so we can launch, or the massive overtime that reduces tester efficacy due to tiredness.

Our company needs to provide a safe, quality product to market quickly.  We can ill afford to delay the launch, lose market share or some other opportunity, or launch a problematic product because we want to grow our personnel.  The smart approach is to find many ways for your team to learn, but not sacrifice throughput or the quality of vetting of the product through testing.


Project Strategy and Decision Matrix

Posted on: February 3rd, 2016 by admin No Comments

There are many ways for us to evaluate the project we have discussed the monetary evaluation techniques in our books. These business measurements provide us with mechanisms to assess the business viability of the product.

There are also ways to evaluate the project strategy with decision-making tools like Pugh. In an earlier post we demonstrated how this tool can be used to evaluate the design against the objective and product requirements. We reviewed a variety of concepts against success factors. Those design factors had priority levels (weighting) so those design attributes that mattered the most received are weighted accordingly. This same approach works for project strategies as well.

Consider a project objective, and our team has generated 3 different strategies that will meet the objective, but we need to determine which one is the best fit (if there is one). We can set the table up with columns representing the strategies, and the rows project success criterion. We provide an example list of those below:

  • project cost
  • product development cost
  • project risk exposure
  • ongoing company support costs
  • time to market
  • present supply base considerations

Once we have determined the scope and have some idea of the business case we can refine our project objectives and way of achieving through the project strategy. Selecting a project strategy with a low probability of success is not helpful even if the business case is positive.


Subscribe to our Newsletter

Subscribe to our Newsletter
Email *
Single Sign On provided by vBSSO