Regression Testing

By Jon M. Quigley


I saw a LinkedIn post yesterday about scope of testing during times of compressed schedule. The position was to test what is new in the software, and of that new, what is the most important, perhaps meaning what if it goes wrong, would be the worst for the client or customer.  Generally this is probably a good idea. However, there are some drawbacks to this approach.  This means no regression testing. Regression testing is testing of the old software features when we add new software features to the product.

Testing as above, is predicated on the belief that those things that we have changed or added, have no implication or impact on those features and functions that were already in place prior to this last iteration of the software. That may not be true.  If we make changes to some software module that is used by other functions, we may miss testing a change in … Continue reading

Product Development Event In Gatlinburg?

I have long had an affinity for nature, having camped with my family when I was a child.  For years, I have been visiting Gatlinburg Tennessee. For someone who has spent much of his time in North Carolina, it is sad that I found this place accidentally when my wife won a three-day trip.  Since then, we have gone back at least once a year.  It is a great balance of nature, a cool town with plenty of things to do and wonderful restaurants.  I have many favorites for breakfast and for dinner.  I will give you many ideas for things to do and restaurants to go based upon my years of vacationing in Gatlinburg.

I have hiked the park, and when my son was younger we explored the myriad of small creeks in the area, looking for salamanders and crayfish.  On a hot day, there are few things more refreshing than sitting next to one of these rocky creeks … Continue reading

State Diagrams and Requirements

State diagrams can also help us develop the requirements and consider software and hardware requirements that may not be so easily evoked from the customer.  The state diagram describes the behavior or the software / system, specifically the number of states in which the product may be.  There will be transitions between states defined by certain actions. For example, an instrument cluster may go into a “sleep mode” when the key is outside of the ignition.  Inserting the key and moving it forward to start the vehicle; will transition the instrument cluster from sleep mode and the features that may be available, to “on” mode where different set if features and performance expectations will reside.  For example, in sleep mode, we may press a button to see the odometer or clock in the instrument cluster.  Moving to key forward we may see pointers move and the instrument cluster to perform a lamp check. That is, do all of the lights … Continue reading

Non-functional Requirements – Extensibility

Software Structure Defined

Non-functional requirements such as software extensibility can be very difficult to document as we likely do not know all of the future features or growth we can anticipate for the product as it matures.  Poorly managed, the code may descend into what is sometimes referred to as spaghetti code.  Instead of the document focusing solely upon the features, we can put some requirements for the structure in place before the coding even begins.  A way to do this is through the use of a software design description document.  This document may be especially important the first time working with the vendor or if there are some questions about the maturity level of the software department.  It may not be necessary to create a software design description document for every project.  An organization of a sufficient maturity level may not require such an artifact.  We can glean some understanding of an organization’s capability through reviews of … Continue reading

Technical Debt and Management

Business Imperative

There was a discussion going on LinkedIn and Twitter about technical debt, and management decisions. Sometimes business imperative trumps technical debt, but you must acknowledge the technical debt, and compare the business positives against the technical debt negatives.

Technical Debt

We should probably start by giving a definition of technical debt.  Technical debt is the future consequences from the deployment of a system or piece of software, specifically the negative consequences that we will likely have to find some way of paying back, hence the name debt.  In the cases where the technical debt is too high, we will lose any profit from the endeavor.

Technical Debt

In some instances we have no idea of the technical debt.  Perhaps our product has not been sufficiently developed (we cut corners) nor vetted (tested and evaluated). In these cases weighing the business upside against the long term downside is very difficult if not impossible. Without some information on … Continue reading

Fifth Dimension Testing

With fifth dimension testing, we use techniques more commonly employed by the “bad guys.” For example, we may execute techniques such as:

Fuzzing Response modification using genetic algorithms Input breakdown Overflows Underflows

This approach allows for evolution of our test collection. We can automate a significant number of these tests if we have a comprehensive, programmed means for discerning that a failure has occurred.

Compliance Testing

            We have discussed compliance testing earlier. This is known as testing to requirement. These requirements can be taken directly from a customer specification (when we have one) or derived internally from a requirements review or even both. Compliance testing is the primary method we use to ensure that we are meeting all specifications and any contractual obligations. Compliance testing is therefore obviously an important part of the project management activities. In the automotive industry, for example, we will verify that all gauges are meeting pointer sweep requirements in our instrument cluster (where we find the gauges on our dashboards).

            Compliance testing has the defect of only testing for specification requirements and may provide an inadequate measure of the true performance of the system.  Compliance testing will not show us how atypical or non-specified interactions with the product will impact the performance or product longevity. Additionally, it can be difficult to record by using specifications all of the unwanted … Continue reading

Securing the Quality of the Product / Project

Requirements management and configuration management are required for anything that even closely resembles effective testing.  Experience suggests failing in these two areas unnecessarily complicates the product verification activities, and we will show some of those traumas in the next few posts. 

An iterative and incremental product development process calls for reviews throughout the development process.  There have been times when people did not understand this concept and have attempted to test the product at the end – just prior to launch. Guess what? That rarely works. You will usually find problems (and sometimes very bad ones) just prior to the time you were going to launch the product. Think about the hundreds or thousands of interactions and interpretations that have to happen just right to launch the product like that! Not probable! 

Just like the project manager monitoring the schedule to evaluate whether the project is on time, so too should there be monitors in place to see … Continue reading

Fundamentals and the FMEA

We have recently had a discussion on what it takes to quality assure software. The discussion focused on FMEA and the role it plays in quality assurance.  The discussion began to sound as if the FMEA was the panacea for poor quality. There is no silver bullet.

To be sure the FMEA (which is essentially a design review) helps, but is not the only thing we can or should do to secure the quality of the software product.  Like many things, you get out what you put into it.  A poorly executed FMEA (the amount of time, rigor, and skill that we put into it) will produce poor results and likely be a time consuming activity with little return.

While FMEA’s are typically more focused upon the hardware (DFMEA) or process (PFMEA) the tool can also be used on Systems level and even Software application.  We have even turned the FMEA methodology into a Project Management Risk Evaluation tool (see … Continue reading

Configuration Identification from Requirements

Our configuration management is borne out of our requirements.  In our earlier blog post, we discussed how the project scope must be traceable to the requirements. In the case we present below, the project scope is a particular function or need to be met, for example a new feature or function for an automobile.  We want to integrate this new feature into our existing architecture.   To show how all of these are linked we show a snippet from an excel table (which could be how we trace our requirements).   

The top level scope is the left most column. We subsequently breakdown the product scope (our WBS will help us disaggregate) into systems requirements, and ultimately the individual component or subsystem requirements. In the case below we see that for Product Requirement 1.1 we have two systems requirements, and three component requirements.

Scope Systems Requirement Component Requirement Prod_Req_1.1 Systems_Req_1.1 Req_ECU1_1.1         Systems Req. 1.2 Req_ECU1  1.2 … Continue reading

Single Sign On provided by vBSSO