Regression Testing

By Jon M. Quigley

 

I saw a LinkedIn post yesterday about scope of testing during times of compressed schedule. The position was to test what is new in the software, and of that new, what is the most important, perhaps meaning what if it goes wrong, would be the worst for the client or customer.  Generally this is probably a good idea. However, there are some drawbacks to this approach.  This means no regression testing. Regression testing is testing of the old software features when we add new software features to the product.

Testing as above, is predicated on the belief that those things that we have changed or added, have no implication or impact on those features and functions that were already in place prior to this last iteration of the software. That may not be true.  If we make changes to some software module that is used by other functions, we may miss testing a change in … Continue reading

Why formalize root cause analysis?

Why formalize root cause analysis?

 

There are many approaches to determining the root of our problems.  In the automotive world, there are two typical approaches, the 8D or the 8 Discipline, or the A3 (named for the paper size).  We will not go into the details of either of these approaches but you can find the templates for these by clicking the links above.

There are some benefits to a formalized root cause analysis.  We can think of 6 reasons as listed below:

Controls jumping to conclusions and just working on the symptom Repeatable Coordinates effort / focus Documents actions so we know what have tried and what is next Traceable for future events

 

 

Controls jumping to conclusions and just working on the symptom

A formalized approach to root cause analysis in such a way that will keep us from jumping to conclusions about the nature of the failure.   Those that have read this … Continue reading

Independent Test and Verification

We are composing a glossary of test terms and we take one from that work today and discuss here.

A development organization can be structured in many ways. The development and testing can be essentially one department under one management structure, or these two areas, development and test can be separated each with a respective hierarchy.  Like most things, there are benefits and drawbacks for each approach.  The trick is to map the biggest benefit to your organization, or reduce the negative effects on your organization.  All of this means the selection for organizational structure is likely not best left to an arbitrary decision.

Communication

I am sure we see the obvious benefit of having the test personnel integrated with the development staff.  Those who have been in development for awhile no doubt understand the communications challenges that can come with separating groups when interaction between those groups is paramount for project success.  So, the benefit of these two disciplines … Continue reading

Testing – End-To-End

There have been some twitter discussions going on about the validity of the term end-to-end (E2E) testing. I have been around this concept for many years and still see the validity in the term and the approach. To that end, I will describe as quick and briefly as I can since this is a blog post and not a book.

The various devices being developed (Device Under Test)

Vehicles are comprised of a myriad of subsystems. In this case we are going to look at the vehicle electrical / electronic architecture that consists of embedded electronic control units (ECU). These are component such as:

Instrument cluster Engine ECU Transmission ECU Antilocking Brakes System (ECU) Door control modules Telemetry systems

Each of these consists of many analog and digital inputs and outputs, along with data link communications. Each have specific expectations under certain conditions and the developers will work to deliver these algorithms and software that will ensure the system works … Continue reading

Verification and Validation

Verification and Validation

The definition for verification and validation can be found at[1]:

 

We must express some disagreement with the activities associated with the individual areas. For example, testing is not limited to Validation. Testing is also a function of verification as we will use these techniques to understand if the instantiation of the product meets the specifications.  Testing will be part of the quality assurance activities for the product – are we building what we said we would – through iterations of the product.  Big bang is dead – meaning build everything at the beginning no longer happens. That method only works in products of the smallest scope.  Rather, we will increment the feature content, verify the product build matches requirements delivered, log bugs and get ready for the next round. The next round fixes bugs, adds more of the requirements to the product content verify and so forth.

Likewise for Validation, there are things … Continue reading

The Un-repeatable Fault

Testing Complex and Embedded Systems What set of conditions could cause this event to occur?

When we have elicited all we can from the customer about fault information, it is time to proceed further in our analysis. This next step requires investigation of the design to understand how the symptom of failure described could happen by breaking down the hardware and software and the interactions within them to understand the improper behavior of the features to the customer. If the investigator is in the automotive, pharmacy, or food industries, they can resort to an immediate perusal of the Design Failure Mode and Effects Analysis (DFMEA) and the Process Failure Mode and Effects Analysis (PFMEA). If our investigator is lucky, they may find pointers to the cause of the issue in these documents.

To be successful, we need to perform a rigorous and systematic critique of the design—with enough follow up to ensure that any correctable issues have been resolved. Usually, … Continue reading

CMMI makes Project Management Easier – Scope

CMMI and Project Management

There are intersections between Capability Maturity Model Integration (CMMI) and project management beyond the specific or obvious project management areas listed below in the staged representation:

Project Planning is Level 2 Project Monitoring and Control Level 2 Risk Management Level 3 Quantitative Project Management Level 5

There are other non-obvious intersections between CMMI and project management.  Consider requirements management and requirements development for example. We submit that CMMI makes project management easier.   Let us first look at requirements management and the connection to project management.

Requirements Management

This is a set of processes (in staged representation level 2) that we will use to evoke and understand the customer needs.  These are in level 2 because these are fundamental to success.  The specific goal is to ensure the requirements are managed and inconsistencies between project plans and work products are identified [CMMI].  Requirements are the detailed incarnation of the project scope. Poor requirements understanding will … Continue reading

FMEA Flashback!

By Jon M Quigley

We have discussed the Failure Mode Effects technique a few times in the past.  Though Failure Mode Effects and analysis seems to be a powerful tool, the problem is you do not know if the FMEA is effective and perhaps you will never know.  The Failure Mode Effects Analysis tool, theoretically, allows us to critique the product before we test.  How do we prove something did not happen because of a well conducted FMEA?  I have seen failures arise in testing and post launch.  In both instances I found it difficult to believe we missed the issue in the development or testing work as the failure seemed obvious.  I would later discover the FMEA was either missing or performed with a check box mentality, both of which are poor execution and not necessarily a testimonial of the apropos of the technique.  Of course this is only anecdotal evidence.

For me the real benefit is when … Continue reading

Technical Debt and Management

Business Imperative

There was a discussion going on LinkedIn and Twitter about technical debt, and management decisions. Sometimes business imperative trumps technical debt, but you must acknowledge the technical debt, and compare the business positives against the technical debt negatives.

Technical Debt

We should probably start by giving a definition of technical debt.  Technical debt is the future consequences from the deployment of a system or piece of software, specifically the negative consequences that we will likely have to find some way of paying back, hence the name debt.  In the cases where the technical debt is too high, we will lose any profit from the endeavor.

Technical Debt

In some instances we have no idea of the technical debt.  Perhaps our product has not been sufficiently developed (we cut corners) nor vetted (tested and evaluated). In these cases weighing the business upside against the long term downside is very difficult if not impossible. Without some information on … Continue reading

Product Testing and Verification Dictionary

Register for the free introduction to product (hardware and software) testing in a course online that covers the topics:

Stakeholders / Sponsors Project Scope Configuration Management Static Testing Techniques (inspections) Dynamic techniques Approaches to testing Regression , Re-test and Human Interactions Evaluation Putting it together with TIEMPO Test Management

This course includes the creation of a product testing dictionary (already more than 130 entries and 300+ more to go).  The dictionary has interaction mechanisms via comments that we will use to modify and adjust the dictionary entry.  Once the entries are secured, these will be turned into a digital dictionary and printed version.  All those that register and contribute to this dictionary will be mentioned in the front matter of the work and will receive a free digital copy of the work.  Presently there are more than 400 entries identified for the work that includes software AND hardware testing terms.

Register now for free at: https://valuetransform.com/training/login/index.php

Single Sign On provided by vBSSO