By Jon M. Quigley
I saw a LinkedIn post yesterday about scope of testing during times of compressed schedule. The position was to test what is new in the software, and of that new, what is the most important, perhaps meaning what if it goes wrong, would be the worst for the client or customer. Generally this is probably a good idea. However, there are some drawbacks to this approach. This means no regression testing. Regression testing is testing of the old software features when we add new software features to the product.
Testing as above, is predicated on the belief that those things that we have changed or added, have no implication or impact on those features and functions that were already in place prior to this last iteration of the software. That may not be true. If we make changes to some software module that is used by other functions, we may miss testing a change in … Continue reading
Why formalize root cause analysis?
There are many approaches to determining the root of our problems. In the automotive world, there are two typical approaches, the 8D or the 8 Discipline, or the A3 (named for the paper size). We will not go into the details of either of these approaches but you can find the templates for these by clicking the links above.
There are some benefits to a formalized root cause analysis. We can think of 6 reasons as listed below:
Controls jumping to conclusions and just working on the symptom Repeatable Coordinates effort / focus Documents actions so we know what have tried and what is next Traceable for future events
Controls jumping to conclusions and just working on the symptom
A formalized approach to root cause analysis in such a way that will keep us from jumping to conclusions about the nature of the failure. Those that have read this … Continue reading
We are composing a glossary of test terms and we take one from that work today and discuss here.
A development organization can be structured in many ways. The development and testing can be essentially one department under one management structure, or these two areas, development and test can be separated each with a respective hierarchy. Like most things, there are benefits and drawbacks for each approach. The trick is to map the biggest benefit to your organization, or reduce the negative effects on your organization. All of this means the selection for organizational structure is likely not best left to an arbitrary decision.
I am sure we see the obvious benefit of having the test personnel integrated with the development staff. Those who have been in development for awhile no doubt understand the communications challenges that can come with separating groups when interaction between those groups is paramount for project success. So, the benefit of these two disciplines … Continue reading
There have been some twitter discussions going on about the validity of the term end-to-end (E2E) testing. I have been around this concept for many years and still see the validity in the term and the approach. To that end, I will describe as quick and briefly as I can since this is a blog post and not a book.
The various devices being developed (Device Under Test)
Vehicles are comprised of a myriad of subsystems. In this case we are going to look at the vehicle electrical / electronic architecture that consists of embedded electronic control units (ECU). These are component such as:
Instrument cluster Engine ECU Transmission ECU Antilocking Brakes System (ECU) Door control modules Telemetry systems
Each of these consists of many analog and digital inputs and outputs, along with data link communications. Each have specific expectations under certain conditions and the developers will work to deliver these algorithms and software that will ensure the system works … Continue reading
Verification and Validation
The definition for verification and validation can be found at:
We must express some disagreement with the activities associated with the individual areas. For example, testing is not limited to Validation. Testing is also a function of verification as we will use these techniques to understand if the instantiation of the product meets the specifications. Testing will be part of the quality assurance activities for the product – are we building what we said we would – through iterations of the product. Big bang is dead – meaning build everything at the beginning no longer happens. That method only works in products of the smallest scope. Rather, we will increment the feature content, verify the product build matches requirements delivered, log bugs and get ready for the next round. The next round fixes bugs, adds more of the requirements to the product content verify and so forth.
Likewise for Validation, there are things … Continue reading
Testing Complex and Embedded Systems What set of conditions could cause this event to occur?
When we have elicited all we can from the customer about fault information, it is time to proceed further in our analysis. This next step requires investigation of the design to understand how the symptom of failure described could happen by breaking down the hardware and software and the interactions within them to understand the improper behavior of the features to the customer. If the investigator is in the automotive, pharmacy, or food industries, they can resort to an immediate perusal of the Design Failure Mode and Effects Analysis (DFMEA) and the Process Failure Mode and Effects Analysis (PFMEA). If our investigator is lucky, they may find pointers to the cause of the issue in these documents.
To be successful, we need to perform a rigorous and systematic critique of the design—with enough follow up to ensure that any correctable issues have been resolved. Usually, … Continue reading
Project Closure and Team Disband
One of my favorites and often poorly executed part of a project is the project closure. Project closure is not just a simple wipe our hands together and smile, then proclaim that we are finished with the project. It is not even handing the customer our results and patting everybody on the back – and on to our congratulatory soda. The roots of the closure of the project happen in the very beginning of the project when we establish the scope of the project. If we are under contract the seeds we sow in the contract will be the harvest we reap during contract closure.
Project Contract Closure
To effectively close the project out, we need to have identified the project scope in a tangible way. That requires a way of tracing the top level scope (customer expectation) to the details of the project, which in the product development world, is typically contained in the … Continue reading
Regression Testing and Scope
A brief exchange on twitter on the topic of regression testing leads me to this post. Twitter, is not the place to really discuss this topic as there are too few characters and there are many factors that go into determining the scope of regression test. Regression testing is the testing of portions of the product that were built previous to this iteration. The testing is to ensure that in the addition of the new features to the product we have not inadvertently broken the product. A good definition of regression testing can be found from ISTQB.
One of the things we will consider in determining the scope of our regression testing will be the level at which our company is comfortable with the risk associated with the product in the field. If our company makes medical or perhaps automotive or aerospace products, we may generally be risk averse since the consequences of … Continue reading
Poor Excuse for Not Automating Testing
Recently I came across and participated in a social media exchange that proposed that automating product testing (software) was really not helpful. Their assertion was backed with comments about personnel new to testing will be unable to learn how to test.
Testing and System Complexity
System and software complexity, the number of interactions, permutations (configurations) and feature content, and failure mode consequences can make comprehensive testing via solely manual techniques very improbable. The only time solely manual methods may work, is for very simple and small scale products. The number of test cases for a moderately complex system can be in the thousands.
Testing and Repeatability
Done correctly, automating testing helps ensure repeatability of testing. This is important when we find a defect and we wish to trace the root cause of the problem, the exact set of steps of the test that evoked the defect. Of course, … Continue reading
We have recently posted how assumptions, left unquestioned can damage a project. It is similarly true for the product when we use models and simulation to generate our product. In the course of building these models, we will know some things for certain. Some attributes of the model we may think we know for certain but may in fact not be valid.
That is why we couple testing with our simulation and models. We will confirm those things we believe to be true and the many things we hope to be true, and probably uncover things we did not even consider to be true through testing. Those attributes that are connected to the desired outcome, we must understand so we can accurately predict the outcome. Below find a graphic that shows the process for building and understanding the model:
Assumptions are deadly when it comes to models. If we believe our simulations and the many … Continue reading