Testing – End-To-End
There have been some twitter discussions going on about the validity of the term end-to-end (E2E) testing. I have been around this concept for many years and still see the validity in the term and the approach. To that end, I will describe as quick and briefly as I can since this is a blog post and not a book.
The various devices being developed (Device Under Test)
Vehicles are comprised of a myriad of subsystems. In this case we are going to look at the vehicle electrical / electronic architecture that consists of embedded electronic control units (ECU). These are component such as:
- Instrument cluster
- Engine ECU
- Transmission ECU
- Antilocking Brakes System (ECU)
- Door control modules
- Telemetry systems
Each of these consists of many analog and digital inputs and outputs, along with data link communications. Each have specific expectations under certain conditions and the developers will work to deliver these algorithms and software that will ensure the system works as desired.
How do the developers know who the subsystem feature content and how it is to be designed?
Often there is a hierarchy of specifications starting with the systems level specifications. These documents describe the interactions and performance expectations at the system level. The scope of the document ranges from the typical function partition, that is what electronic control unit is responsible for what part of the feature visible instantiation. Let us take an easy example. The instrument cluster is required to display the road speed information. Where does this information originate? The system level functional specification may indicate that road speed comes from the ABS ECU, in a specific message. So, the function partition is message originating from the engine, will be sent to the instrument cluster. Ah, but how does the ABS derive the vehicle road speed? Well, the ABS will calculate the road speed based upon the drive axle rotation rate. So the entire function looks like:
- unit measures wheel speed rotation (PWM signal)
- Algorithm looks at the rate of the PWM and calculates the vehicle speed
- Converts calculation into serial communication format (CAN)
- Broadcasts this message using a specific period
- Instrument cluster
- Receives the road speed over the CAN link
- Strips data in the CAN message and converts it to corresponding angular displacement for the speedometer pointer
- Move pointer to that specific speed position (stepper motor)
Subsystem Development, Simulation and Testing
There will be software and hardware engineers working on each of these subsystems. The development and subsystem testing will include simulation of the external interfaces but not the actual interfaces to the subsystem. This is true for nearly all of the other subsystems, and this can get further confounded when the individual subsystems are being designed and developed by a multitude of suppliers. That is, the instrument cluster by supplier A, the ABS module by supplier B and so on. Therefore, each of the tests conducted by these suppliers via simulation of the rest of the system attached to their respective products, have assumptions on how the other portions of the system will work when the subsystems are assembled. These assumptions amount to risk for the product.
At some point, these parts will need to come together and comprise the vehicle. This is where end-2-end testing happens. Each of these tested subsystems will be put together to comprise the entirety of the vehicle systems. The assembling of these sub-assemblies will make it possible for us to build a fully operational (death star) vehicle. Sorry about that, I just watched a couple of the Star Wars movies.
Having the entirety of the system, nothing simulated, makes it possible for us to test the entirety of the system. The real, and the entire system from which we can test, learn and adapt the product to meet the final expected solution. Now we have a system that we then test end to end. The ABS unit, receives and processes the actual signal from the sensor that transforms the mechanical rotation into the pulse stream, that is received by the ABS embedded unit, and the calculation of the road speed is made. These are all measured and compared to calculations to determine the systems capability – does this work as we know it to work. We follow that processed signal, through to the data link, to the instrument cluster and the pointer displacement. We follow every part of that string of the function via testing, to ensure that each of the elements of the system respond as expected.
This is a very simple explanation of end-to-end testing, it is essentially the testing of each segment of the system, through all transition points and inputs. This testing can be distribute or apportioned to many development entities, each testing their segments, or it can tested as a complete system from beginning to end. There are much more complicated parts of the system. For example, think about the system as described along with vehicle telematics systems and perhaps of vehicle systems such as website, or other web-based interaction with the vehicle. End-to-end testing mitigates the risks associated with the subsystems development and testing that is performed using simulation. Additionally, this testing of the entire system, exploring the entire chain of events that deliver the feature, can help find inconsistencies in the system that may originate from many suppliers that are building these individual subsystems.