Archive for April, 2013

Verification or Validation

Posted on: April 21st, 2013 by admin No Comments

In some recent discussions with product development neophytes, we have heard a merging of the concepts of verification and validation.  Let us set the record straight.  Verification and Validation are not synonymous.  The World English Dictionary defines verification as “establishment of the correctness of a theory, fact, etc…”   and validation as “to confirm or corroborate”.  We can thank Barry Boehm (famed American Software Engineer) for the clarifying of these words as they apply to product development.

  • ·         Validation: Are we building the right product?
  • ·         Verification: Are we building the product right?

These two things are not one and the same thing, and in the world of product development, articulating these ideas correctly matters.  Let’s look into each of these closer.

In the definition of verification, the word correctness appears. Are we building the product right (or correctly)?  It is interesting that this word does not appear in the definition of validation.  The notion of correctness is important in that it clearly articulates the scope of verification activities.   Verification is measureable, as is correctness.  Correctness requires a foundational baseline, not subject to opinion. So does verification.  Correctness is black and white.  So should be verification. 

Validation on the other hand does not contain the notion of “correctness”.  Instead it emphasizes agreement or corroboration.  In the case of Boehm’s observation, “are we building or creating the right thing?”  Validation is about achieving agreement from the customer, that their opinion of the product matches your own.  Validation, therefore, is an opinion-driven or softer exercise – not driven by fact but by perspective and opinion, and opinions can change.  In the context of a project, the fluctuating and nebulous nature of customer opinion can create scope creep, poorly defined product requirements or poorly perceived project deliveries if validation activities are not continuous and consistent.  

With these definitions in mind, it is clear that validation must be treated as an on-going and continuous effort throughout the project lifecycle in order to ensure a well-received product.  However, it may not be so clear that so too is it for the verification activities during product development.  We have shown some of the reasons in previous blogs and will discuss at length in future blogs when we turn toward verification in more detail.

How do you ensure a well-received product?

Risk Reserves and Individual Projects

Posted on: April 20th, 2013 by admin No Comments

We have witnessed a disconcerting trend.  That trend is in consolidating risk reserves for projects into one, centrally managed bucket of money.  This bucket was once reserved for the unknown-unknowns and was called a Management Reserve.   However, more businesses are beginning to strip projects of their known-unknown Project Risk Reserves and placing calculated project contributions to this central bucket based upon a pre-defined risk profile for the project.   This practice has the particularly nasty consequence of inflating project IRRs and leading companies to make business decisions without having a sound understanding of the financial risks involved.  Additionally, with the Management Reserve divorced from specific projects and unknown risks, we now have a logistical chain to access to the contingency fund at a time when we have encountered a risk to our project. Typically in these events we are trying to institute project saving actions as prompt as possible, this is one more hurdle to achieving that objective.  Essentially we want to tie the contingency budgets to triggers ( and have as unfettered access as possible for quick response.

Risk reserves have always been a contentious topic.  Executives hate to “tie” up investment capital in projects only to get the money back at the end of the year.  This causes delays in implementing other valuable projects and detracts from shareholder value.  So the move to centrally manage them is not without logic.  However, the practice is extremely dangerous.

Understanding this rationale, project managers can ease executives’ fears by building risk plans which concretely tie the unique risks of the project contained within the risk register to the costs associated with the execution of the risk response plans and the probability of occurrence of the risk triggers.   By doing this, project managers build the case for the Project risk reserve.

One interesting observation I have made using this method is that some executives fail to understand the rationale of multiplying the probability of occurrence with the value of the risk response plan – the time honored way of calculating the management reserve.  They will typically respond, “Doesn’t it cost what it costs?”  This observation is intriguing in that this same method is utilized in decision tree analysis which is commonly taught in B-school classes.   But this is a topic for another Blog.

How do you communicate risk reserve needs to your executive team?

Documenting Dependencies

Posted on: April 19th, 2013 by admin No Comments

By: Rick Edwards

A good carpenter never blames his tools. There is also an aphorism “it is the poor musician who blames his instrument”.  Why do so many good project managers blame their scheduling tool when their project schedule doesn’t fit their desired schedule?

Project managers often struggle with documenting task dependencies utilizing the technology of the day.   In an environment where working “in-parallel” and “multi-tasking” is the norm, sequential relationships are hard to find.   Often, when project managers do define sequential relationships in their project plans, their WBS becomes so large and complex that the schedule becomes difficult to read.  Some project managers solve this problem by utilizing Start-to-Start, Finish-to-Finish and Start-to-Finish dependencies to help conceptualize these dependencies in the project plan.   While this works to a degree, it does add complexity to the plan and makes critical path analysis a bit cumbersome.

It has been my experience that it’s best to design the WBS to the level where I can build “soft” sequential relationships.  These sequential relationships can be as soft as placing relationships on tasks that I would prefer happen in a particular order.  These relationships are admittedly arbitrary.  However, this practice allows the project manager to identify when the project might experience problems, while maintaining a reasonable level of complexity within their WBS.

One important caveat to this approach is that it is extremely important to document in the project scheduling tool the nature of the relationship.  Is it a hard or soft dependency?  Is it technology or resource related?  If it’s soft, what risks would be incurred by the project if the dependency was broken.   All of this information must be well documented.  This will pay huge dividends later when you have to make trade-offs with your schedule.

How do you manage your dependencies in your project plan?

Off Topic Dad

Posted on: April 18th, 2013 by admin No Comments

I know this is way off topic; however I thought we should post this. Below is a letter my brother and I sent to the Veterans Administration. Our father was in the Special Forces and served multiple tours in Vietnam.  The US has been in wars for decades now, and we do not know the total cost in terms of the lives of our veterans.  The numbers are not just those lost in the field, but also those that come home and their families.  For example, in our house, you knew to close the cabinets slowly (I closed them on my finger and then gently pulled my finger from behind the door) so the door does not make a bang sound, awakening or startling our dad.  I thought this was normal until I saw the medical records when he went into the hospital.


I am submitting this claim for VA compensation on behalf of the family of Richard H. Quigley.  Our dad submitted a claim for VA compensation on the 3rd of August 2005, file number <number omitted> and there was no resolution on his claim before his passing on the 21st of October 2005.

One, My father died of small cell lung cancer as a result of exposure to Agent Orange.  My father told my brother and I of several occurrences where, when he was in Vietnam, planes would come over their location and spray Agent Orange on top of them.  He would then have to climb through the foliage and set charges to clear the dead vegetation.  The planes would come and spray again, the process continued until clearing the entire area around them.  The first doctor, Dr.<name omitted>, at the Durham North Carolina VA discussed with my father and us that his cancer was related to his exposure to Agent Orange and noted the same in his medical record.  Also enclosed in this claim is a copy of hand written notes from my father, wrote about the things he observed going on with himself since 1964 (his time in Vietnam).  It is my firm opinion that my father never pursued this claim due to issues discussed in items two and three.

Two, it is obvious from a review of my father’s medical and service record that he suffered from post turmeric stress disorder. It is clearly noted and documented.  However, this issue was insufficiently addressed, at least partially because my father’s belief that it would end his career in the Army.  This belief no doubt was derived from training, in the Army of the 50s’ and 60s’, where recruits were trained to believe that the Army was the only thing that mattered and their devotion to it must be unquestionable and without fail.  Even though he did not pursue treatment for this, the Army should have forced him to seek help.  He was a good soldier, and followed orders.  He never considered that he would be led astray or uncared for by his Army.  However, since his job performance was only minimally affected, it was not a concern for the Army.  This non-treatment of his PSTD only exacerbated the situation of his cancer and seeking treatment.  The culmination of these two items took our father away from us long before the cancer ended his life.

Three, My father felt that there was no reason for attempting to file anything against the Army or VA because, after his service of 22 years, 6 months, and 12 days and winning several Bronze Stars and Purple Hearts, he was awarded 30% disability. This included, 20% for Traumatic Arthritis and 10% for Hemorrhoids.  This documented PTSD, not appearing on his disability assessment is hard for us to understand.  This coupled with the Reduction in Force (RIFT) that reduced him back to the enlisted ranks after a long distinguished service as a Army Officer and just prior to his having sufficient time to retire as a commissioned officer.  The combination of these two items convinced my father that the Nation and Army, which he had given everything he had, betrayed him.  This betrayal coupled with the non-treatment of his PSTD also made him not file for any of the other benefits to which he was entitled to, such as Social Security.

Even at the end, when being assessed by the system for his level of disability, and its relationship to his service, required a second doctor, in addition to the one who already worked for the VA. This required numerous appointments, conflicting with other appointments and emergency trips to the hospital.  At one point, he was in the hospital, when he was to have an appointment with the VA assessment doctor for disability/compensation claim. However, the doctor would not come see him in his room, since he was already in the hospital.  The doctor’s nurse stated that it was because they needed to run test to see if he had cancer.  This would already be determined by the same test they were going to run by other VA doctors, which were treating his cancer.

In summation, there really is no way to monetarily compensation for the loss of our father, and the trials that have been endured as a result of his service to the Army and the United States.  He gave his all, and at no point was a burden to the Army or the government.  I know that the Army and VA owe him greatly for the way they have treated our father and for taking him away from us long before he should have been.  Enclosed you will find a copy of his disability claim from October 1981, his hand written list of symptoms that started in 1964, his DD 214, a copy of the letters of award from the Army for his Bronze Stars and Purple Hearts, his previous claim for compensation associated with his exposure to Agent Orange causing his cancer.

I wonder how many other individuals like Richard H. Quigley are out there given the number and duration of wars the US has been fighting.  The government did not do right by our father. We suspect this has not changed appreciably and more young folks will prematurely age and suffer from PTSD in silence. Their families will be robbed of a life with them even if they manage to come back from the war “in one piece”.   I hope at least when it is found out; it is documented and not summarily ignored as in the case of our dad. We fervently hope the government does something more akin to appropriate rather than wait out a man that is dying after giving so much.

Fundamentals and the FMEA

Posted on: April 17th, 2013 by admin No Comments

We have recently had a discussion on what it takes to quality assure software. The discussion focused on FMEA and the role it plays in quality assurance.  The discussion began to sound as if the FMEA was the panacea for poor quality. There is no silver bullet.

To be sure the FMEA (which is essentially a design review) helps, but is not the only thing we can or should do to secure the quality of the software product.  Like many things, you get out what you put into it.  A poorly executed FMEA (the amount of time, rigor, and skill that we put into it) will produce poor results and likely be a time consuming activity with little return.

While FMEA’s are typically more focused upon the hardware (DFMEA) or process (PFMEA) the tool can also be used on Systems level and even Software application.  We have even turned the FMEA methodology into a Project Management Risk Evaluation tool (see our download section extending beyond the system, product hardware or the manufacturing processes we use to achieve our objectives.

The FMEA is not the only tool.  In football, we do not abruptly start coaching the high-tech plays like the double reverse.  We start with the fundamentals like running, catching, and tackling. In product development we have fundamentals also, and we have been and will continue to write about that in the blog, however, we provide a list below of some of those fundamentals

The things we should be doing to improve the quality are:

1. Requirements Management (which includes – identifying, writing, and critical reviews)
2. Configuration Management
3. Define iterative systems / component package content (iterations and part of configuration management)
4. Sufficient time and rigorous testing (whether model based requirement otherwise)
5. Feedback from the testing and evaluations back into the system (both the project system and the product system)
6. Project planning that recognizes the dependencies and delivery qualities (what attributes of the delivery means the delivery is ”good” quality)
7. Monitoring the detailed project plan for imminent negative impacts
8. When we cut corners – know the impact of the shortcut on our project and product quality

I think we would be well served from the FMEA when we perform it closer to correct. Even for software or systems level critique the method can improve our design and help us understand the risks.

The point is, there is no one silver bullet.  A weak FMEA is part of the problem; a good FMEA is helpful but only address part of the areas of risk. The poor requirements handling and configuration management for example, have nothing to do with the FMEA, and really cause great harm in our work if these are lacking.  We waste time, deliver poor quality and add risks when these things are “broken”.  We would be better served by focusing on the fundamentals (of which an FMEA would be part) than developing new ways.

Requirements and Testing

Posted on: April 16th, 2013 by admin No Comments

We have our requirements, our iterative packages and content defined, and now the developers are producing the iterations. During this time, the verification and test folks have been creating the test cases or details on how we intend to confirm the product performance.

Our requirements provide the fodder for our testing.  For each requirement there must be a way to confirm that requirement has been delivered, fulfilled or otherwise met.  Confirmation the need is satiated is also part of the project closing activities.  We do not counsel testing only to the requirements, however this blog series is on requirements and we want to show the link between the requirements and product verification. How else will are you to know the product meets the desired outcome?  These test cases should be traced all the way back to the project scope via the product requirements (see the table below).

Scope Systems Requirement Component Requirement Test Case
Prod_Req_1.1 Systems_Req_1.1 Req_ECU1_1.1 TC_ECU1_1.1
Systems Req. 1.2 Req_ECU1  1.2 TC ECU1_1.2
Req_ECU2  1.3 TC ECU2_1.3


Traceability from scope through requirements to testing is fundamental and all too frequent missing in many product development organizations. Without this traceability, we end up wasting considerable time testing functions that are not working or delivered. Without the requirements, we do not know what to test independent of the packages delivered or even how to go about the testing. When the product arrives we are scurrying to figure out how to test based upon the myriad of and often based upon opinion on how the product works. The result of this neglect is uncertainty in the product delivery.  We do not know whether our confirmation of the product is adequate – risk in the field performance of the product and all that entails. Further, we find ourselves in the unenviable position of not knowing if our contractual obligations are truly slaked when the contract closure is due.

Requirements Systems Growth

Posted on: April 15th, 2013 by admin No Comments

Once we have our captured our requirements. We identify the substance of the content for the iterations from the development.  In fact, recall from the requirements prioritization blog post. That work has given us some insight into how the iterative packages could be developed and what content or capabilities are to be delivered.  In the end we will essentially have a road map of the functions with defined revision level of the products (hardware and software), associated feature content along with a schedule. We provide a brief example below:

  • Package A content
    • X features
  • Package B content
    • Y features
    • Corrections due to Package A
  • Package C content
    • Z features
    • Corrections due to Package B
  • Package D content
    • Corrections due to Package C

Each subsequent package will add some portion of the remaining requirements as well as corrections from previous package. This is the incremental, iterative development of the product or service.  If we are developing a system, we will manage the packages for all of the system interactions required to produce some defined level of system functionality for each package delivered.  Consider an automotive subsystem for example, that contains instrument cluster aspects as well as supporting elements from another controller on the network such as the brakes.  To produce a functioning set of features we will synchronize the delivery of these two components so we are able to work with the feature and learn and assess the product. We will discuss the assessment at greater length in subsequent blogs.

With each package we will steadily grow the product or service capability until we either:

    •  Meet the total list of product capabilities and the project objective,
    • Realize that we cannot meet the final intention we had desired and terminate the project.
    • Run out of money to develop the product or service
    • Run out of customer mandate for the product or service

Ultimately we learn from each of the iterations, addressing problems found and unexpected or unanticipated circumstances that always attend project work.  We reduce the risk in this iterative incremental approach rather than one release and a test at the end when there is no time to adjust and everybody is shouting to deliver something!

Specification and Requirements Reviews

Posted on: April 14th, 2013 by admin No Comments

One under or ineffectively used tool is the specification or requirements review, which is a form of design review. In this case we are reviewing the design while it is still easy and cheap to make adjustments.  Experience suggests, if we do not just forget to do this activity, it is a hastily arranged activity with little planning.  I know of no study that shows the tangible value of reviews, however, anybody that has ever written anything likely has seen the benefit of others reviewing their work.  The concept of the design review is similar and comparable to testing in that we are looking for defects.

As the old saw goes “as you sow, so shall yea reap” – if we spend little time and effort on the review we will gain little benefit.  To be of value we will need:

      • Available Talent
      • Material to review
      • Time to review material
      • Facilitator to manage the process 
      • Distribute material (with deadline)
      • Schedule the review
      • Have the review taking notes of changes needed
      • Take actions the deemed necessary from the review

To get the most out of the meeting we need a diversity of people to review. A review by the person authoring alone is pointless and better not to waste the time.  We should include those that will use the documents to create or test the end product (or process).  To that end, a systems document, for example, should be reviewed by those that are required to produce the sub-systems requirements as well as the end customer and the test group.

Once we have identified those that should be involved in the reviews, we will distribute the material for them to critique prior to reconvening.  The actual review should be a merciless critique of the document, like piranhas to a “raw steak”.  The author must be able to separate this critique from any personal implication – in other words not be overly defensive.

For what I have seen for reviews, ill prepared participants, I suggest that the day of the meeting those that are required to perform the critique must show their marked up copy of the requirements (specification) to gain entry to the meeting. If they are unable to show this up front diligence then they are not allowed into the discussion and perhaps their manager should talk to them about the reasons for design reviews.

Once we have aggressively yet objectively reviewed the document we will set about making any necessary changes we discovered in the review, always keeping in mind the revision controls. Upon updating we will then release the specification for the next set of activities.

Reviews are beneficial anytime there is a transition point from one group to other groups.  We breakdown some of the barriers and are better able to come to some common understanding by having an open critique of the requirements.

We will share some design review stories in later blogs.

Poor Requirements Portend Project Failure.

Posted on: April 13th, 2013 by admin No Comments

You need not been so technically skilled to be able to see how many project fail due to requirements.  We provide a brief list of the failings below:

    • Requirements control
    • Insufficient time
    • Did not include the sponsor or customer
    • Late delivery of requirements
    • Requirements traceability

We will not write any more on requirements control that is addressed in our other blog posts.  

Insufficient time spent on the requirements gathering is another area of failure.  In our haste to deliver the product we may perform little exploration of the use of the product.  We may have an over reliance upon standards. We cannot build a suitable product if the standards do not address the real world complexities of our application, and many do not. The scope of real world is beyond what can be boiled down to standards in many cases. Standards should be used with a jaundiced eye recognizing these limitations.  Standards help us reduce the time to develop the standards but are double edged sword that can cut us.

We do our sponsor and customer a disservice when we do not include them in the specification work. These people may not be able to speak the technical jargon; however, they are the people who will be the end user of the product.  Learning as much as you can about the application is essentially THE requirements.  Additionally, we will no doubt be contractually bound to meet the sponsors desired end result.

We have been in many projects where the requirements phase consumed so much of the time that the remaining product development time meant certain failure.  This often follows hard with we have no time to document the requirements now – we must work word of mouth.  This is a spiral to a very bad place.  The prioritization of the requirements we have discussed previously can help here.

Requirements traceability is coupled arm-in-arm with our configuration management. When we are unable to trace the requirements back to the scope of the project we are unable to efficiently close any project contract, as by definition closing the project is closing contract (meaning make sure all contractual obligations are accounted).

Requirements and Change Management

Posted on: April 12th, 2013 by admin No Comments

The initial product requirements provide the product baseline. Our project planning will be focused upon delivering meeting these requirements. In a phased development process, we will prioritize the requirements (shown earlier) and deliver iterations.  This staged delivery allows us to gain additional insight into the product.  We may learn things that necessitate changes to the original requirements.  We must manage these changes to secure the design integrity. This is especially true for system development where change to one component may have implications on other sub-systems and subsequently the overall system. 

  1. Identify the change needed
  2. Identify the consequences of the change
    1. Cost – material
    2. Time (schedule / delivery)
    3. Cost – NRE (Non-Recurring Engineering)
    4. Risks associated
  3. Document and review the consequences
  4. 4Accept change
    1. Update documentation
      1. Requirements, 
      2. Configuration Management plan, 
      3. Schedule 
      4. Contracts
  5. Execute to deliver the changes
  6. Confirm change has been made
    1. Verification the change request has made it into the product
    2. Contractual confirmation

Obviously, if the proposed change is not accepted, then all actions after the accept change (step 4) are not needed.  If the change is accepted, we will update the requirements each with new designators and links to the upper level requirements and ultimately the project scope.

Appropriate change management is not just a phone call to the developers and the change happens. We have written about some of the consequences of this method of change management in our configuration management posts on the blog.  Sometimes this method may seem the most expedient, and perhaps it even works – occasionally. However, it is playing roulette with your project and your customer to not manage this properly. More harm than any potential good can come from cutting corners for the sake of speed.

Subscribe to our Newsletter

Subscribe to our Newsletter
Email *
Single Sign On provided by vBSSO