login Register forgot password or username?
Cradle from 3SL, the complete Model Based Systems Engineering Toolsuite, specialising in requirements management, requirements capture, model based systems engineering and for systems engineering software, support and consultancy, the logical choice: Cradle from 3SL.

May 2009 [Cradle 5.7]

Measuring Performance and Effectiveness

In today’s world it’s fairly easy and inexpensive to instrument mechanical or electrical operations to collect performance data. Developers and testers can succumb to the temptation to collect gigabytes of data without first understanding what questions they should be structuring the data collection effort to answer. The questions are called Measures of Performance (MOP) and Measures of Effectiveness (MOE), and together they determine what constitutes successful operations for the system.

Both MOPs and MOEs can be derived from specifications or the Concept of Operations. Their selection should be based on the candidate MOP/MOEs’ ability to discriminate between levels of good performance. For example, the MOP Number of seconds required to go from zero to 60 mph may be very meaningful in selecting which car to buy for a daily commute that requires frequent merges with fast-moving traffic. However, the same measure would not be a good discriminator when selecting the best car for a daily commute on rural roads through small towns where the speed limit varies from 25 to 50 mph.

A MOP is a quantifiable measurement that can take any number of forms, e.g.,:

  • A simple count or measure: How many times did the car break down? How far did the car go before it broke down? How long did it take to repair the car?
  • An average: What was the average number of car trips taken between failures?
  • A rate: How many failures occurred for each hour of travel?
  • A percentage: What percentage of failures resulted in system shut down?

The MOP may be stated as a declarative statement: Average number of car trips between failures. A MOP can be developed for any parameter of interest: speed, acceleration, data flow rates, bit drop rates, ability to fit into and withstand the rigors of transport vehicles, and so on.

The data captured for a MOP indicates the system’s achieved level of performance, and allows the analyst to determine whether the system’s inherent performance requirements have been met. What they don’t provide is an assessment of the impact that level of performance has in terms of being able to accomplish the goal the customer had in mind when he/she started shopping for the system.

A MOE provides that impact assessment. Like the MOP, the MOE can be stated either as a question or a declarative statement. But the MOE is generally more complex, cross-referencing two or more MOPs. The MOE is stated in terms of accomplishment gained per cost incurred – miles travelled per gallon of fuel burned (MPG) is an example of an MOE in everyday use. It correlates the MOPs Miles travelled per tank of fuel burned and Fuel capacity in gallons. Individually these measures are interesting, but not meaningful – knowing you can go 400 miles on a tank of fuel becomes meaningful when you also know whether the tank holds 16 gallons or 30 gallons.

Not every “cost incurred” is as direct as money spent – it could be defective products scrapped, customers lost due to bad quality, time lost due to rework, critical tasks not accomplished, and so on. Customers should be interviewed to understand their specific needs: What are the critical tasks the system must perform? What is the cost of failure to perform those tasks? What is the tolerance for failure to perform? Once the list of MOPs and MOEs is compiled, the customer should be asked to provide a priority ranking of each candidate MOP/MOE relative to one another, and a weight assigned to each MOP and MOE indicating that measure’s contribution to overall performance assessment.

Understanding which MOPs and MOEs are most indicative of success in terms of the customer’s application is crucial to collecting the appropriate data to support a meaningful analysis of performance and effectiveness that provides a cost-effective contribution to demonstrating that the product meets all of the customer’s key function and performance requirements.

Please visit our new online shop to purchase single user versions of Cradle.