Benchmarking Aids Performance Specifications

Benchmarking is well known as a powerful tool for measuring business performance relative to other companies, but it also makes a great tool for setting product performance criteria.

In the course of new product development, it's not uncommon for an engineering team to be asked to incorporate into a product some new features or technology with which they have no experience. While presenting a technical challenge for engineers learning the new subject matter, it also presents a challenge to marketing and quality engineering teams trying to make sure the new features meet the correct performance for the expected customer use. Specifying performance for standard features in one's industry may be second nature, but specifying performance criteria for something new can be a daunting task. Many companies turn to outside experts to help on the technical side. Bringing in outside consultants allows a company to quickly get short-term expertise in a particular area, while providing the in-house design team with a valuable resource as they come up to speed. There is still a need, however, to know how to specify performance criteria for the new features. Benchmarking is a great way to determine the performance requirements, especially if outside consultants are included in the benchmarking activities.

As an example, this consultant recently helped a test equipment company add audio capability to a high-end product. The product team was asked to include in a new piece of test equipment audio for voice prompts, training instructions, and possibly voice recording. Inclusion of audio had been attempted on a previous product but scrapped because the performance was unacceptable. In discussing the product goals with the project manager, it was clear there was no one in the company who knew how to specify the audio performance desired. The performance goal was "laptop computer type audio quality". This makes it difficult even for the consultant since there's no way to measure the quality of the consultant's deliverables. In this case, it was agreed that the first step in the project would involve benchmarking to set the baseline for audio performance. A certain laptop was voted by the project team to provide adequate audio quality for the intended application. This consultant made some general audio quality measurements on the laptop and some other small audio systems including: the consultant's laptop; a small prototype speaker system with amplifier and speakers similar to what was being requested; a commercial portable speaker system designed to work with laptops.

One issue that would impact the audio performance during use was fan noise from the test equipment. As part of the benchmarking activity, all systems (and some mixed combinations of systems) were tested sitting next to an operating piece of test equipment. Testing in a more "real world" environment is critical to benchmarking because it helps the project team understand performance from the customer's perspective. In this case, it was quickly evident that the performance of the laptop chosen by the project team was not good enough for the audio to be heard clearly by the user. The consultant's laptop was established as the bare minimum for audio performance, with a desire for any improvements possible.

With the minimum level of performance established from a qualitative and quantitative standpoint, it becomes much easier to make design trade-offs. In this case, some significant changes were needed to provide the necessary physical volume for the required acoustic elements and the electrical system had to be adjusted to guarantee enough power for the audio system. The resulting system clearly exceeded the performance of the consultant's laptop, but reasons for needing the changes would have been extremely difficult to communicate to the project team without the understanding gained from the benchmarking process.

To see the resume of the expert associated with this case study, see the link below.