Creeping Requirements and Non-Zero Uncertainty

22 May 2022

Earlier in this series, we established that flight test is more than a job title. Rather, flight test exists to answer essential questions during product development related to verification and validation. Similar to the scientific method, flight test uses model validation and the build-up approach as fundamental strategies for efficiency and risk control. The next step to describing flight test should be one of prioritization, since testing everything would take forever and require infinite budget.

Flight test must be driven by requirements. We don’t execute flight test just to see what might happen, show off a product, or to have fun flying around (although flying is fun) – flight tests are engineered to address program requirements. Flight test planning starts with identifying what metrics are important. Program requirements structure the effort to generate model predictions that will be of interest. Depending on the type and phase of program, our source of requirements will vary. Flight test programs can be generally categorized as developmental, civil certification, or military.

Any new aerospace product begins in the developmental phase, and some programs may become fully operational in this status with little need to ever shift their flight test philosophy. Examples of such programs could include commercial space launch vehicles or homebuilt kit planes. For projects in the developmental category, planned flight tests may be quite selective rather than comprehensive, looking only to validate the engineering and address the desired operational concept. This flexibility and fast pace can sometimes tempt a less efficient mentality of “fly-fix-fly” rather than the more disciplined model validation approach. Additionally, “requirements creep” can cause shifting goalposts for development teams, but the tests may be continuously tailored to safely consider the specific flight envelope areas of interest.

Civil flight test is much more structured by the requirements of your civil airworthiness authority. For much of aerospace, the defining requirements are United States Code of Federal Regulations, Title 14. Across the Atlantic, the European Union Aviation Safety Agency (EASA) Certification Specifications will apply, and to the north, the Canadian Aviation Regulations are implemented by Transport Canada. Not every nation has its own airworthiness regulations, and most nations have reciprocal agreements. Depending on the type and size of vehicle – fixed wing/rotorcraft/etc – different rules will apply, and which rules will be applicable to your program must be agreed upon in advance. Although regulations evolve with new legislation, this negotiated certification basis is fixed for five years from project initiation to shield from requirements creep. Unfortunately, published airworthiness regulations don’t fully define rules for evolving technology, so an “issue paper” is often required.

Department of Defense flight test programs define their requirements in the contracting and acquisition process. These program “shall” statements give some direction to the flight test program, but are usually not detailed enough to write a test plan. Military standards and military specifications also provide some detailed guidance for what to test, such as MIL-STD-1797, but often the details of the test plan are customized through conversations between the program office, the developmental test (DT) team, and the operational test (OT) unit. DT supports the engineering data collection effort which results in technical documentation, while OT will assess operational suitability and develop tactical applications for the end users. Cooperative partnership from both flavors of military test is essential throughout the acquisitions process.

In each of these three generalized types of flight test programs, it’s important to consider success criteria. The hunger for data is never satisfied, so defined test objectives to meet specific requirements are essential to scope the effort. Just as risk is never driven to zero, some measure of uncertainty will   always remain – there are high and rapidly increasing costs to minimize both these factors. Next up: “what techniques do test pilots and flight test engineers use to collect data?”