Creeping Requirements and Non-Zero Uncertainty

22 May 2022

Earlier in this series, we established that flight test is more than a job title. Rather, flight test exists to answer essential questions during product development related to verification and validation. Similar to the scientific method, flight test uses model validation and the build-up approach as fundamental strategies for efficiency and risk control. The next step to describing flight test should be one of prioritization, since testing everything would take forever and require infinite budget.

Flight test must be driven by requirements. We don’t execute flight test just to see what might happen, show off a product, or to have fun flying around (although flying is fun) – flight tests are engineered to address program requirements. Flight test planning starts with identifying what metrics are important. Program requirements structure the effort to generate model predictions that will be of interest. Depending on the type and phase of program, our source of requirements will vary. Flight test programs can be generally categorized as developmental, civil certification, or military.

The system engineering approach depends on requirements traceability, using verification and validation to determine that you "build it right" and that you "build the right thing".Any new aerospace product begins in the developmental phase, and some programs may become fully operational in this status with little need to ever shift their flight test philosophy. Examples of such programs could include commercial space launch vehicles or homebuilt kit planes. For projects in the developmental category, planned flight tests may be quite selective rather than comprehensive, looking only to validate the engineering and address the desired operational concept. This flexibility and fast pace can sometimes tempt a less efficient mentality of “fly-fix-fly” rather than the more disciplined model validation approach. Additionally, “requirements creep” can cause shifting goalposts for development teams, but the tests may be continuously tailored to safely consider the specific flight envelope areas of interest.

Civil flight test is much more structured by the requirements of your civil airworthiness authority. For much of aerospace, the defining requirements are United States Code of Federal Regulations, Title 14. Across the Atlantic, the European Union Aviation Safety Agency (EASA) Certification Specifications will apply, and to the north, the Canadian Aviation Regulations are implemented by Transport Canada. Not every nation has its own airworthiness regulations, and most nations have reciprocal agreements. Depending on the type and size of vehicle – fixed wing/rotorcraft/etc – different rules will apply, and which rules will be applicable to your program must be agreed upon in advance. Although regulations evolve with new legislation, this negotiated certification basis is fixed for five years from project initiation to shield from requirements creep. Unfortunately, published airworthiness regulations don’t fully define rules for evolving technology, so an “issue paper” is often required.

Defense contracting for aerospace products requires careful flight testDepartment of Defense flight test programs define their requirements in the contracting and acquisition process. These program “shall” statements give some direction to the flight test program, but are usually not detailed enough to write a test plan. Military standards and military specifications also provide some detailed guidance for what to test, such as MIL-STD-1797, but often the details of the test plan are customized through conversations between the program office, the developmental test (DT) team, and the operational test (OT) unit. DT supports the engineering data collection effort which results in technical documentation, while OT will assess operational suitability and develop tactical applications for the end users. Cooperative partnership from both flavors of military test is essential throughout the acquisitions process.

In each of these three generalized types of flight test programs, it’s important to consider success criteria. The hunger for data is never satisfied, so defined test objectives to meet specific requirements are essential to scope the effort. Just as risk is never driven to zero, some measure of uncertainty will  always remain – there are high and rapidly increasing costs to minimize both these factors. Next up: “what techniques do test pilots and flight test engineers use to collect data?

About the Author

Instructor Test Pilot

David Kern

David Kern is a graduate of the US Air Force Test Pilot School, with experience planning and flying hundreds of flight test missions for civil aircraft certification and military projects. He is an Associate Fellow with the Society of Experimental Test Pilots and Member of the Society of Flight Test Engineers. In his USAF career, he was the USAF F-16 project test pilot for the Collier Trophy-winning Automatic Ground Collision Avoidance System (AGCAS) and served as Instructor Test Pilot and Director of Operations at the USAF Test Pilot School, teaching all parts of the multi-engine and fighter curricula. In civil flight test, he served as a flight test pilot for the Aircraft Certification Service with the Federal Aviation Administration, and is currently a Flight Test Captain for a major airline. He holds a Master of Science in Flight Test Engineering and B.Sci. Electrical Engineering with a minor in Mathematics. He also holds an Airline Transport Pilot certificate with eight type ratings, is an active Certified Flight Instructor for instrument conditions, and has logged piloting time at the flight controls of over 80 different types of aircraft. His publications include "Flight Test Techniques for Active Electronically Scanned Array (AESA) Radar", "Accelerated Development of Flight Tested Sensors and Systems", and “Introduction to Fly-by-Wire Flight Control Systems: The professional pilot’s guide to understanding modern aircraft controls.”