FY16 DOD PROGRAMS - F-35 Joint Strike Fighter

FY16 DOD PROGRAMS

- Insufcient progress in verication of Joint Technical

Data, particularly those for troubleshooting aircraft fault

codes and for support equipment

- Delays in completing the required extensive and

time-consuming modications to the eet of operational

test aircraft which, if not mitigated with an executable plan

and contract, could signicantly delay the start of IOT&E

- Insufcient progress in the following areas which are

required for IOT&E:

▪ Development, integration, and testing of the Air-to-Air

Range Infrastructure instrumentation into the F-35

aircraft

▪ Flight testing to certify the Data Acquisition, Recording,

and Telemetry pod throughout the full ight envelope

▪ Development of other models, including the Fusion

Simulation Model, Virtual Threat Insertion table, and

the Logistics Composite Model

- Delays in providing training simulators in the Block 3F

conguration to the initial training centers and operational

locations

• Based on these ongoing problems and delays, and including

the required time for IOT&E spin-up, the program will not

be ready to start IOT&E until late CY18, at the soonest, or

more likely early CY19. In fact, IOT&E could be delayed

to as late as CY20, depending on the completion of required

modications to the IOT&E aircraft.

Progress in Developmental Testing

• Mission Systems Testing

- The program continues to pursue a cost- and

schedule-driven plan to delete planned mission systems

DT points by using other test data for meeting test point

objectives in order to accelerate SDD close-out. This

plan, if not properly executed with applicable data,

F-35 JSF 47

Executive Summary

Test Strategy, Planning, Activity, and Assessment

• The Joint Strike Fighter (JSF) Program Ofce (JPO)

acknowledged in 2016 that schedule pressure exists for

completing System Development and Demonstration (SDD)

and starting Initial Operational Test and Evaluation (IOT&E)

by August 2017, the planned date in JPO’s Integrated Master

Schedule. In an effort to stay on schedule, JPO plans to

reduce or truncate planned developmental testing (DT) in

an effort to minimize delays and close out SDD as soon as

possible. However, even with this risky, schedule-driven

approach, multiple problems and delays make it clear that

the program will not be able to start IOT&E with full combat

capability until late CY18 or early CY19, at the soonest.

These problems include:

- Continued schedule delays in completing Block 3F

mission systems development and ight testing, which

DOT&E estimates will likely complete in July 2018

- Delayed and incomplete Block 3F DT Weapons Delivery

Accuracy (WDA) events and ongoing weapons integration

issues

- Continued delays in completing ight sciences test points,

particularly those needed to clear the full F-35B Block 3F

ight envelope, resulting in a phased release of Block 3F

envelope across the variants, with the full Block 3F

envelope for F-35B not being released until mid-CY18

- Further delays in completing gun testing for all three

variants and recently discovered gunsight deciencies

- Late availability of veried, validated and tested Block 3F

Mission Data Loads (MDLs) for planned IOT&E and

aircraft delivery dates; DOT&E estimates the rst

validated MDLs will not be available until June 2018

- Continued shortfalls and delays with the Autonomic

Logistics Information System (ALIS) and late delivery of

ALIS version 3.0, the nal planned version for SDD, at

risk of slipping from early CY18 into mid-CY18

- Signicant, well-documented deciencies; for hundreds

of these, the program has no plan to adequately x and

verify with ight test within SDD; although it is common

for programs to have unresolved deciencies after

development, the program must assess and mitigate the

cumulative effects of these remaining deciencies on F-35

effectiveness and suitability prior to nalizing and elding

Block 3F

- Overall ineffective operational performance with multiple

key Block 3F capabilities delivered to date, relative to

planned IOT&E scenarios which are based on various

elded threat laydowns

- Continued low aircraft availability and no indications

of signicant improvement, especially for the early

production lot IOT&E aircraft

F-35 Joint Strike Fighter

FY16 DOD PROGRAMS

48 F-35 JSF

sufcient analytical rigor and statistical condence, would

shift signicant risk to operational test (OT), Follow-on

Modernization (FoM) and the warghter.

- This risky approach would also discard carefully planned

build-up test content in the Test and Evaluation Master

Plan (TEMP) and the Block 3F Joint Test Plan (JTP),

content the program fully agreed was required when

those documents were signed. The program plans to

“quarantine” JTP build-up test points, which are planned

to be own by the test centers, and instead skip ahead

to complex graduation-level Mission Effectiveness Risk

Reduction test points, recently devised to quickly sample

full Block 3F performance. Then, if any of the Block 3F

functionality appears to work correctly during the complex

test points, the program would delete the applicable

underlying build-up test points for those capabilities and

designate them as “no longer required.” However, the

program must ensure the substitute data are applicable and

provide sufcient statistical condence that the test point

objectives had been met prior to deleting any underlying

build-up test points. While this approach may provide a

quick sampling assessment of Block 3F capabilities, there

are substantial risks. The multiple recent software versions

for ight test may prevent the program from using data

from older versions of software to count for baseline test

point deletions because it may no longer be representative

of Block 3F. The limited availability and high cost of

Western Test Range periods, combined with high re-y

rates for test missions completed on the range, make it

difcult for the program to efciently conduct this testing.

Finally, the most complex capabilities in Block 3F have

only recently reached the level of maturity to allow them

to be tested, and they are also some of the most difcult

test points to execute (i.e., full Block 3F capabilities and

ight envelope).

- Historical experience indicates this approach, if not

properly executed, may delay problem discoveries and

increase the risk to completing SDD and increase the risk

of failure in IOT&E (as well as, much more importantly, in

combat). In fact, the program needs to allocate additional

test points – which are not in its current plans – for

characterization, root cause investigations, and correction

of a large number of the open high-priority deciencies

and technical debt described later in this report. The

completion of the planned baseline test points from

the Block 3F JTP, along with correction or mitigation

of signicant deciencies, is necessary to ensure full

Block 3F capabilities are adequately tested and veried

before IOT&E and, more importantly, before they are

elded for use in combat.

- Until recently, the Program Ofce estimated that mission

systems ight testing will complete in October 2017. It

now acknowledges the risk that this testing may extend

into early CY18.

▪ The October 2017 estimate was based on an inated

test point accomplishment rate and optimistically low

regression and re-y rates. The estimate also assumed

that the Block 3FR6 software, delivered to ight test

in December 2016, would have the maturity necessary

to complete the remaining test points and meet

specication requirements without requiring additional

versions of software to address shortfalls in capability.

However, this is highly unlikely, since several essential

capabilities – including aimed gunshots and Air-to-Air

Range Infrastructure – had not yet been ight tested

or did not yet work properly when Block 3FR6 was

released.

▪ The Services have designated 276 deciencies in combat

performance as “critical to correct” in Block 3F, but less

than half of the critical deciencies were addressed with

attempted corrections in 3FR6.

▪ Independent estimates from other Pentagon staff

agencies vary from March 2018 to July 2018 to

complete mission systems testing – all based on the

current number of test points remaining and actual

historic regression and re-y rates from the ight test

program. Even these estimates are optimistic in that

they account for only currently planned testing, which

does not yet include the activities needed to correct the

Services’ remaining high-priority deciencies.

• Flight sciences testing continues to be a source of signicant

discovery, another indication that the program is not nearing

completion of development and readiness for IOT&E. For

example:

- Fatigue and migration of the attachment bushing in the

joint between the vertical tail and the aircraft structure are

occurring much earlier than planned in both the F-35A

and F-35B, even with a newly designed joint developed to

address shortfalls in the original design.

- Excessive and premature wear on the hook point of the

arresting gear on the F-35A, occuring as soon as after only

one use, has caused the program to consider developing a

more robust redesign.

- Higher than predicted air ow temperatures were measured

in the engine nacelle bay during ight testing in portions

of the ight envelope under high dynamic pressure on both

the F-35A and F-35C; thermal stress analyses are required

to determine if airspeed restrictions will be needed in this

portion of the ight envelope.

- Overheating of the horizontal tail continued to cause

damage, as was experienced on BF-3, one of the

F-35B ight sciences test aircraft, while accelerating in

afterburner to Mach 1.5 for a loads test point. The left

horizontal inboard fairing surface reached temperatures

that exceeded the design limit by a signicant amount.

Post-ight inspections revealed de-bonding due to heat

damage on the trailing edge of the horizontal tail surface

and on the horizontal tail rear spar.

- Vertical oscillations during F-35C catapult launches were

reported by pilots as excessive, violent, and therefore a

safety concern during this critical phase of ight. The

program is still investigating alternatives to address this

FY16 DOD PROGRAMS

F-35 JSF 49

deciency, which makes a solution in time for IOT&E and

Navy elding unlikely.

Mission Data Load Development and Testing

• Mission data les, which comprise MDLs, are essential to

enable F-35 mission systems to function properly. Block 3F

upgrades to the U.S. Reprogramming Laboratory (USRL) –

where mission data les are developed, tested and validated

for operational use – are late to meet the needs for Block

3F production aircraft and IOT&E. These upgrades to the

Block 3F conguration, including the associated mission

data le generation tools, are necessary to enable the USRL

to begin Block 3F mission data le development. In spite

of the importance of the mission data to both IOT&E and

to combat, the Program Ofce and Lockheed Martin have

failed to manage, contract, and deliver the necessary USRL

upgrades to the point that fully validated Block 3F MDLs

will not be ready for IOT&E until June 2018, at the earliest.

• Operational units are also affected by the capability shortfalls

in the USRL to create, test and eld MDLs. The complete

set of Block 2B and Block 3i MDLs developed for overseas

areas of responsibility (AORs) have yet to undergo the full

set of lab and ight tests necessary to validate and verify

these MDLs for operational use. Because of the delays

in upgrading the USRL to the Block 3F conguration, the

Services will likely not have Block 3F MDLs for overseas

AORs until late 2018 or early 2019.

• In addition to the late Block 3F USRL upgrades, the required

signal generators for the USRL – with more high-delity

channels to simulate modern elded threats – have not yet

been placed on contract. As a result, the Block 3F MDLs

will not be tested and optimized to ensure the F-35 will

be capable of detecting, locating, and identifying modern

elded threats until 2020, per a recent program schedule.

The program is developing multiple laboratories in order

to produce MDLs tailored for partner nation-unique

requirements, some of which will have more high-delity

signal generator channels earlier than the USRL. The

program is considering using one of these other laboratories

for Block 3F MDL development and testing; however,

the MDL that will be used for IOT&E must be developed,

veried, validated, and tested using operationally

representative procedures, like the MDLs that will be

developed for the operational aircraft in the USRL.

Weapons Integration and Demonstration Events

• Block 3F weapons delivery accuracy (WDA) events are

not complete. These events, required by the TEMP, are key

developmental test activities necessary to ensure the full

re-control capabilities support the “nd, x, track, target,

engage, assess” kill chain. As of the end of November,

only 5 of the 26 events (excluding the gun events) had been

completed and fully analyzed. Several WDAs have revealed

deciencies and limitations to weapons employment (e.g.,

AIM-9X seeker status tone problems and out-of-date launch

zones for AIM-120 missiles). An additional 11 WDAs had

occurred, but analyses were ongoing. Of the 10 remaining

WDAs that had not been completed, 4 were still blocked

due to open deciencies that must be corrected before the

WDA can be attempted. However, the program did not

have time to x the deciencies, complete the remaining

WDAs and analyze them before nalizing Block 3FR6 in

late November for ight testing to begin in December 2016.

For example, recent F-35C ight testing to prepare for a

weapons event with the C-1 version of the Joint Stand-Off

Weapon (JSOW-C1) discovered weapon integration,

Pilot Vehicle Interface (PVI) and mission planning problems

that will prevent full Block 3F combat capability from

being delivered, if not corrected. These discoveries were

made too late to be included in the Block 3FR6 software,

the nal planned increment of capability delivered to ight

test for SDD. Also, multiple changes are being made late

in Block 3F development to mission systems re control

software to correct problems with the British AIM-132

Advanced Short-Range Air-to-Air Missile (ASRAAM)

missile and Paveway IV bomb, changes which could affect

the U.S. AIM-9X air-to-air missile and GBU-31 laser-guided

bomb capabilities, and may require regression testing of the

U.S. weapons.

• Block 3F adds gun capability for all variants. The F-35A

gun is internal; the F-35B and F-35C each use a gun pod.

Ground ring tests have been completed on all variants;

only on the F-35A has initial ight testing of the gun been

accomplished. Early testing of the air-to-ground and

air-to-air symbology have led to discovery of deciencies in

the gunsight and strang symbology displayed in the pilot’s

helmet – deciencies which may need to be addressed before

accuracy testing of the gun, aimed by the HMDS, can be

completed. Because of the late testing of the gun and the

likelihood of additional discoveries, the program’s ability

to deliver gun capability with Block 3F before IOT&E is at

risk, especially for the F-35B and F-35C.

Pilot Escape System

• The program completed pilot escape system qualication

testing in September 2016, which included a set of

modications designed to reduce risk to pilots weighing less

than 136 pounds.

- Modications include:

▪ Reduction in the weight of the pilot’s Generation III

Helmet Mounted Display System (HMDS), referred to

as the Gen III Lite HMDS

▪ Installation of a switch on the ejection seat which

allows lighter-weight pilots to select a slight delay in the

activation of the main parachute

▪ Addition of a Head Support Panel (HSP) between the

risers of the parachute.

- These modications to the pilot escape system were

needed after testing in CY15 showed that the risk of

serious injury or death is greater for lighter-weight pilots.

Because of the risk, the Services decided to restrict pilots

weighing less than 136 pounds from ying the F-35.

FY16 DOD PROGRAMS

50 F-35 JSF

• Twenty-two qualication test cases were completed

between October 2015 and September 2016, with variations

in manikin weight, speed, altitude, helmet size and

conguration, and seat switch setting. Data from tests

showed that the HSP signicantly reduced neck loads

under conditions that forced the head backwards, inducing

a rearward neck rotation, during the ejection sequence.

Data also showed that the seat switch reduced the “opening

shock” by slightly delaying the main parachute for lighter-

weight pilots at speeds greater than 160 knots. The extent

to which the risk has been reduced for lighter-weight pilots

(i.e., less than 136 pounds) by the modications to the

escape system and helmet is still to be determined by a

safety analysis of the test data. If the Services accept the

risk associated with the modications to the escape system

for the lighter-weight pilots, restrictions will likely remain in

effect until aircraft have the modied seat and the HSPs, and

until the lighter-weight Gen III Lite helmets are procured and

delivered to the applicable pilots.

• Based on schedules for planned seat modications,

production cut-in of the modied seat, and the planned

delivery of the Gen III Lite HMDS, the Air Force may be

able to reopen F-35 pilot training to lighter-weight pilots

(i.e., below 136 pounds) in early 2018. DOT&E is not aware

of the plans for the Marine Corps and the U.S. Navy to open

F-35 pilot training to the lighter-weight pilots.

• Part of the weight reduction to the Gen III Lite HMDS

involved removing one of the two installed visors (one

dark, one clear). As a result, pilots that will need to use

both visors during a mission (e.g., during transitions from

daytime to nighttime) will have to store the second visor in

the cockpit. However, there currently is not enough storage

space in the cockpit for the spare visor, so the program is

working a solution to address this problem.

• The program has yet to complete the additional testing and

analysis needed to determine the risk of pilots being harmed

by the Transparency Removal System (which shatters the

canopy rst, allowing the seat and pilot to leave the aircraft)

during off-nominal ejections in other than ideal, stable

conditions (such as after battle damage or during out-of-

control situations). Although the program completed an

off-nominal rocket sled test with the Transparency Removal

System in CY12, several aspects of the escape system have

changed since then (including signicant changes to the

helmet) which warrant additional testing and analyses.

Joint Simulation Environment (JSE)

• JSE is a man-in-the-loop, F-35 mission systems software-in-

the-loop simulation being developed to meet the operational

test requirements for Block 3F IOT&E. However, multiple

aspects of the JSE development effort continue to fall

signicantly behind schedule. The Program Ofce has been

negotiating with the contractor to receive the F-35 aircraft

and sensor models, referred to as “F-35 In A Box (IAB),”

but very limited progress was made in CY16. Also, delays

with security clearances for new personnel limited progress

on several aspects of the development and validation effort.

Although the Naval Air Systems Command (NAVAIR)

government team has begun installing hardware on their

planned timeline (facilities, cockpits, etc.), the team’s

progress in integrating the many different models (i.e.,

multi-spectral environment, threats, weapons) with F-35 IAB

has been severely limited, and the verication, validation and

accreditation of these models within JSE for use in IOT&E,

have effectively stalled. The F-35 program’s JSE schedule

indicates that it plans to provide a fully accredited simulation

for IOT&E use in May 2019; a schedule that carries high

risk of further slips without resolving these issues, and is

not credible. Without a high-delity simulation, the F-35

IOT&E will not be able to test the F-35’s full capabilities

against the full range of required threats and scenarios.

However, for the reasons above, it is now clear that the JSE

will not be available and accredited in time to support the

Block 3F IOT&E. Therefore, the recently approved IOT&E

detailed test design assumes only open-air ight testing will

be possible and attempts to mitigate the lack of an adequate

simulation environment as much as possible. In the unlikely

event the JSE is ready and accredited in time for IOT&E, the

test design has JSE scenarios that would be conducted.

Live Fire Test and Evaluation (LFT&E)

• The F-35 LFT&E program completed one major live re test

series using an F-35C variant full-scale structural test article

(CG:0001). Preliminary test data analyses:

- Demonstrated the tolerance of the vertical tail attachments

to high-explosive incendiary (HEI) projectile threats

- Conrmed the tolerance of the aft boom structures to

Man-Portable Air Defense System (MANPADS) threats

- Demonstrated vulnerabilities to MANPADS-generated

res in engine systems and aft fuel tanks. The data

will support a detailed assessment in 2017 of these

contributions to overall F-35 vulnerability.

• The test plan to assess chemical and biological

decontamination of pilot protective equipment is not

adequate; no plans have been made to test either the Gen II

or the Gen III HMDS. The Program Ofce is on track

to evaluate the chemical and biological agent protection

and decontamination systems in the full-up system-level

decontamination testing in FY17.

• The Navy conducted vulnerability testing of the F-35B

electrical and mission systems to electromagnetic pulses

(EMP).

• The 780th Test Squadron at Eglin AFB, Florida completed

ground-based lethality tests of the PGU-47/U Armor

Piercing High Explosive Incendiary with Tracer (APHEI-T)

round, also known as the Armor Piercing with Explosive

(APEX), against armored and technical vehicles, aircraft, and

personnel-in-the-open targets.

Suitability

• The operational suitability of all variants continues to be

less than desired by the Services. Operational and training

FY16 DOD PROGRAMS

F-35 JSF 51

units must rely on contractor support and workarounds that

would be challenging to employ during combat operations.

In the past year some metrics of suitability performance have

shown improvement, while others have been at or declined.

- Most metrics still remain below interim goals to achieve

acceptable suitability by the time the eet accrues 200,000

ight hours, the benchmark set by the program and dened

in the Operational Requirements Document (ORD)

for the aircraft to meet reliability and maintainability

requirements.

- Reliability growth has stagnated and, as a result, it is

highly unlikely that the program will achieve the ORD

threshold requirements at maturity for the majority

of reliability metrics, most notably Mean Flight

Hours Between Critical Failures, without redesigning

components of the aircraft.

Autonomic Logistics Information System

• The program failed to release any new ALIS capability

in 2016, but did release two updates to the currently elded

ALIS 2.0.1 software to address deciencies and usability

shortfalls. The program planned to test and eld ALIS 2.0.2,

including integration of propulsion data management, in

the summer of 2016, to support the Air Force declaration

of Initial Operational Capability; however, delays in

development and integration have pushed the testing and

elding into 2017.

• Because of the delays with ALIS 2.0.2, Lockheed Martin

shifted personnel to support that product line development.

This caused delays in the development schedule of ALIS 3.0,

the last major SDD software release. The program

acknowledged in August 2016 that it could not execute the

ALIS 3.0 schedule and developed plans to restructure this

ALIS release and the remaining planned ALIS capabilities

into multiple releases, including some that will occur after

SDD completion.

- The program’s restructuring of the ALIS capability

delivery plan divided the planned capabilities and security

updates for ALIS into four more versions: one version for

SDD (ALIS 3.0), with what the Program Ofce considered

to be needed for IOT&E, and three additional software

releases intended to be elded at 6-month intervals after

SDD completion, with the remaining content originally

planned for ALIS 3.0.

- The program plans to release software maintenance

updates midway between each of these four software

releases to address deciencies and usability problems, but

these releases will not include new capabilities.

• The Air Force completed its rst deployment of F-35A

aircraft using the modularized version of the ALIS squadron

hardware, called the Standard Operating Unit Version 2

(SOU v2), and software release 2.0.1 to Mountain Home

AFB, Idaho in February 2016. Difculties integrating the

SOU v2 into the base network interfered with connectivity

between the SOU v2 and the Mountain Home-provided

workstations, but did not affect connectivity of the SOU v2

with the main Autonomic Logistics Operating Unit (ALOU)

in Fort Worth, Texas.

Air-Ship Integration and Ship Suitability

• The program completed the last two ship integration DT

periods in 2016 – both referred to as “DT-III” – one with

the F-35B in November aboard the amphibious assault ship

USS America, and one with the F-35C in August aboard the

aircraft carrier USS George Washington. Test objectives

included expanding the ight clearances for shipboard

operations with carriage of external weapons, night

operations, and Joint Precision Approach Landing System

(JPALS) integration testing. For both periods, operational

and test units accompanied the deployment to develop

concepts of operations for at-sea periods.

• The specialized secure space set aside for F-35-specic

mission planning and the required Offboard Mission

Support (OMS) workstations is likely unsuitable for regular

Air Combat Element (ACE) operations on the Landing

Helicopter Dock (LHD) and Landing Helicopter Assault

(LHA)-class assault ships with the standard complement

of six F-35B aircraft, let alone F-35B Heavy ACE

congurations with more aircraft. Similarly, for F-35C

operations onboard CVN, adequate secure spaces will be

needed to ensure planning and debrieng timelines support

carrier operations.

• The F-35C DT-III included external stores, including bombs,

but only pylons with no AIM-9X missiles on the outboard

stations (stations 1 and 11) due to the F-35C wingtip

structural deciency. The U.S. Navy directed a proof-of-

concept demonstration of an F-35C engine change while

underway, a process that took several days to complete.

ALIS was not installed on USS George Washington, so

reach-back via satellite link to the shore-based ALIS unit was

required, similar to previous F-35C test periods at sea, but

connectivity proved troublesome.

• The F-35B DT-III deployment included an engine

installation due to required maintenance, along with a lift fan

change proof-of-concept demonstration. The Marine Corps

deployed with an operational SOU v2 on USS America.

Cybersecurity Testing

• The JSF Operational Test Team (JOTT) continued to conduct

cybersecurity testing on F-35 systems, in partnership with

certied cybersecurity test organizations and personnel, and

in accordance with the cybersecurity strategy approved by

DOT&E in February 2015. In 2016, the JOTT conducted

adversarial assessments (AA) of the ALIS 2.0.1 SOU, also

known as the Squadron Kit, at Marine Corps Air Station

(MCAS) Yuma, Arizona, and the Central Point of Entry

(CPE) at Eglin AFB, Florida, completing testing that began

in the Fall of 2015. They also completed cooperative

vulnerability and penetration assessments (CVPA) of the

mission systems ALOU at Edwards AFB, California, used

to support developmental testing, and the operational ALOU

in Fort Worth, Texas. The JOTT, with support from the

FY16 DOD PROGRAMS

52 F-35 JSF

Air Force Research Laboratory (AFRL) also completed a

limited cybersecurity assessment of the F-35 air vehicle

in September 2016, on an F-35A aircraft assigned to the

operational test squadron at Edwards AFB. These tests were

not conducted concurrently as originally planned, so end-to-

end testing of ALIS, from the ALOU to the air vehicle, has

not yet been accomplished. An AA of the operational ALOU

was scheduled for early December 2016, which would

complete a full assessment (CVPA and AA) of each ALIS

2.0.1 component.

• The cybersecurity testing in 2016 showed that the program

has addressed some of the vulnerabilities identied during

earlier testing periods; however, much more testing is needed

to assess the cybersecurity structure of the air vehicle and

supporting logistics infrastructure system (i.e., ALOU, CPE,

Squadron Kit) and to determine whether, and to what extent,

vulnerabilities may have led to compromises of F-35 data.

The scope of the cybersecurity testing must also expand to

include other systems required to support the elded aircraft,

including the Multifunction Analyzer Transmitter Receiver

Interface Exerciser (MATRIX) system which is used by

contractor maintenance technicians, the USRL, avionics

integration labs, the OMS and training simulators.

Follow-on Modernization

• The program continued making plans for Follow-on

Modernization (FoM) for all variants, also referred to as

Block 4, which is on DOT&E oversight. The program

intends to award the contract for the modernization effort

in 2QCY18 with developmental ight testing beginning

in 3QCY19. Four increments of capability are planned,

Blocks 4.1 through 4.4. Blocks 4.1 and 4.3 will provide

software-only updates; Blocks 4.2 and 4.4 will include

signicant avionics hardware changes as well as software

updates. Improved Technical Refresh 3 (TR3) processors

with open architecture, designed to make adding, upgrading

and replacing components easier, are planned to be added in

Block 4.2.

• The program’s plans for FoM are not executable for

a number of reasons including, but not limited to the

following:

- Too much technical content for the production-schedule-

driven developmental timeline

- Overlapping increments without enough time for

corrections to deciencies from OT to be included in the

next increment

- High risk due to excessive technical debt and deciencies

from the balance of SDD and IOT&E being carried

forward into FoM because the program does not have a

plan or funding to resolve key deciencies from SDD prior

to attempting to add the planned Block 4.1 capabilities

- Inadequate test infrastructure (aircraft, laboratories,

personnel) to meet the testing demands of the capabilities

planned and the multiple congurations (i.e., TR2, TR3,

and Foreign Military Sales)

- Insufcient resources for conducting realistic operational

testing of each increment

System

• The F-35 Joint Strike Fighter (JSF) program is a tri-Service,

multi-national, single-seat, single-engine family of strike

aircraft consisting of three variants:

- F-35A Conventional Take-Off and Landing (CTOL)

- F-35B Short Take-Off/Vertical-Landing (STOVL)

- F-35C Aircraft Carrier Variant (CV).

• The F-35 is designed to survive in an advanced threat

environment (year 2015 and beyond) using numerous

advanced capabilities. It is also designed to have improved

lethality in this environment compared to legacy multi-role

aircraft.

• Using an active electronically scanned array (AESA) radar and

other sensors, the F-35 with Block 3F is intended to employ

precision-guided weapons, such as the GBU-12 Laser-Guided

Bomb (LGB), GBU-31/32 Joint Direct Attack Munition

(JDAM), GBU-39 Small Diameter Bomb (SDB), Navy Joint

Stand-Off Weapon (JSOW)-C1, and air-to-air missiles such

as AIM-120C Advanced Medium-Range Air-to-Air Missile

(AMRAAM), and AIM-9X infrared guided short-range

air-to-air missile.

• The SDD program was designed to provide mission capability

in three increments:

- Block 1 (initial training; two increments were elded:

Blocks 1A and 1B)

- Block 2 (advanced training in Block 2A and limited combat

capability in Block 2B)

- Block 3 (limited combat capability in Block 3i and full

SDD warghting capability in Block 3F)

• The F-35 is under development by a partnership of countries:

the United States, Great Britain, Italy, the Netherlands, Turkey,

Canada, Australia, Denmark, and Norway.

Mission

• The Combatant Commander will employ units equipped with

F-35 aircraft in joint operations to attack targets during day or

night, in all weather conditions, and in heavily defended areas.

• The F-35 will be used to attack xed and mobile land targets,

surface units at sea, and air threats, including advanced aircraft

and cruise missiles.

Major Contractor

Lockheed Martin, Aeronautics Division – Fort Worth, Texas

FY16 DOD PROGRAMS

F-35 JSF 53

Test Strategy, Planning, and Resourcing

• Preparations for IOT&E. In 2016, the JPO acknowledged

schedule pressure for starting IOT&E in August 2017, as

planned in the Integrated Master Schedule created in 2012.

Due to multiple problems and further delays, the program will

not be able to start IOT&E until late CY18, at the earliest, and

more likely early CY19, but it could be as late as CY20 before

required modications are completed to IOT&E aircraft. The

issues that will not allow IOT&E to start as planned include:

- Continued schedule delays in completing Block 3F mission

systems development and ight testing

▪ The program’s plan to deliver the “Full SDD Warghting

Capability” version of Block 3F software – now referred

to as version 3FR6 – was signicantly delayed. It was

planned for release to ight test in February 2016,

according to the program’s latest mission systems

software and capability release schedule, but did not

begin ight test until early December 2016 (10 months

late). However, during this time, the program released

several “Quick Reaction Cycle” (QRC) versions of

software to quickly resolve deciencies that were

preventing the completion of key test points, like

weapons deliveries. Due to these delays, along with the

recently acknowledged SDD funding shortfall, software

versions 3FR7 and 3FR8 have fallen off the program’s

schedule. However, ongoing delays in maturing some

of the capabilities and new problem discoveries continue

to prevent testing of some planned Block 3F capabilities

and will almost certainly require additional unplanned

releases of Block 3F software.

▪ DOT&E estimates that mission systems ight testing will

not complete prior to July 2018, based on the number of

Block 3F baseline mission systems test points to go, the

monthly average mission systems test point completion

rate observed for CY16 to date, and the average

regression, discovery and developmental test point rate

of 63 percent experienced so far in CY16. This estimate

also includes a decrement of 11 percent for test points to

be designated “no longer required,” the percentage used

by the Program Ofce to account for efciency in CY16

planning of test point accomplishment objectives.

- Delayed and incomplete Block 3F developmental testing

Weapons Delivery Accuracy (WDA) events and ongoing

weapons integration issues

▪ WDA events – key developmental test activities

necessary to ensure the full re-control capabilities

work together to properly support the “nd, x, track,

target, engage, assess” kill chain – are not complete. As

of the end of November, only 5 of the 26 WDA events

(excluding gun events) had been completed and fully

analyzed.

▪ Several WDAs have revealed deciencies and limitations

to weapons employment (e.g., AIM-9X seeker status

tone problems and out-of-date launch zones for AIM-120

missiles). An additional 11 WDAs had occurred, but

analyses are ongoing. Of the 10 remaining WDAs,

4 were still blocked due to open deciencies that must

be corrected before the WDA can be attempted, but the

program did not have time to complete and analyze them

before nalizing Block 3FR6.

- Continued delays in completing ight sciences test points,

particularly those needed to provide the F-35B Block 3F

ight envelope for operational use

▪ Through the end of November, ight sciences testing on

all variants was behind the plan for the year. Although

the program planned to complete Block 3F testing on

the F-35A in October, testing continued into December,

with weapons separations and regression testing of new

software to be completed.

▪ Flight sciences test point completion for CY16 was

5 percent behind for the F-35B and 23 percent behind

for the F-35C as of the end of November. The program

plans to complete Block 3F ight sciences testing

in August 2017 with the F-35C and by the end of

October 2017 with the F-35B, the latter being 10 months

later than planned in the program’s Integrated Master

Schedule.

▪ Due to the delays with completing ight sciences testing,

the program plans a phased release of the Block 3F

envelope across all three variants, with the full Block 3F

envelope for the F-35B not being released until mid-

CY18.

- Further delays in completing gun testing for all three

variants and recently discovered gunsight deciencies

▪ Block 3F adds gun capability for all three variants. The

F-35A gun is internal; the F-35B and F-35C each use a

gun pod. Differences in mounting make the gun pods

unique to a specic variant, i.e., a gun pod designated

for an F-35B cannot be mounted on an F-35C aircraft.

Flight sciences testing of the gun has occurred with the

F-35A; discoveries required control law changes to the

ight control software and delayed the start of mission

systems gun testing on the F-35A from September 2016

to December 2016. Although the F-35B and F-35C have

completed ground rings of their gun pods, airborne

ight sciences gun testing (i.e., airborne ring) for the

F-35B and F-35C has yet to be accomplished.

▪ Besides the ongoing delays with software and gun

modications, both DT and OT pilots have reported

concerns from preliminary test ights that the air-to-

ground gun strang symbology, displayed in the helmet,

is currently operationally unusable and potentially unsafe

to complete the planned testing due to a combination of

symbol clutter obscuring the target, difculty reading

key information, and pipper stability. Also, for air-to-air

employment, the pipper symbology is very unstable

while tracking a target aircraft; however, the funnel

version of the air-to-air gunsight appears to be more

stable in early testing.

▪ Fixing these deciencies may require changes to the

mission systems software that controls symbology

FY16 DOD PROGRAMS

54 F-35 JSF

to the helmet, or the radar software, even though the

program recently released the nal planned version of

ight test software, Block 3FR6. Plans to begin ight

testing of aimed gunshots, integrated with mission

systems, which requires aiming with the helmet, on the

F-35A were planned for fall of 2016, but had slipped to

December 2016, at the soonest, before this new problem

with the gun symbology was discovered.

▪ F-35B ground test ring of its gun pod was accomplished

in July 2016 and ight testing is planned to begin in

January 2017; the F-35C conducted rst ground ring

in November 2016; ight testing is planned to begin in

March 2017.

- Late availability of veried, validated and tested Block 3F

MDLs

▪ Failure by the program to plan for, procure, and provide

the necessary Block 3F upgrades and the associated

Mission Data File Generation (MDFG) tools to the USRL

has caused delays in developing, testing, and verifying

mission data loads for IOT&E.

▪ If Block 3F MDFG tools are delivered in early CY17,

veried, validated and tested MDLs will not be available

for IOT&E until June 2018 (15 months later) at the

soonest, which is late to need for both IOT&E and

elding of Block 3F.

▪ In collaboration with partner nations, the program is

developing multiple laboratories to produce MDLs

tailored for country-unique requirements. Although

these other laboratories may provide additional capacity

for developing and testing MDLs, the MDL that will be

used for IOT&E must be developed, veried, validated,

and tested using operationally representative procedures

involving the USRL.

- Continued shortfalls and delays with ALIS and late

delivery of ALIS software version 3.0, the nal planned

version for SDD, which is at risk of slipping from

early-CY18 into mid-CY18

▪ The program has failed to deliver increments of ALIS

capability as planned. No new capability has completed

testing in 2016, although the program had planned to

eld ALIS 2.0.2, with the propulsion integration module

included, by August 2016 to support the Air Force IOC

declaration, but continued problems caused this to slip

into early CY17.

▪ The program restructured the ALIS capabilities delivery

plan in 2016 and moved content planned for ALIS

3.0 – the last version to be developed during SDD – to

post-SDD ALIS development and elding. Despite the

delays and deferred content, IOT&E will still evaluate

the suitability of the F-35 with ALIS in operationally

realistic conditions.

- Signicant, well-documented deciencies resulting in

overall ineffective operational performance of Block 3F,

hundreds of which will not be adequately addressed with

xes and corrections veried with ight testing within

SDD

▪ The program, Services, JOTT, and DT and OT pilots

recently conducted a review of the status and priority

of open deciency reports (DRs). This review was a

follow-on from a review in the spring of 2016, where

the stakeholders reviewed all the open DRs and created

a rank-ordered list of 263 priority deciencies to be

addressed by the program. The review team later pared

the list down to 176 priority DRs, with 12 being brought

forward to the JPO’s Conguration Steering Board

(CSB); 7 for decision and 5 for CSB awareness. In the

review in the fall of 2016, the stakeholders reviewed

the approximately 1,200 open deciencies, including

the original 176 priority DRs, plus 231 new DRs since

Feb 2016, minus 55 that had been corrected, to create an

updated DR list. This time, however, the team prioritized

the open DRs into one of 4 priorities: priority 1 DRs

are “service critical,” and the Services will not eld

the aircraft unless these DRs are xed; priority 2 have

signicant impact that may, when combined with other

DRs, lead to mission failure; priority 3 carry medium

impact and should be addressed by the program, but

maybe not within SDD; and priority 4 have low impact.

The review team identied 72 DRs as priority 1 and

204 DRs as priority 2, for a total of 276 DRs to address

within SDD or risk elding deciencies that could lead to

operational mission failures during IOT&E or combat.

▪ While these deciencies must be addressed to some

degree during the remaining time in development,

the nal planned software load, Block 3FR6, which

started ight test in December 2016, only included

attempted xes for less than half of the 276 priority

1 and 2 DRs. Corrections to these deciencies will

need to be developed, tested in the labs (if possible)

and then ight tested, since the labs have proven to not

be an adequate test venue for verifying corrections to

deciencies identied during ight testing. However,

the current schedule-driven program plans to close out

SDD testing in 2017 do not include enough time to x

these key deciencies, nor time to verify corrections in

ight test. There is risk in attempting to verify DR xes

only in the lab because the labs proved to not always

be representative of the actual aircraft for detecting

problems or verifying xes for stability problems. The

labs are also not able to adequately replicate the demands

on the mission systems like open air testing does, such as

infrared and radar background clutter and terrain-driven

multipath reections of radio-frequency emissions from

threat emitters, so most xes to deciencies will require

ight testing.

- Overall ineffective operational performance with multiple

key Block 3F capabilities to date

▪ Three independent assessments conducted during the

past 6 months rate the F-35 as red or unacceptable

(not all assessments used the same scoring criteria) in

most critical combat mission areas: The Air Force’s

IOC Readiness Assessment (IRA) of Block 3i, an OT

FY16 DOD PROGRAMS

F-35 JSF 55

community assessment of Block 3FR5.03 based on

observing develomental testing, and an assessment by the

JOTT of the capability of Block 3FR5.05 to perform the

planned mission trials in the IOT&E, based on observing

and assisting with DT.

▪ In July, the Air Force completed their IRA report. The

assessment was based on a limited series of events

conducted with six Block 3i-congured aircraft,

including test missions in Close Air Support (CAS), Air

Interdiction (AI), and Suppression/Destruction of Enemy

Air Defenses (SEAD/DEAD). The assessment noted

unacceptable problems in fusion and electronic warfare

and, concerning the CAS mission, determined that the

Block 3i F-35A does not yet demonstrate equivalent CAS

capabilities to those of fourth generation aircraft.

▪ In August, an F-35 OT pilot from Edwards AFB,

California, briefed the results of an OT community

assessment of F-35 mission capability with Block

3FR5.03, based on observing developmental ight

test missions and results to date. This OT assessment

rated all IOT&E mission areas as “red,” including

CAS, SEAD/DEAD, Offensive Counter Air (OCA)

and Defensive Counter Air (DCA), AI, and Surface

Warfare (SuW). Several DT Integrated Product Team

representatives also briefed the status of different F-35

mission systems capabilities, most of which were rated

“red,” and not meeting the entrance criteria to enter

the “graduation level” mission effectiveness testing.

Trend items from both the OT and IPT briengs were

limitations and problems with multiple Block 3F system

modes and capabilities, including Electro-Optical

Targeting System (EOTS), Distributed Aperture System

(DAS), radar, electronic warfare, avionics fusion,

identication capabilities, navigation accuracy, GPS,

datalinks, weapons integration and mission planning.

▪ In November 2016, the JOTT provided an assessment of

a later version of Block 3F software – version 3FR5.05

– based on observing and assisting with F-35 DT ight

operations and maintenance. The JOTT assessment

made top-level, initial predictions of expected IOT&E

results of the F-35 with Block 3FR5.05 against planned

scenarios and realistic threats. For mission effectiveness,

the assessment predicted severe or substantial operational

impacts across all the planned IOT&E missions (similar

to the list of missions above) due to observed shortfalls

in capabilities, with the exception of the Reconnaissance

mission area, which predicted minimal operational

impact. Unlike the other assessments, the JOTT also

assessed suitability, predicting mixed operational impacts

due to shortfalls for deployability (from minimal to

severe), severe impacts for mission generation, and

substantial impacts for training and logistics support.

- Continued low aircraft availability, especially for the early

production lot IOT&E aircraft. The program has still

not been able to improve aircraft availability, in spite of

reliability and maintainability initiatives, to the goal of

60 percent, which is well short of the 80 percent necessary

to conduct an efcient IOT&E and to support sustained

combat operations. As a result, IOT&E will likely take

longer than currently planned and suitability, along with

elded operations, will be adversely affected.

- Late delivery of the JSE, a man-in-the-loop simulator

expected for IOT&E, which required the test team to create

a test design that attempts to mitigate the high likelihood

that it will not be available. Some IOT&E measures of

effectiveness will not be fully resolved without a veried,

validated and accredited simulator to evaluate the F-35 in an

operationally realistic, dense threat environment.

- Progress in verication of Joint Technical Data (JTD) is

behind plans to complete within SDD, particularly those

for troubleshooting aircraft fault codes and for support

equipment. As of September 2016, the program had veried

approximately 83 percent of all JTD modules, but just over

50 percent of those associated with support equipment.

While symptomatic of an immature system, the lack of

veried JTD makes the completion of aircraft maintenance

more difcult and forces maintainers to rely more heavily

on submitting electronic requests to the contractor for help

or to seek assistance from contractor representatives at eld

locations.

▪ The program has made signicant progress in verifying

JTD for sustaining the aircraft’s low observable signature,

primarily by completing verications on an F-35A

damaged in 2014 by an engine re

▪ All Block 3F JTD must be written and veried prior to the

start of IOT&E

- Delays in completing the extensive and time-consuming

modications required to the eet of operational test aircraft

which, if not mitigated with an executable plan and contract,

could signicantly delay the start of IOT&E.

▪ The program is developing and working plans with

Lockheed Martin and the Services to provide production-

representative operational test aircraft, with the necessary

instrumentation, to start IOT&E. Although it was part of

the agreed-to entrance criteria for IOT&E, the program

currently does not have an adequate plan to provide test

aircraft that meet the TEMP criteria for entering IOT&E

until late-2018, at the earliest, and possibly as late as 2020.

Extensive modications are required on all of the TEMP-

designated OT aircraft; 155 different modications (known

to date) are necessary between all variants and all lots of

aircraft (Lots 3 through 5) to bring the IOT&E aircraft

to the required production-representative conguration,

although no single aircraft requires all 155 modications.

Additional discoveries and modications are likely as the

program nishes SDD.

▪ The Program Ofce and the Services are considering using

later lot aircraft with an alternate instrumentation package.

However, to date, no analyses of the adequacy of the

alternate instrumentation has been completed; nor is there

a contract to design, build and test alternative packages.

- Insufcient progress in the development and testing of

modeling, simulations, and instrumentation required for

IOT&E.

FY16 DOD PROGRAMS

56 F-35 JSF

▪ Flight testing to allow the Data Acquisition Recording

and Telemetry (DART) pod to be used throughout the

full Block 3F ight envelope during IOT&E, including

during simulated weapons releases when the weapons

bay doors will cycle open, has not yet been planned, put

on contract or completed. The DART pod is required for

collecting data during IOT&E.

▪ Flight testing of the Air-to-Air Range Infrastructure

(AARI) – as integrated with the F-35 and required for

adequacy of the open air ight test trials – has not yet

been completed. AARI is used to support battle-shaping

of air-to-air engagements by modeling weapon y-outs

and accounting for endgame effects to remove aircraft

“shot down” by another aircraft or ground threat.

The program must begin testing AARI and allow for

corrections of deciencies during ight testing, to ensure

AARI is adequate for IOT&E.

▪ Integration of AARI and associated range simulators

with the F-35 to indicate inbound missiles on cockpit

displays is required for an adequate evaluation of

open air missions. Within the aircraft, the Embedded

Training (ET) function is intended to support live/virtual/

constructive training using a mixture of real and virtual

entities (e.g., missiles, ground systems, and aircraft). To

avoid intermingling data from real and virtual entities,

as it may cause issues within the F-35, the contractor

developed a separate model, the Fusion Simulation

Model (FSM), to emulate fusion functionality for virtual

entities within ET. The current FSM implementation has

signicant deciencies that make the model so inaccurate

that some required capabilities may not be usable for

IOT&E. Although a properly functioning FSM is

required for IOT&E, the program had not yet completed

contract actions for xes to correct the FSM deciencies

within SDD and prior to IOT&E, but was apparently

developing plans and intended to award contract actions

for at least some of the work on FSM by the end of

January 2017.

▪ Virtual Threat Insertion (VTI) is a function inside of FSM

that correlates virtual threat parametric data supplied by

AARI with data from tables embedded within the FSM

to provide cockpit display indications to the pilot for

threat activity (i.e., a surface-to-air missile launched).

The reference tables for VTI are incomplete and do

not include all threats planned for use in IOT&E. The

program was also apparently planning to update the VTI

tables, but this was also not yet on contract.

▪ The Logistics Composite Model (LCOM), which will

be used to support assessments of suitability measures

including sortie generation rate and logistics footprint

– two key performance parameters in the ORD – is still

under development. Seven versions of the model will

be needed to cover the three variants as well as partner-

unique and shipborne operations.

- The program is behind in developing and elding training

simulators, referred to as F-35 Full Mission Simulators

(FMS), to train pilots, both at the integrated training

centers for initial F-35 pilot training and at the operational

locations. The FMS is a multi-ship, man-in-the-loop, F-35

mission systems software-in-the-loop simulation using

virtual threats, it is used to train both U.S. and partner

pilots.

▪ In 2014, the program moved simulator development

from Akron, Ohio to Orlando, Florida. As a result of the

move, the program lost experienced personnel, suffered

from shortfalls in required stafng, and fell behind in

meeting the hardware and software demands of the

rapidly growing pilot training requirements.

▪ In March 2016, following an inspection of the Block 2B

FMS, evaluators reported 203 test discrepancies;

173 remained open, 4 were canceled, 2 were pending

corrections, and 24 had been closed and corrections

included in the next build of FMS for Block 3i.

▪ The Block 3i FMS is behind the planned schedule

for elding. The rst Block 3i FMS is scheduled for

delivery to Marine Corps Air Station Iwakuni, Japan, in

December 2016, followed by two more FMS delivered to

partner countries.

▪ Because of delays in delivering the Block 3i FMS, the

Block 3F FMS is even further behind schedule. Although

earlier plans included delivering the Block 3F FMS in

CY17, the program is now replanning the schedule.

▪ Since the FMS runs F-35 mission systems software, it

requires Block 3F mission data les, integrated with

virtual threats, to build the threat environment simulation

(TES). It currently takes up to 20 months for the

program to build the TES after new mission data les

are available, hence pilots will not have Block 3F FMS,

with the USRL-produced mission data les, available

for training prior to IOT&E. Alternatively, the program

may elect to use the contractor-developed DT mission

data les for the Block 3F FMS. However, doing so

would make the training in the FMS not operationally

representative, as those mission data les do not

accurately portray the TES to the pilot. Without an

adequate Block 3F FMS, the OT pilots will have to rely

on the available Block 3F OT aircraft for training.

• The JOTT completed detailed test designs for accomplishing

IOT&E. DOT&E approved the designs in August 2016. The

test designs include comparisons of the F-35 with the A-10

in the Close Air Support role, the F-16C (Block 50) in the

Suppression/Destruction of Enemy Air Defenses (SEAD/

DEAD) mission area, and the F-18E/F in the air-to-surface

strike mission area. The JOTT has begun detailed test

planning based on these designs, and will provide these plans

to DOT&E for approval, prior to the start of IOT&E.

• Block Buy. The program and Services continue to pursue

a “Block Buy” for production lots 12 through 14. This

multi-year procurement scheme is based on a partial group

of the partner nations, designated as “Full Participants,”

funding a 2 percent Economic Order Quantity (EOQ) in FY17

and another 2 percent EOQ in FY18. Other partner nations,

FY16 DOD PROGRAMS

F-35 JSF 57

designated “Partial Participants,” would procure Lot 12 as a

single year lot procurement, then commit to procuring Lots 13

and 14 as a part of the Block Buy and provide funding of

4 percent EOQ in FY18. Similar to the Partial Participants, the

Services would procure Lot 12 as a single year procurement

and fund 4 percent EOQ in FY18, but maintain the options

for single year procurements in Lots 13 and 14. Altogether,

452 F-35 aircraft would be procured under the Block Buy

scheme, on top of the 490 aircraft (346 for the U.S. Services)

previously procured in lots 1-11, all purchased without the

informed results of an IOT&E. As reported in the FY15

DOT&E Annual Report, many questions remain on the

prudence of committing to the multi-year procurement of a

Block Buy scheme prior to the completion of IOT&E:

- Is the F-35 program sufciently mature to commit to the

Block Buy with the ongoing rate of discovery while in

development?

- Is it appropriate to commit to a Block Buy given that

essentially all the aircraft procured thus far require

modications to be used in combat? The Services will

have accepted delivery of 346 aircraft through Lot 11,

before the additional aircraft are purchased via the Block

Buy scheme.

- Would committing to a Block Buy prior to the completion

of IOT&E provide the contractor with needed incentives

to x the problems already discovered, as well as those

certain to be discovered during IOT&E?

- Would the Block Buy be consistent with the “y before

you buy” approach to acquisition advocated by the

Administration, as well as with the rationale for the

operational testing requirements specied in title 10,

U.S. Code, or would it be considered a “full rate” decision

before IOT&E is completed and reported to Congress, not

consistent with the law?

• Follow-on Modernization (FoM). The program continued

making plans for all variants for FoM, also referred to as

Block 4, which is on DOT&E oversight. The program intends

to award the contract for the modernization effort in 2QCY18

with developmental ight testing beginning 3QCY19. Four

increments of capability are planned, Blocks 4.1 through

4.4. Blocks 4.1 and 4.3 will provide software-only updates,

Blocks 4.2 and 4.4 will add hardware as well as software

updates. Improved Technical Refresh 3 (TR3) processors are

planned to be added in Block 4.2. However, the plans for FoM

are not executable for a number of reasons including, but not

limited to, the following:

- Too much technical content for the allocated developmental

timeline. Experience with the F-22 modernization program

indicates the planned 18- to 24-month cycle for FoM is

insufcient for the large number of planned additional

capabilities; the F-22 increments had less content plus

software maintenance releases between new capability

releases.

- High risk of carrying excessive technical debt and

deciencies from Block 3F and the balance of SDD into

FoM. The planned 4-year gap between the planned nal

release of Blocks 3F in 2017 and Block 4.1 in 2021 lacks

resources (i.e., funding and time) for a bridge software

maintenance release to reduce technical debt and verify

Block 3F IOT&E corrections of deciencies. Although

the unresolved technical debt is an SDD shortfall, it sets

up FoM to fail due to unrealistic planning and inadequate

resourcing.

- Insufcient time for conducting adequate operational

testing for each increment.

▪ The current plan for F-35 Block 4.2 only has 18 months

for DT ight test and 6 months for OT&E, despite

containing substantially more new capabilities and

weapons than F-22 Block 3.2B.

▪ For comparison, the F-22 Block 3.2B program planned

approximately two years for DT ight test and one

year of OT&E spin-up and ight test; F-22 Blocks 3.1,

3.2A and 3.2B have suffered delays and problems

accomplishing testing due to inadequate test resources

and schedule.

- Inadequate test infrastructure (aircraft, laboratories,

personnel) to meet the testing demands of the capabilities

planned.

▪ The current end-of-SDD developmental test aircraft

drawdown plan is still being developed. However, any

plan that signicantly reduces the F-35 test force in 2017

and 2018 – precisely when the program needs this test

force to nish the delayed SDD Block 3F Joint Test Plan

(JTP) and correct remaining deciencies with additional

Block 3F updates in preparation for IOT&E – would

result in shortfalls of the necessary resources to provide

full Block 3F capability.

▪ A robust test force will also be required to be available

through 2020 to correct the inevitable new discoveries

from IOT&E and produce a nal Block 3F software

release that provides a stable foundation for adding the

new Block 4.1 capabilities.

▪ The program plans to award contracts to start

simultaneous development of Blocks 4.1 and 4.2 in 2018,

well prior to completion of IOT&E and having a full

understanding of the deciencies that will emerge from

IOT&E; without any budget or time to x deciencies

from earlier development.

▪ The requirement to integrate and test multiple

congurations simultaneously (TR2 and TR3) will

require additional time, test aircraft, and lab resources; a

problem that must be addressed as the program considers

plans for the eet of test aircraft for FoM.

▪ As of the writing of this report, the program’s published

FoM plan would have reduced test infrastructure

from 18 DT aircraft and 1,768 personnel, which are

still heavily tasked to complete ongoing Block 3F

development, to just 9 aircraft and approximately

600 personnel to support FoM. Clearly, this plan is

grossly inadequate. However, the program and Services

were in the process of replanning the test infrastructure

for FoM and had not yet provided the results.

FY16 DOD PROGRAMS

58 F-35 JSF

▪ Both the Air Force and the Navy conducted independent

studies in 2016 to determine what infrastructure and test

periods for FoM would be adequate. Neither report had

been released as of the time of this report. DOT&E has

requested to see the preliminary results of the Air Force

study, but the Air Force has refused to provide them, citing

the fact that the results are not nal and the report is in

draft.

- Signicant technical and schedule risk due to Block 4.1

adding new capabilities to the already-stretched TR2

avionics hardware, along with Block 4.2 attempting to

simultaneously migrate to a new open-architecture TR3

processor while adding many signicant new capabilities.

▪ For Block 4.1, the program plans to add multiple new

capabilities to the TR2 avionics hardware, even though

this architecture already has memory and processing

limitations running the full Block 3F capabilities, resulting

in avionics stability issues and capability limitations.

▪ For Block 4.2, the program plans to simultaneously add

multiple signicant new software capabilities while

migrating to a new avionics hardware conguration,

including a new open-architecture TR3 processor and new

electronic warfare (EW) hardware. This will be far more

challenging than the program’s problematic re-hosting of

Block 2B software, designed to run on TR1 processors, on

to TR2 processors to create Block 3i. Although no new

capabilities were added in Block 3i, signicant avionics

stability issues were manifested due to technical debt and

differences with the new architecture.

▪ The program claims the new F-35 Block 4.2 software,

which will be designed to run on new TR3 processors,

will also be backward-compatible to run in the hundreds

of early production aircraft with TR2 processors, but has

not yet presented a plan to demonstrate this. Based on the

current TR2 architecture capacity limitations with Block

3F, this claim is unlikely to be realized.

▪ Instead of adding lab capacity to support testing of

processor loads with the additional mission systems

capabilities, the program plans to reduce the lab

infrastructure supporting development. The program has

already retired the Cooperative Avionics Test Bed aircraft

– a decision that has increased the burden on ight testing

with F-35 aircraft.

▪ Current JPO projections for modifying aircraft with TR2

processors to the TR3 processor conguration extend into

the 2030s. As a result, up to three congurations of test

aircraft and labs may be needed if the program requires

more advanced processors than the TR3 planned for

Block 4 (i.e., the next Block upgrade requiring even more

processing capacity driving the need for new processors).

▪ The program also does not yet have an executable plan

to provide a mission data reprogramming lab in the TR3

conguration in time to support Block 4.2 OT and elding.

- Attempting to proceed with the current unrealistic plans

for FoM would be to completely ignore the costly lessons

learned from Block 2B, 3i and 3F development, as well as

those from the F-22 program. As learned from the F-22

Blocks 3.1, 3.2A and 3.2B, an overly aggressive plan with

inadequate resources ultimately takes longer, costs more

and delays needed capabilities for the warghter.

• This report includes assessments of the progress of testing to

date, including developmental and operational testing intended

to verify performance prior to the start of IOT&E. Test ights

and test points are summarized in two tables on the next page.

- For developmental ight testing, the program creates

test plans by identifying specic test points (discrete

measurements of performance under specic ight

test conditions) for accomplishment, in order to assess

the compliance of delivered capabilities with contract

specications.

▪ Baseline test points refer to points in the test plans that

must be accomplished in order to evaluate if performance

meets contract specications.

▪ Non-baseline test points are accomplished for various

reasons. Program plans include a budget for some of

these points within the capacity of ight test execution.

The following describes non-baseline test points.

» Development points are test points required to

“build up” to, or prepare for, the conditions needed

for assessing specication compliance (included in

non-baseline budgeted planning in CY16).

» Regression points are test points own to ensure

that new software does not introduce shortfalls in

performance for requirements that had previously

been veried using previous software (included in

non-baseline budgeted planning in CY16).

» Discovery points are test points own to investigate

root causes of newly discovered deciencies or to

characterize deciencies so that the program can

design xes for them (not included in planning in

CY16).

▪ As the program developed plans for allocating test

resources against test points in CY16, the program

included a larger budget for non-baseline test points

(development and regression points) for mission

systems testing, as the plans for the year included

multiple versions of software, requiring regression and

developmental test points be completed. For CY16

mission systems testing, planners budgeted an additional

69 percent of the number of planned baseline test points

for non-baseline test purposes (e.g., development and

regression points), the largest margin planned for a CY

to date. This large margin was planned because the

program anticipated the test centers would need points

for building up to the baseline points that would be own

for specication compliance as well as for completing

regression of multiple versions of Block 3F software. In

this report, growth in test points refers to points own

over and above the planned amount of baseline and

budgeted non-baseline points (e.g., discovery points and

any other added testing not originally included in the

formal test plan).

FY16 DOD PROGRAMS

F-35 JSF 59

▪ The continued need to budget for non-baseline test

points in the CY16 plan is a result of the limited

maturity of capabilities in the early versions of

mission systems software. Although the program

planned to complete developmental ight testing in

January 2017, according to their Integrated Master

Schedule, developed after the program was restructured

in 2010, delays in issuing mature software to ight

test made it clear that regression and development test

points would still be needed throughout CY16.

▪ Cumulative SDD test point data in this report refer to the

total progress towards completing development at the end

of SDD.

- Limited operational testing was also conducted throughout

the year to support assessments of weapon capability,

deployment demonstrations, shipborne testing, and the Air

Force’s IOC declaration; results of these limited tests are

used to support assessments throughout this report.

TEST FLIGHTS AS OF NOVEMBER 30, 2016

All Testing Flight Sciences

Mission

Systems

All Variants F-35A F-35B F-35C

2016 Planned 1,221 151 359 237 474

2016 Actual 1,362 226 386 271 479

Dierence from Planned +11.5% +49.7% +7.5% +14.3% +1.1%

Cumulative Planned 7,624 1,587 2,242 1,469 2,326

Cumulative Actual 7,853 1,697 2,318 1,479 2,359

Dierence from Planned +3.0% +6.9% +3.4% +0.7% +1.4%

Prior to CY16 Planned 6,403 1,436 1,883 1,232 1,852

Prior to CY16 Actual 6,492 1,471 1,932 1,209 1,880

TEST POINTS AS OF NOVEMBER 30, 2016

All

Testing

Flight Sciences Mission Systems

All

Variants

F-35A F-35B F-35C

Block 3F

Budgeted

Non-

Baseline

Other

Block 3F

Baseline

Budgeted

Non-

Baseline

Block 3F

Baseline

Budgeted

Non-

Baseline

Block 3F

Baseline

Budgeted

Non-

Baseline

2016 Test Points Planned (by type)

8,774 1,205 159 1,876 115 1,695 146 1,189 1,534 855

2016 Test Points Accomplished (by type)

7,838 1,303 156 1,783 115 1,304 136 975 1,534 532

Dierence from Planned

-10.7% +8.1% -1.9% -5.0% 0.0% -23.1% -6.8% -18.0% 0.0% -37.8%

Points Added Beyond Budgeted Non-Baseline

(Growth Points)

304 0 54 0 250

Test Point Growth Percentage (Growth

Points/Test Points Accomplished)

3.9% 0.0% 3.0% 0.0% 25.6%

Total Points (by type) Accomplished in 2016

8,142 1,459 1,952 1,440 3,291

Cumulative Data

Cumulative System Design and Development

(SDD)

Planned Baseline

51,060 12,225 15,994 12,604 10,237

Cumulative SDD Actual Baseline

50,278 12,327 15,970 12,279 9,702

Dierence from Planned

-1.5% +0.8% -0.2% -2.6% -5.2%

Est. Baseline Test Points Remaining

6,649 100 1,726 1,178 3,645

Est. Non-Baseline Test Points Remaining

2,502 12 136 73 2,281

1. Mission Systems Test Points for CY16 are shown only for Block 3F. Testing conducted to support Block 2B and Block 3i Mission Systems are discussed separately in the text. Cumulative

numbers include all previous Mission Systems activity.

2. These points account for planned development and regression test points built into the 2016 plan; additional points are considered “growth.” The total number of regression, development

and discovery points completed is the sum of budgeted non-baseline test points accomplished plus points added beyond budgeted non-baseline.

3. Represents mission systems activity not directly associated with Block capability (e.g., radar cross section characterization testing, test points to validate simulator).

4. Total Points Accomplished = 2016 Baseline Accomplished + Added Points

FY16 DOD PROGRAMS

60 F-35 JSF

Developmental Testing: F-35A Flight Sciences

Flight Test Activity with AF-1, AF-2, and AF-4 Test Aircraft

• F-35A ight sciences testing focused on:

- Clearing the F-35A Block 3F ight envelope (i.e., to

Mach 1.6, 700 knots, and 9.0 g) for loads, utter, and

weapons environment

- Testing of the internal gun

- Flight envelope clearance for external weapons required

for full Block 3F weapons capability

- Weapons separation testing of the AIM-9X missile

(external only), GBU-12 bomb (external carriage added for

Block 3F)

- High energy braking, high sink rate landings, and arresting

gear engagements

- AF-4 completed all ight testing for which it had been

slated, in July, and transitioned to chemical and biological

testing in August

F-35A Flight Sciences Assessment

• The program planned to complete F-35A ight sciences

testing by the end of October 2016; however, additional

testing for weapons environment and regression of

new software forced testing to continue into at least

December 2016. The program was able to complete baseline

test points to clear the aircraft structure for Block 3F

envelope (up to 9 g, 1.6M and 700 knots), completing utter

testing on AF-2 on September 29 and loads testing on AF-1

on November 4, 2016. Through the end of November,

the test team ew 50 percent more ights than planned

(226 own versus 151 planned) and accomplished 8 percent

more baseline test points than planned for the year (1,303 test

points accomplished versus 1,205 planned). These additional

baseline test points were added by the program throughout

the year and represent testing not originally budgeted for

when the CY16 plans were made. The test team also ew an

additional 156 test points for regression of new air vehicle

software, all of which were within the budgeted non-baseline

test points allocated for the year. As of the end of November

the program had approximately 100 baseline test points

remaining to complete F-35A ight sciences testing for

Block 3F.

• The following discoveries were made during F-35A ight

sciences testing:

- Failure of the attachment joint, as indicated by the

migration of the bushing in the joint, between the

vertical tail and the airframe structure is occurring much

earlier than planned, even with a newly designed joint

developed to address shortfalls in the original design.

In October 2010, the F-35A full scale durability test

article, AJ-1, showed wear in the bushing of this joint

after 1,784 test hours, which indicated that the joint will

fall short of the 8,000 hours of service life required by

the JSF contract specication. The program developed

a redesigned joint and began installing them on the

production line with Lot 6 aircraft, which began delivery

in October 2014. Subsequently, in July 2015, when

inspections showed bushing migrations and signicant

damage to the right and left side attachment joints in

BF-3, one of the F-35B ight sciences developmental test

aircraft, the joint was repaired and the bushing replaced to

replicate the redesigned joint. In August, 2016, inspections

of the joints in AF-2, one of the F-35A ight sciences

developmental test aircraft, showed similar bushing

migration requiring repair and bushing replacement in

accordance with the redesign. On September 1, 2016,

inspections of the vertical tail on BF-3 showed that the

newly designed joint had failed, after only 250 hours

of ight testing since the new joint had been installed,

requiring another repair and replacement. BF-3 completed

repairs and returned to ight on November 10, 2016.

- Vibrations induced by the gun during ring are excessive

and caused the 270 volts DC battery to fail. The program

began qualication testing of a redesigned battery in 2015,

but cracks in the casing discovered after the rst series

of testing required additional redesigning of the battery.

Requalication of a newly designed battery has not yet

occurred as of the writing of this report.

- Limitations to the carriage and employment envelope of

the AIM-120 missile above 550 knots may be required

due to excessive vibrations on the missiles and bombs in

the weapons bay. Analyses of ight test data and ground

vibration test data are ongoing (this applies to all variants).

- Excessive and premature wear on the hook point of

the arresting gear has caused the program to consider

a more robust redesign. In fact, the hook point has

required replacement after only one engagement in some

instances; the longest a hook point has lasted to date is

ve arrestments. This fails to meet the minimum service

life of 15 arrestments. Additionally, failure of the hook

point of the arresting gear on AF-4 occurred in July during

testing of high speed engagements. However, this appears

to be due to a malfunction of the Mobile Aircraft Arresting

System (MAAS), which holds the arresting cable in place

on both sides of the runway. The MAAS is designed to

allow the arresting cable to slide across the hook upon

engagement until the right and left sides are in equilibrium

before the braking action to slow the aircraft takes place

(this helps steer the aircraft toward the center of the

runway during the engagement). For unknown reasons,

only one side of the MAAS released the cable, resulting

in the hook point becoming abraded by the arresting cable

and failing 1.5 seconds after engagement.

- Block 3F envelope testing required an inight structural

temperature assessment, which yielded higher than

predicted air ow temperatures in the engine nacelle bay

in high-speed portions of the ight envelope under high

dynamic pressures. This resulted in higher than expected

nacelle structural temperatures on both the F-35A and

F-35C aircraft. Thermal stress analyses of the affected

parts are necessary before the program can provide the full

FY16 DOD PROGRAMS

F-35 JSF 61

Block 3F ight envelope for eet release. The outcome

may result in restricting elded operational aircraft to

600 knots airspeed below 5,000 feet altitude or a structural

change; this will be determined when the Services review

the analyses and issue the military ight release, which

certies the operational ight envelope.

- All F-35 variants display objectionable or unacceptable

ying qualities at transonic speeds, where aerodynamic

forces on the aircraft are rapidly changing. Particularly,

under elevated “g” conditions, when wing loading causes

the effects to be more pronounced, pilots have reported

the ying qualities as “unacceptable.” The program

adjusted control laws that govern ight control responses

in an updated version of software released to ight test in

March 2016. Results from ight testing of the software

changes have not yet been released. Although the elevated

g “dig-in” apparently affects all three variants, the program

does not plan to develop any additional control law

changes to mitigate these responses to aerodynamic effects

in the transonic region. In operational eet aircraft, g limit

exceedances are annunciated to the pilot and, in peacetime,

result in subsequent restricted maneuvering, mission

termination, and a straight-in approach and landing to

recover the aircraft. The aircraft is then down for some

time for maintenance inspections and potential repairs.

Also, the probability and long-term structural effects of

the g exceedances should be assessed by the program and

mitigated, if necessary.

- Foam insulation around the polyalphaolen (PAO)

coolant tubes that pass through wing and main body

fuel tanks in F-35A aircraft was found to be failing after

exposure to fuel. The discovery was made on a elded

production F-35A aircraft (AF-101) as it was undergoing

depot-level modications for fuel valves in August 2016.

The program determined the cause was a failure of the

manufacturing process with the sealant coating on the

insulation designed to protect the insulation from being

exposed to fuel. Instead, the sealant was permeable to

fuel, permitting the insulation to absorb fuel and expand,

forcing cracking and failure of the sealant coatings and

eventual breakdown and aking of the insulation. This

affected a total of 57 F-35A aircraft; 42 in the production

process and 15 elded aircraft. The Air Force temporarily

grounded the 15 elded aircraft, 10 of which were

designated as Initial Operational Capability aircraft. The

program quickly developed inspections and implemented

procedures to mitigate the insulation problems for elded

aircraft and those too far in the production line to have the

fuel lines replaced with proper insulation. The procedures

vary depending on whether fuel has entered the tank

with the PAO lines. For aircraft in which the fuel tanks

have contained fuel, the procedures involve accessing the

affected fuel tanks, removing the defective insulation,

installing blocking screens to prevent debris from leaving

the tank (and possibly contaminating other tanks, clogging

valves or affecting fuel pump operation). For the aircraft

in the production line that have not yet had fuel in the

tanks, the insulation will be removed from the PAO tubes,

but screens will not be added to the tank. The program

does not plan to re-insulate the PAO tubes, as the Block 3F

avionics – which are cooled by the PAO – apparently have

adequate thermal margin to tolerate the loss of insulation

on the tubes. The program must ensure that deployed

operating locations with high ambient temperatures – such

as those in Southwest Asia – are able to provide the

cooling effect necessary to prevent avionics overheat

conditions, especially for heat-soaked aircraft with hot

fuel tanks and during extended ground operations. The

program will need to conduct another assessment for

Block 4 avionics, and any new processors, to ensure the

thermal margin with that hardware conguration is still

adequate.

- An Air Force F-35A aircraft assigned to Luke AFB,

Arizona, experienced a tailpipe re during engine

start while deployed to Mountain Home AFB, Idaho

in September 2016, causing signicant damage to the

aircraft. The incident is under investigation.

- The program designed and elded an electrical Engine

Ice Protection System (EIPS) to protect the engine from

ice damage when exposed to icing conditions during

ground operations and in ight. Although it was qualied

during SDD engine ground tests, no SDD aircraft have the

system installed in the engine. The program elded the

system with later-lot production aircraft, but deciencies

in the system caused electrical shorting and damage to

the composite blades (referred to as the Fan Inlet Variable

Vanes) on the front of several engines. To prevent further

damage to engines in the eld, the program has disabled

EIPS and is changing the technical orders to require

pilots to shut down the aircraft if icing conditions are

encountered on the ground. DOT&E is not aware of any

corrections to the EIPS planned during SDD.

- The program completed the nal weight assessment of the

F-35A air vehicle for contract specication compliance in

April 2015 with the weighing of AF-72, a Lot 7 aircraft.

The actual empty aircraft weight was 28,999 pounds, 372

pounds below the planned not-to-exceed weight of 29,371

pounds. The actual weights of production aircraft since

then have been stable, with no signicant weight growth

observed. Weight estimates for production Lots 10 and

later indicate an expected weight growth of between 120

and 140 pounds, primarily due to new electronic warfare

(EW) avionics. Weight management of the F-35A is

important for meeting performance requirements and

structural life expectations. The program will need to

continue disciplined management of the actual aircraft

weight beyond the contract specication as further

discoveries during the remainder of SDD may add

weight and result in performance degradation that would

adversely affect operational capability.

FY16 DOD PROGRAMS

62 F-35 JSF

Developmental Testing: F-35B Flight Sciences

Flight Test Activity with BF-1, BF-2, BF-3, BF-4, and BF-5 Test

Aircraft

• F-35B ight sciences focused on:

- Clearing the F-35B Block 3F ight envelope (i.e., to Mach

1.6, 630 knots, and 7.0 g)

- High angle-of-attack testing with external stores

- Air refueling with the British KC-30A Voyager and Air

Force KC-10 aircraft

- Mode 4 (i.e., ight with the lift fan engaged to support

short takeoff and vertical landing operations) envelope

expansion

- Weapons separation testing of the AIM-9X missile

(external only), GBU-12 bomb (external carriage added for

Block 3F); Paveway IV bomb (internal and external) for

the United Kingdom, AIM-132 missile (external only) for

the United Kingdom

- Ground gun re testing with the F-35B gun pod;

accomplished on BF-1 in July

F-35B Flight Sciences Assessment

• Through the end of November, the test team ew 8 percent

more ights than planned (386 own versus 359 planned),

yet accomplished 5 percent less than the planned Block 3F

baseline test points (1,783 points accomplished versus 1,876

planned). The team ew an additional 169 test points for

regression of new air vehicle software, 115 of which were

the budgeted non-baseline points planned for CY16 and 54

points representing growth.

• The following details discoveries in F-35B ight sciences

testing:

- Limitations to the carriage and employment envelope of

the AIM-120 missile above 550 knots may be required

due to excessive vibrations induced on the missiles and

bombs in the weapons bay. Analyses of ight test data and

ground vibration test data are ongoing (this applies to all

variants).

- All F-35 variants display objectionable or unacceptable

ying qualities at transonic speeds, where aerodynamic

forces on the aircraft are rapidly changing. Particularly,

under elevated “g” conditions, when wing loading causes

the effects to be more pronounced, pilots have reported

the ying qualities as “unacceptable.” The program

adjusted control laws that govern ight control responses

in an updated version of software released to ight

test in March 2016. In the F-35B, an uncommanded

aircraft g “dig-in” that exceeds design limits has been

observed while performing elevated-g maneuvers in the

transonic region between 0.9M and 1.05M. Signicant

g exceedances (up to 7.7 g; a 0.7 g exceedance) have

occurred when pilots were attempting to sustain 6.5 g or

greater in this region. Based on ight test data, the F-35B

responses to transonic aerodynamic effects between 0.9M

and 1.05M during rolling or elevated-g maneuvering cause

uncommanded excursions that exceed the designed g limit

as well. Although the elevated g “dig-in” apparently

affects all three variants, the program does not plan to

develop any additional control law changes to mitigate

these responses to aerodynamic effects in the transonic

region. In operational eet aircraft, g limit exceedances

are annunciated to the pilot, and in peacetime, result in

subsequent restricted maneuvering, mission termination,

and a straight-in approach and landing to recover the

aircraft. The aircraft is then down for some time for

maintenance inspections and potential repairs. Also,

the probability and long-term structural effects of the

g exceedances should be assessed by the program and

mitigated, if necessary.

- Horizontal tail overheating was experienced on BF-3

during loads testing while accelerating to 1.5M for a loads

test point. The left horizontal inboard fairing surface

reached temperatures that exceeded the design limit by

a signicant amount. Post-ight inspections revealed

de-bonding on the trailing edge of the horizontal tail

surface and heat damage was noted on the horizontal

tail rear spar. Hardness checks on the rear spar were

performed and were determined to be within the

acceptable range. It is not yet known whether the program

or the Services will impose airspeed or afterburner time

restrictions in the Block 3F envelope due to horizontal tail

overheating.

- Failure of the attachment joint, as indicated by the

migration of the bushing in the joint, between the vertical

tail and the airframe structure, is occurring much earlier

than planned, even with a newly designed joint developed

to address shortfalls in the original design. In October

2010, the F-35A full scale durability test article, AJ-1,

showed wear in the bushing of this joint after 1,784 test

hours, which indicated that the joint will fall short of the

8,000 hours of service life required by the JSF contract

specication. The program developed a redesigned joint

and began installing them on the production line with

Lot 6 aircraft, which began delivery in October 2014.

Subsequently, in July 2015, when inspections showed

bushing migrations and signicant damage to the right and

left side attachment joints in BF-3, one of the F-35B ight

sciences developmental test aircraft, the joint was repaired

and the bushing replaced, to replicate the redesigned joint.

In August 2016, inspections of the joints in AF-2, one

of the F-35A ight sciences developmental test aircraft,

showed similar bushing migration requiring repair and

bushing replacement in accordance with the redesign. On

September 1, 2016, inspections of the vertical tail on BF-3

showed that the newly designed joint had failed, after

only 250 hours of ight testing since the new joint had

been installed, requiring another repair and replacement.

BF-3 completed repairs and returned to ight on

November 10, 2016.

- An F-35B assigned to Marine Corps Air Station Beaufort,

South Carolina, experienced a re within the weapons

bay during a training mission in late October 2016. The

FY16 DOD PROGRAMS

F-35 JSF 63

incident, although still under investigation, resulted in a

Class A mishap (involves loss of life or damage of more

than $2 Million). The Marine Corps did not ground any of

the training eet as a result of the incident.

- The program designed and elded an electrical Engine Ice

Protection System (EIPS) to protect the engine and lift fan

from ice damage when exposed to icing conditions during

ground operations and in ight. Although it was qualied

during SDD engine ground tests, no SDD aircraft have the

system installed in the engine. The program elded the

system with later-lot production aircraft, but deciencies

in the system caused electrical shorting and damage to

the composite blades (referred to as the Fan Inlet Variable

Vanes) on the front of the several engines. To prevent

further damage to engines in the eld, the program has

disabled EIPS and is changing the technical orders to

require pilots to shut down the aircraft if icing conditions

are encountered on the ground. DOT&E is not aware of

any corrections to the EIPS planned during SDD.

- Weight management of the F-35B aircraft is critical to

meeting the Key Performance Parameters (KPPs) in the

Operational Requirements Document (ORD), including

the Vertical Landing Bring-Back (VLBB) requirement,

which will be evaluated during IOT&E. This KPP requires

the F-35B to be able to y an operationally representative

prole and recover to the ship with the necessary fuel and

balance of unexpended weapons (two 1,000-pound bombs

and two AIM-120 missiles) to safely conduct a vertical

landing.

▪ The program completed the nal weight assessment

of the F-35B air vehicle for contract specication

compliance in May 2015 with the weighing of BF-44, a

Lot 7 production aircraft. Actual empty aircraft weight

was 32,442 pounds, only 135 pounds below the planned

not-to-exceed weight of 32,577 pounds and 307 pounds

(less than 1 percent) below the objective VLBB

not-to-exceed weight of 32,749 pounds.

▪ The actual weights of production aircraft through Lot 8

have increased slightly, with the latest Lot 8 aircraft

weighing approximately 30 pounds heavier than BF-44.

Weight estimates for Lot 10 aircraft and later project

weight growth of an additional 90 pounds, primarily due

to additional EW equipment.

▪ Known modications to the 14 Lot 2 through 4 F-35B

aircraft, required to bring those aircraft to the Block 3F

conguration, are expected to potentially add an

additional 350 pounds, which will push their weight

above the objective not-to-exceed weight to meet the

VLBB KPP. This KPP will be evaluated during IOT&E

with an F-35B OT aircraft.

▪ Estimates for FoM weight growth include an additional

250 pounds, which will exceed the vertical landing

structural limit not-to-exceed weight of 33,029 pounds

for the Lot 2 through Lot 4 aircraft. This additional

weight may prevent these aircraft from being upgraded

to the Block 4 conguration.

Developmental Testing: F-35C Flight Sciences

Flight Test Activity with CF-1, CF-2, CF-3, and CF-5 Test Aircraft

• F-35C ight sciences focused on:

- Clearing the F-35C Block 3F ight envelope (i.e., to Mach

1.6, 700 knots, and 7.5 g)

- Air refueling with F/A-18, KC-10, and KC-135 aircraft

- Weapons separation testing of the AIM-9X missile

(external only), Joint Standoff Weapon (JSOW, internal

only), GBU-12 bomb (external carriage added for

Block 3F)

- Shore-based ship suitability testing with external stores,

in preparation for shipborne trials that were conducted in

August

- High angle-of-attack testing with external stores

- Testing of the Joint Precision Approach and Landing

System (JPALS)

- Ground gun re testing with the F-35C gun pod;

accomplished on CF-3 in November

F-35C Flight Sciences Assessment

• Through the end of November, the test team ew 14 percent

more than planned ights (271 own versus 237 planned),

but accomplished 23 percent less than the planned Block 3F

baseline test points (1,304 points accomplished versus 1,695

planned). The team ew an additional 136 test points for

regression of new software, all of which were accounted for

in the budgeted non-baseline points planned for the year.

• The following details discoveries in F-35C ight sciences

testing:

- Flight testing of structural loads with the AIM-9X

air-to-air missile, which will be carried on external

pylons outboard of the wing fold in the F-35C, shows

exceedances above the wing structural design limit

during ight in regions of aircraft buffet (increased

angle-of-attack) and during landings. To address these

deciencies, the program is developing a more robust

outer wing design, which is scheduled for ight testing in

early CY17. Without the redesigned outer wing structure,

the F-35C will have a restricted ight envelope for missile

carriage and employment, which will be detrimental to

maneuvering, close-in engagements.

- Limitations to the carriage and employment envelope of

the AIM-120 missile above 550 knots may be required due

to excessive vibrations induced on the missiles and bombs

due to the acoustics in the weapons bay. Analyses of ight

test data and ground vibration test data are ongoing (this

applies to all variants).

- All F-35 variants display objectionable or unacceptable

ying qualities at transonic speeds, where aerodynamic

forces on the aircraft are rapidly changing. Particularly,

under elevated “g” conditions, when wing loading causes

the effects to be more pronounced, pilots have reported

the ying qualities as “unacceptable.” The program

adjusted control laws that govern ight control responses

in an updated version of software released to ight test

in March 2016. In the F-35C, like the other variants, an

FY16 DOD PROGRAMS

64 F-35 JSF

uncommanded aircraft g “dig-in” that exceeds design

limits has been observed while performing testing of

elevated-g maneuvers in the transonic region of the ight

envelope. While attempting to sustain a maximum g

(7.5g) turn, an F-35C test aircraft experienced 8.2 g – an

exceedance of 0.7 g. The program does not plan to

develop any additional control law changes to address

the ying quality. Similar to the other variants, an over-g

condition requires the pilot to terminate the mission (in

peacetime) and recover the aircraft with a straight-in

approach and landing with minimal maneuvering. The

aircraft is then down for some time for maintenance

inspections and potential repairs. Also, the probability and

long-term structural effects of the g exceedances should be

assessed by the program and mitigated, if necessary.

- Weapons environment testing showed that the

aircraft experienced transient rolling conditions while

asymmetrically opening and closing the weapon bay

doors (WBD). The ight control laws were designed

to compensate for the doors opening and closing

asymmetrically. The program corrected the on-board

aerodynamic models in two vehicle systems software

updates (versions R31.1 and R35.1) to reduce the roll

transients. These corrections resolved the transients for

the subsonic and transonic ight regimes, but not for

supersonic regimes. The operational impact of these

transients will be assessed during IOT&E.

- Block 3F envelope testing required an inight structural

temperature assessment, which yielded higher than

predicted air ow temperatures in the engine nacelle

bay in high-speed portions of the ight envelope under

high dynamic pressures. This resulted in higher nacelle

structural temperatures on both the F-35A and F-35C

aircraft. Thermal stress analyses of the affected parts are

necessary before the program can provide the full Block

3F ight envelope for eet release. The outcome may

result in restricting elded operational aircraft to 600 knots

airspeed below 5,000 feet altitude, or a structural change;

this will be determined when the Services review the

analyses and issue the military ight releases, which will

certify the operational ight envelope.

- As reported in previous DOT&E Annual Reports, the

F-35C experiences buffet and transonic roll off (TRO),

an uncommanded roll, at transonic Mach numbers and

elevated angles of attack. It is caused by the impact of

airow separating from the leading edge of the wing that

“buffets” aft areas of the wing and aircraft during basic

ghter maneuvering. The TRO and buffet occur in areas

of the maneuvering envelope that cannot be sustained

for long periods of time, as energy depletes quickly and

airspeed transitions out of the ight region where these

conditions manifest. However eeting, these areas of the

envelope are used for critical maneuvers. Operational

testing of the F-35C during IOT&E will assess the effect

of TRO and buffet on overall mission effectiveness.

- Due to the stiffness of the landing gear struts,

particularly the nose gear, taxiing in the F-35C results

in excessive jarring of the aircraft and often requires

pilots to stop taxiing if they need to make changes using

the touchscreens on the cockpit displays or to write

information on their kneeboard. Currently, the program

has no plans to correct the deciency of excessive jarring

during F-35C taxi operations.

- Excessive vertical oscillations during catapult launches

make the F-35C operationally unsuitable for carrier

operations, according to eet pilots who conducted

training onboard USS George Washington during the latest

set of ship trials. Although numerous deciencies have

been written against the F-35C catapult launch – starting

with the initial set of F-35C ship trials (DT-I) in

November 2014 – the deciencies were considered

acceptable for continuing developmental testing. Fleet

pilots reported that the oscillations were so severe that

they could not read ight critical data, an unacceptable

and unsafe situation during a critical phase of ight. Most

of the pilots locked their harness during the catapult shot

which made emergency switches hard to reach, again

creating, in their opinion, an unacceptable and unsafe

situation. The U.S. Navy has informed the Program

Ofce that it considers this deciency to be a “must x”

deciency. The program should address the deciency

of excessive vertical oscillations during catapult launches

within SDD to ensure catapult operations can be conducted

safely during IOT&E and during operational carrier

deployments.

- Overheating of the Electro-Hydraulic Actuator System

(EHAS) occurs under normal maneuvering in the F-35C.

The EHAS actuators move the ight surfaces and are

cooled by airow across the control surfaces. Pilots are

alerted in the cockpit of an overheat condition and must

then minimize maneuvering and attempt to cool the

EHAS by climbing, if practical, to an altitude with lower

temperatures to enhance cooling. Recovery and landing

must be completed as soon as possible, terminating the

mission.

- The program designed and elded an electrical Engine Ice

Protection System (EIPS) to protect the engine from ice

damage when exposed to icing conditions during ground

operations and in ight. Although it was qualied during

SDD engine ground tests, no SDD aircraft have the system

installed in the engine. The program elded the system

with later-lot production aircraft, but deciencies in the

system have caused electrical shorting and damage to the

composite blades (referred to as the Fan Inlet Variable

Vanes) on the front of the engine. To prevent further

damage to engines in the eld, the program has disabled

EIPS and is changing the technical orders to require

pilots to shut down the aircraft if icing conditions are

encountered on the ground. DOT&E is not aware of any

corrections to the EIPS planned during SDD.

FY16 DOD PROGRAMS

F-35 JSF 65

- Weight management of the F-35C is important for meeting

air vehicle performance requirements, including the KPP

for recovery approach speed to the aircraft carrier, and

structural life expectations. The program completed

the nal weight assessment of the F-35C air vehicle for

contract specication compliance in May 2016 with the

weighing of CF-28, a Lot 8 aircraft. The actual empty

aircraft weight was 34,581 pounds, 287 pounds below

the planned not-to-exceed weight of 34,868 pounds. The

weights of the other three Lot 8 production aircraft have

been consistent with that of CF-28. Weight estimates for

production Lots 11 and later indicate an expected weight

growth of approximately 160 pounds. The program

will need to continue rigorous management of the actual

aircraft weight through the balance of SDD to avoid

performance degradation that would affect operational

capability.

Developmental Testing: Mission Systems

• Mission systems are developed, tested, and elded in

incremental blocks of capability.

- Block 1. The program designated Block 1 for initial

training capability in two increments: Block 1A for Lot 2

(12 aircraft) and Block 1B for Lot 3 aircraft (17 aircraft).

No combat capability was available in either Block 1

increment. The Services have upgraded all of these

aircraft to the Block 2B conguration through a series of

modications and retrots. Additional modications will

be required to congure these aircraft in the Block 3F

conguration.

- Block 2A. The program designated Block 2A for

advanced training capability and delivered aircraft in

production Lots 4 and 5 in this conguration. No combat

capability was available in Block 2A. The Services

accepted 62 aircraft in the Block 2A conguration

(32 F-35A aircraft in the Air Force, 19 F-35B aircraft in

the Marine Corps, and 11 F-35C aircraft in the Navy).

Similar to the Block 1A and Block 1B aircraft, the

Services have upgraded all of the Block 2A aircraft to the

Block 2B conguration with modications and retrots,

although fewer modications were required. Additional

modications will be required to fully congure these

aircraft in the Block 3F conguration.

- Block 2B. The program designated Block 2B for initial,

limited combat capability with selected internal weapons

(AIM-120C, GBU-31/32 JDAM, and GBU-12). This

block is not associated with the delivery of any lot of

production aircraft, but with an upgrade of mission

systems software capability for aircraft delivered through

Lot 5 in earlier Block congurations. Block 2B is the

software that the Marine Corps accepted for the F-35B IOC

conguration. Corrections to some deciencies identied

during Block 2B and Block 3i mission systems testing

have been included in the latest production release of

Block 2B software – version 2BR5.3 – elded in May 2016

after airworthiness testing in April. The Services began

converting aircraft from these earlier production lots to the

Block 3i conguration by replacing the older Technical

Refresh 1 (TR1) integrated core processor with newer

Technical Refresh 2 (TR2) processors this year. As of the

end of November, 1 F-35A (AF-31) and 1 F-35B (BF-19)

had completed the TR2 modications, both of which are

instrumented operational test aircraft. The Marine Corps

declared IOC with Block 2B-capable aircraft in July 2015.

- Block 3i. The program designated Block 3i for delivery

of aircraft in production Lots 6 through 8, as these aircraft

include a set of upgraded TR2 integrated core processors.

The program delivered Lot 6 aircraft with a Block 3i

version that included capabilities equivalent to Block 2A

in Lot 5. Lot 7 aircraft were delivered with capabilities

equivalent to Block 2B, as are Lot 8 aircraft currently.

Block 3i software began ight testing in May 2014 and

completed baseline testing in October 2015, eight months

later than planned in the Integrated Master Schedule (IMS).

Because of software immaturity and instability during

startup and in ight, the program paused ight testing of

Block 3F software in February 2016 (software version

3FR5) and returned to Block 3i development and ight

testing to address poor mission systems stability. After

completing ight testing in April of another build of Block

3i software, version 3iR6.21, that version was elded to

the operational units with improved stability performance,

which was similar to that seen in the latest build of Block

2B software. By the end of November, the program had

delivered 51 F-35A aircraft to the Air Force, 17 F-35B

aircraft to the Marine Corps, and 13 F-35C to the Navy

in the Block 3i conguration in Lots 6, 7 and 8. The

Air Force declared IOC with Block 3i-capable aircraft in

August 2016.

- Block 3F. The program designated Block 3F as the full

SDD warghting capability for production Lot 9 and later.

Block 3F expands the ight envelope for all variants and

includes additional weapons, external carriage of weapons,

and the gun. Flight testing with Block 3F software on the

F-35 test aircraft rst began in March 2015. Flight testing

of Block 3F mission systems software, version 3FR5, was

paused in February 2016 when the program discovered

that it was too unstable for productive ight testing. The

program elected to reload a previous version of Block 3F

software – version 3FR4 – on the mission systems ight

test aircraft, to allow limited testing to proceed. After

improving the ight stability of the Block 3i software, the

program applied the corrections to deciencies causing

instabilities to the Block 3FR5 software and delivered

another version to ight test – version 3FR5.02 – in March,

to continue Block 3F testing. The program restarted

Block 3F testing in earnest in May with Block 3FR5.03

and released several more Quick Reaction Cycle (QRC)

versions, Blocks 3FR5.04 through 3FR5.07, through

November 2016 in attempts to quickly address key

deciencies that were blocking test points. The program

delivered the nal planned version of Block 3F software –

FY16 DOD PROGRAMS

66 F-35 JSF

3FR6 – to ight testing in December 2016. The program

will then determine, with testing in early 2017, if additional

QRC patches will be adequate to meet specications,

or if another full release of Block 3F software (e.g.,

3FR7) will be required. Of note, all of the aircraft from

earlier production lots, i.e., Lots 2 through 5 will need

to be modied, including structural modications and

the installation of TR2 processors, to have full Block 3F

capabilities. The program plans to begin delivering Lot 9

aircraft in early CY17. The Program Ofce has agreed to

allow the initial Lot 9 aircraft to be delivered with Block

3i software. These provisional acceptances may continue

until August 2017, when the program plans to have Block

3FP8 – the rst version of Block 3F production software –

for delivery of the remainder of Lot 9 and later aircraft.

- Block 4. The program has designated the rst release

of added capabilities following completion of SDD

as Block 4, with four distinct increments (Blocks 4.1,

4.2, 4.3, and 4.4). Current program schedules plan for

testing of Block 4.1 to begin at the end of CY19 with

subsequent increments following at 2-year intervals.

Hardware upgrades are planned in Blocks 4.2 and 4.4,

and will include the next upgrade in processors with

open-architecture Technical Refresh 3 (TR3) processors.

Production cut-in for initial Block 4.1 capabilities is

planned with Lot 13, beginning delivery in 2021, and

Lot 15 for Block 4.2. The post-SDD development

program is referred to as Follow-on Modernization (FoM).

However, for reasons discussed elsewhere in this report,

the program’s initial FoM plan is not executable and is

being re-planned by the program and stakeholders.

Flight Test Activity with AF-3, AF-6, AF-7, BF-4, BF-5, BF-17,

BF-18, CF-3, CF-5, and CF-8 Flight Test Aircraft and Software

Development Progress

• Mission systems testing focused on:

- Attempting to resolve software stability problems with

Block 2B and Block 3i mission systems

- Block 3F mission systems development and testing

- Initial integration testing of the U.S. Navy Joint Standoff

Weapon, version C1 (JSOW-C1)

- Completing weapons separation testing for the Small

Diameter Bomb (SDB) version I (SDB-I), which requires

mission systems-capable aircraft for interfacing with the

SDB

- Weapons integration and testing of the United Kingdom

Paveway IV bomb and Advanced Short-Range Air-to-Air

Missile (ASRAAM); determining root cause and options

to x ASRAAM integration deciencies

- On-Board Inert Gas Generation System (OBIGGS) testing

on CF-8, the only F-35C test aircraft modied with the

necessary hardware to complete testing

- Regression testing of Block 2B software on operational

test aircraft (AF-21, AF-23, BF-16 and BF-20), since the

developmental test aircraft had all already been converted

to the Block 3i or Block 3F conguration

- Joint Precision Approach and Landing System (JPALS)

testing with CF-5

- Testing of the Gen III Helmet Mounted Display System

(HMDS) illumination settings during the third F-35C

developmental test period at sea, designed to correct

excessive “green glow” during night operations onboard

the carrier

- The six mission systems developmental ight test aircraft

assigned to the Edwards AFB test center ew an average

rate of 6.9 ights per aircraft, per month in CY16 through

November, slightly above the planned rate of 6.7 for the

year, and ew slightly more than the planned number of

ights (479 ights accomplished versus 474 planned).

Mission Systems Assessment

• Block 2B

- Although the program completed Block 2B mission

systems testing in 2015 and provided a eet release

version of the software to the elded units, deciencies

remained and were carried forward into Block 3i. This

schedule-driven decision to pass deciencies forward

had consequences. The many deciencies, including

instabilities in both Block 3i and Block 3F mission

systems software, led the program to return to Block 3i

development to make corrections. When the revised

Block 3i software, Block 3iR6.21, demonstrated improved

inight stability, the program developed and tested another

version of Block 2B software – version 2BS5.3 – with

the corrections to the stability deciencies included. This

version was released to elded units in May 2016 for the

F-35A and F-35B, and in August 2016 for the F-35C; the

program expects to complete retrot of all elded aircraft

in the Block 2B conguration with the Block 2BS5.3

software by the end of January 2017.

- Because the test center aircraft had all been upgraded to

the Block 3i/3F conguration (i.e., with the newer TR2

processors), ight testing of the Block 2BS5.3 software

occurred on OT aircraft assigned to the OT squadron at

Edwards AFB, California.

• Block 3i

- Block 3i began with the schedule-driven decision to rehost

the immature Block 2B software and capabilities into

new TR2 avionics processors. Because of the extreme

overlap of development and production, combined with

delays in software development, the program was forced

to create a Block 3i capability to support delivery of Lot

6 and later aircraft, as they were being delivered with the

new processors. Although the program originally intended

that Block 3i would not inherit technical problems from

earlier blocks, this is what occurred, resulting in severe

problems with Blocks 3i and 3F software that needed to be

addressed, affecting both Block 2B and Block 3i elded

aircraft, and stalling the progress of mission systems

testing early in CY16.

- When Block 3i developmental ight testing began in

May 2014, six months later than planned in the program’s

FY16 DOD PROGRAMS

F-35 JSF 67

Integrated Master Schedule (IMS), the combination of

rehosted, immature software and new processors resulted

in severe avionics stability problems that were signicantly

worse than those in Block 2B. Continued delays in

completing Block 2B software development and testing

in support of the Marine Corps IOC, which was a priority

over Block 3i development for the program and the test

centers, combined with the severe stability problems with

the early versions of Block 3i software, caused several

pauses in early Block 3i ight testing. Block 3i ight

testing resumed again in March 2015 and was considered

to be complete in October 2015, eight months later than

planned in the IMS. Despite the continued problems

with avionics stability, sensor fusion, and other inherited

issues from Block 2B, the program terminated Block 3i

developmental ight testing in October 2015, and released

Block 3i software to the elded units. This decision was

made in an attempt to meet the program’s unrealistic

schedule for completing development and ight testing of

Block 3F mission systems.

- The program created an initial version of Block 3F

software by adding the nal required capabilities and

weapons to the problematic Block 3i software. However,

productive and efcient ight testing was not possible

due to inherited instabilities and other deciencies. The

Air Force insisted on xes for seven (ve identied

in 2014 and two more in 2015) of the most severe

deciencies inherited from Block 2B as a prerequisite

to use the nal Block 3i capability in the Air Force IOC

aircraft. Consequently, in February 2016, the program

decided to return to Block 3i development and testing

in another attempt to x key unresolved software

deciencies, including the avionics instabilities troubling

both Block 3i and Block 3F. A new version of mission

systems software, Block 3iR6.21, was quickly developed

and tested, and showed improvement to several of the

“must x” deciencies identied by the Air Force and the

inight stability problems, so it was released to the elded

aircraft in late May 2016. Data collected on start-up and

inight stability of the Block 3iR6.21 mission systems

software showed that both have improved over earlier

versions of Block 3i, and are approximately equivalent to

the nal version of Block 2B software. Based on ights

conducted with the production software through the

end of October 2016, the Air Force reported that, of the

seven “must x” deciencies, ve had been corrected,

one was partially corrected, but needed full Block 3F set

of capabilities to ensure full implementation, and one –

associated with extended post-mission download times

from the aircraft’s portable memory device (PMD) – was

awaiting elding of an upgraded ground data receptacle

(see more detail in the ALIS section below).

• Block 3F

- Block 3F ight testing began in March 2015, six months

later than the date planned in the IMS.

- The emphasis on, and return to, Block 3i testing in March

and April 2016 contributed in part to the program’s

inability to progress with Block 3F ight testing at the

planned rate. As of the end of November, a total of 975

Block 3F baseline test points had been completed in CY16,

compared to 1,189 planned (82 percent of planned). An

additional 1,784 development and regression points were

own, 1,534 of which were accounted for in the budgeted

non-baseline points for the year and 250 representing

growth.

- The lag in completing baseline test points – which are used

to verify capability – is also due to the program delivering

Block 3F software to ight test that was not mature enough

to meet specication compliance, or because deciencies

prevent the specication from being met. In an attempt

to address the deciencies and the lack of maturity in the

software, the program began developing and delivering

QRC versions of software to ight test. These software

versions are built, lab tested, and delivered to ight test

on a shorter timeline than the originally planned series of

software versions for Block 3F.

- Delays in starting Block 3F testing, pausing to redo

Block 3i work, and the immaturity of the Block 3F

software delivered to ight test have all contributed to the

program being well behind the plan to complete Block 3F

ight testing by the end of July 2017, the forecasted

completion date according to the program’s most recent

Mission Systems Software and Capability Release

Schedule. Instead, DOT&E estimates the program will

likely not nish Block 3F development and ight testing

prior to July 2018, based on the following:

▪ Continuing a 6.5 test point per ight accomplishment

rate, which is the CY16 rate observed through the end of

November.

▪ Continuing a ight rate of 6.9 ights per aircraft per

month, as was achieved through the end of November.

▪ Completing all of the baseline test points

(3,645 remaining as of the end of November) and

experiencing a regression, development and discovery

test point work load of 63 percent (historical average,

but well below the rate of 83 percent experienced in

CY16 through November).

- The program plans to truncate the planned testing by

eliminating test points, instead using alternative test

points or old data, in order to meet schedule deadlines

with the expectation of nishing SDD, getting to IOT&E,

and starting full-rate production. While this approach

may provide a quick sampling assessment of Block 3F

capabilities, there are substantial risks. The multiple

recent software versions for ight test may prevent the

program from using data from older versions of software

to count for baseline test point deletions because it may no

longer be representative of Block 3F. Limited availability

and high cost of range periods, combined with high re-y

rates for test missions completed on the Western Test

FY16 DOD PROGRAMS

68 F-35 JSF

Range, make it difcult for the program to efciently

conduct this testing. Finally, the most complex capabilities

in Block 3F have only recently reached the level of

maturity to allow them to be tested, and they are also

some of the most difcult test points to execute (i.e., full

Block 3F capabilities and ight envelope). Such a risky

course of action, if not properly executed with applicable

data, sufcient analytical rigor and statistical condence,

would likely result in failures in IOT&E causing the need

for additional follow-on operational testing, and, most

importantly, deliver Block 3F to the eld with severe

shortfalls in capability – capability that the Department

must have if the F-35 is ever needed in combat against

current threats. In fact, the plan to eliminate or replace test

points is at a point in the development program where the

most difcult, yet some of the most important capabilities,

have just started to reach maturity to begin ight testing.

The program should complete testing of all necessary

Block 3F baseline test points, as dened in the Joint

Test Plans; if the program attempts to use test data from

previous testing or added complex test points to sign off

some of these test points, the program must ensure the data

are applicable and provide sufcient statistical condence

prior to deleting any underlying build-up test points.

Additionally, the program should consider adding another

full version of Block 3F software to develop and deliver to

ight test in order to address more deciencies.

- Deciencies in performance and signicant operational

shortfalls must be resolved if the program is to deliver

the expected full Block 3F capability by the end of

SDD. Based on operational test pilot observations of

developmental test missions own in June and July 2016,

an assessment of the operational utility of Block 3FR5.03

software to support planned IOT&E missions, including

Close Air Support, Suppression/Destruction of Enemy

Air Defenses, Offensive and Defense Counter-Air,

Air Interdiction, and Surface Warfare, rated each of

the mission areas “red” and unacceptable overall.

Additionally, the JOTT provided an assessment of the

Block 3F capabilities, based on observing and assisting

with F-35 developmental testing with Block 3FR5.05

software, which began ight testing in August. The team’s

assessment made top-level, initial predictions of expected

IOT&E results of the F-35 for each of the mission areas.

The team predicted severe or substantial operational

impacts across all the planned IOT&E missions, similar

to the list of missions above, due to shortfalls and

deciencies, with the exception of the Reconnaissance

mission area, which predicted minimal operational impact.

The program should ensure adequate resources remain

available (personnel, labs, ight test aircraft) through

the completion of IOT&E to develop, test and verify

corrections to deciencies identied during ight testing

that may cause operational mission failures during IOT&E

or in combat.

- The program plans to provide full Block 3F capability,

as dened in the TEMP, with the rst Lot 10 aircraft

delivery in January 2018. In fact, as required by the

National Defense Authorization Act (NDAA) for FY16,

the Secretary of the Air Force certied to Congress in

September 2016 that these aircraft will have full combat

capability, as determined as of the date of the enactment

of the NDAA, with Block 3F hardware, software, and

weapons carriage. However, for many reasons, it is clear

that the Lot 10 aircraft will not initially have full Block 3F

capability. These reasons include, but are not limited to,

the following:

▪ Envelope limitations will likely restrict carriage and

employment of the AIM-120 missile and bombs well

into 2018, if not later.

▪ The full set of geographically specic area of

responsibility MDLs will not be complete, i.e.,

developed, tested and veried, until 2019, at the soonest,

due to the program’s failure to provide the necessary

equipment and software tools for the USRL.

▪ Even after they are delivered, the initial set of MDLs

will not be tested and optimized to deal with the full set

of threats present in operational test, let alone in actual

combat, which is part of full combat capability.

▪ The program currently has more than 270 Block 3F

unresolved high-priority (Priority 1 and Priority 2, out of

a 4-priority categorization) performance deciencies, the

majority of which cannot be addressed and veried prior

to the Lot 10 aircraft deliveries; less than half of these

deciencies were being actively worked in Block 3F.

▪ The program currently has 17 known and acknowledged

failures to meet the contract specication requirements,

all of which the program is reportedly planning to get

relief from the SDD contract due to lack of time and

funding.

▪ Dozens of contract specication requirements are

projected to be open into FY18; these shortfalls in

meeting the contract specications will translate into

limitations or reductions to full Block 3F capability.

▪ Estimates to complete Block 3F mission systems that

extend into the summer of 2018 have been put forth

not just from DOT&E, but also from other independent

Department agencies (e.g., CAPE), afrming that

delivery of full capability in January 2018 will be nearly

impossible to achieve, unless testing is prematurely

terminated, which would increase the likelihood that the

full Block 3F capabilities will not be adequately tested

and priority deciencies xed.

▪ Deciencies continue to be discovered at a rate of about

20 per month, and many more will undoubtedly be

discovered before and during IOT&E.

▪ ALIS version 3.0, which is necessary to provide full

combat capability, will not be elded until mid-2018,

and a number of capabilities that had previously been

designated as required for ALIS 3.0 are now being

FY16 DOD PROGRAMS

F-35 JSF 69

deferred to later versions of ALIS (i.e., after summer of

2018).

▪ The Department has chosen to not fund the program

to the CAPE estimate that the completion of Block 3F

mission systems testing will last until mid-2018, a time

span which is much later than, and at a cost that is at

least double, the Program Ofce’s latest unrealistic

estimate to complete SDD. This guarantees the program

will attempt a premature resource- and schedule-driven

shutdown of mission systems testing which will

increase the risk of mission failures during IOT&E and,

more importantly, if the F-35 is used in combat.

▪ Finally, rigorous operational testing in IOT&E, which

provides the most credible means to predict combat

performance in advance of actual combat, will not be

completed until at best the end of 2019 – and more

likely later.

Assessment of Block 2B and 3i “Initial Warghting” Fielded

Capability

• Using aircraft in the Block 2B conguration, both the

Air Force, with the F-35A, and the Marine Corps, with

the F-35B, have own simulated combat missions during

training or in support of training exercises. These training

missions have highlighted numerous shortfalls in Block 2B

capability.

- Unlike legacy aircraft, Block 2B aircraft will need

to make substantial use of voice communications to

receive targeting information and clearance to conduct

an attack during Close Air Support (CAS) missions due

to the combined effects of digital data communications

deciencies, lack of infrared pointer capability, limited

ability to detect infrared pointer indications from a

controller (which may be improved in the Generation

III Helmet Mounted Display System (Gen III HDMS)),

and inability to conrm coordinates loaded to GPS-aided

weapons. Each of these shortfalls limit effectiveness and

increase the risk of fratricide in combat.

- Many pilots assess and report that the Electro-Optical

Targeting System (EOTS) on the F-35 is inferior to those

currently on legacy systems, in terms of providing the

pilot with an ability to discern target features and identify

targets at tactically useful ranges, along with maintaining

target identication and laser designation throughout the

attack. Environmental effects, such as high humidity,

often forced pilots to y closer to the target than desired

in order to discern target features and then engage for

weapon employment, much closer than needed with

legacy systems, potentially alerting the enemy, exposing

the F-35 to threats around the target area or requiring

delays to regain adequate spacing to set up an attack.

However, due to design limitations, there are no signicant

improvements to EOTS planned for Block 3F.

- When F-35 aircraft are employed at night in combat, pilots

are restricted from using the current limited night vision

camera in the Generation II helmet with Block 2B aircraft.

This restriction does not apply to pilots equipped with the

Generation III helmet, which is elded with the Block 3i

aircraft. In general, if used in combat, pilots ying

Block 2B aircraft would operate much like early fourth

generation aircraft using cockpit panel displays, with the

Distributed Aperture System providing limited situational

awareness of the horizon, and heads-up display symbology

projected on the helmet.

• Because Block 3i is an interim capability based on Block 2B,

it inherited numerous limitations that will reduce operational

effectiveness and require workarounds if F-35 in the Block 3i

conguration are used in combat. The Air Force conducted

an IOC Readiness Assessment (IRA), using F-35A aircraft

with four different versions of Block 3i mission systems

software. Based on observations from elded units and

from the Air Force’s IRA, the following mission areas

will be affected by limitations, which may affect overall

effectiveness:

- Close Air Support (CAS). In many ways, the F-35 in

the Block 3i conguration does not yet demonstrate

CAS capabilities equivalent to those of fourth generation

aircraft. The F-35A in the Block 3i conguration has

numerous limitations that make it less effective overall

in the CAS mission role than most currently elded

ghter aircraft like the F-15E, F-16, F-18 and A-10 in a

permissive or low-threat environment, which is where

CAS is normally conducted. These limitations, consistent

with observations made by the Air Force in its IRA report,

include:

▪ The limited weapons load of two bombs (along with two

missiles for self-defense) constrains the effectiveness of

the Block 3i F-35 for many CAS missions. Compared

to a legacy ghter with multiple weapons on racks, and

multiple weapons types per aircraft, the limited Block 3i

load means that only a limited number and type of

targets can be effectively attacked.

▪ No gun capability. An aircraft-mounted gun is a key

weapon for some CAS scenarios when a bomb cannot

be used due to collateral damage concerns or when

the enemy is “dangerously close” to friendly troops.

The gun can also be an effective weapon for attacking

moving targets. However, even though an internal gun

is installed in the Block 3i F-35A, it cannot be used

until signicant modications to both the gun system

and aircraft are completed, and a version of Block 3F

software is tested and delivered to elded aircraft. Gun

weapons delivery accuracy (WDA) testing, aimed by the

HMDS, with the required modications and software,

has slipped from September 2016 to early 2017. Initial

build-up testing for the gun WDA was being planned for

December 2016 at the time of writing this report.

▪ Limited capability to engage moving targets. Even

though the Block 3i F-35A does not have a functioning

gun, it can carry the GBU-12 laser guided bomb which

has limited moving target capability. However, Block

3i (and Block 3F because it is currently not planned

to be addressed) does not have an automated targeting

FY16 DOD PROGRAMS

70 F-35 JSF

function with lead-laser guidance (i.e., automatically

computing and positioning the laser spot proportionately

in front of the moving target to increase the likelihood

of hitting the target) to engage moving targets with the

GBU-12, like most legacy aircraft have that currently

y CAS missions. Instead, F-35 pilots can only use

basic rules-of-thumb when attempting to engage moving

targets with the GBU-12, resulting in very limited

effectiveness. Also, limitations with cockpit controls

and displays have caused the pilots to primarily use

two-ship “buddy lasing” for GBU-12 employment,

which is not always possible during extended CAS

engagements when one of the aircraft has to leave to

refuel on a tanker. To meet the ORD requirement for

engaging moving targets, the Air Force is considering

integrating the GBU-49, a elded weapon that has

similar size, weight and interfaces as the GBU-12, or a

similar weapon that does not require lead-laser guidance,

in Block 3F. Otherwise, the program plans to develop

and eld lead-laser guidance in Block 4.2, which

would be delivered in CY22, at the earliest. However,

because of the similarities, the GBU-49 could be quickly

integrated with Block 3F to provide a robust moving

target capability for the F-35 much earlier.

▪ Voice communications are sometimes required to

validate digital communications. Problems with

Variable Message Format (VMF) and Link-16 datalink

messaging – including dropped or hidden information

or incorrect formats – sometimes require pilots to

use workarounds by validating or “reading back”

information over the radio that prevent them from

conducting digital (only) CAS, a capability that is

common in most legacy CAS aircraft. Recent use

of VMF digital communications during weapons

demonstration events by the operational test teams

has been more successful; however, data analyses are

ongoing.

▪ Limited night vision capability. Although Lot 7 and

later aircraft are elded with the Gen III HMDS, which

has shown improvement to the deciencies with the

earlier Gen II HMDS, limitations with night vision

capability remain. Pilots using the Gen III helmet for

night operations report that visual acuity is still less than

that of the night vision goggles used in legacy aircraft,

which makes identication of targets and detecting

markers more difcult, if not impossible. Also, “green

glow” – a condition where light leakage around the

edge of the display during low-light conditions makes

reading the projected information difcult – is improved

over the Gen II HMDS, but is still a concern during low

ambient illumination conditions. The program currently

has two open “Category 1 High” deciency reports for

“green glow,” with the most signicant safety concerns

pertaining to nighttime carrier operations.

▪ Lack of target marking capability – a key capability

for both Forward Air Controller-Airborne (FAC-A)

and CAS missions. Legacy CAS platforms can mark

targets with rockets, ares, and/or infrared (IR) pointers,

none of which are currently available on the F-35. The

F-35 has a laser designator as part of its Electro-Optical

Targeting System (EOTS), but the laser is used for

targeting from ownship when using the GBU-12 laser

guided bomb or to “buddy-guide” a weapon from

another aircraft. This limitation is not planned to be

xed during SDD.

- Other mission areas. In addition to the Block 3i

limitations listed above that affect the CAS mission area,

the following inherent Block 3i limitations will also affect

the capability of the F-35 in other mission areas:

▪ Poor ability to accurately locate (i.e., determine

geographic location with precision needed for weapons

employment) and identify threat emitters.

▪ No standoff weapon. With only direct attack bombs,

the F-35 in the Block 3i conguration will be forced

to y much closer to engage ground targets and,

depending on the threat level of enemy air defenses and

acceptable mission risk, it may be limited to engaging

ground targets that are defended by only short-range air

defenses, or by none at all.

▪ The limited weapons loadout of the Block 3i F-35 makes

effective attack of many expected types of targets in a

typical theater a challenge. For example, unlike legacy

aircraft, the Block 3i F-35 has no mixed weapons load

capability, which limits exibility to attack targets with

appropriately matched weapons. Block 3i F-35 aircraft

can only employ two internally carried bombs, and

although internal carriage reduces the susceptibility of

the F-35 relative to legacy aircraft, by virtue of the low

observability it provides, it does not provide the ability

to attack more than one or two targets.

▪ Pilots report that inadequacies in Pilot Vehicle Interfaces

(PVI) in general, and deciencies in the Tactical

Situation Display (TSD) in particular, which displays

the results of sensor fusion and is designed to provide

increased situation awareness, continue to degrade

battlespace awareness and increase pilot workload.

Workarounds to these deciencies are time-consuming

for the pilot and detract from efcient and effective

mission execution.

- Block 3i has signicant deciencies that must still be

addressed, despite the additional software release to the

eld, Block 3iP6.21, in May 2016. In addition to the

limitations listed above, Block 3i also has hundreds of

other deciencies, the most signicant of which must be

xed in Block 3F to realize the full warghting capability

required of the F-35. These deciencies include, but are

not limited to, the following:

▪ Avionics sensor fusion performance is still unacceptable.

» Air tracks often split erroneously or multiple false

tracks on a single target are created when all sensors

contribute to the fusion solution. The workaround

during early developmental testing was to turn off

FY16 DOD PROGRAMS

F-35 JSF 71

some of the sensors to ensure multiple tracks did

not form, which is unacceptable for combat and

violates the basic principle of fusing contributions

from multiple sensors into an accurate track and clear

display to gain situational awareness and to identify

and engage enemy targets.

» Similarly, multiple false ground tracks often are

displayed when only one threat emitter is operating.

In addition, tracks that “time out” and drop from the

display cannot be recalled, which can cause pilots

to lose tactical battleeld awareness on enemy air

defense radars that turn on only intermittently, as is

typical of missile engagement radars.

» Sharing erroneous tracks over the Multifunction

Advanced Data Link (MADL) between aircraft in

the F-35 formation multiplies the problems described

above.

» The Air Force IOC Readiness Assessment (IRA)

report also identied deciencies with fusion in

Block 3i.

▪ Electronic warfare (EW) capabilities, including

electronic attack (EA), are inconsistent and, in some

cases, not effective against required threats.

» Although the details of the deciencies are classied,

effective EW capabilities are vital to enable the F-35

to conduct Suppression/Destruction of Enemy Air

Defenses (SEAD/DEAD) and other missions against

elded threats.

» The Air Force IRA report also identied signicant

EW deciencies in Block 3i.

▪ Datalinks do not work properly. Messages sent across

the MADL are often dropped or pass inaccurate offboard

inter-ight fusion tracks based on false or split air tracks

and inaccurate ground target identication and positions.

▪ Reduced on-station time and greater reliance on tanker

aircraft. Although this limitation is not unique to the

Block 2B or Block 3i conguration, the F-35 has high

fuel burn rates and slow air refueling rates that extend

air refueling times and decrease overall on-station time,

which may reduce overall mission effectiveness.

- The program was able to improve stability of the mission

systems software to support the Air Force’s plan to declare

IOC. The Program Ofce reported improvements in Mean

Flight Hours Between Instability Events (MFHBIE) for

both start-up and in-ight of Block 2B and Block 3i. The

latest inight stability metrics from the Program Ofce

are provided in the table to the right. Note that “2BS”

versions of software refer to Block 2B versions delivered

to ight test. For Block 3i, the program adopted a naming

convention where a “P” version refers to software released

for production aircraft and an “R” version is for ight

testing. An “R” version of software has additional coding

that permits data to be collected from data buses on the

aircraft and stored on the DART pod or transmitted to

ground stations for recording or playback. For IOT&E,

since data will be collected with the instrumentation

packages on the OT aircraft, IOT&E will be own with an

“R” version of software where selected data and messages

can be directed for recording for post-ight analyses.

- The operational effect of mission systems software

instabilities on the F-35 will not be well understood

before the completion of formal operational testing. One

of the objectives of the Air Force IRA was to examine

the frequency and effect of these instability events. The

Air Force dened and scored instability events during

the IRA in the same way as the Program Ofce and the

contractor for comparison purposes and observed similar

trends. An instability event is generally the initial failure,

or the primary system failure, and does not account for

subsequent failures of the same system or failures of

subsystems. In addition, the Air Force collected data on

instability occurrences, which includes a broader set of

instabilities. An instability occurrence accounts for all

failures of systems and associated subsystem failures,

when each of the failures could have affected the mission

capability of the aircraft. The Air Force collected data on

instability occurrences with F-35A aircraft ying the most

current Block 3i software and counted 25 occurrences

in 34.1 ight hours, resulting in a Mean Flight Hours

Between Instability Occurrences of 1.4 hours. During

IOT&E, all relevant stability events and occurrences,

on the ground or in the air, which impact mission

effectiveness or suitability, including repeat events (unless

attributed to a hardware failure) will be counted to assess

overall mission effect. Similar to the table below, stability

data from IOT&E will be compared with data from elded

aircraft with the “P” version of Block 3F software to assess

any differences.

• The Air Force IRA test team at Nellis AFB ew a total of 18

mission scenarios (72 aircraft sorties) covering the mission

sets of CAS, Air Interdiction (AI), and SEAD/DEAD.

The missions were own over the Western Test Ranges

from March 1 through April 29, 2016. Additionally, the

assessment included observations from an Air Force-led

deployment to Mountain Home AFB, Idaho, with six F-35A

MISSION SYSTEMS SOFTWARE INFLIGHT STABILITY METRICS

DATA AS OF NOVEMBER 27, 2016

Software

Release

Number of

Inight

Stability Events

Cumulative

Flight Hours

Mean Flight

Hours Between

Instability Events

2BS5.2 31 224.8 7.3

2BS5.3 1 28.5 Insucient data

3iP6.21 13 349.5 26.9

3iR6.21 (Edwards

OT Aircraft)

6 75.8 12.6

3FR5* 222 950.1 4.3

* 3FR5 metrics are a summation of 8 versions of software used in ight testing: 3FR5,

3FR5.02, 3FR5.03, 3FR5.03QRC, 3FR5.04QRC, 3FR5.05, 3FR5.06, and 3FR5.07

FY16 DOD PROGRAMS

72 F-35 JSF

aircraft from Edwards, supported by an ALIS SOU v2 with

software 2.0.1. Although the Air Force has determined that

the F-35A with Block 3i mission systems software provides

“basic” capabilities for IOC, many signicant limitations

and deciencies remain. In comparison to a dedicated

operational test and evaluation, this was a brief, but

revealing assessment of mission capability. However, until a

full operational test and evaluation of the F-35 is completed,

we will have low condence that we understand all of the

limitations in the system.

- The detailed results of the IRA, as reported by the Air

Force, are consistent with the assessments in this Annual

Report.

- Inight stability of the Block 3i mission systems was

assessed to be back to a level comparable to that in

Block 2B, as measured by the number of inight

instability events per ight hour.

- If used in combat, F-35 aircraft will need support to locate

and avoid modern threat ground radars, acquire targets,

and engage formations of enemy ghter aircraft, due to

unresolved performance deciencies and limited weapons

carriage available (i.e., two bombs and two air-to-air

missiles).

- Unresolved Block 3i deciencies in fusion, EW, and

weapons employment continue to result in ambiguous

threat displays, limited ability to effectively respond to

threats, and, in some cases, a requirement for offboard

sources to provide accurate coordinates for precision

attack.

- Concerning the CAS mission area, the team concluded that

the Block 3i F-35A does not yet demonstrate equivalent

CAS capabilities to those of fourth generation aircraft.

Mission Data Load Development and Testing

• F-35 effectiveness in combat relies on mission data loads

(MDL) – which are a compilation of the mission data les

needed for operation of the sensors and other mission

systems – working in conjunction with the system software

data load to drive sensor search parameters so that the F-35

can identify and correlate sensor detections, such as threat

and friendly radar signals. The contractor team produced

an initial set of mission data les for developmental testing

during SDD, but the operational MDLs – one for each

potential major geographic area of operation – are being

created, tested, and veried by a U.S. government lab, the

U.S. Reprogramming Lab (USRL), located at Eglin AFB,

Florida, which is operated by government personnel from the

Services. The Air Force is the lead Service. These MDLs

will be used for operational testing and elded aircraft,

including the Marine Corps and Air Force IOC aircraft. The

testing of the USRL MDLs is an operational test activity,

as was arranged by the Program Ofce after the restructure

that occurred in 2010. The Department must have a

reprogramming lab that is capable of rapidly creating, testing

and optimizing MDLs, and verifying their functionality

under stressing conditions representative of real-world

scenarios, to ensure the proper functioning of F-35 mission

systems and the aircraft’s operational effectiveness in both

combat and the IOT&E of the F-35 with Block 3F.

• Despite the critical requirement for developing and elding

F-35 MDLs, signicant ongoing software and hardware

deciencies in the USRL have yet to be addressed,

which continue to prevent efcient creating, testing, and

optimization of the MDLs for operational aircraft elded in

the Block 2B and Block 3i conguration, and are preventing

the development of MDLs for Block 3F.

- The current reprogramming hardware and software tools

are so cumbersome that it takes months for the USRL

to create, test, optimize, and verify a new MDL. This

time-consuming process was still not complete for the

complete set of Block 3i AOR-specic MDLs.

- The program has mismanaged sustainment and upgrades

of the USRL to the point that it currently does not have the

ability to start creating MDFs for Block 3F and will not

have that capability until February 2017, at the earliest.

Once the USRL can start creating Block 3F MDFs, it will

take approximately 15 months to deliver a veried MDL for

IOT&E and for elded Block 3F aircraft.

- The program plans to start delivering production aircraft

in the Block 3F conguration in May 2017. Because the

USRL will not be able to develop, test, and validate a

Block 3F MDL until mid-2018, the Services will have to

eld Block 3F-capable aircraft with either Block 3i, or with

a Block 3F test MDL provided by the contractor; however,

either course of action will likely restrict these elded

Block 3F aircraft from use in combat.

• Additionally, the Program Ofce and Lockheed Martin have

failed to complete necessary contracting actions to address

current shortfalls in signal generation capability within

the USRL, including the key hardware upgrades needed to

create, test, and verify Block 3F MDFs to detect and identify

emissions from currently elded threat systems in scenarios

with realistic threat densities. This failure occurred in spite

of the requirement being clearly identied in 2012 and

the Department programming $45 Million in the FY13-16

budgets to address it. The JPO sponsored a gap analysis

study of USRL capabilities to determine the lab upgrade

requirements at the engineering level before beginning

contracting actions. When completed in 2014, the study

concluded that between 16 and 20 upgraded radio frequency

(RF) signal generator channels would be needed for the USRL

to adequately create and test MDFs in the USRL for the

elded threats examined in the study, using realistic scenarios

and threat densities. After receiving a proposal for the

upgrades from the contractor priced at over $200 Million in

May 2016, the JPO requested a new proposal, reportedly with

options only for up to 12 upgraded signal generator channels,

which the contractor indicated would not be answered until

July 2017. Furthermore, once on contract, it would then take

approximately 3 years after ordering the equipment for it

to be delivered and installed, which will be late to need for

FY16 DOD PROGRAMS

F-35 JSF 73

both IOT&E and elding of Block 3F aircraft. As a result,

even though the USRL will eventually have the capability

to create MDLs for Block 3F in 2017, it still will not have

the required signal generators to test and optimize the MDLs

to ensure adequate performance against currently elded

threats.

• To provide the necessary and adequate Block 3F

mission data development capabilities for the USRL, the

Program Ofce must immediately fund and expedite the

contracting actions for the necessary hardware and software

modications, including an adequate number of additional

RF signal generator channels and the other required

hardware and software tools. Unless these actions are taken

immediately, the USRL will not be congured to create,

test, and verify Block 3F MDLs for aircraft for current

threat systems and threat scenarios until sometime in 2020,

placing the operational aircraft at risk in combat against

elded threats and the program at risk of failing IOT&E.

The program is working to nd alternative facilities with

the required signal generators to mitigate this lab capability

shortfall for Block 3F.

• Signicant additional investments are also required

within 2-3 years to further upgrade the USRL to support

F-35 Block 4 Follow-on Modernization (FoM) MDL

development. Block 4.2 is currently planned to include

new Technical Refresh 3 (TR3) processors and other new

hardware which, due to the overlapping Block 4 increments,

will require the USRL, or an additional reprogramming lab,

to have two different avionics congurations simultaneously

– a TR2 line for Blocks 3F and 4.1, plus a TR3 line for

Block 4.2 and later. Although the Block 4 hardware

upgrades in the USRL will need to begin soon to be ready in

time, the reprogramming requirements for Block 4 have yet

to be fully dened. The Program Ofce must expeditiously

undertake the development of those requirements and plan

for adequate time and resources within the DOD budget

cycle, in order to ensure the USRL is able to meet Block 4

MDL requirements.

• The USRL, with JOTT observers, held an “Urgent

Reprogramming Exercise (URE)” from April 20 to July 25,

2016. This type of exercise is intended to test the USRL’s

ability to respond to an urgent request from a Service to

modify the mission data in response to a new threat or

new mode of an existing threat. Due to USRL’s ongoing

production efforts, the URE was conducted concurrently

with the lab’s effort to produce an operational MDL, which

is why the exercise period was several months, instead of

a few days. The JOTT and USRL carefully tracked hours

that were specic to the URE as they occurred and surveyed

USRL personnel to identify process issues. The total hours

recorded were double the Air Force standard for rapidly

reprogramming a mature system. The JOTT identied

several key process problems, many of which are described

above, including the lack of necessary hardware, analysis

tools that were not built for operational use, and missing

capabilities, like the ability to quickly determine ambiguities

in the mission data. These problems must be corrected in

order to bring the USRL’s ability to react to new threats up to

the identied standards routinely achieved on legacy aircraft.

• In addition to the above deciencies that involve overall

laboratory capability and tools to develop MDLs, there are

also deciencies in the program’s sustainment efforts to

ensure a high state of readiness, particularly if the Services

have an urgent reprogramming requirement at any time.

To meet these tasks, the USRL must have all necessary

equipment in a functioning status, similar to aircraft

availability. Inadequacies in the current level of sustainment

include, but are not limited to:

- Insufcient number of Field Service Engineers (FSE) to

assist in maintenance and operation of the lab equipment,

which include both specialized equipment and aircraft

mission equipment

- Inadequate or insufcient training for most laboratory

personnel, which is hindered by the insufcient number of

FSEs

- No engineering drawings or JTD for many critical

components, making troubleshooting of failures of those

components difcult and lengthening the time required to

return the laboratory to full operational status

- Insufcient spare parts for many critical components

- Low supply priority, equivalent to that of a unit in training,

resulting in long delays to receive required parts

- Missing part numbers for many components, forcing

USRL personnel to submit an Action Request (AR) rst to

determine the part number before a replacement part can

be ordered through supply.

Weapons Integration and Demonstration Events

Block 3F Developmental Testing

• After the release of Block 3iP6.21 software in May 2016,

the program focused on completing development of Block

3F capabilities, including weapons envelope and integration

testing. To provide an operational employment ight

envelope, the program accomplished ight sciences testing

of external weapons carriage and employment, as well as

integrating bombs (SDB-I, JSOW C-1, and PW-IV) and

missiles (AIM-9X and AIM-132 ASRAAM) not previously

integrated on the F-35 in Block 2B or 3i.

• The TEMP requires 26 Block 3F weapons delivery

accuracy (WDA) events be completed as part of the Block

3F developmental testing effort. These WDAs are key

developmental test activities necessary to ensure the full

Block 3F re-control capabilities support the “nd, x,

track, target, engage, assess” kill chain. As of the end of

November, only 5 of the 26 events (excluding the gun

events) had been completed and fully analyzed. Several

WDAs have revealed deciencies and limitations to weapons

employment. An additional 11 WDAs have occurred, but

analyses are ongoing. Of the 10 remaining WDAs, 4 are

still blocked due to open deciencies that must be corrected

before the WDA can be attempted. The program should

correct deciencies that are preventing completion of all of

FY16 DOD PROGRAMS

74 F-35 JSF

the TEMP-required Block 3F WDA events and ensure they

are completed prior to nishing SDD.

• Discoveries from the Block 3F WDA events include:

- AIM-9X and AIM-132 ASRAAM seeker status tone

problems

- Out-of-date launch zones for AIM-120 missiles

- Pilot Vehicle Interface (PVI) and mission planning

problems with the U.S. Navy’s JSOW-C1 missile that, if

not corrected, may cause signicant weapon employment

limitations in the eet’s ability to attack moving ship

targets and enable exible engagement of land-based

targets of opportunity

- Ongoing radar and fusion deciencies affecting air-to-air

target track stability and accuracy, which could cause

reduced missile lethality

- Multiple hung stores, which typically result in an inight

emergency, occurred with the AIM-9X due to mission

systems software and weapon integration deciencies

- Problems with integrating the British AIM-132 ASRAAM

missile and Paveway IV bomb; changes to address these

problems could have unintentionally affected the U.S.

AIM-9X and laser-guided bomb capabilities, which may

require regression testing of these U.S. weapons.

• In an effort to efciently accomplish the WDA events, the

program dedicated several test aircraft to a WDA surge

period during June through August. Although the program

had planned to begin WDA events as early as February 2016,

the rst live weapons event did not occur until July. Delays

in starting the Block 3F WDAs were caused by immature

software and deciencies affecting weapons employment.

The following table lists the Block 3F WDA events, software

versions, scheduled and completion dates, overall results

and assessments for each completed live re event through

the end of November. Many of the events were originally

blocked from completion due to software deciencies that

had to be addressed using QRC versions of software in order

to allow the weapons events to proceed.

FY16 DOD PROGRAMS

F-35 JSF 75

Block 3F Developmental Testing Weapons Events Accomplished Through November 2016

WDA Number Weapon Event

Software

Conguration

Scheduled Date

Result Assessment

Completion Date

301 AMRAAM 3FR5.03

Feb 16 Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Initial data analysis indicates that there was an

inight issue that may have aected targeting

accuracy. Analysis in process to determine the

root cause and impact(s).

Jul 16

302

AMRAAM with

AIM-9X

3FR5.03

Feb 16 Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Initial data review indicated that the AIM-9X

tones were not as expected and there was no

missile post-launch timer indication to the pilot.

Jul 16

303

AMRAAM red

with target

o-boresight

3FR5.03

Feb 16

Partially successful

accomplishment; shot

captured key radar capability

data but failed primary test

objective; shot required

control room intervention.

Known issues with outdated F-35 AMRAAM

Attack Model in mission systems software

resulted in no shoot cues or dynamic launch

zone displayed to pilot requiring the control

room to provide a “shoot” call to the pilot. Initial

data review indicates that there was also no

post-launch timer indication to the pilot. Also,

weapon quality track was erratic pre- and

post-launch. More detailed analyses are pending,

following data to be provided by the missile

vendor.

Aug 16

307 2 X AMRAAM 3FR5.03

Jun 16

Partially successful

accomplishment; shot

required control room

intervention.

The cockpit indication was a guidance failure

on the missiles and required control room

intervention to conrm the shot parameters and

direct the pilot to shoot. More detailed analyses

are pending, following data to be provided by

the missile vendor.

Aug 16

308

2 X SDB-I

(GBU-39) and 1 X

AMRAAM

3FR5.06

Jun 16

Successful accomplishment

of event.

All weapons initially appear to have functioned

successfully. Analysis ongoing.

Nov 16

311 2 X AMRAAM 3FR5.03

Apr 16

Pending Data Review; shot

required control room

intervention.

Unsuccessful; also the pilot indications in the

cockpit indicted a guidance fail resulting in

control room intervention to accomplish the

shot. More detailed analyses are pending,

following data to be provided by the missile

vendor.

Jul 16

316

AIM-9X red

against a non-

maneuvering

target

3FR5.03

Feb 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Inight weapon failed on rst missile attempt

(built-in test failure and no missile tone to the

pilot); back-up missile functioned as expected.

Deciency report was written on missile tone

anomalies.

Jul 16

317

AIM-9X red

against a

maneuvering

target

3FR5.03

Jun 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Initial data review indicates that the missile

tones were not correct, no dynamic launch

zone indication in Dogght mode and the

gun symbology occluded the target in the

helmet-mounted display. More detailed analyses

on radar track accuracy and radar ranging

accuracy following data to be provided by the

missile vendor.

Aug 16

320

JDAM (GBU-31)

delivered against

a single target

using Synthetic

Aperture Radar

(SAR) map

coordinates

3FR5.03

Feb 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team planned to use a known

workaround for minor Launch Acceptability

Region (LAR) inaccuracy due to an outdated

LAR model in mission systems software. Pilot

released the bomb using a “rule of thumb”

guidance to determine “in-zone.” JDAM LAR

model update in mission systems software is

required.

Jul 16

321

JDAM (GBU-31)

delivered against

a single target

using Bomb-

on-Coordinate

employment

3FR5.03

Apr 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team planned to use a known

workaround for a minor LAR inaccuracy due

to an outdated LAR model in mission systems

software. Pilot released the bomb using a “rule

of thumb” guidance to determine “in-zone.”

Post-mission initial data review indicates that the

target elevation values available to the pilot were

not consistent between the mission planned

terrain elevation, the displayed elevation on the

cockpit displays, and the value loaded into the

JDAM in the transfer alignment.

Jul 16

FY16 DOD PROGRAMS

76 F-35 JSF

Block 3F Developmental Testing Weapons Events Accomplished Through November 2016 (CONTINUED)

WDA Number Weapon Event

Software

Conguration

Scheduled Date

Result Assessment

Completion Date

322

JDAM (GBU-31) X

2 Ripple release

on two targets

3FR5.03

Jun 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team planned to use a known

workaround for a minor LAR inaccuracy due to an

outdated LAR model in mission systems software.

Pilot released the bomb using a “rule of thumb”

guidance to determine “in-zone.” Pilot released

weapons on rule-of-thumb with minor impact

for this DT scenario and Service representatives

have stated that the rule-of-thumb workaround

may be adequate for operations. Post mission

data analysis showed a SAR map coordinate

inaccuracy, but within the Circular Error Probable

(CEP) of the weapon.

Aug 16

323

JDAM (GBU-31)

Pattern on

target (multiple

weapons)

3FR5.05

Jul 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Weapons impacted as expected with the

selections made by the pilot and with accurate

PVI indications. Dual voltage bomb rack unit

(BRU) functioned properly with no power

distribution issues.

Oct 16

324

SBD-I (GBU-39) X

2 on two targets

3FR5.03

May 16 Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team used a planned workaround for

BRU-61; using the new dual-voltage BRU in

single-voltage mode due to a mission systems

software limitation.

Aug 16

325

SDB-I (GBU-39)

Single release

3FR5.03

Feb 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team used a U.S. non-operationally

representative BRU-61, one with only a single

voltage unit, to complete this WDA event. This

older BRU-61 is representative for partner

operations.

Jul 16

328

UK Paveway IV

bomb

3FR5.05

Jul 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

Weapons integration deciencies were identied

during this event and deciency reports

completed.

Oct 16

SDB Seps

SDB-I (GBU-39)

multiple ripple

release for

ight sciences

separation

test points,

completed on

mission systems

aircraft.

3FR5.03

Feb 16

Successful accomplishment

of event and sucient

data collected for weapons

integration analyses.

The test team used a U.S. non-operationally

representative BRU-61, one with only a single

voltage unit, to complete this WDA event. This

older BRU-61 is representative for partner

operations. Awaiting data delivery for detailed

analysis.

Jul 16

• The remaining 10 events are planned to be completed over

the next several months, as the program provides versions of

Block 3F software with necessary deciency xes to allow

the rest of the events to proceed. The remaining events are

complex multi-weapon, multi-target, and advanced threat

presentations. Whether all WDAs will be completed with

the nal planned increment of Block 3F software – version

3FR6 – released in December is still to be determined, but

several key deciency xes related to weapons employment

are apparently not included and the probability of additional

discoveries during the remaining weapons test events is high,

based on results to date.

Gun Testing

• All three variants add gun capability with Block 3F. The

F-35A gun is internal; the F-35B and F-35C each use a gun

pod. Differences in the outer mold-line faring mounting

make the gun pods unique to a specic variant, i.e., a gun

pod designated for an F-35B cannot be mounted on an F-35C

aircraft.

• Flight sciences testing of the F-35A internal gun was

completed in May 2016. The rst ring of the gun in ight

occurred October 30, 2015, and the entire ight sciences test

effort consisted of 11 ights over the 7-month period. Testing

revealed that the small doors that open when the gun is red

induce a yaw (i.e., sideslip), resulting in gun aiming errors that

exceed accuracy specications. As a result, software changes

to the ight control laws were needed to enable adjustments,

which are still to be determined by ight testing, to cancel

out the yaw when the gun doors are open. These control law

changes, and the resulting regression testing, delayed the start

of gun accuracy ight testing on mission systems test aircraft

until December 2016, at the earliest. Since no mission-

systems-capable developmental test aircraft were built with

an internal gun, the program modied one of the operational

test F-35A aircraft (AF-31) to conduct the needed gun testing

events. Until testing is completed on AF-31, it is unknown if

the F-35 gun system, aimed by the Gen III HMDS, will meet

accuracy requirements for effective air-to-air and air-to-

ground gun employment.

FY16 DOD PROGRAMS

F-35 JSF 77

• The program has conducted ground testing of the F-35B

gun pod and plans to start airborne testing in January 2017.

Initial ground ring of the F-35C gun pod occurred in

mid-November 2016 and airborne gun testing is planned

to start in March 2017. New discoveries, as well as

determining the amount of adjustment to the ight control

laws to counter the pitching moments induced by ring the

gun pod, are likely.

• Accuracy testing of the gun with the HMDS has not yet been

completed and continues to be delayed as new discoveries

are made. Hence, the effectiveness of the gun, aimed

via the gunsight in the HMDS, is still unproven for both

air-to-air and air-to-ground gun employment. The effects

of the canopy transparency on gun aiming – i.e., the pilot

aiming the gun via the HMDS gunsight looking through the

thick canopy material, associated distortions, and attempted

software-programmed corrections – are not yet characterized.

• Although aimed ring of the gun had yet to occur, both

DT and OT pilots have own with the air-to-ground gun

strang symbology displayed in the helmet and reported

concerns that it is currently operationally unusable and

potentially unsafe to complete the planned aimed gun re

testing. These deciencies may cause further delays to

the start of gun accuracy ight testing. Also, testing of the

air-to-air symbology by both DT and OT pilots revealed that

the gunsight is very unstable when tracking a target aircraft.

Fixing these deciencies may require changes to the mission

systems software that controls symbology to the helmet, or to

the radar software, as the program is working to nalize the

last version of Block 3F. Plans to begin aimed ight testing

of the gun on the F-35A were planned for this fall, but will

likely not start until December 2016, at the earliest.

• Because of the late testing of the gun and likelihood of

additional discoveries, the program’s ability to deliver gun

capability with Block 3F before IOT&E is at risk, especially

for the F-35B and F-35C, which have not yet red the gun in

ight.

Weapons Demonstration Events by the Operational Test Teams

• The JOTT and the associated Service operational test

squadrons (VMX-1, 31TES, and 422TES) assigned

to Edwards AFB, California, and Nellis AFB, Nevada

accomplished 6 air-to-air missile events, 19 GBU-31/32

JDAM air-to-ground events, and 28 GBU-12 laser guided

bomb events during 2016. For one of these events, the

team accomplished one combined AMRAAM missile with

one GBU-12 laser guided bomb event, as described in the

AMRAAM Air-to-Air Missile Event Table on the following

page. These weapon delivery events were accomplished

on range complexes at the Naval Weapons Center China

Lake, California; Marine Corps Air Station Yuma, Arizona;

and Eglin AFB, Florida. All of the OT weapon events were

planned and accomplished in operationally representative

scenario proles constructed to evaluate the F-35’s ability

to nd-x-track-target-engage-assess airborne and xed and

moving ground targets.

• The following tables and accompanying assessments show

the weapon events, aircraft Block conguration, date

accomplished, and results.

FY16 DOD PROGRAMS

78 F-35 JSF

AMRAAM Air-to-Air Missile Events Accomplished by Operational Test Teams

Event

Identier

Event Description

Aircraft Block

Software

Conguration

Date

Accomplished

Results

WDA-108 Cruise Missile Defense 3iR6.01 May 16

This event was a re-shoot of a developmental test event. The reshoot

was required by the operational test community because of control

room workarounds needed during the DT event. The OT prole was

successful.

OT 2.1

2 F-35 aircraft in MADL network attacking

one F-16 drone target with jamming

2BR5.3 Aug 16

Prole did not meet test objectives due to issues with the target

presentation. Data analysis in progress.

OT 2.2

2 F-35 aircraft in MADL network defending

against an o-boresight attacker

2BR5.3 Aug 16

Partially successful. Missile guided to objective target, however

secondary objective compromised due to issues with the target

presentation. Data analysis in progress.

OT 2.3

2 F-35 aircraft in MADL network vs 2

jamming equipped F-16 drones

2BR5.3 Aug 16

Prole did not meet test objectives due to issues with the target

presentation. Data analysis in progress.

OT 2.4

F-35 combined Air-to-Air AMRAAM and

GBU-12 Air-to-Ground prole

2BR5.3 Aug 16

Primary test objective to conrm ability of the F-35 to support a laser

guided bomb to impact while simultaneously supporting a missile

inight was successful. Secondary objective was unsuccessful due to

issues with the target presentation.

MAWTS-2

2 F-35 aircraft attacking a high closure rate

supersonic target

2BR5.3 Aug 16

This prole was a USMC engagement scenario to support ongoing

tactics development. Prole objective was successful

Air-to-Air General Observations

• The operational test teams completed the missile proles in

accordance with the DOT&E-approved test plan; however,

some weapons integration objectives were not successful

due to the drone target presentation failures (details are

classied). The failures in the drone target presentations

prevented either the primary or secondary test objectives to

verify the F-35’s capability to complete the nd-x-track-

target-engage-assess re control thread. The test team is

conducting data analyses to determine whether engineering

characterization runs or re-shooting of the proles are

required.

• Although four of the ve missile events fell short of

addressing all of the specic data objectives, they were

successful in identifying key deciencies in the ability

of the aircraft to support selected missile functionality,

stores management system anomalies, and the instability

of the shoot cues provided to the pilot to support missile

employment. Data analyses to identify root cause for all the

noted deciencies are ongoing and the operational test team

will recommend specic mission systems software xes to

address the noted deciencies.

GBU-31/32 Joint Direct Attack Munition (JDAM) and GBU-12 Laser Guided Bomb (LGB) Air-to-Ground Event Summary

Weapon Type Number of Weapons Events

F-35

Variant****

Date

Accomplished

Results

GBU-12 LGB 28 Laser Guided Bomb (LGB) Events*

21 F-35A

Jan to July

2016

22 successful/6 partially successful*** events.

7 F-35B

GBU-31 or

GBU-32 JDAM

15 GBU-31 (BLU-109) Events (8 inert/7 live)** F-35A 10 successful/5 partially successful***

3 inert GBU-32 (Mk-83) Events** F-35B 2 successful/1 partially successful***

*GBU-12 OT events were conducted against an operationally representative mix of xed and moving targets; self-, airborne buddy-, and ground tactical control party

target-lasing; target cueing via voice, VMF digital, and F-35 shoot-list sharing via MADL.

**JDAM GBU-31/32 events were accomplished against an operationally representative mix of xed target coordinates consisting of: pre-planned targeted coordinates,

F-35 self-targeting using SAR map and EOTS derived coordinates, and target cueing via voice, VMF digital, and F-35 shoot-list sharing via MADL.

***Air-to-Ground fully successful missions achieved weapon miss distances within expected mean radial error. Partially successful missions were cases where the

weapon was employed but with larger miss distances and observed mission systems issues described below.

****Mission Systems software for all variants was 2BS5.2 or 2BS5.3

FY16 DOD PROGRAMS

F-35 JSF 79

Air-to-Ground General Observations

• Although initial observations from weapons integration

can be characterized in general, detailed data analyses are

ongoing to determine precise mean radial error results for

both the LGB and JDAM weapons delivery events, and

to identify root causes for the observed mission systems

deciencies and weapon delivery issues.

• The JDAM predictive launch acceptability region (LAR)

and dynamic launch zone (DLZ) information were

consistently in error compared to the expected pilot drop

cues calculated from both the JDAM truth model and initial

DT characterizations. In the majority of the OT JDAM

drops, there were wide discrepancies between the LAR

presentations to the pilot via the HMDS, the corresponding

presentations on the in-cockpit controls and displays, and

the actual JDAM in-weapon LAR. In a number of cases,

the mission systems bombing cues available to the pilot via

the Tactical Situation Display on the Panoramic Cockpit

Display were in conict with the HMDS shoot cues and the

DLZ. This inconsistency is both confusing to the pilot and

can result in erratic and inaccurate weapon impact relative

to the target desired impact point. Also, the tactical displays

available to the pilot did not allow the pilot to conrm

the actual target coordinates passed to the weapon. This

conrmation of the in-weapon target coordinates is usually

required by rules of engagement (ROE) in operational areas

in order to enable positive target information conrmation to

the ground controllers prior to clearance to drop any weapon.

The F-35 in the Block 2B or Block 3i conguration is not

currently able to comply with these ROE.

• In general, pilots were able to use the F-35 Synthetic

Aperture Radar (SAR) mapping function to derive

weapons quality coordinates, which are adequate to deliver

ordinance on target. Pilots were also able to share the

SAR-map-derived coordinates between ight members to

validate and conrm target positions and coordinates prior to

releasing weapons.

• The EOTS was not able to provide the pilot with sufcient

resolution at tactical employment ranges to enable a positive

ID on the intended target. However, the EOTS generally

was able to track targets, both moving and stationary, but

only after the target identication was conrmed by an

external source or multiple sources. However, there are still

signicant tracking limitations, as evidenced by a new, open

Category 1-High deciency titled “EOTS TFLIR Tracker

Unable to Point or Area Track.” The EOTS system also was

able to generate accurate weapon quality coordinates when

cued to the correct target.

• The lack of any lead-point-compute or lead-laser guidance

in the F-35 EOTS system required rule-of-thumb pilot

techniques to provide limited capability with the GBU-12

on moving targets. The OT moving target attacks were

generally successful; however, the successes relied on high

levels of pilot experience and were not enabled by the F-35

mission systems. While the rule-of-thumb procedures

allowed the technical requirements of the weapons delivery

event to be met, they did not allow the pilot to maintain

positive target ID using the PVI procedures to designate,

track, and employ the weapon for the full attack timeline.

Most importantly, these procedures would likely not

have met the current positive target ID requirements for

operational employment rules of engagement. Due to these

limitations, which threaten the effectiveness of the F-35

to engage moving targets, the program and Services are

exploring other options to meet this ORD requirement. One

option, which is being considered by the Air Force, is to

integrate the GBU-49, a elded weapon that has similar size,

weight, and interfaces as the GBU-12, or a similar weapon

that does not require lead-laser guidance, in Block 3F.

Otherwise, the program plans to develop and eld lead-laser

guidance in Block 4.2, which would be delivered in CY22, at

the earliest. However, because of the similarities, the GBU-

49 could be quickly integrated with Block 3F to provide a

robust moving target capability for the F-35 much earlier.

• Pilots were able to use the digital Variable Message Format

(VMF) system to communicate between F-35 aircraft

and tactical ground controllers. The VMF links and data

provided the expected data to both the pilot and the ground

parties. In previous developmental testing, the VMF

has exhibited signicant issues with both reliability and

accuracy; however, in the OT events the system was both

reliable and accurate. Data analysis is ongoing to determine

the differences between the uses of VMF in developmental

testing compared to the operational weapons test events. The

ground parties used in the operational testing were equipped

with the most up-to-date software, rmware, and hardware

and were staffed by fully qualied ground controllers.

• Pilots experienced multiple inight failures of the Fuselage

Remote Interface Unit (FRIU), an electronic component that

provides the interface between the aircraft avionics and all

weapon stations, which often disrupted the ground attack

prole. The failures resulted in degraded weapons at critical

phases of the target attack prole and required the pilots

to abort the attack, reset the FRIU to regain control and

communications with the weapon, and then recommit to a

follow-on target attack. Such target attack interruptions are

unacceptable for combat operations.

• Pilots consistently rated the Offboard Mission Support

(OMS) mission planning system as cumbersome, unusable,

and inadequate for operational use. As a result, the time

required for operational planners to build a mission plan

is excessive and cannot support current planning cycle

requirements for multiple aircraft combat missions.

Additionally, the post-mission download times are too long

to support operational debrieng requirements.

Pilot Escape System

• Testing of the pilot escape system in CY15 showed that the

risk of serious injury or death is greater for lighter-weight

pilots, which led to the decision by the Services to restrict

pilots weighing less than 136 pounds from ying the F-35.

FY16 DOD PROGRAMS

80 F-35 JSF

In an effort to reduce this risk, the program developed three

modications associated with the escape system and began

testing them in late CY15 and throughout CY16. These

modications include:

- Reduction in the weight of the pilot’s Generation III helmet

(the new helmet is called Gen III Lite) to reduce the effect

of forces on the pilot’s neck during the ejection sequence.

- Installation of a switch in the seat that allows

lighter-weight pilots to select a slightly delayed activation

of the main parachute. This delay allows the drogue chute,

which deploys almost immediately during the ejection

sequence, to further slow and align the pilot before the

main parachute deploys. This delay is designed to reduce

the severity of loads on the neck experienced during

opening shock.

- The addition of a Head Support Panel (HSP) between the

risers of the parachute designed to prevent the pilot’s neck

from “snapping back” through the risers during the opening

of the main parachute.

• Concerned with the problems with the escape system and

the possibility of more discoveries, the U.S. Air Force asked

the JPO in June 2016 to gather and provide information on

potential costs and challenges to changing ejection seats

from the Martin Baker US16E seat currently installed in all

F-35 variants to the United Technologies ACES 5 seat as an

alternative for the F-35A.

• After prototypes of the design changes were available,

twenty-two qualication test cases were completed between

October 2015 and September 2016, with variations in manikin

weight, speed, altitude, helmet size and conguration, and the

seat switch settings. Seven of the tests were accomplished

with the lightweight (103 lbs) manikin. Data from these

tests showed that the HSP signicantly reduced neck loads

under conditions that forced the head backwards, inducing a

rearward neck rotation, during the ejection sequence. Data

also showed that the seat switch delay reduced the opening

shock from the main parachute for lighter-weight pilots at

speeds greater than 160 knots. Results of the additional tests

were provided to the Services in late CY16 to update their risk

assessments associated with ejections. Despite the improved

results, the extent to which risks have been reduced to lighter-

weight pilots (i.e., less than 136 pounds) by the modications

to the escape system and helmet is still to be determined by

these analyses. If the Services accept the risk associated with

the modications to the escape system for pilots weighing

less than 136 pounds, restrictions will likely remain in effect

until aircraft have the modied seat with the switch and

HSP installed, and the Gen III Lite helmets are procured and

delivered to the applicable pilots in the eet.

• The program plans to start retrotting elded F-35s with

the modications to the ejection seats in February 2017 and

delivering aircraft with the upgraded seat in Lot 10, starting in

January 2018. The Gen III Lite helmets will be included with

the Lot 10 aircraft delivery, and will be delivered starting in

November 2017. If these delivery timelines are met, the Air

Force may open F-35 pilot training to lighter-weight pilots

(i.e., below 136 pounds) as early as December 2017.

• Part of the weight reduction to the Gen III Lite HMDS

involved removing one of the two visors (one dark, one clear).

As a result, pilots that will need to use both visors during a

mission (e.g., during transitions from daytime to nighttime),

will have to store the second visor in the cockpit. However,

there currently is not adequate storage space in the cockpit for

the visor; the program is working a solution to address this

problem.

• The program has yet to complete additional testing and

analysis needed to determine the risk of pilots being harmed by

the Transparency Removal System (which shatters the canopy

rst, allowing the seat and pilot to leave the aircraft) during

ejections in other than ideal, stable conditions (such as after

battle damage or during out-of-control situations). Although

the program completed an off-nominal rocket sled test with the

Transparency Removal System in CY12, several aspects of the

escape system have changed since then, including signicant

changes to the helmet, which warrant additional testing

and analyses. DOT&E recommends the program complete

these tests, in a variety of off-nominal conditions, as soon as

possible, so that the Services can better assess risk associated

with ejections under these conditions.

Static Structural and Durability Testing

• Structural durability testing of all variants using full-scale test

articles continues, with plans for each variant to complete three

full lifetimes (one lifetime is 8,000 equivalent ight hours, or

EFH). Although all variants are scheduled to complete testing

before the end of SDD, the complete teardown, analyses,

and damage assessment and damage tolerance reporting is

not scheduled to be completed until August 2019. Testing

on all variants has led to discoveries requiring repairs and

modication to production designs and retrots to elded

aircraft.

• F-35A durability test article (AJ-1) completed the second

lifetime of testing, or 16,000 EFH in October 2015. After

completing second lifetime inspections, third lifetime testing

began on March 11, 2016. As of November 16, 2016,

20,000 EFH, or 50 percent of the third lifetime had been

completed. Third lifetime testing is projected to complete in

December 2017.

• F-35B durability test article (BH-1) completed 14,051 EFH

by November 17, 2016, which is 6,051 hours (76 percent) into

the second lifetime. Due to the amount of modications and

repairs to bulkheads and other structures in the current F-35B

ground test article, it may not be adequate to continue testing

and a new one may be needed and durability testing repeated

to ensure adequate lifetime testing is completed. The program

needs to conduct an assessment to determine the extent to

which the results of further durability testing are representative

of production aircraft and if necessary procure another test

article for the third life testing.

FY16 DOD PROGRAMS

F-35 JSF 81

- Two main wing carry-through bulkheads, FS496 and

FS472, are no longer considered production-representative

due to the extensive repairs that have been required. The

program plans to continue durability testing, repairing

the bulkheads as necessary, through the second lifetime

(i.e., 8,001 through 16,000 EFH), which is projected to be

complete in February 2017.

- Prior to CY16, testing was halted on September 29,

2013, at 9,056 EFH, when the FS496 bulkhead severed,

transferred loads to, and caused cracking in the adjacent

three bulkheads (FS518, FS472, and FS450). The repairs

and an adequacy review of the repairs to support further

testing were completed on December 17, 2014, when the

program determined that the test article could continue

testing. Testing restarted on January 19, 2015, after a

16-month delay.

- The program determined that several of the cracks

discovered from the September 2013 pause at 9,056 EFH

were initiated at etch pits. These etch pits are created by

the etching process required prior to anodizing the surface

of the structural components; anodizing is required for

corrosion protection. Since the cracks were not expected,

the program determined that the etch pits were more

detrimental to fatigue life than the original material design

suggested. The program is currently developing an

analysis path forward to determine the effect on the overall

fatigue life.

- After the durability test completed 11,915 EFH on August

13, 2015, the load cycling was stopped to allow removal

and replacement of the FS496 bulkhead outer segments

(both left- and right-hand sides), removal and replacement

of the left-hand-side aft fuselage close-out frame, repairs

to the engine thrust mount shear webs, installation of

fasteners at the FS518 frame, maintenance of the right-

hand-side EHAS panel, repairs to the right-hand-side of the

mid-fairing longeron, and repairs to the FS556 upper arch.

The entire repair activity took about 9 months, with an

85-EFH testing effort conducted in early March 2016 that

reached 12,000 EFH.

- Testing resumed in early May 2016, reached 13,000 EFH

in mid-June 2016, and then stopped for another month to

repair the FS472 lower ange.

- Testing resumed in mid-July. At 13,086 EFH, cracks

were discovered on the forward fuselage including FS236

bulkhead, left-hand-side FS223 frame, and right-hand-side

FS191 upper frame.

- Testing continued with buffet loads until it reached 13,980

EFH before stopped to implement fuselage repairs in

August 2016.

- Testing resumed on September 17 and had reached

14,051 EFH on November 17, 2016.

• F-35C durability test article (CJ-1) completed the second

lifetime of testing, or 16,000 EFH on October 29, 2016.

The third lifetime testing is scheduled to begin in late

December 2016.

- In October 2015 with 13,731 EFH accomplished, cracks

were discovered on the left-hand side and right-hand

side of one wing front spar and one left-hand-side wing

forward root rib; this discovery was considered signicant

because wing spar and wing root rib are primary structural

components and the cracks were not predicted by the

nite element model (FEM) used in the design of these

components. The repairs took over 3 months before the test

resumed in early February 2016.

- On February 9, 2016, with 13,827 EFH accomplished, a

crack was found on the left-hand-side inverter/converter/

controller and power distribution center/inverter bay oor.

Testing continued with catapult and trap load cycling.

- In late February 2016 with 13,931 EFH accomplished,

cracks were found on the left- and right-hand sides of the

FS496 bulkhead anges, which were deemed signicant.

The repairs took another 3 months to complete before the

test resumed in May 2016.

- In August 2016 with 14,831 EFH accomplished, small

cracks were found on the right-hand-side armpit (below

wing root) and were quickly repaired with a simple blend.

- In August 2016 with 14,892 EFH accomplished, cracks

were found on the FS518 lower frame and some nearby

broken fasteners. A weld repair for the titanium frame

was completed. Further investigation revealed cracks

on the right- and left-hand-side wing rear spars. While a

repair disposition was being developed, the durability test

resumed with loading only for catapult takeoffs and carrier

trap landings.

• The program plans to use Laser Shock Peening (LSP), a

mechanical process designed to add compressive residual

stresses in the materials, in an attempt to extend the lifetime

of the FS496 and FS472 bulkheads in the F-35B. The

rst production line cut-in of LSP will start with Lot 11

F-35B aircraft. Earlier Lot F-35B aircraft will undergo

LSP processing as part of a depot modication. Testing is

proceeding in three phases: rst, coupon-level testing to

optimize LSP parameters; second, element-level testing to

validate LSP parameters and quantify life improvement; and

third, testing of production and retrot representative articles

to verify the service life improvements. All three phases are

in progress, with full qualication testing scheduled to be

completed in August 2017. As of December 1, 2016, 122 of

211 durability tests had been conducted with results within

expectations, which is a 58 percent completion.

Joint Simulation Environment (JSE)

• The JSE is a man-in-the-loop, mission systems software-in-

the-loop simulation developed to meet the operational test

requirements for Block 3F IOT&E. The Program Ofce made

the decision in September 2015 to stop development on the

contractor’s effort to build a similar system, the Verication

Simulation (VSim), instead tasking the Naval Air Systems

Command (NAVAIR) to lead the building of a government-

owned Joint Simulation Environment (JSE), with the

FY16 DOD PROGRAMS

82 F-35 JSF

contractor providing only the F-35 aircraft and sensor models.

However, negotiations for the F-35 models have not yet been

successful, which has prevented NAVAIR from fully dening

the simulation’s architecture and environment (the virtual

software environment in which aircraft, sensor, and threat

models interact with one another).

• While the Program Ofce continued to negotiate with

the contractor, and had success in meeting the hardware

requirements (facilities, cockpits, etc.), the lack of denition

of the simulation environment makes any integration schedule

not credible. In the next year, the program must acquire the

F-35 models, integrate them into an as-yet undened and

undeveloped battlespace environment, complete development

of several dozen threat aircraft and surface system models,

ensure that aircraft sensor models correctly perceive the threat

system models, and validate the entire simulation. Previous

efforts of this magnitude have taken several years, so it is

unlikely that NAVAIR will complete the project as planned

in time to support IOT&E. Current Program Ofce estimates

are that JSE will deliver late to need in May 2019, but before

the end of IOT&E. Verication, Validation, and Accreditation

(VV&A) activities remained effectively stalled in 2016 and are

also a very high risk to timely completion of the simulation.

• Without a high-delity simulation, the F-35 IOT&E will not be

able to test the F-35’s full capabilities against the full range of

required threats and scenarios. Nonetheless, because aircraft

continue to be produced in substantial quantities (essentially

all of which require modications and retrots before being

used in combat), the IOT&E must be conducted without

waiting for the JSE, to demonstrate F-35 combat effectiveness

under the most realistic conditions that can be obtained in

ight testing, once the aircraft hardware and software meet

the IOT&E entrance criteria, which is expected to occur long

before the completion and successful VV&A of JSE. It is

now clear that the JSE will not be available and accredited in

time to support the Block 3F IOT&E. The currently approved

IOT&E detailed test design, which was developed entirely

around open-air ight testing, mitigates the lack of an adequate

simulation environment as much as possible.

Live Fire Test and Evaluation (LFT&E)

F-35C Full-Scale Aft Fuselage and Empennage Structure Test

• The F-35 LFT&E program completed the F-35C full-scale

aft fuselage and empennage structure tests. The Navy’s

Weapons Survivability Laboratory in China Lake, California,

accomplished three test events using the CG:0001 full scale

structural test article. The tests evaluated the ability of the

vertical tail and aft boom structure to withstand damage from

high-explosive incendiary (HEI) projectile and simulated

Man-Portable Air Defense System (MANPADS) threats. A

preliminary review of the test results indicates that:

- The F-35 vertical tail is capable of withstanding an HEI

projectile impact. The threat can target and fail one

attachment lug but the remaining lugs demonstrated their

ability to handle normal ight loads after the impact.

However, the pilot receives no alerts from the Integrated

Caution, Advisory and Warning (ICAW) system from this

type of structural damage, so there is a potential that a

damaged vertical tail could fail without warning the pilot

if the pilot demands higher than normal ight loads on the

vertical tail after the damage occurs.

- Two MANPADS shots were completed against the aft

boom structures, which support the horizontal and vertical

tails. Combined with results from earlier tests on an

F-35A and F-35B test articles, these tests showed that the

structures are sufciently robust against these threats to

retain all control surfaces. Although damage to a single

control surface actuator is possible, earlier ight control

tests showed sufcient controllability within a limited

ight envelope to allow controlled ight back to a safe

area where the pilot could eject.

- The MANPADS tests demonstrated the potential for

damage to the fueldraulics system – the engine fuel-based

hydraulics system – which can result in a sustained

re leading to further damage to the aircraft and a pilot

ejection over enemy territory. The data will be used

to support an assessment in 2017 that will determine

the contribution of this issue to the overall aircraft

vulnerability.

- While extended res occurred in the MANPADS tests,

there has been no effort expended to determine what

catastrophic damage might result and the timeframe for

that to occur. Current procedures are for an immediate

ejection upon determination of a sustained re. However,

if the time-to-failure could be established for this sort of

re, it might allow the pilot time to depart a combat area

and eject somewhere relatively safe. Further analysis of

these test results and the related issue are needed.

PAO Shut-O Valve

• The program has not provided an ofcial decision to

reinstate this vulnerability reduction feature. There has

been no activity on the development of the PAO-shut-off

valve technical solution to meet criteria developed from

2011 live re test results. As stated in several previous

reports, this aggregate, 2-pound vulnerability reduction

feature, if installed, would reduce the probability of pilot

incapacitation, decrease overall F-35 vulnerability, and

prevent the program from failing one of its vulnerability

requirements.

Vulnerability to Unconventional Threats

• The full-up, system-level chemical-biological

decontamination test on an SDD aircraft, which began

4QFY16 and is scheduled to end in 2QFY17 at Edwards

AFB, was supported by two risk-reduction events:

- A System Integration Demonstration of the proposed

decontamination equipment and shelter was conducted on

an F-16 test article during 1QFY15 at Edwards AFB to

simulate both hot air chemical and hot/humid air biological

decontamination operations. Extensive condensation

inside the shelter and on the test article during the

FY16 DOD PROGRAMS

F-35 JSF 83

hot/humid air biological decontamination event indicated

the need for process and shelter modications.

- A 2QFY16 event demonstrated that a modied system

process and a better insulated shelter can maintain

adequate temperature and humidity control inside the

shelter, even in a cold-weather environment.

• The test plan to assess chemical and biological

decontamination of pilot protective equipment is not

adequate. Compatibility testing of protective ensembles

and masks has shown that the materials survive exposure

to chemical agents and decontamination materials and

processes, but the program has neither tested nor provided

plans for testing the HMDS currently being elded.

Gen II HMDS compatibilities were determined by analysis,

comparing HMDS materials with those in an extensive DOD

aerospace materials database. A similar analysis is planned

for the Gen III HMDS design. However, even if material

compatibilities were understood, there are no plans to

demonstrate a process that could adequately decontaminate

either HMDS from chemical and biological agents.

• The Joint Program Executive Ofce for Chemical and

Biological Defense approved initial production of the F-35

variant of the Joint Service Aircrew Mask (JSAM-JSF)

during 1QFY16. This ofce and the F-35 Joint Program

Ofce are integrating the JSAM-JSF with the HMDS, which

is undergoing Safety of Flight testing.

• The Navy evaluated an F-35B aircraft to the EMP threat

level dened in Military-Standard-2169B. Follow-on tests

on other variants of the aircraft, including a test series to

evaluate any Block 3F hardware/software changes, are

planned for FY16-17.

Gun Ammunition Lethality and Vulnerability

• The 780th Test Squadron at Eglin AFB, Florida, completed

the ground-based lethality test of the PGU-47/U Armor

Piercing High Explosive Incendiary with Tracer (APHEI-T)

round (also known as Armor Piercing with Explosive

(APEX)) against armored and technical vehicles, aircraft,

and personnel-in-the-open targets. Ground-based lethality

tests for the APEX correlated well with pre-test predictions

for the round penetrations, but potential problems were

discovered with fuze functioning when impacting rolled

homogeneous armor at high obliquity. Nammo, the

Norwegian manufacturer, conducted additional testing to

identify the cause of the dudded rounds during the ground

tests and subsequently modied the fuze design to increase

reliability. The program will determine the effect of the

ground-based lethality test data on the ammunition lethality

assessment.

• Per the current mission systems software schedule, the

weapons integration characterization of the gun and sight

systems will not be ready for the air-to-ground gun strafe

lethality tests until December 2016, at the earliest. Strang

targets will include a small boat, light armored vehicle,

technical vehicle (pickup truck), and plywood mannequins

for each round type tested.

Operational Suitability

• The operational suitability of all variants continues to be less

than desired by the Services. Operational and training units

must rely on contractor support and workarounds that would

be challenging to employ during combat operations. In the

past year some metrics of suitability performance have shown

improvement, while others have been at or declined. Most

metrics still remain below interim goals to achieve acceptable

suitability by the time the eet accrues 200,000 ight hours,

the benchmark set by the program and dened in the

Operational Requirements Document (ORD) for the aircraft to

meet reliability and maintainability requirements. This level

of maturity is further stipulated as 75,000 ight hours for the

F-35A, 75,000 ight hours for the F-35B, and 50,000 ight

hours for the F-35C.

• Reliability growth has stagnated, so it is highly unlikely that

the program will achieve the ORD threshold requirements at

maturity for the majority of reliability metrics, most notably

the Mean Flight Hours Between Critical Failures, without

redesigning aircraft components.

• Aircraft eet-wide availability averaged 52 percent for 12

months ending October 2016, compared to the modest goal of

60 percent. It is important to note that the expected combat

sortie rates will require signicantly greater availability than

60 percent; therefore, if the F-35 is to replace legacy aircraft

for combat taskings, availability will likely need to improve to

near 80 percent.

• Monthly availability had been averaging in the mid-30s to

low-40s percent for the 2-year period ending September 2014.

Monthly availability then increased rapidly and signicantly

from October to December, peaking at 56 percent in December

2014. However, since then it has remained at, centering

around the low-50s percent with no strong improving trend

over time.

• Only two out of nine reliability metrics that have ORD

requirement thresholds have improved since last year’s report.

All nine are below the interim goals that were set to determine

if the metrics will meet the thresholds by maturity. None are

within 5 percent of their interim goal, whereas previously,

several of these metrics were reported as being above or

within 5 percent of their interim goal. In particular, reliability

metrics related to critical failures have decreased over the

past year. This decrease in reliability correlates with the

simultaneously observed decline in the Fully Mission Capable

(FMC) rate for all variants, which measures the percentage of

aircraft not in depot status that are able to y all dened F-35

missions. The eet-wide FMC rate peaked in December 2014

at 62 percent and has fallen steadily since then to 21 percent in

October 2016.

• In addition to the nine ORD metrics, there are three contract

specication metrics, Mean Flight Hours Between Failure

scored as “design controllable,” or DC, one for each

variant. DC failures are equipment failures due to design

aws considered to be the fault of the contractor, such as

components not withstanding stresses expected to be found

FY16 DOD PROGRAMS

84 F-35 JSF

in the normal operational environment. It does not include

failures caused by improper maintenance, or caused by

circumstances unique to ight test. This metric exhibited the

highest rate of the growth in the past and, for this metric, all

variants are currently above program target values for this stage

in development. However, since May 2015, DC reliability has

generally decreased or remained at as well.

• Although most measures of reliability have not improved

signicantly over the past year, three of six measures of

maintainability have improved slightly. Maintainability metrics

record the amount of time required to troubleshoot and repair

faults on the aircraft. Additionally, the number of ight hours

each aircraft ies per month, known as the utilization rate, has

also increased marginally.

• F-35 aircraft spent 9 percent more time down for maintenance

than intended (eet average of 16.4 percent compared

to 15 percent goal), and waited for parts from supply for

71 percent longer than the program targeted (eet average

of 17 percent compared to goal of 10 percent). At any given

time, from 10 to 20 percent of aircraft were in a depot facility

or depot status at the home base for major rework or planned

upgrades. Of the remaining aircraft not in any depot status, on

average less than a third were able to y all missions of even

a limited capability set that is associated with the Block 2B or

Block 3i aircraft.

• Accurate suitability measures rely on adjudicated data from

elded operating units. A Joint Reliability and Maintainability

Evaluation Team (JRMET), composed of representatives

from the Program Ofce, the JOTT, the contractor (Lockheed

Martin), and Pratt and Whitney (for engine records), reviews

maintenance data to ensure consistency and accuracy for

reporting measures; government representatives chair the

team. However, the Lockheed Martin database that stores

the maintenance data, known as the Failure Reporting and

Corrective Action System (FRACAS), was not in compliance

with U.S. Cyber Command information assurance policies

implemented in August 2015 through late summer of 2016.

Because of this non-compliance, government personnel were

not able to access the database via government networks,

preventing the JRMET from holding regularly scheduled

reviews of maintenance records for nearly a year, other than

a few ad hoc reviews. Regular JRMET meetings resumed

in September 2016, but the program is currently working

through reviewing a large backlog of un-adjudicated eld

data. The program restarted publishing monthly reliability and

maintainability (R&M) status reports from adjudicated data in

October 2016, after roughly a year-long hiatus.

F-35 Fleet Availability

• Aircraft availability is determined by measuring the percent of

time individual aircraft are in an available status, aggregated

over a reporting period (e.g., monthly). The program assigns

aircraft that are not available to one of three categories of

status: Not Mission Capable for Maintenance (NMC-M); Not

Mission Capable for Supply (NMC-S); and depot status.

- Program goals for these not-available categories have

remained unchanged since 2014, at 15 percent for

NMC-M, 10 percent for NMC-S, and 15 percent of

the eet in depot status. Depot status is primarily for

completing the modications required to bring currently

elded aircraft in compliance with their expected

airframe structural lifespans of 8,000 ight hours and to

incorporate additional mission capability. The majority

of aircraft in depot status are located at dedicated depot

facilities for scheduled modication periods that can

last several months, and they are not assigned as a part

of the operational or training eet during this time. A

small portion of depot activity can occur in the eld

when depot eld teams conduct a modication at a main

operating base, or affect repairs beyond the capability of

the local maintenance unit. Similar to being at a depot

facility, aircraft are temporarily assigned to depot status

during these periods and are not considered a part of the

operational or training eet.

- These three not-available category goals sum to

40 percent, resulting in a eet-wide availability goal of

60 percent for 2016.

- In addition to these overall program goals, the program

has implemented a Performance Based Logistics (PBL)

construct with Lockheed Martin that ties contract

incentive awards to a slightly different set of tailored eet

performance targets. These tailored targets prioritize

improvement efforts for Marine Corps F-35B performance

as the rst branch to declare Initial Operational Capability

(IOC), and also because the F-35B variant has shown the

lowest overall availability performance. Current PBL-

based goals are 53 percent availability, 35 percent FMC,

and 70 percent mission effectiveness rates for the F-35B

training and operational eets assigned to Marine Corps

Air Station (MCAS) Beaufort and MCAS Yuma. The

majority of the incentive structure is tied to these goals.

To ensure Lockheed Martin continues to try to improve

performance across the board, a smaller portion of the

incentive fee is tied to overall eet performance metrics

of 60 percent F-35A, 50 percent F-35B, and 60 percent

F-35C availability, regardless of operating site.

• Aircraft monthly availability averaged 52 percent for the

12-month period ending October 2016 in the training and

operational eets, with a maximum availability of 55 percent

in May 2016 and a minimum availability of 44 percent in

October 2016. This is only a minor improvement over the

average 51 percent monthly availability reported in the

FY15 DOT&E Annual Report for the 12 months ending

October 2015. Further, some groups of aircraft continue to

experience minimum availability well below 50 percent.

- In no month did the overall eet exceed its goal of

60 percent availability. Only the F-35C variant exceeded

the 60 percent goal, in 6 of 12 months, with a maximum

availability of 71 percent in April 2016. The F-35A and

F-35B variants never exceeded 60 percent, but the F-35A

FY16 DOD PROGRAMS

F-35 JSF 85

achieved 59 percent in May 2016 and the F-35B reached a

maximum 50 percent in January, April, and July 2016.

- The table below summarizes aircraft availability by

operating location for the 12-month period ending

October 2016. The rst column indicates the average

availability achieved for the whole period, while the

maximum and minimum columns represent the range

of monthly availabilities reported over the period. The

number of aircraft assigned at the end of the reporting

period is shown as an indicator of potential variance in

availability. Sites are arranged in order of when each

site began operation of any variant of the F-35, and then

arranged by variant for sites operating more than one

variant. The Marine Corps terminated F-35B operations

at Eglin AFB in February 2015, so there were no F-35Bs

at that site for the 12-month period of this report; thus,

that entry, previously reported in the FY15 DOT&E

Annual Report, has been removed. The Navy operational

test squadron at Edwards AFB received its rst F-35C in

August 2016, the only new operating site to stand up since

the FY15 DOT&E Annual Report.

- Trend analysis of monthly eet availability from

August 2012 through October 2016 showed a weak rate

of improvement of approximately 5 percent growth per

year over this period. This is consistent with the growth

rate reported in the DOT&E FY15 Annual Report – but,

again, the growth was neither steady nor continuous. The

majority of this growth still results from a concentrated

increase in availability that occurred during the months

of September 2014 through December 2014. Analysis of

availability from January 2015 through October 2016, the

time period after this concentrated increase, shows a more

modest less than 1 percent annual growth rate, which is in

better agreement with recent observations.

- The combined eet of designated, instrumented OT

aircraft currently at Edwards AFB, which was built in

F35 AVAILABILITY FOR 12MONTH PERIOD ENDING OCTOBER 2016

Operational

Site

Average Maximum Minimum

Aircraft

Assigned

Whole Fleet 52% 55% 44% 178

Eglin F-35A 38% 49% 32% 25

Eglin F-35C 60% 71% 54% 21

Yuma F-35B 55% 62% 40% 19

Edwards F-35A 53% 74% 40% 8

Edwards F-35B 46% 64% 30% 7

Edwards F-35C

27% 40% 4% 2

Nellis F-35A 50% 62% 42% 13

Luke F-35A 61% 68% 44% 44

Beaufort F-35B 43% 53% 33% 24

Hill F-35A 57% 80% 22% 15

1. Data do not include SDD aircraft.

2. Aircraft assigned at the end of October 2016.

3. Edwards AFB F-35C operations began August 2016.

Lots 3 to 5, averaged 48 percent availability from January

to October 2016. Seventeen instrumented OT aircraft

were assigned to Edwards AFB as of October 2016. This

is well-short of the target of 80 percent that will be needed

to conduct an efcient IOT&E, or combat operations.

• Due to concurrent development and production, which

resulted in delivering operational aircraft before the program

has completed development and nalized the aircraft

design, the Services must send the current eet of F-35

aircraft to depot facilities. This is to receive modications

that have been designed since the aircraft were originally

manufactured and are now required for full capability. Some

of these modications are driven by faults in the original

design that were not discovered until after production had

started, such as major structural components that do not

meet the requirements for the intended lifespan, and others

are driven by the continuing improvement of the design of

combat capabilities that were known to be lacking when the

aircraft were rst built. These modications are a result of

the concurrency of production and development and cause

the program to expend resources to send aircraft for major

re-work, often multiple times, to keep up with the aircraft

design as it progresses. Since SDD will continue at least

to the middle of 2018, and by then the program will have

delivered nearly 200 aircraft to the Services in other than

the 3F conguration, the depot modication program and its

associated concurrency burden will be with the Services for

years to come.

- Sending aircraft to depot facilities for several months

at a time to bring them up to Block 3i capability from

Block 2B (i.e., upgrading avionics processors) and to meet

life limit requirements, and eventually to the Block 3F

conguration, reduces the number of aircraft at eld sites

and thus decreases eet availability. For the 12-month

period ending October 2016, the proportion of the eet in

depot status averaged 15 percent, compared to 16 percent

for the 12-month period ending October 2015 stated in

the DOT&E FY15 Annual Report. The proportion of

aircraft in depot status was relatively at over the majority

of this period with little overall trend, ranging between a

maximum monthly value of 22 percent and a minimum

value of 11 percent. The maximum value of 22 percent

occurred in October 2016, and was partly driven by

one-time repairs to shedding foam insulation around

PAO lines in the fuel tanks for 15 elded F-35A aircraft.

DOT&E expects this rise in the depot rate to be a one-time

occurrence, and not indicative of a general trend.

- There is evidence from Program Ofce reports, however,

that later production lot aircraft achieve higher availability

rates than earlier lots. For example, for the period from

October 2015 to September 2016, accounting for 30 Lot 4

aircraft of all variants, each variant averaged a monthly

availability between 43 and 44 percent. For the same time

period and accounting for 33 Lot 7 aircraft of all variants,

each variant averaged a monthly availability between

64 and 68 percent, which was a statistically signicant

FY16 DOD PROGRAMS

86 F-35 JSF

increase. However, a signicant amount of this increase

in availability can be attributed to the newer lot aircraft

requiring fewer depot modications. Over this period

the Lot 4 aircraft averaged a monthly depot rate between

19 and 26 percent, depending on variant, whereas the

Lot 7 aircraft averaged a monthly depot rate between

0 and 6 percent, considering variant.

- Projections of depot rates beyond 2016 are difcult, since

testing and development are ongoing and discoveries

continue, including the need for redesigned outer wing

structure on the F-35C to accommodate AIM-9X missile

carriage. This structural modication was installed on an

F-35C developmental test aircraft for testing in late 2016.

Also, the program does not yet know the full suite of

modications that will be necessary to bring currently

produced aircraft up to the nal Block 3F conguration.

However, as the program continues to ramp up production

rates, the later lot aircraft, which generally require fewer

modications, will comprise a larger proportion of the

eet and may exert a downward inuence on the depot

percentage rate.

• To examine the suitability performance of elded aircraft,

regardless of how many are in the depot, the program reports

on the Mission Capable (MC) and Fully Mission Capable

(FMC) rates for the F-35 eet. The MC rate represents the

proportion of the eet that is not in depot status and that is

ready to y any type of mission (as opposed to all mission

types). This rate includes aircraft that are only capable of

ying training ights, however, and not necessarily a combat

mission. The FMC rate calculates only the proportion

of aircraft not in depot status that are capable of ying

all assigned missions and can give a better view into the

potential combat capability available in the elded units.

- F-35 aircraft averaged a 62 percent MC rate for the

12-month window ending in October 2016 considering

all variants, a slight decrease from the 65 percent reported

in the FY15 DOT&E Annual Report. The rate showed

little change over time, ranging from a minimum value of

57 percent to a maximum value of 66 percent

for the whole eet, and was relatively consistent

across variants as well. The F-35A achieved

the highest variant-specic rate at 64 percent,

followed by 63 percent for the F-35C, and

59 percent for the F-35B.

- The FMC rate continued to exhibit a steady

decline rst observed in 2015, and averaged

only 29 percent over the period, compared

to 46 percent reported in the FY15 DOT&E

Annual Report. The rate started at 32 percent in

November 2015, which was close to the peak of

33 percent in April 2016, but generally dropped

month over month to a minimum value of 21 percent by

October 2016. The FMC rate has not been consistent

across variants. The F-35A eet achieved the highest

average FMC rate for the period at 37 percent, followed by

the F-35C at 24 percent. The F-35B eet exhibited only

a 14 percent average FMC rate, however. Failures in the

Distributed Aperture System (DAS), electronic warfare

(EW) system, and Electro-Optical Targeting System

(EOTS) were the highest drivers pushing aircraft into

Partial Mission Capable (PMC) status.

- Analysis of the MC rate of each production lot reveals

that later lot aircraft have a greater MC rate than earlier

lot aircraft; the difference is less pronounced than the

comparison of availability, but still signicant. The

30 Lot 4 aircraft averaged between 52 and 61 percent MC

over this period by variant, compared to 68 to 73 percent

for the Lot 7 aircraft by variant.

- The OT eet at Edwards AFB averaged an MC rate of

53 percent from January to October 2016.

• The rst table below shows F-35 MC and FMC rates for

the total eet and each variant for the 12-month period

ending October 2016, including the average, maximum, and

minimum monthly values observed. The second table shows

F-35 availability and MC rates by production lot and by

variant for the 12-month period ending September 2016.

F35 MC AND FMC RATES BY VARIANT FOR 12MONTH PERIOD

ENDING OCTOBER 2016

Variant

MC FMC

Avg. Max Min Avg. Max Min

Fleet 62% 66% 57% 29% 33% 21%

F-35A 64% 70% 55% 37% 42% 27%

F-35B 59% 65% 53% 14% 17% 10%

F-35C 63% 73% 55% 24% 44% 13%

F35 AVAILABILITY AND MISSION CAPABLE RATES BY LOT

OCTOBER 2015 TO SEPTEMBER 2016

Lot

No. of Aircraft Availability Mission Capable

F-35A F-35B F-35C Total F-35A F-35B F-35C F-35A F-35B F-35C

2/3 14 13 - 27 33% 37% N/A 57% 54% N/A

4 10 17 3 30 44% 44% 43% 61% 59% 52%

5 22 3 7 32 51% 50% 57% 62% 52% 60%

6 23 6 7 36 62% 60% 67% 63% 66% 68%

7 22 7 4 33 67% 64% 68% 73% 68% 68%

8 14 3 3 20 49% 65% 79% 68% 65% 80%

FY16 DOD PROGRAMS

F-35 JSF 87

• The monthly NMC-M rate averaged 16 percent over the period

and was relatively stable, with a minimum value of 14 percent

and a maximum value of 20 percent. This rate achieved the

program goal of 15 percent, or lower, in 4 of the 12 months of

the period. It also shows a slight decreasing (improving) trend

over time that indicates with further improvement it may be

possible to achieve and sustain program targets within the next

calendar year.

- Completing directed modications or upgrades on

still-possessed aircraft in the eld also affects the NMC-M

rate. In such cases, squadron-level maintainers, instead of

the depot or contractor eld teams, are tasked to complete

Time Compliance Technical Directives (TCTDs). The

“time compliance” limits for these directives vary,

normally allowing the aircraft to be operated for a certain

period of time without the modication. This permits

maintenance personnel to do the work at an opportune time,

without taking the aircraft off the ight schedule to do so,

such as by combining the TCTD with other maintenance

activities. While maintainers accomplish these TCTDs, the

aircraft are designated as NMC-M status, and not in depot

status. Incorporating these TCTDs will drive the NMC-M

rate up (worse) until these remaining modications are

completed. Publishing and elding new TCTDs is

expected for a program under development and is needed

to see improvement in reliability and maintainability;

however, they inherently add to the maintenance burden in

the elded operational units.

• The NMC-S rate averaged 17 percent and showed no

signicant trend over the period. In no month did the

rate achieve the program goal of 10 percent or less, with

a minimum value of 14 percent and a maximum value of

20 percent.

- Several factors have contributed to the NMC-S rate

underperforming relative to its goal more than either the

NMC-M or depot not-available categories. First, the

program originally funded spares to a 20 percent NMC-S

rate. To determine the quantity and type of spares needed

to achieve this, the program used incorrect engineering

predictions that overestimated component reliability (eet

data were not available when this modeling was done early

in the program). Actual mean time between failures for

many components is lower than the forecasted values used

in the spares model. Second, contracting for spares has

often been late to need to support the rst aircraft delivery

for several of the initial production lots. Third, the program

has been late to stand up organic depot capabilities to repair

existing parts that have failed but can be refurbished instead

of being replaced with new parts. Such a capability would

reduce the strain on suppliers to produce more spare parts.

- The lack of spares available in the supply system is

driving operating units to take good parts from one NMC

aircraft and install them in other aircraft down for those

parts, bringing the latter back to available status. This

process, known as cannibalization, is performed by units

when supply cannot provide needed parts in a timely

manner. Cannibalization results in a signicant increase

in maintenance man-hours compared to replacing a bad

part with a new or repaired part. For the 12-month period

ending in October 2016, the monthly cannibalization rate

averaged 9.8 cannibalization actions for every 100 sorties

against a program goal of no more than 8 actions

for every 100 sorties. The eet met this goal in only

1 month, performing 6.2 cannibalizations per 100 sorties

in December 2015, but analysis over this period does

not demonstrate a statistically signicant trend in the

cannibalization rate.

- Modifying aircraft also has an effect on the NMC-S rate

as the Services can cannibalize parts from aircraft in the

depots to support eld units when replacement parts are

not otherwise available from normal supply channels or

stocks of spare parts on base. With the large number of

aircraft in depot status, the program may have been able to

improve the NMC-S rate by using depot cannibalizations,

instead of procuring more spare parts, or reducing the

failure rate of parts installed in aircraft, or improving

how quickly failed parts are repaired and returned to

circulation. If the Services endeavor to bring all of the

early lot aircraft into the Block 3F conguration, the

program will continue to have an extensive modication

program for several years. While this will continue to

provide opportunities for depot cannibalizations during

that time, once the Block 3F modications are complete,

there will be fewer aircraft in the depot serving as spare

parts sources and more in the eld requiring parts support.

If demand for spare parts remains high, this will put

pressure on the supply system to keep up with demand

without depot cannibalization as a source.

- While the eet was much closer to achieving the NMC-M

goal than the NMC-S goal, these two rates are not

necessarily completely independent. Specically, poor

diagnostics or difcult-to-conduct troubleshooting – issues

that are maintainability problems at root cause – can

drive the NMC-S rate up as well. For example, if

troubleshooting efforts initially isolate faults to incorrect

parts, units may inadvertently take good parts off the

aircraft, return them to the supply system for depot or

manufacturer checks, and demand replacement parts,

unnecessarily straining the supply system for repair actions

that will not resolve the fault. Units will report aircraft

in NMC-S status until these replacement parts arrive.

Once the unit receives and installs these parts, it would

discover that the original problem remains, and return the

aircraft to NMC-M status until further troubleshooting

hopefully isolates the correct part. Thus, actions to reduce

higher-than-targeted NMC-S rates may include improving

the accuracy of diagnostics and troubleshooting procedures

as well as increasing the availability of spare parts.

• The following table summarizes depot, NMC-M, and

NMC-S rates for the total F-35 eet and each variant for

the 12-month period ending October 2016, including the

average, maximum, and minimum monthly values observed.

FY16 DOD PROGRAMS

88 F-35 JSF

• Low availability is preventing the eet of elded operational

F-35 aircraft from achieving the originally planned, Service-

funded ying hour goals. The original Service beddown

plans were based on F-35 squadrons ramping up to a steady

state, xed number of ight hours per tail per month,

allowing for the projection of total eet ight hours.

- Since poor availability in the eld has shown that these

original plans were unexecutable, the Program Ofce has

since produced modeled-achievable projections of total

eet ight hours, basing these projections on demonstrated

eet reliability and maintainability data, as well as

expectations for future improvements. The most current

modeled-achievable projection is from March 2016.

- Through November 21, 2016, the eet had own

approximately 91 percent of the modeled-achievable

hours. This is an improvement since November 2015, the

date used in the FY15 DOT&E Annual Report, when the

eet had own 82 percent of modeled-achievable hours;

however, recent updates to the model revised the projected

hours downward. The completion of actual ight hours

against modeled-achievable ight hours was consistent

across all three variants, with each variant completing

between 90 or 96 percent of its variant-specic projection.

By comparison, the eet had own only 72 percent of

the original beddown plan hours, with wide discrepancy

between variants. The F-35A had own 82 percent of its

original beddown plan hours, while the F-35C had own

only 49 percent, for example.

- The following table shows the planned versus achieved

ight hours by variant for both the original plans and the

modeled-achievable projections for the elded production

aircraft through November 21, 2016.

F35 FLEET PLANNED VS. ACHIEVED FLIGHT HOURS

AS OF NOVEMBER 21, 2016

Variant

Original Beddown Plan

Cumulative Flight Hours

“Modeled Achievable”

Cumulative Flight Hours

Est.

Planned

Achieved

Percent

Planned

Est.

Modeled

Achieved

Percent

Planned

F-35A 41,000 33,754 82% 36,788 33,754 92%

F-35B 29,000 19,644 68% 21,935 19,644 90%

F-35C 12,500 6,070 49% 6,348 6,070 96%

Total 82,500 59,469 72% 65,071 59,469 91%

F-35 Fleet Reliability

• Aircraft reliability assessments include a variety of metrics,

each characterizing a unique aspect of overall weapon

system reliability.

- Mean Flight Hours Between Critical Failures (MFHBCF)

includes all failures that render the aircraft not safe to

y, and any equipment failures that would prevent the

completion of a dened F-35 mission. It includes failures

discovered in the air and on the ground.

- Mean Flight Hours Between Removal (MFHBR) gives

an indication of the degree of necessary logistical support

and is frequently used in determining associated costs.

It includes any removal of an item from the aircraft for

replacement. Not all removals are failures, and some

failures can be xed on the aircraft without a removal. For

example, some removed items are later determined to have

not failed when tested at the repair site. Other components

can be removed due to excessive signs of wear before a

failure, such as worn tires.

- Mean Flight Hours Between Maintenance Event

Unscheduled (MFHBME_Unsch) is a useful reliability

metric for evaluating maintenance workload due to

unplanned maintenance. Maintenance events are either

scheduled (e.g., inspections, planned removals for part

life) or unscheduled (e.g., maintenance to remedy failures,

troubleshooting false alarms from fault reporting or defects

reported but within limits, unplanned servicing, removals

for worn parts— such as tires). One can also calculate the

mean ight hours between scheduled maintenance events,

or total events including both scheduled and unscheduled.

However, for this report, all MFHBME_Unsch metrics

refer to the mean ight hours between unscheduled

maintenance events only, as it is an indicator of aircraft

reliability and the only metric with an ORD requirement

for mean ight hours between maintenance event.

- Mean Flight Hours Between Failures, Design Controllable

(MFHBF_DC) includes failures of components due to

design aws under the purview of the contractor, such

as the inability to withstand loads encountered in normal

operation. Failures induced by improper maintenance

practices are not included.

• The F-35 program developed reliability growth projection

curves for each variant throughout the development

period as a function of accumulated ight hours. These

projections were established to compare observed reliability

with target numbers to meet the threshold requirement at

maturity, dened by 75,000 ight hours for the F-35A and

F-35B, and by 50,000 ight hours for the F-35C, for a total

200,000 cumulative eet ight hours. In November 2013,

the program discontinued reporting against these curves for

all ORD reliability metrics, and retained only the curve for

MFHBF_DC, which is the only reliability metric included in

the JSF Contract Specication (JCS). DOT&E reconstructed

the growth curves for the other metrics analytically for this

report. The following discussion and tables compare the

F35 DEPOT, NMCM, AND NMCS RATES BY VARIANT FOR 12MONTH

PERIOD ENDING OCTOBER 2016

Variant

Depot

(Goal of 15% or less)

NMC-M

(Goal of 15% or less)

NMC-S

(Goal of 10% or less)

Avg. Max Min Avg. Max Min Avg. Max Min

Fleet 15% 22% 11% 16% 20% 14% 17% 20% 14%

F-35A 14% 27% 8% 17% 24% 12% 17% 21% 12%

F-35B 20% 25% 14% 17% 25% 11% 16% 20% 13%

F-35C 6% 15% 2% 14% 20% 9% 20% 27% 13%

FY16 DOD PROGRAMS

F-35 JSF 89

3-month reliability metrics to the growth goals required to be

on track to meet threshold requirements at maturity.

- As of the end of July 2016, the F-35 eet, including

operational and ight test aircraft, had accumulated

nearly 60,300 ight hours, or approximately 30 percent

of the total 200,000-hour maturity mark dened in the

ORD. Unlike the above table, which accounts only for

elded production aircraft, the ight test aircraft are

included in the eet hours which count toward reliability

growth and maturity. By variant, the F-35A had own

approximately 32,400 hours, or just over 43 percent of

its individual 75,000-hour maturity mark; the F-35B had

own approximately 20,300 hours, or 27 percent of its

maturity mark; and the F-35C had own approximately

7,600 hours, or 15 percent of its maturity mark.

• The program reports reliability and maintainability metrics

on a 3-month rolling window basis. This means, for

example, the MFHBR rate published for a month accounts

only for the removals and ight hours of that month and

the two previous months. This rolling 3-month window

provides enough time to average out variability often seen

in month-to-month reports, while providing a short enough

period to distinguish current trends.

• The rst table, below, compares the most recently reported

and projected interim goal MFHBCF values, with associated

ight hours. It shows the ORD threshold requirement at

maturity and the values for May 2015, the month used in the

FY15 DOT&E Annual Report, for reference as well.

• The three similar tables on the next page compare the most

recently reported and projected interim goals for MFHBR,

MFHBME_Unsch, and MFHBF_DC rates for all three

variants. MFHBF_DC is contract specication, and its JCS

requirement is shown in lieu of an ORD threshold.

• Note that data more current than July 2016 were not

available at the time of this report due to the backlog of

maintenance events awaiting JRMET review as a result

of the Lockheed Martin database (FRACAS) not being

compliant with all applicable DOD information assurance

policies mandated by U.S. Cyber Command.

• Reliability values decreased (worsened) for 8 of 12 metrics

between the May 2015 and the July 2016 values. All

three MFHBCF metrics decreased between May 2015

and July 2016, and usually showed the greatest degree of

reduction compared to the other reliability metrics. This

aligns with the declining FMC rates for all variants. Of the

remaining metrics, F-35A MFHBR and MFHBME_Unsch,

and F-35A and F-35B MFHBF_DC, improved slightly.

A more in-depth trend analysis over the 12-month period

showed that all three variants exhibited declining MFHBCF;

F-35B and F-35C MFHBR and MFHBME_Unsch were either

at or decreasing slowly; and MFHBF_DC for all variants

were also either at or decreasing. Only F-35A MFHBR and

MFHBME_Unsch increased over this period.

• All nine of the ORD metrics are below interim program

goals based on their planned reliability growth curves to

meet threshold values by maturity. Furthermore, none of

the ORD metrics are within 5 percent of their interim goals.

Of the ORD metrics, F-35B MFHBME, at 86 percent, was

the closest to its interim goal, while F-35C MFHBCF, at 39

percent, was the farthest. All of the JCS metrics, which are

the MFHBF_DC for each variant, are above their growth

curve interim values, ranging from 12 percent above for

the F-35A to 28 percent above for the F-35B. This pattern

indicates that the performance of the contract specication

reliability metrics exceeding their interim values is not

translating into the ORD reliability metrics showing the same

improvement, which are operational requirements that will be

evaluated during IOT&E.

• The fact that all the contract specication metrics are above

their growth curve does not necessarily imply that the F-35

will deliver desired reliability in the eld, especially in light

of the fact that all ORD requirements are below their growth

curves. The ORD requirements reect how the aircraft will

perform in combat, while the JCS metrics are limited to

failures that are denitively the fault of component design.

However, several situations can divorce improvement in the

JCS metrics to similar improvements in the ORD metrics or

availability. For example, components that are easily broken

during maintenance, such as nutplates, may not be scored

as design-controllable failures, but repairing and replacing

these fragile components will adversely affect the ORD

reliability metrics. Likewise, when old versions of redesigned

components fail in the eld, depending on circumstances,

these failures may not be reported in the reliability metrics,

but the effect on downing the aircraft will always be reected

in the availability metrics.

• The effect of lower (poorer) MFHBCF values is reduced

aircraft fully mission capable, mission capable, and

F35 RELIABILITY: MFHBCF HOURS

Variant

ORD Threshold Values as of July 31, 2016 Values as of May 2015*

Flight

Hours

MFHBCF

Cumulative Flight

Hours

Interim Goal

to Meet ORD

Threshold

MFHBCF

Observed MFHBCF

(3 Mos. Rolling

Window)

Observed Value as

Percent of Goal

Cumulative Flight

Hours

Observed MFHBCF

(3 Mos. Rolling

Window)

F-35A 75,000 20 32,358 17.8 8.0 45% 15,845 8.8

F-35B 75,000 12 20,256 10.0 4.6 46% 11,089 7.2

F-35C 50,000 14 7,648 10.9 4.2 39% 3,835 7.5

* The JPO revised past R&M metrics based on applying the current JRMET scoring rules to past data. As a result, values reported for May 2015 in this report may be

dierent than the values for the same month in the FY15 DOT&E Annual Report. See the Reliability Growth section below for more details.

FY16 DOD PROGRAMS

90 F-35 JSF

availability rates. MFHBR values lagging behind planned

growth targets drive a higher demand for spare parts from

the supply system than originally envisioned. When

MFHBME_Unsch values are below expectation, there is a

higher demand for maintenance manpower than anticipated.

Reliability Growth

• In the fall of 2016, the Program Ofce revised reliability

and maintainability (R&M) metrics that had been previously

reported by applying new or updated JRMET scoring rules

that had been created or modied at different times over the

course of system development, and agreed to by the JRMET

members, to historical maintenance event data. Scoring

rules determine such criteria as when a maintenance event is

considered relevant and should be included in R&M metrics,

when an event is not relevant and will not be included in

metrics, such as failures in test-specic instrumentation that

will not be installed in operational aircraft, and when an

event is chargeable to the design-controllable metric as being

the fault of the design as opposed to induced by improper

maintenance. There are many detailed scoring rules to

ensure similar maintenance situations are scored consistently.

As the JRMET developed new scoring rules and changed

some existing ones, the program realized that previously

reported metrics needed to be revised – scored by the new

rule set – in order to ensure current R&M metrics could be

compared more accurately with past R&M performance.

The effects on each reliability metric of this revision were

mixed, with 7 of 12 of the May 2015 metrics being revised

downward (worsening), and the remaining 5 increasing

compared to their originally reported values; however,

4 of these improved metrics decreased, or worsened, by

July 2016. Note the values in the tables above reect the

JPO revised past R&M metrics based on applying the

current JRMET scoring rules to past data. As a result, values

reported for May 2015 in this report may be different than

the values for the same month in the FY15 DOT&E Annual

Report.

• In the two prior Annual Reports, DOT&E reported the

results of reliability growth analysis based on the Duane

Postulate, using R&M data provided by the Program

Ofce, to determine the rate of growth for MFHBR and

MFHBME_Unsch. In 2016, DOT&E conducted an updated

analysis of reliability growth using the more rened U.S.

Army Materiel Systems Analysis Activity (AMSAA)-Crow

model, examining data from the start of the program to

July 2016. The AMSAA-Crow model characterizes growth

by a single growth parameter, using a method that is similar

to the Duane Postulate. A growth rate between zero and

one implies improvement in reliability, a growth rate of zero

F35 RELIABILITY: MFHBR HOURS

Variant

ORD Threshold Values as of July 31, 2016 Values as of May 2015

Flight

Hours

MFHBR

Cumulative Flight

Hours

Interim Goal

to Meet ORD

Threshold MFHBR

Observed MFHBR

(3 Mos. Rolling

Window)

Observed Value as

Percent of Goal

Cumulative Flight

Hours

Observed MFHBR

(3 Mos. Rolling

Window)

F-35A 75,000 6.5 32,358 5.8 4.7 81% 15,845 4.4

F-35B 75,000 6.0 20,256 5.0 2.8 56% 11,089 4.0

F-35C 50,000 6.0 7,648 4.7 2.3 49% 3,835 3.9

F35 RELIABILITY: MFHBME_Unsch HOURS

Variant

ORD Threshold Values as of July 31, 2016 Values as of May 2015

Flight

Hours

MFHBME_

Unsch

Cumulative Flight

Hours

Interim Goal

to Meet ORD

Threshold

MFHBME_Unsch

Observed

MFHBME_Unsch

(3 Mos. Rolling

Window)

Observed Value as

Percent of Goal

Cumulative Flight

Hours

Observed

MFHBME_Unsch

(3 Mos. Rolling

Window)

F-35A 75,000 2.0 32,358 1.77 1.36 77% 15,845 1.13

F-35B 75,000 1.5 20,256 1.25 1.08 86% 11,089 1.10

F-35C 50,000 1.5 7,648 1.13 0.74 65% 3,835 0.98

F35 RELIABILITY: MFHBF_DC HOURS

Variant

JCS Requirement Values as of July 31, 2016 Values as of May 2015

Flight

Hours

MFHBF_

Cumulative Flight

Hours

Interim Goal

to Meet JCS

Requirement

MFHBF_DC

Observed

MFHBF_DC

(3 Mos. Rolling

Window)

Observed Value as

Percent of Goal

Cumulative Flight

Hours

Observed

MFHBF_DC

(3 Mos. Rolling

Window)

F-35A 75,000 6.0 32,358 5.2 5.8 112% 15,845 5.4

F-35B 75,000 4.0 20,256 3.2 4.1 128% 11,089 3.6

F-35C 50,000 4.0 7,648 2.9 3.3 114% 3,835 4.2

FY16 DOD PROGRAMS

F-35 JSF 91

implies no growth, and a growth rate less than zero implies

reliability decay. Since it is logarithimic, a growth rate

of 0.40 represents much faster than twice the growth of a

rate of 0.20.

• Unlike the Duane Postulate, the AMSAA-Crow model

enables the determination of statistical condence intervals

on its estimated growth rate based on the underlying

mathematics in the model. Further, the expected growth rate

is determined by Maximum Likelihood Estimator (MLE)

methods, rather than linear regression as in the Duane

Postulate, allowing for the quantity of data to have an effect

on the growth parameter estimate.

- Previous DOT&E Annual Report reliability growth

analyses included only the F-35A and F-35B variants,

and only for the MFHBR and MFHBME metrics, due to

a small amount of hours on the F-35C, and fewer critical

failures than removals and unscheduled maintenance

events. For this year’s updated analysis, sufcient data for

the MFHBCF metric and the F-35C variant were available

for these metrics and estimates to be included.

- The rst table below shows the most likely growth rate

and 95 percent upper and lower condence bound growth

rates, providing a range of likely values for the actual

growth rate, for all three variants and all three ORD

reliability metrics. It also includes the projected values

of these three metrics for each variant based on the most

likely, upper, and lower bound growth rates at maturity;

i.e., 75,000 ight hours for the F-35A and F-35B and

50,000 ight hours for the F-35C.

Metric

Variant

July 2016

Growth Rates

Projections at Maturity

ORD

Threshold

Most

Likely

Lower

Bound

Upper

Bound

Most

Likely

Lower

Bound

Upper

Bound

MFHBCF

F-35A 0.137 0.109 0.164 9.6 9.0 10.2 20.0

F-35B -0.051 -0.089 -0.014 N/A * 12.0

F-35C -0.107 -0.180 -0.039 N/A * 14.0

MFHBR

F-35A 0.192 0.173 0.211 6.1 5.8 6.4 6.5

F-35B 0.126 0.103 0.148 4.1 3.9 4.4 6.0

F-35C -0.068 -0.119 -0.020 N/A * 6.0

MFHBME

_Unsch

F-35A 0.170 0.161 0.179 1.38 1.35 1.41 2.0

F-35B 0.359 0.351 0.367 2.01 1.96 2.08 1.5

F-35C 0.189 0.174 0.205 1.26 1.20 1.33 1.5

* No estimates for projections at maturity were made for metrics with negative growth rates.

Aircraft MFHBME_Unsch Growth Rate

F-15 0.14

F-16 0.14

F-22 (at 35,000 ight hours) 0.22

B-1 0.13

“Early” B-2 (at 5,000 ight hours) 0.24

“Late” B-2 0.13

C-17 (at 15,000 ight hours) 0.35

- The growth rates listed in the rst table were calculated

with approximately 32,400 hours for the F-35A,

20,300 hours for the F-35B, and 7,600 hours for

the F-35C. For comparison, historically observed

MFHBME_Unsch growth rates for several currently

elded aircraft are shown in the second table. Analogous

rates for MFHBR and MFHBCF are not available.

• The updated reliability growth analysis through July 2016,

using the AMSAA-Crow model, accounts for the recent

tapering off of reliability growth better than the Duane

Postulate. As a result, most of the growth rates in the table

above are lower than those reported in prior DOT&E Annual

Reports. For the nine ORD metrics, the current growth

analysis predicts that only one will meet or surpass the

ORD threshold value at maturity, F-35B MFHBME_Unsch.

As the analysis showed no growth for F-35B and F-35C

MFHBCF, and F-35C MFHBR, no projections out to

maturity were made for those metrics and current estimates

do not meet threshold requirements.

- Comparing the currently exhibited MFHBME_Unsch

growth rates to historical aircraft shows that from program

initiation to July 2016, F-35 reliability has improved faster

than average for all variants. However, F-35 reliability

remains below program interim goals for its current stage

of development in all cases, and is not projected to achieve

threshold values by maturity in most cases, due to very

low initial reliability at the start of the program, well

below the assumed initial reliability values that informed

program interim goals.

- Although there were approximately 7,600 hours on the

F-35C eet for this year’s analysis, usually enough time

to establish a growth trend, the lack of evidential growth

in the MFHBCF and MFHBR metrics may be explained

by the fact that the F-35C eet has only recently begun

to send aircraft to the depot for modications. Also,

the F-35C eet has the least hardware improvements

incorporated relative to the F-35A and F-35B eets. The

relatively strong growth in the MFHBME metric, by

contrast, can be partly explained for all variants by a

reduction in false alarms from the aircraft Prognostics and

Health Management (PHM) system, driving fewer overall

unscheduled maintenance actions, in addition to the natural

learning curve process.

• Based on current reliability trends, projections to maturity

may not be appropriate. Reliability growth projection

methodologies often assume that a system is in a single

phase of testing, characterized by a nearly constant operating

mode and environment, and gets reliability improvements

incorporated while the system is under test. For most of

the F-35 program, these conditions have held sufciently

true such that reliability growth displayed consistent

behavior; however, with the release of Block 2B capabilities,

including increased ight envelope, beginning in 2015,

both the operating mode and environment apparently

changed enough to constitute a new phase for the purpose of

analyzing reliability growth. Programs with multiple phases

FY16 DOD PROGRAMS

92 F-35 JSF

of development, where each phase is dened by different

environments or operational usage, normally generate

separate reliability planning curves (used to determine

interim goals during that phase) and separate reliability

growth tracking curves for each phase, as a single curve is

not sufcient to mathematically represent reliability growth

behavior across multiple phases. Because the reliability

projections are based on data that span the periods of time,

both before and after the Block 2B eet release, they may not

best capture reliability trends.

- For programs with multiple phases, it is common for

reliability to decrease or level off at the start of a new phase

when the system is subjected to a more stressing operating

mode or environment that exposes new failure modes. As a

result, reliability growth can come to a halt or even decline;

however, after a while, growth may resume as the program

starts to implement reliability improvements for these new

failure modes.

- Reliability growth may resume as a result of ongoing

program reliability improvement initiatives, continuing

to send aircraft through the depot modications program,

replacing lower reliability components with higher

reliability versions via TCTDs, and other reliability

initiatives. However, DOT&E also expects that the

Block 3F envelope and capabilities release, incrementally

released between CY17 and CY18, will reveal new failure

modes (e.g., new weapons, higher airspeeds and g with

Block 3F envelope) that will limit the overall effect of these

reliability improvement initiatives.

- Despite the difculty projecting accurate reliability values

at maturity, given the phased introduction of F-35 block

capabilities, DOT&E does not expect any variant to achieve

interim threshold goals for MFHBCF by the start of

IOT&E, considering the recent decline in this metric over

the past year. In fact, indications are that for each variant,

this metric is the furthest from its current interim goal.

• Failing to grow reliability sufciently by the start of IOT&E

will make achieving the necessary 80 percent availability

to accomplish all mission trials within the planned time

span very difcult. Further, a failure to achieve adequate

MFHBCF reliability in particular will impede the ability of

the Operational Test Squadrons (OTS) to generate multiple

four-ship formations with all required mission systems

functional, a necessary condition for a set of the planned

mission trials.

• A number of components have demonstrated reliability much

lower than predicted by engineering analysis. This drives

down the overall system reliability and can lead to long wait

times for resupply as the eld demands more spare parts

than the program planned to provide. Aircraft availability is

also negatively affected by longer-than-predicted component

repair times. The table at top right shows some of the

high-driver components affecting low availability and

reliability, grouped by components common to all variants,

followed by components failing more frequently on a

particular variant or which are completely unique to it.

HIGHDRIVER COMPONENTS AFFECTING LOW AVAILABILITY

ANDRELIABILITY

Variant

Common to All Variants Additional High Drivers by Variant

F-35A

• Avionics Processors

• Low Observable

Maintenance

• Shock Struts

• Cold Air Duct

• IPP Vent Fan Controller

• Main Landing Gear Tires

• Nutplates

• On-Board Oxygen

Generating System

• Horizontal Tail Actuation

• Vertical Tail Bulb Seal

• Electronic Warfare Receiver

F-35B

• Fuel System Components and

Mods

• Flexible Linear Shaped Charge

F-35C

• Main Landing Gear Retract

Actuator *

• Nose Landing Gear Steering

Motor *

* Unique to the F-35C

IPP – Integrated Power Package

- The composition of the list of some of the high-driver

components has changed as the program has progressed

and either elded more reliable components, or new

failures have occurred to displace previous high drivers.

For example, compared to the list reported in previous

DOT&E Annual Reports, the 270V DC battery and

associated components, the F-35B Upper Lift Fan Door

Actuator, and the exhaust nozzle assembly components

used on the F-35A and F-35C, are no longer high drivers.

Improving aircraft availability can be realized by more

than just improving the reliability of components and

restocking supply with improved, redesigned parts;

updating JTD and improving repair procedures can

contribute to increased aircraft availability as well.

However, in the current reporting period, overall reliability

has not increased and new components have become

high drivers, such as the Electronic Warfare Receiver and

the Vertical Tail Bulb Seal. Note also that the program

released Block 2B capabilities and ight envelope to the

eet in the period of this report. As the ight envelope

is expanding and the eet uses more mission system

capabilities, new failure modes will likely emerge

to dampen the overall effect of individual reliability

improvements, consistent with recent trends observed in

reliability growth analysis.

Maintainability

• The amount of time needed to repair aircraft and return

them to ying status remains higher than the requirement

for the system when mature, but has improved over the past

year. The program assesses this time with several measures,

including Mean Corrective Maintenance Time for Critical

Failures (MCMTCF) and Mean Time To Repair (MTTR)

for all unscheduled maintenance. MCMTCF measures

active maintenance time to correct only the subset of

failures that prevent the F-35 from being able to perform a

specic mission; it indicates how long it takes, on average,

for maintainers to return an aircraft from NMC to Mission

Capable (MC) status. MTTR measures the average active

maintenance time for all unscheduled maintenance actions;

it is a general indicator of the ease and timeliness of repair.

FY16 DOD PROGRAMS

F-35 JSF 93

Both measures include active touch labor time and cure times

for coatings, sealants, paints, etc., but do not include logistics

delay times, such as how long it takes to receive shipment of

a replacement part.

• The tables below compare measured MCMTCF and MTTR

values for the 3-month period ending in July 2016 to the

ORD threshold and the percentage of the value to the

threshold for all three variants. The tables also show the

value from May 2015, the month reported in the FY15

DOT&E Annual Report, for reference. [Note that the

May 2015 values may be different than those in the FY15

DOT&E Annual Report due to the revision of the scoring

rules described at the beginning of the Reliability Growth

section above.] For maintainability, lower repair times are

better. Three of six metrics improved marginally, while three

metrics, F-35B and F-35C MCMTCF, and F-35A MTTR,

increased or worsened. Currently, all mean repair times are

at least or nearly twice as long as their ORD threshold values

for maturity, reecting a heavy maintenance burden currently

being carried by elded units.

F35 MAINTAINABILITY: MCMTCF HOURS

Variant

ORD

Threshold

Values as of

July 31, 2016

(3 Mos. Rolling

Window)

Observed

Value as

Percent of

Threshold

Values as of

May 2015

(3 Mos. Rolling

Window)

F-35A 4.0 10.6 265% 11.4

F-35B 4.5 13.2 293% 12.7

F-35C 4.0 10.1 253% 8.4

F35 MAINTAINABILITY: MTTR HOURS

Variant

ORD

Threshold

Values as of

July 31, 2016

(3 Mos. Rolling

Window)

Observed

Value as

Percent of

Threshold

Values as of

May 2015

(3 Mos. Rolling

Window)

F-35A 2.5 6.3 252% 4.7

F-35B 3.0 7.3 243% 7.7

F-35C 2.5 4.9 196% 5.3

- A more in-depth analysis of data from between

August 2015 and July 2016, in order to capture

longer-term 1-year trends, shows that for the MCMTCF

metric, the F-35A and F-35B repair times are decreasing,

while for the F-35C it is relatively at. For overall

mean repair times, however, the F-35A exhibited a slight

increasing, or worsening trend; the F-35B showed a

slight decreasing, or improving, trend; and the F-35C was

relatively stable. Prior to May 2015, all six metrics were

improving. In contrast, the more recent trend from this

period generally indicates a slowing of improvement in the

maintainability metrics.

- All six maintainability metrics exhibit high month-to-

month variability. Due to this variability, it is difcult

to make projections in trends for maintenance metrics;

however, it will be challenging for the program to meet the

threshold values by maturity with the rate of improvement

slowing and when current values for repair times are at

least twice as high as requirements.

- Several factors negatively inuenced the ability to conduct

quick and efcient maintenance. Extensive adhesive cure

times for structural repairs, such as attaching hardware

(e.g., nutplates and installing heat blankets around the

engine bay), as well as long material cure times for low

observable (LO) repairs, remain drivers. The cure time

for some LO materials can be as high as 168 hours, for

example, although units can accelerate this if they have

appropriate tools.

- Other factors that indirectly affect maintainability

metrics have also been raised as concerns by maintainers.

Maintainers must physically connect Portable Maintenance

Aid (PMA) laptops to the aircraft in order to conduct most

maintenance activities. The PMAs enable the maintainers

to get status and conguration information from the

aircraft, as well as control aircraft functions to enable other

maintenance, such as opening the bomb bay doors where

the cooling-air receptacle is located in order to apply

air conditioning while running avionics on the ground.

Maintainers also access the Anomaly Fault Resolution

System (AFRS), which automatically troubleshoots Health

Reporting Codes (HRCs) generated by the on-aircraft

PHM system, and access JTD, which tells maintainers

how to effect repairs identied by AFRS, via the PMA.

Finally, maintainers record their work with the PMAs

as well. However, synching the PMAs to the aircraft to

conduct maintenance has been difcult, time-consuming

and, in many instances, maintainers must attempt to

synch several PMAs with an aircraft before nding one

that will successfully connect. These connections are

called Maintainer Vehicle Interface (MVI) sessions.

Occasionally PMAs disconnect in the middle of an MVI

session, which also hampers efcient maintenance.

Recently, the program introduced improved MVI cable

adapters to prevent accidental physical disconnection,

which has helped. Software-related problems persist as

well, such as PMAs taking anywhere from seconds to

minutes to connect. This occasionally leads maintainers

to disconnect a PMA they incorrectly believe is failing to

connect, which prevents that PMA from connecting to an

aircraft until an Automatic Logistics Information System

(ALIS) administrator resets it, which can be a lengthy

process.

• Maintainers have reported several difculties with

troubleshooting the aircraft, which is the rst step in many

maintenance actions. Normally, the aircraft PHM system

produces HRCs and then maintainers use AFRS to identify

possible root causes for those HRCs as well as determine

the appropriate repair action. Often, AFRS will provide a

“solution set,” which lists several possible root causes for

an HRC, rank ordered by probability of occurrence. While

AFRS coverage is improving, it currently provides effective

solution sets only approximately 70 percent of the time.

FY16 DOD PROGRAMS

94 F-35 JSF

Particularly, when an aircraft fails a Vehicle Systems (VS)

Built-In Test (BIT), an aircraft self-check conducted pre- and

post-ight, there is no specic HRC produced, making these

relatively frequent occurrences difcult to troubleshoot.

When there is no HRC, such as in a VS BIT failure or

manually reported fault, or AFRS does not produce a solution

set for an HRC, or all the solutions offered by AFRS fail to

resolve a fault, units must use other resources to troubleshoot

the discrepancy. The primary method is to submit Action

Requests (AR) to the joint JPO-Lockheed Martin Lightning

Support Team (LST), whose engineers will further

troubleshoot the aircraft remotely. The AR response times

vary signicantly, depending on category and urgency, but

average several days to get a nal response. Alternatively,

or in conjunction, maintainers can use experience to

troubleshoot on their own; however, in most cases they lack

any system theory-of-operation or troubleshooting manuals

that tell them how aircraft systems work. The current JTD

are primarily dedicated to instructions only for repair actions

for which AFRS has already identied a solution, and not

for teaching maintainers the details of systems operations.

Recently, the program and Lockheed Martin have started to

provide some troubleshooting manuals to eld maintainers

for select mission systems to try to improve the poor eet

FMC performance. The extent to which these manuals will

help troubleshooting and result in higher FMC rates remains

to be determined.

• F-35 ying squadrons also have a heavy burden of scheduled

maintenance. In particular, maintenance units have reported

that daily servicing and inspection tasks, known as the

Before-Operations Servicing (BOS), Inter-Operations

Servicing (IOS), and Post-Operations Servicing (POS),

are very time-consuming compared to similar inspections

on legacy aircraft. Some of these daily inspections also

require power and cooling air application on the aircraft, so a

unit’s ability to perform them is a function of the amount of

Support Equipment (SE) assigned or available when needed.

As the eet matures and more data become available, the

Services may be able to increase intervals between certain

scheduled inspection tasks to reduce the man-hours that units

must dedicate to this type of maintenance, if eld experience

warrants this. However, it is not clear the scheduled

maintenance burden will reduce in the near future.

Autonomic Logistics Information System (ALIS)

• The program continues to fall behind in ALIS development

and elding. Although the program planned to test and eld

the next iteration of capability, designated ALIS 2.0.2, in

2016 to support the Air Force’s decision to declare Initial

Operational Capability (IOC) in August, the program failed

to do so. Additionally, the program continued to defer

planned content from ALIS 3.0 to post-SDD development.

• ALIS includes hardware and software that connects with

all aspects of F-35 operations, including maintenance

management, aircraft health, supply chain management,

Offboard Mission Support (OMS) mission planning, along

with tracking and management of pilot and maintainer

training. Units rely on ALIS for planning and executing

deployments by managing the data required to transfer

aircraft, materiel, and personnel from home station to a

deployed or expeditionary environment. Similar to the

manner in which the program develops and elds mission

systems capability in the air vehicle, it elds ALIS in

increments.

- The program elded ALIS software version 2.0.1.1 in

late 2015. Since that time, the program has released

two updates, 2.0.1.2 and 2.0.1.3, to address previously

identied, usability-related deciencies. These software

updates include xes to existing deciencies and usability

problems, but do not add new capabilities to ALIS. Prior

to the release of the rst update with ALIS 2.0.1.2, the

program attempted to eld ALIS software versions with

both new capabilities and deciency corrections, a process

which tended to add new problems while xing some

existing problems. Instead, the program now plans to

continue elding updates dedicated only to correcting

deciencies every three months until the release of ALIS

3.0, the nal release scheduled for SDD.

- Although the program had planned to eld a new

version of ALIS software, version 2.0.2, in the second

half of 2016, in time to support the U.S. Air Force IOC

declaration, it was unable to do so. ALIS 2.0.2 includes

propulsion integration, a key capability the Air Force

had planned to have for IOC; however, the Air Force

declared IOC with ALIS 2.0.1 in August, forgoing those

capabilities. Because the program continued to experience

technical difculties integrating propulsion functionality

into ALIS, elding of 2.0.2 slipped into CY17. As a result,

operational units began 2016 with ALIS 2.0.1.1 and will

nish the year with ALIS 2.0.1.3; receiving only updates to

address deciencies and without any additional capability

elded in ALIS.

- Delays in ALIS 2.0.2 have affected the development of the

next, and last, major release of ALIS software within SDD,

ALIS 3.0, because Lockheed Martin shifted personnel

from ALIS 3.0 development to support completing

ALIS 2.0.2 development. Because the program can

no longer complete ALIS 3.0 with all of the additional

capability development planned by the end of SDD, it

has restructured the planned ALIS increments for the

remainder of SDD and for Follow-on Modernization

(FoM). This restructuring reduces the content of ALIS 3.0

from earlier plans, defers content from ALIS 3.0 that the

program has now determined is not required for IOT&E

to post-SDD development, and also adds Service and

partner priorities and emerging requirements for security

updates. The resulting plan from the restructuring was

to eld four increments of software at 6-month intervals;

the rst, ALIS 3.0, scheduled to eld in mid-to-late 2018,

which is required for IOT&E, followed by the remaining

three after SDD. These incremental software releases are

also intended to resolve ALIS deciencies and usability

FY16 DOD PROGRAMS

F-35 JSF 95

problems. At the mid-point between each of these major

releases, the program plans to deliver software updates to

continue addressing usability problems and deciencies.

Because no elding or Logistics Test and Evaluation

(LT&E) events of additional ALIS capability have occurred

for over a year, the program’s plan to develop, test, and

eld these ALIS 3.0 and later versions appears overly

ambitious with a low likelihood of actually being realized.

Regardless of whether ALIS 3.0 or a later version has

been elded, or which capabilities are included, IOT&E

will evaluate the suitability of the F-35 and ALIS in

operationally realistic conditions.

• Until 2016, formal testing of ALIS software only took

place at the Edwards AFB, California, ight test center on

non-operationally representative ALIS hardware, which

relied on reach-back capability to the Lockheed Martin

facilities at Fort Worth, Texas. Although some formal testing

will continue to occur in this manner, the program developed

and elded a dedicated end-to-end developmental testing

venue for ALIS located in part at Edwards AFB and in part at

Lockheed Martin in Fort Worth in 2016. This venue, referred

to as the Operationally Representative Environment (ORE),

reects the end-to-end Autonomic Logistics infrastructure

used to support elded operations, including one Autonomic

Logistics Operating Unit (ALOU), which represents the

main hub at Lockheed Martin Fort Worth, two Central

Points of Entry (CPEs), representing the country-unique

portal from the main hub, and two Standard Operating Units

(SOUs), representing squadron-level ALIS components, all

networked together in a closed environment. Although the

ORE provides for more realistic developmental testing of

ALIS hardware and software for early problem discovery

and xing deciencies, the current closed environment does

not adequately represent the variety of ways in which the

Services operate ALIS in different environments. ALIS

testing at the ight test center is limited in several ways.

First, the inability of ALIS to support their engines and lift

fans, which differ from production models, so LT&E of

propulsion functionality in ALIS cannot take place there.

Also, the ight test center does not use ALIS capabilities

routinely, such as Squadron Health Management (SHM),

AFRS, or the Computerized Maintenance Management

System (CMMS), as operational units do. Finally, the ight

test center does not use PHM capabilities, as they are used by

operational units, since the ight test aircraft have additional

sensors and onboard instrumentation that provide the ight

test center with more information than is available through

PHM.

ALIS Software Testing and Fielding in 2016

• Although the program planned to test and eld new capability

with ALIS 2.0.2 software release in 2016, it failed to do so.

The plans for added capability in ALIS 2.0.2 include:

- Life Limited Parts Management (LLPM), which includes:

▪ Propulsion integration. Currently propulsion data are

downloaded from aircraft portable memory devices and

provided to Pratt & Whitney Field Service Engineers for

processing and generation of maintenance work orders.

Propulsion integration will allow ALIS to process

propulsion data in the same manner as aircraft data.

▪ Production Aircraft Inspection Requirements (PAIRs).

ALIS 2.0.2 will include the rst phase of the PAIRs

system. The program added PAIRs as part of the

PHM after eliminating most of the originally planned

prognostic algorithms. The program plans to include

8 prognostic algorithms in ALIS 2.0.2 and 8 in ALIS 3.0

out of the originally planned 128 SDD algorithms.

- Sub-squadron reporting. This will allow the air vehicle

to report its status back to the home squadron SOU

even when it is deployed away from the majority of a

squadron’s assets.

- SOU-to-SOU communication. Currently, information on

one U.S. SOU is transferred to another by routing les

from the originating SOU through the CPE at Eglin AFB,

Florida, to the ALOU at Fort Worth, Texas, back through

the CPE and to the receiving SOU. This new capability

will permit targeted routing of les between SOUs under

specic circumstances and is geared primarily toward

making aircraft deployments more efcient.

- Deployability improvements. This includes improved

deployment planning and the bulk transfer of all deploying

assets at once. The current release of ALIS makes

deployment planning inefcient as it does not provide a

centralized location in ALIS for this function. During

deployments, squadrons currently transfer aircraft, supply,

and support equipment data les individually.

- Commercial Off-the-Shelf (COTS) hardware replacement.

This allows the program to plan for hardware obsolescence

and substitute newer hardware over time.

- ALIS Readiness Check. Improves the health monitoring

of ALIS processes.

• Testing of ALIS 2.0.2 will occur in multiple stages at

multiple venues. The program plans to conduct an LT&E on

the air vehicle portion of the ALIS 2.0.2 software package in

early 2017, including initial testing of the propulsion module

of the software in the ORE. Once those tests are complete,

the program plans to do a validation and verication of

the process to upgrade to ALIS 2.0.2, including the data

migration, at an operational unit – possibly Luke AFB,

Arizona – before elding ALIS across the rest of the F-35

operating locations.

• Releasing ALIS 2.0.2 to eld units will require signicant

manual intervention and data verication efforts to transition

each site, which will likely affect ight operations. The

data migration effort for ALIS 2.0.2 will be more complex

and will take longer than previous ALIS releases because of

propulsion integration and changes in data structures. For

example, the Program Ofce noted that one ALIS domain

alone, Customer Relationship Management, will require

40 man-hours for data migration and verication. Currently,

the program estimates that each site will require 8 days

FY16 DOD PROGRAMS

96 F-35 JSF

to complete the transition of all assets. Lockheed Martin

will conduct the migration and has plans to complete the

transition at each site by using the Friday through Monday

time period of two consecutive weeks. Whether or not the

affected squadron can continue ying operations between

the two transition periods is unknown. As of September

2016, the program must transition 56 sites—either SOUs or

CPEs—through this process. As of the time of this report,

the program had not released a comprehensive transition

plan.

Assessment of ALIS Support to Deployment Demonstrations

with Operational Units

• Because of delays in ALIS release 2.0.2, elded units have

operated with ALIS 2.0.1 since October 2015. As planned,

the Marine Corps used this release for a deployment

demonstration to the Marine Corps Air Ground Combat

Center (MCAGCC) Twentynine Palms, California, in

December 2015, which DOT&E reported on in the FY15

DOT&E Annual Report. Similarly, the Air Force conducted

a deployment demonstration to Mountain Home AFB, Idaho,

in February 2016. The operational test squadrons from

Edwards AFB participated in each of these demonstrations;

however, the ALIS hardware came from operational units

(a Marine Corps squadron from MCAS Yuma for the

MCAGCC demonstration and an Air Force squadron from

Hill AFB, Utah, for the Mountain Home demonstration).

• The Air Force completed its rst F-35A deployment away

from Edwards AFB, California, with six aircraft from the

31st Test and Evaluation Squadron (31TES) to Mountain

Home AFB, which has no organic F-35 capability, from

February 8 to March 2, 2016. All aircraft that participated

in the deployment were in Block 2B conguration

with software version 2BR5.2. This deployment was a

Service-led assessment.

- This deployment was the rst time the Air Force deployed

with a modularized, more transportable version of the

ALIS hardware, referred to as SOU v2. ALIS software

version 2.0.1 was used for this deployment, as well as

for the Marine Corps’ deployment to Twentynine Palms;

the previous “cross ramp” deployment at Edwards

AFB in May 2015 used the bulky SOU v1.

Deployed

personnel had no difculty setting up and conguring the

ALIS network at Mountain Home AFB; however, they

had a great deal of difculty using ALIS on the local

base network. After several days of troubleshooting,

Information Technology (IT) personnel and ALIS

administrators determined that they had to change several

settings on the base network at Mountain Home and in

the web interface application (i.e., Internet Explorer) to

permit users to log on to ALIS. One of these changes

The 31st TES previously conducted a “cross ramp deployment” at its home

base, Edwards AFB, from April 27 to May 8, 2015, to support deployment

concept of operations development. DOT&E reported on this activity in the

FY15 Annual Report.

involved lowering the security setting on the base network,

an action that may not be compatible with required

cybersecurity and network protection standards in place.

- Data le transfers took place more quickly than in the

previous F-35 deployment demonstrations, (i.e., the

F-35A cross ramp deployment and the Marine Corps’

deployment demonstration to MCAGCC Twentynine

Palms). However, Lockheed Martin provided the ve

ALIS administrators normally assigned to the 31TES and

three additional, highly experienced ALIS administrators

from other locations to provide deployment support, more

than for any previous deployment. Whether the Service’s

concept of operations for deploying ALIS will call for this

level of ALIS administrative support, to ensure timely

and accurate transfer of aircraft data at the deployed

location, is still not known. Although the process was

time-consuming and labor-intensive, they completed

the transfer of all data to the deployed SOU v2 before

deployed ight operations were scheduled to begin. To

account for the expected extended time for data transfers,

the 31TES allocated the ferry date and two additional

days to complete the transfers; ight operations began

on the third day of the deployment, as planned. Service

deployment concepts of operations may need to account

for time to transfer aircraft data les and ensure accuracy

before beginning – or at least sustaining – operations at

deployed locations.

- Because of ambiguity in the ordnance loading technical

data, one aircraft experienced major damage to a weapons

bay door and horizontal tail early in the deployment when

a bomb, which was incorrectly loaded, struck the aircraft

following release. Aircraft repairs were extensive enough

to require most of the remainder of the deployment to

complete. The Marine Corps had previously discovered

this ambiguity in the technical data, but the program did

not disseminate this information across the F-35 enterprise.

- Preparations to redeploy back to Edwards AFB began on

March 1, 2016, with aircraft departing on March 2 and

aircraft data le transfer from the deployed SOU beginning

as soon as the aircraft took off from Mountain Home AFB.

Though ALIS administrators transferred all data off the

deployed SOU at Mountain Home AFB, administrators

at Edwards AFB did not nish inducting aircraft les

back onto the Edwards AFB SOU until March 4. The

redeployed aircraft were ready for ight at Edwards on

March 5, a 4-day transition period.

• Since the Services have not yet completed ALIS Concept of

Operations (CONOPs) development, they will likely need

to take into account the results of these deployments when

determining the procedures and timing of F-35 deployments.

Although the aircraft may be own for short periods of

time without ALIS, operational planners may need to allow

for additional time between aircraft deployment and the

beginning of deployed ight operations, compared to legacy

FY16 DOD PROGRAMS

F-35 JSF 97

platforms. Deployed operations, including the set-up and

support from ALIS, will be evaluated during IOT&E.

• The challenges facing the Services and program in making

ALIS deployable now involves software. Previously, the

program identied the need to move from the bulkier, heavy

SOU version 1 (v1) racks, which weighed approximately

1,600 pounds each, to the more customizable, modularized,

two-man portable components in the SOU v2, so that

the ALIS “footprint” could meet F-35 deployability

requirements. Although the SOU v2 has improved

the deployability of the ALIS hardware, these recent

deployments show that lack of exibility exhibited in

integrating ALIS into new or existing networks, along

with deciencies in ALIS functionality and usability,

contribute more to deployability problems than just the

previously-identied hardware limitations.

ALIS Software and Hardware Development Planning from

2016 through the End of SDD

• In CY16, the program continued to struggle with providing

the planned increments of capability to support the scheduled

releases of ALIS software 2.0.2 to such an extent that

the program now cannot accomplish the original plan for

ALIS 3.0 development. As the objective date for Air Force

IOC neared, the program considered releasing ALIS 2.0.2

in two increments: the rst with all capabilities aside from

propulsion integration in time to support an August 2016

Air Force IOC declaration; the second with propulsion

integration, when the program overcame technical problems

and completed formal testing. When the Air Force declared

IOC without ALIS 2.0.2, using the already-elded version

of ALIS 2.0.1.3 instead, the need for a two-phase release

no longer existed. As a result, the program now plans to

conduct the LT&E of ALIS 2.0.2 in two parts in early 2017;

the rst with all functionality except propulsion integration at

the ight test center, then propulsion integration in the ORE.

ALIS 2.0.2 has been delayed for over a year from the release

schedule approved in CY15.

• The Program Ofce planned for the release of ALIS 3.0

in June 2017, in time to support its planned start date for

IOT&E, but now plans to release it in mid-to-late 2018.

However, the ongoing delays with ALIS 2.0.2 and the

resulting restructuring of ALIS 3.0 and beyond, have caused

the program to defer capability that had been planned to be

delivered with ALIS 3.0. The following list includes major

capabilities the program planned for ALIS 3.0 inclusion, and

identies which ones are now being deferred – in full or in

part – out of SDD:

- Decentralized maintenance. This will enable execution

of the sortie generation cycle with a deployable PMA for

independent maintenance workow while maintainers

work in the shadow of the aircraft. Decentralized

maintenance is now divided into two parts, both deferred

to post-SDD software versions.

- Resource sharing. This capability will allow the sharing

of tools, support equipment, pilots, and training records

across squadrons without requiring the transfer of data

between SOUs. Deferred to post-SDD software release.

- Security enhancements. This includes additional ALIS

readiness checks to validate and monitor user accounts and

additional penetration testing.

- Offboard Prognostic Health Management (PHM).

Additional algorithms to assess materiel condition

independently of ALIS releases and to implement a

correlation function between the Integrated Caution,

Advisory and Warning (ICAW) system and HRCs.

Partially deferred to post-SDD software release; only 16

of 128 planned prognostic algorithms are now included

within SDD.

- Life Limited Parts Management (phase 2). Adds an

Identify Locate (IDLO) viewer for product life-cycle

management, support for lightning protection and

On-Board Inert Gas Generation System (OBIGGS),

Illustrated Parts Breakdown product, Complex PAIRs to

manage remaining life of aircraft components, support

for quick engine changes, the HMDS, and back-shop

visibility for supply chain management. Full Life Limited

Parts Management in ALIS was a capability the program

originally planned for ALIS 2.0.0 to support Marine

Corps IOC; however, the re-baselining of this technically

difcult-to-implement capability has resulted in it not

being elded for at least 2 years after IOC declaration.

- COTS hardware replacement.

- Corrosion Management System. Will improve the ability

of ALIS to track and report the corrosion conditions of

aircraft using two sensors located in designated positions

within the aircraft and includes corrosion HRCs in ALIS.

Deferred to post-SDD software release.

- Low Observable Health Assessment System (LOHAS)

enhancements. Partially deferred to post-SDD release.

Prognostic Health Management (PHM) within ALIS

• The PHM system is designed to collect performance data to

determine the operational status of the air vehicle and, upon

reaching maturity, will use data collected across the F-35

enterprise and stored within PHM to predict maintenance

requirements based on trends. The PHM system is

designed to provide the capability to diagnose and isolate

failures, track and trend the health and life of components,

and enable autonomic logistics using air vehicle HRCs

collected during ight and saved on aircraft PMDs. The

F-35 PHM system has three major components: fault and

failure management (diagnostic capability), life and usage

management (prognostic capability), and data management.

PHM diagnostic and data management capabilities remain

immature. The program has yet to integrate any prognostic

capabilities; the rst set of algorithms is planned for ALIS

2.0.2.

• Diagnostic capability should detect true faults within the

air vehicle and accurately isolate those faults to a line

replaceable component. However, to date, F-35 diagnostic

capabilities continue to demonstrate poor accuracy, low

FY16 DOD PROGRAMS

98 F-35 JSF

detection rates, and also have high false alarm rates.

Although coverage of the fault detection has grown with

the elding of each Block of F-35 capability, all metrics of

performance remain below threshold requirements. The

table below compares specic diagnostic measures from the

ORD with current values of performance through April 2016.

• PHM monitors nearly every on- and offboard system on the

F-35. It must be highly integrated to function as intended

and requires continuous improvements for the system to

mature.

• Poor diagnostic performance increases maintenance

downtime. Maintainers often conduct BITs to see if the

fault codes detected by the diagnostics are true faults. False

failures (diagnostics detecting a failure when one does not

exist) require Service personnel to conduct unnecessary

maintenance actions and often rely on contractor support

to diagnose system faults more accurately. These actions

increase maintenance man-hours per ight hour, which

in turn can reduce aircraft availability rates and sortie

generation rates. Poor accuracy of diagnostic tools can also

lead to desensitizing maintenance personnel to actual faults.

• The number of false alarms recorded within ALIS can be

articially lowered, as qualied maintenance supervisors can

defer or cancel an HRC without generating a work order for

maintenance actions, if they know that the HRC corresponds

to a false alarm not yet added to the nuisance lter list. The

deferred or canceled HRC will not result in the generation

of a work order, and it will not count as a false alarm in the

metrics in the table below. The program does not score an

HRC as a false alarm unless a maintainer signs off a work

order indicating that the problem described by the HRC did

not occur. Because PHM is immature and this course of

action saves time for the maintainers, it occurs regularly at

eld locations; however, this means the number of recorded

false alarms is not always an accurate reection of the HRC

false alarm rate.

• Comparing the values in the table below with those in the

FY15 DOT&E Annual Report shows improvement in Fault

Detection Coverage, Fault Detection Rate, Fault Isolation

Rate for non-electronic faults to one Line Replaceable

Component (LRC), and – most signicantly – Mean Flight

Hours Between Safety Critical False Alarms. Mean Flight

Hours Between False Alarms and Fault Isolation Rate

for non-electronic faults to three or fewer LRCs show

no signicant improvement, and Fault Isolation Rate for

electronic faults to one LRC has gotten worse since last

year’s report. At this time, Mean Flight Hours Between

Flight Safety Critical False Alarm and Fault Isolation Rate

for non-electronic faults to one LRC are the only diagnostic

metrics which appear to be improving adequately toward

meeting their threshold requirements. The program planned

for accurate diagnostics to support a planned level of

sustainment; poor diagnostics contribute to poor reliability

and maintainability metrics, reducing aircraft availability and

increasing aircraft downtime.

• Following are the systems most likely to result in missed

fault detections, incorrect fault isolations, and false alarms as

of April 2016.

- Missed detections: Integrated Core Processor (ICP),

Communications, Navigation, and Identication (CNI)

rack modules, Panoramic Cockpit Display, Power and

Thermal Management System (PTMS), and vehicle system

processing.

- Incorrect isolation: ICP, PTMS, EW, electric power, and

hydraulic power system.

- False alarms: Propulsion, CNI system, EW, ICP, and

displays and indicators in general.

• The Program Ofce initiated a PHM maturation plan in

2015 to improve the performance of each of the three major

components of PHM:

- Improving BIT functionality, PHM software handling of

BIT results, and off-aircraft lter lists and fault isolation

instructions; also focusing on identied high-fault drivers

to prioritize developing AFRS solutions with the greatest

impact on fault detection and isolation, false alarm

METRICS OF DIAGNOSTIC CAPABILITY

6-month rolling window as of April 2016. Data provided by Program Oce considered “preliminary” as they have not completed formal adjudication process by the data review board.)

Diagnostic Measure

Threshold

Requirement

Demonstrated Performance

Block 1 Block 2 Block 3

Developmental Test and Production Aircraft

Fault Detection Coverage (percent mission critical failures detectable by PHM) N/A 88 88 93

Fault Detection Rate (percent correct detections for detectable failures) 98 88 88 93

Fault Isolation Rate (percentage): Electronic Fault to One Line Replaceable Component (LRC) 90 65 64 42

Fault Isolation Rate (percentage): Non-Electronic Fault to One LRC 70 71 73 86

Fault Isolate Rate (percentage): Non-Electronic Fault to Three or Fewer LRCs 90 87 87 100

Production Aircraft Only

Mean Flight Hours Between False Alarms 50 0.09 0.41 0.50

Mean Flight Hours Between Flight Safety Critical False Alarms 450 61 537 437

Accumulated Flight Hours for Measures N/A 61 6,440 6,111

Ratio of False Alarms to Valid Maintenance Events N/A 135:1 19:1 19:1

FY16 DOD PROGRAMS

F-35 JSF 99

performance, unnecessary maintenance, high maintenance

man-hours, aircraft availability, and excess cost

- Improving the functionality of PAIRS and algorithms

which assess materiel condition based on usage and repair

feedback, potentially adding new life tracking items based

on eet experience

- Improving or adding data collection from the air vehicle,

improving data downloading and processing from the

aircraft to ALIS, and improving distribution and storage of

data to better support user needs

• Structural PHM (SPHM) is a key element of overall

airframe life-cycle management. It includes conditional

event detection and analysis, including over-g, hard landing,

overspeed, and overload conditions, and is planned to

provide a corrosion monitoring and predictive modeling

capability. The air vehicle currently includes two corrosion

sensors—one on the forward face of the radome bulkhead

and the other on the wall of the bay housing the fuel/heat

exchanger. ALIS 2.0.0 included a logging function for these

corrosion sensors. A Program Ofce study completed in

November 2015 determined that 27 percent of the corrosion

sensors in the eet had failed, so the program is in the

process of developing a new sensor manufactured with more

precise sealing applications to be used during production

instead of upon installation.

Air-Ship Integration and Ship Suitability Testing

F-35B

• The integrated test team from Patuxent River, Maryland,

conducted the third and nal planned set of F-35B ship

trials, referred to as Developmental Test III (DT-III), from

October 28 through November 17, 2016, on USS America.

The objectives for this 3-week developmental test event

included:

- Expanding the vertical landing ight envelope for both day

and night operations (higher wind-over-deck conditions

and operations at higher sea states than earlier ship trials,

as well as operating from additional landing spots farther

forward on the ight deck)

- Evaluating the Gen III HMDS for nighttime landings, with

or without landing aids on the ship

- Assessing Joint Precision Approach Landing System

(JPALS) functionality

- Conducting vertical landings and short take-offs with

symmetric and asymmetric external loads carriage

- Expanding vertical take-off capability

- Evaluating environmental effects from ight operations,

such as the thermal tolerance and response of the ight

deck to vertical landings and noise surveys from various

ship locations

- Conducting maintenance demonstrations – including

engine and lift fan removal and replacement actions, and a

power module maintenance demonstration – and loading

and unloading of external stores

- Evaluating the operational capability of the rst

deployment of an ALIS SOU v2 on the ship

• Besides the two developmental test aircraft from the Patuxent

River test force (BF-1 and BF-5), the Marine Corps also

supported the test activities by providing an additional three

instrumented operational test aircraft assigned to VMX-1,

the operational test unit at Edwards AFB, California, and two

eet aircraft from VMFA-211, one of the two operational

units at MCAS Yuma, Arizona. Although primarily a

developmental test event, the Marine Corps embarked eet

and operational test squadron personnel for training, and

to inform the JSF Ship Integration Team in preparation for

the rst operational F-35B deployment onboard USS Wasp,

planned for late 2017. From November 17 – 21, the Marine

Corps also conducted a “Lightning Carrier” proof of concept

demonstration, with an additional ve F-35B eet aircraft

plus two MV-22 and two H-1 Air Combat Element (ACE)

assets deployed to the ship to assess interoperability and

the suitability of F-35B “Heavy” ACE congurations on

LHA-class ships. Observations from this testing included:

- The specialized secure space set aside for F-35-specic

mission planning and the required Offboard Mission

Support (OMS) workstations is likely too small and

therefore unsuitable for regular ACE operations with the

standard complement of six F-35B aircraft – let alone

F-35B Heavy ACE congurations with more aircraft. Due

to the classication of certain F-35 capabilities, pilots

must conduct mission planning in a secure space. The

ALIS SOU v2, which has several classied components,

was also located in this space. However, pilots, the ALIS

administrator, and security personnel commented that the

compartment designated for the secure workspace onboard

USS America was too small to accommodate enough

OMS workstations and a sufcient brieng and debrieng

area. Marine Corps and ship personnel are investigating

using this compartment for ALIS only, and designating an

alternate compartment for mission planning.

- The power module maintenance demonstration was

intended to show that a deployed unit could conduct

modular engine maintenance at-sea. The F135 engine

is modular, with a fan and compressor section; a power

section with the combustion chamber and turbine stages;

an afterburner section, which on the F-35B consists

of a Three-Bearing Swivel Module (3BSM) that can

rotate downward to more than 90 degrees for vertical

ight; and a nozzle section. The general maintenance

concept for a failed engine is to replace only the defective

module on any given engine to return the overall engine

to service more quickly, and send the defective module

to depot-level repair. The demonstration consisted of

splitting open an F135 engine mounted on two aligned

Maintenance and Transportation Trailers (MTTs) into its

modularized sections, removing a “bad” power module,

taking a “good” spare power module out of its shipping

and storage container, placing the good module into the

FY16 DOD PROGRAMS

100 F-35 JSF

engine, and containerizing the bad module, all with the

use of an overhead bridge crane in the aft high bay of the

hangar bay. The demonstration showed that maintainers

could swap a module at sea; however, the evolution took

up a large amount of space in the hangar bay and occurred

without a full ACE onboard. The Navy and Marine

Corps should conduct some further analyses, such as an

operational logistics footprint study which simulates ight

deck and hangar bay spotting with a full ACE onboard,

using data from this evolution to determine what the

impact of this maintenance would be on integrated ship

and ACE operations with a full ACE onboard.

- The detachment planned to stage an F135 engine removal

and installation (R&I) demonstration, but early in the

deployment maintainers discovered, during a Post-

Operations Servicing, that one of the OT aircraft (BF-20)

had a thrust pin that had unseated. There are several thrust

attachments between the engine and the airframe that

transfer the propulsive forces produced by the engine to

the airframe, and this was the rst time in program history

that maintainers discovered a thrust pin had backed out

of full engagement, a serious safety of ight concern. As

a result, the unit submitted an AR to request disposition.

The AR response directed that the engine be removed from

the aircraft, and the thrust pin attachment points on both

the engine and airframe be thoroughly inspected. This

provided a natural opportunity to evaluate an actual engine

R&I as opposed to a staged demonstration. The unit

provided photos and dimensional data to the Lightning

Support Team (LST), initiating a long investigation

process to determine the root cause, but there were no

immediately obvious signs of wear or damage. The LST

eventually directed the squadron to replace the engine, as

there was a full spare engine onboard, and the lift fan drive

shaft. The squadron completed this maintenance in the

hangar bay and, on November 16, conducted a High-Speed

Low-Thrust (HSLT) engine operation on the ight deck

to conrm that the new engine was installed correctly and

fully functional. The unusual circumstances of this event

primarily drove the 2-week long R&I process, as opposed

to specic shipboard conditions and, by the time of this

report, the program had not yet determined a root cause.

However, the engine R&I was practically aided by the fact

that, for this detachment, a full spare engine was available

for immediate installation. Currently, the program’s

planned Aoat Spares Package of spare parts that will

be loaded onboard the USS Wasp for the rst F-35B

deployment in 2017 will not have a full spare engine, only

spare propulsion modules. See the F-35C ship suitability

section for further details on F135 engine R&I concerns at

sea.

- The squadron also conducted a staged lift-fan R&I

demonstration on BF-20 while it was in an NMC status in

the hangar bay for the engine R&I. Maintainers positioned

the aircraft along the ship’s centerline and directly beneath

the bridge crane in the forward of two high bays. Organic

Marine squadron personnel rst used a collapsible, portable

oor crane and an assembled support frame to cradle the

upper lift fan door and remove it from the aircraft, and then

place it on the deck. After maintainers attached another

assembled frame to the top and sides of the lift fan, ship

personnel used the overhead bridge crane to raise the lift

fan out of the aircraft cavity and, via attached tether ropes

to each of the four top corners of the frame to guide the

lift fan, lowered it to a support cradle on the deck. Service

personnel then reversed this process to reinstall the lift fan.

After the upper lift fan door was reinstalled and maintainers

were disassembling the support frame that attaches the

door to the crane, a portion of this assembly fell onto the

lift fan, damaging a stator strut at the top of the lift fan.

Repairs to this strut took another couple of days to complete.

Maintenance personnel noted several improvements that

should be incorporated into this process; most importantly,

the tether points for the lift fan support assembly need to

be moved to the bottom four corners for better control, as

the tethers provided very little control near the hook point

of the crane; also the program should provide a protective

maintenance cover for the lift fan to prevent damage during

future lift fan R&I’s or upper lift fan door maintenance.

- On November 15 and 16, a single eet aircraft from

VMFA-211 departed from USS America to drop live

ordnance on targets on an inland range, hot-pitted for fuel

from MCAS Yuma, Arizona, and returned to the ship each

day. Both sorties dropped one GBU-12 laser-guided bomb

and one GBU-32 JDAM. The Marine Corps originally

intended to y two loaded aircraft each day, but the lack of

available mission-capable aircraft drove the detachment to

launch only a single aircraft each day.

- While the set of sea trials were not focused on operational

realism, several aspects were more operationally

representative than the 2015 F-35B deployment

demonstration onboard USS Wasp. The aircraft had a full

suite of Block 2B electronic mission systems installed,

unlike onboard USS Wasp; however, like the USS Wasp

demonstration, these aircraft mission systems were not

maintained to a full combat-mission-capable state of

readiness. Unlike in 2015, the OT and eet aircraft were

cleared to carry live ordnance on the ight deck, with

some workarounds. With this clearance, the test team

intended to employ live ordnance on missions. Production-

representative support equipment (SE) was onboard ship

for the rst time as well for use on the non-DT aircraft.

Similar to the 2015 demonstration, the operational logistics

support system, known as the Autonomic Logistics Global

Sustainment system, was still not available. As a result,

spares provisioning and supply support were not necessarily

the same as would be expected on a combat deployment.

F-35C

• The third and nal phase of F-35C ship suitability testing,

designated Developmental Test III (DT-III), was conducted

by VX-23, the developmental test team from Patuxent River,

FY16 DOD PROGRAMS

F-35 JSF 101

from August 10 – 26, 2016, aboard USS George Washington.

The primary objective of DT-III was to complete

characterization of the ying qualities of the F-35C aircraft

for catapult launches and arrested recoveries, building on the

results from two previous at-sea developmental test periods.

The test team explored aircraft ight operations around the

carrier in high crosswind conditions and, for the rst time,

with external ordnance, including asymmetric load-outs.

Both day and night operations were conducted, allowing for

assessments of the Gen III HMDS for night approaches and

landings under varying light conditions. These investigations

will help develop aircraft launch and recovery bulletins to an

expanded envelope to support eet operations. Also, while

the ship was underway, VFA-101, the Navy’s F-35C training

squadron at Eglin AFB, Florida, participated in the event for

other test objectives, including a Commander of Naval Air

Forces (CNAF)-directed proof-of-concept demonstration

of an F-35C engine R&I in the ship’s hangar bay as well

as initial day carrier qualications for 12 pilots that would

assess overall suitability of catapult launches and the Delta

Flight Path capability for carrier approaches and landings.

- Initially, only developmental test aircraft CF-3 and CF-5

(transient aircraft needed for logistical support) and search

and rescue helicopters deployed to the carrier. No air wing

was present. Five VFA-101 aircraft deployed onboard

the ship from August 14 – 18. The major contractor and

test team were responsible for maintenance of CF-3 and

CF-5, although eet maintenance personnel supported

the VFA-101 carrier qualications and the engine R&I

demonstration. ALIS was not installed on the carrier; it

was accessed via satellite link to a location ashore.

- The developmental test team conducted night operations

with modications to the Helmet Display Unit for the

Gen III HMDS that permitted lower illumination settings,

intended to reduce the amount of “green glow” in the

helmet display that makes seeing the lights on the carrier

difcult during night operations. The test pilots reported

that the rened brightness control somewhat improved the

night carrier approaches; however, “green glow” was still

a signicant problem and is the subject of two Category 1

deciency reports.

- From the carrier qualications, the VFA-101 pilots found

the F-35C catapult shot not operationally suitable due

to excessive vertical (Nz) oscillations during launch.

Although numerous deciencies have been written against

the F-35C catapult shot oscillations – starting with the

initial set of F-35C ship trials (DT-I) in November 2014 –

the deciencies were considered acceptable for continued

developmental testing. The eet pilots reported that the

oscillations were so severe that they could not read ight

critical data, an unacceptable and unsafe situation during

a critical phase of ight. Most of the pilots locked their

harness during the catapult shot, which made emergency

switches hard to reach, again creating an unacceptable and

unsafe situation.

- The VFA-101 pilots reported that the Delta Flight Path

mode of operation made carrier approaches easier on

pilot workload and touchdown points more consistent.

During the qualications, pilots made 154 approaches and

landings with 100 percent boarding rate and no bolters.

- The engine R&I proof-of-concept demonstration took

55 hours to complete and used about one-third to one-half

of one of the three hangar bay partitions; this is much

more space than that needed for an F/A-18 engine change.

Because it was the rst F-35C engine R&I demo at-sea,

maintainers moved through all required steps at a slow

pace to ensure safety rst, which may have extended

the timeline relative to what an experienced crew could

achieve during routine maintenance operations. On the

other hand, the maintainers had practically free use of

most of the hangar bay space, which may have facilitated

speedier maintenance relative to conducting an engine R&I

with a full air wing onboard. As a result, actual engine

R&I’s during deployments may not differ drastically in

time from this demonstration.

- While the proof-of-concept demonstration showed that

an engine could physically be swapped at sea, it also

revealed that such a major maintenance evolution would

be very difcult, time consuming, take up a large amount

of space, and be a drastic change from the engine R&I on

legacy aircraft. The F-35C engine change is also more

labor- and space-intensive than the F-35B engine R&I,

such as conducted onboard the USS America. The F-35B

engine R&I is aided by the aircraft’s 3BSM doors, which

open during regular operation to enable the exhaust nozzle

to rotate downward to more than 90 degrees for vertical

ight. Opening these doors for engine maintenance avoids

the need to remove xed panels, such as on the F-35A

and F-35C. For the F-35C, many more skin panels and

a large piece of structure known as the tail hook trestle,

although not the tail hook itself, must be removed for an

engine R&I. Storing these items, and the associated tubes

and wire harnesses, so they will not be damaged while off

the aircraft, also takes up additional space. The fact that

the demonstration was conducted without a full air wing

on the ship additionally limited the test team’s ability to

assess the likely impact of an F-35C engine change on

integrated carrier-air wing operations. Such an assessment

will be needed for IOT&E. Because of the complexity

and time required to conduct an engine change, the Navy

and JPO should investigate alternatives for determining

the impact of an R&I while conducting carrier-air wing

operations as well as improving the maintainability of the

F-35 system at sea.

• Both the F-35B engine R&I onboard USS America and the

F-35C engine R&I onboard USS George Washington were

hampered by the lack of suitable strut locks approved for

at-sea use, considering the rolling and pitching motion that

may be experienced while underway. Since the engine is

a signicant part of the aircraft weight, without strut locks

FY16 DOD PROGRAMS

102 F-35 JSF

the airframe would raise up on the pressurized landing

gear struts as soon as the engine was detached. This could

potentially damage either the engine or airframe due to

tight tolerances, or injure maintainers with hands in the

area. In both cases, maintainers put the aircraft up on jacks

to de-service the struts before the engine change, and then

raised the aircraft back up on jacks to re-service the struts

after the change, adding signicant time to the process.

Further, ship maneuvering is restricted when raising and

lowering aircraft on jacks; engine R&I times could be

decreased if the program develops, and the Navy approves,

appropriate strut locks for at-sea use.

• Maintainers conducted a less extensive power

module maintenance demonstration onboard

USS George Washington than the one performed on

USS America, consisting of removing a power module

from its container in the hangar bay, moving it to the

engine repair shop aft of the hangar bay, and returning it

to its container. To open the container, maintainers used a

motorized, wheeled, mobile crane that is part of the ship’s

SE complement to raise the container lid, which is composed

of the roof and four side walls, over the encapsulated power

module, and set it to the side in the hangar bay. A specialized

Electric Pallet Jack (EPJ) was then used to move the power

module, still attached to the container bottom, to the engine

repair shop, where it could be transferred to an MTT via an

overhead bridge crane. Maintainers expressed dissatisfaction

with the container design, which required a large amount

of space and a large piece of SE to remove, and stated that,

while suspended on a possibly pitching and rolling ship, such

a heavy item could present a safety hazard. They stated a

preference for the type of container used for the T56 engine,

installed on the E-2 Hawkeye and C-2 Greyhound aircraft.

This type of container has a door on one side that opens

outward, with the engine mounted on rails inside. An MTT

can be wheeled up to the container and the engine slid onto

it by hand. This conguration takes up less space to remove

an engine, doesn’t require any SE, is quicker, and presents

fewer hazards. The current container is designed to a very

high standard of structural integrity in order to withstand

a fall if ever resupplied by moving it across a wire strung

between a resupply ship and a carrier, a standard form of

resupply at sea. However, only the planned heavy E-Stream

wire system was capable of moving the heavy power module

container, but this program is now canceled. The Navy

now plans to resupply un-containerized power modules

via internal carriage on a CV-22 aircraft, and containerize

any spare modules onboard ship if needed for storage. The

program and the Navy should investigate if the heavy power

module container should be redesigned for better usability at

sea.

• Current program plans do not provide a full spare engine

for the envisioned Aoat Spares Package of parts that will

go onboard Navy CVN and L-class ships to support F-35C

and F-35B squadrons, respectively. This will signicantly

increase the amount of time required to conduct an actual

engine change. The 55-hour timeline measured during the

proof-of-concept demonstration provided above assumed

a full spare engine ready for immediate install once the

down engine is removed from the aircraft. Without a spare,

the time required to troubleshoot the down engine to a bad

module, disassemble the engine to swap that module, and

then reassemble the engine to reinstall it into the aircraft

must be added to the overall process; this can easily add

several more days of downtime to the affected aircraft.

Further, the probability of Foreign Object Damage (FOD)

to engines is higher at sea than ashore, which may drive

more frequent engine R&Is at sea. This is due to the close

proximity of aircraft maintenance to the ship landing areas

allowing foreign objects to migrate, and the more stressing

arrested or vertical landings at sea, which can increase the

probability of items like fasteners falling off an aircraft into

the landing area.

• Access to ALIS offboard the ship via the ship’s satellite

communications was intermittent and troublesome, making

transmitting large le sizes difcult. For example, a 200 MB

le required 2 days to successfully transfer due to bandwidth

limitations and inconsistent connectivity. These issues

drove VFA-101 to operate in an ALIS ofine mode for the

majority of the detachment. While the root cause appeared

to be due to limitations with the shipboard communications

equipment vice ALIS directly, and deployed units will have

an SOU onboard ship, the SOU will occasionally have to

transmit large les to the CPE due to how data-intensive

ALIS is. This requirement to communicate large amounts of

information will likely be exacerbated after a ship emerges

from a restricted Emissions Control (EMCON) period where

transmissions from the ship are severely limited or cut-off

completely. The program and the Navy should investigate

potential options to improve ship-based communications

bandwidth dedicated to ALIS connectivity off-ship, such as

increasing the priority of ALIS transmissions, or reserving

low-use times of the day for transmitting large volumes of

ALIS message trafc.

• VFA-101 brought a suite of production-representative SE to

the aircraft carrier, including electrically powered hydraulic,

air conditioning, and polyalphaolen (PAO) carts for use

in the hangar bay. Personnel use the PAO cart to service

the aircraft with this special uid that cools the radar and

some other avionics. The Navy prefers that SE for use in

hangar bays be electrical vice diesel powered because of

the enclosed environment. They also brought an engine

R&I trailer and an engine maintenance trailer, needed for

the engine maintenance demo. Collectively, these items

of SE were larger than legacy items and took up a large

amount of deck space. Hangar bay personnel commented

that the size of the SE would also make them more difcult

to move around a crowded hangar bay with a full air wing

onboard. The Navy should investigate any efcient, multi-

use opportunities for F-35 SE, such as using legacy SE on

the F-35 or F-35 SE on legacy aircraft, to try to limit the

impact on the overall SE footprint for an air wing with F-35

FY16 DOD PROGRAMS

F-35 JSF 103

included. Additionally, the JOTT will evaluate SE operation

and movement around the ight deck and hangar bay during

IOT&E.

• Since the hangar-bay SE items are electrical, they rely

on 440V power from outlets in the walls of the ship.

Maintenance on a single F-35C can sometimes require

external power, provided by a small transformer power cart

that converts the 440V wall power to the 270V and 28V

DC power used by the aircraft, along with air conditioning

and hydraulic power, each requiring separate carts. Such

maintenance activities would require the use of three wall

outlets. However, most hangar bay partitions had four

outlets, which would make simultaneous maintenance on

more than one F-35C in a partition a coordination challenge.

The Navy should investigate options for increasing the

number of wall power outlets in hangar bays to help facilitate

simultaneous maintenance on multiple F-35Cs, or the ability

to interconnect multiple pieces of support equipment from a

single outlet to permit simultaneous operations.

• The Navy is working on the following air-ship integration

issues, primarily for carrier operations. Some of the

following issues also apply to F-35B operations on L-class

ships:

- Flight deck Jet Blast Deectors (JBDs) will require

additional side panel cooling in order to withstand regular,

cyclic limited afterburner use, during F-35C catapult

launches. JBDs are retractable panels that redirect hot

engine exhaust up and away from the rest of the ight

deck when an aircraft is at high thrust for take-off.

During IOT&E, an F-35C detachment will deploy to a

CVN to evaluate sortie generation rate capability within

an air wing context. The CVN used for IOT&E must

have additional side panel cooling installed in the JBDs

to enable the most operationally representative test to

evaluate this Key Performance Parameter of the F-35C.

- The Navy continues to procure a replacement mobile

Material Handling Equipment crane for several purposes

onboard carriers, including lifting the power module

container lid as described above. This crane will only be

used on CVNs, for F-35 maintenance only, as they lack

the hangar-bay overhead cranes that L-class ships come

equipped with. Since the FY15 DOT&E Annual Report,

the crane acquisition has proceeded at a pace such that

sufcient articles should be in the eet in order to support

a rst F-35C deployment in the 2020 timeframe.

- Two methods of shipboard aircraft reghting for the F-35

with ordnance in the weapons bays are being developed,

one for doors open and one for doors closed. Each method

will use an adapter that can t to the nozzle of a standard

hose. The open door adapter will also attach to a 24-foot

aircraft tow bar so reghters can slide it underneath

the aircraft and spray cooling water up into the bay.

Development of this open door adapter is proceeding well

and it was deployed to the USS America to support live

ordnance carry by the OT and eet F-35B aircraft during

DT-III. However, the closed bay adapter, which intends

to use water pressure to drive a saw to cut into the aircraft

and lock a hose in place to douse a loaded weapons bay

during a ight deck re, was not yet ready for deployment.

As a workaround, F-35B aircraft on USS America with

live ordnance taxied with their weapons bay doors open,

closing them only right before take-off, to mitigate the

risk, but this will not be a standard practice for combat

deployments.

Cybersecurity Operational Testing

• The JOTT continued to accomplish testing based on the

cybersecurity strategy approved by DOT&E in February 2015,

with some modications due to test limitations, discussed

below. In accordance with this strategy, in FY16 the JOTT

conducted adversarial assessments (AA) of the ALIS 2.0.1

Squadron Kit and Central Point of Entry (CPE), completing

testing that began in Fall 2015, and conducted cooperative

vulnerability and penetration assessments (CVPA) of the

mission systems Autonomic Logistics Operating Unit (ALOU)

used to support developmental testing (referred to as the

DT-ALOU), and the operational ALOU. The JOTT also

completed a limited cybersecurity assessment of the F-35

air vehicle. These tests were not conducted concurrently

as originally planned; therefore, end-to-end testing of

ALIS, from the ALOU to the air vehicle, has not yet been

accomplished. The JOTT initially tested the DT-ALOU in

lieu of the operational ALOU because the JPO did not approve

an Interim Authority to Test for the ALOU due to concerns

that cybersecurity testing would adversely affect the ALOU’s

operations; however, a limited test of the operational ALOU

was completed in October 2016 and an AA was scheduled for

December 5 – 9, 2016.

- The U.S. Navy’s Commander, Operational Test and

Evaluation Force (COTF) conducted a CVPA and limited

AA against the DT-ALOU, from April 1 – 15, 2016, at

Lockheed Martin’s Fort Worth facility. The COTF testing

veried that the DT-ALOU, congured with ALIS 2.0.1.3,

had mitigated several key vulnerabilities discovered on

ALIS 2.0.1.1 systems during fall 2015 testing. However,

this testing of the DT-ALOU was not operationally

representative because several key systems and external

interfaces, from which cyber-attacks might originate,

were not present. The testing was further constrained

because the Program Ofce and Lockheed Martin only

permitted testing to occur during overnight hours while the

DT-ALOU was disconnected from external networks to

minimize interference with operations. The COTF testing

still discovered several minor security problems with the

DT-ALOU. The operational ALOU is still congured with

ALIS 2.0.1.1.

- The U.S. Marine Corps Information Assurance Red Team

(MCIART) conducted an AA of the Marine Fighter Attack

Squadron 211 (VMFA-211) ALIS 2.0.1.3 Squadron Kit

at Marine Corps Air Station Yuma, Arizona, April 25

through May 6, 2016. The unit’s Squadron Kit was in

the process of being stood up, so it was not in a fully

operational conguration during the test. The operational

FY16 DOD PROGRAMS

104 F-35 JSF

VMA-121 Squadron Kit was declared off-limits by Marine

Corps personnel. MCIART veried that several key

vulnerabilities discovered during the 2015 Squadron Kit

testing had been mitigated; however, MCIART discovered

several new vulnerabilities from insider and outsider threat

postures.

- The U.S. Air Force 177th Information Aggressor Squadron

(IAS) conducted an AA against the ALIS 2.0.1.3 Central

Point of Entry (CPE) at Eglin AFB, Florida, from

June 2 – 10, 2016. The 177 IAS assessed the system

as an outsider and near-sider threat, and discovered

vulnerabilities with various components of the CPE,

despite the fact that Lockheed Martin administrators and

ALIS users had implemented new operating procedures

during the test to improve the CPE security posture.

The

CPE classied servers were not adequately assessed due

to time constraints and a lack of approval for connecting

177 IAS equipment to the classied CPE network.

- The JOTT, with support from the Air Force

Research Laboratory (AFRL), conducted a limited

CVPA of the F-35A Block 2B air vehicle, from

September 26 – 27, 2016, at Edwards AFB, California.

The CVPA tested the process by which the air vehicle

validates the digital signature of les within the operational

ight program when it is loaded onto the aircraft via the

aircraft media device. This test was one of the test cases

proposed by cybersecurity subject matter experts, and was

the rst cybersecurity assessment of an operational F-35

air vehicle. The successful accomplishment of this initial

test should encourage the Program Ofce to examine

other planned test cases in future air vehicle cybersecurity

assessments. Analyses of the test results are ongoing.

- The COTF and the JOTT conducted a CVPA of the

operational ALOU October 17 – 28, 2016, at Lockheed

Martin’s Fort Worth facility. The test team was augmented

by Lockheed Martin Red Team members so that the

ALOU could be examined for vulnerabilities from the

Lockheed Martin Intranet (LMI). COTF and the JOTT

were not permitted to conduct any test activities on the

ALOU unless it was disconnected from the LMI, limiting

the operational realism of the test and precluding certain

vulnerabilities from being assessed. Detailed analyses of

the data collected are ongoing.

• In response to DOT&E’s recommendation that active

intrusion discovery and forensics, referred to as a Blue

Hunt, be conducted on the Squadron Kit and CPE, the JOTT

has scheduled the 855th Cyber Protection Team (CPT) to

conduct two events for the end of CY16. Current plans are to

perform mostly vulnerability assessment and traditional Red

Team activities against these systems —not active intrusion

discovery and forensics—and so it is still unclear whether

these events will fulll DOT&E’s request. Additionally, the

JOTT will need to conduct a Blue Hunt on the ALOU once

ALIS 2.0.2.4 is loaded and then additional Blue Hunts on all

ALIS levels (ALOU, CPE, and Squadron Kit) each time a full

increment of ALIS software is released.

• While progress towards fullling missed test opportunities in

2015 was considerable in 2016, full end-to-end cybersecurity

testing of the ALIS architecture, from the operational ALOU

to the air vehicle, remains to be completed. The JOTT is

planning concurrent assessments of the ALIS 2.0.2 Squadron

Kit, CPE, and ALOU in 2017. The JOTT is also exploring

testing opportunities on the F-35 training systems, and has

begun exploring options for testing systems at the U.S.

Reprogramming Laboratory, which generates mission data

les for the F-35.

• The JPO continued to develop its Operationally Representative

Environment (ORE); it plans to perform verication,

validation, and accreditation (VV&A) testing in order to

conduct future operational testing on ALIS components within

the ORE. Regardless of whether the ORE completes VV&A,

the JOTT is working with the JPO and Lockheed Martin

to plan cybersecurity testing of ALIS components within

the ORE for purposes of risk reduction ahead of continued

cybersecurity testing of the operational ALIS systems.

DOT&E Response to Senator McCain’s Questions Regarding

the Completion of SDD

In a letter to the SECDEF on November 3, 2016, Senator McCain

asked the Department to respond to questions regarding the

completion of SDD. The letter was prompted by, and cited,

recent revelations that the program would be experiencing yet

another delay in completing SDD and cost overruns that may be

upwards of $1 Billion.

Although USD(AT&L) responded to the Senator on behalf of the

Department in a letter dated December 19, 2016, the following

are DOT&E’s responses to each of the questions.

Question #1: When will the Department complete the SDD

phase of the F-35?

• DOT&E Answer: SDD will close out in multiple phases.

Developmental ight testing is projected to end no

earlier than mid-2018, based on independent estimates

on completing mission systems ight testing – the testing

that will likely take the longest to complete. These

estimates—from the Director of Cost Assessment and

Program Evaluation (CAPE) of March 2018, the Director

of Developmental Test and Evaluation of March to

June 2018, Deputy Assistant Secretary of Defense for

Systems Engineering of July 2018, and my ofce of July

2018—are all later than the program’s estimate, based

on the amount of planned mission systems test points

remaining. (These estimates are optimistic because they

do not fully account for the corrections and verication

testing needed for the more than 270 high-priority

deciencies in Block 3F performance identied by a recent

review.) Then, incremental deliveries of the Block 3F

Outsider threats have neither physical access nor account privileges to a

network; near-sider threats have physical access to a system, but no account or

log-in privileges to a network.

FY16 DOD PROGRAMS

F-35 JSF 105

capabilities (i.e., ight envelope, weapons, and avionics)

for each variant will likely not be completed until late

2018 due to continued delays and discoveries with F-35B

and C ight sciences testing, along with weapons testing.

Finally, contract close out actions, including specication

compliance and verication and validation, will complete

no earlier than late 2019. Completion of all required

contracting action for the SDD phase will likely continue

for a number of years.

Question #2: How many additional funds, in each

upcoming scal year budget, will be required to complete

F-35 SDD?

• DOT&E Answer: Although DOT&E does not conduct

independent cost estimates, CAPE estimated that the

program would need an additional $550 Million in FY18 to

nish the necessary and planned developmental test points

and produce additional software versions to x and verify

the important known and documented deciencies, then an

additional $425 Million in FY19 and $150 Million in FY20

to complete SDD. These estimates add up to an additional

$1.125 Billion required to complete SDD. The Program

Ofce estimate is about one-half of the CAPE estimate.

Question #3: What other Service priorities will not receive

funding in scal year 2018 due to the SDD delay and cost

overrun?

• DOT&E Answer: Although the program recently claimed

that their estimated SDD overrun can be covered by

reallocating existing JSF program funding (other than

$100 Million in ight test risk), the SDD cost increase will

be much larger than the current program estimate for the

reasons described in this report. Therefore, the overrun

will not be completely covered with only program funds

and the Services will likely need to address the SDD cost

increase from within their budgets, or funding currently

designated for Follow-on Modernization (FoM) will need to

be reallocated to complete SDD.

Question #4: Is Secretary James’ Block 3F full combat

capability certication, as required by the Fiscal Year 2016

NDAA, still valid?

• DOT&E Answer: For many reasons, it is clear that the

Lot 10 aircraft that will begin delivery in early 2018 will

not initially have full Block 3F capability. These reasons

include, but are not limited to, the following:

- Envelope limitations will likely restrict the full planned

Block 3F carriage and employment envelopes of the

AIM-120 missile and bombs well into 2018, if not later.

- The full set of geographically specic area of

responsibility mission data loads (MDLs) will not be

complete, i.e., developed, tested and veried, until 2019,

at the soonest, due to the program’s failure to provide

the necessary equipment and software tools for the U.S.

Reprogramming Laboratory (USRL).

- Even after the MDLs are delivered, they will not be

tested and optimized to deal with the full set of threats

present in IOT&E, let alone in actual combat, which is

part of full combat capability.

- The program currently has more than 270 Block 3F

unresolved high-priority (Priority 1 and Priority 2, out of

a 4-priority categorization) performance deciencies, the

majority of which cannot be addressed and veried prior

to the Lot 10 aircraft deliveries.

- The program currently has 17 known and acknowledged

failures to meet the contract specication requirements,

all of which the program is reportedly planning to get

relief from the SDD contract due to lack of time and

funding.

- Dozens of contract specication requirements are

projected to be open into FY18; these shortfalls in

meeting the contract specications will translate into

limitations or reductions to full Block 3F capability.

- Estimates to complete Block 3F mission systems extend

into the summer of 2018, not just from DOT&E, but

other independent Department agencies, making delivery

of full capability in January 2018 nearly impossible to

achieve, unless testing is prematurely terminated, which

increases the likelihood the full Block 3F capabilities

will not be adequately tested and priority deciencies

xed.

- Deciencies continue to be discovered at a rate of about

20 per month, and many more will undoubtedly be

discovered during IOT&E.

- ALIS version 3.0, which is necessary to provide full

combat capability, will not be elded until mid-2018;

also, a number of capabilities that had previously been

designated as required for ALIS 3.0 are now being

deferred to later versions of ALIS (i.e., after summer of

2018).

- The Department has chosen to not fund the CAPE

estimate for the completion of Block 3F mission systems

testing lasting until mid-2018, an estimate which is

at least double the Program Ofce’s latest unrealistic

estimate to complete SDD. This guarantees the program

will attempt a premature resource- and schedule-driven

shutdown of mission systems testing, which will increase

the risk of mission failures during IOT&E and, more

importantly, if the F-35 is used in combat.

- Finally, rigorous operational testing, which provides the

sole means to evaluate actual combat performance, will

not complete until at best the end of 2019—and more

likely later.

Question #5: How will this delay and cost overrun affect

the current overall schedule for Joint Strike Fighter

deliveries to the Services?

• DOT&E Answer: The Program Ofce currently has no

plans to delay the production and delivery schedule of

aircraft to the Services. However, since Lot 10 aircraft

will not initially be delivered with full combat capability,

including operational MDLs for Block 3F, the Services

will need to plan for accepting aircraft with less capability,

FY16 DOD PROGRAMS

106 F-35 JSF

possibly with Block 3i capability, until full Block 3F

capability can be delivered.

Question #6: When will you complete the operational test

and evaluation phase?

• DOT&E Answer: The IOT&E is planned to cover a

span of approximately 12 months, and will start after the

program is able to meet the TEMP entrance criteria and

the Department certies that the program is ready for test.

These entrance criteria are common-sense and carefully

dened requirements that were well-coordinated with the

Services and JPO as the TEMP was being staffed. Meeting

these criteria to enter IOT&E is necessary to ensure the

test is conducted efciently and effectively within the time

span planned and to minimize the risk of failing IOT&E,

or causing a “pause test” and having to reaccomplish costly

test trials, which would only further delay the completion of

IOT&E and increase program costs. Since the program will

not be ready to start IOT&E until late 2018, at the earliest,

and more likely 2019, completion of IOT&E will not occur

until late 2019 or early 2020.

Question #7: When will you make the

Milestone C/Full-Rate Production decision?

• DOT&E Answer: Since the Milestone C/Full-Rate

Production decision cannot be made until after IOT&E is

completed and DOT&E has issued its report, it cannot occur

by the threshold date of October 2019 and will likely not

occur until early 2020, at the soonest.

Question #8: Will you defer any planned F-35 capabilities

from SDD into the F-35 Follow-on Modernization (FoM)

program?

• DOT&E Answer: Multiple F-35 capabilities will be

deferred from SDD or not function properly in Block 3F

unless the program continues testing and xing deciencies.

The program currently has hundreds of unresolved

deciencies and immature capabilities, including 17

documented failures to meet specication requirements

for which the program acknowledges and intends to seek

contract specication changes in order to close out SDD.

Question #9: How will the SDD delay affect the Follow-on

Modernization (FoM) program?

• DOT&E Answer: Delays to the completion of SDD will

impact both the FoM program schedule and content. While

FoM is critical for the capabilities needed with the F-35 and

the program is attempting to minimize delays, the program

does not appear to be ready to complete all prerequisites to

start full development in FY18, as planned. Also, IOT&E

will not be complete until late 2019 or early 2020, which

overlaps with the planned test periods for Block 4.1.

Finally, the program’s current plans for FoM are not

executable, for many reasons, which include the following:

- Too much technical content for the production-schedule-

driven developmental timeline

- Overlapping capability increments without enough time

for deciencies from OT to be xed prior to releasing the

next increment

- High risk due to excessive technical debt and

deciencies from the balance of SDD and IOT&E being

carried forward into FoM because the program does not

have a plan or funding to resolve key deciencies from

SDD prior to attempting to add the planned Block 4.1

capabilities

- Inadequate test infrastructure (aircraft, laboratories,

personnel) in the current FoM plan to meet the testing

demands of the capabilities planned and the multiple

congurations (i.e., TR2, TR3, and Foreign Military

Sales)

- Insufcient time for conducting adequate DT and OT for

each increment

Question #10: When will you provide your nal response

either to validate the current requirement for the F-35 Joint

Strike Fighter total program of record quantity or identify

a new requirement for the total number of F-35 aircraft

that the Department would ultimately procure?

• DOT&E Answer: DOT&E is not aware of when the

Department will complete these actions.

Recommendations

• Status of Previous Recommendations. The program

adequately addressed 5 of the 14 previous recommendations.

As discussed in the appropriate sections of this report, the

program did not, and still should:

1. Acknowledge schedule pressures that make the start of

IOT&E in August 2017 unrealistic and adjust the program

schedule to reect the start of IOT&E no earlier than late

CY18.

2. The Department should carefully consider whether

committing to a “block buy” is prudent given the state of

maturity of the program, as well as whether the block buy is

consistent with a “y before you buy” approach to defense

acquisition and the requirements of title 10 U.S. Code.

3. Plan and program for additional Block 3F software builds

and follow-on testing to address deciencies currently

documented from Blocks 2B and 3i, deciencies discovered

during Block 3F developmental testing, and during IOT&E,

prior to the rst Block 4 software release planned for 2020.

4. Ensure the testing of Block 3F weapons prior to the start

of IOT&E leads to a full characterization of re-control

performance using the fully integrated mission systems

capability to engage and kill targets.

5. Provide the funding and accelerate contract actions to

procure and install the full set of upgrades recommended

by DOT&E in 2012, correct stimulation problems, and x

all of the tools so the USRL can operate efciently before

Block 3F mission data load development begins.

6. Complete the planned testing detailed in the

DOT&E-approved USRL mission data optimization

operational test plan and amendment. Although some

FY16 DOD PROGRAMS

F-35 JSF 107

testing was completed, the program should ensure all

operational Block 3i MDLs are tested per the approved test

plan.

7. Along with the Navy and Marine Corps, conduct an actual

operational test of the F-35B onboard an L-class ship

before conducting a combat deployment with the F-35B.

This test should have the full Air Combat Element (ACE)

onboard, include ordnance employment and the full use

of mission systems, and should be equipped with the

production-representative support equipment.

8. Develop a solution to address the modication and retrot

schedule delays for production-representative operational

test aircraft for IOT&E. These aircraft must be similar to, if

not from, the Lot 9 production line.

9. Develop an end-to-end ALIS test venue that is production

representative of all ALIS components. Although the

program has developed the ORE, only limited testing has

occurred.

• FY16 Recommendations.

1. The program should complete all necessary Block 3F

baseline test points. If the program uses test data from

previous testing or added complex test points to sign off

some of these test points, the program must ensure the data

are applicable and provide sufcient statistical condence

prior to deleting any underlying build-up test points.

2. In light of the fact that the program is unable to correct

all open deciencies prior to IOT&E, the program should

assess and mitigate the cumulative effects of the many

remaining SDD deciencies on F-35 effectiveness and

suitability, especially those deciencies that, in combination

or alone, may cause operational mission failures during

IOT&E or in combat, prior to nalizing and elding Block

3F. The program will need to add test points to troubleshoot

and address deciencies that are currently not resolved.

3. The program should consider developing another full

version of Block 3F software to deliver to ight test in order

to address more known deciencies.

4. The program should ensure adequate resources remain

available (personnel, labs, ight test aircraft) through

the completion of IOT&E to develop, test, and verify

corrections to deciencies identied during ight testing.

5. The program should address the deciency of excessive

F-35C vertical oscillations during catapult launches within

SDD to ensure catapult operations can be conducted safely

during IOT&E and during operational carrier deployments.

6. The Program Ofce must immediately fund and expedite

the contracting actions for the necessary hardware and

software modications to provide the necessary and

adequate Block 3F mission data development capabilities

for the USRL, including an adequate number of additional

radio frequency signal generator channels and the other

required hardware and software tools.

7. The program should address the JOTT-identied shortfalls

in the USRL that prevent the lab from reacting to new

threats and reprogramming mission data les consistent

with the standards routinely achieved on legacy aircraft.

8. The program should correct deciencies that are preventing

completion of all of the TEMP-required Block 3F Weapons

Delivery Accuracy (WDA) events and ensure the events are

completed prior to nishing SDD.

9. The program should ensure Block 3F is delivered with

capability to engage moving targets, such as that provided

by the GBU-49, or other bombs that do not require

lead-laser guidance.

10. The program should complete additional testing and

analysis needed to determine the risk of pilots being harmed

by the Transparency Removal System (which shatters the

canopy rst, allowing the seat and pilot to leave the aircraft)

during ejections in other than ideal, stable conditions (such

as after battle damage or during out-of-control situations).

The program should complete these tests as soon as

possible, with the new equipment, including the Gen III

Lite helmet in a variety of off-nominal conditions, so that

the Services can better assess risk associated with ejections

under these “off-nominal” conditions.

11. The program needs to conduct an assessment to determine

the extent to which the results of further durability

testing with BH-1, the F-35B durability test article, are

representative of production aircraft and, if necessary,

procure another test article for the third life testing.

12. The Navy and the Program Ofce should investigate

alternatives for determining the operational impact of an

engine removal and install while conducting carrier air wing

operations at sea.

13. The Navy and Marine Corps should conduct an analysis,

such as an operational logistics footprint study, which

simulates ight deck and hangar bay spotting (aircraft

placement) with a full ACE onboard, using data from the

DT-III ship trials to determine what the impact of an engine

removal and installation would be on integrated ship and

ACE operations with a full ACE onboard.

14. The program and the Navy should investigate if the heavy

power module container should be redesigned for better

usability at sea.

15. The program and the Navy should investigate potential

options to improve ship-based communications bandwidth

dedicated to ALIS connectivity off-ship, such as increasing

the priority of ALIS transmissions, or reserving low-use

times of the day for handling large volumes of ALIS

message trafc.

16. The Navy should investigate any efcient, multi-use

opportunities for F-35 support equipment (SE) such as

using legacy SE on the F-35 or F-35 SE on legacy aircraft.

17. The Navy should investigate options for increasing the

number of wall power outlets in CVN hangar bays to help

facilitate simultaneous maintenance on multiple F-35Cs,

or the ability to interconnect multiple pieces of support

equipment from a single outlet to permit simultaneous

operations.

FY16 DOD PROGRAMS

108