Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Correlation Does Not Mean Causation:
Improving Army Software Maintenance Cost Estimates
Presented to
Joint Software and IT Cost Forum
September 2020
1
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Presentation Overview
• Objectives and Strategy
• Data Demographics
• Causal Relationship Driven CER
• Next Steps
Sept 2020 2
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
SWS Initiative Objective and Strategy
Accurately estimate Army system Software Sustainment (SWS) costs to:
- Effectively estimate and justify software and system life cycle costs
- Objectively evaluate Army system software sustainment execution costs
- Inform and optimize the allocation of available sustainment resources across the
Army
Effective software sustainment cost estimation is the basis for
Army system software life cycle cost management
Collect and evaluate SWS cost and
technical data for all Army
operational systems (Phase I and
Phase II data call)
Generate and validate cost
estimating relationships from
Phase I and Phase II data
collection
Implement systemic Army SWS
data collection via the SRDR-M:
Populate cost and technical data
repository
Improve Army SWS policy,
business, and technical
requirements
Sept 2020 3
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Summary of Accomplishments
• Established Software Sustainment Data Collection Mechanisms
− Army Software Data Collection Questionnaire
− Software Sustainment WBS Used to Collect Sustainment Costs
− Annual Data Collection
• Created Comprehensive Software Sustainment Data Repository
− 192 Systems
− 700 Capability Releases
− 300 IAVA Releases
− 3,200 records on software license data
• Established Robust Foundation for Software Sustainment Fact-Based
Decisions
− Allocations of Costs by WBS Elements
− Continue to improve Software Sustainment Cost Estimating Relationships
• Data and Analysis Results provided to DoD Community
Sept 2020 4
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
DASA-CE SWS WBS
Software Sustainment
1.0 Software Change
Product
1.1 Capability Changes
Change requirements
Change development
B/L Integration & Test
IV&V
1.2 IAVA Changes
On-Site technical
assistance
Problem Troubleshooting
S/W Installation
Operational Assistance
On-site Training
Operations
Organization management
Personnel management
Financial management
Information management
Process management
Change management
3.0 Software
Licenses
4.0 Certification &
Accreditation
8.0 Other
7.0 Field
Software Eng.
Version 5.0
5.0 System
Facilities
6.0 Sustaining
Engineering
Non-System Specific
2.0 Project
Management
Planning
Execution management
Configuration management
Resource & team management
Contracting management
Measurement - reporting
System Specific
System Specific System/Non-System Specific
System Specific System Specific
System/Non-System Specific System/Non-System Specific
License - Right to Use
COTS/GOTS
NDI
Open Source
Other
License - Maintenance
Security
Safety
Networthiness
Airworthiness
Hardware
Software development
assets/workstations
System integration & test facilities
Test equipment - tools
Facility Operations
6.1 Help desk
6.2 Hosting
6.3 Engineering and User
Support
Test Support
Software Delivery
Technical Studies
Training
Sept 2020 5
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Army Software Sustainment Definition
• Software sustainment (SWS) includes all software change activities and
products associated with modifying a software system after a software
release has been provided to an external party
• The release is the primary SWS change product - a composite of one or
more changes - it can be either a formal release or an engineering
release
• SWS includes software enhancements, software maintenance, and
cybersecurity updates
• Software maintenance includes defect repair, rehosting, adaptations,
updates, and reconfiguration
• SWS may be funded by multiple funding sources
• Costs include both Fixed and Variable costs accrued at both the system
and organizational levels
• Costs include both organic (government) and contractor resources
Sept 2020 6
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Data Demographics
Releases by Change Type
Distribution of System AgeSystems by Super Domain
Releases by Size Measure
192 Systems
1,040 Releases
3,434 Licenses
411K Data Fields
Overview
Sept 2020 7
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Causal Relationship Driven CERs
Sept 2020 8
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Correlation versus Causation
• It is well known that correlations among factors does not necessarily
mean causation.
− For example, an increase in ice cream sales is correlated with shark attacks
• Because of this, regression models are often the wrong tool to use for
causal search, i.e., identifying which factors affect the outcome.
• These models may use predictor variables that are influenced by
va
riables outside the model, confounding variables. The model may
have a good fit to the data but will not be accurate making estimates
Sept 2020 9
X
1
X
2
X
3
X
4
X
5
X
6
Y
Confounding Variable Example
• X
2
influences both X
1
and Y and X
1
influences Y. A change in X
2
will
produce a change in both X
1
and Y
• X
6
influences Y as does X
5
but X
6
also
influences X
5
. A change in X
6
will
produce a change in both X
5
and Y
• Conclusion: choose only 2 of the 4
influencing factors on Y
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Causal Relationship Analysis
• The Army Software Maintenance Initiative collaborated with the
Software Engineering Institute (SEI), Pittsburgh, PA, to investigate cause
and effect relationships in collected maintenance data.
• The large number of factors in the software maintenance data make it
challenging to identify which ones are useful for grouping data
• As a result of causal analysis, the data was segmented into two tiers
− First tier was data segmented by Super Domain
− Second tier was segmented by ACAT level within each super domain
• Unit cost (total release hours per software change) was used as the
variable of interest, Y, in the analysis
• Different factors appear in different groups meaning different predictors
are used in CERs
• This presentation only shows the causal relationship graphs and CERs
for the Real-Time super domain and the three ACAT level releases
Sept 2020 10
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Causal Analysis Influencing Factors
Sept 2020 11
Super Domain ACAT I ACAT II ACA III+
RT
• Phase
• Inter-Service
Partner Count
• HW Platforms
• Inter-Service
Partner Count
• Hardware Variants
• Maintenance Phase
• Software Baseline
ENG (None)*
• Number of
Appropriations
• Hardware Variants
• Maintenance
Phase
• Inter-Service
Partner Count
• Software Baseline
• Duration Type
• Hardware Variants
• Maintenance Phase
AIS
• Inter-Service
Partner Count*
(No info)
• Duration Type
• Inter-Service Partner
Count
* There were very few observations in two of three ACAT I datasets making
casual effects harder to analyze.
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Real-Time ACAT I Releases
Sept 2020 12
Data Model Conditions Obs Adj R
2
ACAT I
ACAT II
ACAT III+
Thr = 399 * (TSC)
0.91
Thr = 364 * (TSC)
0.91
Thr = 569 * (TSC)
0.91
10% | RT | ACAT Levels 68 0.71*
THrs = 577 * (TSC)
0.81
10% | RT | ACAT I 23 0.74
THrs = 164 * (TSC)
1.0
* ServCnt
1.10
10% | RT | ACAT I |
ServCnt
23 0.84
THrs = 91 * (TSC)
0.94
* HW_Plat
0.18
10% | RT | ACAT I |
HW_Plat
22 0.80
MS C
MS C - FRP
O&S
THrs = 260 * (TSC)
0.74
THrs = 394 * (TSC)
0.74
THrs = 787 * (TSC)
0.74
10% | RT | ACAT I |
Phases (Ordinal)
23 0.71
THrs = 161 * (TSC)
1.00
* ServCnt
1.05
*
HW_Plat
0.0011
10% | RT | ServiceCnt |
HW_Plat
22 0.84*
* High P-Values for one or more coefficients
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Real-Time ACAT II
Sept 2020 13
Data Model Conditions Obs Adj R
2
ACAT I
ACAT II
ACAT III+
Thr = 399 * (TSC)
0.91
Thr = 364 * (TSC)
0.91
Thr = 569 * (TSC)
0.91
10% | RT | ACAT Levels 68 0.71*
THrs = 308 * (TSC)
0.95
10% | RT | ACAT II 23 0.75
THrs = 298 * (TSC)
0.94
* ServCnt
0.86
10% | RT | ACAT II |
ServiceCnt
23 0.75
* High P-Values for one or more coefficients
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Real-Time ACAT III+
Sept 2020 14
Data Model Conditions
Obs
Adj R
2
ACAT I
ACAT II
ACAT III+
Thr = 399 * (TSC)
0.91
Thr = 364 * (TSC)
0.91
Thr = 569 * (TSC)
0.91
10% | RT | ACAT Levels 68
0.71*
THrs = 467 * (TSC)
0.99
10% | RT | ACAT III+
22
0.61
THrs = 280 * (TSC)
1.02
* SW_Base
0.46
10% | RT | ACAT III+ | SW_Base
22
0.65
THrs = 252 * (TSC)
1.01
* HW_Var
0.74
10% | RT | ACAT III+ | HW_Var
22
0.64*
MS C - FRP
MS C - LRP
O&S
THrs = 272 * (TSC)
0.92
THrs = 465 * (TSC)
0.92
THrs = 776 * (TSC)
0.92
10% | RT | ACAT III+ | Phases
(Ordinal)
22
0.67*
MS C - FRP
MS C - LRP
O&S
THrs = 138 * (TSC)
1.07
* SW_Base
0.51
THrs = 70 * (TSC)
1.07
* SW_Base
0.51
THrs = 328 * (TSC)
1.07
* SW_Base
0.51
10% | RT | ACAT III+ | SW_Base
| Phases (Ordinal)
22
0.67*
* High P-Values for one or more coefficients
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Conclusions on Causal-driven CERs
• Causal relationship analysis provided insight into which independent
variables should be examined for predicting total release hours
• This saved a lot of random analysis time
• The discovered relationships also suggested other relationships that
could answer different information needs such as which data does not
contribute to CER formulation and can be eliminated from data
collection
• Segmenting data as suggested by causal analysis generally shows more
CER accuracy in each segment versus trying to find a one-size-fits-all
CER.
• It also highlights poor performing members in the segment that need
further investigation.
• Causal analysis should proceed regression analysis to save time and
eliminate confounding variables
Sept 2020 15
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Next Steps
• Next Steps
• Annual data collection
− Collection of FY18 PPSS actual execution data by Army G4
− Development of Army OSMIS data repository for data collection and storage
− The Software Resources Data Reporting for Maintenance (SRDR-M*) closely aligns
to the DASA-CE SWS WBS and data requirements
o Moving forward, the SRDR-M will be utilized to collect SWS data from Army programs
and perform analysis
• Annualized release data will continue to be analyzed for benchmarks,
annual changes in software changes, and for estimating relationships
with the other WBS elements
• The causal relationships will be updated using both new and old data
and the CERs will be revised based on the discovered relationships
Sept 2020 16
*See http://cade.osd.mil/policy/dids for more information
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Contributors
Kimberly Roye
Senior Analyst
Galorath Federal
703- 966-3192
Cheryl Jones
Software Measurement Analyst
US Army AFC
973-724-2644
Paul Janusz
Software Measurement Analysis
US Army AFC
973-724-2644
Dr. Bradford Clark
Vice President
Software Metrics, Inc.
(703) 754-0115
Dr. Robert Charette
President
ITABHI Corporation
(540) 972-8150
Sept 2020 17
Dr. Christian Smart
Chief Scientist
Galorath Federal
256-457-3354
Mr. James Judy
U.S. Army DASA CE
703-697-1612
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Acronyms -1
ACAT Acquisition Category
AIS Automated Information System super domain
BL Software Change Backlog
BY Base Year
C&A Certification and Accreditation
C5ISR Command, Control, Communications, Computers, Cyber, Intelligence,
Surveillance, and Reconnaissance
CADE Cost Assessment Data Enterprise
CER Cost Estimating Relationship
COTS Commercial Off The Shelf
CRED Uncertainty Estimation Determination
CSCI Computer Software Configuration Item
Cyber% Percent of the release that is Cybersecurity updates
DASA-CE Deputy Assistant to the Secretary of the Army for Cost and Economics
DBS Defense Business System commodity
DIACAP DoD Information Assurance Certification and Accreditation Process
DISA Defense Information Systems Agency
DoD US Department of Defense
DSLOC Delivered Source Lines of Code
ECP Engineering Change Proposal
El_Mod External Interfaces Modified
ENG Engineering super domain
Enh% Percent of the release that is Enhancements to the system
EW Electronic Warfare
Sept 2020 18
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Acronyms -2
FSE Field Software Engineering
FTE Full Time Equivalent
IAVA Information Assurance Vulnerability Alert
IAVM Information Assurance Vulnerability Management
ICEAA International Cost Estimating and Analysis Association
Maint% Percent of the release that is Maintenance changes
NVD National Vulnerability Database
O&S Operations and Sustainment
ODC Other than Direct Costs
OMA Operations and Maintenance Army funding
OPA Other Program Army funding
OSMIS Operation/Sustainment Management Information System
PDSS Post-Deployment Software Support
PEO Program Executive Office
POM Program Objective Memorandum
PPSS Post-Production Software Support
PTR Problem Trouble Report
Sept 2020 19
Unclassified
Distribution Statement A: Approved for Public Release Distribution is Unlimited
Acronyms - 3
RDT&E Research, Development, Testing, and Evaluation
RMF Risk Management Framework
RT Real-Time super domain
SC Software Changes
SEC Software Engineering Center
SER Schedule Estimating Relationship
SLOC Source Lines of Code
SRDR Software Resources Data Report
SRDR-M Software Resources Data Report for Maintenance
STIG Security Technical Implementation Guides
SUP Mission Support super domain
SW Software
SWBase Software Baseline SLOC
SWS Software Sustainment
TDEV Time to Develop
THrs Total release hours
TReqts Total Requirements in a system
TReqts_Imp Total Requirements Implemented in a release
TSC Total Software Changes for a release
WBS Work Breakdown Structure
Sept 2020 20
