Understanding the Cost of Poor Software Quality: A Report, Essays (high school) of Computer science

Download Understanding the Cost of Poor Software Quality: A Report and more Essays (high school) Computer science in PDF only on Docsity!

Herb Krasner Member, Advisory Board Consortium for IT Software Quality (CISQ) www.it-cisq.org [email protected] Date: September 26, 2018

The Cost of Poor

Quality Software

in the US:

A 2018 Report

Contents

• 1. Forward
• 2. Executive Summary
• 3. Introduction
     • How much are we spending on IT software in the world today?
     • Illuminating a fundamental but unseen problem in IT systems
     • The cost of quality approach adapted to IT software
     • The Iceberg Model...........................................................................................................
• 4. The Landscape, Looking: Backwards, Forwards and at Present
     • Looking backwards: Legacy systems hold us captive
     • Looking forward: Tech innovations coming faster and faster..................................
     • Looking at today: Highly vulnerable and deficient systems of systems
     • The era of 9-digit failures and defects.
     • Troubled/challenged projects
     • Technical debt..................................................................................................................
     • Landscape summary
• 5. Human Talent Perspective on CPSQ
     • Defining the information technology workforce
     • Computer and information technology occupations in the US today
     • Impact of the IT gig economy
     • Implications
• 6. Cost of Software Quality: Definitions and Model
     • Definition of software quality
     • Good versus poor-quality software
     • The cost of software quality model and its evolution
     • Categories of CPSQ.........................................................................................................
     • Categories of CGSQ
  • 7. Conclusions
    • What the various sources have revealed—the cost of poor-quality software
    • Summary of poor software quality costs
    • Other observations
    • What to do
• 8. Acknowledgements
  • 9. Section References
    • Introduction section references
    • Landscape section references
    • Human talent section references
    • CoSQ section references.................................................................................................
    • Conclusion section references

2. Executive Summary

This project performs a comprehensive research study, evaluating the cost of software—specifically poor-quality software—on the US economy as a whole. Existing sources of public data were used in this report with all sources cited. This report fills a gaping hole in our understanding of the financial implications of poor-quality software effecting society today and into the future. This report is primarily for C-suite executives, CTOs, CIO’s and other IT professionals who are interested in quantifying their costs of poor-quality software. The report body describes the primary motivations for doing this study, including software’s critical importance to modern society and illuminating the fundamental issues causing problems. The iceberg model is used to show which software quality costs are usually hidden from sight. Next the landscape of software quality problem areas are described by 1) looking backwards in time, 2) forward into the future, and 3) identifying current issues facing us. The issues described include:

1. Legacy systems that hold our personnel and budgets captive;
2. Technical innovations that attempt to move us forward at accelerating rates;
3. Today’s highly vulnerable “Systems of Systems”;
4. Today’s era of 9-digit software systems’ failures and defects; and
5. The growing burden of technical debt. Once the landscape is defined, the labor force impacts are addressed by covering the following topics:
6. Defining the Information Technology Workforce;
7. Computer and Information Technology Occupations in the US Today (BLS);
8. Impact of the IT gig economy; and
9. Implications for quality and costs. Formal definitions of software quality and the cost of software quality model are introduced by defining:
10. Software Quality;
11. Good versus Poor-quality Software;
12. The cost of software quality model and its evolution;
13. Categories of Cost of Poor Software Quality (CPSQ); and
14. Categories of Cost of Good Software Quality (CGSQ).

The Cost of Software Quality (CoSQ) model identifies the component costs of quality and how those add up to form a notional total. A summary of cost categories for poor- quality software and data, and what these numbers are telling us in order to improve the situation are summarized. The management actions necessary to attack the problems and make a significant improvement in various organizational situations conclude the report. In summary, the cost of poor quality software in the US in 2018 is approximately $2.84 trillion , the main components of which are seen in the following graph. If we remove the future cost of technical debt, the total becomes $2.26 trillion. For simplification, the various cost categories are, at this time, assumed to be mutually exclusive. Clearly a deeper level of intersection analysis is warranted. We therefore view this amount as a potential upper bound. It was our intention to use this result as a starting point for community discussion and future in-depth benchmarking studies. The methods for arriving at these category amounts and total is presented in the sections of the body of this report. General recommendations for improvement depend on each organization’s unique situational context. These recommendations include:

1. Find and fix problems and deficiencies as close to the source as possible, or better yet, prevent them from happening in the first place. This is in line with industry movements such as early work product appraisals and continuous testing.
2. Measure the CPSQ. With these numbers in hand, you have the basis for a business case to invest smartly in software quality improvement.
3. Attack the problem by focusing on the different results of good vs. poor software quality in your shop and relevant benchmark organizations.
4. Economic target areas will likely include: cost of ownership, profitability, human performance impact, enabling innovation, and effectiveness of mission critical IT systems. FIGURE 1: AREAS OF COST RELATING TO POOR IT/SOFTWARE QUALITY IN THE US 18.22% Technical debt

Finding/fixing defects

Losses from SW failures 21.42% Legacy system problems 6.01% Troubled/ cancelled projects

The most recent survey of computer ownership was conducted in 2012. It revealed that in the three decades since the first survey, the percentage of homes with a computer increased almost tenfold, to nearly 80%. Moreover, a poll conducted in February 2018 by the Pew Research Center found that 77% of all Americans, and 94% of all Americans aged 18-29, own a smartphone. Worldwide smart phone usage is predicted to be 2.53 billion this year. Computing devices and software are the main tools that enable our personal lives, our society, industry and government; therefore, software quality and software security are among the most important topics of this decade. The importance of both quality and security will increase over the next decade. Computing technology is integral to all today’s activities. Software quality matters.

How much are we spending on IT software

in the world today?

According to IDC^9 , global information technology spending will top $4.8 trillion in 2018, with the US accounting for approximately $1.5 trillion of that market. They state that the United States is the largest technology market in the world, representing 31% of the global total. In the US according to CompTIA, the IT sector is poised for another strong year, 5.0% growth projected. The optimistic upside forecast is in the 7.2% range, with a downside floor of 2.8%. For the most recent year of available data, US exports of tech products and services were an estimated at $309 billion in 2016. Exports account for approximately $1 out of every $4 generated in the US tech industry. Forecasts of IT growth from various sources include:

- Gartner 4.5% worldwide forecast - IDC 5.3% worldwide forecast - Forrester 5.8% US forecast - CompTIA 5.0% worldwide forecast According to Gartner^5 , about $3.7 trillion dollars will be spent worldwide on IT enterprise systems in 2018; an increase of 6.2% over last year. Their study, focusing on purchased products and services, covers the cost of: Data Center Systems, Enterprise Software, Computing Devices, IT Services and Communications Services. Enterprise software spending is forecast to experience the highest growth in 2018 with an 11. percent increase, at about $400 billion. Application software spending is expected to continue to rise through 2019, and infrastructure software will also continue to grow, bolstered by modernization initiatives. When considering the additional expenses of labor and support costs, it is asserted by Apptio^7 that global IT spending is actually closer to $6.3 trillion —because in most companies, the cost of labor accounts for close to 45% of their IT spending. It would appear that this number covers IT products, services and labor; but probably does not cover things like embedded systems, IoT, lost market share, stock declines, legal costs, etc.—and other costs associated with problematic IT systems and services.

The US share of that $6.3 T would be $1.953 trillion, or approximately 9.56% of US GDP ($20.4 T). It would not be unreasonable to suggest that US total IT spend in these products, services and labor, is about 10% of GDP. The following graph is freely available on the Gartner web site^5. Accepting Gartner’s report with Apptio’s enhancement, a more correct representation of total worldwide IT spending is $6.3 trillion in 2018. The US share of that would be $1.95 trillion. Adding in potential missing categories described above, the US IT spending amount for products, services and labor is probably at least $2 trillion or about 10% of GDP. US GDP in 2018 is $20.4 trillion, or about 23.3% of the world economy. Software quality is important—just about every C-suite executive now knows that. But recognizing that concept in the abstract is one thing, while actually investing time and resources toward procuring, developing, releasing and/or evolving high- quality software is quite another. The fact of the matter is that, many executives don’t ultimately make software quality a top-level priority goal. This can be a serious mistake. Just ask Equifax! The reality of the situation is that there are serious costs associated with poor- quality software. It’s not just a question of undermining a company/organization’s reputation—although that has its own costs—it’s also a matter that’s directly reflected in the bottom line. Data Center Systems Software Devices IT Services Communications Services 2016 2017 2018 2019 2020 2021 2022 FIGURE 2: GARTNER’S FORCAST FOR 2018 WORLDWIDE DOLLAR-VALUED IT SPENDING $4, $4, $ $3, $2, $2, $1, $1, $ $ Growth Increased 1.8% pts to 6.2% $ US Billions

CoQ is a proven technique in manufacturing and service industries, both for communicating the value of quality initiatives and for identifying quality initiative candidates. CoSQ offers the same promise for the software industry but could be used more than currently. Initial uses of CoSQ indicate that it represents a very large percentage of development costs—60 percent and higher for organizations that are unaware of improvement opportunities^14. CoSQ use demonstrates significant cost savings for software organizations willing to undertake quality improvement initiatives. Perhaps more importantly, the use of CoSQ enables an understanding of the economic tradeoffs that accompany activities and expenditures made for improving the quality of delivered software.

The Iceberg Model

Many of the costs of poor IT software quality are hidden and difficult to identify with formal measurement systems. The iceberg model (figure below) is very often used to illustrate this concept: Only a minority of the costs of poor software quality are obvious—appearing above the surface of the waterline. But there is a huge potential for reducing costs under the waterline. Identifying and improving these costs will significantly reduce the costs of operating a business/organization. FIGURE 3: THE ICEBERG MODEL Costs Usually Visible Costs Usually Not Visible

**- Customer problem reports

• Customer service calls
• Lawsuits/warrantee claims
• QA & test department costs
• Service outages
• Finding & fixing internal problems/defects
• Cancelled and troubled projects
• Unaccounted overtime (crisis mode)
• Waste and rework
• Successful cyber attacks
• Staffing problems (e.g.turnover)
• Poor teamwork
• Lack of good planning
• Dubious project value/ROI
• Excessive systems costs
• Lost market opportunities
• Lack of good practices & quality standards
• Understanding complex code
• Technical debt
• Poor quality data**

4. The Landscape:

looking backwards, forwards

and at present

Looking backwards:

Legacy systems hold us captive

In 2002, NIST reported that estimates of the economic costs of faulty software in the US range in the tens of billions of dollars per year and have been estimated to represent approximately just under 1 percent of the nation’s gross domestic product (GDP). How has that changed in the 16 years since? In most companies and organizations, the Operation & Maintenance (O&M) of existing IT systems consumes the majority of the IT budget, roughly 75% of the total IT spend per year. For a particular system, software maintenance costs^23 will typically form 75-80% of the Total Cost of Ownership (TCO). In either case this leaves only about 25% for the development of new capabilities, products and systems. Respondents to a 2013 Forrester Research survey of IT leaders at more than 3, companies estimated they spent an average of 72% of their budgets on just keeping- the-lights-on functions. In 2016 the US Government Accountability Office found that 5,233 of the government’s almost 7,000 IT projects systems were spending “all of their funds on operations and maintenance”. Legacy IT systems reflect an organization’s past and present; they mirror both the complexity of the world they were developed for and that they currently operate in. If you peel away a system’s layers you see code and data flows that reflect rules governing the organization—some nuanced, some long forgotten—which determine how the software should process information. As the organization changes, new code is layered over existing code. Embedded systems, starting with military airplanes, ships, motor vehicles, railway signaling, telecommunications, the electricity grid, gas/oil analysis and even traffic lights, contain more software. Legacy systems become unwieldy due to aging, varying by particular type of system. One reason is the technology itself. The result of different departmental approaches, and inadequate IT strategy and governance leads to an assortment of diverse mainframes, servers, databases, computer languages and packages from multiple vendors. The resulting fragmented architecture—with thousands of interlinked subsystems—becomes costly to maintain and, as it ages, fewer people know how to work on it. IN THIS SECTION:

• Legacy systems that hold our personnel and budgets captive
• Technical innovations that attempt to move us forward at accelerating rates
• Today’s highly vulnerable “Systems of Systems”
• Today’s era of 9-digit software systems’ failures and defects
• The growing impact of technical debt

4. The Landscape:

Looking Backwards, Forwards

and at Present

Looking forward:

Tech innovations coming faster and faster

The Fourth Industrial Revolution, representing a transition to a new set of systems, bringing together digital, biological, and physical technologies in new and powerful combinations, is upon us. The term ‘Fourth Industrial Revolution’ was first used in 2016 at the World Economic Forum. New systems are being built on the infrastructure of the digital revolution (3rd). Just as the digital revolution was built on the heart of the second industrial revolution—electricity, mass communication systems, and modern manufacturing—the new systems that mark the Fourth Industrial Revolution are being built on the infrastructure of the third, digital revolution—the availability of global, digital communications; low-cost processing and high-density data storage; and an increasingly connected population of active users of digital technologies. The Fourth Industrial Revolution represents new ways in which technology becomes embedded within societies and even the human body. It is marked by emerging technology breakthroughs in a number of fields, including:

- Robotics - Nanotechnology - Quantum computing - Artificial Intelligence (AI)/Machine Learning (ML) - biotechnology - blockchain/cryptocurrencies - location-based platforms - Internet of Things (IoT) - virtual/augmented/mixed reality - e-learning - BYOD (Bring Your Own Device) - mobile computing - wearables/implantables - e-payment systems - autonomous vehicles - digital security technologies (especially multilayer authentication) These technologies, directly enabled by new computer software, challenge the systems of the past. They have great potential to connect billions of more objects/people to the Internet, drastically improve the efficiency of business and organizations and help regenerate the natural environment through better asset management. They hold unique opportunities to improve human communication and conflict resolution, while at the same time potentially causing large disruptions in our modern societies, especially when they fail massively. E.g., what happens when a self-driving auto kills a pedestrian without stopping? Is that a software flaw? Very likely, YES—probably one of omission—and it’s already happened.

The inherent characteristics of these new software systems will have increased: complexity, conformity, changeability and connectedness; requiring us to take a fresh look at how poor-quality software impacts future scenarios of developing and deploying these new technologies. These new technologies are primarily in the research and development (R&D) stage of their lifecycle. Software quality problems frequently occur when research prototypes are prematurely thrust into a product development stream.

Looking at today:

Highly vulnerable and deficient systems of systems

On average, software developers make 100 to 150 errors for every thousand lines of code^23. Of course, this number varies from developer to developer and project to project. Even if only a small fraction—say 10 percent—of these errors are serious, then a relatively small application of 20,000 lines of code will have roughly 200 serious coding errors. Not to place the blame solely on software developers, the Meta Group reports that up to 80 percent of the issues leading to customer dissatisfaction can be traced to poor understanding of requirements. Poor architecture causes a wide array of quality problems including fragility, lack of scalability, and resistance to modification. In summary, the whole software development process is fraught with opportunities to introduce problems and deficiencies. The main culprits in most problematic IT systems of today are sheer size and complexity. For example^43 , the Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google’s entire 18-year existence. The repository contains 86 terabytes of data, including approximately two billion lines of code in nine million unique source files. In terms of the largest single product, that’s probably Microsoft Windows at 500 million LOC. As of 2017, Microsoft announced what they believe is the world’s largest Git repository^44 :

- approximately 3.5M files that - result in a Git repository of about 300 gigabytes in size - with 4,000 engineers producing 1,760 daily “lab builds” across 440 branches, plus thousands of pull request validation builds. Even your smart phone has millions of LOC in it. Not to mention your new automobile with dozens of systems embedded, all interacting with each other.

Wells Fargo Bank In early August 2018, Wells Fargo admitted that as many as 400 homeowners were accidentally foreclosed upon after a “calculation error” in their accounting software denied them a mortgage loan modification. In their latest SEC filing, the bank promised to continue to assess any customer harm and provide remediation as appropriate. To that end, they have set aside $8 million for affected customers. PSA Airlines In June 2018, at PSA Airlines, issues with a crew-scheduling program caused thousands of flights to be cancelled for days last week. The computer problem was tied to the crew scheduling and tracking system at PSA Airlines, a wholly owned subsidiary that operates flights under the American Eagle brand. Those flights carry passengers to and from regional airports to major hubs like Charlotte, North Carolina. This was a significant IT systems issue that caused both PSA’s main systems and backup systems to slow down beyond a usable state. During the outage, American Airlines cancelled about 3,000 flights, with 2,500 of those to and from the Charlotte airport. Those cancellations stranded passengers in Charlotte and elsewhere, while drawing widespread ire from travelers, including on social media. The airline industry has been particularly hard hit with numerous IT failures this past year. Uber Technologies, Inc. In March 2018, a self-driving Uber SUV struck and killed a pedestrian in suburban Phoenix, Arizona, in the first death involving a fully autonomous test vehicle. Uber determined the likely cause of the fatal collision was a problem with the software that decides how the car should react to objects it detects. The car’s sensors apparently detected the pedestrian, but the software decided it did not need to react right away. Uber executives believe the system was tuned so it would be less responsive to objects in its path, such as plastics bags. How much this will eventually cost Uber internally and externally is yet to be determined. Another crash on March 23, 2018 of a Tesla Model X in Mountain View, California caused that company’s stock price to drop 3.3% the following day. TSB Bank Millions of TSB customers were locked out of their accounts after an IT upgrade led to an online banking outage. A planned system upgrade was expected to shut internet and mobile banking services down for one weekend in April 2018 but ended up causing weeks of disruption. The problems arose from TSB’s move to a new banking platform following its split from Lloyds Banking Group. Immediately after the new system was switched on, many customers experienced problems logging in, while others were shown details from other people’s accounts or inaccurate credits and debits on their own. Customers remained locked out of their accounts two weeks after the initial outage. TSB said it was handling their complaints on a case-by-case basis. Welsh NHS IT failure Doctors and hospital staff of the Wales NHS experienced a widespread computer failure that led to them being unable to access patient files. According to the National Cyber Security Centre, the failure was due to technical issues as opposed to a cyber-attack, yet it still caused wide disruption as GPs were unable to access blood and X-Ray results. It also caused a backlog as patients could not be contacted to cancel appointments, and notes could not be typed up and saved on NHS systems. TABLE 1: TOP 2018 IT FAILURES IN THE NEWS

Troubled/challenged projects^30

There are also thousands of troubled projects within organizations that rarely make the news. Looking at 25 years of historical projects in the Standish database, in 2017 they reported the following: that only 29% were fully successful with respect to time and budget. Their data says nothing about the quality of the result; presumably those had successful outcomes. The Standish Group research shows a staggering 19% of projects will be cancelled before they ever get completed. Further results indicate 52% of projects will cost 189% of their original estimates. The number and % of challenged projects (over budget, behind schedule, low quality deliverables) has barely changed over 25 years. The cost of these cancellations and overruns are usually hidden just below the tip of the proverbial iceberg. Meltdown & Spectre At the start of 2018, Google researchers revealed CPU hardware vulnerabilities called Meltdown and Spectre, which affected almost all computers on the market. Meltdown primarily affects Intel processors, while Spectre affects Intel, AMD and ARM processors. Although these are both primarily hardware vulnerabilities, they communicate with the operating system to access locations in its memory space. Meltdown breaks the most fundamental isolation between user applications and the operating system. This allows a program to access the memory, and also the secrets, of other programs and the operating system. Spectre meanwhile breaks the isolation between different applications—it allows an attacker to trick error-free programs, which follow best practices, into leaking their secrets. New Spectre flaws are still being discovered. Hawaii Sends Out State-Wide False Alarm About a Missile Strike On January 13, 2018 the citizens of Hawaii were notified to take immediate cover in the face of an inbound ballistic missile strike. It turned out to be a false alarm, although it took over 30 minutes (and, presumably, several thousand heart attacks) before the alert was retracted. Investigations found that while the problem was largely due to human error, there were “troubling” design flaws in the Hawaii Emergency Management Agency’s alert origination software. US CBP For a second year in succession, the US Customs and Border Protection (CBP) computer systems experienced an outage that left thousands of passengers across the United States waiting in long lines to clear customs. This time, the outage was only for about two hours, while last year’s incident lasted four hours and affected more than 13,000 passengers on 109 flights, according to a Department of Homeland Security Inspector General report. The 2017 New Year’s problem was caused by an inadequately tested software change related to CBP’s long-running IT modernization effort. Another report indicated that the main CPB computer system used to screen international passengers has seen its performance “greatly diminished over the past year as a result of ongoing efforts to modernize (its) underlying system architecture.” Before this latest outage, there were three other service disruptions in 2017, according to the DHS IG report.

Technical debt

Technical Debt is a forward-looking metric that represents the effort required to fix problems that remain in the code when an application is released. The CAST Appmarq benchmarking repository^29 was used to benchmark Technical Debt across different technologies, based on the number of engineering flaws and violations of good architectural and coding practices in that code base. CAST bases the Technical Debt calculation in an application as the cost of fixing the structural quality problems in an application that, if left unfixed, put the business at serious risk. Technical Debt includes only those problems that are highly likely to cause severe business disruption; it does not include all problems, just the most serious ones. Based on this definition and the 2011 analysis of 1400 applications containing 550 million lines of code submitted by 160 organizations, CAST estimates that the Technical Debt of an average-sized application of 300,000 lines of code (LOC) is $1,083,000. This represents an average Technical Debt of $3.61 per LOC; and that is just the principal owed. Java applications were higher at $5. per LOC. Assuming that this debt is generally true of all software applications and systems, then estimating the LOC existing in the US today, identifies the contribution of this area to the total CPSQ! According to Grady Booch^30 in 2005, when asked “how many LOC are written each year around the world?”; he stated that about 35 billion LOC are written each year. This is based on about 15 million or so software professionals worldwide, of which about 30% actually cut code. Each developer contributes about 7,000 LOC a year. Over history, there are about a trillion lines of code written so far up to 2005! How much has this number of LOC and technical debt grown over the last 13 years? If we conservatively assume that the worldwide code growth rate is steady at 35 billion new LOC per year, then in 2018 there would be 1.455 trillion LOC worldwide. And assuming that there is $4 of technical debt per LOC then the debt in 2018 would be $5.82 trillion; and in 2020, $6.1 trillion. And if the US share of that debt is roughly 31% (see the section on IT spending), US debt figures would be $1.8 trillion and $1.9 trillion respectively. Looking at the Open Source area for an example of where we have good data on source code in available repositories, we see that code growth follows an exponential growth pattern. Research in 2008^14 shows that the additions to open source projects, the total project size (measured in LOC), the number of new open source projects, and the total number of open source projects are growing at an exponential rate. The total amount of source code and the total number of projects double about every 14 months. This growth rate may not be indicative of non-open source projects since they have so many contributors. Figured another way, in 2011 CAST Software^42 estimated that the global IT technical debt was $500 billion and would rise to $1 trillion by 2015. If the doubling period is 4 years, then the debt in 2019 would be $2 trillion; and in 2018 $1.75 trillion. The US share of that would be $.54 trillion—a lower figure than our previous calculations would indicate. The total amount of source code and the total number of projects double about every 14 months.

Deciding that the latter technique is probably closer to the truth, we therefore conclude that the amount of IT technical debt in the US in 2018 is approximately $.54 trillion. That represents just the debt principal.

Landscape summary

Other important areas that are contributing major problems to the above landscape are:

- Cybersecurity vulnerabilities and the rapid rise in cybercrime - Commonly used open source software defects - Purchased software product and multi product suite deficiencies - Large systems of systems comprised of custom code mixed with COTS and Open source These issues are not elaborated here due to report size considerations. In May 2017, analyst firm Forrester produced a detailed forecast for the US technology market, titled US Tech Market Outlook For 2017 And 2018: Mostly Sunny, With Clouds And Chance Of Rain. Key takeaways from the report are that spending on software will grow by nearly 10 percent in 2017 and 2018, thanks mainly to increased cloud adoption, while consulting services and staff budgets will rise by 6-7 percent. However, total US tech spending growth will only be around 5 percent, thanks to barely rising budgets for outsourcing, hardware and telecom services. The outlook is that software is currently the main growth area in enterprise IT spending, while security and privacy remains a major priority. What is the cost of poor-quality software in these major buckets of the landscape of problem areas facing us right now? In summary, they are: - Legacy system CPSQ—$.635 trillion - Massive failures and 9-digit defects—$1.275 trillion - Troubled and cancelled projects—$.178 trillion - Technical debt—$.54 trillion It is not yet clear how the above categories intersect, implying the need for deeper research. Assuming the above categories are mutually exclusive, the CPSQ covering the landscape in 2018 is about $4.234 trillion. …the CPSQ covering the landscape in 2018 is about $2.4 trillion.