Economics of Assurance Techniques: A Review of 17 Contemporary Assurance Schemes, Study notes of Economics

Download Economics of Assurance Techniques: A Review of 17 Contemporary Assurance Schemes and more Study notes Economics in PDF only on Docsity!

Lancaster

Security

The Economics of Assurance Activities

Security Lancaster — Lancaster University

Contributors: Dr. Jose M. Such (Principal Investigator), Dr. Antonios Gouglidis, William Knowles, Gaurav Misra, Prof. Awais Rashid Security Lancaster Infolab21 SCC Lancaster University Lancaster LA1 4WA United Kingdom

Cite as: Such J.M., Gouglidis A., Knowles W., Misra G, Rashid A. The Economics of Assurance Activities. Technical Report SCC-2015-03, Security Lancaster, Lancaster University, 2015.

Acknowledgements: This Cyber Security research project was funded by the UK Government.

Disclaimer: This material is provided for general information purposes only. You should make your own judgement as regards use of this material and seek independent professional advice on your particular circumstances. Neither the publisher, nor the author, nor any contributors assume any liability to anyone for any loss or damage caused by any error or omission in the work, whether such error or omission is the result of negligence or any other cause.

Executive Summary

At the heart of the information assurance process lie the “assurance techniques” that are used in its assessments. Despite this, and against the backdrop of the year-on-year annual increases of security expenditures for organisations of all sizes, such assurance techniques remain largely unstudied holistically to understand them and their main characteristics, especially from the perspective of the economics of their use. This leaves some lingering questions unanswered: (i) which are these assurance techniques and what are their main characteristics? (ii) how are these techniques being used within particular assurance schemes? (iii) how do we ensure that the increasing number of trained professionals, products, and services in the information assurance space are deployed and utilised in a cost-effective manner?

This project intends to address this gap through a comprehensive review of the use of assurance techniques within 17 contemporary assurance schemes, and a large- scale stakeholder-supported study including 14 interviews as well as an on-line survey with 115 respondents on their perception of the use and value of such techniques in practice, in order to inform the design of future assurance schemes.

In order to mitigate against the subjectivity over what constitutes an assurance technique, a set of 25 assurance techniques were defined that spanned 6 categories: Review ; Interview ; Observe; Test; Independent Validation; Individual Competence. Relationships between assurance techniques were then described, e.g., where one contributed to another.

A framework was further defined to establish criteria for analysing assurance techniques, both independently, and within the context of specific schemes. The framework’s design was informed by the stakeholder interviews. These interviews were also used to collate scheme-specific information. This resulted in a mapping of the usage of assurance techniques within each of the 17 assurance schemes. In order to facilitate the design of security evaluation criteria for future assurance schemes, a mapping was also made between the defined assurance techniques and the security control families of ISO/IEC

An online survey was then conducted which received responses from a further 115 stakeholders. An analysis of stakeholder characteristics found 64% of respondents to be security practitoners (e.g., penetration testers) and 91.81% of all stakeholders had over 5 years of industry experience. Stakeholder representation across our range of chosen assurance schemes was high, in particular for ISO/IEC 27001 and Cyber Essentials.

For individual qualifications, “Oral Examination” was perceived to be the most effective assurance technique, with multiple-choice examination the least effective. A further review found “Oral Examination” and “Employment History and Qualification Review” to be the most cost- effective combination for assessing individual competence.

An analysis of assurance techniques for assessing security controls was also conducted. A baseline “medium” size target was chosen for the survey (e.g., a company with 250 employees or infrastructure with 16 external IPs or 150 internal IPs). The analysis included factors such as the number of people required, expertise required, time required, effectiveness, cost, complementary assurance techniques, and stakeholder confidence in their answer.

Stakeholders perceived “Penetration Tests” and “Red Team Exercises” to be the most effective assurance techniques, but also categorised them as “Expensive”. In contrast, both “Review of Client-Completed Self- Assessment Forms” and “Public Reviews” were perceived to be the least effective, but also the cheapest to conduct. A further analysis suggested the most cost- effective assurance techniques to be “Architectural Review” and “Vulnerability Scans” and “Penetration Tests”. The least cost-effective assurance techniques were perceived to be “Public Review”, “Emanation Security Analysis”, “Fuzzing”, “Static Analysis” and “Dynamic Analysis”.

A case study for a “special” environment was also described, in the form of Industrial Control Systems (ICSs). Stakeholders interviewed as part of this process perceived an endemic lack of security risk management processes in ICS environments, with security assessments (where they occurred) often providing limited assurance about an environment’s security. In order to encourage the development of ICS security risk management processes a series of practical “next steps” were identified.

A high level analysis of the economics of assurance schemes and incentives in the assurance scheme ecosystem, which could hamper/facilitate cost-effective assurance schemes and techniques, was also reported. A series of assurance scheme case studies were also conducted. Notably, this involved an analysis and comparison of the assurance ecosystem and incentives for ISO/IEC 27001 and Cyber Essentials certification.

Finally, the aggregate findings of the study were synthesised and consolidated into a series of conclusions and recommendations for improvement. This includes recommendations for assurance technique use in current and future assurance schemes.

Introduction

A notable trend in the body of literature on information assurance schemes is the focus on the operational benefits and challenges of using the scheme, or debate on the security controls that they outline. The assurance techniques used in the assessment of conformance to assurance schemes have largely escaped rigorous analysis. Where existing literature exists on assurance techniques, the focus has largely fallen on their role within software assurance. In particular, assurance techniques and their use within the Software Development Life Cycle (SDLC) [4]), or in rare cases, their use within specific product-focused assurance schemes (e.g., the classification of assurance techniques for use within Common Criteria [11]). The predominant body of work in this area has been instigated by the National Institute of Standards and Technology (NIST) project, Software Assurance Metrics And Tool Evaluation (SAMATE)^1 , which is sponsored by the U.S. Department of Homeland Security (DHS). An abundance of publications have been produced under this umbrella; notably around the topic of source code analysis, with a particular focus on static analysis^2. A comprehensive review of existing software security assessment tools is presented in [19], focusing on when they can be used, their required skills, and their benefits and drawbacks.

The role of economics within information assurance is a small but growing area of research focus; however, the majority of this research has focused on factors such as incentives (e.g., [3]), and limited attention has been paid to the economics of assurance techniques. Where this exists, the focus has again fallen on software assurance. For instance, [17] investigated the economic impact of having an inadequate infrastructure for software testing and [6] elaborated on existing approaches that are able to model and assess the cost and value of software. The scope of assurance techniques falls beyond software assurance, however, and it is in this broader application that this document is concerned: the multitude of assurance techniques, both non- technical (e.g., interviews and observation) and technical (e.g., penetration tests), which can be used in the assessment of security controls (be they technical, organisational or physical) or individual competence, and the economic factors inherent within this.

This study is the first one to report a comprehensive and extensive study of assurance techniques and their economics. Figure 1 depicts a high-level overview of the main steps of the methodology we used to produce this report. The initial process involved information gathering using three information sources. Firstly, publicly available information about the 17 assurance schemes shown in Table 1 and related literature was considered. Secondly, 14

interviews with security experts were conducted to retrieve information not publicly available, to validate information collected from publicly available information, and to check collected information for completeness. Interviews were also used to study the economics, incentives, and the assurance ecosystem, along with the ICS case study. Thirdly, an on- line survey was used to gather further information from 115 security professionals.

Start of project

Information gathering

Public information Interviews^ Survey

Analysis

Report

End of project

Terminology

External feedback

Framework

Figure 1: Methodology

Scheme Scope Target CBEST/STAR National (UK) Organisational security CEH International Individual qualification CESG CAPS National (UK) Organisational security CESG CAS National (UK) Organisational security CESG CCP National (UK) Individual qualification CESG CHECK National (UK) Individual qualification CESG CLAS National (UK) Individual qualification CESG CPA National (UK) Organisational security CESG CTAS National (UK) Organisational security CISSP International Individual qualification Common Criteria International Organisation security CREST National (UK) Individual qualification Cyber Essentials National (UK) Organisational security Cyber Scheme National (UK) Individual qualification ISO/IEC 27001 International Organisational security PCI DSS International Organisational security Tiger Scheme National (UK) Individual qualification Table 1: Assurance Schemes Reviewed

All of the gathered information was used to: (i) define a consistent and coherent assurance terminology to clearly define assurance schemes, targets, techniques, evidence and the relationships between them; (ii) define a full assurance technique framework, including 25 assurance techniques classified into 6 assurance technique categories, and the relationships between them (e.g., how the outputs from some are used as inputs to others); (iii) analyse and study the current assurance technique landscape; and (iv) propose recommendations for future assurance schemes. (^1) http://samate.nist.gov/Main Page.html (^2) A comprehensive list of SAMATE publications can be found at: http://samate.nist.gov/index.php/SAMATE Publications.html

Assurance Techniques

Potential variations of assurance techniques are abundant. Therefore, the definition of a consolidated set of assurance techniques is paramount to allow for consistency within the survey and ensuing analysis. This study defines 25 high-level assurance techniques, which are split over 6 categories. Four of these categories represent the broad techniques for assessing assurance targets, in the traditional sense of a security control: Review; Interview; Observe and Test. This is supplemented by a fifth category, Independent Validation, which represents third-party assessment. The final category is Individual Competence, which contains assurance techniques that assess an individual’s competence for using other assurance techniques (e.g., as part of a qualification).

This set of assurance techniques must be distinguished from two meta-techniques. The first of these is the audit, which is more appropriately defined as a process in which other assurance techniques are used to determine conformance to a specification. Assurance techniques in this context generate audit evidence. Such assurance techniques may be used directly by auditors (i.e., one or more individuals conducting an audit), although equally, an auditee (i.e., the client undergoing the audit) may also use assurance techniques, or procure services that use them (e.g., penetration tests), for which the audit evidence may be used by an auditor.

The second is risk assessment, which can be broken down into the consolidated steps of: asset identification; threat assessment; vulnerability assessment; risk evaluation (i.e., computing a measure of “risk”); and the recommendation of countermeasures. The assurance techniques that we have defined here are predominantly concerned with that of vulnerability assessment, although some assurance techniques contribute in full or part to the two prior steps (e.g., asset identification is a fundamental step of architectural reviews of operational systems, while threat assessment is explicitly defined here). The appropriate choice of assurance techniques here is paramount, as it is the outputs of these techniques that provide the variables for risk computation, which ultimately influences choices surrounding risk treatment (e.g., the implementation of new security controls). This importance for appropriate assurance technique choice can be extended when examining their role in risk management, which goes beyond the scope of a single risk assessment through monitoring and reviewing organisational risk over time. Controls may be implemented as part of the risk assessment process; the level of risk, pre and post-treatment, will then influence the choice of assurance techniques that are used within subsequent iterations of risk assessments. Therefore, if inappropriate assurance techniques are used it can have a wider impact on the risk management process.

The definition of the 25 high-level assurance techniques organised in 6 categories is provided below. Figure 3 visualises assurance techniques’ categorisation and their relationships.

Review

Review of Documented Policies, Procedures, and Processes - The process of analysing the documented specifications (e.g., procedures and security properties) and processes (e.g., managerial) for a component or system under assessment.

Review of Client-Completed Self-Assessment Form

• An analysis of a client submitted review of their implementation of assurance targets as set out within an assurance scheme. Self-assessment forms typically consist of a multitude of questions that a client must answer is multiple choice or narrative form.

Threat Assessment - A multi-stage process used to identify and rank the threats to computer software, a component, or IT system. Threat analysis builds upon the analysis of sub-processes such as asset identification and architectural reviews against a security policy.

Architectural Review - An analysis of the components (type, quantity, configuration, etc.) and their relationships within a piece of software, component, or system to determine if their implementation meets a desired security policy.

Configuration Review - A review of the way a system or its software has been configured to see if this leads to known vulnerabilities. Configuration reviews can be passive (e.g., manually checking software versions for known vulnerabilities) or active (e.g., automated build review scanners).

Source Code Review - The examination of source code to discover faults that were introduced during the software development process. Source code reviews are predominantly manual; however, they may be supplemented with automated techniques (e.g., using static analysis tools).

Observe

Observe - The process of watching a live, operational system to identify real-world deviations from documented assurance targets.

Individual Competence

Simulated Attack

Penetration Test

Red Team Exercise

Vulnerability Scan

Paper-Based Examination (Narrative)

Paper-Based Examination (Multiple Choice)

Oral Examinations (Viva Voce)

Employment History and Qualification Review

Configuration Review

Static Analysis

Dynamic Analysis Fuzzing

Cryptographic Validation

Architectural Review

Social Engineering

Threat Assessment (^) Formal Verification

Public Review Witnessed Test

Review of Documented Policies, Procedures, and Processes

Observe Interview

Emanation Security Analysis

Virtual Lab Examination

Review of Client-Completed Self-Assessment Form

Source Code Review

Test

Optional Contributing Assurance Technique Optional Parallel Assurance Technique

Independent Validation

Review

Figure 3: Assurance Activities

Interview

Interview - The process of questioning one or more individuals about security-related matters within the organisation being assessed through any medium (e.g., in person or virtually).

Test

Red Team Exercise - A simulated attack on a system that is given more freedom than is available during a penetration test, in order to more realistically simulate a real-world malicious attacker. This freedom is given in terms of the engagement’s duration (e.g., often months in duration), available human resources (e.g., large teams built around

individuals with different specialisms), allowed use of tools (e.g., a heavy use of social engineering is common), and restriction of defender knowledge to test their day-to-day responses to cyber threats.

Penetration Test - A simulated attack on a component or system using similar techniques to that of a real- world malicious attacker. A penetration test may build upon a vulnerability assessment; however, it differs in having an implicit or explicit goal that the assessment attempts to realise (e.g., compromise sensitive data or obtain a certain level of network access). Typically this requires vulnerabilities to be exploited, which would not be undertaken within a vulnerability assessment.

Vulnerability Scan - The process of using an automated scanner on a web application or network to identify

Use of Assurance Techniques within

Assurance Schemes

To further understand how assurance techniques are used in practice, it is required to study the role they play in particular assurance schemes. In this section, a descriptive analysis of the use of assurance techniques within assurance schemes is performed. Data for this was collected through an in-depth review of publicly available information about the 17 assurance schemes mentioned earlier, and targeted interviews to confirm and/or complete missing/incomplete information.

For each of the 25 assurance techniques, data was gathered about which of the 17 assurance schemes uses them. Then, for each assurance technique within each assurance scheme, the following data was gathered:

• Intended Outcome: A qualitative description of what an assurance technique is intended to achieve for a particular assurance scheme and how the results are reported (e.g., pass or fail for an examination, or the choice of metrics to report vulnerabilities).
• Lifecycle Stage: The stage of a component or system’s lifecycle in which an assurance technique is predominantly used. Five criteria are outlined: - Pre-Deployment - Before a component or system has been put into an operational environment. - Operational - Once the system is live. - Acquisition - An assessment prior, during, or after a component or system has been procured, but before it is deployed operationally by the purchasing organisation. - End of Life - When a system is being is being removed from active use. - N/A - Not applicable (e.g., for assurance techniques that assess individual competence).
• Qualifications and/or Certifications needed: The required prerequisites to be allowed to conduct an assurance technique. These can be applied at two levels: that of the individual (e.g., personal qualifications or security clearance) or that of the organisation (e.g., to be a certification body or other “approved” company).
• Sensitivity of Input Material: This study uses the data classifications mentioned below and outlined by the UK Cabinet Office’s 2013 publication, “Government Security Classifications” [18] (readers are referred to the UK Cabinet Office publication for a full description of each classification). 1. OFFICIAL (a) OFFICIAL-SENSITIVE COMMERCIAL

(b) OFFICIAL-SENSITIVE PERSONAL.

2. SECRET
3. TOP SECRET

• Extent of Contribution: Three criteria are defined to determine a level of extent that an assurance technique contributes to the collective assurance targets set out by an assurance scheme.

1. Xsig - An assurance technique is mandatory and its contribution to the scheme is significant. The term significant is qualified as an assurance technique that provides assessment to a large proportion of

Assurance Techniques and Security Controls

technique that is a necessary prerequisite to another Xsig activity, regardless of the proportion of security controls and requirements assessed.

2. Xmin - An assurance technique is mandatory; however, it’s contribution to the scheme in minor. The term minor is qualified as an assurance technique that is only applicable to the assessment of a small proportion of security controls or requirements, and is not a necessary prerequisite to an Xsig assurance technique.
3. Xop - An assurance technique is suggested, but an alternative could be used in its place to assess the outlined security controls and requirements.

Appendix A: Assurance Technique Characteristics per Assurance Scheme details all the results obtained. The criteria represent the columns, and each row describes the characteristics of an assurance technique within the context of a particular scheme. A tabular approach enables ease of analysis, and if interactive, the sorting and filtering by particular characteristics. Such functionality enables it to serve as a valuable descriptive resource on the contemporary usage of assurance techniques, both for the design of future schemes, and if in the public domain, those wishing to procure assurance techniques for use within assurance schemes.

Next, a high level analysis of the table in Appendix A is reported. First, Figure 4 lists all the assurance techniques and how often they are used within assurance schemes (only reported values where explicit mentions of use of assurance technique was found within an assurance scheme). It can be seen that Review of Documented Policies, Procedures and Processes was found to be the most widely used assurance technique across all the organisational security schemes that were surveyed during this research. On the other hand, none

of the assurance schemes reviewed included Static Analysis, Dynamic Analysis and Public Reviews.

Figure 4: Number of Assurance Schemes in which each Assurance Technique is employed.

Having a closer look at individual variables reported in Appendix A, the intended outcome variable contains a qualitative description of what the technique is aiming to achieve for that particular assurance scheme. This information is important to contextualize the effectiveness of any technique, as the effectiveness of an assurance technique is perceived with respect to its intended outcome. It can be seen that the intended outcome of the assurance technique often depends on the assurance scheme it is employed in. For example, Review of Documented Policies, Procedures and Processes is used to perform an assessment statement to outline risks and recommendations when it is used as part of the CESG Tailored Assurance Scheme (CTAS). On the other hand, it is used ensure compliance with established standards and provide audit trails for other assurance schemes like ISO/IEC 27001 and CESG Assured Services (CAS). Thus, the same technique can be used for different objectives depending on the assurance scheme.

An interesting observation regarding the Lifecycle stage is that most techniques are used for Operational systems regardless of the assurance scheme they are used in. One notable exception to this is the Common Criteria assurance scheme. It can be seen from the table that assurance techniques like Review of Documented Policies, Procedures and Processes, Source Code Review, Penetration Testing, Vulnerability Scans and Cryptographic Validation are used in the Pre-Deployment phase even though they are used for Operational systems when employed in other assurance schemes.

Regarding extent of contribution, a general observation is that most assurance techniques that are explicitly mentioned to be used within particular assurance schemes

are mandatory and its contribution to the scheme is significant (Xsig). There are only few exceptions to this ( out 92 cases), in which assurance techniques were deemed to be either mandatory but with a minor contribution (7 cases) or were optional (9 cases). Notable cases were those of Penetration Testing and Vulnerability Scans, which are both optional in ISO/IEC 27001, yet they were rated among the most cost-effective assurance techniques by the security practitioners that filled out the aforementioned on-line survey. Another interesting observation is that an assurance technique can be a significant part of an assessment for a particular assurance scheme while it may be optional for an assessment for a different assurance scheme. For example, Source Code Review are mandatory and a significant part of a Common Criteria evaluation but they are an optional part of a CTAS evaluation and they may or may not be employed. Moreover, in other assurance schemes such as PCI DSS, for example, Source Code Reviews are not employed at all.

Assurance Techniques and Security Controls

Assurance schemes like Cyber Essentials clearly dictate the assurance technique to be used to assess the security controls it mandates (e.g., review of self-assessment forms to check the 5 Cyber Essentials security controls). However, there are many other assurance schemes in which this is unclear. Furthermore, the effectiveness of an assurance technique is obviously relational to the security control (i.e., assurance target) in which it is assessing.

A preliminary mapping of assurance techniques to the high-level security families of ISO/IEC 27001 has been produced. It is believed that such a mapping will aid in the development of compliance evaluation criteria for the security controls outlined in future assurance schemes. ISO/IEC 27001 was chosen due to its widespread international adoption and position as the de facto MSS for information security, and the frequent use as a baseline for other assurance schemes. However, there are mappings of ISO/IEC 27001 to other schemes, like Appendix H of [14], which is a mapping between the security controls of ISO/IEC 27001 to NIST 800-53, and then from NIST 800- 53 to ISO/IEC 15408 (Common Criteria).

Appendix B: Mapping of Assurance Techniques to Assurance Controls outlines the mapping between 20 assurance techniques and the 35 ISO/IEC 27001 (Annex A) control families. Assurance techniques within ISO/IEC 27001 broadly fall into two categories: First, those used or procured (from a third party) by a client (i.e., the auditee) which generate audit evidence. Second, those used by an auditor. In some cases, assurance techniques may bridge the two categories (e.g., for internal audits). It is important to clarify for the reader, that in standards such as ISO/IEC 27001, auditors are free to use any assurance technique

Perceptions of Assurance Techniques

Expert knowledge from 115 security professionals was gathered via an on-line survey focusing on economic-related variables, including experts’ perceptions of requirements (number of people, expertise, and time) and cost to conduct each assurance technique as well as effectiveness and complementary assurance techniques. Note that these variables can largely vary depending on the assurance target being assessed. Indeed, many of the techniques depend on the nature and size of the organization to be assessed, the environment and conditions of evaluation, etc.

In order to enable meaningful comparisons across techniques and with a view to maximising fairness of any such comparison, survey respondents where suggested to consider a commercial medium-size scenario for all assurance techniques as follows:

“For each assurance technique, assume a commercial target of medium size. Examples: company with 250 employees; infrastructure with 16 external IPs or 150 internal IPs; web application with one database and 100 static or dynamic pages; product like a Firewall, Router or Switch.”

Stakeholder Composition

Primary Role: Figure 5 shows the distribution of the different roles that the respondents of the survey have in their day to day jobs. As can be seen from the figure, 64% of respondents in our sample are Security Practitioners. This is an advantage for our research as the practitioners actually perform the assurance techniques, which are analyzed in this project, and have a fair idea about how they work and therefore have provided valuable insight from their point of view.

10% (^) 2% 3%

22% 63%

Auditor

Chief Information Security Officer Competence Assessor (e.g: for qualifications) Information Security Manager

Security Practitioner (e.g: a penetration tester, security architect) Figure 5: Primary Role of Survey Participants

Assurance Experience: Figure 6 shows the number of years respondents spent in the information security industry. Notably, 56,45% respondents have spent over 15 years in the security industry, and 91.81% over 5 years.

8%

15%

21%

31%

25%

0% 20% 40% 60% 80% 100%

<5 yrs

5-9 yrs

10-14 yrs

15-19 yrs

20+ yrs

Number of Years

Figure 6: Number of years spent in security industry

Assurance Schemes: Respondents were also asked about the assurance schemes they are involved in their day-to-day role. Figure 7 shows the results. As can be seen in the figure, we found a reasonably large variety of assurance schemes that the respondents are familiar with, covering most of the assurance schemes reviewed in this document.

19

20

22

21

24

25

54

87

0 20 40 60 80 100

CPA

CTAS

CAS

PCI DSS

PGA

CC

Cyber Essentials

ISO/IEC 27000 Series

Figure 7: Assurance Schemes

Individual Qualifications: Figure 8 shows the number of instances of each of the individual qualifications encountered. It is to be noted here that the total number of responses to this question is more than the total number of respondents, because respondents were allowed to choose multiple qualifications to be able to list all their qualifications.

2

5

6

9

16

21

43

44

76

81

0 20 40 60 80 100

PCI DSS

CISA

CEH

CISM

CREST

CHECK

ISO/IEC 27001 Auditor

CISSP

CCP

CLAS

Figure 8: Individual Qualifications

Confidence Level: Respondents were asked to select their level of confidence in the answers they provided for each of the assurance technique. The results are shown in Table 2. The respondents were able to select 3 levels, namely, Low, Medium and High. Architectural Reviews and Penetration Testing have been found to be the two assurance techniques where the highest proportion of respondents answered with High level of confidence (62% and 61% respectively).

Assurance Technique Confidence Level Total Low Med High Resp. Review of Policies, etc. 4% 40% 56% 72 Review of Client Forms 16% 53% 31% 64 Architectural Review - 38% 62% 64 Configuration Review 6% 55% 39% 56 Source Code Review 18% 47% 35% 49 Observation 12% 64% 24% 41 Interviews 9% 41% 50% 54 Red Team Exercises 7% 52% 41% 42 Penetration Tests 5% 34% 61% 56 Vulnerability Scan 7% 42% 51% 55 Social Engineering 25% 40% 35% 40 Threat Assessment 4% 46% 50% 54 Static Analysis 30% 67% 3% 30 Dynamic Analysis 28% 65% 7% 29 Fuzzing 41% 48% 11% 27 Formal Verification 16% 53% 31% 32 Cryptographic Validation 26% 52% 22% 31 Emanation Security Analysis 35% 54% 11% 26 Witnessed Test 10% 63% 27% 30 Public Review 46% 46% 8% 26

Table 2: Confidence of respondents in their input

Assurance Techniques Characteristics

Assurance Technique Number of People Total 1 2 3 4+ Resp. Review of Policies 54% 37% 8% 1% 73 Review of Client Forms 81% 13% 3% 3% 64 Architectural Review 74% 17% 6% 3% 64 Configuration Review 61% 30% 5% 4% 57 Source Code Review 43% 33% 10% 15% 49 Observation 61% 32% 5% 2% 41 Interviews 35% 56% 9% - 54 Red Team Exercises 11% 30% 28% 31% 43 Penetration Tests 18% 64% 16% 2% 56 Vulnerability Scan 80% 16% 4% - 55 Social Engineering 40% 42% 5% 13% 40 Threat Assessment 72% 22% 2% 4% 54 Static Analysis 70% 20% 7% 3% 30 Dynamic Analysis 62% 24% 7% 7% 29 Fuzzing 66% 26% - 8% 27 Formal Verification 31% 41% 13% 15% 32 Cryptographic Validation 58% 26% 7% 9% 31 Emanation Sec. Analysis 46% 46% 8% - 26 Witnessed Test 50% 33% 17% - 30 Public Review 48% 28% 8% 16% 25 Table 3: Number of people required

Number of People Required: The results are shown in Table 3. It can be seen from the results that most respondents believed that almost all the assurance techniques can be successfully performed for the scenario given with 2 people. Furthermore, a vast majority stated that Review of Client-Completed Self-Assessment

Forms, Architectural Reviews, Vulnerability Scans, Threat Assessment, and Static Analysis can be successfully performed for the scenario given with only 1 person.

A notable exception is Red Team Exercises where more than 50% of the respondents (59% to be exact) believe that it requires more than 2 people to complete this technique. For all other techniques, at least 50% of the respondents believe that at most 2 people are required for the technique to be completed for the given example scenario.

Expertise Required: Table 4 shows the results obtained regarding the level of expertise respondents thought was required to perform the particular assurance techniques successfully. Looking at the results, we find that different levels of expertise are required for different techniques in the type of scenario we described to the respondents. Techniques such as Architectural Review, Interviews, Threat Assessment and Cryptographic Validation seemingly require Senior professionals (72%, 66%, 61% and 61% respectively).

Assurance Expertise Required Total Technique P P(W) S Pr Resp. Review of Policies 33% 35% 32% - 72 Review of Client Forms 45% 26% 27% 2% 64 Architectural Review 8% 9% 72% 11% 64 Configuration Review 21% 46% 33% - 57 Source Code Review 19% 18% 45% 18% 49 Observation 27% 46% 22% 5% 41 Interviews 11% 16% 66% 7% 55 Red Team Exercises 9% 10% 50% 31% 42 Penetration Tests 12% 29% 52% 7% 56 Vulnerability Scan 44% 40% 16% - 55 Social Engineering 20% 40% 35% 5% 40 Threat Assessment 6% 20% 61% 13% 54 Static Analysis 27% 33% 40% - 30 Dynamic Analysis 21% 34% 45% - 29 Fuzzing 30% 33% 33% 4% 27 Formal Verification 12% 25% 47% 16% 32 Cryptographic Validation 6% 10% 61% 23% 31 Emanation Sec. Analysis 8% 46% 35% 11% 26 Witnessed Test 7% 37% 53% 3% 30 Public Review 36% 28% 24% 12% 25

Table 4: Expertise required to perform each technique — P: Practitioner; P(W): Practitioner with Supervision; S: Senior; Pr: Principal.

Another interesting observation is that some techniques are more likely to be performed by Practitioners if they are provided with supervision. Looking at the table, we find a big jump in the proportion of respondents who think that techniques like Configuration Review, Social Engineering and Emanation Security Analysis can be performed by Practitioners with supervision as compared to without supervision. This is an important aspect to consider as it has implications in terms of the resources required for the performance of the technique which would eventually contribute towards its cost. There also seems to be 3 assurance techniques that could be conducted most of the time by practitioners alone or with little supervision: Review of Client-Completed Self-Assessment Forms, Vulnerability Scans, and Public review.

Assurance Cost Total Technique Extremely Very Expensive Moderate Cheap Responses Expensive Expensive Review of Policies - - 14% 69% 17% 72 Review Client Forms - - 5% 36% 59% 64 Architectural Review - 5% 28% 58% 9% 64 Configuration Review - 2% 21% 67% 10% 57 Source Code Review 18% 20% 29% 29% 4% 49 Observation - - 17% 63% 20% 41 Interviews 2% 2% 25% 55% 16% 55 Red Team Exercises 2% 17% 52% 24% 5% 42 Penetration Tests 2% 10% 52% 34% 2% 56 Vulnerability Scan - 2% 20% 29% 49% 55 Social Engineering - 2% 23% 55% 20% 40 Threat Assessment - 4% 28% 57% 11% 54 Static Analysis - 3% 23% 64% 10% 30 Dynamic Analysis - - 35% 55% 10% 29 Fuzzing 4% 7% 15% 67% 7% 27 Formal 22% 25% 22% 31% - 32 Verification Cryptographic 13% 26% 29% 26% 6% 31 Validation Emanation 4% 23% 31% 34% 8% 26 Sec. Analysis Witnessed Test - 10% 40% 37% 13% 30 Public Review 4% 8% 15% 31% 42% 26

Table 7: Cost of the Assurance Techniques

Effectiveness: We asked the respondents to state how effective they thought the assurance techniques were “in achieving its objectives”. The results are shown in Table 6 and they show that most of the assurance techniques have at least Good effectiveness according the respondents (13 out of 20 techniques). However, there are 5 assurance techniques for which the majority (at least 50%) of respondents think that the effectiveness is Fair at best. These techniques are Review of Client-Completed Self-Assessment Forms, Static Analysis, Dynamic Analysis, Fuzzing and Public Review.

Penetration Tests are the only assurance technique for which the majority of the respondents (50%) feel that the effectiveness is Very Good. The two assurance techniques which have a considerable proportion of respondents rating the effectiveness as Excellent are Penetration Tests and Red Team Exercises. These can be considered to be the best perceived techniques in terms of effectiveness by the respondents in our sample.

The two assurance techniques which have a comparatively higher proportion of respondents who rated their effectiveness as Poor are Review of Client-Completed Self-Assessment Forms and Public Reviews. These are considered the least effective assurance techniques by the respondents.

Cost: Respondents could also express their opinion on the cost of conducting each assurance technique in the type of scenarios given. The results are shown in Table 7.

We find that Review of Client-completed Self-assessment Forms is considered to be by far the cheapest assurance

technique by a large majority of the respondents (59%) in the described scenario, followed by Vulnerability Scans, and Public Review.

There is also a group of assurance techniques, whose cost for the scenarios described is perceived to be moderate: Review of Documented Policies, Procedures and Processes, Architectural Review, Configuration Review, Observation, Interviews, Social Engineering, Threat Assessment, and Dynamic and Static Analysis.

There are 4 techniques which are considered to be at least expensive by over 60% of the respondents. These techniques are Source Code Review (67%), Red Team Exercises (71%), Penetration Testing (66%) and Formal Verification (69%).

Complementary Assurance Techniques

One of the primary objectives of this research was to identify assurance techniques which are complementary to each other, providing insights on which are the assurance techniques that are used together more often than others. To this aim, we asked the respondents of the on-line survey to list up to 3 complementary assurance techniques for every assurance technique they were familiar with, which when performed together could achieve high effectiveness.

Appendix C: Complementary Assurance Techniques contains all the details of the results obtained, reporting individual bar charts showing the number of complementary assurance techniques suggested by respondents for each of

the 25 assurance techniques studied. For the sake of clarity and brevity, only aggregated high-level results are reported here.

Most Commonly Chosen Techniques: Table 8 summarises the number of times each assurance technique was chosen by respondents as the first, second and third most complementary technique for other assurance techniques.

Assurance Technique 1st 2nd 3rd Total Review of Policies 4 5 4 13 Observation 3 2 2 7 Architectural Review 0 4 3 7 Interviews 3 1 2 6 Penetration Tests 3 0 2 5 Source Code Review 1 1 2 4 Static Analysis 3 0 0 3 Configuration Review 1 1 1 3 Vulnerability Scan 1 1 0 2 Dynamic Analysis 1 1 0 2 Review of Client Forms 0 0 2 2 Fuzzing 0 2 0 2 Witnessed Test 0 0 2 2 Threat Assessment 0 1 0 1 Formal Verification 0 1 0 1 Red Team Exercises 0 0 0 0 Social Engineering 0 0 0 0 Cryptographic Validation 0 0 0 0 Emanation Security Analysis 0 0 0 0 Public Review 0 0 0 0

Table 8: Most commonly chosen complementary techniques

Being chosen as the most common complementary technique can be interpreted as an added value to the utility of the assurance technique. If a particular assurance technique is a complementary technique for another assurance technique, the chances of it being included in different assurance schemes is higher. This supports the analysis in the previous section, which presented Review of Documented Policies, Procedures and Processes and Penetration Tests as the two most commonly used assurance techniques across various assurance schemes. The likelihood of their being chosen as complementary techniques may be a contributing factor of such widespread use across schemes.

Groups of Complementary Techniques: On further analysis of the individual distributions of the complementary techniques, 3 main clusters of assurance techniques have been identified.

1. Observation, Interviews and Review of Documented Policies, Procedures and Processes: Looking at Figure C1 in Appendix C: Complementary Assurance Techniques for Review of Documented Policies, Procedures and Processes, Figure C3 for Observation and Figure C4 for Interviews, we find that all these techniques are the top two most commonly selected complementary techniques of each

other. This suggests that these techniques have a higher chance of being performed together for assurance schemes. Looking at Appendix A, we find that all these three assurance techniques in both the PCI DSS and

ISO/IEC 27001 Case Study

2. Vulnerability Scans and Penetration Testing: Looking at Figure C6 in Appendix C: Complementary Assurance Techniques, we find that Vulnerability Scans (25) are the most common complementary technique for Penetration Tests. Similarly, we can see in Figure C7 that Penetration Tests (25) are the most common complementary assurance technique for Vulnerability Scans. Looking at Appendix A, we find that both these techniques are used in 5 assurance schemes, namely, ISO/IEC 27001 (though optionally), PCI DSS, Common Criteria, CTAS and CPA.
3. Static Analysis and Dynamic Analysis: From Figure C9 and C10 in Appendix C: Complementary Assurance Techniques, we see that Static Analysis and Dynamic Analysis are the most commonly chosen complementary assurance technique for each other. Looking at Appendix A, it seems none of assurance schemes reviewed uses these techniques.

Cost-Effectiveness of Assurance Techniques

Collected data on perceived cost and effectiveness obtained via the on-line survey was then used to derive a measure of cost-effectiveness. Details about this measure as well as all the calculations performed to get cost-effectiveness values for each assurance technique are in Appendix E: Cost-Effectiveness Calculations. Because of their difference in nature, it was decided to split the analysis between assurance techniques targeting security controls and assurance techniques targeting individual competences.

0.000 0.050 0.100 0.150 0.200 0.250 0.300 0.350 0.

Review of policies (72)

Review Client forms (64)

Arc>tectural review (64)

Configura>on review (57)

Source code review (49)

Observa>on (41)

Interview (55)

Red team exercise (42)

Penetra>on tes>ng (56)

Vulnerability scan (55)

Social engineering (40)

Threat assessment (54)

Sta>c analysis (30)

Dynamic analysis (29)

Fuzzing (27)

Formal verifica>on (32)

Cryptogtaphic valida>on (31)

Emana>on security analysis (26)

Witnessed test (30)

Public review (26)

Figure 9: Cost-effectiveness of assurance techniques.

Figure 9 depicts cost-effectiveness for each of the 20 analysed assurance techniques for security controls. Architectural review, penetration testing, and vulnerability scans were perceived to be the most cost-effective assurance

(i.e., Red Team Exercise; Penetration Tests; Reviewing Documented Policies, Procedures, and Processes; and, Vulnerability Scans) and “Comb. 11”, which includes Threat Assessment; Architectural Review; Interviews; and, Reviewing Documented Policies, Procedures, and Processes.

Cost-Effectiveness of Assurance Techniques for Individual Competences

Figure 12 depicts the overall effectiveness of techniques to assure individual competence to conduct the other assurance techniques described above based on the values respondents provided for the on-line survey. For each technique to assure individual competence, Figure 12 also includes the number of people who provided their perception of how effective each technique was.

0.000 0.100 0.200 0.300 0.400 0.500 0.600 0.700 0.

Virtual Lab Examina9on (74)

Oral Examina9on (Viva-­‐Voce) (93)

Paper Based Examina9on (Narra9ve form) (92)

Paper Based Examina9on (Mul9ple choice) (97)

Employment History and Qualifica9on Review (100)

Figure 12: Perceived effectiveness of competence ATs

Oral Examination (Viva-Voce) was perceived to be the most effective one, closely followed by Employment History and Qualification Review. However, the differences of these two with respect to Paper Based Examination (Narrative

form) and Virtual Lab Examination, though existing, were minimal. There was a more substantial difference with respect to Paper Based Examination (Multiple choice), which was clearly considered as the least effective technique to assure individual competence.

0 5 10 15 20 25

Comb. 1 (2)

Comb. 2 (1)

Comb. 3 (1)

Comb. 4 (4)

Comb. 5 (2)

Comb. 6 (2)

Comb. 7 (3)

Comb. 8 (2)

Comb. 9 (2)

Comb. 10 (2)

Comb. 11 (2)

Comb. 12 (1)

Comb. 13 (3)

Comb. 14 (9)

Comb. 15 (2)

Comb. 16 (1)

Comb. 17 (9)

Comb. 18 (4)

Comb. 19 (6)

Comb. 20 (1)

Comb. 21 (17)

Figure 13: Perceived cost-effectiveness of competence ATs

As part of the on-line survey, another question was also asked, this time about what combination of techniques would be the most cost-effective in assessing individual competence. The results are reported in Figure 13 and Table 13 provides the mapping of each label in Figure 13 with the corresponding combination. Most respondents (76 out of

115. selected the combination of Oral Examination (Viva- Voce) and Employment History and Qualification Review, which actually consists of a combination of the top two highest rated techniques in Figure 12.

Special Scenario: ICS Case Study

This study examines the application of assurance techniques within Industrial Control System (ICS) environments. To contextualise the opportunities and challenges of applying such techniques, interviews with ICS security practitioners (including CESG, penetration testing providers, and a non-academic research institute) were conducted to discover how ICS operators address security risks in practice. A framework for future improvement in ICS security is outlined from this review’s findings. Three phases of the ICS system development lifecycle are then examined (during product development; during procurement; once operational) to determine when and how the assurance techniques defined within this project can be applied, and what challenges are present in conducting such security assessments.

Critical infrastructure such as that of utility industries (e.g., oil and gas) is a frequently cited example of an ICSs, although their usage is considerably more diverse and widespread. Service industries (e.g., logistics), and manufacturing industries (e.g., aerospace) make heavy use of ICS technologies. The technologies that support ICSs are largely similar in concept, and in many cases, identical. The technological similarity can be further expanded to small- scale installation, such as Building Automation Systems, although they are not addressed here.

Automation Network

Enterpris e Network

Safety

Physical Processes

Basic Monitoring and Control

Area Supervisory Controls

Site Manufacturing Operations

Business Planning and Logistics

Enterprise Systems

Demilitarised Zone

0

1

2

3

4

5

Operational Sensitivity to Failure

Lowest

Highest

Operational Goals: CIA

Availability

Integrity

Confidentiality

Operational Goals: SRA

Availability

Reliability

Safety

Figure 14: A Conceptual Model of an ICS: Safety and Security Goals (Adapted from [5, 13])

At a conceptual level, an ICS can be seen as a series of layers, split into two areas (Figure 14). Layers 0-3 constitute the “automation network”. Present in layers 0-2 are safety systems, the sensors and actuators that monitor and manipulate physical processes, and the devices enforcing the intended logic of such processes. Multiple instances of layers 0-2 may exist, which may be geographically clustered or dispersed (e.g., a utility network may have many thousand “field sites”). In both cases, they have been conceptually labelled ”Cell Zones”. Layer 3 manages automation network

wide functions. Layer 3 systems capture and archive cell zone process data, monitor these processes, and take managerial action as necessary. Layers 4-5 are known as the “enterprise network”. Centralised IT services are found here (e.g., business-to-customer services). Both the automation network and enterprise network may be physically isolated from each other, in what is known as an “air gap” which can act as a security feature. However, these networks in contemporary ICSs are frequently interconnected, due to the potential to facilitate core business functions (e.g., to enable automation in a manufacturing system, through linking the consumer purchasing system to the production line).

Risk Management, Risk Perception and Standards for ICS Security

The use of appropriate assurance techniques within the risk management process is paramount, as it is the output of these techniques that influences the way that risk is perceived, assessed and treated in a cyclical fashion. Therefore, understanding the current practices and challenges of securing ICS environments must be contextualised, in order to understand the potential application for assurance techniques within ICS environments. Academic surveys of publications are available from alternative sources (e.g., [13]). Instead, this review intends to collate the perceptions and experiences about ICS security of those with experience of the realities of these environments.

Industry surveys such as that of ENISA [7] (EU-centric) and SANS [15] (US-centric), highlighted low utilisation of standards, with a greater preference for guidelines. Standards where used included: ISO/IEC 27002^3 ; ISA/IEC 62443; and NERC CIP. In both surveys, fulfilment criteria is not qualified as to the extent to which it constitutes (e.g., how close to achieving certification). Despite this, positive respondent count remained low, with 10-20% current implementation or utilisation and 10-45% planned. Such findings raise question around security risk management practices; more so if non-response bias is considered.

Survey results represent a snapshot in time, and may not reflect the current status. This study does not purport to be a comprehensive or quantitative reflection of what is; however, interviewed practitioners, with experiences of assessing many environments, expressed views that largely paralleled the findings of surveys: strict conformance to standards within automation networks was scarce, with verified compliance or certification only where there was a mandatory requirement for it. Notably this was (^3) ISO/IEC 27002 here is notable, as it outlines controls, rather than ISO/IEC 27001 which focuses on managing security risk.