Automated Test Framework (ATF) for ServiceNow: Development, Components, and Best Practices, Exams of Advanced Education

Download Automated Test Framework (ATF) for ServiceNow: Development, Components, and Best Practices and more Exams Advanced Education in PDF only on Docsity!

ATF v2 FINAL EXAM QUESTIONS AND

ANSWER

Goals guiding ATF development: - - Increase ServiceNow customer upgrade value by reducing the time required for instance validation

• Drive quality by performing instance verification in exactly the same way, as often as needed
• Make the framework user interface independent and upgrade safe, all while requiring no extra licensing cost
• Manage risk by providing more productive upgrades, allowing customers to stay current Benefits of the Automated Test Framework - - Increased productivity
• Ease of use
• Better test accuracy Benefits of the Automated Test Framework: Increased productivity - - ATF can provide coverage for most manual test cases, reducing ServiceNow upgrade cycles and accelerating time to production.
• Regression testing can be run by a single system administrator without specific business process expertise.
• Reusable test templates can also be created to expedite development of commonly performed testing patterns. Benefits of the Automated Test Framework: Ease of use - - ATF is designed for both technical and non-technical users.
• Although scripting is supported for complex tests, ATF is targeted as a no-code or low- code solution.
• No additional tools or software is required to use ATF. Benefits of the Automated Test Framework: Better test accuracy - - ATF provides the ability to create reusable testing plans to reduce manual efforts.
• Tests are developed based on reliable best practices while avoiding inconsistent common user testing errors. Types of testing targeted by ATF - - Functional business logic testing
• Regression testing
• Browser compatibility testing
• Server-side Jasmine testing Types of testing targeted by ATF: Functional business logic testing - - verifies a process is fit for its intended purpose and meets its defined acceptance criteria
• can provide functional testing across the Now Platform. It supports delivered applications, all scoped applications, Service Portal, custom tables, and custom forms.

• testing can be performed on the user interface or directly against instance data on the server. Types of testing targeted by ATF: Regression testing - - create comprehensive testing plans to ensure existing processes still work as expected following an upgrade or development cycle
• run the same comprehensive testing plan before and after major changes to get a baseline
• provides assurance that the same tests will run in the same manner, allowing routine instance checks to be performed whenever necessary Types of testing targeted by ATF: Browser compatibility testing - - targets compatibility testing across multiple browsers and operating systems
  • write an ATF test once, and then verify the same functionality works as expected for all supported environments. Types of testing targeted by ATF: Server-side Jasmine testing - - Jasmine => an open source testing framework for JavaScript
• Jasmine tests are called to verify server-side functionality from the steps of an ATF test. Main areas where Cloud Dimensions pulls in manual testing and other third-party testing tools include: - - Load testing
• Performance testing
• Report testing
• Email format testing *** Match the following testing challenges to their corresponding ATF solution CHALLENGES
• Exposure to project risk increased when not current on new ServiceNow releases
• Automated tests created by third-party tools can break when the ServiceNow UI changes
• Consistent repeatability across manual testers cannot be managed well
• Upgrade cots are driven up by the resources spent on manual testing ATF SOLUTION
• ATF manages risk with more productive upgrades, allowing you to stay current
• ATF increases upgrade value by reducing the reliance on manual testing
• ATF verifies your instance in exactly the same way, as often as needed
• The ATF framework is user interface independent and upgrade safe - - Automated tests created by third-party tools can break when the ServiceNow UI changes => The ATF framework is user interface independent and upgrade safe

• Consistent repeatability across manual testers cannot be managed well => ATF verifies your instance in exactly the same way, as often as needed
• Upgrade cots are driven up by the resources spent on manual testing => ATF increases upgrade value by reducing the reliance on manual testing
• Exposure to project risk increased when not current on new ServiceNow releases => ATF manages risk with more productive upgrades, allowing you to stay current *** What types of testing is ATF designed to cover? Select all that apply?
• Browser compatibility testing for the application
• Regression testing following an update
• Business logical functional testing to meet unique criteria
• User Acceptance Testing (UAT) for sign-off approval - - Browser compatibility testing for the application
• Regression testing following an update
• Business logical functional testing to meet unique criteria The important elements used to develop ATF testing include: - - Test suites
• Tests
• Test steps Test suite - groups tests together that are typically performed at the same time Test suites - - represent a grouping of one or more tests or other test suites.
• The hierarchy of child test suites can be to any level.
• The same test can be included within multiple test suites.
• However, a test suite can only be assigned one parent test suite. Test - represents a set of actions and assertions to verify an expected end result
• Each action or assertion within the test is represented by an individual test step.
• Although there is no limit to the number of test steps contained within a test, it is typically better to create smaller focused tests, rather than larger broad tests. Client and server - - Some of the test steps within a test are performed on the client user interface, while others are performed on the server.
• A single test can include both client and server actions and assertions.
• Test steps performed on the client simulate user interactions, such as opening a form.
• Test steps performed on the server execute in the background and cover a wide range of possibilities.

Create User - - dynamically adds a new user, assigns it specified roles, and then impersonates the user.

• once a test completes, the user created is removed. To help your tests remain self- contained without dependencies to specific users and roles

• recommended to use Create a User at the the beginning of a test, rather than Impersonate from New York forward. Test step - - represents a single action or assertion within a test

• ex: impersonating a user, opening a form, or validating a field value Different types of test steps are created using different test step configurations. A test step configuration: - defines the behavior and characteristics of each type of action or assertion. Test Designer - - most common ATF role on a ServiceNow project
• can create and run tests and test suites
• most application developers are assigned the role since ATF tests should be created for all important customizations.

• Although coding skills are not mandatory, debugging tests takes an advanced level of ServiceNow and ATF knowledge. Coding skills required: Low-code or No-code Test Designers can perform the following capabilities: - - Create, edit, and delete tests suites

• Create, edit, and delete tests
• Add test steps to tests using existing test step configurations (new test step configurations can only be created by a Test Administrator)
• Run tests and test suites
• View test results
• View ATF system properties Test Administrators - - ensure ATF system properties are set appropriately to control how the ATF environment operates within an instance
• also define retention policies to determine how long testing results should be kept.
• large % of the time required involves creating new test step configurations => once created, test step configurations can be reused by any ATF role.
• The more new test step configurations required by the project, the more impact there will be on the Test Administrator.
• Since new test step configurations are created using script => should have advanced coding experience.

• Due to their experience, they are often pulled in to help Test Designers debug failing tests or create new ones. Test Administrators can perform all the capabilities of a Test Designer, plus the following: - - Edit ATF system properties
• Create test step configurations Coding skills required: Pro-code Web Service Testers - - build web service tests
• role provides access to the web service modules required to facilitate test development. Typically, web service tests make use of REST integration. However, technologies like SOAP can be used through scripting.
• number of Web Service Testers required => dependent on amount of web service integration customization implemented
• Since web service integrations might require the use of script => should have advanced coding experience.
• Due to their experience, they are often pulled in to help Test Designers debug failing tests or create new ones Web Service Testers can perform all the capabilities of a Test Designer, plus the following: - - Build web service tests Coding skills required: Pro-code ATF Development Process - 1. Identify essential functionality per process / application

2. Create tests to validate essential functions
3. Package Tests into Suites
4. Run or schedule test execution
5. Monitor and fix issues ATF Development Process:
6. Identify essential functionality per process / application - - focus on functionality UNIQUE to SN instance

• when determining what is essential, consider exposure to risk
• weight detail of validation against on-going test maintenance required ATF Development Process:

2. Create tests to validate essential functions - - begin test creation once changes to new feature are minimal

• if develop new functionality and ATF tests in tandem, realize any iterative changes could result in additional ATF development effort ATF Development Process:

3. Package Tests into Suites - - organize Tests into Test Suites that can be run together

• Use existing Test Suite or create a new one
• Same Test can be grouped into multiple Test Suites ATF Development Process:

4. Run or schedule test execution - - ATF Test Suites can be run on-demand OR scheduled

• ATF Tests are ONLY run ON-DEMAND
• Test Steps within a test are run on the client or the server
• Test Steps performed on client UI are executed by a test client runner within a separate browser window
• Test Steps performed on the server execute against the instance ATF Development Process:

5. Monitor and fix issues - - once test runs, review test results

• detailed test results are provided at the Test Suite, Test, and Test Step level
• Runtime screenshots are included in test results to help monitor and debug any issues Starting in the New York release, tests can run - at the same time as other tests. However, tests run within a test suite still run sequentially, one after the other. Running ATF Test Suites: Values passed between test steps - - output values from one test step can be used as input values to another test step WITHIN the SAME test
• output values CAN NOT be used as input values for test steps WITHIN OTHER tests Running ATF Test Suites: Each test is isolated - - each ATF test runs INDEPENDENT and ISOLATED from any other test
• values from one test cannot be transferred as values to another test
• test results are the only thing persisted after tests run Running ATF Test Suites: Impersonate users are unimpersonated - - ATF tests can impersonate a user doing the work
• impersonated user is automatically unimpersonated later during the test Running ATF Test Suites: Test data is rolled back - Once each ATF has completed, any instance changes made during the individual test are rolled back
• new records are deleted
• deleted records are restored
• updated records are reverted to previous values The ATF framework is not meant to run in ServiceNow __________ instance environments - production ATF tests and test suites should only be run in non-production instances. Verifying production instances involves User Acceptance Testing (UAT) performed by stakeholders

Some of the important reasons ATF is prevented from running on their production instance include: - - Running ATF tests slows down instance performance

• Changes made within an ATF test would be visible to users before the rollback
• Actionable events, such as email notifications to customers or approvals to managers, would be triggered from data changed by ATF tests
• Sensitive production data could be seen when impersonating a user
• Audit history of data changes would remain even after data rollback *** Match the ATF components to their corresponding definitions. ATF COMPONENTS
• Test Suite
• Test Step
• Test Step Configuration
• Test
• Client Test Runner DEFINITIONS
• Simulates UI actions or assertions within a separate browser window when a test is run
• Definition of the behavior and characteristics for a type of action or assertion
• A single action or assertion with configured input values
• A set of ordered actions or assertions, whose changes rollback after a run
• A set of tests and test suites that can be scheduled or run on-demand - - Client Test Runner => Simulates UI actions or assertions within a separate browser window when a test is run
• Test Step Configuration => Definition of the behavior and characteristics for a type of action or assertion
• Test Step => A single action or assertion with configured input values
• Test => A set of ordered actions or assertions, whose changes rollback after a run
• Test Suite => A set of tests and test suites that can be scheduled or run on-demand *** What are the only ServiceNow instances within a deployment stack that should run ATF?
• Only non-production
• Only test and production
• Any instance environment
• Only development - - Only non-production Significant factors that increase ATF test development effort: - - Level of test detail
• Quality of existing development
• Number of client-side tests
• Uniqueness of each test

Factors that are not significant to the increase ATF test development effort: - - Number of steps

• Number of test cases Level of test detail - SIGNIFICANT FACTOR
• can add time and complexity. Find the right balance between detailed testing and ATF maintenance. Think about:
• How frequently will existing functionality be updated?
• Who will build or update the tests? Quality of existing development - SIGNIFICANT FACTOR
• can have an impact if functionality requires redesign in order for ATF tests to be built Number of client-side tests - SIGNIFICANT FACTOR
• typically take longer to execute than server-side tests. Therefore, they also take longer to unit test. Think about:
• Is the same user interface getting tested multiple times?
• How can I test this on the server, if the client user interface has been tested? Number of test steps - NOT A SIGNIFICANT FACTOR
• does not necessarily translate into test development impact.
• ATF tests can simply be copied and modified for similar test cases. Uniqueness of each test - SIGNIFICANT FACTOR
• MOST significant indicator of ATF development effort.
• The more unique a test is, the longer it will take to develop.
• Every unique test must be carefully designed and built, one step at a time. However, similar tests can be developed quickly by copying another. Think about:
• How many unique test cases do you have?
• Are the test steps different?
• Are the expected outputs different?
• Are the tables and forms different? Number of test cases - NOT A SIGNIFICANT FACTOR
• does not necessarily translate into test development impact. If many of the cases are similar, ATF tests can be copied and modified.
• uniqueness of tests is a more significant indicator. Think about:
• Do you have a few test cases covering a lot of different functionality (high uniqueness)?

• Do you have a lot of test cases covering a small amount of functionality (low uniqueness)? ServiceNow quick start tests - - sample baseline ATF tests
• delivered by ServiceNow beginning with the Madrid release
• focus on the same essential test cases required most frequently by customers Quick start tests: - - Represent a small set of critical use cases
• Install easily with the demo data of a ServiceNow application
• Can be used as templates to build and customize test and test suites matching your unique processes
• Run against your instance specific data once copied and configured
• Annotate your copy of the test if ServiceNow ever changes the original
• Quick start tests are inactive, read-only templates.
• They only produce a pass result when run with the demo data provided by their corresponding ServiceNow plugin.
• To allow quick start tests to run successfully against your instance data, you must - first create a copy of the quick start test. Once the quick start test is copied, you can apply any desired changes and run the test. *** ServiceNow delivers baseline ATF tests focusing on the same essential test cases required most frequently by customers. These tests can be copied and configured to match the unique business process of the customer. What are these tests called?
• Start smart tests
• Quick start tests
• Jump start tests
• Quick jump tests - - Quick start tests Steps to enable tests and test suites to run on your personal developer instance: - 1. Select "Automated Test Framework" > "Tests," then open any test

2. In upper-right corner of test form, notice 'Run Test" button does not appear => good indicator that ATF tests and test suits are not enabled on instance
3. Navigate to Automated Test Framework properties by selecting Enable tests and test suites in the blue highlighted annotation at the top of the form. You can also navigate to Automated Test Framework properties by selecting "Automated Test Framework" > "Administration" > "Properties in the application navigator."
4. At the top of the Automated Test Framework properties form, select "Yes" to Enable test/test suite execution. This allows ATF tests and test suites to run on the instance on- demand.

5. Select "Yes" to Enable schedule test suite execution. This allows ATF test suites to be scheduled, rather than just run on-demand. You will require this property to be enabled for the ATF: Schedule Test Suites course.
6. Select "Save" to update the new property settings on the instance. 7 Select "Automated Test Framework" > "Tests," and then open any active test. 8 In the upper-right corner of the test, notice the "Run Test" button now appears. This indicates ATF tests and test suites are now enabled to run on the instance. Tests run a series of test steps in an ___________ order defined by the ___________ - execution, Test Designer Tests execute until one test step fails or all test steps pass. A test step failure rolls up to the test. If one of the test steps fail... - the test fails If a previous test step fails... - all subsequent test steps are skipped Tests can impersonate a user performing the work. Unimpersonation occurs at the end of a test run. If a test is cancelled before it finishes running... - the last impersonation will still be active Once a test completes, only its ____________________ are persisted. The ___________________________ variables of each test step are not retained. - test results, test data and outputs Running ATF tests in parallel using a ServiceNow Developer Portal personal instance.. . - is NOT possible due to predefined instance configurations that cannot be adjusted Tables excluded from rollback by ServiceNow - Any records modified in the following tables are excluded from the rollback following a test run. Emails and reports are typically excluded, so their content and styling can be checked manually.

• History [sys_history_line]
• ECC Queue [ecc_queue]
• Email [sys_email]
• Email Log [sys_email_log]
• Report Execution [report_executions]
• Report Stats [report_stats]
• or any table extending these tables Tables excluded from rollback by you - Additional tables can be excluded from the rollback after a test run by adding an excludedFromRollback=true Attribute to their Collection record in "System Definition" > "Dictionary".

Asynchronous data changes - - Some asynchronous changes are not always finished in time to be included within the rollback following a test run.

• This would include things like event processing, email notifications, and workflow updates. Do not rely on the rollback of asynchronous changes to complete before the next test is run. If you need to make instance updates while tests are running there is a way to keep the two from impacting each other in the same browser session... - - Just use a different browser application to perform the updates => such as Firefox if tests are running in Chrome
• Another option is to make the updates using the "incognito" feature of many browsers. *** Which of the following updates might be rolled back after a test run completes?
• Record changes to a table with a Dictionary attribute of excludeFromRollback = true
• Record changes made asynchronously
• Record changes to a table that extends ECC Queue [ecce_que]
• Record changes to the Email [sys_email] table - - Record changes made asynchronously *** What determines how the execution of a test is divided into batches? Select all that apply.
• Test step execution order
• Load balancing
• If test steps are executed on the client
• If test steps are executed on the server - - Test step execution order
• If test steps are executed on the client
• If test steps are executed on the server Match each element included in Test Results to its proper description. DESCRIPTION
• Show server transaction log
• Ensure Test Results are kept longer than 30 days
• Provides a quick way to isolate the failure point
• Show browser console log for UI
• Show execute frame contents for Test Steps ELEMENT
• Retain indefinitely
• Test Step Results
• Test Log
• Test Transactions
• Screenshots - - Ensure Test Results are kept longer than 30 days => Retain indefinitely

• Provides a quick way to isolate the failure point => Test Step Results
• Show browser console log for UI => Test Log
• Show server transaction log =>Test Transactions
• Show execute frame contents for Test Steps => Screenshots When viewing the list of waiting and running test suites, keep in mind multiple test suites can run at the same time. However, the tests included in any of the test suites... - still only run one at a time. Therefore, if a parent test suite is run, its child test suites also appear in the list as running.