Moto Documentation: A Comprehensive Guide to Mocking AWS Services for Testing, Schemes and Mind Maps of Logic

Download Moto Documentation: A Comprehensive Guide to Mocking AWS Services for Testing and more Schemes and Mind Maps Logic in PDF only on Docsity!

Moto Documentation

Release 2.2.18.dev

Steve Pulec

Dec 12, 2021

ii

A library that allows you to easily mock out tests based on AWS infrastructure.

GETTING STARTED 1

CHAPTER

ONE

GETTING STARTED

If you’ve never used moto before, you should read the Getting Started with Moto guide to get familiar with moto and its usage.

4 Chapter 1. Getting Started

Decorator

With a decorator wrapping, all the calls to S3 are automatically mocked out.

import boto from moto import mock_s from mymodule import MyModel

@mock_s def test_my_model_save(): conn = boto3.resource('s3', region_name='us-east-1')

We need to create the bucket since this is all in Moto's 'virtual' AWS account

conn.create_bucket(Bucket='mybucket')

model_instance = MyModel('steve', 'is awesome') model_instance.save()

body = conn.Object('mybucket', 'steve').get()[ 'Body'].read().decode("utf-8")

assert body == 'is awesome'

Context manager

Same as the Decorator, every call inside the with statement is mocked out.

def test_my_model_save(): with mock_s3(): conn = boto3.resource('s3', region_name='us-east-1') conn.create_bucket(Bucket='mybucket')

model_instance = MyModel('steve', 'is awesome') model_instance.save()

body = conn.Object('mybucket', 'steve').get()[ 'Body'].read().decode("utf-8")

assert body == 'is awesome'

Raw

You can also start and stop the mocking manually.

def test_my_model_save(): mock = mock_s3() mock.start()

conn = boto3.resource('s3', region_name='us-east-1') conn.create_bucket(Bucket='mybucket')

model_instance = MyModel('steve', 'is awesome') (continues on next page)

6 Chapter 2. Additional Resources

(continued from previous page) model_instance.save()

body = conn.Object('mybucket', 'steve').get()[ 'Body'].read().decode("utf-8")

assert body == 'is awesome'

mock.stop()

Unittest usage

If you use unittest to run tests, and you want to use moto inside setUp , you can do it with .start() and .stop() like:

import unittest from moto import mock_s import boto

def func_to_test(bucket_name, key, content): s3 = boto3.resource('s3') object = s3.Object(bucket_name, key) object.put(Body=content)

class MyTest(unittest.TestCase): mock_s3 = mock_s3() bucket_name = 'test-bucket' def setUp(self): self.mock_s3.start()

you can use boto3.client('s3') if you prefer

s3 = boto3.resource('s3') bucket = s3.Bucket(self.bucket_name) bucket.create()

def tearDown(self): self.mock_s3.stop()

def test(self): content = b"abc" key = '/path/to/obj'

run the file which uploads to S

func_to_test(self.bucket_name, key, content)

check the file was uploaded as expected

s3 = boto3.resource('s3') object = s3.Object(self.bucket_name, key) actual = object.get()['Body'].read() self.assertEqual(actual, content)

It is possible to use Moto as a class-decorator. Note that this may behave differently then you might expected - it currently creates a global state on class-level, rather than on method-level.

2.1. Getting Started with Moto 7

Example on usage

If you are a user of pytest, you can leverage pytest fixtures to help set up your mocks and other AWS resources that you would need.

Here is an example:

@pytest.fixture(scope='function') def aws_credentials(): """Mocked AWS Credentials for moto.""" os.environ['AWS_ACCESS_KEY_ID'] = 'testing' os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing' os.environ['AWS_SECURITY_TOKEN'] = 'testing' os.environ['AWS_SESSION_TOKEN'] = 'testing'

@pytest.fixture(scope='function') def s3(aws_credentials): with mock_s3(): yield boto3.client('s3', region_name='us-east-1')

In the code sample above, all of the AWS/mocked fixtures take in a parameter of aws_credentials , which sets the proper fake environment variables. The fake environment variables are used so that botocore doesn’t try to locate real credentials on your system.

Next, once you need to do anything with the mocked AWS environment, do something like:

def test_create_bucket(s3):

s3 is a fixture defined above that yields a boto3 s3 client.

Feel free to instantiate another boto3 S3 client -- Keep note of the region though.

s3.create_bucket(Bucket="somebucket")

result = s3.list_buckets() assert len(result['Buckets']) == 1 assert result['Buckets'][0]['Name'] == 'somebucket'

What about those pesky imports

Recall earlier, it was mentioned that mocks should be established BEFORE the clients are set up. One way to avoid import issues is to make use of local Python imports – i.e. import the module inside of the unit test you want to run vs. importing at the top of the file.

Example:

def test_something(s3): from some.package.that.does.something.with.s3 import some_func # <-- Local import␣ ˓→for unit test

^^ Importing here ensures that the mock has been established.

some_func() # The mock has been established from the "s3" pytest fixture, so this␣ ˓→function that uses

a package-level S3 client will properly use the mock and not reach␣

˓→out to AWS.

2.1. Getting Started with Moto 9

Patching the client or resource

If it is not possible to rearrange imports, we can patch the boto3-client or resource after the mock has started. See the following code sample:

The client can come from an import, an init-file, wherever..

client = boto3.client("s3") s3 = boto3.resource("s3")

@mock_s def test_mock_works_with_client_or_resource_created_outside(): from moto.core import patch_client, patch_resource patch_client(outside_client) patch_resource(s3)

assert client.list_buckets()["Buckets"] == []

assert list(s3.buckets.all()) == []

This will ensure that the boto3 requests are still mocked.

Other caveats

For Tox, Travis CI, and other build systems, you might need to also perform a touch ~/.aws/credentials command before running the tests. As long as that file is present (empty preferably) and the environment variables above are set, you should be good to go.

2.2 Non-Python SDK’s / Server Mode

Moto has a stand-alone server mode. This allows you to use Moto with any of the official AWS SDK’s.

Install the required dependencies using:

pip install moto[server]

You can then start it like this:

$ moto_server

You can also pass the port:

$ moto_server -p

• Running on http://127.0.0.1:3000/

If you want to be able to use the server externally you can pass an IP address to bind to as a hostname or allow any of your external interfaces with 0.0.0.0:

$ moto_server -H 0.0.0.

• Running on http://0.0.0.0:5000/

Please be aware this might allow other network users to access your server.

To use Moto in your tests, you can pass the endpoint_url -parameter to the SDK of your choice.

10 Chapter 2. Additional Resources

2.2.2 Use ServerMode using the decorators

It is possible to call the MotoServer for tests that were written using decorators. The following environment variables can be set to achieve this:

TEST_SERVER_MODE=true

Whenever a mock-decorator starts, Moto will:

1. Send a reset-request to http://localhost:5000, removing all state that was kept
2. Add the endpoint_url parameter to boto3, so that all requests will be made to http://localhost:5000.

Calling the reset-API ensures the same behaviour as normal decorators, where the complete state is removed. It is possible to keep the state in between tests, using this environment variable:

MOTO_CALL_RESET_API=true

2.2.3 Dashboard

Moto comes with a dashboard to view the current state of the system:

http://localhost:5000/moto-api/

2.2.4 Reset API

An internal API endpoint is provided to reset the state of all of the backends. This will remove all S3 buckets, EC servers, etc.:

requests.post("http://motoapi.amazonaws.com/moto-api/reset")

2.2.5 Install with Homebrew

Moto is also available to install using Homebrew, which makes it much easier to manage if you’re not using Python as your primary development language.

Once Homebrew is installed, you can install Moto by running:

brew install moto

To make the Moto server start up automatically when you log into your computer, you can run:

brew services start moto

12 Chapter 2. Additional Resources

2.2.6 Caveats

The standalone server has some caveats with some services. The following services require that you update your hosts file for your code to work properly:

1. s3-control

For the above services, this is required because the hostname is in the form of AWS_ACCOUNT_ID.localhost. As a result, you need to add that entry to your host file for your tests to function properly.

2.3 FAQ

2.3.1 Is Moto concurrency safe?

No. Moto is not designed for multithreaded access/multiprocessing.

2.3.2 Why am I getting RUST errors when installing Moto?

Moto has a dependency on the pip-module cryptography. As of Cryptography >= 3.4, this module requires Rust as a dependency. Most OS/platforms will support the installation of Rust, but if you’re getting any errors related to this, see the cryptography documentation for more information: https://cryptography.io/en/latest/installation/#rust

2.4 Boto vs Boto

Boto3 is the latest Python SDK, and as such the SDK targeted by Moto. All our @mock_ -decorators should be usable against any boto3-version.

Still stuck on boto, the former SDK? Moto does have some support, in the form of our deprecated services:

from moto import mock_ec2_deprecated import boto

@mock_ec2_deprecated def test_something_with_ec2(): ec2_conn = boto.ec2.connect_to_region('us-east-1') ec2_conn.get_only_instances(instance_ids='i-123456')

When using both boto2 and boto3, one can do this to avoid confusion:

from moto import mock_ec2_deprecated as mock_ec2_b from moto import mock_ec

If you want to use Server Mode, the easiest way is to create a boto config file ( ~/.boto ) with the following values:

[Boto] is_secure = False https_validate_certificates = False proxy_port = 5000 proxy = 127.0.0.

2.3. FAQ 13

2.6 AWS Config Support

An experimental feature for AWS Config has been developed to provide AWS Config capabilities in your unit tests. This feature is experimental as there are many services that are not yet supported and will require the community to add them in over time. This page details how the feature works and how you can use it.

2.6.1 What is this and why would I use this?

AWS Config is an AWS service that describes your AWS resource types and can track their changes over time. At this time, moto does not have support for handling the configuration history changes, but it does have a few methods mocked out that can be immensely useful for unit testing.

If you are developing automation that needs to pull against AWS Config, then this will help you write tests that can simulate your code in production.

2.6.2 How does this work?

The AWS Config capabilities in moto work by examining the state of resources that are created within moto, and then returning that data in the way that AWS Config would return it (sans history). This will work by querying all of the moto backends (regions) for a given resource type.

However, this will only work on resource types that have this enabled.

Current enabled resource types

1. S3 (all)
2. IAM (Role, Policy)

2.6.3 Developer Guide

There are several pieces to this for adding new capabilities to moto:

1. Listing resources
2. Describing resources

For both, there are a number of pre-requisites:

Base Components

In the moto/core/models.py file is a class named ConfigQueryModel. This is a base class that keeps track of all the resource type backends.

At a minimum, resource types that have this enabled will have:

1. A config.py file that will import the resource type backends (from the init.py )
2. In the resource’s config.py , an implementation of the ConfigQueryModel class with logic unique to the resource type
3. An instantiation of the ConfigQueryModel
4. In the moto/config/models.py file, import the ConfigQueryModel instantiation, and update RESOURCE_MAP to have a mapping of the AWS Config resource type to the instantiation on the previous step (just imported).

2.6. AWS Config Support 15

An example of the above is implemented for S3. You can see that by looking at:

1. moto/s3/config.py
2. moto/config/models.py

Testing

For each resource type, you will need to test write tests for a few separate areas:

• Test the backend queries to ensure discovered resources come back (ie for IAM::Policy , write tests.tests_iam.test_policy_list_config_discovered_resources ). For writing these tests, you must not make use of boto to create resources. You will need to use the backend model methods to provision the resources. This is to make tests compatible with the moto server. You must make tests for the resource type to test listing and object fetching.
• Test the config dict for all scenarios (ie for IAM::Policy , write tests.tests_iam.test_policy_config_dict ). For writ- ing this test, you’ll need to create resources in the same way as the first test (without using boto ), in every meaningful configuration that would produce a different config dict. Then, query the backend and ensure each of the dicts are as you expect.
• Test that everything works end to end with the boto clients. (ie for IAM::Policy , write tests.tests_iam.test_policy_config_client ). The main two items to test will be the boto.client(‘config’).list_discovered_resources() , boto.client(‘config’).list_aggregate_discovered_resources() , moto.client(‘config’).batch_get_resource_config() , and moto.client(‘config’).batch_aggregate_get_resource_config(). This test doesn’t have to be super thorough, but it basically tests that the front end and backend logic all works together and returns correct resources. Beware the aggregate methods all have capital first letters (ie Limit ), while non-aggregate methods have lowercase first letters (ie limit )

Listing

S3 is currently the model implementation, but it also odd in that S3 is a global resource type with regional resource residency.

But for most resource types the following is true:

1. There are regional backends with their own sets of data
2. Config aggregation can pull data from any backend region – we assume that everything lives in the same account

Implementing the listing capability will be different for each resource type. At a minimum, you will need to return a List of Dict that look like this:

[ { 'type': 'AWS::The AWS Config data type', 'name': 'The name of the resource', 'id': 'The ID of the resource', 'region': 'The region of the resource -- if global, then you may want to have the␣ ˓→calling logic pass in the aggregator region in for the resource region -- or just us-east-1 :P' } , ... ]

16 Chapter 2. Additional Resources