Using AWS CLI (and AWS SDK)

• The AWS CLI is an open source tool built using the AWS SDK for Python (Boto3) that provides commands to interact with AWS services. With minimal configuration, you can use all the features provided by the AWS management console from your favorite terminal.

• AWS SDKs provide an API for different programming languages (Python, Java, JavaScript, C++, .NET, GO, PHP, Ruby,…) to programmatically build and use AWS services.

In this article, we will see some tips to make the best use of these 2 tools.

• CLI AWS Installation
• AWS CLI Profiles
• Temporary session
• Priorities of Credentials
  • For AWS CLI
  • For AWS SDK
  • Good Practices
• Signing HTTP Request
• Debugging
  • Policy Simulator
  • Dry Run
  • Message
• AWS EC2 Instance Metadata
  • Examples of use
    • Data Types
    • Services
    • Some instance metadata
• AWS SDKs
• AWS Limits and Backoff
  • Limits / Quotas
  • Exponential Backoff

CLI AWS Installation

AWS CLI is available in 2 versions:

• Version 2: the latest and supports the latest features
• Version 1: original version, should no longer be used

To install AWS CLI Version 2 on Docker, Linux, macOS or Windows, refer to the AWS documentation https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

After a successful installation, you should be able to run the following commands:

% which aws
/usr/local/bin/aws

% aws --version
aws-cli/2.2.16 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off

AWS CLI Profiles

It is possible to register multiple user accounts in AWS CLI. This is called Profiles.

To do this, once your default account is configured, you can add a new one by executing the command:

% aws configure --profile <name-of-other-profile>

Now you can run an AWS CLI command under this new Profile by adding the parameter:

% aws iam get-user --profile <name-of-other-profile>

Temporary session

When an AWS account is protected by an MFA code, it is possible to create a temporary session using AWS Security Token Service (AWS STS) to request temporary identifiers with limited privileges.

To do this, run the following command:

% aws sts get-session-token --serial-number <arn-of-your-mfa-device> --token-code <token-code-from-your-mfa-device> --duration-seconds <session-duration-in-seconds>

{
    "Credentials": {
        "AccessKeyId": "ASIH9FS4N0N76VVTDFDG5C",
        "SecretAccessKey": "4h0f4j88H9L/HT4I0xjvE9-jrf72jkISjklbjEvRHfn",
        "SessionToken": "EJu6g3JpZ2luX2VjEDoaCWV.../...wvA5a5mfFWUOrH62fDFYbmUW5j31k1r/igIgP14W374njGO+mbO5+MeYKGoaLvHAC0SFdXh9sjopz2wq7gEIQxABGgw3OTc1MTc4.../...k+4b6sGGym82jlaPI=",
        "Expiration": "2021-07-26T10:10:59+00:00"
    }
}

You get new Credentials, valid, the time of the session.

Priorities of Credentials

It is possible to set Credentials in several places. There is therefore a priority order that must be known to fully understand the adverse effects that this can cause.

For AWS CLI

1. Passed in command line (–region, –output, –profile)
2. Passed in ENVIRONMENT VARIABLES (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
3. Saved in the /. aws/credentials file generated by the aws configure command
4. Saved in the /. aws/config­ file generated by the aws configure command
5. Recorded in Container Credentials (for ECS Tasks)
6. Saved in the EC2 Instances Profiles

For AWS SDK

1. Passed in Language System Properties
2. Passed in ENVIRONMENT VARIABLES (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
3. Saved in the default /. aws/credentials file in many SDKs
4. Recorded in Container Credentials (for ECS Tasks)
5. Saved in the EC2 Instances Profiles

Good Practices

To avoid any pitfalls with Credentials, here are some rules to follow:

1. NEVER save Credentials in the code!!
2. But rather define the Credentials in the best place in the priority chain:
   • If you are inside AWS, use Roles IAM (EC2 Instance Roles, ECS Roles, Lambda Roles)
   • If you are outside the AWS network, use environment variables or Profiles

Signing HTTP Request

When using AWS CLI or AWS SDK, HTTP calls to AWS are automatically signed. The protocol used is called Signature Version 4 (SigV4) and comes from AWS.

It comes in two possible forms:

• HTTP headers (Authorization header)
• URL parameters (Query string)

Debugging

Here are some tools that will help you debug and understand what’s going on if there are problems:

Policy Simulator

It may be useful to check the access rights to an AWS resource based on a User, Group or Role. There is a tool in AWS that can run these tests, the Policy Simulator: https://policysim.aws.amazon.com/

Dry Run

It may also be useful to test an AWS CLI command by simulating its execution. AWS CLI commands have an option for this:

• Example of simulated creation of an EC2 instance:

% aws ec2 run-instances --dry-run --region eu-west-3 --image-id ami-062fdd189639d3e93 --instance-type t2.micro

An error occurred (DryRunOperation) when calling the RunInstances operation: Request would have succeeded, but DryRun flag is set.

Because the command is run in dry-run mode, if successful, it returns DryRunOperation. If failed, it would return UnauthorizedOperation.

Message

Some AWS CLI commands return a encoded authorization message describing the problem. This message must be decoded to be understandable.

To do this, you can use the command:

% aws decode-authorization-message --encoded-message <encoded-message>

AWS EC2 Instance Metadata

Instances Metadata are data relating to an EC2 instance: they are accessible from the instance and allow you not to have to use IAM Role since this data has already been loaded into the instance for its configuration or creation.

They are available at: http://169.254.169.254/latest/meta-data/

Note that this is a local address and therefore only accessible from the EC2 instance.

Examples of use

Data Types

ec2-user ~ % curl http://169.254.169.254/latest/
dynamic
meta-data
user-data

There are 3 types of data accessible from an EC2 instance as we can see in the output of this command.

Services

You can access the instance metadata from a running instance using one of the following methods:

• Instance Metadata Service Version 1 (IMDSv1) – request/response method
• Instance Metadata Service Version 2 (IMDSv2) – session-oriented method

When using session-oriented requests (IMDSv2), you create a session token that defines the duration of the session, which must be a minimum of one second and a maximum of six hours. During the specified period, you can use the same session token for subsequent requests.

Here’s how to retrieve a session token:

ec2-user ~ % TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

You can then use it in the following commands, the time of the session:

ec2-user ~ % curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ami-id

Some instance metadata

Here is a description of some metadata of instances that may be useful:

AWS SDKs

An AWS SDK (Software Development Kit) is used to interact with AWS within an application. There are therefore many AWS SDKs depending on the different programming languages (Python, Java, C++, JavaScript, Ruby, PHP,…):

• The AWS CLI is itself encoded from the Python AWS SDK (also called Boto3).
• Some AWS services are only accessible by an AWS SDK: DynamoDB, Lambda Function,…

Note that if you do not have a default Region, the AWS SDKs will interact with the default Region us-east-1.

AWS Limits and Backoff

Limits / Quotas

There are Limits or Quotas in AWS that you must be aware of:

• API Rate Limits: depending on the AWS APIs, one cannot make more than a certain number of calls per second to an API
• Service Quotas (or Service Limits): According to the AWS Services, you cannot consume more than a certain number of AWS Services (e.g. 1152 vCPU per AWS account)

Exponential Backoff

When you get ThrottlingException errors when calling AWS Services, you must use the Exponential Backoff.

It is a retry mechanism with a duration between each attempt that increases exponentially. It is described more precisely in this article https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

• AWS SDKs already implement it, so there is nothing to do
• But if you make calls to the AWS APIs by another means, you MUST set up such a mechanism:
  • In case of ThrottlingException or 5xx errors
  • Not in case of 4xx errors

Written by

Jean-Jerome Levy

DevOps Consultant

Seasoned professional in the field of information technology, I bring over 20 years of experience from working within major corporate IT departments. My diverse expertise has played a pivotal role in a myriad of projects, marked by the implementation of innovative DevOps practices.

You may also like...

AWS CloudFormation: Infrastructure-as-Code by Amazon

AWS Streaming - Amazon Kinesis

AWS Messages - Amazon SQS and SNS

Monitoring and Audit in AWS - CloudWatch, X-Ray et CloudTrail