AWS certified developer associate hints. AWS CI/CD, CLI, SDK, CDK, other AWS services (part VII)

AWS CI/CD

AWS Code Commit

  • Version control
  • Enabled by using control systems such as GIT
  • Git repo can be synchronized between computers
  • Offers private repositories
  • Fully managed and highly available
  • Code only in AWS cloud account
  • Security
  • Integrated with Jenkins, CodeBuild and others

AWS Code Commit security

  • SSH keys
  • HTTPS
  • IAM policies
  • Encryption
  • Encryption in transit
  • Cross-account access

AWS CodePipeline

  • Visual workflow to orchestrate your CICD
  • Source- CodeCommit, ECR, S3, Bitbucket, GitHub
  • Build - CodeBuild, Jenkins, CloudBees, TeamCity
  • Test - CodeBuild, AWS Device Farm
  • Deploy - CodeDeploy, Elastic Beanstalk, CloudFormation Consist of stages including manual approvals

AWS CodeBuild

  • Source - code commit, s3, gutbucket or githab
  • Build instructions: code file buildspec.yml
  • Output logs can be stored to S3 & CloudWatch logs
  • Use CloudWatch metrics to monitor build statistic
  • Use CloudWatch events to detect failed builds
  • Use CloudWatch alarms to notify if you need “threshhold”

CodeBuild - buildspec.yml description:

  • buldspec.yaml must be in a root
  • Env - plain text or SSM or AWS Secret Manager
  • Phases: install, pre_build, build, post_build
  • Artifacts
  • Cache
  • Possible to run locally, possible to run inside VPC
  • CloudFormation is used to deploy complex infrastructure using API
  • CREATE_UPDATE - crease or update an existing stack
  • DELETE_ONLY - delete a stack if it exists

AWS Codedeploy

Each EC2 instance must be running CodeDeploy Agent.

CodeDeploy components:

  • Application - a unique name function as a container(revision, deployment configuration)
  • Compute platform - EC2/On-Premises, AWS Lambda or Amazon ECS
  • Deployment configuration - EC2/On-premises - specify the minimum number of healthy instances for deployment - AWS Lambda or Amazon ECS - specify how traffic is router to your updated version
  • Deployment Group - group of tagged EC2 instances
  • Deployment Type - method used to deploy the application (In-place deployment only for EC2/on-premises, blue/green - EC2 instances only, AWS Lambda and ECS)
  • IAM instance profile - give ec2 instances the permissions to accesss both s3/github
  • Application revision - application code + appspec.yml file
  • Service Role - an IAM Role for CodeDeploy to perform operation on EC2
  • Target revision - the most recent version that you want to deploy to a Deployment Group

CodeDeploy - appspec.yml

  • Files - how to source and copy from s3/github (source/destination)
  • Hooks - set of instructions to do to deploy the new version (ApplicationStop, DownloadBundle, BeforeInstall, Install, AfterInstall, ApplicationStart, ValidateService)

Deployment Configuration

  • One at a time - one EC2 instance at a time, if one instance fails then deployment stops
  • Half at a time - 50%
  • All at once - quick but no healthy host downtime
  • Custom

EC2 instances stay in “Failed” state.

New deployment will first be deployed to failed instances, to rollback, redeploy, old deployment or enable automated rollback for failure.

CodeDeploy - Deploy to an ASG

In place deployment:

  • Updates existing EC2 instances
  • Newly created EC2 instances by an ASG will also get automated deployment

Blue/Green deployment:

  • A new Auto-Scaling group is created (settings are copied)
  • Choose how long to keep the old EC2 instances (old ASG)
  • Must be using an ELB

AWS CodeStar

  • Quickly creates CICD ready project for EC2, Lambda, Elastic Beanstalk
  • Support C#, Go, HTML5, Java, Node,PHP, Python, Ruby

AWS CodeArtifact

  • Secure scalable and cost-effective artifacts management for software deployment
  • Works with: maven, grade, nom, yarn, twine, pipe and NuGet
  • Support own packages

Event is created when a package version is created, modified or deleted into EventBridge.

AWS CodeGuru

  • An ML-powered service for automated code reviews and application performance recommendations
  • Provides two functions
  • CodeGuru Reviewer: automated code reviews for static code analysis
  • CodeGuru Profiler: visible/recommendations about application performance runtime

CodeGuru - Agent configurations

  • MaxStackDepth
  • MemoryUsageLimitPercent
  • MinimumTimeForeReportingInMilliseconds
  • ReportingIntervalInMilliseconds
  • SamplingIntervaInMilliseconds

AWS Cloud9

  • Cloud-based Integrated development environment
  • Prepackaged with essential tools for popular languages
  • Share your development env with your team
  • Fully integrated with AWS SAM & Lambda

AWS CLI, SDK and CDK

AWS CLI possible to run with —dry-run flag to ensure we have enough permissions to do something.

STS Decode

To decode encoded responses from AWS CLI, we need to run STS command line. STS service need to be granted permissions

aws sts decode-authorization-message —encoded-message “Message”

Instance metadata

AWS CLI Profiles

aws configure —profile - gives a way to have multiple aws accounts

/.aws/credentials => will have several profiles

/.aws/config => will have several configs

aws s3 ls —profile “name” => execute command in not default profile

MFA with CLI

To use MFA with CLI you must create a temp session, it’s possible using STS GetSessionToken.

aws sts get-session-token —duration-seconds 360 => returns secret access key + token.

aws sts get-session-token —serial-number arn:aws:iam::xxxx:mfa/name —token-code code.

AWS Limits

  • API Rate limits, possible to request API limit request to AWS support
  • Service Quotas (Service limits), possible to request programmatically and by doing a request to AWS support

In case you see ThrottlingExeption, you need to do exponential backoff.

Logic is already integrated in the AWS SDK.

If you are making call to API, you need to implement retries on 5xx server errors yourself (1/2/4/16 seconds).

AWS CLI credentials provider chain

The CLI will look for credential in this order.

  1. Command line options (region, output, profile)
  2. ENV variables
  3. CLI credential file (/.aws/credentials)
  4. CLI configuration files (/.aws/config)
  5. Container credentials - for ECS tasks

AWS signing API request

  • If you use the SDK or CLI HTTP requests are signed for you
  • You should sign an AWS HTTP request using Signature v4 (SigV4)
  • Possible to send in headers (Authorization/SignedHeaders/Signature)
  • Possible to use query string (?X-Amz-Algorithm=AWS4-HMAC-SHA256X-Amz-Credential&X-Amz-Date&X-Amz-Signature)

AWS Cloud development Kit (CDK)

  • Define your cloud infrastructure using familiar language
  • Contains high level components called constructs
  • The code is compiled into CloudFormation template (JSON/YAML)
  • You can therefore deploy infrastructure and application runtime code together

AWS CDK Constructs:

  • CDK Constrsruct is a component that encapsulate everything CDK needs to create the final CloudFormation stack
  • Can represent a single AWS resource or multiple related resources
  • AWS construct library contains Constructs for every AWS resource
  • Constructs divided in 3 different level

Layer 1 Constructs:

  • Can be called CFN Resources which represent all resources directly available in CloudFormation
  • Constructs are periodically generated from CloudFormation Resource Specification
  • Construct names start with Con
  • You must explicitly configure all resources properties

Layer 2 Constructs:

  • represent AWS resources but with a higher level
  • Similar functionality as L1 but with convenient defaults and boilerplate

Layer 3 Constructs:

  • Can be called Patterns, which represents multiple resources
  • Helps you complete common tasks in AWS

CDK important commands

npm install -g aws-cdk-lib - install CDK CLI
cdk init app - create new CDK project
cdk synth - synthesizes and prints the CloudFormation template
cdk bootstrap - deploys the CDK toolkit staging Stack
cdk deploy - deploy the stack
cdk diff - view difference of local CDK and deployed Stack
cdk destroy - destroy Stack

Bootrsraping, to start CDK you need to deploy S3 and roles first, it’s done with command.

cdk bootstrap aws://1234323232/eu-west-1

AWS Step Functions

  • Model your workflow as state machine
  • Written in JSON
  • Visualization of the workflow and the execution of the workflow, as well as history
  • Start workflow withSDK call, API Gateway, Event Bridge

Step function task states:

  • Invoke one AWS service
  • Can invoke lambda function
  • Run a AWS batch job
  • Run ECS task and wait for it to complete
  • Enter an item from DynamoDB
  • Publish message to SNS, SQS
  • Launch another Step function workflow
  • Run an once activity (EC2, or on-premise)

Step function states:

  • Choice state - test for a condition to send to a branch
  • Fail or Success state - stop execution
  • Pass state - simply pass its input to its output or inject some fixed data
  • Wait state - provide a delay
  • Map state - dynamically iterate step
  • Parallel state - begin parallel branches of execution

Error handling in step functions:

  • Any state can encounter runtime errors for various reasons
  • Use retry (to retry failed state) and Catch (transition to failure path) in the state machine to handle the errors instead of inside the application code
  • Predefined error codes: State.ALL/State.Timout/State.TaskFailed/State.Permission

Step function Retry:

  • Evaluated from top to bottom
  • ErrorEquals : match a specific kind of error
  • IntervalSeconds: initial delay before retrying
  • BackoffRate: multiply the delay after each retry
  • MaxAttempts: default to 3, set to 0 for never retried
  • When max attempts are reached, the Catch kicks in

Step function Catch:

  • Evaluated from top to bottom
  • ErrorEquals: match a specific kind of error
  • Next: state send to
  • ResultPath - a path that determines what input is sent to the state specified in the next field

Step function - ResultPath:

  • Include the error in the input

Step function - wait for task token:

  • Allow you to pause step function during a task until a task token is returned
  • Task might wait for other AWS services, human approval, 3rd party integration, call legacy systems
  • Append .waitForTaskToken to the Resource field to tell Step Functions to wait for the Task Token to be returned

Step function - activity tasks:

  • Enables you to have the Task work performed by an Activity Worker
  • Activity Worker apps can be running on EC2, Lambda, mobile device
  • Activity Worker poll for a Task using GetActivityTask API
  • After activity worker completes its work, it sends a response of its success/failure using SendTaskSuccess or SendTaskFailure

Step Functions - Standard vs Express

Standard:

  • Max duration 1 year
  • Execution model exactly-once execution
  • Execution rate over 2000 /second
  • Execution history: up to 90 days or using CloudWatch
  • Pricing, number of state tansitions

Express:

  • Max duration 5 mins
  • Execution rate over 100 000 /second
  • History: only CloudWatch logs
  • Pricing, number of executions, duration and memory consumption
  • Async execution - at-least once, sync execution - at-most once

AWS AppSync

  • App sync is a managed service that uses graphql
  • Retrieve data in real-time with WebSocket or MQTT on WebSocket
  • For mobile apps: local data access & data synchronization
  • Integration with CloudWatch Metrics and logs
  • Possible to use as a source DynamoDB, Aurora, OpenSearch, Lambda, HTTP
  • Possible to authorize using API_KEY, AWS_IAM, OPENID_CONNECT, AMAZON_COGNITO_USER_POOLS

AWS Amplify

  • Set of tools to get started with creating mobile applications
  • Elastic beanstalk for mobile and web applications
  • "Must have" features such as, authentication, storage, and machine-learning all powered by AWS services
  • Front-end libraries with ready-to-use components for React, Cue, JS, iOS, Android , Flutter
  • Incorporate AWS best practices for reliability, security, scalability

Features:

  • Authentication (cognito, mia, sing-in, rebuild ui)
  • Datastore (Appsync, DynamoDB, Graphql, offline and real-time)
  • Hosting (CI/CD, PR, monitoring, custom domains)
  • Unit and e2e tests cypress

AWS SES - simple email service

  • Smtp interfac
  • AWS SDK
  • Integration with s3, SNS, Lambda
  • Integrated with IAM for allowing to send emails

Amazon OpenSearch service

  • Amazon open search is successor to amazon elasticSearch
  • With open search you can search any field, even partial matches
  • It’s common to use open search as a complement to another database
  • Do not support SQL
  • Ingestion from Kinesis data firehose, AWS IoT, CloudWatch logs
  • Security through cognito, IAM, KMS, TLS
  • Comes with OpenSearch Dashboard

Amazon Athena

  • Serverless query service to analyze data store in S3
  • Uses SQL lang to query the files
  • Support CSV, JSON, ORC, avro and parquet
  • Commonly used with Amazon Quicksight for reporting/dashboard
  • Used for business intelligence, analysis, reporting, query VPC flow logs, ELB logs
  • Use columnar data for cost saving (Apache parquet or ORC)
  • Use glue to convert your data to Parquet or ORC
  • Compress data
  • Partition datasets using by data or other formats
  • Federated query, combine queries using data source connector(lambda function)

Amazon managed streaming for Apache Kafka (Amazon MSK)

  • Alternative to apache Kafka on AWS
  • Fully managed apache Kafka on AWS
  • Allow you to create, update, delete clusters
  • MSK creates & manages Kafka brokers nodes & Zookeeper nodes for you
  • Deploy the MSK cluster to VCP multi-az
  • Automatically recover from common Kafka failures
  • Data is stored on EBS volumes for as long as you want
  • Possible to use MSK serverless
  • 1mb default message size, possible to have 10mb
  • Possible to consume using Kinesis data analytics for Apache Flink, AWS glue, Lambda, custom consumer

AWS certificate manager (ACM)

  • Let’s you easily provision, manage and deploy SSL/TLS certificates
  • Used to provide in-flight encryption for website (https)
  • Private and public certificates
  • Automatic TLS certificates renewal
  • Free of charge for public TLS certificates
  • Integration with ELB, CloudFront, API Gateway

AWS Private certification Authority (CA)

  • Managed service allows you to create private certificate authorities (CA), including root and subordinaries CAs
  • Can issue and deploy end-entity X.509 certificates
  • Certificates are trusted by your organization (not public internet)
  • Works for AWS services that are integrated with ACM

AWS AppConfig

  • Configure, validate and deploy dynamic configuration
  • Deploy dynamic configuration changes to your applications independently of any code deployments
  • Feature flag, application using, allow/block listing
  • Use with apps on EC2 instance, Lambda, ECS, EKS
  • Gradually deploy the configuration changes and rollback if issues occurs
  • Validate configuration changes before deployment using JSON schema pr lambda function