AWS Go Through

Posted on | In , , |

I have been used for AWS for a while, but always there are someone who want to touch this however don’t know how to start or what to follow.

Recently I read this link, and I was doing it all through and it was a good one to start. So, you want to learn AWS? AKA, “How do I learn to be a Cloud Engineer?”

I won’t paste and tell you everything how to do it, but will point the ones that might block you or you might need to Google it.

Most of the words comes from the above link, I will add my own where necessary.

1. Introduction

So many people struggle with where to get started with AWS and cloud technologies in general. There is popular “How do I learn to be a Linux admin?” post that inspired me to write an equivalent for cloud technologies. This post serves as a guide of goals to grow from basic AWS knowledge to understanding and deploying complex architectures in an automated way. Feel free to pick up where you feel relevant based on prior experience.

1.1. Assumptions:

  • You have basic-to-moderate Linux systems administration skills
  • You are at least familiar with programming/scripting. You don’t need to be a whiz but you should have some decent hands-on experience automating and programming.
  • You are willing to dedicate the time to overcome complex issues.
  • You have an AWS Account and a marginal amount of money to spend improving your skills.

1.2. How to use this guide:

  1. This is not a step by step how-to guide.
  2. You should take each goal and “figure it out”. I have hints to guide you in the right direction.
  3. Google is your friend. AWS Documentation is your friend. Stack Overflow is your friend.
  4. Find out and implement the “right way”, not the quick way. Ok, maybe do the quick way first then refactor to the right way before moving on.
  5. Shut down or de-provision as much as you can between learning sessions. You should be able to do everything in this guide for literally less than $50 using the AWS Free Tier. Rebuilding often will reinforce concepts anyway.
  6. Skip ahead and read the Cost Analysis and Automation sections and have them in the back of your mind as you work through the goals.
  7. Lastly, just get hands on, no better time to start then NOW.

1.3. Project Overview

This is NOT a guide on how to develop websites on AWS. This uses a website as an excuse to use all the technologies AWS puts at your fingertips. The concepts you will learn going through these exercises apply all over AWS.

This guide takes you through a maturity process from the most basic webpage to an extremely cheap scalable web application. The small app you will build does not matter. It can do anything you want, just keep it simple.

Need an idea? Here: Fortune-of-the-Day - Display a random fortune each page load, have a box at the bottom and a submit button to add a new fortune to the random fortune list.

2. Account Basics

Create an IAM user for your personal use.

It will deduct for 1$ for validation, don’t worry. You are not consuming anything.

Set up MFA for your root user, turn off all root user API keys.

  1. You need Google Authenticator, download it from your Android/iPhone app store.
  2. Wait for a while and then a new token will be generated, you need type both.
  3. After doing that, refresh the page, the green checkmark will be shown on aws IAM home page.

Set up Billing Alerts for anything over a few dollars.

The linke is In the Billing Alerts Preferences link, and then click Manage Billing Alerts to setup the billing alarm using CloudWatch. I used $10 as a threshold.

Configure the AWS CLI for your user using API credentials.

  1. Go to IAM service and create a new user, I use admin. Should be a programmable user.
  2. Install aws-cli using pip install
  3. Configure ~/.aws/credentials using 
    1
    2
    3
    4
    5
    6
    7
    [default]
    aws_access_key_id = xxxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    [corp]
    aws_access_key_id = xxxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Also configure the ~/.aws/config using

1
2
3
[default]
output = json
region = us-east-1

  1. Use aws --profile corp help to test.

Checkpoint: You can use the AWS CLI to interrogate information about your AWS account.

  1. Bind the arn:aws:iam::aws:policy/IAMReadOnlyAccess or type iam in the permission set filter in admin‘s permission set. You will find a readonly one.
  2. Type aws --profile corp iam get-user to test.

3. Web Hosting Basics

Deploy a EC2 VM and host a simple static “Fortune-of-the-Day Coming Soon” web page.
Take a snapshot of your VM, delete the VM, and deploy a new one from the snapshot. Basically disk backup + disk restore.
Checkpoint: You can view a simple HTML page served from your EC2 instance.

3.1. Deploy a EC2 VM and put a simple web server

  1. Lauched with one free-tire Ubuntu 16.04 server, just select free tier only and then click into Ubuntu and then reveiw and finish. It will ask you to use an existing ssh keys or create a new pair. I choose to create a new pair and then download that.
  2. Keep that new generated key safe. That’s the way to ssh into the new launched server.
  3. chmod 400 to your new generated pem file. Eg, it could be new-ec2-admin.pem
  4. Login into using ssh -i new-ec2-admin.pem 34.205.74.0 or ssh -i new-ec2-admin.pem ec2-34-205-74-0.compute-1.amazonaws.com. You can get the login method from right click the ec2 machine, and then click Connect. Then use sudo -i to get root permission without password.
  5. Install Nginx/Apached or other webserver you would like, even you can setup your own webserver using NodeJS or Python, etc. I am using Python since python is supported by default. So, just type python3 -m http.server 8080 to start a small http server and will listen to 8080 port.
  6. Open port for 8080 using security groups. Add a new inboud rule for TCP port of 8080 from anywhere, i.e., 0.0.0.0
  7. Open your browser and access http://34.205.74.0:8080/, Directory listing page should come up.
  8. Add python server to auto start. Add @reboot cd ~/ && python3 -m http.server 8080 > /dev/null 2>&1 by using crontab -e.
  9. Reboot your server and testing if http://34.205.74.0:8080/ could be accessed again.

3.2. Take a snapshot and recover from snapshot

  1. Stop the EC2 instance.
  2. Create a image from the stopped EC2. It could be a HVM image, or ParaVirtualized. The HVM can be launched with t2.micro, however, ParaVirtualized can’t, but able to be launched using t1.micro type.
    It will create a snapshot and then create a image.
  3. Add EC2 permission to your aws-cli user.

  4. Run one new instance with the image-id created above, using next command:

    1
    2
    3
    4
    5
    6
    7
    aws --region us-east-1  \
        --profile corp ec2 run-instances \
        --image-id ami-04e6460ff83cc6750 \
        --instance-type t2.micro \
        --security-groups test-sg \
        --key-name aws-mw-corp  \
        --count 1

  5. Access the http://34.205.74.0:8080/(use your own IP address) again, it should work by default.

4. Auto Scaling

Create an AMI from that VM and put it in an autoscaling group so one VM always exists.

Put a Elastic Load Balancer infront of that VM and load balance between two Availability Zones (one EC2 in each AZ).

Checkpoint: You can view a simple HTML page served from both of your EC2 instances. You can turn one off and your website is still accessible.

5. External Data

Create a DynamoDB table and experiment with loading and retrieving data manually, then do the same via a script on your local machine.
Refactor your static page into your Fortune-of-the-Day website (Node, PHP, Python, whatever) which reads/updates a list of fortunes in the AWS DynamoDB table. (Hint: EC2 Instance Role)
Checkpoint: Your HA/AutoScaled website can now load/save data to a database between users and sessions

6. Web Hosting Platform-as-a-Service

Retire that simple website and re-deploy it on Elastic Beanstalk.
Create a S3 Static Website Bucket, upload some sample static pages/files/images. Add those assets to your Elastic Beanstalk website.
Register a domain (or re-use and existing one). Set Route53 as the Nameservers and use Route53 for DNS. Make www.yourdomain.com go to your Elastic Beanstalk. Make static.yourdomain.com serve data from the S3 bucket.
Enable SSL for your Static S3 Website. This isn’t exactly trivial. (Hint: CloudFront + ACM)
Enable SSL for your Elastic Beanstalk Website.
Checkpoint: Your HA/AutoScaled website now serves all data over HTTPS. The same as before, except you don’t have to manage the servers, web server software, website deployment, or the load balancer.

7. Microservices

Refactor your EB website into ONLY providing an API. It should only have a POST/GET to update/retrieve that specific data from DynamoDB. Bonus: Make it a simple REST API. Get rid of www.yourdomain.com and serve this EB as api.yourdomain.com
Move most of the UI piece of your EB website into your Static S3 Website and use Javascript/whatever to retrieve the data from your api.yourdomain.com URL on page load. Send data to the EB URL to have it update the DynamoDB. Get rid of static.yourdomain.com and change your S3 bucket to serve from www.yourdomain.com.
Checkpoint: Your EB deployment is now only a structured way to retrieve data from your database. All of your UI and application logic is served from the S3 Bucket (via CloudFront). You can support many more users since you’re no longer using expensive servers to serve your website’s static data.

8. Serverless

Write a AWS Lambda function to email you a list of all of the Fortunes in the DynamoDB table every night. Implement Least Privilege security for the Lambda Role. (Hint: Lambda using Python 3, Boto3, Amazon SES, scheduled with CloudWatch)
Refactor the above app into a Serverless app. This is where it get’s a little more abstract and you’ll have to do a lot of research, experimentation on your own.
The architecture: Static S3 Website Front-End calls API Gateway which executes a Lambda Function which reads/updates data in the DyanmoDB table.
Use your SSL enabled bucket as the primary domain landing page with static content.
Create an AWS API Gateway, use it to forward HTTP requests to an AWS Lambda function that queries the same data from DynamoDB as your EB Microservice.
Your S3 static content should make Javascript calls to the API Gateway and then update the page with the retrieved data.
Once you have the “Get Fortune” API Gateway + Lambda working, do the “New Fortune” API.
Checkpoint: Your API Gateway and S3 Bucket are fronted by CloudFront with SSL. You have no EC2 instances deployed. All work is done by AWS services and billed as consumed.

9. Cost Analysis

Explore the AWS pricing models and see how pricing is structured for the services you’ve used.
Answer the following for each of the main architectures you built:
Roughly how much would this have costed for a month?
How would I scale this architecture and how would my costs change?

10. Architectures

Basic Web Hosting: HA EC2 Instances Serving Static Web Page behind ELB
Microservices: Elastic Beanstalk SSL Website for only API + S3 Static Website for all static content + DynamoDB Table + Route53 + CloudFront SSL
Serverless: Serverless Website using API Gateway + Lambda Functions + DynamoDB + Route53 + CloudFront SSL + S3 Static Website for all static content

11. Automation

!!! This is REALLY important !!!

These technologies are the most powerful when they’re automated. You can make a Development environment in minutes and experiment and throw it away without a thought. This stuff isn’t easy, but it’s where the really skilled people excel.
Automate the deployment of the architectures above. Use whatever tool you want. The popular ones are AWS CloudFormation or Teraform. Store your code in AWS CodeCommit or on GitHub. Yes, you can automate the deployment of ALL of the above with native AWS tools.
I suggest when you get each app-related section of the done by hand you go back and automate the provisioning of the infrastructure. For example, automate the provisioning of your EC2 instance. Automate the creation of your S3 Bucket with Static Website Hosting enabled, etc. This is not easy, but it is very rewarding when you see it work.

12. Continuous Delivery

As you become more familiar with Automating deployments you should explore and implement a Continuous Delivery pipeline.
Develop a CI/CD pipeline to automatically update a dev deployment of your infrastructure when new code is published, and then build a workflow to update the production version if approved. Travis CI is a decent SaaS tool, Jenkins has a huge following too, if you want to stick with AWS-specific technologies you’ll be looking at CodePipeline.

13. Miscellaneous / Bonus

These didn’t fit in nicely anywhere but are important AWS topics you should also explore:

IAM: You should really learn how to create complex IAM Policies. You would have had to do basic roles+policies for for the EC2 Instance Role and Lambda Execution Role, but there are many advanced features.
Networking: Create a new VPC from scratch with multiple subnets (you’ll learn a LOT of networking concepts), once that is working create another VPC and peer them together. Get a VM in each subnet to talk to eachother using only their private IP addresses.
KMS: Go back and redo the early EC2 instance goals but enable encryption on the disk volumes. Learn how to encrypt an AMI.

14. Final Thoughts

I’ve been recently recruiting for Cloud Systems Engineers and Cloud Systems Administrators. We’ve interviewed over a dozen local people with relevant resume experience. Every single person we interviewed would probably struggle starting with the DynamoDB/AutoScaling work. I’m finding there are very few people that HAVE ACTUALLY DONE THIS STUFF. Many people are familiar with the concepts, but when pushed for details they don’t have answers or admit to just peripheral knowledge. You learn SO MUCH by doing.

If you can’t find an excuse or get support to do this as part of your job I would find a small but flashy/impressive personal project that you can build and show off as proof of your skills. Open source it on GitHub, make professional documentation, comment as much as is reasonable, and host a demo of the website. Add links to your LinkedIn, reference it on your resume, work it into interview answers, etc. When in a job interview you’ll be able to answer all kinds of real-world questions because you’ve been-there-done-that with most of AWS’ major services.

I’m happy to hear any feedback. I’m considering making THIS post my flashy/impressive personal project in the form of a GitHub repo with sample code for each step, architecture diagrams, etc.

0%