Table of Content

  1. Introduction
    • Brief overview of the problem: Managing web servers and logs at scale
    • Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination
    • Why This Stack Works Together
  2. Prerequisites
  3. Infrastructure Setup
    • Create security group
    • Security Group configuration for web traffic
    • Verify the security group rules
    • Create your key pair
    • Launch EC2 instance using created security group
    • Copy keypair to ssh directory
  4. Ansible Configuration
    • Folder Structure
    • Prepare workspace environment
    • Create the host.yaml file
    • Test ansible connection
  5. Set Up Roles
  6. Common Task
  7. Nginx Playbook
  8. Security Playbook
  9. Log Management FluentD Playbook
  10. Deployment and Testing

Introduction

Brief overview of the problem: Managing web servers and logs at scale

In today's cloud environments, managing web servers and logs at scale presents significant challenges.

DevOps teams struggle with manual server configurations, inconsistent deployments, and the overwhelming task of processing massive log data for insights and security analysis.

Our automated solution combines AWS, Ansible, Nginx, and Fluentd to streamline these operations efficiently.

Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination

AWS EC2 + Ansible + Nginx + Fluentd creates a robust web infrastructure by combining cloud scalability with automation and efficient logging.

AWS EC2 provides the flexible compute resources, Ansible automates deployment and configuration tasks, Nginx serves as a high-performance web server, and Fluentd handles comprehensive log collection and processing.

Why This Stack Works Together

AWS EC2 (Infrastructure)

  • Provides on-demand, scalable computing resources
  • Offers multiple instance types to match workload needs
  • Integrates seamlessly with other AWS services
  • Enables global deployment with multiple regions

Ansible (Automation)

  • Automates server configuration and application deployment
  • Uses simple YAML syntax for easy maintenance
  • Requires no agents on managed servers (agentless)
  • Ensures consistent configurations across all servers

Nginx (Web Server)

  • Delivers high-performance web serving capabilities
  • Handles concurrent connections efficiently
  • Provides reverse proxy and load balancing features
  • Offers robust security features and SSL/TLS support

Prerequisites

  1. Basic Knowledge of AWS & Ansible
  2. AWS CLI installation on local machine
  3. Configure AWS CLI
  4. Install Ansible

Infrastructure Setup

Before we proceed, with setting up infrastructure. It is best we confirm Ansible is installed on our machine and AWS CLI is well configured.

ansible --version



aws sts get-caller-identity

Now we can go ahead with setting up infrastructure.

Create security group

Next step is to create the security group, copy and paste the below code in your terminal

aws ec2 create-security-group \
    --group-name nginx-web-server-sg \
    --description "Security group for Nginx web server and SSH access"

This command would create a security group and output the

  • GroupId
  • and SecurityGroupArn;
{
    "GroupId": "sg-0ee2b6c700c11e902",
    "SecurityGroupArn": "arn:aws:ec2:us-east-1:910883278292:security-group/sg-0ee2b6c700c11e902"
}

You can login to your AWS console to confirm the creation of the Security Group

Security Group configuration for web traffic

Lets Add the necessary inbound rules for HTTP (80), HTTPS (443), and SSH (22):

  • Export the Security Group ID to a variable (from the previous command output; Replace with your actual Security Group ID)
export SG_ID="sg-0ee2b6c700c11e902"

confirm you have exported your SG_ID
echo $SG_ID

  • Allow HTTP (port 80)
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

The above command should output similar results (Remember your security ID is different from mine)

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-081264c3c8fe3e58c",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 80,
            "ToPort": 80,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-081264c3c8fe3e58c"
        }
    ]
}
  1. Allow HTTPS (port 443)
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

Command output should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06591dee28547a3eb",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06591dee28547a3eb"
        }
    ]
}
  • Allow SSH (port 22) - Best practice is to limit this to your IP
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0
Note: for a more secure environment, its advisable to only allow your specific IPs instead of using 0.0.0.0/0 as the cidr block

Command output should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06c1f6c3205247aea",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "105.113.64.38/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06c1f6c3205247aea"
        }
    ]
}

Verify the security group rules

aws ec2 describe-security-groups --group-ids $SG_ID

Due to the how lengthy the command outputs for the security group verification is, I wont be posting it here.

Create your key pair

This command would create your key pair, and save it on your local machine

aws ec2 create-key-pair --key-name nginx-server-key --query 'KeyMaterial' --output text > nginx-server-key.pem

Confirm the created key
cat nginx-server-key.pem

Launch EC2 instance using created security group

aws ec2 run-instances \
    --image-id ami-0e1bed4f06a3b463d \
    --instance-type t2.micro \
    --key-name nginx-server-key \
    --security-group-ids $SG_ID \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nginx-fluentd-server}]'
  • *Image AMI: * ami-0e1bed4f06a3b463d : installs Ubuntu 22
  • *Instance Type: * vCPUs: 1, Memory: 1 GiB
  • Key Pair: Key=Name,Value=nginx-fluentd-server : identifies as the name of the server

Copy keypair to ssh directory

cp nginx-server-key.pem ~/.ssh

SSH into EC2 instance

You need this step to confirm you can actually log into the machine.

  • Change key pair permissions chmod 400 ~/.ssh/nginx-server-key.pem
  • Connect using ssh ssh -i ~/.ssh/nginx-server-key.pem ubuntu@<instance-ip-address>

Once you can log into the instance via ssh, we can move to the next section.

Ansible Configuration

Folder Structure

.
├── hosts.yaml
├── site.yml
└── roles/
    ├── common/
    ├── └── defaults/
    │   |   └── main.yml
    │   └── tasks/
    │   |    └── main.yml
    │   └── handlers/
    │       └── main.yml
    ├── nginx/
    │   └── tasks/
    │       └── main.yml
    ├── security/
    |   └── defaults/
    │   |   └── main.yml
    │   └── tasks/
    │       └── main.yml
    └── fluentd/
        └── defaults/
        |   └── main.yml
        └── tasks/
        |   └── main.yml
        └── files/
        |   └── denylist.txt
        └── templates/
            └── td-agent.conf.j2

Prepare workspace environment

Open your preferred terminal or use vscode terminal, create a new folder called AWS-Nginx-Ansible-FluentD and navigate into that folder

mkdir AWS-Nginx-Ansible-FluentD
cd AWS-Nginx-Ansible-FluentD

Create the host.yaml file

The hosts.yaml file is an Ansible inventory file that defines the target servers (hosts) Ansible will manage.
This file tells Ansible where and how to connect to the managed nodes for executing tasks or running playbooks.

touch host.yaml

add the below code

---
nginx_server:
  hosts:
    nginx-server-1:
      ansible_host: <your-instance-public-ip>
      ansible_user: ubuntu
      ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/nginx-server-key.pem"
      ansible_ssh_common_args: "-o StrictHostKeyChecking=no"
Note: change the value of ansible_host to your created instance IP

Test ansible connection

Run the below command in the project folder terminal

ansible nginx-server -i hosts.yaml -m ping

If connection is successfull, you should see the below result

nginx-server-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.10"
    },
    "changed": false,
    "ping": "pong"
}

Set Up Roles

In Ansible, roles are a way to organize and reuse configurations by breaking them into modular, reusable components.
Each role contains specific;

  • Tasks,
  • Variables,
  • Templates,
  • Files,
  • and handlers required to perform a particular function, such as installing a web server or configuring a database.

By using roles, you can structure your playbooks more efficiently, making them cleaner, more scalable, and easier to maintain

  1. Create site.yaml: This file is the root directory for all roles.
touch site.yaml
  1. Add the below snippet to the playbook
---
- name: Configure Web Server with Nginx and Fluentd
  hosts: nginx_server
  become: yes

  roles:
    - common
    - nginx
    - security
    - fluentd

Common Task

Create Required directories and files

  • In the root directory, create a folder called common
mkdir common
  • Create defaults, handler & task folders
mkdir defaults handler tasks
  • Create the main files in defaults, handler & tasks
touch defaults/main.yaml handler/main.yaml tasks/main.yaml

Common Playbook

  • variable definition: Add the below code to common\defaults\main.yaml The content of the file defines settings for your Ansible tasks. In simple terms, it’s configuring paths and settings for Nginx and Fluentd, as well as setting a rule for how long logs are retained.
---
nginx_html_root: /var/www/html
fluentd_config_dir: /etc/td-agent
log_retention_days: 5
  • Handler Task List: Add the below code to common\handler\main.yaml Restart Nginx: Restarts the nginx service to apply any changes or ensure it is running. Restart Fluentd: Restarts the fluentd service to reload its configuration or ensure it is operational.
---
- name: restart nginx
  service:
    name: nginx
    state: restarted

- name: restart fluentd
  service:
    name: fluentd
    state: restarted
  • Main Common Task: Add the below code to common\tasks\main.yaml The content of the file updates the package cache on systems using apt (like Debian or Ubuntu).
---
- name: Update apt cache
  apt:
    update_cache: yes
    cache_valid_time: 3600

update_cache: yes: Refreshes the list of available packages from repositories.
cache_valid_time: 3600: Ensures the cache is valid for 3600 seconds (1 hour), skipping updates if the cache is still recent.

Nginx Playbook

  • In the root directory, create a folder called nginx
mkdir nginx
  • Create nginx task folder
mkdir nginx/tasks
  • Create the nginx tasks files
touch nginx/tasks/main.yaml
  • Nginx Task playbook: Add the below code to nginx/tasks/maAuthor Of article : Ukeme David Eseme Read full article