How to Create an External Data Store for Your Docker-to-EC2 Deployment

We've recently demonstrated how to use a GitHub Action workflow to deploy an application to an AWS EC2 instance. While this is an excellent starting point, it does not provide the ability to store non-ephemeral data.

With the need to store non-ephemeral data in mind, the Bitovi DevOps team is proud to introduce the first major enhancement to our docker-to-ec2 GitHub Action, which adds support for non-ephemeral storage via EFS.

Knowledge Check

What Are the Benefits of Non-Ephemeral Storage?

Non-ephemeral storage offers a number of benefits for you. It allows for data to be stored and accessed for longer periods of time, meaning that it can be used to store important information that needs to be kept for extended periods of time / indefinitely.

Additionally, non-ephemeral storage can help ensure that data remains safe and secure. It is stored in an isolated environment that is not subject to the same risks associated with ephemeral storage, such as loss of data upon recreation, AWS resource shifting, or other destructive actions on running resources. Non-ephemeral storage can provide peace of mind for you that your data is safe and secure.

What Is AWS EFS?

AWS EFS (Elastic File System) is a cloud storage service offered by Amazon Web Services. It provides you with the ability to store and access non-ephemeral data on the cloud, allowing data access from any location with an internet connection. EFS provides you with a secure, reliable, and highly scalable storage solution, making it an ideal choice for businesses and organizations that require a cloud-based storage solution.

Instructions

1. Create an EFS in a Single Zone
2. Create a Highly Available EFS
3. Mount an EFS

Prerequisites

• A GitHub repo
• An AWS account
• An application repo that’s been prepped to deploy through the docker-to-ec2 github action

Create an EFS in a Single Zone

Summary of Work

1. Toggle Create EFS option

2. Update docker-compose file to use volume mounts

3. Specify the volume mounts

4. (Optional) Specify a zone mapping

1. Toggle Create EFS Option

Update your deployment workflow to include the create_efs: true input.

.github/workflows/deploy.yaml

steps:
  - id: deploy
    name: Deploy
    uses: bitovi/github-actions-deploy-docker-to-ec2@v0.4.5
    with:
      aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID}}
      aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY}}
      app_port: 3000
      create_efs: true

Run the deployment workflow, and voilà! 🎉 The EFS, along with all its additional resources, will be created and added to the Terraform-managed state. Additional resources include security groups that bidirectionally whitelist the EC2 security group and EFS mount targets, which will be used to mount the EFS to the EC2 within the deployment.

2. Update docker-compose File to Use Volume Mounts

Update the volume mounts used by the docker-compose file to be: $HOST_DIR:$TARGET_DIR

The HOST_DIR and TARGET_DIR values will be added to the .env file before docker-compose up is run to begin the application.

docker-compose.yaml

services:
  app:
    container_name: example-application
    env_file: .env
    restart: always
    build: .
    ports:
      - 3334:3334
    stdin_open: true
    tty: true
    volumes:
      - $HOST_DIR:$TARGET_DIR

3. Specify the volume mounts

There are 2 inputs that control the volume mount paths. As always, the defaults will get you working quickly, but specificity is offered.

application_mount_target

The application_mount_target input represents the folder path within the EC2 instance to the data directory. The default is; /user/ubuntu/<application_repo>/data.

Additionally, the application_mount_target is the HOST_DIR which is added to the .env that docker-compose uses to start the application.

data_mount_target

The data_mount_target input represents the target volume directory within the docker-compose container. The default is /data

Additionally, the data_mount_target is the TARGET_DIR which is added to the .env that docker-compose uses to start the application.

4. (Optional) Specify a Zone Mapping

Zone Mappings are a list of JSON objects that represent EFS Mount Targets. This is the most complex use case for an EFS. We recommend you stick to using our defaults if you’re just starting out.

Example Zone Mapping

{
  "a": {
    "subnet_id": "subnet-foo123",
    "security_groups: ["sg-foo123", "sg-bar456"]
  },
  "b": {
    "subnet_id": "subnet-goo456",
    "security_groups: ["sg-goo456", "sg-roo789"]
  }
}

The following zone mapping translates to 2 mount targets. The first mount target is in the “a” availability zone, and an ENI is created within the subnet (subnet-foo123), and finally, the mount target attaches the security group list ("sg-foo123", "sg-bar456").

The second mount target is in the “b” availability zone. An ENI is created within the subnet (subnet-goo456), then the mount target attaches the security group list ("sg-goo456", "sg-roo789").

Create a Highly Available EFS

Toggle Create HA EFS Option

Creating a high availability, EFS does incur more cost however is a reliable solution for those who need their data highly available. The HA option will create mount targets in all availability zones within the region it is being deployed to.

Mount an Existing EFS

Summary of Work

1. Specify the Mount ID of an existing EFS
2. (Optional) Specify the EFS Primary Security Group

Specify the Mount ID of the Existing EFS

If you have a preexisting EFS that you’d like to mount to an EC2, simply add the resource identifier as an aws_mount_efs_id input.

.github/workflows/deploy.yaml

steps:
  - id: deploy
    name: Deploy
    uses: bitovi/github-actions-deploy-docker-to-ec2@v0.4.5
    with:
      aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID}}
      aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY}}
      app_port: 3000
      aws_mount_efs_id: fs-1a2b3c4d

Much like the create_efs option, this will whitelist the Security Groups bidirectionally between the EFS and the EC2. However, it will not create additional resources, be advised that the deployed EC2 needs a mount target within its availability zone. The default security group will be used unless mount_efs_security_group_id is specified.

(Optional) Specify the EFS Primary Security Group

A list of security groups to attach to the mounted efs' mount targets.

mount_efs_security_group_id=[ "sg-123", "sg-456"]

Conclusion

You now have all the information and configuration required to create or mount an EFS to your deployed EC2, which provides you with a non-ephemeral storage option. You could say that this change will… last 🥁 ba dom tss. 

Thanks! Keep an eye out for our next blog in the series, which will be introducing a much more powerful storage option, RDS!

Need Help?

Drop into Community Discord and let us know! Bitovians (and our community) love to solve problems and chat about exciting projects.

Need DevOps Consulting or Platform Engineering services? Bitovi has DevOps consultants who can assist with all aspects of your development and DevOps journey.