Terraform AWS EC2 user data troubleshooting

In AWS EC2 (Elastic Compute Cloud), user data refers to the information or scripts that you can provide to an EC2 instance during its launch. User data can be used to automate instance configuration and perform various tasks on the instance.

When launching an EC2 instance, you can specify user data in the form of a script or commands.

This user data will be executed on the instance during the bootstrapping process. It allows you to customize the instance’s configuration, install software packages, run initialization scripts, and perform other actions.

User data is typically provided in the form of a shell script, but it can also be in other formats such as PowerShell scripts or cloud-init directives. The user data script runs with root privileges, allowing you to perform administrative tasks on the instance.

Some common use cases of user data include:

• Installing and configuring software packages on the instance.
• Running scripts to set up the instance as a web server, database server, or any other specific role.
• Mounting and formatting additional volumes.

Use multi-line commands in user data using Terraform

Here’s an example of how you can provide user data when launching an AWS EC2 instance using Terraform:

“`

resource "aws_instance" "example" {
ami = "ami-12345678" # Specify the desired Amazon Machine Image (AMI) ID
instance_type = "t2.micro" # Specify the instance type

user_data = <<-EOF
#!/bin/bash
echo "Hello, World!"
apt-get update
apt-get install -y nginx
service nginx start
EOF

# Other instance configuration...
}

“`

In this example, the `user_data` attribute is set to a multi-line command using the `<<-EOF` heredoc syntax. The user data script is written in Bash and contains several commands:

1. It outputs “Hello, World!” to the instance’s console log.
2. It updates the package repositories using `apt-get update`.
3. It installs the Nginx web server using `apt-get install -y nginx`.
4. It starts the Nginx service using `service nginx start`.

When Terraform provisions the EC2 instance, it will pass the provided user data script to the instance. The instance will execute the script during its bootstrapping process.

Use script in user data using Terraform

Here’s an example of providing user data with a script file when launching an AWS EC2 instance using Terraform:

“`

resource "aws_instance" "example" {
ami = "ami-12345678" # Specify the desired Amazon Machine Image (AMI) ID
instance_type = "t2.micro" # Specify the instance type

user_data = "${file("init.sh")}"

# Other instance configuration...
}

“`

In this example, the `user_data` attribute is set to the content of the `init.sh` script file using the `${file()}` interpolation function. The `init.sh` file should be located in the same directory as your Terraform configuration file.

The `init.sh` script can contain any commands or scripts that you want to run on the EC2 instance during its initial bootstrapping process. This allows you to perform custom configuration, install software, set up environment variables, and perform other tasks according to your requirements.

Make sure to replace the `ami-12345678` with the actual ID of the Amazon Machine Image (AMI) you want to use, and configure other instance settings as needed for your specific use case. Additionally, ensure that the `init.sh` script file exists in the same directory as your Terraform configuration file or provide the correct path if it’s located elsewhere.

This is the example content for the `init.sh` script file that you can use as user data when launching an AWS EC2 instance with Terraform:

#!/bin/bash

# Update the package repositories
apt update

# Install necessary packages
apt install -y nginx

# Configure Nginx
echo "Hello, world!" > /var/www/html/index.html

# Start Nginx service
systemctl start nginx

In this example, the `init.sh` script performs the following actions:

1. Updates the package repositories using `apt update`.
2. Installs the Nginx web server using `apt install -y nginx`.
3. Configures Nginx by creating an `index.html` file with the content “Hello, world!” in the default web root directory (`/var/www/html/`).
4. Starts the Nginx service using `systemctl start nginx`.

You can customize the `init.sh` script based on your requirements. This is just a simple example to demonstrate the basic usage of user data with a script file in Terraform.

What is behind the scene of user data in AWS EC2?

Behind the scenes, when user data is provided for an EC2 instance in AWS, it plays a crucial role in the instance initialization process. Here’s an overview of what happens:

1. Launching the EC2 instance: When you launch an EC2 instance and provide user data, either through the AWS Management Console, AWS CLI, or infrastructure-as-code tools like Terraform, the user data is associated with the instance.

2. User data injection: During the instance launch process, the user data is injected into the instance metadata service. This metadata service runs on the instance and provides information and configuration data about the instance.

3. Retrieving user data: Once the instance is up and running, it can retrieve the injected user data from the instance metadata service. The user data is made available to the instance through a well-known URL, typically `http://169.254.169.254/latest/user-data`.

4. Execution of user data: The user data retrieved by the instance is executed as specified. It can be a script or configuration instructions written in a specific language such as Bash, PowerShell, or any other scripting language supported by the instance’s operating system.

It’s important to note that user data is executed only once during the initial launch of the instance. If changes to the user data are made later, they won’t take effect unless the instance is stopped and started again.

• User data shell scripts must start with the #! characters and the path to the interpreter you want to read the script (commonly /bin/bash). For a great introduction on shell scripting, see the BASH Programming HOW-TO at the Linux Documentation Project (tldp.org).
• Scripts entered as user data are run as the root user, so do not use the sudo command in the script.
• The cloud-init output log file captures console output so it is easy to debug your scripts following a launch if the instance does not behave the way you intended. To view the log file, connect to the instance and open /var/log/cloud-init-output.log.
• When a user data script is processed, it is copied to and run from /var/lib/cloud/instances/instance-id/. The script is not deleted after it is run. Be sure to delete the user data scripts from /var/lib/cloud/instances/instance-id/ before you create an AMI from the instance. Otherwise, the script will exist in this directory on any instance launched from the AMI.

Retrieve instance user data in AWS

To retrieve user data from within a running instance, use the following URI.

http://169.254.169.254/latest/user-data

A request for user data returns the data as it is (content type application/octet-stream).

This example returns user data that was provided as comma-separated text.

To retrieve the user data for an instance, use the describe-instance-attribute command. With describe-instance-attribute, the AWS CLI does not perform base64 decoding of the user data for you.

aws ec2 describe-instance-attribute --instance-id i-1234567890abcdef0 --attribute userData

The following is example output with the user data base64 encoded.

{
“UserData”: {
“Value”: “IyEvYmluL2Jhc2gKeXVtIHVw”
},
“InstanceId”: “i-1234567890abcdef0”
}

On a Linux computer , use the –query option to get the encoded user data and the base64 command to decode it.

aws ec2 describe-instance-attribute --instance-id i-1234567890abcdef0 --attribute userData --output text --query "UserData.Value" | base64 --decode

Cloud-init implementation in EC2

cloud-init in AWS consists of 4 services in a target Linux system. These 4 services start cloud-init software and take user data given from AWS to install softwares and configuring softwares when an EC2 instance is launched.

By the way, user data scripts and cloud-init directives run only during the boot cycle when you first launch an instance. You can update your configuration to ensure that your user data scripts and cloud-init directives run every time you restart your instance. Here’s the how-to link of the official documentation.

$ systemctl list-unit-files –type service| grep ^cloud
cloud-config.service enabled
cloud-final.service enabled
cloud-init-local.service enabled
cloud-init.service enabled

It’s important how systemd takes precedence about the services, cloud-init related services are invoked with a few dependencies each other. systemctl list-dependencies command shows each service’s dependencies in a visualized way.

$ systemctl list-dependencies

What are dependencies of cloud-init related services and what do these services do? systemctl cat command can show service’s systemd configuration file.

$ systemctl cat cloud-init-local.service

Or you can search the configuration file location and just cat that file on your console.

$ find /etc/systemd/system -name cloud-init-local.service
/etc/systemd/system/cloud-init.target.wants/cloud-init-local.service

$ sudo cat /etc/systemd/system/cloud-init.target.wants/cloud-init-local.service

As a result, we will know 4 services start in specific order as configured in systemd configuration files. As long as I checked, here’s the order how the systemd starts the cloud-init related services. This is also explained in cloud-init official document as “Boot Stages”.

• cloud-init-local.service, runs cloud-init init –local
• cloud-init.service, runs cloud-init init
• cloud-config.service, runs cloud-init modules –mode=config
• cloud-final.service, runs cloud-init modules –mode=final

There is a configuration file of cloud-init /etc/cloud/cloud.cfg that determines what modules each service run. Now you should understand cloud-init related services and dependencies, what modules are run inside each service. This information is taken from Amazon Linux2 so the result might not be the same if you check other cloud provider’s configuration file.

$ sudo cat /etc/cloud/cloud.cfg

# WARNING: Modifications to this file may be overridden by files in
# /etc/cloud/cloud.cfg.d

users:
- default

disable_root: true
ssh_pwauth: false

mount_default_fields: [~, ~, 'auto', 'defaults,nofail', '0', '2']
resize_rootfs: noblock
resize_rootfs_tmp: /dev
ssh_deletekeys: false
ssh_genkeytypes: ~
syslog_fix_perms: ~

datasource_list: [ Ec2, None ]
repo_upgrade: security
repo_upgrade_exclude:
- kernel
- nvidia*
- cuda*

# Might interfere with ec2-net-utils
network:
config: disabled

cloud_init_modules:
- migrator
- bootcmd
- write-files
- write-metadata
- growpart
- resizefs
- set-hostname
- update-hostname
- update-etc-hosts
- rsyslog
- users-groups
- ssh
- resolv-conf

cloud_config_modules:
- disk_setup
- mounts
- locale
- set-passwords
- yum-configure
- yum-add-repo
- package-update-upgrade-install
- timezone
- disable-ec2-metadata
- runcmd

cloud_final_modules:
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change

system_info:
# This will affect which distro class gets used
distro: amazon
distro_short: amzn
default_user:
name: ec2-user
lock_passwd: true
gecos: EC2 Default User
groups: [wheel, adm, systemd-journal]
sudo: ["ALL=(ALL) NOPASSWD:ALL"]
shell: /bin/bash
paths:
cloud_dir: /var/lib/cloud
templates_dir: /etc/cloud/templates
ssh_svcname: sshd

mounts:
- [ ephemeral0, /media/ephemeral0 ]
- [ swap, none, swap, sw, "0", "0" ]
# vim:syntax=yaml

user data file is pass under /var/lib/cloud/instance and interpreted by cloud-init to execute instructions in each module. So the order of your instructions in user data (cloud config) might not be the same order when instructions are executed in an instance. It depends on 4 service dependencies and what modules are taken in each service.

$ sudo cat /var/lib/cloud/instance/user-data.txt # user data

If the user-data is a set of commands to be executed at launch time, they will also be copied over to another file named part-001 on the instance. In fact, it is this file that will be used for execution by cloud-init in the EC2 instance.

sudo cat /var/lib/cloud/instances/i-0bd6fa076d81be7af/scripts/part-001

We will explore this a bit deeper in the next section.

Checking if the startup script was executed

To verify if the startup script was executed during EC2 instance launch using cloud-init, you can follow these steps:

• Connect to your EC2 instance using SSH or any other method.
• Open the cloud-init log file using the command:
• sudo vi /var/log/cloud-init.log
• Inside the log file, you will find detailed information about the cloud-init initialization process. Look for references to “part-001” in the log. This part refers to the user data script that was provided during instance launch.

Below excerpt lists the references to part-001 in the log.

Sep 12 02:10:07 cloud-init[2421]: __init__.py[DEBUG]: {'Content-Type': 'text/x-shellscript', 'Content-Disposition': 'attachment; filename="part-001"', 'MIME-Version': '1.0'}
Sep 12 02:10:07 cloud-init[2421]: __init__.py[DEBUG]: Calling handler ShellScriptPartHandler: [['text/x-shellscript']] (text/x-shellscript, part-001, 2) with frequency once-per-instance
Sep 12 02:10:07 cloud-init[2421]: util.py[DEBUG]: Writing to /var/lib/cloud/instance/scripts/part-001 - wb: [700] 655 bytes
...
Sep 12 02:10:29 cloud-init[2910]: util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-001'] with allowed return codes [0] (shell=True, capture=False)

The 1st line in the above log identifies the set of commands to be shellscript. This is most likely done using the shebang on top. The 2nd line invokes the ShellScriptPartHandler, while the 3rd line copies over the contents of user-data to the part-001 file. The 4th line comes way later in the log, as is noted from the time logged, this is when the startup script is executed.