How To: Wait for loooooooooooooong AWS operations

Many AWS operations can take a long time. Starting up an EC2 instance, taking a database backup, or deploying a CloudFormation stack. It all takes time.

Sometimes, you have to wait for these operations to finish before you can move on with your task. For example, I was recently working on running an ECS task using bash. I wanted to display the logs of the ECS task in stdout once the ECS task was finished. To do this, I would have to:

1. Start the ECS task
2. Wait for my task to finish running
3. Fetch the logs from CloudWatch

AWS CLI comes with the handy wait command for use cases like this.

AWS wait

In the AWS CLI, you can "wait" until a particular condition is satisfied.

The syntax is aws <service> wait <condition>. The wait command is implemented on services with long-running operations. If you google aws <service> wait, you can find the CLI official documentation to see which conditions you can wait for:

• EC2
• RDS
• CloudFormation
• ECS

The specific condition I was waiting for was aws ecs wait tasks-stopped. I passed in the task ID to the command, and it will wait until my task has finished running. It comes in handy when you want to make sure that a certain condition is met before proceeding with your code. However, the command does come with some drawbacks.

AWS is impatient

Unfortunately, AWS's wait command can't wait forever. There are parameters within the CLI that determine how long the CLI waits before it times out. And these parameters are not something that we can adjust.

The particular ECS task I was working on was deploying Python programs to a web application platform. During deployment, the ECS task detects the user's Python environment and installs all the necessary Python libraries on the server. This can take a long time, depending on the size of the Python program and whether the libraries are cached or not. When the deployment took too long, my task exited with a Max attempts exceeded error.

Loop it

The solution to this issue is pretty simple, really. You just have to keep waiting until it works. Kudos to antonbormotov (https://stackoverflow.com/questions/30977532/aws-command-line-interface-aws-ec2-wait-max-attempts-exceeded)

while [ "${exit_status}" != "0" ]
do
    aws ecs wait tasks-stopped --tasks "${TASK_ID}"
    exit_status="$?"
done

In this while loop, we first run the AWS wait command. Once the wait command is finished, we evaluate the exit code (by inspecting the $? variable). We only exit out of the loop if the exit code is 0 (successful).

GOTCHA: -e

When you are using this pattern, you want to be careful with the bash set -e builtin! When this is set, your script will exit immediately when a command outputs a non-zero status. In our example, the wait command will output a non-zero status and exit without completing the while loop.

Be especially careful in CI/CD environments! My GitHub Actions runner had this set and my loop didn't work.

To work around this, you can temporarily set +e to turn off the error-on-exit mode.

# Turn off error-on-exit
set +e

while [ "${exit_status}" != "0" ]
do
    aws ecs wait tasks-stopped --tasks "${TASK_ID}"
    exit_status="$?"
done

set -e

And that's it! Happy hacking.