โšก Onwuachi Control Plane

ASG Lifecycle Hooks: Pending:Wait and Bootstrap Control

Overview

ASG lifecycle hooks intercept instance state transitions โ€” at launch or termination โ€” and hold the instance in a Waiting state until your code signals completion or the timeout fires. Without hooks, instances enter InService immediately on launch regardless of whether your application is actually ready.

Why It Matters

A fleet that routes traffic to instances before bootstrap completes causes intermittent failures that are hard to debug. Lifecycle hooks give you a guaranteed window to finish bootstrap logic โ€” pull secrets, register with monitoring, warm a cache, drain connections โ€” before the instance enters or exits the traffic pool.

Where It Fits

DOP-C02 Domain 3 โ€” Resilient Cloud Solutions

EC2 Launch | v Pending:Wait <– lifecycle hook fires here | v Your code runs (Lambda / SSM / script) | v CONTINUE signal (or timeout fires DefaultResult) | v InService

Termination hooks mirror this โ€” Terminating:Wait before the instance is destroyed.


The Big Picture

ASG Lifecycle Hook | +– Transition: autoscaling:EC2_INSTANCE_LAUNCHING +– HeartbeatTimeout: 120s (your code must signal within this window) +– GlobalTimeout: 12000s (hard ceiling, even with heartbeat extensions) +– DefaultResult: CONTINUE (safe for labs) | ABANDON (safe for production)

ABANDON on launch = instance gets terminated if bootstrap fails silently. Use this in production so broken instances never enter service.


Core Concepts

HeartbeatTimeout โ€” how long ASG waits for your code to call complete-lifecycle-action or record-lifecycle-action-heartbeat. Clock starts when the hook fires.

GlobalTimeout โ€” AWS-enforced hard ceiling (~3.3 hours). No matter how many heartbeat extensions your code sends, the hook resolves at this limit.

DefaultResult โ€” what ASG does when the timeout fires with no signal:

Signaling CONTINUE manually (what Lambda does in production):

aws autoscaling complete-lifecycle-action \
  --auto-scaling-group-name <asg-name> \
  --lifecycle-hook-name <hook-name> \
  --instance-id <instance-id> \
  --lifecycle-action-result CONTINUE

Notification targets โ€” how your code gets called:


Real-World Example

Live lab in devopslab-vpc, ASG dop-lab-asg:

  1. Added hook dop-lab-launch-hook โ€” EC2_INSTANCE_LAUNCHING, HeartbeatTimeout=120, DefaultResult=CONTINUE
  2. Scaled ASG to desired=1 โ€” replacement instance i-0f949c176bba871bb caught in Pending:Wait immediately
  3. Attempted manual complete-lifecycle-action โ€” hook had already timed out and auto-fired CONTINUE (120s elapsed during polling)
  4. Second launch attempt: instance again caught in Pending:Wait, confirmed in describe output, then auto-transitioned to InService after timeout

Key observation: complete-lifecycle-action returns No active Lifecycle Action found when the timeout has already fired โ€” the hook resolved itself via DefaultResult. This is expected behavior, not an error in the bootstrap path.


Engineering Analogy

Your systemd.sh Packer script runs systemctl enable but not systemctl start โ€” services are staged on the AMI, not running. They start on first boot at runtime. A lifecycle hook is the same pattern at the ASG layer: the instance exists and is staged, but held in Pending:Wait until external logic confirms it is ready to serve traffic.

Your platform-rehydrate.sh is exactly what a lifecycle hook would invoke in an ASG-managed fleet โ€” pull config, start services, verify health โ€” then signal CONTINUE.


Best Practices


Common Mistakes


Pro Tip

If you see No active Lifecycle Action found when calling complete-lifecycle-action, the HeartbeatTimeout already fired and DefaultResult kicked in. This is not an error โ€” it means your signal window closed. Increase HeartbeatTimeout or speed up your bootstrap logic.


Key Takeaways


Related Articles


References

System Context

โ† Back to Kb