How can I get my Lambda function to work correctly with my Amazon EC2 instance or the AWS Instance Scheduler?

Last updated: 2021-09-08

I use an Amazon Elastic Compute Cloud (Amazon EC2) instance or the AWS Instance Scheduler with an AWS Lambda function. But my Lambda function times out, fails to invoke, or throws errors.

Short description

The AWS Instance Scheduler can cause your Lambda function to work incorrectly for the following reasons:

  • You get an SDK timeout. To resolve this issue, complete the steps in the Fix SDK timeouts section.
  • You get a function timeout. To resolve this issue, complete the steps in the Prevent function timeouts section.
  • An Amazon CloudWatch Events rule didn't trigger. To resolve this issue, complete the steps in the Verify that your scheduled rule was invoked section.
  • Your function is throttling. To resolve this issue, complete the steps in the Prevent function throttling section.
  • You get duplicate Lambda invokes. To resolve this issue, complete the steps in the Prevent errors by using idempotency section.

Note: The AWS Instance Scheduler runs Lambda functions at set intervals using a CloudWatch Events rule to schedule the Lambda function. This results in an asynchronous invocation of the Lambda function, which has an impact on how function errors are handled. If the function fails due to an error when running, the same event is retried up to two more times by default. Then, the event is discarded. In the event of throttles, the event can be queued for up to six hours before the event is discarded.

Resolution

Fix SDK timeouts

The default SDK timeout isn't overridden in earlier versions of the AWS Instance Scheduler. If the API call to start or stop an instance takes longer than 60 seconds to respond in the earlier versions, then the request times out.

To resolve this issue, get the latest version of the AWS Instance Scheduler, or modify the read_timeout value of your Boto3 Config object (on the Boto3 website). For example:

import boto3
from botocore.config import Config

client_config = Config(read_timeout=890)

ec2 = boto3.client('ec2', config=client_config)

Prevent function timeouts

If your Lambda function timed out, check if the function is waiting to verify that the EC2 instance has launched. For more information, see Why can't I start or launch my EC2 instance?

Instances can take a few minutes to reach the RUNNING state. Your function can time out if the time to reach the RUNNING state is longer than the configured timeout for the function. To resolve this issue, increase the timeout for your function by modifying the Timeout value of 300 (5 minutes) in your AWS CloudFormation template. You must have enough time for the instance to launch before the function times out. For example:

"Runtime": "python3.7",
"Timeout": 300,
"TracingConfig": {
    "Mode": "Active"
}

Note: Lambda functions have a maximum configurable timeout value of 15 minutes.

If your function is in a virtual private cloud, you must allow your function to access the internet to make calls to the Amazon EC2 API.

Verify that your scheduled rule was invoked

To troubleshoot a Lambda function that's not being invoked by CloudWatch Events, see Why wasn't my Lambda function triggered by my CloudWatch Events rule?

Prevent function throttling

To prevent function throttling, see How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?

For asynchronous invokes, Lambda throttles on your account can affect the ability of your function to be invoked. All asynchronous invocations are sent to an event queue before they are invoked. If another function received a high number of asynchronous invokes too, then your event queue can get backlogged. This backlog doesn't mean that your function failed to invoke. This backlog means that the event has been delayed. You can see multiple invokes after the event queue backlog is cleared. By default, events in your asynchronous event queue remain in the asynchronous event queue for up to six hours.

Prevent errors by using idempotency

To prevent errors from duplicate invokes, make your Lambda function idempotent when it's configured with an asynchronous invoke.

Keep in mind that asynchronous Lambda invokes can occur more than once due to eventual consistency with the event queue.


Did this article help?


Do you need billing or technical support?