Why can't I find my DataSync task logs in the CloudWatch log group?

3 minute read
1

I ran an AWS DataSync task, but I can't find the logs in the relevant Amazon CloudWatch log group. How can I troubleshoot this?

Resolution

Confirm that CloudWatch Logs has a resource policy that allows DataSync to upload logs. Follow these steps to review the CloudWatch Logs resource policies in the AWS Region of your DataSync agent:

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

1.    Run the describe-resource-policies command using the AWS Command Line Interface (AWS CLI):

aws logs describe-resource-policies --region us-east-1

Note: Each AWS account is allowed up to 10 resource policies per Region for CloudWatch Logs. If you exceed this limit, then you receive an error message when creating your resource policy.

2.    Review the output of the command. If a resource policy isn't set up, then the output is similar to the following:

{
  "resourcePolicies": []
}

Important: Confirm that the resource policy for DataSync is enabled in the correct AWS Region. The policy must be in the same Region as the DataSync agent that you're using.

Follow these steps to create a resource policy that grants DataSync permissions for uploading logs:

1.    Create a JSON file that grants DataSync the minimum permissions for uploading logs:

{
  "Statement": [
    {
      "Sid": "DataSyncLogsToCloudWatchLogs",
      "Effect": "Allow",
      "Action": [
        "logs:PutLogEvents",
        "logs:CreateLogStream"
      ],
      "Principal": {
        "Service": "datasync.amazonaws.com"
      },
      "Resource": "*"
    }
  ],
  "Version": "2012-10-17"
}

You can name the file policy.json.

2.    Run the put-resource-policy command using the AWS CLI to create a resource policy using the JSON file:

aws logs put-resource-policy --policy-name trustDataSync --policy-document file://policy.json --region <Region>

Important: Set the AWS Region of your DataSync agent as the value for --region.

3.    Run the describe-resource-policies command to confirm that the resource policy was created:

aws logs describe-resource-policies --region <Region>

Note: Each AWS account is allowed up to 10 resource policies per Region for CloudWatch Logs. If you exceed this limit, you receive an error message when creating your resource policy. Use the put-resource-policy command to verify if you've reached the limit.

4.    After you create the resource policy, the command output is similar to the following:

{
  "resourcePolicies": [
    {
      "policyName": "trustDataSync",
      "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"DataSyncLogsToCloudWatchLogs\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"datasync.amazonaws.com\"},\"Action\":[\"logs:PutLogEvents\",\"logs:CreateLogStream\"],\"Resource\":\"*\"}]}",
      "lastUpdatedTime": 1577448776606
    }
  ]
}

5.    To test the resource policy, run a DataSync task. A few minutes after the task runs, confirm that you're seeing the log stream from the relevant CloudWatch log group.


Related information

Overview of managing access permissions to your CloudWatch Logs resources

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago
2 Comments

Hi,

I followed these steps and now my Datasync tasks are pushing logs to cloudwatch when they are executed manually. But they aren't creating log steams and pushing logs when triggered automatically based on their schedule. Any reason why it's not working for scheduled tasks?

replied 7 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 7 months ago