How do I set special parameters in an AWS Glue job using AWS CloudFormation?

Last updated: 2022-08-16

I want to enable special parameters, such as --enable-metrics, for my job in AWS Glue. However, I get a template validation or "null values" error from AWS CloudFormation when I try to run my job. How do I resolve these errors?

Short description

To set special parameters for your job in AWS Glue, you must supply a key-value pair for the DefaultArguments property of the AWS::Glue::Job resource in CloudFormation. If you supply a key only in your job definition, then CloudFormation returns a validation error.

Resolution

1.    In your CloudFormation template, set the value of your special parameter to an empty string for the DefaultArguments property of your job definition.

JSON:

"MyJob": {
  "Type": "AWS::Glue::Job",
  "Properties": {
    "Command": {
      "Name": "glueetl",
      "ScriptLocation": "s3://my-test//test-job1"
    },
    "DefaultArguments": {
      "--job-bookmark-option": "job-bookmark-enable",
      "--enable-metrics": ""
    },
    "ExecutionProperty": {
      "MaxConcurrentRuns": 2
    },
    "MaxRetries": 0,
    "Name": "cf-job3",
    "Role": {
      "Ref": "MyJobRole"
    }
  }
}

YAML:

MyJob:
  Type: 'AWS::Glue::Job'
  Properties:
    Command:
      Name: glueetl
      ScriptLocation: 's3://my-test//test-job1'
    DefaultArguments:
      '--job-bookmark-option': job-bookmark-enable
      '--enable-metrics': ''
    ExecutionProperty:
      MaxConcurrentRuns: 
    MaxRetries: 0
    Name: cf-job3
    Role: !Ref MyJobRole

Note: In the preceding example JSON and YAML templates, the value of --enable-metrics is set to an empty string. The empty string validates the template and launches the resource that's configured with the special parameter.

2.    To activate your special parameter, run your job.


Did this article help?


Do you need billing or technical support?