How do I enable special parameters in an AWS Glue job using AWS CloudFormation?

Last updated: 2019-05-20

I want to enable special parameters, such as --enable-metrics, for my job in AWS Glue. However, I get a template validation or "null values" error from AWS CloudFormation when I try to run my job. How can I resolve these errors?

Short Description

To enable special parameters for your job in AWS Glue, you must supply a key-value pair for the DefaultArguments property of the AWS::Glue::Job resource in AWS CloudFormation. If you supply a key only in your job definition, then AWS CloudFormation returns a validation error.

Resolution

1.    In your AWS CloudFormation template, for the DefaultArguments property of your job definition, set the value of your special parameter to an empty string.

In the following example JSON and YAML templates, the value of --enable-metrics is set to an empty string.

JSON:

    "MyJob": {
      "Type": "AWS::Glue::Job",
      "Properties": {
        "Command": {
          "Name": "glueetl",
          "ScriptLocation": "s3://my-test//test-job1"
        },
        "DefaultArguments": {
          "--job-bookmark-option": "job-bookmark-enable",
	  "--enable-metrics": ""
        },
        "ExecutionProperty": {
          "MaxConcurrentRuns": 2
        },
        "MaxRetries": 0,
        "Name": "cf-job3",
        "Role": {
          "Ref": "MyJobRole"
        }
      }
    }

YAML:

MyJob:
  Type: 'AWS::Glue::Job'
  Properties:
    Command:
      Name: glueetl
      ScriptLocation: 's3://my-test//test-job1'
    DefaultArguments:
      '--job-bookmark-option': job-bookmark-enable
      '--enable-metrics': ''
    ExecutionProperty:
      MaxConcurrentRuns: 
    MaxRetries: 0
    Name: cf-job3
    Role: !Ref MyJobRole

The empty string validates the template and launches the resource configured with the special parameter.

2.    To activate your special parameter, run your job.


Did this article help you?

Anything we could improve?


Need more help?