Data is essential for businesses to make informed decisions, improve operations, and innovate. Integrating data from different sources can be a complex and time-consuming process. AWS offers AWS Glue to help you integrate your data from multiple sources on serverless infrastructure for analysis, machine learning (ML), and application development. AWS Glue provides different authoring experiences for you to build data integration jobs. One of the most common options is the notebook. Data scientists tend to run queries interactively and retrieve results immediately to author data integration jobs. This interactive experience can accelerate building data integration pipelines.
Recently, AWS announced general availability of Amazon CodeWhisperer. Amazon CodeWhisperer is an AI coding companion that uses foundational models under the hood to improve developer productivity. This works by generating code suggestions in real time based on developers’ comments in natural language and prior code in their integrated development environment (IDE). AWS also announced the Amazon CodeWhisperer Jupyter extension to help Jupyter users by generating real-time, single-line, or full-function code suggestions for Python notebooks on Jupyter Lab and Amazon SageMaker Studio.
Today, we are excited to announce that AWS Glue Studio notebooks now support Amazon CodeWhisperer for AWS Glue users to improve your experience and help boost development productivity. Now, in your Glue Studio notebook, you can write a comment in natural language (in English) that outlines a specific task, such as “Create a Spark DataFrame from a json file.”. Based on this information, CodeWhisperer recommends one or more code snippets directly in the notebook that can accomplish the task. You can quickly accept the top suggestion, view more suggestions, or continue writing your own code.

This post demonstrates how the user experience on AWS Glue Studio notebook has been changed with the Amazon CodeWhisperer integration.
Prerequisites
Before going forward with this tutorial, you need to complete the following prerequisites:
- Set up AWS Glue Studio.
- Configure an AWS Identity and Access Management (IAM) role to interact with Amazon CodeWhisperer. Attach the following policy to your IAM role for the AWS Glue Studio notebook:
Getting Started
Let’s get started. Create a new AWS Glue Studio notebook job by completing the following steps:
- On the AWS Glue console, choose Notebooks under ETL jobs in the navigation pane.
- Select Jupyter Notebook and choose Create.
- For Job name, enter
codewhisperer-demo
.
- For IAM Role, select your IAM role that you configured as a prerequisite.
- Choose Start notebook.
A new notebook is created with sample cells.

At the bottom, there is a menu named CodeWhisperer. By choosing this menu, you can see the shortcuts and several options, including disabling auto-suggestions.

Let’s try your first recommendation by Amazon CodeWhisperer. Note that this post contains examples of recommendations, but you may see different code snippets recommended by Amazon CodeWhisperer.
Add a new cell and enter your comment to describe what you want to achieve. After you press Enter, the recommended code is shown.

If you press Tab, then code is chosen. If you press arrow keys, then you can select other recommendations. You can learn more in User actions.
Now let’s read a JSON file from Amazon Simple Storage Service (Amazon S3). Enter the following code comment into a notebook cell and press Enter:
CodeWhisperer will recommend a code snippet similar to the following:
Now use this method to utilize the suggested code snippet:
The proceeding code returns the following output:
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+
|birth_date| contact_details|death_date|family_name|gender|given_name| id| identifiers| image| images| links| name| other_names| sort_name|
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+
|1944-10-15| null| null| Collins| male| Michael|0005af3a-9471-4d1...|[{C000640, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Mac Collins|[{bar, Mac Collin...|Collins, Michael|
|1969-01-31|[{fax, 202-226-07...| null| Huizenga| male| Bill|00aa2dc0-bfb6-441...|[{Bill Huizenga, ...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Bill Huizenga|[{da, Bill Huizen...| Huizenga, Bill|
|1959-09-28|[{phone, 202-225-...| null| Clawson| male| Curtis|00aca284-9323-495...|[{C001102, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Curt Clawson|[{bar, Curt Claws...| Clawson, Curtis|
|1930-08-14| null|2001-10-26| Solomon| male| Gerald|00b73df5-4180-441...|[{S000675, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Gerald Solomon|[{null, Gerald B....| Solomon, Gerald|
|1960-05-28|[{fax, 202-225-42...| null| Rigell| male| Edward|00bee44f-db04-4a7...|[{R000589, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| E. Scott Rigell|[{null, Scott Rig...| Rigell, Edward|
|1951-05-20|[{twitter, MikeCr...| null| Crapo| male| Michael|00f8f12d-6e27-4a2...|[{Mike Crapo, bal...|https://theunited...|[{https://theunit...|[{Wikipedia (da),...| Mike Crapo|[{da, Mike Crapo,...| Crapo, Michael|
|1926-05-12| null| null| Hutto| male| Earl|015d77c8-6edb-4ed...|[{H001018, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Earl Hutto|[{null, Earl Dewi...| Hutto, Earl|
|1937-11-07| null|2015-11-19| Ertel| male| Allen|01679bc3-da21-482...|[{E000208, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Allen Ertel|[{null, Allen E. ...| Ertel, Allen|
|1916-09-01| null|2007-11-24| Minish| male| Joseph|018247d0-2961-423...|[{M000796, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Joseph Minish|[{bar, Joseph Min...| Minish, Joseph|
|1957-08-04|[{phone, 202-225-...| null| Andrews| male| Robert|01b100ac-192e-4b5...|[{A000210, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Robert E. Andrews|[{null, Rob Andre...| Andrews, Robert|
|1957-01-10|[{fax, 202-225-57...| null| Walden| male| Greg|01bc21bf-8939-487...|[{Greg Walden, ba...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Greg Walden|[{bar, Greg Walde...| Walden, Greg|
|1919-01-17| null|1987-11-29| Kazen| male| Abraham|02059c1e-0bdf-481...|[{K000025, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...|Abraham Kazen, Jr.|[{null, Abraham K...| Kazen, Abraham|
|1960-01-11|[{fax, 202-225-67...| null| Turner| male| Michael|020aa7dd-54ef-435...|[{Michael R. Turn...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Michael R. Turner|[{null, Mike Turn...| Turner, Michael|
|1942-06-28| null| null| Kolbe| male| James|02141651-eca2-4aa...|[{K000306, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Jim Kolbe|[{ca, Jim Kolbe, ...| Kolbe, James|
|1941-03-08|[{fax, 202-225-79...| null| Lowenthal| male| Alan|0231c6ef-6e92-49b...|[{Alan Lowenthal,...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Alan S. Lowenthal|[{null, Alan Lowe...| Lowenthal, Alan|
|1952-01-09|[{fax, 202-225-93...| null| Capuano| male| Michael|0239032f-be5c-4af...|[{Michael Capuano...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...|Michael E. Capuano|[{null, Mike Capu...|Capuano, Michael|
|1951-10-19|[{fax, 202-225-56...| null| Schrader| male| Kurt|0263f619-eff8-4e1...|[{Kurt Schrader, ...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Kurt Schrader|[{bar, Kurt Schra...| Schrader, Kurt|
|1947-06-13|[{fax, 202-225-69...| null| Nadler| male| Jerrold|029e793d-ec40-4a1...|[{N000002, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Jerrold Nadler|[{ca, Jerrold Nad...| Nadler, Jerrold|
|1970-02-03|[{fax, 202-225-82...| null| Graves| male| Tom|02b621fc-0523-449...|[{Tom Graves, bal...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Tom Graves|[{bar, Tom Graves...| Graves, Tom|
|1932-05-09| null| null| McMillan| male| John|03018f7c-f866-419...|[{M000566, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| John McMillan|[{null, Alex McMi...| McMillan, John|
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+
only showing top 20 rows
As you can see from the result, you can quickly utilize the code snippet recommended by Amazon CodeWhisperer.
Examples of code recommendations
In this section, we provide additional examples of code recommendations. Note that these are just our examples, and different code snippets may be suggested by Amazon CodeWhisperer.
Add a column with a calculation
In extract, transform, and load (ETL) use cases, it’s common to add new columns from existing columns. When we need to add columns to our Spark DataFrame, we can articulate with a high level of detail to Amazon CodeWhisperer what type of column we need added and its respective attributes:
Amazon CodeWhisperer will recommend a code snippet similar to the following:
It can be utilized as follows:
The preceding code returns the following output:
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+------------------+--------------------+
|birth_date| contact_details|death_date|family_name|gender|given_name| id| identifiers| image| images| links| name| other_names| sort_name| age| current_date|
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+------------------+--------------------+
|1944-10-15| null| null| Collins| male| Michael|0005af3a-9471-4d1...|[{C000640, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Mac Collins|[{bar, Mac Collin...|Collins, Michael| 78.71506849315068|2023-06-14 06:12:...|
|1969-01-31|[{fax, 202-226-07...| null| Huizenga| male| Bill|00aa2dc0-bfb6-441...|[{Bill Huizenga, ...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Bill Huizenga|[{da, Bill Huizen...| Huizenga, Bill| 54.4027397260274|2023-06-14 06:12:...|
|1959-09-28|[{phone, 202-225-...| null| Clawson| male| Curtis|00aca284-9323-495...|[{C001102, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Curt Clawson|[{bar, Curt Claws...| Clawson, Curtis| 63.75342465753425|2023-06-14 06:12:...|
|1930-08-14| null|2001-10-26| Solomon| male| Gerald|00b73df5-4180-441...|[{S000675, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Gerald Solomon|[{null, Gerald B....| Solomon, Gerald| 71.24931506849315|2023-06-14 06:12:...|
|1960-05-28|[{fax, 202-225-42...| null| Rigell| male| Edward|00bee44f-db04-4a7...|[{R000589, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| E. Scott Rigell|[{null, Scott Rig...| Rigell, Edward|63.087671232876716|2023-06-14 06:12:...|
|1951-05-20|[{twitter, MikeCr...| null| Crapo| male| Michael|00f8f12d-6e27-4a2...|[{Mike Crapo, bal...|https://theunited...|[{https://theunit...|[{Wikipedia (da),...| Mike Crapo|[{da, Mike Crapo,...| Crapo, Michael| 72.11780821917809|2023-06-14 06:12:...|
|1926-05-12| null| null| Hutto| male| Earl|015d77c8-6edb-4ed...|[{H001018, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Earl Hutto|[{null, Earl Dewi...| Hutto, Earl| 97.15616438356165|2023-06-14 06:12:...|
|1937-11-07| null|2015-11-19| Ertel| male| Allen|01679bc3-da21-482...|[{E000208, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Allen Ertel|[{null, Allen E. ...| Ertel, Allen| 78.08493150684932|2023-06-14 06:12:...|
|1916-09-01| null|2007-11-24| Minish| male| Joseph|018247d0-2961-423...|[{M000796, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Joseph Minish|[{bar, Joseph Min...| Minish, Joseph| 91.2904109589041|2023-06-14 06:12:...|
|1957-08-04|[{phone, 202-225-...| null| Andrews| male| Robert|01b100ac-192e-4b5...|[{A000210, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Robert E. Andrews|[{null, Rob Andre...| Andrews, Robert| 65.9041095890411|2023-06-14 06:12:...|
|1957-01-10|[{fax, 202-225-57...| null| Walden| male| Greg|01bc21bf-8939-487...|[{Greg Walden, ba...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Greg Walden|[{bar, Greg Walde...| Walden, Greg| 66.46849315068494|2023-06-14 06:12:...|
|1919-01-17| null|1987-11-29| Kazen| male| Abraham|02059c1e-0bdf-481...|[{K000025, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...|Abraham Kazen, Jr.|[{null, Abraham K...| Kazen, Abraham| 68.91232876712328|2023-06-14 06:12:...|
|1960-01-11|[{fax, 202-225-67...| null| Turner| male| Michael|020aa7dd-54ef-435...|[{Michael R. Turn...|https://theunited...|[{https://theunit...|[{Wikipedia (comm...| Michael R. Turner|[{null, Mike Turn...| Turner, Michael|63.465753424657535|2023-06-14 06:12:...|
|1942-06-28| null| null| Kolbe| male| James|02141651-eca2-4aa...|[{K000306, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Jim Kolbe|[{ca, Jim Kolbe, ...| Kolbe, James| 81.01643835616439|2023-06-14 06:12:...|
|1941-03-08|[{fax, 202-225-79...| null| Lowenthal| male| Alan|0231c6ef-6e92-49b...|[{Alan Lowenthal,...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Alan S. Lowenthal|[{null, Alan Lowe...| Lowenthal, Alan| 82.32328767123288|2023-06-14 06:12:...|
|1952-01-09|[{fax, 202-225-93...| null| Capuano| male| Michael|0239032f-be5c-4af...|[{Michael Capuano...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...|Michael E. Capuano|[{null, Mike Capu...|Capuano, Michael| 71.47671232876712|2023-06-14 06:12:...|
|1951-10-19|[{fax, 202-225-56...| null| Schrader| male| Kurt|0263f619-eff8-4e1...|[{Kurt Schrader, ...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Kurt Schrader|[{bar, Kurt Schra...| Schrader, Kurt| 71.7013698630137|2023-06-14 06:12:...|
|1947-06-13|[{fax, 202-225-69...| null| Nadler| male| Jerrold|029e793d-ec40-4a1...|[{N000002, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Jerrold Nadler|[{ca, Jerrold Nad...| Nadler, Jerrold| 76.05479452054794|2023-06-14 06:12:...|
|1970-02-03|[{fax, 202-225-82...| null| Graves| male| Tom|02b621fc-0523-449...|[{Tom Graves, bal...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| Tom Graves|[{bar, Tom Graves...| Graves, Tom|53.394520547945206|2023-06-14 06:12:...|
|1932-05-09| null| null| McMillan| male| John|03018f7c-f866-419...|[{M000566, biogui...|https://theunited...|[{https://theunit...|[{Wikipedia (de),...| John McMillan|[{null, Alex McMi...| McMillan, John| 91.15890410958905|2023-06-14 06:12:...|
+----------+--------------------+----------+-----------+------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+------------------+--------------------+----------------+------------------+--------------------+
only showing top 20 rows
Sort and extract records
You can use Amazon CodeWhisperer for sorting data and extracting records within a Spark DataFrame as well:
Amazon CodeWhisperer will recommend a code snippet similar to the following:
It can be utilized as follows:
The preceding code returns the following output:
Generate sample datasets in a Spark DataFrame
Amazon CodeWhisperer is powerful enough to generate sample Spark DataFrames as well, which can be done like so:
Amazon CodeWhisperer will recommend a code snippet similar to the following:
It can be utilized as follows:
The preceding code returns the following output:
Generate transformations in SQL
We can also use Amazon CodeWhisperer to create a code snippet for transformation in SQL and create a new table from the SQL query results (CTAS) like so:
Amazon CodeWhisperer will recommend a code snippet similar to following:
Conclusion
In this post, we demonstrated how AWS Glue Studio notebook integration with Amazon CodeWhisperer helps you build data integration jobs faster. This integration is available today in US East (N. Virginia). You can start using the AWS Glue Studio notebook with Amazon CodeWhisperer to accelerate building your data integration jobs. To get started with AWS Glue, visit AWS Glue.
Learn more
To learn more about using AWS Glue notebooks and Amazon CodeWhisperer, check out the following video.
About the authors
Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team. He works based in Tokyo, Japan. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike.
Gal Heyne is a Product Manager for AWS Glue with a strong focus on AI/ML, data engineering, and BI, and is based in California. She is passionate about developing a deep understanding of customers’ business needs and collaborating with engineers to design easy-to-use data products. In her spare time, she enjoys playing card games.