What's New in Amazon Redshift

Starting today, Amazon QuickSight customers can leverage Amazon Redshift Spectrum to visualize and analyze vast amounts of unstructured data in their Amazon S3 “data lake” – without having to load or transform any data. In addition, customers can now visualize combined data sets that include frequently accessed data stored in Amazon Redshift and bulk data sets stored cost-effectively in Amazon S3 using the same SQL syntax of Amazon Redshift. 

AWS Schema Conversion Tool (SCT) can now extract data from a Microsoft SQL Server data warehouse for direct import into Amazon Redshift. This follows the recently announced capability to convert SQL Server data warehouse schemas.

You can use the new Amazon Redshift query monitoring rules feature to set metrics-based performance boundaries for workload management (WLM) queues, and specify what action to take when a query goes beyond those boundaries. For example, for a queue that’s dedicated to short running queries, you might create a rule that aborts queries that run for more than 60 seconds. To track poorly designed queries, you might have another rule that logs queries that contain nested loops. We also provide pre-defined rule templates in the Amazon Redshift management console to get you started.

Today we announced the general availability of Amazon Redshift Spectrum, a new feature that allows you to run SQL queries against exabytes of data in Amazon Simple Storage Service (Amazon S3). With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake” — without having to load or transform any data. Redshift Spectrum applies sophisticated query optimization, scaling processing across thousands of nodes so results are fast – even with large data sets and complex queries.

We are pleased to announce that the AWS Schema Conversion Tool (SCT) can now extract data from Teradata and Oracle data warehouses for direct import into Amazon Redshift. Amazon Redshift is a fast, fully managed, petabyte scale data warehouse that was designed for the cloud from the ground up. AWS SCT will run an analysis of your data warehouse, automate the schema conversion, apply the schema to the Amazon Redshift target and extract your warehouse data, regardless of volume. You can use Amazon S3 or Amazon Snowball to move your exports to the cloud, where Amazon Redshift can natively import the data for use.

The Amazon Redshift UNLOAD command now supports Amazon S3 server-side encryption using an AWS KMS key. The UNLOAD command unloads the results of a query to one or more files on Amazon S3. You can let Amazon Redshift automatically encrypt your data files using Amazon S3 server-side encryption, or you can specify a symmetric encryption key that you manage. With this release, you can use Amazon S3 server-side encryption with a key managed by AWS KMS. In addition, the COPY command loads Amazon S3 server-side encrypted data files without requiring you to provide the key. For more information, see COPY and UNLOAD in the Amazon Redshift Database Developer Guide.

Amazon Redshift Workload Management (WLM) enables you to flexibly manage priorities within workloads so that short, fast-running queries don't get stuck in queues behind long-running queries. Today we are announcing an improved WLM experience in the Amazon Redshift console. The new features include in-line validations, simpler error messages, and more so you can easily create WLM queues and manage workloads. For more information, see Workload Management in the Amazon Redshift Database Developer Guide.

Amazon Redshift now supports Zstandard (ZSTD) column compression encoding, which delivers better data compression thereby reducing the amount of storage and I/O needed. With the addition of ZSTD, Amazon Redshift now offers seven compression encodings to choose from depending on your dataset.

You can now configure Amazon Kinesis Firehose to prepare your streaming data before it is loaded to data stores. With this new feature, you can easily convert raw streaming data from your data sources into formats required by your destination data stores, without having to build your own data processing pipelines. 

You can now use the standard Python logging module to log error and warning messages from Amazon Redshift user-defined functions (UDF). You can then query the SVL_UDF_LOG system view to retrieve the messages logged from your UDF’s and troubleshoot your UDF’s easily.

AWS is excited to announce immediate availability of the new Canada (Central) Region. Canada joins Northern Virginia, Ohio, Oregon, Northern California, and AWS GovCloud as the sixth AWS Region in North America and as the fifteenth worldwide, bringing the total number of AWS Availability Zones to 40 globally.

You can now record configuration changes to your Amazon Redshift clusters with AWS Config. The detailed configuration recorded by AWS Config includes changes made to Amazon Redshift clusters, cluster parameter groups, cluster security groups, cluster snapshots, cluster subnet groups, and event subscriptions. In addition, you can run two new managed Config Rules to check whether your Amazon Redshift clusters have the appropriate configuration and maintenance settings. These checks include verifying that your cluster database is encrypted, logging is enabled, snapshot data retention period is set appropriately, and much more. 

You can now use multibyte (UTF-8) characters in Amazon Redshift table, column, and other database object names. For more information, see Names and Identifiers in the Amazon Redshift Database Developer Guide. To support this new feature, we have updated the Amazon Redshift ODBC and JDBC drivers. The driver updates include support for multibyte characters and other enhancements. For details, see Amazon Redshift JDBC Release Notes and Amazon Redshift ODBC Release Notes.

We are excited to announce four new Amazon Redshift features that improve data compression, connection management, and data loading. 

We are excited to announce that Amazon Redshift is now available in the South America (São Paulo) Region.

AWS is excited to announce immediate availability of the new US East (Ohio) Region. Ohio joins Northern Virginia, Oregon, Northern California, and AWS GovCloud as the fifth AWS Region in North America and as the fourteenth worldwide, bringing the total number of AWS Availability Zones to 38 globally. 

You can now use time zones as part of time stamps in Amazon Redshift. The new TIMESTAMPTZ data type allows you to input timestamp values that include a time zone. Amazon Redshift automatically converts timestamps to Coordinated Universal Time (UTC) and stores the UTC values. Also, the COPY command now recognizes timestamp values with time zones in the source data and automatically converts them to UTC.  You can retrieve and display timestamps in Amazon Redshift by setting your preferred time zone at the session level, user level or client connection level.  

You can now use Amazon Redshift’s Enhanced VPC Routing to force all of your COPY and UNLOAD traffic to go through your Amazon Virtual Private Cloud (VPC).  Enhanced VPC Routing supports the use of standard VPC features such as VPC Endpoints, security groups, network ACLs, managed NAT and internet gateways, enabling you to tightly manage the flow of data between your Amazon Redshift cluster and all of your data sources. In particular, when your Amazon Redshift cluster is on a private subnet and you enable Enhanced VPC Routing, all the COPY and UNLOAD traffic between your cluster and Amazon S3 will be restricted to your VPC. You can also add a policy to your VPC endpoint to restrict unloading data only to a specific S3 bucket in your account, and monitor all COPY and UNLOAD traffic using VPC flow logs. 

AWS Cost and Usage Report data is now available for easy and quick upload directly into Amazon Redshift and Amazon Quicksight. 

You can now get up to 60% higher query throughput (as measured by standard benchmarks TPC-DS, 3TB) in Amazon Redshift as a result of improved memory allocation, which reduces the number of queries spilled to disk. This new improvement is available in version 1.0.1056 and above. Combined with the I/O and commit logic enhancement released in version 1.0.1012, it delivers up to 2 times faster performance for complex queries that spill to disk, and queries like SELECT INTO TEMP TABLE that create temporary tables. 

UNION ALL performance improvement: Business analytics often involves time-series data, which is data generated or aggregated daily, weekly, monthly or at other intervals. By storing time-series data in separate tables—one table for each time interval—and using a UNION ALL view over those tables, you can avoid potentially costly table updates. Amazon Redshift now runs UNION ALL queries up to 10 times faster if they involve joins, and up to 2 times faster if they don’t involve any joins. This performance improvement is automatic and requires no action on your part and is available in version 1.0.1057 and above. For more information about UNION ALL views and time-series tables, see Using Time-Series Tables in the Amazon Redshift Database Developer Guide.

AWS Database Migration Service now supports Amazon Redshift as a migration target. This allows you to stream data to Amazon Redshift from any of the supported sources including Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle, and SQL Server, enabling consolidation and easy analysis of data in the petabyte-scale data warehouse.

BACKUP NO option when creating tables: You can now use the BACKUP NO option with the CREATE TABLE command, and improve data loading and cluster performance. For tables, such as staging tables, which contain only transient and pre-processed data, specify BACKUP NO to save processing time when creating snapshots and restoring from snapshots. This option also reduces storage space used by snapshots.

You now have the ability to quickly and easily create new Amazon Redshift datasources in Amazon Machine Learning (Amazon ML) by copying settings from an existing Amazon Redshift datasource. A new option on the Amazon ML console allows you to select an existing Redshift datasource to copy the Redshift cluster name, database name, IAM role, SQL query and staging data location, to automatically populate these fields in the Create Datasource wizard. You can modify the settings before the new datasource is created, for example, to change the SQL query, or to specify a different IAM role to access the cluster.

We are excited to announce that Amazon Redshift is now available in the AWS China (Beijing) Region.

You can now assign one or more AWS Identity and Access Management (IAM) roles to your Amazon Redshift cluster for data loading and exporting. Amazon Redshift assumes the assigned IAM roles when you load data into your cluster using the COPY command or export data from your cluster using the UNLOAD command. It uses the resulting credentials to access other AWS services, such as Amazon S3, securely during these operations. IAM roles enhance security of your cluster and simplify data loading and exporting by eliminating the need for you to embed AWS access credentials within SQL commands. They also enable your cluster to periodically re-assume an IAM role during long-running operations. Handling of data encryption keys for COPY and UNLOAD commands remains unchanged.

You can now more easily set up or select your Identity and Access Management (IAM) role when connecting to an Amazon Redshift cluster from the Amazon Machine Learning (Amazon ML) console. To streamline the process of setting up your connection to Amazon Redshift, Amazon ML now pre-populates an interactive drop-down menu of existing IAM roles that have an Amazon ML managed policy for Amazon Redshift, and other IAM roles that you might prefer. From the Amazon ML console, you have the option of dynamically creating a new IAM role, enabling you to quickly connect to your Amazon Redshift cluster. 

You can now restore a single table from an Amazon Redshift snapshot instead of restoring the entire cluster. This new feature enables you to restore a table that you might have dropped accidentally, or reconcile data from a table that you might have updated or deleted unintentionally. To restore a table from a snapshot, simply navigate to the “Table Restore” tab for a cluster and click on the “Restore Table” button.

We are excited to announce that Amazon Redshift is now available in the AWS US West (N. California) Region.

You can now use the Amazon Machine Learning (Amazon ML) console to retrieve data from Amazon Redshift with an improved data schema conversion functionality. Data types supported by Amazon ML are not equivalent to Amazon Redshift’s supported data types, requiring a schema conversion when creating an Amazon ML datasource. Using the Amazon ML console, you will now be able to take advantage of more accurate rules for this schema conversion process, based on the data type information provided by Amazon Redshift. For more information about using Amazon Redshift with Amazon ML, please reference the documentation in the Amazon ML developer guide.

Append rows to a target table: Using the ALTER TABLE APPEND command, you can now append rows to a target table. When you issue this command, Amazon Redshift moves the data from the source table to matching columns in the target table. ALTER TABLE APPEND is usually much faster than a similar CREATE TABLE AS or INSERT INTO operation because it moves the data instead of duplicating it. This could be particularly useful in cases where you load data into a staging table, process it, and then copy the results into a production table. For more details, refer to the ALTER TABLE APPEND command. 

You can now configure Amazon Redshift Work Load Management (WLM) settings to move timed-out queries automatically to the next matching queue and restart them. The matching queue has the same Query Group or User Group as the original queue. Please see the WLM Queue Hopping section of our documentation for more detail.

Tag-based, resource-level permissions and the ability to apply default access privileges to new database objects make it easier to manage access control in Amazon Redshift. In addition, you can now use the Amazon Redshift COPY command to load data in BZIP2 compression format. More details on these features below:

We are pleased to announce two new features for Amazon Redshift, making it easier for you to control access to your clusters and expanding query capabilities.

You can now create and run scalar user-defined functions (UDFs) in Amazon Redshift. With scalar UDFs, you can perform analytics that were previously impossible or too complex for plain SQL.

We are excited to announce two new features for Amazon Redshift that make it easier to manage your clusters and expand query capabilities.

You can now configure Amazon Redshift to automatically copy snapshots of your KMS-encrypted clusters to another region of your choice. By storing a copy of your snapshots in a secondary region, you have the ability to restore your cluster from recent data if anything affects the primary region. For details on how to enable automatic cross-region backups for your KMS-encrypted clusters, refer to the Snapshots section of the Amazon Redshift management guide.

Amazon Redshift makes it easy to launch a high-performance, petabyte-scale data warehouse for less than $1000/TB/year. Get started with a free 2-month trial.

We are excited to announce that you can now ingest AVRO files directly into Amazon Redshift. Use the COPY command to ingest data in AVRO format in parallel from Amazon S3, Amazon EMR, and remote hosts (SSH clients). For details, refer to the data ingestion section of the documentation.

Amazon Redshift makes it easy to launch a high-performance, petabyte-scale data warehouse for less than $1000/TB/year. Get started with a free 2-month trial.

You can now launch Amazon Redshift clusters on second-generation Dense Storage (DS2) instances.  DS2 has twice the memory and compute power of its Dense Storage predecessor, DS1 (formerly DW1), and the same storage capacity. DS2 also supports Enhanced Networking and provides 50% more disk throughput than DS1. On average, DS2 provides 50% better performance than DS1, but is priced the same as DS1. To move from DS1 to DS2, simply restore a DS2 cluster from a snapshot of a DS1 cluster of the same size.

You can use Interleaved Sort Keys to quickly filter data without the need for indices or projections in Amazon Redshift. A table with interleaved keys arranges your data so each sort key column has equal importance. While Compound Sort Keys are more performant if you filter on the leading sort key columns, interleaved sort keys provide fast filtering no matter which sort key columns you specify in your WHERE clause. To create an interleaved sort, simply define your sort keys as INTERLEAVED in your CREATE TABLE statement.

The performance benefit of interleaved sorting increases with table size, and is most effective with highly selective queries that filter on multiple columns. For example, assume your table contains 1,000,000 blocks (1 TB per column) with an interleaved sort key of both customer ID and product ID. You will scan 1,000 blocks when you filter on a specific customer or a specific product, a 1000x increase in query speed compared to the unsorted case. If you filter on both customer and product, you will only need to scan a single block.

The interleaved sorting feature will be deployed in every region over the next seven days. The new cluster version will be 1.0.921.

For more information, please see our AWS Blog Post on Interleaved Sorting and review our documentation on Best Practices for Designing Tables.

You can now use the Amazon Mobile Analytics Auto Export feature to automatically export your app event data to Amazon Redshift. With your app event data in Amazon Redshift, you can run SQL queries, build custom dashboards, and gain deep insights about your application usage. Additionally, you can use your existing business intelligence and data warehouse tools to report on your app event data.

You can turn on the Auto Export feature from your Amazon Mobile Analytics Console. To learn more, visit our webpage and check out the documentation.

Amazon Redshift’s new custom ODBC and JDBC drivers make it easier and faster to connect to and query Amazon Redshift from your Business Intelligence (BI) tool of choice. Amazon Redshift’s JDBC driver features JDBC 4.1 and 4.0 support, a 35% performance gain over open source options, and improved memory management. Amazon Redshift’s ODBC drivers feature ODBC 3.8 support, a 6% performance gain, and better Unicode data and password handling, among other benefits. Additionally, AWS partners Informatica, Microstrategy, Pentaho, Qlik, SAS, and Tableau will be supporting these Redshift drivers with their solutions. For more information please see Connecting to a Cluster in our documentation. If you need to distribute these drivers to your customers or other third parties, please contact us at redshift-pm@amazon.com so we can arrange an appropriate license.

We are delighted to announce that Amazon Redshift is now available in the AWS GovCloud (US) Region.

Amazon Redshift has added a number of features this week and over the past month, including the ability to tag resources and cancel queries from the console, enhancements to data load and unload, and sixteen new SQL commands and functions. Amazon Redshift is a fast, easy-to-use, petabyte-scale data warehouse service that costs as little as $1,000/TB/Year. To get started for free with Amazon Redshift and partner tools, please see our Free Trial page.

AWS is delighted to announce a free trial and reserved instance price reductions for Amazon Redshift, a fast, fully-managed, petabyte-scale data warehouse for as little as $1,000/TB/Year. You can now try Amazon Redshift's SSD node for free for two months. What's more, a number of Business Intelligence and Data Integration partners are offering free trials of their own to help you ingest and report on your data in Amazon Redshift. Amazon Redshift has also reduced three year Reserved Instance prices in Japan, Singapore, and Sydney by over 25%.

Two Month Free Trial
If you are new to Amazon Redshift, you may be eligible for 750 free hours per month for two months to try the dw2.large node - enough hours to continuously run one node with 160GB of compressed SSD storage. You can also build clusters with multiple dw2.large nodes to test larger data sets, which will consume your free hours more quickly. Please see the Amazon Redshift Free Trial Page for more details.

Price Reductions in Asia Pacific
You can now purchase a three year reserved dw1.8xlarge instance in Japan for $30,000 upfront and $1.326 per hour, down 28% from $30,400 upfront and $2.288 hourly. A three-year reserved dw1.8xlarge instance in Singapore and Sydney now costs $32,000 upfront and $1.462 per hour, down 26% from $32,000 upfront and $2.40 hourly. The dw1.xlarge instance price has also decreased and continues to be one eighth the cost of dw1.8xlarge. Please see the Amazon Redshift Pricing Page for more details.

To learn more about Amazon Redshift, please visit our detail page and getting started page. To find out about recently released features, please visit the Developer Guide and the Management Guide history. To receive alerts when new features are announced, please subscribe to our feature announcements thread in the Amazon Redshift forum.

We are delighted to announce cross region ingestion and improved query functionality for Amazon Redshift, a fast, easy-to-use, petabyte-scale data warehouse service in the cloud that costs as little as $1,000/TB/Year. Customers can now COPY data directly into Amazon Redshift from an Amazon S3 bucket or Amazon DynamoDB table that is not in the same region as the Amazon Redshift cluster. We've also launched new numeric SQL functions, greatest and least, as well as new window functions, percentile_cont and percentile_disc, for more advanced analytics. These features will be rolling out to all new and existing Amazon Redshift customers over the next week, during maintenance windows.

To get started with Amazon Redshift, please visit our detail page. To learn more about recently released features, please visit the Developer Guide and the Management Guide history. To receive alerts when new features are announced, please subscribe to our feature announcements thread in the Amazon Redshift forum.