如何使用 AWS CLI 和 jq 创建我的 AWS DMS 任务设置的备份？

2 分钟阅读

我想备份我的 AWS Database Migration Service (AWS DMS) 复制任务设置，以便稍后还原或用作其他任务的模板。我该如何操作？

简短描述

使用 AWS 命令行界面 (AWS CLI) 和 jq 命令行 JSON 处理器创建您的 AWS DMS 复制任务设置的备份。

运行 AWS CLI describe-replication-tasks 命令以获取复制任务设置。
使用 jq 命令行 JSON 处理器筛选和转换任务设置。
保存 JSON。

解决方案

**注意：**如果在运行 AWS CLI 命令时遇到错误，请确保您使用的是最新版本的 AWS CLI。

先决条件

安装 AWS CLI。
下载 jq 工具。

获取复制任务设置

要获取 AWS DMS 复制任务设置，请使用 AWS CLI 运行 describe-replication-tasks 命令并使用 --filters 标志。通过使用 filters 标志，您只返回与以下项目之一匹配的任务：

复制任务 ID
复制任务 ARN

运行此查询，使用复制任务 ID 获取 AWS DMS 任务的复制任务设置。将 <dms_task_id> 替换为 AWS DMS 复制任务 ID。

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_id> --output json

运行此查询，使用复制任务 ARN 获取 AWS DMS 任务的复制任务设置。将 <dms_task_arn> 替换为 AWS DMS 复制任务 ARN。

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_arn> --output json

使用 jq 筛选和转换 JSON 输出

接下来，筛选和转换上一步的 JSON 输出。您需要筛选和转换此 JSON，使其兼容以后的恢复。

1. 使用 jq 命令行 JSON 处理器对输出 JSON 进行以下修改：

添加键 CdcStartPosition，然后将其设置为 RecoveryCheckpoint.。
移除以下数据：

ReplicationTaskArn
ReplicationTaskStartDate
ReplicationTaskStats
Status
StopReason
ReplicationTaskCreationDate
CloudWatchLogGroup
CloudWatchLogStream
RecoveryCheckpoint

2. 运行以下命令之一将输出 JSON 传送到 jq，然后筛选和转换复制任务数据：

使用 AWS DMS 复制任务 ID：

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_id> --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([
    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])
    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'

使用 AWS DMS 复制任务 ARN：

aws dms describe-replication-tasks --filter Name=replication-task-arn,Values=<dms-task-arn> --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([
    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])
    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'

将输出保存到 JSON 文件

要将生成的输出另存为备份，请将其重定向到 JSON 文件。例如，此命令将任务 ID 为 dms_original_task 的 AWS DMS 任务的任务设置保存到名为 task_backup.json 的 JSON 文件中：

aws dms describe-replication-tasks --filter Name=replication-task-id,Values=original-dms-task --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([ ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]]) | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring' > task_backup.json

生成的 JSON 文件显示类似于以下内容的输出：

{
  "ReplicationTaskIdentifier": "original-dms-task",
  "SourceEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:QKMQN2TCULCPYIK3CMGF2CRDC5DIWZ433BCKHZY",
  "TargetEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:6LEFAZCDPQD2HWHJIXJYUU4PGMTKA2AKKFN4KCA",
  "ReplicationInstanceArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:rep:C4CAB7ANKQU2AU7WFH4BYCJSLPS3YYUGCTHL3UA",
  "MigrationType": "full-load-and-cdc",
  "TableMappings": "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"293111713\",\"rule-name\":\"293111713\",\"object-locator\":{\"schema-name\":\"%\",\"table-name\":\"%\"},\"rule-action\":\"include\",\"filters\":[]}]}",
  "ReplicationTaskSettings": "{\"Logging\":{\"EnableLogging\":true,\"LogComponents\":[{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TRANSFORMATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_UNLOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"IO\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_LOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"PERFORMANCE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_CAPTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SORTER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"REST_SERVER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"VALIDATOR_EXT\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_APPLY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TASK_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TABLES_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"METADATA_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_FACTORY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMON\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"ADDONS\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"DATA_STRUCTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMUNICATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_TRANSFER\"}]},\"StreamBufferSettings\":{\"StreamBufferCount\":3,\"CtrlStreamBufferSizeInMB\":5,\"StreamBufferSizeInMB\":8},\"ErrorBehavior\":{\"FailOnNoTablesCaptured\":true,\"ApplyErrorUpdatePolicy\":\"LOG_ERROR\",\"FailOnTransactionConsistencyBreached\":false,\"RecoverableErrorThrottlingMax\":1800,\"DataErrorEscalationPolicy\":\"SUSPEND_TABLE\",\"ApplyErrorEscalationCount\":0,\"RecoverableErrorStopRetryAfterThrottlingMax\":true,\"RecoverableErrorThrottling\":true,\"ApplyErrorFailOnTruncationDdl\":false,\"DataTruncationErrorPolicy\":\"LOG_ERROR\",\"ApplyErrorInsertPolicy\":\"LOG_ERROR\",\"EventErrorPolicy\":\"IGNORE\",\"ApplyErrorEscalationPolicy\":\"LOG_ERROR\",\"RecoverableErrorCount\":-1,\"DataErrorEscalationCount\":0,\"TableErrorEscalationPolicy\":\"STOP_TASK\",\"RecoverableErrorInterval\":5,\"ApplyErrorDeletePolicy\":\"IGNORE_RECORD\",\"TableErrorEscalationCount\":0,\"FullLoadIgnoreConflicts\":true,\"DataErrorPolicy\":\"LOG_ERROR\",\"TableErrorPolicy\":\"SUSPEND_TABLE\"},\"ValidationSettings\":{\"ValidationPartialLobSize\":0,\"PartitionSize\":10000,\"RecordFailureDelayLimitInMinutes\":0,\"SkipLobColumns\":false,\"FailureMaxCount\":10000,\"HandleCollationDiff\":false,\"ValidationQueryCdcDelaySeconds\":0,\"ValidationMode\":\"ROW_LEVEL\",\"TableFailureMaxCount\":1000,\"RecordFailureDelayInMinutes\":5,\"MaxKeyColumnSize\":8096,\"EnableValidation\":true,\"ThreadCount\":5,\"RecordSuspendDelayInMinutes\":30,\"ValidationOnly\":false},\"TTSettings\":{\"TTS3Settings\":null,\"TTRecordSettings\":null,\"EnableTT\":false},\"FullLoadSettings\":{\"CommitRate\":10000,\"StopTaskCachedChangesApplied\":false,\"StopTaskCachedChangesNotApplied\":false,\"MaxFullLoadSubTasks\":8,\"TransactionConsistencyTimeout\":600,\"CreatePkAfterFullLoad\":false,\"TargetTablePrepMode\":\"DROP_AND_CREATE\"},\"TargetMetadata\":{\"ParallelApplyBufferSize\":0,\"ParallelApplyQueuesPerThread\":0,\"ParallelApplyThreads\":0,\"TargetSchema\":\"\",\"InlineLobMaxSize\":0,\"ParallelLoadQueuesPerThread\":0,\"SupportLobs\":true,\"LobChunkSize\":0,\"TaskRecoveryTableEnabled\":false,\"ParallelLoadThreads\":0,\"LobMaxSize\":32,\"BatchApplyEnabled\":false,\"FullLobMode\":false,\"LimitedSizeLobMode\":true,\"LoadMaxFileSize\":0,\"ParallelLoadBufferSize\":0},\"BeforeImageSettings\":null,\"ControlTablesSettings\":{\"historyTimeslotInMinutes\":5,\"HistoryTimeslotInMinutes\":5,\"StatusTableEnabled\":false,\"SuspendedTablesTableEnabled\":false,\"HistoryTableEnabled\":false,\"ControlSchema\":\"\",\"FullLoadExceptionTableEnabled\":false},\"LoopbackPreventionSettings\":null,\"CharacterSetSettings\":null,\"FailTaskWhenCleanTaskResourceFailed\":false,\"ChangeProcessingTuning\":{\"StatementCacheSize\":50,\"CommitTimeout\":1,\"BatchApplyPreserveTransaction\":true,\"BatchApplyTimeoutMin\":1,\"BatchSplitSize\":0,\"BatchApplyTimeoutMax\":30,\"MinTransactionSize\":1000,\"MemoryKeepTime\":60,\"BatchApplyMemoryLimit\":500,\"MemoryLimitTotal\":1024},\"ChangeProcessingDdlHandlingPolicy\":{\"HandleSourceTableDropped\":true,\"HandleSourceTableTruncated\":true,\"HandleSourceTableAltered\":true},\"PostProcessingRules\":null}",
  "CdcStartPosition": "checkpoint:V1#102#00000000/B803EBA0#0#0#*#0#277"
}

现在，您可以使用 create-replication-task AWS CLI 命令编辑和还原 JSON 文件。例如，此命令使用 task_backup.json 文件创建任务 ID 为 restored-dms-task 的 AWS DMS 复制任务：

aws dms create-replication-task --replication-task-identifier restored-dms-task --cli-input-json file://task_backup.json

注意：在命令中包含 --replication-task-identifier 时，就会覆盖 task_backup.json 文件中的 --replication-task-identifier 值。