AWS Typescript CDK,尝试将 kinesis 流作为来源添加到 firehose

Posted

技术标签:

【中文标题】AWS Typescript CDK,尝试将 kinesis 流作为来源添加到 firehose【英文标题】:AWS Typescript CDK, Trying to add kinesis stream as a source to firehose 【发布时间】:2021-11-06 03:00:36 【问题描述】:

我正在尝试执行此流程:

DynamoDB -> Kinesis 数据流 -> Kinesis firehose -> S3 -> Redshift

我在控制台中尝试过,一切正常,但是当我尝试使用 is typescript CDK 编写代码时,我收到了这个错误

Resource handler returned message: "KinesisSourceStreamConfig is only applicable for KinesisStreamAsSource stream type. (Service: Firehose, Status Code: 400, Request ID: c3b3
9dc4-c6a7-3c31-9a00-68c863659e53, Extended Request ID: bgcK523kXRumygU0/xX5cjDlUZgF6LLwSYo7sqyfQ2GIjzW9kJGWY3ZFm9pHoZlPuLanXnLlSqOr3xhyfHIEW7GYONByP5Fe)" (RequestToken: 1c9c1
56b-c2a4-9600-28f5-52af5664e3bb, HandlerErrorCode: InvalidRequest)

这是我要实现的代码:

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as dynamodb from '@aws-cdk/aws-dynamodb';
import * as kinesis from '@aws-cdk/aws-kinesis';
import * as  firehose from '@aws-cdk/aws-kinesisfirehose';
import * as iam from "@aws-cdk/aws-iam";
import  CfnResource  from '@aws-cdk/core';

const DynamoDBArn = 'arn:aws:dynamodb:us-east-1:207627709836:table/blog-srsa-ddb-table';

export class DynamoStreamer extends cdk.Stack 
  constructor(
    scope: cdk.Construct,
    id: string
  ) 
    super(scope, id);
    const bucket = new s3.Bucket(this, 'data-lake', 
      publicReadAccess: false
    );
    const dataStream=new kinesis.Stream(this,"dataStream",shardCount:1,streamName:"stevensu_cdk_stream")

    const myTable = new dynamodb.Table(this, 'dynamo-table', 
      tableName: 'blog-srsa-ddb-table2',
      partitionKey:  name: 'id', type: dynamodb.AttributeType.STRING ,
      sortKey:  name: 'name', type: dynamodb.AttributeType.STRING ,
      removalPolicy: cdk.RemovalPolicy.RETAIN,
      kinesisStream: dataStream
    );

    const deliveryStreamRole = new iam.Role(this, `DeliveryStreamRole`, 
      assumedBy: new iam.ServicePrincipal("firehose.amazonaws.com"),
    );


    const firehoseDeliveryStream = new firehose.CfnDeliveryStream(this, 'Delivery Stream', 
      redshiftDestinationConfiguration: 
        clusterJdbcurl: 'jdbc:redshift://redshift-cluster-1.cicrdvtr7ouk.us-east-1.redshift.amazonaws.com:5439/dev',
        username: 'admin',
        password: '2131636+aA',
        roleArn: 'arn:aws:iam::207627709836:role/service-role/KinesisFirehoseServiceRole-KDS-RED-vTWqG-us-east-1-1630965194702',
        copyCommand: 
          dataTableName: 'users_test',
          copyOptions: 'format as json \'auto\''
        ,
        s3Configuration: 
          bucketArn: bucket.bucketArn,
          roleArn: deliveryStreamRole.roleArn
        ,
      ,
      kinesisStreamSourceConfiguration: 
        kinesisStreamArn: dataStream.streamArn,
        roleArn: deliveryStreamRole.roleArn
      ,

    );
  

这部分代码发生了错误:

 kinesisStreamSourceConfiguration: 
        kinesisStreamArn: dataStream.streamArn,
        roleArn: deliveryStreamRole.roleArn
      ,

【问题讨论】:

【参考方案1】:

不确定是否有更好的方法来执行此操作,但错误来自您需要在 CfnDeliveryStream 实例中包含 deliveryStreamType 键并将其设置为 kinesisStreamAsSource

另请参阅有关允许值的 cfn 文档:https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesisfirehose-deliverystream.html#cfn-kinesisfirehose-deliverystream-deliverystreamtype

【讨论】:

是的,它成功了,谢谢!我不知道为什么我还没有读到这个属性

以上是关于AWS Typescript CDK,尝试将 kinesis 流作为来源添加到 firehose的主要内容,如果未能解决你的问题,请参考以下文章

如何将生命周期策略添加到 AWS CDK Typescript 中的现有 S3 存储桶

使用 AWS CDK 在 AWS Codepipeline 中部署 Python Lambda 函数

如何在 TypeScript 中使用 aws-cdk 在每次部署时不重建 DockerImageAsset?

AWS CDK - 云观察

如何在部署前从@aws-cdk/aws-appsync 检索 schema.graphql 文件

AWS-CDK 基础初始化