[AWS] ElasticSearch, Sync data between DynamoDB and ElasticSearch by using DyanmoDB Stream

Posted Answer1215

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了[AWS] ElasticSearch, Sync data between DynamoDB and ElasticSearch by using DyanmoDB Stream相关的知识,希望对你有一定的参考价值。

Full-Text Search

  • Find images by names
  • What this usually means
    • Support search by any field
    • Tolerate typos
    • Support highlight
    • Rank search result

So when we upload a image, we save the information into DynamoDB already.

Now if we want to support full-text search, we also need to save the data into ElasticSearch.

Which means we will have two databases. And we need to keep both database in sync! Sounds simple but actually NOT AT ALL!

  • Command and Query Responsibillity Segregation (CQRS)
    • Command means a write operation
      • Command means a write operation
      • Query means a read operation

Decouple DynamoDB Updates

  • Ideally
    • Decouple updating both datastores
    • We should have a stream of updates from DynamoDB
    • Asynchronously execute each update
  • Why is this better?
    • User is decoupled from the updating ElasticSearch operation
    • ElasticSearch will be updated eventally when DynamoDB finish udpating
    • Worst case scenario, Elasticsearch is updated a bit later.

Solution

  • Kinesis
  • DynamoDB Stream

 

Kinesis

When image uploaded, will write to Kinesis stream.

  • Then one `write-to-dynamodb` lambda function will read from stream and add record to DynamoDB
  • Another lambda function`write-to-es` will also read from Kinesis stream and write record to Elasticsearch

DynamoDB Stream

When we write data to DynamoDB

  • We will config DynamoDB Stream
  • and `write-to-es` lambda function will read from the DynamoDB Stream
  • Then write data to ElasticSearch

Scaling a Data Stream

A data stream is split into shards, and every shard stores a subset of data. Different shards can spread across multiple hosts, which allows to scale up a data stream.

Every record that is added to a data stream has key and data, A key of a data record is used to decide what shard it will be written to. All items with the same key will be written to the same shard and will be available in the same order they were added to a stream.

A single stream is splited into multiple parts, each part is called a shard

The way a shard is selected: Every record has a key, key is hased, those hased key is used to select a shard.

 

How to Enable Stream

In serverless.yml: resources

 

DynamoDB JSON

Read: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.LowLevelAPI.html#Programming.LowLevelAPI.ResponseFormat

 

Elasticsearch

You can read more about Elasticsearch in the official documentation.

You can download and run Elasticsearch yourself, or use one of the managed Elasticsearch solutions. AWS provides a managed Elasticsearch service that we will use in this course, but there are other companies like Elastic that provide similar service.

Here is how to create an Elasticsearch client:

import * as elasticsearch from \'elasticsearch\'
 import * as httpAwsEs from \'http-aws-es\'

 const esHost = process.env.ES_ENDPOINT

 const es = new elasticsearch.Client({
   hosts: [ esHost ],
   connectionClass: httpAwsEs
 })

Here is how to store a document in Elasticsearch

await es.index({
   index: \'images-index\',
   type: \'images\',
   id: \'id\', // Document ID
   body: {  // Document to store
     title: \'title\',
     imageUrl: \'https://example.com/image.png\'
   }
 })

 

Kibana Access 

AWS provides multiple ways to restrict access to a Kibana dashboard. One option to restrict access to Kibana would be to use Amazon Cognito, a service for authentication, authorization, and user management. 

 

以上是关于[AWS] ElasticSearch, Sync data between DynamoDB and ElasticSearch by using DyanmoDB Stream的主要内容,如果未能解决你的问题,请参考以下文章

`aws s3 sync` 如何确定文件是不是已更新?

由于 MQTT 错误,AWS App Sync 订阅不起作用

使用 AWS Amplify 和 App Sync 与使用 Mongodb 的现有节点服务器

签署 ElasticSearch AWS 调用

无法关闭 AWS 上的 ElasticSearch 索引?

如何在 graphQL 中编写 JOIN 或从多种类型中获取结果 - AWS App sync iOS