如何将弹性搜索索引的响应保存到 s3 存储桶中

Posted 2023-03-22

【中文标题】如何将弹性搜索索引的响应保存到 s3 存储桶中

【英文标题】：How to save the response from the elastic search index into s3 bucket

【发布时间】：2020-10-26 20:07:26

【问题描述】：

如何在my-index下面生成

在本地创建一个环境

使用 pip 安装必要的库（elasticsearch、requests、requests_aws4auth、boto3）

使用 lambda_function.py 在env\Lib\site-packages\ 中创建文件并添加以下代码

压缩上述文件夹并将其命名为 lambda_function.zip 并上传到 lambda 函数中，您可以在其中创建具有必要 IAM 角色的函数

 import boto3
 from requests_aws4auth import AWS4Auth
 from elasticsearch import Elasticsearch, RequestsHttpConnection
 session = boto3.session.Session()
 credentials = session.get_credentials()

 awsauth = AWS4Auth(credentials.access_key,
                    credentials.secret_key,
                    session.region_name, 'es',
                    session_token=credentials.token)
 es = Elasticsearch(
     ['https://search-testelastic-2276kyz2u4l3basec63onfq73a.us-east-1.es.amazonaws.com'],
     http_auth=awsauth,
     use_ssl=True,
     verify_certs=True,
     connection_class=RequestsHttpConnection
 )


 def lambda_handler(event, context):
     es.cluster.health()
     es.indices.create(index='my-index', ignore=400)
     r = ['Name': 'Dr. Christopher DeSimone', 'Specialised and Location': 'Health',
  'Name': 'Dr. Tajwar Aamir (Aamir)', 'Specialised and Location': 'Health']
     for e in enumerate(r):
          es.index(index="my-index", body=e[1])

回复如下

"took":2,"timed_out":false,"_shards":"total":1,"successful":1,"skipped":0,"failed":0,"hits":"total":"value":3,"relation":"eq","max_score":1.0,"hits":["_index":"my-index_1","_type":"_doc","_id":"elqrJHMB10jKFvejVaNM","_score":1.0,"_source":"Name":"Dr. Christopher DeSimone","Specialised and Location":"Health","_index":"my-index_1","_type":"_doc","_id":"e1qrJHMB10jKFvejVqMK","_score":1.0,"_source":"Name":"Dr. Tajwar Aamir (Aamir)","Specialised and Location":"Health","_index":"my-index_1","_type":"_doc","_id":"fFqrJHMB10jKFvejVqMR","_score":1.0,"_source":"Name":"Dr. Bernard M. Aaron","Specialised and Location":"Health"]

如何将上述响应以 json 格式保存在 s3 存储桶的文件夹中

存储桶名称 = test20220elastic

【参考方案1】：

您可以重用您的 session 对象来创建 S3 资源：

es = ...
s3 = session.resource('s3')
bucket = s3.Bucket('test20220elastic')

def lambda_handler(event, context):
    ...
    for e in enumerate(r):
        result = es.index(index="my-index", body=e[1])
        bucket.put_object(Body=json.dumps(result), Key="my_folder/my_result.json",
                          ContentType="application/json")

不过，您可能希望为每个结果建立一个不同的键名。

【讨论】：

你能在下面回答***.com/questions/62808166/…