drill 集成开源s3 存储minio

Posted rongfengliang

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了drill 集成开源s3 存储minio相关的知识,希望对你有一定的参考价值。

drill 支持s3数据的查询,同时新版的通过简单配置就可以实现minio 的集成

测试使用docker 运行drill 参考 https://www.cnblogs.com/rongfengliang/p/9846899.html,minio 安装也可以使用docker
minio bucket 创建以及文件上传可以使用UI 或者mc

配置

  • s3 配置说明
{
  "type": "file",
  "connection": "s3a://app/",
  "config": {
    "fs.s3a.access.key": "<key>",
    "fs.s3a.secret.key": "<secret>",
    "fs.s3a.endpoint": "<minio server>",
    "fs.s3a.connection.ssl.enabled": "false", // default isn‘t  ssl
    "fs.s3a.path.style.access": "true" 
  },
  "workspaces": {
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "	"
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  },
  "enabled": true
}
  • 查询
select * from s3.`root`.`user.json`
  • minio 添加数据
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}

效果

技术分享图片
技术分享图片

参考资料

https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5?_branch_match_id=520116032687566918
https://www.cnblogs.com/rongfengliang/p/9846899.html
http://drill.apache.org/docs/s3-storage-plugin/

以上是关于drill 集成开源s3 存储minio的主要内容,如果未能解决你的问题,请参考以下文章

【Minio】基于AWS S3协议搭建个人云存储服务

分片上传Minio存储服务的问题集锦[推荐收藏]

openresty + gor+minio 集成

使用 S3 Java SDK 与 S3 兼容存储通信 (minio)

MinIO 搭建

通过S3协议实现通用的文件存储服务中间件