Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换
Posted rongfengliang
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换相关的知识，希望对你有一定的参考价值。
Singer 可以方便的进行数据的etl 处理，我们可以处理的数据可以是api 接口，也可以是数据库数据，或者
是文件
备注： 测试使用docker-compose 运行&&提供数据库内容,使用virtualenv && python 3.5 以及以上
环境准备
- docker-compose 文件
version: "3"
services:
gogs-service:
image: gogs/gogs
ports:
- "10022:22"
- "10080:3000"
image: mysql:5.7.16
ports:
- 3306:3306
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
environment:
MYSQL_ROOT_PASSWORD: dalongrong
MYSQL_DATABASE: gogs
MYSQL_USER: gogs
MYSQL_PASSWORD: dalongrong
TZ: Asia/Shanghai
postgres:
image: postgres:9.6.11
ports:
- "5432:5432"
environment:
- "POSTGRES_PASSWORD:dalong"
- postgres target 配置
target.json
{
"host": "localhost",
"port": 5432,
"dbname": "postgres",
"user": "postgres",
"password": "postgres",
"schema": "public"
}
- 创建gitlab virtualenv
virtualenv gitlab
source ./gitlab/bin/activate
pip install tap-gitlab
- 创建access_token
从gitlab 官方网站创建即可 - gitlab tap 配置文件
格式如下,因为隐私没有暴露：
{
"api_url": "https://gitlab.com/api/v4",
"private_token": "xxxxxxx",
"groups": "",
"projects": "",
"start_date":"2010-01-01T00:00:00Z"
}
运行&&效果
- 运行
./gitlab/bin/tap-gitlab -c gitlab.json | ./postgres/bin/target-postgres -c target.json
- 效果
INFO Starting sync
INFO GET https://gitlab.com/api/v4/projects/dalongrong%2Fdemoapp?private_token=XXXXXXXXX
INFO Table ‘projects‘ does not exist. Creating
_url" character varying, "builds_enabled" boolean, "container_registry_enabled" boolean, "created_at" timestamp without time zone, "creator_
id" bigint, "default_branch" character varying, "description" character varying, "forks_count" bigint, "http_url_to_repo" character varying,
"id" bigint, "issues_enabled" boolean, "last_activity_at" timestamp without time zone, "lfs_enabled" boolean, "merge_requests_enabled" bool
ean, "name" character varying, "name_with_namespace" character varying, "namespace__id" bigint, "namespace__kind" character varying, "namespace__name" character varying, "namespace__path" character varying, "only_allow_merge_if_all_discussions_are_resolved" boolean, "only_allow_merge_if_build_succeeds" boolean, "open_issues_count" bigint, "owner_id" bigint, "path" character varying, "path_with_namespace" character varying, "public" boolean, "public_builds" boolean, "request_access_enabled" boolean, "shared_runners_enabled" boolean, "shared_with_groups" jsonb, "snippets_enabled" boolean, "ssh_url_to_repo" character varying, "star_count" bigint, "tag_list" jsonb, "visibility_level" bigint, "web_url" character varying, "wiki_enabled" boolean, PRIMARY KEY ("id"))
INFO Table ‘branches‘ does not exist. Creating
INFO Table ‘commits‘ does not exist. Creating
INFO Table ‘issues‘ does not exist. Creating
说明
使用类似的方法，我们也可以转换github 的以及jira 等基于api 开发的模型
参考资料
https://github.com/singer-io/tap-gitlab
https://github.com/rongfengliang/singer-mysql2postges-demo
以上是关于Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换的主要内容，如果未能解决你的问题，请参考以下文章
Singer 学习六 运行&&开发tapstargets (一 taps 运行说明)