pyhton 操作hive数据仓库

Posted sunshinekimi

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了pyhton 操作hive数据仓库相关的知识,希望对你有一定的参考价值。

使用库Pyhive

安装:pip   install Pyhive   -i http://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com

from pyhive import hive   # or import hive
conn = hive.Connection(host=‘****‘, port=****, username=‘****‘, database=‘****‘)
cursor.execute(‘‘SELECT * FROM my_awesome_data LIMIT 10‘‘)
for i in range(****):
    sql = "INSERT INTO **** VALUES ({},‘username{}‘)".format(value, str(username))
    cursor.execute(sql)
 
 
# 下面是官网代码:
from pyhive import presto  # or import hive
cursor = presto.connect(‘localhost‘).cursor()
cursor.execute(‘SELECT * FROM my_awesome_data LIMIT 10‘)
print(cursor.fetchone())
print(cursor.fetchall())

  

impyla

安装:

pip   install Pyhive   -i http://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com

from impala.dbapi import connect 
conn = connect(host =‘****‘,port = ****)
cursor = conn.cursor()
cursor.execute(‘SELECT * FROM mytable LIMIT 100‘)
print cursor.description   # 打印结果集的schema 
results = cursor.fetchall()

  impyla交互hive 与pandas

from pyhive import hive
import pandas as pd
def LinkHive(sql_select):
    connection = hive.Connection(host=‘localhost‘)
    cur = connection.cursor()      
    cur.execute(sql_select)
    columns = [col[0] for col in cursor.description]
    result = [dict(zip(columns, row)) for row in cursor.fetchall()]
    Main = pd.DataFrame(result)
    Main.columns = columns 
    return Main
 
sql = "select * from 数据库.表名"
df  = LinkHive(sql)
或者

rom impala.dbapi import connect
from impala.util import as_pandas
conn = connect(host=‘10.161.20.11‘, port=21050)
cur = conn.cursor()
cur.execute(‘SHOW TABLES‘)
cur.execute(‘SELECT * FROM businfo‘)
data = as_pandas(cur)
print (data)
print (type(data))

 

Usage

Impyla implements the Python DB API v2.0 (PEP 249) database interface (refer to it for API details):

from impala.dbapi import connect
conn = connect(host=‘my.host.com‘, port=21050)
cursor = conn.cursor()
cursor.execute(‘SELECT * FROM mytable LIMIT 100‘)
print cursor.description  # prints the result set‘s schema
results = cursor.fetchall()

The Cursor object also exposes the iterator interface, which is buffered (controlled by cursor.arraysize):

cursor.execute(‘SELECT * FROM mytable LIMIT 100‘)
for row in cursor:
    process(row)

Furthermore the Cursor object returns you information about the columns returned in the query. This is useful to export your data as a csv file.

import csv

cursor.execute(‘SELECT * FROM mytable LIMIT 100‘)
columns = [datum[0] for datum in cursor.description]
targetfile = ‘/tmp/foo.csv‘

with open(targetfile, ‘w‘, newline=‘‘) as outcsv:
    writer = csv.writer(outcsv, delimiter=‘,‘, quotechar=‘"‘, quoting=csv.QUOTE_ALL, lineterminator=
)
    writer.writerow(columns)
    for row in cursor:
        writer.writerow(row)

You can also get back a pandas DataFrame object

from impala.util import as_pandas
df = as_pandas(cur)
# carry df through scikit-learn, for example

 

以上是关于pyhton 操作hive数据仓库的主要内容,如果未能解决你的问题,请参考以下文章

Hive数据仓库--Java代码JDBC方式访问Hive中表的数据

数据仓库 Hive 从入门到小牛

数据仓库工具——Hive表操作

数据仓库工具——Hive表操作

数据仓库工具——Hive表操作

数据仓库Hive