简单示例用例(Simple Example Use Cases)--hive GettingStarted用例翻译

Posted 2020-09-25 ChavinKing

1、MovieLens User Ratings

First, create a table with tab-delimited text file format:

首先，创建一个通过tab分隔的表：

CREATE TABLE u_data (   userid INT,   movieid INT,   rating INT,   unixtime STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘ STORED AS TEXTFILE;

Then, download the data files from MovieLens 100k on the GroupLens datasets page (which also has a README.txt file and index of unzipped files):

然后，下载数据文件从下面方法：

wget http://files.grouplens.org/datasets/movielens/ml-100k.zip or: curl --remote-name http://files.grouplens.org/datasets/movielens/ml-100k.zip

Note:  If the link to GroupLens datasets does not work, please report it on HIVE-5341 or send a message to the [email protected] mailing list.

Unzip the data files:

解压缩这个文件：

unzip ml-100k.zip

And load u.data into the table that was just created:

并且加载数据到刚刚创建的u_data表中：

LOAD DATA LOCAL INPATH ‘<path>/u.data‘ OVERWRITE INTO TABLE u_data;

Count the number of rows in table u_data:

统计表u_data的行数：

SELECT COUNT(*) FROM u_data;

Note that for older versions of Hive which don‘t include HIVE-287, you‘ll need to use COUNT(1) in place of COUNT(*).

Now we can do some complex data analysis on the table u_data:

现在我们可以做一些复杂的数据分析针对表u_data：

Create weekday_mapper.py:

import sys import datetime for line in sys.stdin:   line = line.strip()   userid, movieid, rating, unixtime = line.split(‘\t‘)   weekday = datetime.datetime.fromtimestamp(float(unixtime)).isoweekday()   print ‘\t‘.join([userid, movieid, rating, str(weekday)])

Use the mapper script:

使用这个脚本：

CREATE TABLE u_data_new (   userid INT,   movieid INT,   rating INT,   weekday INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘;

add FILE weekday_mapper.py;

INSERT OVERWRITE TABLE u_data_new SELECT   TRANSFORM (userid, movieid, rating, unixtime)   USING ‘python weekday_mapper.py‘   AS (userid, movieid, rating, weekday) FROM u_data; 解释：这里通过python脚本清洗表u_data中数据，使用python脚本通过 TRANSFORM (userid, movieid, rating, unixtime)   --输入字段   USING ‘python weekday_mapper.py‘              --脚本处理   AS (userid, movieid, rating, weekday)         --输出字段

SELECT weekday, COUNT(*) FROM u_data_new GROUP BY weekday;

Note that if you‘re using Hive 0.5.0 or earlier you will need to use COUNT(1) in place of COUNT(*).

2、Apache Weblog Data

The format of Apache weblog is customizable, while most webmasters use the default.
For default Apache weblog, we can create a table with the following command.

More about RegexSerDe can be found here in HIVE-662 and HIVE-1719.

CREATE TABLE apachelog ( host STRING, identity STRING, user STRING, time STRING, request STRING, status STRING, size STRING, referer STRING, agent STRING) ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.RegexSerDe‘ WITH SERDEPROPERTIES ( "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[^\\]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\".*\") ([^ \"]*|\".*\"))?" ) STORED AS TEXTFILE;