公共数据集：Amazon Customer Reviews ( s3://amazon-reviews-pds )

Posted 2023-03-05 Laurence　

Amazon Customer Reviews ( s3://amazon-reviews-pds ) 是一份不错的单表公共数据集，本文仅关注Parquet格式数据，总计48GB，给出分类统计，建表脚本和经裁剪的子集数据表。注意：该公共数据源在中国区无法通过S3直接访问，如需在国内使用，可手动复制部分数据至国内S3。

① 数据集介绍

https://s3.amazonaws.com/amazon-reviews-pds/readme.html

② 数据集概况

本文仅关注Parquet格式数据，总计48GB，以下是各分类数据情况：

[ec2-user@ip-10-0-13-30 amazon-reviews-pds]$ du -hs * | sort -h
17M	    product_category=Gift_Card
25M	    product_category=Personal_Care_Appliances
28M	    product_category=Digital_Software
31M	    product_category=Mobile_Electronics
33M	    product_category=Major_Appliances
39M	    product_category=Digital_Video_Games
79M	    product_category=Luggage
135M	product_category=Software
200M	product_category=Furniture
216M	product_category=Video
227M	product_category=Watches
265M	product_category=Home_Entertainment
265M	product_category=Musical_Instruments
341M	product_category=Jewelry
390M	product_category=Digital_Music_Purchase
454M	product_category=Tools
492M	product_category=Baby
553M	product_category=Grocery
619M	product_category=Outdoors
622M	product_category=Camera
662M	product_category=Video_Games
666M	product_category=Lawn_and_Garden
690M	product_category=Home_Improvement
697M	product_category=Pet_Products
705M	product_category=Office_Products
784M	product_category=Digital_Video_Download
810M	product_category=Automotive
896M	product_category=Shoes
901M	product_category=Mobile_Apps
959M	product_category=Electronics
1.2G	product_category=Apparel
1.2G	product_category=Sports
1.2G	product_category=Toys
1.3G	product_category=Beauty
1.3G	product_category=Kitchen
1.4G	product_category=Health_&_Personal_Care
1.5G	product_category=Home
2.1G	product_category=PC
2.4G	product_category=Wireless
2.8G	product_category=Music
3.0G	product_category=Video_DVD
5.9G	product_category=Digital_Ebook_Purchase
11G	    product_category=Books

③ 建表语句

可以使用如下Spark SQL创建外部表，直接读取数据：

-- 建表
-- Author: https://laurence.blog.csdn.net
create external table if not exists all_reviews(
    marketplace string, 
    customer_id string, 
    review_id string, 
    product_id string, 
    product_parent string, 
    product_title string, 
    star_rating int, 
    helpful_votes int, 
    total_votes int, 
    vine string, 
    verified_purchase string, 
    review_headline string, 
    review_body string, 
    review_date date, 
    year int
)
PARTITIONED BY (product_category string)
STORED AS PARQUET
LOCATION 's3://amazon-reviews-pds/parquet/';

-- 修复分区元数据
msck repair table all_reviews;

-- 查询数据
select * from all_reviews limit 10;

④ 对数据进行裁剪

以Video_DVD品类为例，3GB的数据量，用于普通测试，同时减少列数：

-- 建表
-- Author: https://laurence.blog.csdn.net
create external table if not exists dvd_reviews(
    marketplace string, 
    review_id string, 
    customer_id string,
    product_title string,
    star_rating int,
    review_date date
)
STORED AS PARQUET
LOCATION 's3://amazon-reviews-pds/parquet/product_category=Video_DVD';

-- 查询数据
select * from dvd_reviews limit 10;

本文地址：https://laurence.blog.csdn.net/article/details/129182298，转载请注明出处！