如何获得猪行的平均值

Posted 2023-03-01

通过以下处理

REGISTER 's3://jmh-dtg-2016/jeon_dtg/test.py' USING jython as test;
raw01  = LOAD 's3://jmh-dtg-2016/jeon_dtg/test_pig.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'NO_MULTILINE', 'UNIX', 'SKIP_INPUT_HEADER');
raw02 = FOREACH raw01 GENERATE (chararray)$2 as date, (chararray)$3 as code, (chararray)$4 as car_num, (chararray)$5 as pre_time, (FLOAT)$8 as vel, (chararray)$18 as link_id;
raw03 = GROUP raw02 BY (link_id, car_num);
raw04 = FOREACH raw03 GENERATE group, test.my_fun(raw02.vel) AS val;
dump raw04;

得到这些结果

enter image description here

我想获得每行的平均值。总之，我想要这样的结果：

(39.0),(45.0)) -> 42 
(1.0)) -> 1

这是我使用的python函数。

@outputSchema('num01:float')
def my_fun(data01):
    a = data01
    b = sorted(a)
    c = int((len(b)/100.0) * 10.0)
    d = int((len(b)/100.0) * 90.0)
    e = b[c:d]
    return e

而且不可能

 @outputSchema('num01:float')
    def my_fun(data01):
        a = data01
        b = sorted(a)
        c = int((len(b)/100.0) * 10.0)
        d = int((len(b)/100.0) * 90.0)
        e = b[c:d]
        return sum(e)

[请帮帮我..

答案

听起来您只需要从一袋值中获取平均值？如我错了请纠正我。 PIG运算符AVG应该做到这一点，并且比Python UDF更具性能。

raw04 = FOREACH raw03 GENERATE group, AVG(raw02.vel) AS val;