以列表为列将 CSV 导入熊猫数据框

Posted 2023-03-12

【中文标题】以列表为列将 CSV 导入熊猫数据框

【英文标题】：Import CSV into pandas dataframe with list as column

【发布时间】：2020-04-02 22:51:47

【问题描述】：

您好，我有一个类似这样的“sudo”csv 文件：

id, Wave ID, Time Stamp, Number of Samples, Sample Data Array
123, 317, 1567191561.8044672, 128, 79, 17, 162, 165, 66, 3, 40, 191, 68, 56, 59, 142, 143, 7, 150, 14, 120, 172, 76, 167, 55, 27, 198, 115, 50, 87, 38, 185, 199, 74, 43, 4, 133, 114, 89, 10, 136, 46, 85, 187, 182, 170, 149, 9, 25, 128, 39, 175, 102, 45, 33, 35, 129, 156, 20, 118, 108, 72, 111, 99, 122, 140, 93, 155, 54, 63, 189, 173, 171, 134, 163, 159, 91, 193, 64, 8, 97, 34, 80, 11, 121, 145, 190, 135, 144, 31, 29, 179, 125, 116, 196, 67, 152, 112, 148, 103, 132, 106, 78, 75, 28, 174, 119, 98, 110, 86, 123, 141, 84, 83, 178, 12, 169, 113, 48, 131, 52, 180, 100, 117, 6, 77, 69, 146, 18, 157, 127, 164
123, 20,  1567191562.0020044, 16, 779, 788, 801, 817, 835, 855, 875, 895, 916, 933, 946, 956, 963, 965, 962, 952
123, 20,  1567191561.8064446, 0,
123, 317, 1567191561.8044672, 100, 132, 48, 195, 78, 190, 124, 38, 99, 87, 1, 66, 6, 106, 18, 180, 197, 59, 148, 41, 128, 125, 194, 175, 81, 21, 115, 184, 30, 71, 77, 166, 3, 107, 114, 52, 55, 186, 5, 103, 145, 19, 8, 69, 64, 122, 90, 129, 83, 165, 79, 178, 2, 14, 74, 25, 133, 147, 158, 75, 146, 20, 140, 101, 97, 10, 143, 88, 50, 168, 112, 118, 9, 137, 155, 24, 89, 144, 16, 13, 156, 196, 113, 183, 34, 120, 142, 130, 49, 86, 46, 138, 191, 192, 189, 70, 123, 159, 108, 7, 95

所以前 4 列是一个普通的 csv，然后剩下的就是一些长度的列表。 “样本数”列表示列表的长度，每行以换行符结束。

最终的数据框看起来像：

id, Wave ID, Time Stamp, Sample Data Array
123, 317, 1567191561.8044672, [1,2,3,4,5,...]
123, 317, 1567191561.8044672, [1,2,3,4,5,...]
123, 20, 1567191561.8044672, []
123, 317, 1567191563.8044672, [1,2,3,4]

有什么方法可以在 pandas 或其他东西中使用 read_csv 导入它吗？我写了一个简单的解析器，它逐行读取文件，但速度很慢。希望最后有一个熊猫数据框，这样我就可以对列进行分组/排序。

谢谢

【参考方案1】：

您可以将数据读取为 一个 列（使用保证数据中不存在的分隔符），然后拆分为 5 列。然后，您可以删除最后一列并将最后一列转换为列表：

import pandas as pd
import io
import datetime

s="""id, Wave ID, Time Stamp, Number of Samples, Sample Data Array
123, 317, 1567191561.8044672, 128, 79, 17, 162, 165, 66, 3, 40, 191, 68, 56, 59, 142, 143, 7, 150, 14, 120, 172, 76, 167, 55, 27, 198, 115, 50, 87, 38, 185, 199, 74, 43, 4, 133, 114, 89, 10, 136, 46, 85, 187, 182, 170, 149, 9, 25, 128, 39, 175, 102, 45, 33, 35, 129, 156, 20, 118, 108, 72, 111, 99, 122, 140, 93, 155, 54, 63, 189, 173, 171, 134, 163, 159, 91, 193, 64, 8, 97, 34, 80, 11, 121, 145, 190, 135, 144, 31, 29, 179, 125, 116, 196, 67, 152, 112, 148, 103, 132, 106, 78, 75, 28, 174, 119, 98, 110, 86, 123, 141, 84, 83, 178, 12, 169, 113, 48, 131, 52, 180, 100, 117, 6, 77, 69, 146, 18, 157, 127, 164
123, 20,  1567191562.0020044, 16, 779, 788, 801, 817, 835, 855, 875, 895, 916, 933, 946, 956, 963, 965, 962, 952
123, 20,  1567191561.8064446, 0,
123, 317, 1567191561.8044672, 100, 132, 48, 195, 78, 190, 124, 38, 99, 87, 1, 66, 6, 106, 18, 180, 197, 59, 148, 41, 128, 125, 194, 175, 81, 21, 115, 184, 30, 71, 77, 166, 3, 107, 114, 52, 55, 186, 5, 103, 145, 19, 8, 69, 64, 122, 90, 129, 83, 165, 79, 178, 2, 14, 74, 25, 133, 147, 158, 75, 146, 20, 140, 101, 97, 10, 143, 88, 50, 168, 112, 118, 9, 137, 155, 24, 89, 144, 16, 13, 156, 196, 113, 183, 34, 120, 142, 130, 49, 86, 46, 138, 191, 192, 189, 70, 123, 159, 108, 7, 95"""

tmp = pd.read_csv(io.StringIO(s), sep='§', engine='python')
df = tmp.iloc[:,0].str.split(', *', 4, expand=True)
df.columns = [c.strip() for c in tmp.columns[0].split(',')]
df = df.drop('Number of Samples', 1)

df.id = df.id.astype(int)
df['Wave ID'] = df['Wave ID'].astype(int)
df['Time Stamp'] = df['Time Stamp'].astype(float).map(datetime.datetime.fromtimestamp)
df['Sample Data Array'] = df['Sample Data Array'].str.split(', *')

结果：

    id  Wave ID                 Time Stamp                                  Sample Data Array
0  123      317 2019-08-30 20:59:21.804467  [79, 17, 162, 165, 66, 3, 40, 191, 68, 56, 59,...
1  123       20 2019-08-30 20:59:22.002004  [779, 788, 801, 817, 835, 855, 875, 895, 916, ...
2  123       20 2019-08-30 20:59:21.806444                                                 []
3  123      317 2019-08-30 20:59:21.804467  [132, 48, 195, 78, 190, 124, 38, 99, 87, 1, 66...

【讨论】：

大部分情况下都有效。在处理可能格式错误的行时遇到一些问题，但我认为这会让我做我想做的事情，直到我使数据输出更好