Pandas 按行中的值和其他列中的值在行之间进行差异

Posted 2023-03-12

【中文标题】Pandas 按行中的值和其他列中的值在行之间进行差异

【英文标题】：Pandas diff between rows by value in row and value in other columns

【发布时间】：2019-04-13 03:17:54

【问题描述】：

我获得了与员工签订合同的历史数据框。 员工可能会多次出现在记录中。 目标文件代表 3 种类型。 目标是计算特定员工在公司工作的时间。 我找到了解决方案。但代码的执行时间几乎是 2 小时。 有没有更快捷方便的方法来做到这一点？

原始表大约有 200000+ 行

这是它的结构示例：

import pandas as pd

df = pd.DataFrame(
                    'name': ['John Johnson', 'John Johnson', 'John Johnson', 'John Johnson', 'Tom Thompson', 'Tom Thompson',
                            'Steve Stevens', 'Steve Stevens', 'Steve Stevens', 'Steve Stevens', 'Steve Stevens', 
                            'Tom Thompson', 'Tom Thompson', 'Tom Thompson', 'Tom Thompson'], 
                   'doc_type': ['opening_document','any_other_document','any_other_document','closing_document2','opening_document','any_other_document',
                                'opening_document','any_other_document','closing_document1','opening_document','closing_document2',
                               'any_other_document','closing_document1','any_other_document','opening_document'], 
                   'date': pd.to_datetime(['2017-1-1', '2017-1-2', '2017-1-10', '2017-1-15', '2017-1-16', '2017-1-17',
                                '2018-1-2', '2018-1-10', '2018-1-15', '2018-1-16', '2018-1-30',
                                '2017-2-1', '2017-2-4', '2017-3-10', '2017-5-15'])
                  )

# sort by date
df = df.sort_values(by='date').reset_index(drop=True)

输出：

+----+---------------+--------------------+---------------------+
|    |     name      |      doc_type      |        date         |
|----+---------------+--------------------+---------------------|
|  0 | John Johnson  |  opening_document  | 2017-01-01 00:00:00 |
|  1 | John Johnson  | any_other_document | 2017-01-02 00:00:00 |
|  2 | John Johnson  | any_other_document | 2017-01-10 00:00:00 |
|  3 | John Johnson  | closing_document2  | 2017-01-15 00:00:00 |
|  4 | Tom Thompson  |  opening_document  | 2017-01-16 00:00:00 |
|  5 | Tom Thompson  | any_other_document | 2017-01-17 00:00:00 |
|  6 | Tom Thompson  | any_other_document | 2017-02-01 00:00:00 |
|  7 | Tom Thompson  | closing_document1  | 2017-02-04 00:00:00 |
|  8 | Tom Thompson  | any_other_document | 2017-03-10 00:00:00 |
|  9 | Tom Thompson  |  opening_document  | 2017-05-15 00:00:00 |
| 10 | Steve Stevens |  opening_document  | 2018-01-02 00:00:00 |
| 11 | Steve Stevens | any_other_document | 2018-01-10 00:00:00 |
| 12 | Steve Stevens | closing_document1  | 2018-01-15 00:00:00 |
| 13 | Steve Stevens |  opening_document  | 2018-01-16 00:00:00 |
| 14 | Steve Stevens | closing_document2  | 2018-01-30 00:00:00 |
+----+---------------+--------------------+---------------------+

我需要计算 opening_document 和 (closing_document1 或 closing_document2) 之间的时间差 所有文档（不仅是目标文档）都代表类似的行

我的脚本输出正确：

%%time

# since name is not enough for correct JOIN we need to make a new unique key
# logic is based on information according to which before closing doc_type there always opening type (because you cant lay off who you not hired yet)

df['key'] = np.nan                   # create new empty column

count_key = 0                        # key counter
df['key'][count_key] = count_key     # assign key 0 for row 0 

for i in range(1, len(df)):          # start with row 1
    store = df['doc_type'][i] 
    if store != 'opening_document':
        df['key'][i] = count_key     # if row is NOT 'opening_document' then keep key the same
    else:
        count_key += 1               # else change key
        df['key'][i] = count_key     # and assing it for current row
        
  # just statusbar for make sure that something happening
    sys.stdout.write('\r')             
    sys.stdout.write("[%-20s] %d%%" % ('='*round(20*(i/(len(df)-1))), (100/(len(df)-1))*i))
    sys.stdout.flush()
    
print('\n')

在原始数据帧中Wall time：1h 29min 53s

它为我们提供了一个额外的键，您可以通过它明确地确定如何加入

+----+---------------+--------------------+---------------------+-------+
|    |     name      |      doc_type      |        date         |   key |
|----+---------------+--------------------+---------------------+-------|
|  0 | John Johnson  |  opening_document  | 2017-01-01 00:00:00 |     0 |
|  1 | John Johnson  | any_other_document | 2017-01-02 00:00:00 |     0 |
|  2 | John Johnson  | any_other_document | 2017-01-10 00:00:00 |     0 |
|  3 | John Johnson  | closing_document2  | 2017-01-15 00:00:00 |     0 |
|  4 | Tom Thompson  |  opening_document  | 2017-01-16 00:00:00 |     1 |
|  5 | Tom Thompson  | any_other_document | 2017-01-17 00:00:00 |     1 |
|  6 | Tom Thompson  | any_other_document | 2017-02-01 00:00:00 |     1 |
|  7 | Tom Thompson  | closing_document1  | 2017-02-04 00:00:00 |     1 |
|  8 | Tom Thompson  | any_other_document | 2017-03-10 00:00:00 |     1 |
|  9 | Tom Thompson  |  opening_document  | 2017-05-15 00:00:00 |     2 |
| 10 | Steve Stevens |  opening_document  | 2018-01-02 00:00:00 |     3 |
| 11 | Steve Stevens | any_other_document | 2018-01-10 00:00:00 |     3 |
| 12 | Steve Stevens | closing_document1  | 2018-01-15 00:00:00 |     3 |
| 13 | Steve Stevens |  opening_document  | 2018-01-16 00:00:00 |     4 |
| 14 | Steve Stevens | closing_document2  | 2018-01-30 00:00:00 |     4 |
+----+---------------+--------------------+---------------------+-------+

按名称和新键将“转换”行合并为列，然后计算打开和关闭天数之间的差异

df_merged = pd.merge(df.loc[df['doc_type']=='opening_document'],
                     df.loc[df['doc_type'].isin(['closing_document1','closing_document2'])], 
                     on=['name','key'], 
                     how='left')

df_merged['time_diff'] = df_merged['date_y'] - df_merged['date_x']

最终正确输出：

    name           doc_type_x        date_x                 key  doc_type_y         date_y               time_diff
--  -------------  ----------------  -------------------  -----  -----------------  -------------------  ----------------
 0  John Johnson   opening_document  2017-01-01 00:00:00      0  closing_document2  2017-01-15 00:00:00  14 days 00:00:00
 1  Tom Thompson   opening_document  2017-01-16 00:00:00      1  closing_document1  2017-02-04 00:00:00  19 days 00:00:00
 2  Tom Thompson   opening_document  2017-05-15 00:00:00      2  nan                NaT                  NaT
 3  Steve Stevens  opening_document  2018-01-02 00:00:00      3  closing_document1  2018-01-15 00:00:00  13 days 00:00:00
 4  Steve Stevens  opening_document  2018-01-16 00:00:00      4  closing_document2  2018-01-30 00:00:00  14 days 00:00:00

我发现不使用循环的最佳解决方案是 diff() 方法 但事实证明，我们不知道我们减去了哪个“块”

不要循环这样做：

df1 = df.loc[df['doc_type'].isin(['opening_document','closing_document1','closing_document2'])].sort_values(by='date').reset_index(drop=True)
df1['diff'] = df1['date'].diff(-1)*(-1)
df1 = df1[df1['doc_type']=='opening_document'].reset_index(drop=True)

输出：

+----+---------------+------------------+---------------------+-------------------+
|    |     name      |     doc_type     |        date         |       diff        |
|----+---------------+------------------+---------------------+-------------------|
|  0 | John Johnson  | opening_document | 2017-01-01 00:00:00 | 14 days 00:00:00  |
|  1 | Tom Thompson  | opening_document | 2017-01-16 00:00:00 | 19 days 00:00:00  |
|  2 | Tom Thompson  | opening_document | 2017-05-15 00:00:00 | 232 days 00:00:00 |
|  3 | Steve Stevens | opening_document | 2018-01-02 00:00:00 | 13 days 00:00:00  |
|  4 | Steve Stevens | opening_document | 2018-01-16 00:00:00 | 14 days 00:00:00  |
+----+---------------+------------------+---------------------+-------------------+

索引为 2 的行中的值是错误的。没有任何结案文件。

如何提高性能并保存正确的输出？

【问题讨论】：

也许这有帮助 - ***.com/questions/52295963/…

我最后使用了 diff() 。主要问题是没有钥匙。

【参考方案1】：

要提高您在循环for 中执行的操作的性能，您可以在'name' 列上使用shift 来查找它更改的位置，或者'opening_document' 在“doc_type”中的位置，加上使用cumsum 增加值，例如：

df['key'] = ((df.name != df.name.shift())|(df.doc_type == 'opening_document')).cumsum()

然后像你一样使用merge 可能就足够有效了。如果您希望密钥从 0 开始，只需在上面代码的末尾添加 -1

编辑：每次更改名称时，'doc_type'中的值为opening_document，可以只保留第二个条件，例如：

df['key'] = (df.doc_type == 'opening_document').cumsum()

【讨论】：

@cbccbd good :) 只说一句，如果你确定每次改名，'doc_type'的第一行是opening_document，你甚至可以去掉列名上的这个条件比如df['key'] = (df.doc_type == 'opening_document').cumsum()就够了

@ben-t 太棒了！是，我确定。因为在其他程序之前过滤。再次感谢（：