带有大熊猫数据框的python代码很慢
Posted
技术标签:
【中文标题】带有大熊猫数据框的python代码很慢【英文标题】：python code with large pandas DataFrame is to slow 【发布时间】：2018-09-28 22:31:59 【问题描述】：我有以下代码，算法很慢。
我尝试对numpy (1.14.3) 使用预分配并将pandas (0.23.0) 中的字符串列转换为category 以加快代码速度，但它仍然很慢。
我的df 是一个大号pd.DataFrame。 len(df) 返回 1342058 并有 25 列。
df 包含带有特定事件和位置的时间戳。数据的频率是可变的，从秒数据到每小时数据。我基本上想重新采样每个 client_id 和给定日期箭头 (rng) 的数据（start 和 end 之间每 5 分钟一次）。
df = pd.DataFrame()
df['client_id']=['40:a3:cc:XX:XX:XX','28:c6:3f:XX:XX:XX','40:a3:cc:XX:XX:XX','40:a3:cc:XX:XX:XX','28:c6:3f:XX:XX:XX','40:a3:cc:XX:XX:XX','40:a3:cc:XX:XX:XX','28:c6:3f:XX:XX:XX','28:c6:3f:XX:XX:XX','40:a3:cc:XX:XX:XX']
df['seen_time'] = ['2018-09-01 00:00:03+00:00', '2018-09-01 00:00:04+00:00','2018-09-01 00:00:05+00:00','2018-09-01 00:00:06+00:00','2018-09-01 00:00:08+00:00','2018-09-01 00:00:09+00:00','2018-09-01 00:00:09+00:00','2018-09-01 00:00:14+00:00','2018-09-01 00:00:19+00:00','2018-09-01 00:00:25+00:00']
df['location_x'] = [7.488775,20.163136,19.485196,12.841458,15.508627,5.708157,13.451071,19.1871,65,9.015443,28.266964]
这是我的代码：
start = "2018-09-01 00:00"
end = "2018-09-28 00:00"
rng = pd.date_range(start=start, end=end, freq='5T') # date range in x min.
df['client_id'] = df['client_id'].astype('category')
# --> this line already improved the speed of the algorithm by 20x
def processDataSingle(client_id):
df_ = pd.DataFrame() # temporariy dataframe.
data_ = np.zeros((len(rng),)).tolist() # initialize.
dataSize = len(rng)
for idx,d in enumerate(rng):
t0 = d.tz_localize('UTC')
t1 = (d + pd.to_timedelta(filter_min,unit='m')).tz_localize('UTC')
# t0 and t1 are start and end time, with 5 min difference for the query.
df_client = df[df['client_id']==client_id]
# selects the data of the required client_id only.
df_client_range = df_client[(df_client['seen_time']>=t0)&(df_client['seen_time']<t1)]
# selects only the data for this client and between t0 and t1
content =
'x_mean':df_client_range['location_x'].mean(),
'x_median':df_client_range['location_x'].median(),
'x_min': df_client_range['location_x'].min(),
'x_max':df_client_range['location_x'].max()
data_[idx] = content
# end of looping over date range.
df_[client_id] = data_
return df_
def processDataArrayWrapper(client_ids):
results = np.zeros((len(client_ids),)).tolist() # preallocate!!! a=np.zeros((2,))
for iclient,client_id in enumerate(client_ids):
results[iclient] = processDataSingle(client_id)
# end looping over client_ids
return results
#end of processDataArrayWrapper
client_ids = ['firstID','secondID','thirdID'] # in real world around 1000 IDs
%prun results = processDataArrayWrapper(client_ids)
prun 的结果如下：
158001796 function calls (155889787 primitive calls) in 483.930 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
44934 209.957 0.005 209.957 0.005 pandas._libs.algos.take_2d_axis1_object_object
381942 44.206 0.000 44.379 0.000 built-in method pandas._libs.algos.ensure_int64
22468 20.656 0.001 95.913 0.004 categorical.py:2437(_recode_for_categories)
67402 20.583 0.000 20.583 0.000 pandas._libs.algos.take_1d_int16_int16
3 20.274 6.758 483.715 161.238 <ipython-input-49-c130ea2c69e4>:23(processDataSingle)
44934 9.177 0.000 9.177 0.000 pandas._libs.algos.take_2d_axis1_int64_int64
44934 9.107 0.000 9.107 0.000 pandas._libs.algos.take_2d_axis1_float64_float64
36873382 7.775 0.000 15.205 0.000 built-in method builtins.isinstance
44934 7.592 0.000 7.592 0.000 method 'nonzero' of 'numpy.ndarray' objects
11899203 4.604 0.000 5.136 0.000 built-in method builtins.getattr
22471 4.368 0.000 4.450 0.000 method 'get_indexer' of 'pandas._libs.index.IndexEngine' objects
44934 3.644 0.000 3.644 0.000 pandas._libs.algos.take_1d_int64_int64
22467 3.479 0.000 3.776 0.000 categorical.py:64(f)
643042 3.380 0.000 3.380 0.000 method 'reduce' of 'numpy.ufunc' objects
148422 3.016 0.000 3.016 0.000 method 'copy' of 'numpy.ndarray' objects
2557465/2534997 2.740 0.000 4.622 0.000 common.py:1835(_get_dtype_type)
337007/292073 2.727 0.000 318.981 0.001 algorithms.py:1545(take_nd)
7351676 2.672 0.000 5.713 0.000 generic.py:7(_check)
404409/359475 2.359 0.000 20.467 0.000 series.py:165(__init__)
8155845/6470735 2.152 0.000 2.873 0.000 built-in method builtins.len
471840 2.111 0.000 2.111 0.000 built-in method numpy.core.multiarray.empty
89874 2.109 0.000 4.487 0.000 internals.py:3363(_rebuild_blknos_and_blklocs)
2610562 2.028 0.000 5.330 0.000 base.py:61(is_dtype)
1046990/1002055 1.888 0.000 2.184 0.000 built-in method numpy.core.multiarray.array
44935 1.830 0.000 1.830 0.000 method 'take' of 'numpy.ndarray' objects
4713590 1.757 0.000 1.773 0.000 built-in method builtins.hasattr
134826/89883 1.750 0.000 7.191 0.000 base.py:250(__new__)
247139 1.647 0.000 6.430 0.000 cast.py:257(maybe_promote)
1087307 1.523 0.000 5.693 0.000 common.py:1688(is_extension_array_dtype)
134802/89868 1.475 0.000 18.290 0.000 datetimes.py:329(__new__)
2233231 1.437 0.000 2.797 0.000 <frozen importlib._bootstrap>:997(_handle_fromlist)
718950 1.411 0.000 2.764 0.000 generic.py:4374(__setattr__)
763890/629088 1.361 0.000 2.083 0.000 method 'format' of 'str' objects
224670 1.314 0.000 254.948 0.001 internals.py:1237(take_nd)
629082 1.249 0.000 2.836 0.000 internals.py:116(__init__)
314539 1.145 0.000 303.889 0.001 frame.py:2661(__getitem__)
...
67401 0.634 0.000 134.602 0.002 ops.py:1175(wrapper)
44934 0.625 0.000 277.776 0.006 internals.py:4518(take)
...
67404 0.558 0.000 101.225 0.002 categorical.py:267(__init__)
...
89868 0.527 0.000 11.864 0.000 datetimelike.py:523(take)
44934 0.522 0.000 285.513 0.006 generic.py:2780(_take)
1392955 0.517 0.000 0.599 0.000 internals.py:352(dtype)
44934/22467 0.514 0.000 12.605 0.001 built-in method _operator.ge
...
44937 0.425 0.000 264.034 0.006 internals.py:4423(<listcomp>)
...
763912 0.296 0.000 1.993 0.000 numeric.py:424(asarray)
44934 0.295 0.000 22.606 0.001 datetimes.py:109(wrapper)
381946 0.289 0.000 2.251 0.000 method 'any' of 'numpy.ndarray' objects
89892 0.286 0.000 0.960 0.000 common.py:298(_asarray_tuplesafe)
67404 0.281 0.000 1.299 0.000 cast.py:971(maybe_cast_to_datetime)
89868 0.280 0.000 2.962 0.000 internals.py:4108(get)
629082 0.276 0.000 0.276 0.000 internals.py:127(_check_ndim)
44934 0.276 0.000 295.073 0.007 frame.py:2704(_getitem_array)
107568 0.267 0.000 2.324 0.000 missing.py:494(na_value_for_dtype)
179742 0.264 0.000 0.827 0.000 dtypes.py:459(construct_from_string)
44937 0.260 0.000 270.290 0.006 internals.py:4388(reindex_indexer)
...
67401 0.220 0.000 128.302 0.002 ops.py:1073(dispatch_to_index_op)
...
1 0.194 0.194 483.930 483.930 <string>:1(<module>)
...
44934 0.179 0.000 20.583 0.000 base.py:89(cmp_method)
...
44934 0.110 0.000 10.841 0.000 datetimes.py:39(_maybe_cache)
...
1 0.003 0.003 483.719 483.719 <ipython-input-49-c130ea2c69e4>:61(processDataArrayWrapper)
...
1 0.000 0.000 483.930 483.930 built-in method builtins.exec
注意：我删除了一堆 cumtime 很少的行。我相信大部分时间都花在take_2d_axis1_object_object 上，但我很难理解这个问题。代码的哪一部分导致了这个问题？
我在我的 Mac book pro 上使用 Intel Core i7 2,7 GHz 4 核运行代码，python 进程使用了近 400% 的 CPU，这意味着它以某种方式在后面使用了多核。
【问题讨论】：
你到底想在这里完成什么？您能否就您要解决的问题提供一些背景或见解？查看您的数据框样本会有所帮助。 @rahlf23 我希望我的算法目标现在很明确。 查看数据框示例仍然会有所帮助。这里的目标是一个最小可重现的例子。此时，您似乎可以简单地使用groupby() w/resample() 和agg() 来完成您的任务
DataFrame 很大。我将尝试在问题中添加一些数据，但不是所有原始列。
df[df['client_id']==client_id] 在循环的每次迭代中迭代整个数据帧。至少，你应该groupby('client_id')
【参考方案1】：
我想出了一个非常酷的解决方案，将执行时间从几个小时减少到不到 15 秒。
def resampleData():
results =
for idx,thisIDIndex in df.groupby('client_id').groups.items():
df_id = df.loc[thisIDIndex]
df_id_sel = df_id[['seen_time','location_x','location_y']] # only the stuff I really need.
df_id_sel = df_id_sel.set_index('seen_time')
df_id_sel_resampled_mean = df_id_sel.resample('5T').mean()
df_id_sel_resampled_max = df_id_sel.resample('5T').max()
df_id_sel_resampled_min = df_id_sel.resample('5T').min()
df_id_sel_resampled_median= df_id_sel.resample('5T').median()
df_id_sel_resampled_mean = df_id_sel_resampled_mean.rename(columns='location_x':'x_mean', 'location_y':'y_mean')
df_id_sel_resampled_max = df_id_sel_resampled_max.rename(columns='location_x':'x_max', 'location_y':'y_max')
df_id_sel_resampled_min = df_id_sel_resampled_min.rename(columns='location_x':'x_min', 'location_y':'y_min')
df_id_sel_resampled_median = df_id_sel_resampled_median.rename(columns='location_x':'x_med', 'location_y':'y_med')
DF = pd.concat([df_id_sel_resampled_min, df_id_sel_resampled_max,df_id_sel_resampled_mean, df_id_sel_resampled_median], axis=1, sort=False)
#print('--- Index ' + idx + ' done ')
results[idx] = DF
# end loop over ids
return results
#end of doStupidLoop
我现在只循环遍历 id，然后通过groupby 访问数据，正如所指出的那样。我还使用resample 来完成工作，而不是在时间轴上进行迭代。我可能可以进一步优化代码，但我现在对解决方案很满意，它仍然非常明确且易于理解和遵循。
【讨论】：
以上是关于带有大熊猫数据框的python代码很慢的主要内容，如果未能解决你的问题，请参考以下文章