pandas 读取大文件 read_table C-engine CParserError: Error tokenizing data

Posted 2021-01-14 cbugs

解决办法：

pd_data = pd.read_table(comment_file,header=None,encoding=‘utf-8‘, engine=‘python‘)

官网解析：

　　　 engine : {‘c’, ‘python’}, optional

Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.

 

1、

iterator : boolean, default False

Return TextFileReader object for iteration or getting chunks with get_chunk().

或者通过chunk 获取

pd_data = pd.read_table(comment_file,header=None,encoding=‘utf-8‘,iterator=True)
# print(pd_data)
# pd_data_t = pd.read_table(comment_file,header=None,encoding=‘utf-8‘, engine=‘python‘)
# return;
loop = True
chunk_data = []
chunk_size = 1024
while loop:
   try:
      pd_data_tmp = pd_data.get_chunk(chunk_size)
      chunk_data.append(pd_data_tmp)
   except StopIteration:
      loop = False
df = pd.concat(chunk_data,ignore_index=True)