pandas中分隔符由多个字符组成

Posted 2021-10-14 JasonLiu1919

背景

在使用pandas过程由于文本中存在形如, 、| 等常规字符，所以需要自定义分隔符，特别是自定义由多个字符组成的分隔符。那么此时在使用 pandas.read_csv()的时候要如何设置？

解决

比如当生成文件的时候使用#|#作为分隔符，直接使用df = pd.read_csv(raw_file, sep='#|#', quoting=3)会报错：

    df = pd.read_csv(raw_file, sep='#|#', quoting=3)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 460, in _read
    data = parser.read(nrows)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 1198, in read
    ret = self._engine.read(nrows)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 2585, in read
    alldata = self._rows_to_cols(content)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 3237, in _rows_to_cols
    self._alert_malformed(msg, row_num + 1)
  File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 2998, in _alert_malformed
    raise ParserError(msg)

需要将其改为：

df = pd.read_csv(raw_file, sep='\\#\\|\\#', quoting=3)

官方文档是这么说明的：

 In addition, separators longer than 1 character and different from '\\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\\r\\t'.