stanfordcorenlp处理中文含有 % 时报错解决

Posted 2022-01-22 SanjinLiu

报错

使用python第三方库stanfordcorenlp，调用StanfordCoreNLP接口处理中文时报错

Traceback (most recent call last):
  File "process_data.py", line 136, in <module>
    main()
  File "process_data.py", line 126, in main
    key2gloss, key2sentence = get_dict()
  File "process_data.py", line 39, in get_dict
    for key, pos_tag, gloss in tqdm(get_sentence()):
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "process_data.py", line 28, in get_sentence
    pos_tag = nlp.pos_tag(context)
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/site-packages/stanfordcorenlp/corenlp.py", line 185, in pos_tag
    r_dict = self._request(\'pos\', sentence)
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/site-packages/stanfordcorenlp/corenlp.py", line 239, in _request
    r_dict = json.loads(r.text)
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/lx2021/miniconda3/envs/wsd_env/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

调试时发现报错是因为被处理的文本中含有 %

解决

处理含有%的文本会报错是因为StanfordCoreNLP内部对 % 和 + 做了特殊处理，将 % 和 + 分别替换为 %25 和 %2B 即可正常使用。

这个替换可以在调用时先对要处理的文本做好替换，也可以直接修改stanfordcorenlp源码，一劳永逸。

参考

https://blog.csdn.net/l18848956739/article/details/87605125