无法在正则表达式和 nltk 的帮助下删除特殊字符

Posted 2023-02-23

【中文标题】无法在正则表达式和 nltk 的帮助下删除特殊字符

【英文标题】：Unable to remove special characters with the help of regex with nltk

【发布时间】：2021-07-21 15:50:47

【问题描述】：

无法在 nltk 的正则表达式的帮助下删除特殊字符。

代码是

X, y = data.comments, data.sentiment

紧随其后

documents = []

from nltk.stem import WordNetLemmatizer

stemmer = WordNetLemmatizer()

for sen in range(0, len(X)):
    # Remove all the special characters
    document = re.sub(r'\W', ' ', str(X[sen]))
    
    # remove all single characters
    document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)
    
    # Remove single characters from the start
    document = re.sub(r'\^[a-zA-Z]\s+', ' ', document) 
    
    # Substituting multiple spaces with single space
    document = re.sub(r'\s+', ' ', document, flags=re.I)
    
    # Removing prefixed 'b'
    document = re.sub(r'^b\s+', '', document)
    
    # Converting to Lowercase
    document = document.lower()
    
    # Lemmatization
    document = document.split()

    document = [stemmer.lemmatize(word) for word in document]
    document = ' '.join(document)
    
    documents.append(document)

它返回的错误如下所示

KeyError                                  Traceback (most recent call last)
~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   3079             try:
-> 3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 9

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-16-f9320c54a8cb> in <module>
      7 for sen in range(0, len(X)):
      8     # Remove all the special characters
----> 9     document = re.sub(r'\W', ' ', str(X[sen]))
     10 
     11     # remove all single characters

~\anaconda3\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
    822 
    823         elif key_is_scalar:
--> 824             return self._get_value(key)
    825 
    826         if is_hashable(key):

~\anaconda3\lib\site-packages\pandas\core\series.py in _get_value(self, label, takeable)
    930 
    931         # Similar to Index.get_value, but we do not fall back to positional
--> 932         loc = self.index.get_loc(label)
    933         return self.index._get_values_for_loc(self, loc, label)
    934 

~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:
-> 3082                 raise KeyError(key) from err
   3083 
   3084         if tolerance is not None:

KeyError: 9

我不知道为什么它不起作用，仅删除特殊字符代码不起作用。我正在尝试清理数据，目标变量是 y，问题案例是二进制分类。仅删除特殊字符不起作用。

【问题讨论】：

尝试使用iloc 索引而不是纯切片：document = re.sub(r'\W', ' ', str(X.iloc[sen]))

谢谢你，iloc 正在工作。我认为熊猫版本存在一些问题。

太好了，我会发表评论作为答案

【参考方案1】：

将此行上的索引从纯切片更改为iloc：

document = re.sub(r'\W', ' ', str(X.iloc[sen]))