如果行只包含停用词中的任何一行，则从文本文件中删除这些行

Posted 2021-04-27

我想从Myfile.txt文件中删除那些行，如果该行只包含且仅包含任何来自停用词的行

例如，Myfile.txt文件的样本是

Adh Dhayd
Abu Dhabi is      # here is "is" stopword but this line should not be removed because line contain #Abu Dhabi is
Zaranj
of                # this line contains just stop word, this line should be removed
on                # this line contains just stop word, this line should be removed
Taloqan
Shnan of          # here is "of" stopword but this line should not be removed because line contain #Shnan of
is                # this line contains just stop word, this line should be removed
Shibirghn
Shahrak
from              # this line contains just stop word, this line should be removed

我以此代码为例

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize



example_sent = "This is a sample sentence, showing off the stop words filtration."

stop_words = set(stopwords.words('english'))

word_tokens = word_tokenize(example_sent)

filtered_sentence = [w for w in word_tokens if not w in stop_words]

filtered_sentence = []

for w in word_tokens:
    if w not in stop_words:
        filtered_sentence.append(w)

print(word_tokens)
print(filtered_sentence)

那么根据上面提到的Myfile.txt的解决方案代码是什么。

答案

您可以查看该行是否与任何停用词匹配，如果不将其附加到过滤后的内容中。也就是说，如果要过滤所有仅包含一个stop_word的行。如果还应过滤具有多个停用词的行，请尝试对该行进行标记，并使用stop_words构建交集：

f = open("test.txt","r+")
filtered_content = []
stop_words = set(stopwords.words('english'))
for line in f.read().splitlines():
    if not line in stop_words:
        filtered_content.append(line)
g = open("test_filter.txt","a+")
g.write("
".join(filtered_content))
g.close()
f.close()

如果要删除多个停用词，请使用此if语句。这将删除仅包含停用词的行。如果一个单词不是停用词，则保留该行：

if not len(set(word_tokenize(line)).intersection(stop_words)) == len(word_tokenize(line)):