TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process

Posted 2022-02-14 AI架构师易筋

机器学习基础：第 8 集 - 自然语言处理的标记化

练习

https://bit.ly/tfw-nlp1

from tensorflow.keras.preprocessing.text import Tokenizer

sentences = [
    'i love my dog',
    'I, love my cat',
    'You love my dog!',
    'hello, hello, hello, hello I am in China now'
]

tokenizer = Tokenizer(num_words = 100)
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
print(word_index)

参考

https://youtu.be/f5YJA5mQD5c