令牌索引序列长度大于此模型的指定最大序列长度 (651 > 512)，具有拥抱面部情感分类器

Posted 2023-03-29

【中文标题】令牌索引序列长度大于此模型的指定最大序列长度 (651 > 512)，具有拥抱面部情感分类器

【英文标题】：Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512) with Hugging face sentiment classifier

【发布时间】：2021-07-01 09:54:46

【问题描述】：

我正在尝试借助拥抱面部情绪分析预训练模型来获取 cmets 的情绪。它返回类似Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512) with Hugging face sentiment classifier 的错误。

下面我附上代码请看

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
import transformers
import pandas as pd

model = AutoModelForSequenceClassification.from_pretrained('/content/drive/MyDrive/Huggingface-Sentiment-Pipeline')
token = AutoTokenizer.from_pretrained('/content/drive/MyDrive/Huggingface-Sentiment-Pipeline')

classifier = pipeline(task='sentiment-analysis', model=model, tokenizer=token)

data = pd.read_csv('/content/drive/MyDrive/DisneylandReviews.csv', encoding='latin-1')

data.head()

输出是

    Review
0   If you've ever been to Disneyland anywhere you...
1   Its been a while since d last time we visit HK...
2   Thanks God it wasn t too hot or too humid wh...
3   HK Disneyland is a great compact park. Unfortu...
4   the location is not in the city, took around 1...

紧随其后

classifier("My name is mark")

输出是

['label': 'POSITIVE', 'score': 0.9953688383102417]

后跟代码

basic_sentiment = [i['label'] for i in value if 'label' in i]
basic_sentiment

输出是

['POSITIVE']

将总行数追加到空列表中

text = []

for index, row in data.iterrows():
    text.append(row['Review'])

我正在尝试获取所有行的情绪

sent = []

for i in range(len(data)):
    sentiment = classifier(data.iloc[i,0])
    sent.append(sentiment)

错误是：

Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512). Running this sequence through the model will result in indexing errors
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-19-4bb136563e7c> in <module>()
      2 
      3 for i in range(len(data)):
----> 4     sentiment = classifier(data.iloc[i,0])
      5     sent.append(sentiment)

11 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1914         # remove once script supports set_grad_enabled
   1915         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1917 
   1918 

IndexError: index out of range in self

【参考方案1】：

数据框Review 列中的某些句子太长。当这些句子被转换为标记并在模型中发送时，它们超出了model 的512 seq_length 限制，sentiment-analysis 任务中使用的模型的嵌入在512 标记嵌入上进行了训练。

要解决此问题，您可以过滤掉长句子并只保留较小的句子（令牌长度

或者你可以用truncating = True截断句子

sentiment = classifier(data.iloc[i,0], truncation=True)