无法在 keras 的 BERT 之上为 NER 添加 CRF 层

Posted 2023-03-12

【中文标题】无法在 keras 的 BERT 之上为 NER 添加 CRF 层

【英文标题】：Cannot add CRF layer on top of BERT in keras for NER

【发布时间】：2021-07-09 16:45:37

【问题描述】：

我在为 NER 训练 BERT-CRF 模型时遇到了一个未知问题。我将 keras.contrib 用于 CRF 模型。

这里是导入的库。

!pip install transformers
!pip install git+https://www.github.com/keras-team/keras-contrib.git
import pandas as pd
import numpy as np
from transformers import TFBertModel, BertTokenizer, BertConfig
import tensorflow as tf
from tensorflow import keras
from keras_contrib.layers import CRF
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from tqdm import tqdm

模型创建代码。

input_ids = keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32)
token_type_ids = keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32)
attention_mask = keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32)
bert_output = bert(
       [input_ids,
       attention_mask,
       token_type_ids]
   )[0]    
bert_output = keras.layers.Dropout(0.3)(bert_output)
dense_layer_output = keras.layers.Dense(num_classes+1, activation='softmax', name='output')(bert_output)
crf = CRF(num_classes)
outputs = crf(dense_layer_output)
model = keras.Model(
       inputs=[input_ids, token_type_ids, attention_mask],
       outputs=[outputs],
   )
model.compile(
   loss=crf.loss_function,
   metrics=[crf.accuracy],
   optimizer=keras.optimizers.Adam(5e-5)
   )

model.fit(
    x_train,
    y_train,
    epochs=1,
    verbose=1,
    batch_size=32,
    validation_data=(x_test, y_test)
)

在尝试训练模型时出现此错误。我无法理解它的来源和原因。

WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-f369b38eb91d> in <module>()
      5     verbose=1,
      6     batch_size=32,
----> 7     validation_data=(x_test, y_test)
      8 )

9 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    975           except Exception as e:  # pylint:disable=broad-except
    976             if hasattr(e, "ag_error_metadata"):
--> 977               raise e.ag_error_metadata.to_exception(e)
    978             else:
    979               raise

AttributeError: in user code:

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.7/dist-packages/keras_contrib/losses/crf_losses.py:54 crf_loss  *
        crf, idx = y_pred._keras_history[:2]

    AttributeError: 'Tensor' object has no attribute '_keras_history'

我在互联网上读到 keras.contrib 已被贬低，但我不知道如何在 BERT 之上使用 CRF 层。如果在 keras 中有更好的方法，请建议我。

我不知道这个问题是否有意义，但我们将不胜感激。 提前致谢！

【问题讨论】：

Keras-contrib 已弃用。请改用 TensorFlow Addons(tfa)。在您的情况下使用tfa.layers.CRF 而不是from keras_contrib.layers import CRF。更多信息可以参考here。谢谢！

【参考方案1】：

最简单的方法是使用 TensorFlow 插件的 CRF 层。然后利用它的输出来计算损失。

import tensorflow_addons as tfa
crf = tfa.layers.CRF(len(num_labels)+1)

此外，您也可以通过创建自己的模型类来利用它来创建模型。

from tensorflow_addons.text.crf import crf_log_likelihood

def unpack_data(data):
    if len(data) == 2:
        return data[0], data[1], None
    elif len(data) == 3:
        return data
    else:
        raise TypeError("Expected data to be a tuple of size 2 or 3.")


class ModelWithCRFLoss(tf.keras.Model):
    """Wrapper around the base model for custom training logic."""

    def __init__(self, base_model):
        super().__init__()
        self.base_model = base_model

    def call(self, inputs):
        return self.base_model(inputs)

    def compute_loss(self, x, y, sample_weight, training=False):
        y_pred = self(x, training=training)
        _, potentials, sequence_length, chain_kernel = y_pred

        # we now add the CRF loss:
        crf_loss = -crf_log_likelihood(potentials, y, sequence_length, chain_kernel)[0]

        if sample_weight is not None:
            crf_loss = crf_loss * sample_weight

        return tf.reduce_mean(crf_loss), sum(self.losses)

    def train_step(self, data):
        x, y, sample_weight = unpack_data(data)

        with tf.GradientTape() as tape:
            crf_loss, internal_losses = self.compute_loss(
                x, y, sample_weight, training=True
            )
            total_loss = crf_loss + internal_losses

        gradients = tape.gradient(total_loss, self.trainable_variables)
        self.optimizer.apply_gradients(zip(gradients, self.trainable_variables))

        return "crf_loss": crf_loss, "internal_losses": internal_losses

    def test_step(self, data):
        x, y, sample_weight = unpack_data(data)
        crf_loss, internal_losses = self.compute_loss(x, y, sample_weight)
        return "crf_loss_val": crf_loss, "internal_losses_val": internal_losses

你可以按照这些代码行编写

decoded_sequence, potentials, sequence_length, chain_kernel = crf(dense_layer_output, mask=attention_mask)

base_model = tf.keras.Model(
       inputs=[input_ids, attention_mask],
       outputs=crf_layer_outputs,
   )

model = ModelWithCRFLoss(base_model)
model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=5e-3, epsilon=1e-08),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)