{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 注意力机制与Transformer\n", "\n", "循环网络的一个主要缺点是序列中的所有单词对结果的影响相同。这会导致标准的LSTM编码器-解码器模型在处理序列到序列任务（如命名实体识别和机器翻译）时表现不佳。实际上，输入序列中的某些特定单词对输出序列的影响往往比其他单词更大。\n", "\n", "以序列到序列模型为例，比如机器翻译。它通过两个循环网络实现，其中一个网络（**编码器**）将输入序列压缩成隐藏状态，另一个网络（**解码器**）将隐藏状态展开成翻译结果。这种方法的问题在于，网络的最终状态很难记住句子的开头部分，从而导致模型在处理长句子时质量较差。\n", "\n", "**注意力机制**提供了一种方法，可以对每个输入向量对RNN输出预测的上下文影响进行加权。它的实现方式是通过在输入RNN的中间状态和输出RNN之间创建快捷路径。这样，在生成输出符号$y_t$时，我们会考虑所有输入隐藏状态$h_i$，并赋予不同的权重系数$\\alpha_{t,i}$。\n", "\n", "![显示带有加性注意力层的编码器/解码器模型的图像](../../../../../lessons/5-NLP/18-Transformers/images/encoder-decoder-attention.png)\n", "*[Bahdanau等人，2015](https://arxiv.org/pdf/1409.0473.pdf)中的编码器-解码器模型，图片引用自[这篇博客](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n", "\n", "注意力矩阵$\\{\\alpha_{i,j}\\}$表示某些输入单词在生成输出序列中某个单词时的影响程度。下面是一个这样的矩阵示例：\n", "\n", "![显示RNNsearch-50找到的样本对齐的图像，取自Bahdanau - arviz.org](../../../../../lessons/5-NLP/18-Transformers/images/bahdanau-fig3.png)\n", "\n", "*图片取自[Bahdanau等人，2015](https://arxiv.org/pdf/1409.0473.pdf)（图3）*\n", "\n", "注意力机制是当前或接近当前自然语言处理领域的先进技术的核心。然而，添加注意力机制会显著增加模型参数的数量，这导致了循环网络的扩展问题。循环网络扩展的一个关键限制是模型的循环特性使得批量处理和训练并行化变得困难。在循环网络中，序列的每个元素都需要按顺序处理，这意味着它无法轻松并行化。\n", "\n", "注意力机制的采用结合这一限制，促成了如今我们所熟知和使用的先进Transformer模型的诞生，从BERT到OpenGPT3。\n", "\n", "## Transformer模型\n", "\n", "与将每次预测的上下文传递到下一步评估不同，**Transformer模型**使用**位置编码**和**注意力机制**来捕获给定输入在提供的文本窗口中的上下文。下图展示了位置编码与注意力机制如何在给定窗口中捕获上下文。\n", "\n", "![显示Transformer模型中如何进行评估的动画GIF](../../../../../lessons/5-NLP/18-Transformers/images/transformer-animated-explanation.gif)\n", "\n", "由于每个输入位置可以独立映射到每个输出位置，Transformer比循环网络更容易并行化，这使得构建更大、更具表现力的语言模型成为可能。每个注意力头可以用来学习单词之间的不同关系，从而提升下游自然语言处理任务的效果。\n", "\n", "## 构建简单的Transformer模型\n", "\n", "Keras中没有内置的Transformer层，但我们可以自己构建。和之前一样，我们将专注于AG News数据集的文本分类，但值得一提的是，Transformer模型在更复杂的自然语言处理任务中表现最佳。\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow import keras\n", "import tensorflow_datasets as tfds\n", "import numpy as np\n", "\n", "ds_train, ds_test = tfds.load('ag_news_subset').values()\n", "\n", "def extract_text(x):\n", " return x['title']+' '+x['description']\n", "\n", "def tupelize(x):\n", " return (extract_text(x),x['label'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在 Keras 中，新层应继承 `Layer` 类，并实现 `call` 方法。让我们从 **位置嵌入** 层开始。我们将使用[官方 Keras 文档中的一些代码](https://keras.io/examples/nlp/text_classification_with_transformer/)。我们将假设我们将所有输入序列填充到长度 `maxlen`。\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "class TokenAndPositionEmbedding(keras.layers.Layer):\n", " def __init__(self, maxlen, vocab_size, embed_dim):\n", " super(TokenAndPositionEmbedding, self).__init__()\n", " self.token_emb = keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n", " self.pos_emb = keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n", " self.maxlen = maxlen\n", "\n", " def call(self, x):\n", " maxlen = self.maxlen\n", " positions = tf.range(start=0, limit=maxlen, delta=1)\n", " positions = self.pos_emb(positions)\n", " x = self.token_emb(x)\n", " return x+positions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这一层由两个`Embedding`层组成：一个用于嵌入标记（以我们之前讨论的方式），另一个用于嵌入标记位置。标记位置是通过使用`tf.range`从0到`maxlen`生成的一系列自然数创建的，然后通过嵌入层处理。两个生成的嵌入向量随后相加，生成输入的基于位置嵌入的表示，其形状为`maxlen`$\\times$`embed_dim`。\n", "\n", "现在，让我们实现Transformer块。它将接收之前定义的嵌入层的输出：\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "class TransformerBlock(keras.layers.Layer):\n", " def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n", " super(TransformerBlock, self).__init__()\n", " self.att = keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim, name='attn')\n", " self.ffn = keras.Sequential(\n", " [keras.layers.Dense(ff_dim, activation=\"relu\"), keras.layers.Dense(embed_dim),]\n", " )\n", " self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)\n", " self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)\n", " self.dropout1 = keras.layers.Dropout(rate)\n", " self.dropout2 = keras.layers.Dropout(rate)\n", "\n", " def call(self, inputs, training):\n", " attn_output = self.att(inputs, inputs)\n", " attn_output = self.dropout1(attn_output, training=training)\n", " out1 = self.layernorm1(inputs + attn_output)\n", " ffn_output = self.ffn(out1)\n", " ffn_output = self.dropout2(ffn_output, training=training)\n", " return self.layernorm2(out1 + ffn_output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在，我们已经准备好定义完整的 Transformer 模型：\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "text_vectorization (TextVect (None, 256) 0 \n", "_________________________________________________________________\n", "token_and_position_embedding (None, 256, 32) 648192 \n", "_________________________________________________________________\n", "transformer_block (Transform (None, 256, 32) 10656 \n", "_________________________________________________________________\n", "global_average_pooling1d (Gl (None, 32) 0 \n", "_________________________________________________________________\n", "dropout_2 (Dropout) (None, 32) 0 \n", "_________________________________________________________________\n", "dense_2 (Dense) (None, 20) 660 \n", "_________________________________________________________________\n", "dropout_3 (Dropout) (None, 20) 0 \n", "_________________________________________________________________\n", "dense_3 (Dense) (None, 4) 84 \n", "=================================================================\n", "Total params: 659,592\n", "Trainable params: 659,592\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "embed_dim = 32 # Embedding size for each token\n", "num_heads = 2 # Number of attention heads\n", "ff_dim = 32 # Hidden layer size in feed forward network inside transformer\n", "maxlen = 256\n", "vocab_size = 20000\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_sequence_length=maxlen, input_shape=(1,)),\n", " TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim),\n", " TransformerBlock(embed_dim, num_heads, ff_dim),\n", " keras.layers.GlobalAveragePooling1D(),\n", " keras.layers.Dropout(0.1),\n", " keras.layers.Dense(20, activation=\"relu\"),\n", " keras.layers.Dropout(0.1),\n", " keras.layers.Dense(4, activation=\"softmax\")\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training tokenizer\n", "938/938 [==============================] - 45s 39ms/step - loss: 0.4978 - acc: 0.8068 - val_loss: 0.2808 - val_acc: 0.9124\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print('Training tokenizer')\n", "model.layers[0].adapt(ds_train.map(extract_text))\n", "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", "model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## BERT Transformer 模型\n", "\n", "**BERT**（双向编码器表示的变换器）是一个非常大的多层变换器网络，*BERT-base* 有 12 层，*BERT-large* 有 24 层。该模型首先在大规模文本数据（维基百科 + 书籍）上进行无监督训练（预测句子中被遮盖的单词）。在预训练过程中，模型吸收了大量的语言理解能力，随后可以通过微调与其他数据集结合使用。这一过程被称为 **迁移学习**。\n", "\n", "![图片来源于 http://jalammar.github.io/illustrated-bert/](../../../../../lessons/5-NLP/18-Transformers/images/jalammarBERT-language-modeling-masked-lm.png)\n", "\n", "变换器架构有许多变体，包括 BERT、DistilBERT、BigBird、OpenGPT3 等，这些都可以进行微调。\n", "\n", "接下来，我们来看如何使用预训练的 BERT 模型来解决传统的序列分类问题。我们将借用 [官方文档](https://www.tensorflow.org/text/tutorials/classify_text_with_bert) 中的一些思路和代码。\n", "\n", "为了加载预训练模型，我们将使用 **Tensorflow hub**。首先，让我们加载 BERT 专用的向量化工具：\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'tensorflow_text'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_41180/4216669875.py\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_hub\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKerasLayer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow_text'" ] } ], "source": [ "import tensorflow_text \n", "import tensorflow_hub as hub\n", "vectorizer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'input_type_ids': ,\n", " 'input_word_ids': ,\n", " 'input_mask': }" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vectorizer(['I love transformers'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "使用与原始网络训练时相同的向量化工具是非常重要的。此外，BERT 向量化工具会返回三个组件：\n", "* `input_word_ids`，这是输入句子的标记编号序列\n", "* `input_mask`，用于显示序列中哪些部分是实际输入，哪些部分是填充。这与 `Masking` 层生成的掩码类似\n", "* `input_type_ids`，用于语言建模任务，允许在一个序列中指定两个输入句子。\n", "\n", "然后，我们可以实例化 BERT 特征提取器：\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "bert = hub.KerasLayer('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pooled_output -> (1, 128)\n", "encoder_outputs -> 4\n", "sequence_output -> (1, 128, 128)\n", "default -> (1, 128)\n" ] } ], "source": [ "z = bert(vectorizer(['I love transformers']))\n", "for i,x in z.items():\n", " print(f\"{i} -> { len(x) if isinstance(x, list) else x.shape }\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "因此，BERT 层会返回一些有用的结果：\n", "* `pooled_output` 是通过对序列中所有标记进行平均得到的结果。你可以将其视为整个网络的智能语义嵌入。它等同于我们之前模型中 `GlobalAveragePooling1D` 层的输出。\n", "* `sequence_output` 是最后一个 Transformer 层的输出（对应于我们上面模型中的 `TransformerBlock` 的输出）。\n", "* `encoder_outputs` 是所有 Transformer 层的输出。由于我们加载了一个 4 层的 BERT 模型（你可能已经从名字中包含的 `4_H` 猜到了），它有 4 个张量。最后一个张量与 `sequence_output` 相同。\n", "\n", "现在我们将定义端到端的分类模型。我们将使用*函数式模型定义*，在定义模型输入后，通过一系列表达式计算其输出。我们还会将 BERT 模型的权重设置为不可训练，仅训练最终的分类器：\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model\"\n", "__________________________________________________________________________________________________\n", "Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", "input_1 (InputLayer) [(None,)] 0 \n", "__________________________________________________________________________________________________\n", "keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n", "__________________________________________________________________________________________________\n", "keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n", " keras_layer[0][1] \n", " keras_layer[0][2] \n", "__________________________________________________________________________________________________\n", "dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n", "__________________________________________________________________________________________________\n", "dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n", "==================================================================================================\n", "Total params: 4,782,981\n", "Trainable params: 516\n", "Non-trainable params: 4,782,465\n", "__________________________________________________________________________________________________\n" ] } ], "source": [ "inp = keras.Input(shape=(),dtype=tf.string)\n", "x = vectorizer(inp)\n", "x = bert(x)\n", "x = keras.layers.Dropout(0.1)(x['pooled_output'])\n", "out = keras.layers.Dense(4,activation='softmax')(x)\n", "model = keras.models.Model(inp,out)\n", "bert.trainable = False\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "938/938 [==============================] - 528s 559ms/step - loss: 0.8056 - acc: 0.6983 - val_loss: 0.5953 - val_acc: 0.7888\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", "model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "尽管可训练参数很少，但过程仍然相当缓慢，因为 BERT 特征提取器计算量很大。看起来我们无法达到合理的准确性，可能是由于训练不足或模型参数不足。\n", "\n", "让我们尝试解冻 BERT 的权重并对其进行训练。这需要非常小的学习率，同时还需要更谨慎的训练策略，包括**预热**，并使用 **AdamW** 优化器。我们将使用 `tf-models-official` 包来创建优化器：\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model\"\n", "__________________________________________________________________________________________________\n", "Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", "input_1 (InputLayer) [(None,)] 0 \n", "__________________________________________________________________________________________________\n", "keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n", "__________________________________________________________________________________________________\n", "keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n", " keras_layer[0][1] \n", " keras_layer[0][2] \n", "__________________________________________________________________________________________________\n", "dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n", "__________________________________________________________________________________________________\n", "dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n", "==================================================================================================\n", "Total params: 4,782,981\n", "Trainable params: 4,782,980\n", "Non-trainable params: 1\n", "__________________________________________________________________________________________________\n", "938/938 [==============================] - 629s 664ms/step - loss: 0.6344 - acc: 0.7658 - val_loss: 0.4876 - val_acc: 0.8247\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from official.nlp import optimization \n", "bert.trainable=True\n", "model.summary()\n", "epochs = 3\n", "opt = optimization.create_optimizer(\n", " init_lr=3e-5,\n", " num_train_steps=epochs*len(ds_train),\n", " num_warmup_steps=0.1*epochs*len(ds_train),\n", " optimizer_type='adamw')\n", "\n", "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer=opt)\n", "model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "正如你所看到的，训练过程相当缓慢——但你可能想尝试训练模型几个周期（5-10），看看与我们之前使用的方法相比，是否能获得最佳结果。\n", "\n", "## Huggingface Transformers库\n", "\n", "另一种非常常见（且稍微简单一些）的使用Transformer模型的方法是[HuggingFace包](https://github.com/huggingface/)，它为不同的NLP任务提供了简单的构建模块。该库同时支持Tensorflow和PyTorch，这两个非常流行的神经网络框架。\n", "\n", "> **注意**：如果你对了解Transformers库的工作原理不感兴趣——可以直接跳到笔记本的最后部分，因为你不会看到与我们之前所做的有实质性不同的内容。我们将重复使用不同的库和更大的模型来训练BERT模型的相同步骤。因此，这个过程涉及一些相当长时间的训练，所以你可能只需要浏览代码即可。\n", "\n", "让我们看看如何使用[Huggingface Transformers](http://huggingface.co)来解决我们的问题。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "首先，我们需要选择将要使用的模型。除了内置的一些模型之外，Huggingface还提供了一个[在线模型库](https://huggingface.co/models)，社区中有许多预训练模型可供选择。所有这些模型都可以通过提供模型名称来加载和使用。模型所需的所有二进制文件会自动下载。\n", "\n", "有时候，你可能需要加载自己的模型，这种情况下可以指定包含所有相关文件的目录，包括分词器的参数、包含模型参数的`config.json`文件、二进制权重等。\n", "\n", "通过模型名称，我们可以实例化模型和分词器。让我们从分词器开始：\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import transformers\n", "\n", "# To load the model from Internet repository using model name. \n", "# Use this if you are running from your own copy of the notebooks\n", "bert_model = 'bert-base-uncased' \n", "\n", "# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n", "# prepared all required files for you.\n", "#bert_model = './bert'\n", "\n", "tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n", "\n", "MAX_SEQ_LEN = 128\n", "PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n", "UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`tokenizer` 对象包含 `encode` 函数，可直接用于编码文本：\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[101, 23435, 12314, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.encode('Tensorflow is a great framework for NLP')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们还可以使用分词器以适合传递给模型的方式对序列进行编码，例如包括 `token_ids`、`input_mask` 字段等。我们还可以通过提供 `return_tensors='tf'` 参数来指定我们需要 Tensorflow 张量：\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'input_ids': , 'token_type_ids': , 'attention_mask': }" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer(['Hello, there'],return_tensors='tf')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在我们的案例中，我们将使用预训练的BERT模型，名为`bert-base-uncased`。*Uncased*表示该模型对大小写不敏感。\n", "\n", "在训练模型时，我们需要提供经过分词的序列作为输入，因此我们将设计数据处理管道。由于`tokenizer.encode`是一个Python函数，我们将采用与上一单元相同的方法，通过调用`py_function`来实现：\n" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "def process(x):\n", " return tokenizer.encode(x.numpy().decode('utf-8'),return_tensors='tf',padding='max_length',max_length=MAX_SEQ_LEN,truncation=True)[0]\n", "\n", "def process_fn(x):\n", " s = x['title']+' '+x['description']\n", " e = tf.py_function(process,inp=[s],Tout=(tf.int32))\n", " e.set_shape(MAX_SEQ_LEN)\n", " return e,x['label']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们可以使用`BertForSequenceClassification`包加载实际模型。这确保了我们的模型已经具有分类所需的架构，包括最终的分类器。您会看到一条警告消息，指出最终分类器的权重尚未初始化，模型需要进行预训练——这完全没问题，因为这正是我们即将要做的！\n" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "model = transformers.TFBertForSequenceClassification.from_pretrained(bert_model,num_labels=4,output_attentions=False)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"tf_bert_for_sequence_classification_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "bert (TFBertMainLayer) multiple 109482240 \n", "_________________________________________________________________\n", "dropout_75 (Dropout) multiple 0 \n", "_________________________________________________________________\n", "classifier (Dense) multiple 3076 \n", "=================================================================\n", "Total params: 109,485,316\n", "Trainable params: 109,485,316\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从 `summary()` 可以看出，该模型包含将近 1.1 亿个参数！据推测，如果我们希望在相对较小的数据集上进行简单的分类任务，我们不希望训练 BERT 基础层：\n" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"tf_bert_for_sequence_classification_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "bert (TFBertMainLayer) multiple 109482240 \n", "_________________________________________________________________\n", "dropout_75 (Dropout) multiple 0 \n", "_________________________________________________________________\n", "classifier (Dense) multiple 3076 \n", "=================================================================\n", "Total params: 109,485,316\n", "Trainable params: 3,076\n", "Non-trainable params: 109,482,240\n", "_________________________________________________________________\n" ] } ], "source": [ "model.layers[0].trainable = False\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们可以开始训练了！\n", "\n", "> **注意**：训练完整规模的BERT模型可能会非常耗时！因此，我们只会训练前32个批次。这只是为了展示模型训练的设置方式。如果你有兴趣尝试完整规模的训练，只需移除 `steps_per_epoch` 和 `validation_steps` 参数，然后准备好等待吧！\n" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "32/32 [==============================] - 142s 4s/step - loss: 1.3896 - acc: 0.2500 - val_loss: 1.3863 - val_acc: 0.2480\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile('adam','sparse_categorical_crossentropy',['acc'])\n", "tf.get_logger().setLevel('ERROR')\n", "model.fit(ds_train.map(process_fn).batch(32),validation_data=ds_test.map(process_fn).batch(32),steps_per_epoch=32,validation_steps=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如果你增加迭代次数并等待足够长的时间，同时进行多个训练周期，你可以预期 BERT 分类会给我们带来最佳的准确率！这是因为 BERT 已经对语言结构有了相当好的理解，我们只需要微调最终的分类器即可。然而，由于 BERT 是一个大型模型，整个训练过程需要很长时间，并且需要强大的计算能力！（GPU，最好是多块 GPU）。\n", "\n", "> **Note:** 在我们的示例中，我们使用的是最小的预训练 BERT 模型之一。还有更大的模型可能会带来更好的结果。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 要点\n", "\n", "在本单元中，我们了解了基于**transformers**的最新模型架构。我们将其应用于文本分类任务，但同样，BERT模型也可以用于实体提取、问答以及其他自然语言处理任务。\n", "\n", "Transformer模型代表了当前自然语言处理领域的最先进技术。在大多数情况下，当你开始尝试实现定制的自然语言处理解决方案时，它应该是首选。然而，如果你希望构建高级的神经网络模型，理解本模块中讨论的循环神经网络的基本原理是非常重要的。\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**免责声明**： \n本文档使用AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。原始语言的文档应被视为权威来源。对于关键信息，建议使用专业人工翻译。我们不对因使用此翻译而产生的任何误解或误读承担责任。\n" ] } ], "metadata": { "interpreter": { "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" }, "kernelspec": { "display_name": "py38_tensorflow", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "coopTranslator": { "original_hash": "ab59c532409774988ab875f2260e8e53", "translation_date": "2025-08-31T10:42:25+00:00", "source_file": "lessons/5-NLP/18-Transformers/TransformersTF.ipynb", "language_code": "zh" } }, "nbformat": 4, "nbformat_minor": 4 }