{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 文本分类任务\n", "\n", "在本模块中，我们将从一个简单的文本分类任务开始，基于 **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** 数据集：我们将把新闻标题分类为以下四个类别之一：国际、体育、商业和科技。\n", "\n", "## 数据集\n", "\n", "为了加载数据集，我们将使用 **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API。\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow import keras\n", "import tensorflow_datasets as tfds\n", "\n", "# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n", "# we will set tensorflow option to grow GPU memory allocation when required.\n", "physical_devices = tf.config.list_physical_devices('GPU') \n", "if len(physical_devices)>0:\n", " tf.config.experimental.set_memory_growth(physical_devices[0], True)\n", "\n", "dataset = tfds.load('ag_news_subset')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们现在可以通过使用 `dataset['train']` 和 `dataset['test']` 分别访问数据集的训练和测试部分：\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Length of train dataset = 120000\n", "Length of test dataset = 7600\n" ] } ], "source": [ "ds_train = dataset['train']\n", "ds_test = dataset['test']\n", "\n", "print(f\"Length of train dataset = {len(ds_train)}\")\n", "print(f\"Length of test dataset = {len(ds_test)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "让我们打印出数据集中前10条新标题：\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n", "1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n", "2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n", "3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n", "1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n" ] } ], "source": [ "classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n", "\n", "for i,x in zip(range(5),ds_train):\n", " print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 文本向量化\n", "\n", "现在我们需要将文本转换为可以表示为张量的**数字**。如果我们想要词级别的表示，需要完成以下两件事：\n", "\n", "* 使用一个**分词器**将文本拆分为**标记**。\n", "* 构建这些标记的**词汇表**。\n", "\n", "### 限制词汇表大小\n", "\n", "在 AG News 数据集的例子中，词汇表的大小相当大，超过了 10 万个单词。一般来说，我们不需要那些在文本中很少出现的单词——只有少数句子会包含它们，而模型也无法从中学习。因此，通过向向量化器构造函数传递一个参数，将词汇表大小限制为一个较小的数字是合理的：\n", "\n", "以上两个步骤都可以通过 **TextVectorization** 层来处理。我们可以实例化一个向量化器对象，然后调用 `adapt` 方法来遍历所有文本并构建词汇表：\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "vocab_size = 50000\n", "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n", "vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **注意** 我们仅使用整个数据集的一部分来构建词汇表。这样做是为了加快执行速度，避免让您等待太久。然而，这样做存在一定风险，即整个数据集中的某些词可能不会被包含在词汇表中，并在训练过程中被忽略。因此，使用完整的词汇表大小并在`adapt`过程中遍历整个数据集应该可以提高最终的准确性，但提升幅度不会太大。\n", "\n", "现在我们可以访问实际的词汇表：\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n", "Length of vocabulary: 5335\n" ] } ], "source": [ "vocab = vectorizer.get_vocabulary()\n", "vocab_size = len(vocab)\n", "print(vocab[:10])\n", "print(f\"Length of vocabulary: {vocab_size}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "使用向量化器，我们可以轻松地将任何文本编码为一组数字：\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vectorizer('I love to play with my words')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 词袋文本表示法\n", "\n", "由于单词承载着意义，有时我们可以通过单独查看单词，而不考虑它们在句子中的顺序，来理解一段文本的含义。例如，在新闻分类中，像 *天气* 和 *雪* 这样的词可能表明 *天气预报*，而像 *股票* 和 *美元* 这样的词则可能属于 *财经新闻*。\n", "\n", "**词袋** (BoW) 向量表示法是最容易理解的传统向量表示法。每个单词都与一个向量索引相关联，向量中的元素表示某个文档中每个单词出现的次数。\n", "\n", "![展示词袋向量表示法在内存中如何表示的图片。](../../../../../lessons/5-NLP/13-TextRep/images/bag-of-words-example.png) \n", "\n", "> **注意**: 你也可以将 BoW 理解为文本中每个单词的独热编码向量的总和。\n", "\n", "下面是一个使用 Scikit Learn Python 库生成词袋表示法的示例：\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "sc_vectorizer = CountVectorizer()\n", "corpus = [\n", " 'I like hot dogs.',\n", " 'The dog ran fast.',\n", " 'Its hot outside.',\n", " ]\n", "sc_vectorizer.fit_transform(corpus)\n", "sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们还可以使用上面定义的Keras向量化器，将每个单词编号转换为独热编码，并将所有这些向量相加：\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def to_bow(text):\n", " return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n", "\n", "to_bow('My dog likes hot dogs on a hot day.').numpy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **注意**：您可能会惊讶地发现结果与之前的示例不同。原因在于，在 Keras 示例中，向量的长度对应于词汇表的大小，而该词汇表是基于整个 AG News 数据集构建的；而在 Scikit Learn 示例中，我们是根据示例文本动态构建词汇表的。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 训练 BoW 分类器\n", "\n", "现在我们已经了解了如何构建文本的词袋表示，让我们训练一个使用它的分类器。首先，我们需要将数据集转换为词袋表示。这可以通过以下方式使用 `map` 函数来实现：\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "batch_size = 128\n", "\n", "ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n", "ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在让我们定义一个简单的分类器神经网络，它包含一个线性层。输入大小是`vocab_size`，输出大小对应于类别数量（4）。由于我们正在解决一个分类任务，最终的激活函数是**softmax**：\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n", "])\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train_bow,validation_data=ds_test_bow)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "由于我们有4个类别，超过80%的准确率是一个不错的结果。\n", "\n", "## 将分类器训练为一个网络\n", "\n", "因为向量化器也是一个 Keras 层，我们可以定义一个包含它的网络，并进行端到端的训练。这样我们就不需要使用 `map` 来向量化数据集，只需将原始数据集传递到网络的输入即可。\n", "\n", "> **注意**: 我们仍然需要对数据集应用映射操作，以将字段从字典（例如 `title`、`description` 和 `label`）转换为元组。然而，当从磁盘加载数据时，我们可以一开始就构建一个具有所需结构的数据集。\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_1 (InputLayer) [(None, 1)] 0 \n", " \n", " text_vectorization (TextVec (None, None) 0 \n", " torization) \n", " \n", " tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n", " \n", " tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n", " bda) \n", " \n", " dense_2 (Dense) (None, 4) 21344 \n", " \n", "=================================================================\n", "Total params: 21,344\n", "Trainable params: 21,344\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def extract_text(x):\n", " return x['title']+' '+x['description']\n", "\n", "def tupelize(x):\n", " return (extract_text(x),x['label'])\n", "\n", "inp = keras.Input(shape=(1,),dtype=tf.string)\n", "x = vectorizer(inp)\n", "x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n", "out = keras.layers.Dense(4,activation='softmax')(x)\n", "model = keras.models.Model(inp,out)\n", "model.summary()\n", "\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 二元组、三元组和 n 元组\n", "\n", "词袋模型的一个局限性在于，有些词是多词表达的一部分。例如，“hot dog”这个词的含义与单独的“hot”和“dog”在其他语境中的含义完全不同。如果我们始终用相同的向量表示“hot”和“dog”，可能会让模型感到困惑。\n", "\n", "为了解决这个问题，**n 元组表示法**常被用于文档分类方法中，其中每个单词、双词组或三词组的频率是训练分类器的一个有用特征。例如，在二元组表示中，我们会将所有的单词对添加到词汇表中，除了原始单词之外。\n", "\n", "以下是一个使用 Scikit Learn 生成二元组词袋表示的示例：\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary:\n", " {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n" ] }, { "data": { "text/plain": [ "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n", " dtype=int64)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n", "corpus = [\n", " 'I like hot dogs.',\n", " 'The dog ran fast.',\n", " 'Its hot outside.',\n", " ]\n", "bigram_vectorizer.fit_transform(corpus)\n", "print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n", "bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "n-gram 方法的主要缺点是词汇表的大小会迅速增长。在实际应用中，我们需要将 n-gram 表示与降维技术（例如 *embeddings*）结合使用，这将在下一单元中讨论。\n", "\n", "为了在我们的 **AG News** 数据集中使用 n-gram 表示，我们需要将 `ngrams` 参数传递给 `TextVectorization` 构造函数。二元组词汇表的长度会**显著增加**，在我们的例子中，它超过了 130 万个标记！因此，将二元组标记限制在一个合理的数量范围内是有意义的。\n", "\n", "我们可以使用与上面相同的代码来训练分类器，但这样做会非常占用内存。在下一单元中，我们将使用 embeddings 来训练二元组分类器。同时，你可以在这个笔记本中尝试训练二元组分类器，看看是否能获得更高的准确率。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 自动计算 BoW 向量\n", "\n", "在上面的例子中，我们通过手动将单个词的一次性编码相加来计算 BoW 向量。然而，最新版本的 TensorFlow 允许我们通过向向量化器构造函数传递 `output_mode='count` 参数来自动计算 BoW 向量。这使得定义和训练我们的模型变得更加简单：\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training vectorizer\n", "938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n", " keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n", "])\n", "print(\"Training vectorizer\")\n", "model.layers[0].adapt(ds_train.take(500).map(extract_text))\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 词频-逆文档频率 (TF-IDF)\n", "\n", "在 BoW 表示法中，无论单词本身如何，单词出现的权重都使用相同的技术进行计算。然而，很明显，像 *a* 和 *in* 这样的高频词对于分类的意义远不如一些专业术语。在大多数 NLP 任务中，有些词比其他词更重要。\n", "\n", "**TF-IDF** 是 **词频-逆文档频率** 的缩写。它是袋子模型（bag-of-words）的一个变体，其中不是用二进制的 0/1 值来表示单词是否出现在文档中，而是使用一个浮点值，该值与单词在语料库中出现的频率相关。\n", "\n", "更正式地说，单词 $i$ 在文档 $j$ 中的权重 $w_{ij}$ 定义为：\n", "$$\n", "w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n", "$$\n", "其中：\n", "* $tf_{ij}$ 是单词 $i$ 在文档 $j$ 中出现的次数，即我们之前看到的 BoW 值\n", "* $N$ 是语料库中的文档总数\n", "* $df_i$ 是整个语料库中包含单词 $i$ 的文档数量\n", "\n", "TF-IDF 值 $w_{ij}$ 随着单词在文档中出现的次数成比例增加，同时会根据语料库中包含该单词的文档数量进行调整。这有助于解决某些单词比其他单词更频繁出现的问题。例如，如果某个单词出现在语料库的*每一篇*文档中，那么 $df_i=N$，且 $w_{ij}=0$，这些词将被完全忽略。\n", "\n", "你可以使用 Scikit Learn 轻松创建文本的 TF-IDF 向量化：\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n", " 0.43381609, 0. , 0. , 0. , 0. ,\n", " 0. , 0. , 0. , 0. , 0. ,\n", " 0. ]])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer\n", "vectorizer = TfidfVectorizer(ngram_range=(1,2))\n", "vectorizer.fit_transform(corpus)\n", "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在 Keras 中，通过传递参数 `output_mode='tf-idf'`，`TextVectorization` 层可以自动计算 TF-IDF 频率。让我们重复上面的代码，看看使用 TF-IDF 是否能提高准确性：\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training vectorizer\n", "938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n", " keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n", "])\n", "print(\"Training vectorizer\")\n", "model.layers[0].adapt(ds_train.take(500).map(extract_text))\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 结论\n", "\n", "尽管TF-IDF表示法为不同的词赋予了频率权重，但它无法表达词语的含义或顺序。正如著名语言学家J.R. Firth在1935年所说：“一个词的完整含义总是与上下文相关，任何脱离上下文的意义研究都不值得认真对待。”在课程的后续部分，我们将学习如何通过语言模型从文本中捕捉上下文信息。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**免责声明**： \n本文档使用AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)进行翻译。尽管我们努力确保准确性，但请注意，自动翻译可能包含错误或不准确之处。应以原始语言的文档作为权威来源。对于关键信息，建议使用专业人工翻译。因使用本翻译而导致的任何误解或误读，我们概不负责。\n" ] } ], "metadata": { "interpreter": { "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" }, "kernel_info": { "name": "conda-env-py37_tensorflow-py" }, "kernelspec": { "display_name": "py37_tensorflow", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "nteract": { "version": "nteract-front-end@1.0.0" }, "coopTranslator": { "original_hash": "19b43951d55b377a76209c24c1f017e4", "translation_date": "2025-08-31T11:00:44+00:00", "source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb", "language_code": "zh" } }, "nbformat": 4, "nbformat_minor": 4 }