{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 使用 ConceptNet 的概念图\n",
    "\n",
    "> **注意：** 原始的 [Microsoft Concept Graph](https://concept.research.microsoft.com/) API 已不再可用。该笔记本已更新为使用作为替代的 [ConceptNet](https://conceptnet.io/)，这是一个免费开放的知识图谱，具有概念之间类似的 `is-a` 关系。\n",
    "\n",
    "[ConceptNet](https://conceptnet.io/) 是一个包含诸如 `IsA`、`PartOf`、`UsedFor` 等关系的大型概念语义网络。它可通过以下方式获得：\n",
    " * 可下载的数据文件\n",
    " * REST API（无需 API 密钥）\n",
    "\n",
    "ConceptNet 统计数据：\n",
    " * 超过 800 万个节点\n",
    " * 跨 83 种语言的 2100 多万条边\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 ConceptNet Web 服务\n",
    "\n",
    "[ConceptNet](https://conceptnet.io/) 提供了一个 REST API 用于探索概念之间的 `is-a`（IsA）关系。无需 API 密钥。\n",
    "以下是调用的示例 URL：`https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "import json\n",
    "\n",
    "def http(x):\n",
    "    response = urllib.request.urlopen(x)\n",
    "    data = response.read()\n",
    "    return data.decode('utf-8')\n",
    "\n",
    "def query(x):\n",
    "    concept = x.lower().replace(' ', '_')\n",
    "    url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n",
    "        urllib.parse.quote(concept))\n",
    "    try:\n",
    "        result = json.loads(http(url))\n",
    "    except Exception:\n",
    "        return {}\n",
    "    edges = result.get('edges', [])\n",
    "    if not edges:\n",
    "        return {}\n",
    "    total_weight = sum(edge['weight'] for edge in edges)\n",
    "    if total_weight == 0:\n",
    "        return {}\n",
    "    return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n",
    "\n",
    "query('microsoft')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们尝试使用父概念对新闻标题进行分类。为了获取新闻标题，我们将使用[NewsApi.org](http://newsapi.org)服务。您需要获得自己的API密钥才能使用该服务——请访问该网站并注册免费开发者计划。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "trusted": true
   },
   "outputs": [],
   "source": [
    "newsapi_key = '<your API key here>'\n",
    "def get_news(country='us'):\n",
    "    res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n",
    "    return res['articles']\n",
    "\n",
    "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n",
       " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n",
       " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n",
       " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n",
       " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n",
       " '2 teachers killed at Swedish high school, student arrested - ABC News',\n",
       " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n",
       " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n",
       " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
       " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
       " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n",
       " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n",
       " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n",
       " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n",
       " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n",
       " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n",
       " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n",
       " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n",
       " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n",
       " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n",
       " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n",
       " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n",
       " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n",
       " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n",
       " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n",
       " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n",
       " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n",
       " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n",
       " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n",
       " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n",
       " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n",
       " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
       " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
       " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n",
       " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n",
       " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n",
       " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n",
       " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n",
       " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n",
       " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_titles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我们希望能够从新闻标题中提取名词。我们将使用 `TextBlob` 库来实现这一点，该库简化了许多典型的自然语言处理任务。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n",
      "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n",
      "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n",
      "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n",
      "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n",
      "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n",
      "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n",
      "Finished.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package conll2000 to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package conll2000 is already up-to-date!\n",
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package movie_reviews is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install textblob\n",
    "!{sys.executable} -m textblob.download_corpora\n",
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "trusted": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'covid-19 live updates': 1,\n",
       " 'vaccines': 1,\n",
       " 'boosters': 1,\n",
       " 'york': 4,\n",
       " 'ukrainians flee mariupol': 1,\n",
       " 'forces push': 1,\n",
       " 'port city': 1,\n",
       " 'wall street journal': 3,\n",
       " 'bond yields': 1,\n",
       " 'futures rise': 1,\n",
       " 'powell says fed': 1,\n",
       " 'ready': 1,\n",
       " 'be': 1,\n",
       " 'aggressive': 1,\n",
       " 'putin': 3,\n",
       " 'alexei navalny': 2,\n",
       " 'russian': 2,\n",
       " 'supreme court nominee': 1,\n",
       " 'ketanji brown jackson': 1,\n",
       " \"confirmation hearing 's\": 1,\n",
       " 'cnn': 1,\n",
       " 'swedish': 1,\n",
       " 'high school': 1,\n",
       " 'abc': 1,\n",
       " 'clues': 1,\n",
       " 'covid-19': 1,\n",
       " '’ s': 2,\n",
       " 'moves': 1,\n",
       " 'sewers': 1,\n",
       " 'roll dice': 1,\n",
       " 'jackson': 1,\n",
       " 'decisions |': 1,\n",
       " 'thehill': 1,\n",
       " 'clear': 2,\n",
       " 'chemical weapons': 2,\n",
       " 'ukraine': 3,\n",
       " 'claims president': 2,\n",
       " 'biden': 2,\n",
       " 'nasa': 2,\n",
       " 'solar system': 2,\n",
       " 'daily mail': 3,\n",
       " 'us stocks': 1,\n",
       " 'fed chair powell': 1,\n",
       " \"'s remarks\": 1,\n",
       " 'fox': 1,\n",
       " \"'we 've\": 1,\n",
       " 'tests': 1,\n",
       " 'covid': 1,\n",
       " 'politico': 1,\n",
       " 'duchess': 1,\n",
       " 'cambridge': 1,\n",
       " 'swaps khaki jungle gear': 1,\n",
       " 'vampire': 1,\n",
       " 'wife': 1,\n",
       " 'belize': 1,\n",
       " 'china': 2,\n",
       " 'flight recorders': 1,\n",
       " 'plane crash': 1,\n",
       " 'reuters': 2,\n",
       " 'russian oligarch': 1,\n",
       " 'abramovich': 1,\n",
       " 'live': 1,\n",
       " 'russia': 2,\n",
       " 'stops talks': 1,\n",
       " 'japan': 1,\n",
       " 'español': 1,\n",
       " 'powers remain': 1,\n",
       " 'threats lurk': 1,\n",
       " 'set': 1,\n",
       " 'webb': 1,\n",
       " 'telescope begins multi-instrument alignment': 1,\n",
       " 'scitechdaily': 1,\n",
       " 'uconn': 1,\n",
       " 'ucf': 1,\n",
       " 'ncaa': 1,\n",
       " \"women 's tournament second-round highlights\": 1,\n",
       " 'march madness': 1,\n",
       " 'bucking republican trend': 1,\n",
       " 'indiana': 1,\n",
       " 'vetoes transgender': 1,\n",
       " 'bill': 1,\n",
       " 'maggie fox': 1,\n",
       " 'coronation': 1,\n",
       " 'shameless': 1,\n",
       " \"'sudden accident\": 1,\n",
       " 'mirror online': 1,\n",
       " 'mirror': 2,\n",
       " 'plane crash –': 1,\n",
       " 'search': 1,\n",
       " 'moment flight': 1,\n",
       " 'daniel morgan': 1,\n",
       " 'report condemns': 1,\n",
       " 'met': 1,\n",
       " 'guardian': 6,\n",
       " 'rishi sunak': 1,\n",
       " '’ s spring': 1,\n",
       " 'statement': 1,\n",
       " 'bbc.com': 1,\n",
       " 'uk': 3,\n",
       " 'ireland': 1,\n",
       " 'euro': 1,\n",
       " 'vladimir putin': 2,\n",
       " \"'s 'lover\": 1,\n",
       " 'brass eye': 1,\n",
       " '’ s outtakes': 1,\n",
       " 'brutal tv comedy': 1,\n",
       " 'threatens civilians': 1,\n",
       " 'mariupol': 1,\n",
       " \"'s spirit\": 1,\n",
       " 'shell u-turn': 1,\n",
       " 'cambo': 1,\n",
       " 'green targets': 1,\n",
       " 'st helens': 1,\n",
       " 'dog attack': 1,\n",
       " 'girl': 1,\n",
       " 'bbc': 1,\n",
       " 'playstation': 1,\n",
       " \"'assassin 's\": 1,\n",
       " 'creed': 1,\n",
       " 'jade raymond': 1,\n",
       " 'haven studios': 1,\n",
       " 'nme': 1,\n",
       " 'nintendo switch': 1,\n",
       " 'folders •': 1,\n",
       " 'eurogamer.net': 2,\n",
       " 'fa': 1,\n",
       " 'solution ”': 1,\n",
       " 'liverpool': 1,\n",
       " 'fan group blasts “ shambolic ”': 1,\n",
       " 'wembley': 1,\n",
       " 'anfield': 1,\n",
       " 'manchester': 1,\n",
       " 'live erik': 1,\n",
       " 'hag': 1,\n",
       " 'utd': 1,\n",
       " 'manager updates': 1,\n",
       " 'manchester evening': 1,\n",
       " 'inflation': 1,\n",
       " 'government borrowing': 1,\n",
       " 'february': 1,\n",
       " 'crude oil': 1,\n",
       " '– business': 1,\n",
       " 'kremlin': 1,\n",
       " 'large-scale fraud': 1,\n",
       " 'sky': 1,\n",
       " 'natural gas': 1,\n",
       " 'gazprom': 1,\n",
       " 'retail unit': 1,\n",
       " 'insider': 1,\n",
       " 'zaghari-ratcliffe': 1,\n",
       " 'hunt': 1,\n",
       " 'iran': 1,\n",
       " 'debt payment': 1}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w = {}\n",
    "for x in all_titles:\n",
    "    for n in TextBlob(x).noun_phrases:\n",
    "        if n in w:\n",
    "            w[n].append(x)\n",
    "        else:\n",
    "            w[n]=[x]\n",
    "{ x:len(w[x]) for x in w.keys()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到，名词并没有给我们带来大型的主题组。让我们用从概念图中获得的更通用的术语替换名词。这将花费一些时间，因为我们对每个名词短语都在执行REST调用。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "trusted": true
   },
   "outputs": [],
   "source": [
    "w = {}\n",
    "for x in all_titles:\n",
    "    for noun in TextBlob(x).noun_phrases:\n",
    "        terms = query(noun)\n",
    "        for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
    "            if term in w:\n",
    "                w[term].append(x)\n",
    "            else:\n",
    "                w[term]=[x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "trusted": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': 9,\n",
       " 'brand': 4,\n",
       " 'place': 9,\n",
       " 'town': 4,\n",
       " 'factor': 4,\n",
       " 'film': 4,\n",
       " 'nation': 11,\n",
       " 'state': 5,\n",
       " 'person': 4,\n",
       " 'organization': 5,\n",
       " 'publication': 10,\n",
       " 'market': 5,\n",
       " 'economy': 4,\n",
       " 'company': 6,\n",
       " 'newspaper': 6,\n",
       " 'relationship': 6}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ECONOMY:\n",
      "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
      "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
      "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
      "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
      "\n",
      "NATION:\n",
      "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
      "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
      "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
      "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
      "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
      "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
      "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n",
      "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n",
      "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
      "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
      "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n",
      "\n",
      "PERSON:\n",
      "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
      "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
      "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n",
      "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n"
     ]
    }
   ],
   "source": [
    "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n",
    "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n",
    "print('\\nPERSON:\\n'+'\\n'.join(w['person']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免责声明**：  \n本文件由 AI 翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 翻译而成。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能存在错误或不准确之处。原始语言版本的文件应被视为权威来源。对于重要信息，建议使用专业人工翻译。因使用本翻译内容所引起的任何误解或曲解，我们概不负责。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.4 64-bit (conda)",
   "metadata": {
    "interpreter": {
     "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
    }
   },
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}