{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用强化学习训练平衡小车杆\n",
    "\n",
    "本笔记本是 [AI for Beginners Curriculum](http://aka.ms/ai-beginners) 的一部分，灵感来源于 [这篇博客文章](https://medium.com/swlh/policy-gradient-reinforcement-learning-with-keras-57ca6ed32555)、[官方 TensorFlow 文档](https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic) 和 [Keras RL 示例](https://keras.io/examples/rl/actor_critic_cartpole/)。\n",
    "\n",
    "在这个示例中，我们将使用强化学习（RL）训练一个模型，使其能够在水平轨道上移动的小车上平衡一根杆。我们将使用 [OpenAI Gym](https://www.gymlibrary.ml/) 环境来模拟这个场景。\n",
    "\n",
    "> **注意**: 你可以在本地运行本课程的代码（例如使用 Visual Studio Code），此时模拟环境会在一个新窗口中打开。如果在线运行代码，你可能需要对代码进行一些调整，具体说明请参考 [这里](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7)。\n",
    "\n",
    "我们将从确保 Gym 已安装开始：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: gym in /home/leo/.local/lib/python3.10/site-packages (0.25.0)\n",
      "Requirement already satisfied: pygame in /home/leo/.local/lib/python3.10/site-packages (2.1.2)\n",
      "Requirement already satisfied: gym-notices>=0.0.4 in /home/leo/.local/lib/python3.10/site-packages (from gym) (0.0.7)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /home/leo/.local/lib/python3.10/site-packages (from gym) (2.1.0)\n",
      "Requirement already satisfied: numpy>=1.18.0 in /usr/lib/python3/dist-packages (from gym) (1.21.5)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install gym pygame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们创建 CartPole 环境，并了解如何操作它。一个环境具有以下属性：\n",
    "\n",
    "* **动作空间** 是我们在模拟的每一步中可以执行的所有可能动作的集合  \n",
    "* **观察空间** 是我们能够进行的所有可能观察的集合  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action space: Discrete(2)\n",
      "Observation space: Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/leo/.local/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n",
      "/home/leo/.local/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import pygame\n",
    "import tqdm\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "print(f\"Action space: {env.action_space}\")\n",
    "print(f\"Observation space: {env.observation_space}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们看看模拟是如何运行的。以下循环会运行模拟，直到 `env.step` 不返回终止标志 `done`。我们将使用 `env.action_space.sample()` 随机选择动作，这意味着实验可能会非常快地失败（当小车的速度、位置或角度超出某些限制时，CartPole 环境会终止）。\n",
    "\n",
    "> 模拟会在新窗口中打开。你可以多次运行代码，观察它的表现。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/leo/.local/lib/python3.10/site-packages/gym/core.py:57: DeprecationWarning: \u001b[33mWARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.\n",
      "If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.\n",
      "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
      "  deprecation(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.00425272 -0.19994313  0.00917169  0.34113726] -> 1.0\n",
      "[ 0.00025386 -0.00495286  0.01599443  0.05136059] -> 1.0\n",
      "[ 1.5480528e-04  1.8993615e-01  1.7021643e-02 -2.3623335e-01] -> 1.0\n",
      "[ 0.00395353  0.38481084  0.01229698 -0.5234989 ] -> 1.0\n",
      "[ 0.01164974  0.18951797  0.001827   -0.22696657] -> 1.0\n",
      "[ 0.0154401   0.38461378 -0.00271233 -0.51907265] -> 1.0\n",
      "[ 0.02313238  0.5797738  -0.01309379 -0.812609  ] -> 1.0\n",
      "[ 0.03472786  0.38483363 -0.02934597 -0.5240733 ] -> 1.0\n",
      "[ 0.04242453  0.580356   -0.03982743 -0.8258571 ] -> 1.0\n",
      "[ 0.05403165  0.38580072 -0.05634458 -0.54596174] -> 1.0\n",
      "[ 0.06174766  0.19151384 -0.06726381 -0.27155042] -> 1.0\n",
      "[ 0.06557794 -0.00258703 -0.07269482 -0.00081817] -> 1.0\n",
      "[ 0.0655262  -0.19659522 -0.07271118  0.26807207] -> 1.0\n",
      "[ 0.0615943  -0.00051497 -0.06734974 -0.04662942] -> 1.0\n",
      "[ 0.061584    0.19550486 -0.06828233 -0.3597784 ] -> 1.0\n",
      "[ 0.06549409  0.00141663 -0.0754779  -0.08938391] -> 1.0\n",
      "[ 0.06552242 -0.19254686 -0.07726558  0.17856352] -> 1.0\n",
      "[ 0.06167149  0.00359088 -0.0736943  -0.1374588 ] -> 1.0\n",
      "[ 0.0617433   0.19968675 -0.07644348 -0.45245075] -> 1.0\n",
      "[ 0.06573704  0.3958018  -0.0854925  -0.7682167 ] -> 1.0\n",
      "[ 0.07365308  0.20195423 -0.10085683 -0.50361156] -> 1.0\n",
      "[ 0.07769216  0.0083876  -0.11092906 -0.24433874] -> 1.0\n",
      "[ 0.07785992 -0.18498953 -0.11581583  0.01139782] -> 1.0\n",
      "[ 0.07416012  0.01158649 -0.11558788 -0.31546465] -> 1.0\n",
      "[ 0.07439185  0.20814891 -0.12189718 -0.64224803] -> 1.0\n",
      "[ 0.07855483  0.01491799 -0.13474214 -0.3903015 ] -> 1.0\n",
      "[ 0.07885319 -0.17806001 -0.14254816 -0.14295265] -> 1.0\n",
      "[ 0.07529199  0.01878517 -0.14540721 -0.47699296] -> 1.0\n",
      "[ 0.07566769 -0.17401667 -0.15494707 -0.23344138] -> 1.0\n",
      "[ 0.07218736  0.0229406  -0.1596159  -0.57071024] -> 1.0\n",
      "[ 0.07264617  0.21989843 -0.1710301  -0.9091196 ] -> 1.0\n",
      "[ 0.07704414  0.02745241 -0.1892125  -0.6747003 ] -> 1.0\n",
      "[ 0.07759319 -0.16460665 -0.20270652 -0.4470505 ] -> 1.0\n",
      "[ 0.07430106 -0.35637102 -0.21164753 -0.22448184] -> 1.0\n",
      "Total reward: 34.0\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "\n",
    "done = False\n",
    "total_reward = 0\n",
    "while not done:\n",
    "   env.render()\n",
    "   obs, rew, done, info = env.step(env.action_space.sample())\n",
    "   total_reward += rew\n",
    "   print(f\"{obs} -> {rew}\")\n",
    "print(f\"Total reward: {total_reward}\")\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你会注意到观测值包含4个数字，它们是：\n",
    "- 小车的位置\n",
    "- 小车的速度\n",
    "- 杆子的角度\n",
    "- 杆子的旋转速率\n",
    "\n",
    "`rew` 是我们在每一步中获得的奖励。在 CartPole 环境中，每进行一步模拟你都会获得1分奖励，目标是最大化总奖励，也就是让小车尽可能长时间保持平衡而不倒下。\n",
    "\n",
    "在强化学习中，我们的目标是训练一个**策略** $\\pi$，它会根据每个状态 $s$ 告诉我们应该采取的动作 $a$，本质上就是 $a = \\pi(s)$。\n",
    "\n",
    "如果你想要一个概率性的解决方案，可以将策略视为为每个动作返回一组概率，即 $\\pi(a|s)$ 表示在状态 $s$ 下采取动作 $a$ 的概率。\n",
    "\n",
    "## 策略梯度方法\n",
    "\n",
    "在最简单的强化学习算法中，称为**策略梯度**，我们将训练一个神经网络来预测下一步的动作。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/tensorflow/__init__.py:29: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives\n",
      "  import distutils as _distutils\n",
      "2022-07-24 16:50:47.597258: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:47.597280: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "/usr/local/lib/python3.10/dist-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  import imp\n",
      "2022-07-24 16:50:49.838826: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-07-24 16:50:49.839078: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839143: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839194: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839245: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839295: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839345: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839392: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839441: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839449: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n",
      "Skipping registering GPU devices...\n",
      "2022-07-24 16:50:49.839649: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "num_inputs = 4\n",
    "num_actions = 2\n",
    "\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Dense(128, activation=\"relu\",input_shape=(num_inputs,)),\n",
    "    keras.layers.Dense(num_actions, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将通过运行许多实验来训练网络，并在每次运行后更新我们的网络。让我们定义一个函数来运行实验并返回结果（所谓的**轨迹**）——所有状态、动作（及其推荐概率）和奖励：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_episode(max_steps_per_episode = 10000,render=False):    \n",
    "    states, actions, probs, rewards = [],[],[],[]\n",
    "    state = env.reset()\n",
    "    for _ in range(max_steps_per_episode):\n",
    "        if render:\n",
    "            env.render()\n",
    "        action_probs = model(np.expand_dims(state,0))[0]\n",
    "        action = np.random.choice(num_actions, p=np.squeeze(action_probs))\n",
    "        nstate, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "        states.append(state)\n",
    "        actions.append(action)\n",
    "        probs.append(action_probs)\n",
    "        rewards.append(reward)\n",
    "        state = nstate\n",
    "    return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "您可以运行一个未训练网络的单集，并观察到总奖励（即单集长度）非常低：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward: 27.0\n"
     ]
    }
   ],
   "source": [
    "s,a,p,r = run_episode()\n",
    "print(f\"Total reward: {np.sum(r)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "政策梯度算法的一个棘手方面是使用**折扣奖励**。其思想是我们在游戏的每一步计算总奖励的向量，并在此过程中使用某个系数 $gamma$ 对早期奖励进行折扣。我们还对结果向量进行归一化，因为我们将使用它作为权重来影响我们的训练：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "eps = 0.0001\n",
    "\n",
    "def discounted_rewards(rewards,gamma=0.99,normalize=True):\n",
    "    ret = []\n",
    "    s = 0\n",
    "    for r in rewards[::-1]:\n",
    "        s = r + gamma * s\n",
    "        ret.insert(0, s)\n",
    "    if normalize:\n",
    "        ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在开始实际训练！我们将运行300次实验，每次实验中我们将执行以下步骤：\n",
    "\n",
    "1. 运行实验并收集轨迹数据。\n",
    "2. 计算所采取的动作与预测概率之间的差异（`gradients`）。差异越小，说明我们越确信采取了正确的动作。\n",
    "3. 计算折扣奖励，并将梯度乘以折扣奖励——这样可以确保高奖励的步骤对最终结果的影响比低奖励的步骤更大。\n",
    "4. 我们的神经网络的目标动作部分来源于运行期间的预测概率，部分来源于计算出的梯度。我们将使用`alpha`参数来确定梯度和奖励在多大程度上被考虑——这被称为强化算法的*学习率*。\n",
    "5. 最后，我们基于状态和目标动作训练网络，并重复这一过程。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 -> 29.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-07-24 16:50:51.475024: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 -> 135.0\n",
      "200 -> 484.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-07-24 16:51:35.910774: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:37.151017: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:39.284311: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:42.235074: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:44.691458: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:48.381946: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f40201e7b20>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABUUElEQVR4nO29d5gk1Xnv/z0VOk6OO7uzOcDuEpcVUWCiQEIGhYsuSsYWEvfK2ALLsi0s+1qyjS3bkvzzlSVb2JKMdLEAJYNkBCIIhERcwsImNofZmZ20k6dj1fn9UXWqT1VXh5npng77fp5nn+2uru4+1d3zrbe+73vewzjnIAiCIOoLpdIDIAiCIEoPiTtBEEQdQuJOEARRh5C4EwRB1CEk7gRBEHWIVukBAEBHRwdftWpVpYdBEARRU7zyyisjnPNOv8eqQtxXrVqFbdu2VXoYBEEQNQVj7Eiux8iWIQiCqENI3AmCIOoQEneCIIg6hMSdIAiiDiFxJwiCqEOKEnfG2GHG2JuMsdcZY9vsbW2MsccZY/vs/1ul/e9ijO1njL3FGLu2XIMnCIIg/JlL5H4F5/wczvlW+/5nATzJOV8P4En7PhhjmwDcDGAzgOsAfJ0xppZwzARBEEQBFlLnfiOAy+3b9wJ4GsCf2Nvv55wnABxijO0HcD6A5xfwXgRB5OC5AyN44cAorji9C+eusC6gXz82jqd2D+LCNe24eF0H9g9NYWgqgZ7mMPrGZrGqPYoDw9NY09GAH77aB845VnVE8b4tvRiaiuN7Lx6DYZq+79cU1vHB81fgO88fQSyZBgCEAxp+++JV+Okb/bjhnKUIaip+9Gofrt28BE/sHsQVp3ehKaRnvdaTuwexaWkTeprDzrZn9w1jaUsYazsb8Oy+Ybx86CSu2bQEZ/Y2Y/uxcTAGnNXbgl39k4ilDAQ1BT/feQJbVrbi8tO6Sva5pgwTP371OP7Heb1QFIa0YeJbvz6ElMFxy8Wr8P1txzA2k4SuKvjIhSvxyI4BDE7EneczxvDus3rw2tFx9I3H8P4ty7CyPVqy8RWiWHHnAH7OGOMAvsE5vwdAN+d8AAA45wOMMfGpLgPwgvTcPnubC8bYbQBuA4AVK1bMc/gEQdz937uxs38S2/smcO/HzgcA/N8n9+GpPUN4YvcQHrnjUlz9lV+6ntMc1jERS+HWt6/GN391yNl+3RlL8INX+vCPT+wFY9nvJZZ/mE0a+Mrje12PaQrD3Y/sRnNYx8aeJnz6we343LuSuPuR3fjb952JD56f/Xf+u/e9ik9cugafufY0Z9tHv/kSAODwF6/HX/90N94anMJbg1P4xke34sav/dp57Es/fwsj0wn0tobxyJsnsKYjWlJxf+7AKP74h29gXXcDtqxoxcuHx/A3j+wBACiM4e8e3ePsO51M4xvPHAQA53PjHPjqU/tg2p9ZLJnG567fVLLxFaJYcb+Ec95vC/jjjLE9efb1+Ukga0UQ+wRxDwBs3bqVVgwhiHkyk7Ci55SRibTF7Xja8H3ORCwFANg/NI3VHVHcefV63HH/6+gfj+H4WAytER2v/Z93ZD3v5cMncdO/Po/DIzMAgKc/czkm4ync8M+/xlTces1YysBs0nrf0ZkkACCZzr4K4JwjkTaRNPyvEDjnznGkjGyJmE6kEU8ZiKesfXK9znyZtD+juH0sh+xjBoDR6QQA4BsfPQ933P8anj8wCgD47q3n49L1VjeAExNx/MXDO3DJug58+ed7fT+DclKU584577f/HwLwY1g2yyBjrAcA7P+H7N37ACyXnt4LoL9UAyYIws2MLT5pMyOAhn1bCEpAc/+pa4oVg+3sn0RnQxBLWyxb5Ph4HAMTcZdNItMctqyVoydnnfuq/VoJ+70SadN5X3HikcfmHaPh8xhgianJc++TSBlISu+V9jkBLIRZ23ISJ41DI9POY+LkGNJV9LZGsOP4BACgtzXi7LOkOYRvfHQrfuuiVdBVhlSO4ywXBcWdMRZljDWK2wDeAWAHgIcB3GLvdguAh+zbDwO4mTEWZIytBrAewEulHjhBEBYxIe5S5CrEVAjuao/X2xIJAABGphPobApimRD3sRj6x2NY2hLyfS/hmwtxbwxp0BTF9V7JtImkYY1JiLuff58uIO6vHBmDYYu76bMcaDxlImXwjLiXWDxnEtYxiNc/NDKLzsYgAEncNQW9rWHHesn1uWmK4vp+FoNibJluAD9mlpGkAfhPzvmjjLGXATzIGLsVwFEANwEA53wnY+xBALsApAHczjn3vzYkCGJBcM6dCFMWSdMTubdG3cnMprCGEdta6GoMoqsxCFVh6B+3xP1tq9p8309E7kNTCTQENWiq4kTu8ZT1Z25F7tb7TzmWUbbw5hL3oKYgkTbx+rFxiHOC3wkgljJctk46RwJ4vsx4xn5oZBqnL2nE8FTCEfegrqK31ToxdjcFEdT8CwM1lZX8yqIQBcWdc34QwNk+20cBXJXjOXcDuHvBoyMIIi+JtOlEjXLkmvaIu1dYGoKZP/3OxiA0VcGSphD2Dk5hMp5GT44INKQrCKgKkobpCL3msWWSkuBOx7NPPM4Y7X0MT1QuovSZRNp5njcqT6QNxFMGUkb5bBlhd6UME4bJcfTkLK7a2I1n941ItoyCZS2WFSNbMl40hZX8yqIQVdHylyCIufP4rkGokrHqity5sGVsa8FjCShSKUxXoyXky1rDeOXIGABgaQ7PnTGGprCOkekEmmxxz47cjYznnsztuYuI2PQ85kT0PCP83n2m42lb3DNJ11JH7o7nnjZxfCyGlMGxrrMBusqcZGtQy0Tu4n8/NFUp+fgKQeJOEDXKJ77jXgNBrpYRQm9yK0JOpk1cvbEbl6xrxxd+sssRLsCyZQBgWUsYLx06CQDoafaP3AGg2bZ0msOWfOiqj+du357O47n7JVQNkzvllqbJHVH3RvdWpYwJk/OMLVPiyF2MPWmYGJyy6td7WkIIaqorci9K3BXma02VE+otQxB1gKqwLJEUJA0TKcNEUFccT1gkCwE4SUJZnET1jB/Cjmn2RO7iKiEhJVSFLeMnvOJkZLjsJPcJKlfkPhFLIWmYSJvcuWJImxzcJ/E6X2alhKpIWkcCKkK64lg2IU3Fmo4GNAQ1nLmsJedr6Wp1JlQJgqhymkKabykkYIlTyuAIqIpTEimiUiATuX/0opXYPTCFkzMJLMkbubvFXXNsmdyRe95SSO4/boNz330AYHQ66dyelU5UaZNDV/2m2swdYSmlDBMx+wQS0lVX0jSoK4gENGz7s6sR1HLHyppKnjtBEPOgMaS7JsnIIinqznWVOcInKkF0laHVLovsagzh32/ZikIIr12URaqqN3LPeO5iMpNvQtXMjtxl68KULBpv0DtsV/oA1uxQ+bj1EnWykieHiauDsK4iqGdEXAh9qMCb6oriss0WA7JlCKIOaAp7InfujdxN6KpV6QJYEe4Zy5rwpZvOhqLMLdLNFbknpMg9kXYLmV8y0UmoFhO5e54/PJURd84z1lApBVScmGRbJhzIRO66ypz3LUQlSiFJ3AmiDmgM6i4BNCR7QtSCBzTFNVP1tO4m3HhOVtungjjiHhHi7k6o+rUU8BM2v4Rq2vD33L2R/4gUuQNAxI6cSymgM84M1YyvH9Isz13cLha1AqWQJO4EUQc0+njuYVvwROQeUBWnsgUAAtr8vOncnrs8ickbuRebUHWXc4pEqtgsTlgjkucOWBE1AKRKWG4oz1CN2VclVuRufYayPVMIvQKlkCTuBFEHNIV1V9RqmByRgJVSEx64rrojd1no5/pe8v+KwsBYRtzFyUTGz3PPVQop3/ZG7mLMI1PuyD1qT8rK1cpgPsieu0ioBjXF8ddzzUb1Q1PIliEIYh40hfQsYYzY0WwsZcDkljC6Ivd5intHg5WAbY8GnG2awiRbxigycs9MVspsc9symYSqW+S9tky4xLZM2sjkDURCNayrYIw5dsxcI3dKqBIEMScUZtVfu2rEOXciTGEvBDTFJeh6ntK9fFy2vhP/8uEtOHNZs7NNlcQ96WPL+DcOs7aZOSJ3V+WMp4FYlucubJkSCaioYwcyCVVh/QTn4blXohSSxJ0gapxIQIOmMpg8I5Ry5D6dsGZT6ioriS2jqQreeWYPmNTCQCRVAf+EqizUYzNJpO0JSIC7kibtEvfs0k7x+NhsyvX6kRLbMvIM3qRty4irg5BT/lj852d1hSRxJwhiDoQDqpPUlDsthh1xz0Tu8gSffJNu5oomva5fKaQjzoaJy7/0NH7wSp8jdnJQLwugN4cg1717EdUypZriL8/gtRKqhhOxO//P1XOnhCpBEHMhGlCh2VG47E2HHVtGTFjyJlRLM5MTyFTMAPmrZRJpExOxFAYnE75dIYUAWr1Y3DaTd5aqjLhKKZWAzkgzeFOGiYQcuevziNyrseUvQRDVTTigSZG7CUB12zJxSdxLkFD1Qy0g7sJzFyKfkmwZv1LIoKa4rB3D5Hktl0iwxJF7UhZ3qzlZ2KmSmXvkTglVgiCKQm6QFQmojrimnQoUjrBdCin6u3gnMc03oeqH23M3cnruhkvcferc7f0CWkYMVYXBlMS9JeJeeASAU/ZZKs/dmZGqq1kJ1XlF7jSJiSCIYpB1IiLZMmnJmxaRpiPuKnMlUeebUPVDjtz9q2XcydOkYWZORD7VMpa4W7c1hblsmWU+HSszpZCliY7Fe0eDmp1QNaX69nl47iolVAmCKAJZEHtbw44tI0/8EbaM7LnLgl7ShGqRnrvhY8vIvWXEDNOgpiIlFvdWFZhmphLIT9yjwpYpUXQsTkKRgOqqcwespfWAuUXu1gLZ7s/k+QOj2D80neMZC4fEnSBqECGIn7pyHT5/w+aMLWOajoBmqmVkcc+IcCkjd82TnJXrxAHJc7ej11SaZxKqcuQu2TLC2glo1tR9cTJY5rMohrCgShW5i3FGApIt45RC2nXuc2g/qSoMnLuP9a4fvYGvP72/JOP1g8SdIGoQIRLRoIagJpVCGhlvWlOsFr9T8YznzhhzEqmlTai6X0vU1gu8FowroepTLROUPHddVWBKwugXuTuNw0oUuYv3diL3tOFE6kGPPVMMumObZU4+8VSmlXA5IHEniBpERO5iLVTZcxdiqSqWkMu2DAAnqVrahKo7chcnFIF3wlIuzz0tee5OszDN3c5XXtxb4JRClqr9gHTyFJF7KOCO3INziNw1T8Lbeg8TyXT5fHgSd4KoQUQAKHqxy5678KZVhSGoq1JC1RZ120IpZZ27t6/5tEfcvTNMc5VCOglVn8SvSHL69VAXM1RLVecu7J2wriJhT8ryeu5zidydk68k7kmfmbylhMSdyMlXHt+L2+97tdLDIHxwonNb5+TFKtKmO3J36tw1Ieqi2qN8kXva5JC6Ezii63juRsZzl3vLCAGXo2LdtnxEklZVGM5e3oLTlzQ67xstceSekjx3cRWyEM9dnEhTnlYLqTSJO1EB9p6Ywu6ByUoPg/DBMN1RrF/krjCrl4xYhi7gtWXKmFAFgGggY5/4ee4pH89dJF7dDc6s15bF/aHbL8Gjd17mHEu4xDNUM0lpzYmuM43D5hG5K9mRe8rIbo1cSmiGKpETs8CUb6JyOJ67ku25p6WEalBTnH4suieRGijTJCZBQ1DLWiBb/J9Mm46Q+3WClNvpivEmDSv5KNsy1jEZziSmUs1QTUmlkAIRqTfYZZdRH+8/F5p0ZfXxe1/G29d1ODNfywWJO5ETk/NFn3hBFIcTubPsyF08pijuLpDidjkidz8fvCGkAZPu8bqqZfJMYgr6eO4i+ahKfo84lkxCtbSlkFEfcV/b2YCvfOBsXHl6V9GvJ65sDJPjlSNjTlLYOx+glJC4Ezkp1M+DqByygAOQ2g+YTlSvKf4tfr0RfCnweu6Au6pFiK6wTVIG902oytUyAnE7KbUjcB5T3f53qUohxXhl71947owxvG9L75xeT5NKIRNp05kHUE5bhjx3IicmL90fC1FahNUioliRsJNtGZFQFYh9vBF8KfB6/4Bb3DMtf+XI3U6oSj8xR1R9TkqptI+42wuQyMdfClKmtcC4PA7ZopkrumPLcCTTptMvnqpliIpgcu67gg5ReQzHc7fui0lE3oSqHHlm7Bh31UwpELaD7EOLlgBApi2AEPmklFD1W6zDrzVxKkfkHtAUKWFZulJI1XPl4zd5qlg0x1qyqplEv/gU1bkTlcDknCL3KsWQBBxwJ+zkhKorclc8tkwZEqrruhqcbQ1BewFt5l/nLloNyPGD47lLTbkcYfQRd12zZuFqUmRcClIGh664e/H4tT0oFnHym7XtGBG5ky1DVATy3KsXU5qFKv/vTaiKRaw1hTn+fNATwZcCp948qOGs3mYAmaqSSECzF7vOXAmm0typSHG1HzBMMOYurRQnqISfLWNH7orCXCeRhWKYHJrqtbXmL5fixCpmC4vIvZwJ1aJHyxhTGWOvMcZ+at9vY4w9zhjbZ//fKu17F2NsP2PsLcbYteUYOFF+5H4eRHXhrZaRPWc5obq+u8HZLnASqz7li/NF9twv39AJADgxGQeQqQ83pHxAyjCzKmjEODWFuStiJEsDyK6WEVcgmqpkdV6cL2nThKYqTouGpc2hBb2e+HzEIiDV5rnfAWC3dP+zAJ7knK8H8KR9H4yxTQBuBrAZwHUAvs4Ym38mgqgYJkXuVUt2tUzGc09Lj63vbsx6rlhLVfGpcJkvItLWFIZbL12Dqzd24yMXrgQgL4HHXZ67XGZrSraNqrjH5u0t461zz5ysSreUnWXLZCL3pQvw24HMyVcsAlI11TKMsV4A1wP4d2nzjQDutW/fC+A90vb7OecJzvkhAPsBnF+S0RKLivDcOU1kqjocW8bHczelqH5Dd0PWc7193UuBE7mrDM1hHf9+y1a8fV0H/uja03Dd5iUA7EoeqVpGFjZnkpPtdWdPVHLPUBU0BDVnJqymKiULRtKGFbmLCLtngeIu8gZC1MWxmLx0SeCs9yxyv/8PwB8DkMOAbs75AABwzgcYY6KifxmAF6T9+uxtRI0hgiCTZ3qYENVBVvsBaZKMnFBd0pRtJ1iRe2nFXSRU5da/jDHcfsU6fOtXh6yxGdxV5y4LsThZGaYJVfW3Zfwi9z++7nRHgL2Lai+ElO25T8as1sUr2yILej1x8p1NpLMeSxkcc1jUqfj3LLQDY+zdAIY4568wxi4v4jX9ZCDrdMoYuw3AbQCwYsWKIl6WWGwyl8omVIWctWpC6KLQwMxiHVIppMLAWPaf403n9WJTT1NJx6Mp7isI12NOPsDdCdK7ADZgi6qiuG2ZPJH76o6o631K1vLXMKEpDB9423L0T8Txu1esXdDr6Z7IXSaZNp28RCkpJnK/BMANjLF3AQgBaGKM/T8Ag4yxHjtq7wEwZO/fB2C59PxeAP3eF+Wc3wPgHgDYunUrXfdXIZloir6easNbLaNJnrshJVQB4I+uPQ394zHnueeuaMW5K1pRSlTVXbXjekw68ci/pZgkdGLMhiESqpnnC2FM+ETuMppSuoSqYZ9kIgENf/qujQt+PXGCm/GJ3MuVVC14bcY5v4tz3ss5XwUrUfoU5/wjAB4GcIu92y0AHrJvPwzgZsZYkDG2GsB6AC+VfORE2fGrZiCqA2+1jF/LXxH93n7FOtz93jPLOh5xIvErr9QkcZcj61lZ3IUXb1qTh7y17EBmwo/f1QFgCWipfqspg5elVFRUy7jfq7Keux9fBPAgY+xWAEcB3AQAnPOdjLEHAewCkAZwO+e8fGtJEWVD5FFJ3KsP0yPguuS5mx7hXwxUx3P3E3f7qsLjs8dSPpG7Pe1ftmW8XSGVHMellbBaRpRClgrxWrMJf1umHMxJ3DnnTwN42r49CuCqHPvdDeDuBY6NqDDiD45mqVYfRo5JTLL1kcu+KAeaxx5yPebjuQNuW8ZbCimfmDK9ZfIfl64qpUuo2vZQqRC9ZWZ91kwtV+ROM1SJnJDnXr1ktx/ILAZREXEvwnO3Zs9mhGxWsiicQMIwoav+CVW/ahnv+5SyK2Rpe++IyL2KPHfi1EWOpojqwptQFXpnmGZWQnUxyFstI/V9kX9L8VR2tYzhG7lbtwsmVFWldOJuj6NUiJPftJ+4l8mWIXEnciJXMBDVhQiAhQgyZjXQkm2ZUs5ALYTw3P2W25MreVwLRPuVQopqGU9bX8C/FFLGmqFaOlumlAlV0eph1qcUslTNzryQuBM5EQJCS+1VH+I7kXOLqsJcjcMWM6GqOd5/tqSoOTx3IDN+OXLXctgyfr1lXGMoYZ27YZq++YP5Ik5Is4tYLUPiTuREnjVIVBemj6+uKYpr5udieu5+i3VkxuXvuQOZDpWmk7w3cydUbRHMJbqaopRsgey0wX2vQuaL7tS5L161DIk7kROTqmWqFm+1DCDqvM2KiLteREL1yT1DGJtNuR4TfdtF8Jq27RA5lyleW4hgroBaU0uXUE2ZpU2oMmZZTTGfaplyJVRpDVUiJ/IfHFFdeKtlALvO22eG6mLgeO4+7ylE8l+ePpD1mIjcRcSdNjkiiuI6Lm9vmXyRe6n863SJSyEB67PxqzyjyJ1YdDiVQlYt3moZcVsuhVzMhKpTLeMT7ea7ggjqti0j8jtmdkJVdxbIFseVewylTKiW0pYBci/2UY0zVIk6hyYxVS+Gp1oGsCLXB7YdA7ZlP1ZuivHc/RBRufitpeyGXf4JVcvSyJdQLVnL3xInVAFrTVm/UkhKqBKLDvWWqV4y7Qcy27yRplriyDMfmWqZ3J67HxnPXa6W8a9z91tDVSakq5iMp0uy/kCpE6oA0BoJ+G4nW4ZYdKi3TPUibBnZm055RKJaIvd8iUlhy8gLaKuexTrkZfYUBt82xgBw7ooWjEwncGB4en4HIZEucUIVANqiOcSd6tyJxYYi9+rFr1qmfyLu2mdxq2VyNw7zbgvrmd7lq9qtfuwZcTehK8x10sqUQvK8Vsll6621W5/ZOzKfQ3BRjoRqq0fcQ7o7UVxqSNyJnMi1x0R1YfpUy3ipRJ27X7TrFUl5YYqzepsBSHMqDO7T8leK3PMo1vK2CNZ0RPHLvcPzOwgbbi8vWcqukADQ5rFlxPKAZMsQiw41Dqteiqllr8wM1blF7qctsVbuNEyOlGFiKp5GUFf869yNwknO81a2YtfA5LyOQSAKCPQyRe7i8wjpKhRGkTtRAYSmU7VM9SFs2nwCvri9ZfJVy7hlRtgRQEboDc7x852DmEqkceXpXW5bRnF77vloieiYjmdXpMwFMa+j1AnptogOIHNi1lUGXVWoKySx+JDnXr34VctcvbErx97lJ1/LX2+wLdsyTjtgg+O7LxxGb2sYv7Ghy9XtUhbZQlZTY0hHLGUsKBoWS/XpJS6F9HrumqogoCpkyxCLi1xORuJeffglVP/9lrfhb99X3uX0ctEeDYIxoKMxmPVYV2MI//nxC3DB6jYAQEhTcdN5vbjv4xc4EbrBObYfm8A1m7qhSglVb58Zv8ZkMg1By8feOziFFw+OzutYRORe6lJIb7WMrioIaKVbYMQLiTvhiyzoJO7Vh18pJABEpKh4MVnVEcXzn70KW3IsvH3xug6nzltTGf7hprNxyboO5+QUSxqIpQx0NFgnB0VqZeyehZt/HI0hS9z/4bG38Mn7Xs277/HxGL765L6sunhRQFDqhKq3zl3YMmKFqVJD4k74Irf5Jc+9+shVLRMJVG7S+ZLmUN7HRU27XFEjPPqR6QQAyzMHpOUDGfP0z8kvWULcD43MYHw2mXdC06cfeB1ffnwv9pyYcm0XkXupE6p+kbuuMfLcicVF/puglr/Vh9N+wCNA0QpF7sUQsmejymMWSd/RmSSATHQr9N9bFlnIBm8MWSeH42MxmNx/cQyBuPqZiLk7VWZsmfJG7prCLM+dessQi4lsxVDkXn0Yji3j3h4JVu+ftKiSkStqhJ8+KiL3sCXOIlpXmPsYi43cxW92OpFGNMdn0mSfCCY94i4SqqWexBT2nHgDmoIfffIS6Fp5qpoocid8MSmhWtWYJvedil8pz70YQnp25C5uj05bkXtLxF0LrirM5bsX0tsGj5BP5SmLFCeC8ZyRe+lF98/fvQnf/98XQVMYNIWhOaKXzUqr3tM8UVFkJ4bEvfowuP8CztUs7kEfcRe2zIiwZaLuyF323g0UXrRa2DICvy6M3n1P2u8tKNQ3fiHc+vbVAGy/vcS2jxcSd8IXityrGyty9xP36v2TFraMvPiLprhtmVZP5O7YMwoAo3AppIjGBX4TmoYm43h234h01ZBwPZ6WJhmVi4CmOG0VykX1/hKIikLVMtWNYdZe5B7ytPcFMuI9Op1ESFeyrBsngnci+fzvEdQU6CpzVmSaiqey9rnl2y9j98Akrj+rx3lvGaNMpZAyuqqUvBrHC3nuhC8m1blXNSb3bxoWLHM0uBCEcKek35NT554yXNUkXltGcTz4/MfHGHNZM1M+tsxBuyXwrP3YsCdyT5WpFFJmWUsIS5rDZXt9gCJ3IgeyntMaqtWHyblvcjFXr/NqIOT0bs8kdOTZpy2SuGdE3X2/GKekIag5PrqfLZOwp/uLZKs3cnd6y5RR3O+/7aKyd+0kcSd8MVyeO9W5Vxu5bJlqxoncpWBB7hvTGslE3KrHjnHWaC0iySn77vkSqo64z3gi90WwZbxlkeWgeq/hiIpiUp17VZOrWqaaySRU/SN3ly2jiP99EqsFkMsh84u75cePTrtnsjozVBdxmcJyQOJO+OKqlinBmpREaclVLVPNiISqHCzIYt0sR+6KN5Gau+ukF+G5t0R034SqQETuaZO7Zqm+2TcOxoCeMnvi5YZsGcIXOVg3yHOvOmrRlgk6y8pJtox0glrVHnFuZyVU7fthvbBkNYU0RAMq2iKBvJOY5GTrRCyFWMrAd54/gp9s78clazvQ6dPhspYgcSeyeODlo3jlyJhzn2yZ6uJLj72Fh7f3Ox0UvfzeFevK1q9kIQSdUkjJlpFOUGf3tmRtZ053SGv7ms5owfe5/PQuRIMatveNZ9kyuRqJTcbS+PvH9uDZfdb6q3devaHg+1Q7BcWdMRYC8EsAQXv/H3DO/4Ix1gbgAQCrABwG8AHO+Zj9nLsA3ArAAPApzvljZRk9URYeefMEnjuQWWSYSiGri++/cgyJtJkzcv/Mtact8oiKQyRU5eorubrnjGXNzm1vXfsJe/HvtUWI+w1nL8UNZy/Fh//9haxqmRlPI7HGoIapRBqP7BjAs/tG8NsXr4KuMlx/Zs8cjqw6KSZyTwC4knM+zRjTAfyKMfYzAO8D8CTn/IuMsc8C+CyAP2GMbQJwM4DNAJYCeIIxtoFznrs9G1FVWOtZUkK1WomnrMi3xlwZJ0GZylF9JTf4Ujyeu/gNru1sKPr9GoIaRqdnXdvGZ91lj+0NAUwl0th+bBwA8PtXrkN7jiuiWqNgQpVbTNt3dfsfB3AjgHvt7fcCeI99+0YA93POE5zzQwD2Azi/lIMmyot3ZRgqhawu4ikrTlrMNVJLgSgtLHbehKqwrGOcm7jrWZ77+Kw7wSqE/NiYdRJoDrt709QyRXnujDEVwCsA1gH4Guf8RcZYN+d8AAA45wOMMbGA4zIAL0hP77O3eV/zNgC3AcCKFSvmfwREyfHaMFVo356ycM6dSTj5FseuRkSJ4paV7tWavvHR87Cppylrf5WxrGP0rkOaj8aQllUtkyXu9uv1j8fRGNTKWtu+2BQl7ralcg5jrAXAjxljZ+TZ3e8Xl3Wq5pzfA+AeANi6dStd91cRqSxxJ3WvFhJp/2RkLdAc1vGzOy7Fqna3b37t5iW++yvKwq5OQrrq+rwAYDyWbcsAVkDTEq2fqB2YY7UM53ycMfY0gOsADDLGeuyovQfAkL1bH4Dl0tN6AfSXYrDE4pD2hOrkuVcPwpIB/HvLVDsbfSL0XFhL7Fm37/v4BU5CtliCmoJE2gTn3EnceiP3kK4iGlAxkzTQEi7+qqAWKHgNwhjrtCN2MMbCAK4GsAfAwwBusXe7BcBD9u2HAdzMGAsyxlYDWA/gpRKPmygj2bYMiXu1IJKpQO1F7nNFkZbYu2RdB87z2DmFEHX1clmod0m9gKa4Jj3VE8VE7j0A7rV9dwXAg5zznzLGngfwIGPsVgBHAdwEAJzznYyxBwHsApAGcDtVytQW3oQqRe7VQyKd+VMy63zmsKqwBV2dBGz/PJE2nRr7eMqAwqzkbjJtIqgqaAxpODHpblxWDxQUd875GwDO9dk+CuCqHM+5G8DdCx4dURG8Yk6Re/UgR+6xVH3HTCpjC7o6ESs/JVImELK2xZIGQroKTWFIpk07crdksKWOKmUA6i1D+OAtVaPIvXqQPfd4sr7FXVGyq2XmguhtL1/txNMGwrrqCH9AU9AUPnVtGeIUI+2pjjFJ3KsGWdxPhch9IdUyGXGXrnaSJkK66jQsC6gZz72eatwBEnfCB+8al16xJypHPH0K2TILjtwlW8YmnjYQ1BXHy9clW6b1VPPciVMP2YbRVYU89yrCZcuk6vuka9W5z//5olomkTYwMZvCwZFpxJOWLSMIqAqa6rRahjx3Igu5zl1XGXnuVcB0Io2UYbrEvd4JqIpT8TIfZFvmP547jP95zwuYtROq4jFXQrXOxJ0idyILeYZqQFMpcq8CfvOrv8INZy/FspbaXkBiLvzNe89E2xzaDXhxbJm0iZMzCSTTJsZmk+hoCDplpEFNQZMj7vVly1Dkfgryy73D2Nk/kfNxWcwDKqMFsitMPGXg0MgMjozOuCo/6p0L1rRjfXfjvJ/vRO4pA9MJ63MbmU4ipCuuyP3y07rwWxetxMq2SM7XqkVI3E9BPv+TnfjXZw76PsY5d4u7ptT9ZJlqR/QyH4+l6t5nLyUhPWPLzNiLdpycSSCoq05UH1BVLG+L4C9vPKOumoYBZMuckiRSJpI5IsCUJ0oPaAp57hVmQIj7bOqU8twXimzLiBWZTA6EdRUpxTpJBrT6EnSZ+j0yIicpw8xptXj9dV1VshqJEYvLickYAKsvSjxt1H1PmVIhT2KSl9sL6YqzWDeJO1FXpAwzq62v85inpl1XFRhky1SUExMJALa4p0yE6liQSolc5z4jibs1Q9X6DMXqUPUI2TKnICmD54zGvRG9rjLQHKbKcmLCitzHZ5OIpQwEdRVfuulsLGkOVXhk1U1Q8tzdkXumzj1YxydKEvdTkHy2jHc2qqZQQrXSCM/d5MDIVAIhTcE762AB53KT6QrptWVUiJ90QJ1bj/haon5PW0ROLFvGLeJ/+uM38Wf/9WZ25K5ZM1RfOzqGH77St5jDJGxOTMad24OT8TkvWnGqoigMAVVB3GPLeCcx1SsUuZ9iGCaHybMTp7sHJqEr2a0GAiqDyTne+/XnAADvP6930cZKWAxMxNEWDeDkTBKDk4k5rSN6qhPUFEzEkpB/1iFdgWlSQpWoM8RCHN6Sx5RhIm2aWQt1aB7B55JF8+PX+vDer/+6jKMlOOcYn01ibae17ujgVNyp3yYKE9QVjE67100N6yresbkbd169Hq111nJAhn4lpxhiyTFvQjWZNpE2eVZNu7BlBLJ3uat/Eq8dHaf2BGUkkTaRMrjTdoBzOGV8RGGCmoqTM25xD+kqVrZHcefVG5y1VesREvdTjJTdMtYr4lYFDc/23BXmuqSVFxgWfbJpYk35ECfTZa2ZnjIUuRdPUFMwmiXup8bnd2ocJeEgRN1rvyTTJgyTZ1XLiJa/oh5YXmBY9Mmu977ilWQ6bot7S6bvCSVUiyegKRidTri2nSqfH4n7KUZSRO6eCD3peO7u7ZrKYHDurFbjEne7hUGszpd7qyQicu9oCDilffVcm11qgrqKSfsEKZKnJO5EXSIidm+EbiVUuW/7AdPkTs9rsmUWlylbmBpCGi5c2w4AOKu3pYIjqi3kE2F3UxAAXIt11DNUClmnfPSbL2LH8Ql8/obNuPGcZc52EZl7Pfdk2rQ9d7foBzSr/UBD0Bb3WMa/FKJOtkz5EJF7Y1DHdz52foVHU3u4xL0xhGMnYxS5E7UL5xzP7hvB2GwK2w6PuR5zInefUkjD5Fk9ZzSFgXNkxN03cqf+BOViOmF93g0hisPmQ1CqLOpusto1nCqRO4l7HZKrdBGQ69wzgsw5t6plTBOGT0IVgNOJ0O25U0K13IhFJsTJlZgbor8MY1beAjh1qmXoF1OHJCXhFp6tQLZlxmaS0FTmJJrSJs9KqIoqGZGIHZ/N2DKUUC0/olqmkSL3eSFsmdO6GxG1T5Cnii1Dv5g6RAgxAFdPDSATsRsmx//+f6+gtzWCL9y42drmU+eu2svPi+f5lUJSQrV8TCdS0BRGFTLzRJwcr928BBeuaUf/eOyU+SxJ3OsQWdy9towc1Q9MWE2oktLEJrmKRlUYxMpjIqL389zJlikf0/E0GkJaXc+kLCfbjlg5p3ds7sbmpc24yK44OhU4NU5hpxiJfJG79NhsMo1k2nSVR8qRu8IAxRYV38idbJmyM5VIk9++AP72fWfiytO7sKmnqdJDWXToV1OHiOg8ElAx5RF3uQRyNmkgaZg5I3eFMSeRKsT92MlZ7B6YxMaepky1TI71WImFMx0ncV8I125egms3L6n0MCoCRe51iBDi1kjA8Ry9jwGWuKcM0zkZcA4kpcjdsmWEuFvNq6JBDbd9dxuAjNcep8i9bExT5E7MExL3OkRE4u0NAcRShqs0UvbjxX1Z8BOSf64w5ni9ScPExp4mfOTClTh2MoZE2iDPfRGYTqSpxp2YFyTudYgQ8DZ7UQc5qeotdZRtGSDjn+sqg8IAVfLcVSUzhbt/PO4sVUbiXj4ocifmS0FxZ4wtZ4z9gjG2mzG2kzF2h729jTH2OGNsn/1/q/Scuxhj+xljbzHGri3nARDZOOIe8RP3ApG7/dyQpkKRqmWSaROqwtBlz/I7enLWeU4sSTNUy8EH73kBB4dnTpkZlURpKSZyTwP4Q875RgAXAridMbYJwGcBPMk5Xw/gSfs+7MduBrAZwHUAvs4Yo1/nIpIw3JH7TB5xTxkmkulMNC989FBAhcqYq1pGYQxdjVbkfkwSd6pzLz2JtIHnD44CAJrC9btaEFE+Coo753yAc/6qfXsKwG4AywDcCOBee7d7AbzHvn0jgPs55wnO+SEA+wFQx6NFRJQ7irU25VmqWbZM2nTVvovKl5CugDF3QlVTmNOfg8S9vIjv7PqzevDJy9dWeDRELTInz50xtgrAuQBeBNDNOR8ArBMAgC57t2UAjklP67O3eV/rNsbYNsbYtuHh4XkMnchF0sjnuXsjd+6qfY+nTCgMCKgKVCXTUwawVpNviwSgKQzHxiRbhsS95Eza8wnesakbHQ3BCo+GqEWKFnfGWAOAHwK4k3M+mW9Xn21Zi2xyzu/hnG/lnG/t7OwsdhhEEXgTqvlsmazIPWVAUxToqgJFsmUAK7mqKAydjUG3555H3GNJAz97c8C1sDZRGLHARFOILBlifhQl7owxHZaw38c5/5G9eZAx1mM/3gNgyN7eB2C59PReAP2lGS5RDE4ppIjcJVsm6RV3T7VMIm1CUxk0lbkmMQGZKL6rKYSjo5a4B1Ql7wzV/3r9OD5536t49ej4wg7qFENE7k1hqpQh5kcx1TIMwDcB7Oacf0V66GEAt9i3bwHwkLT9ZsZYkDG2GsB6AC+VbshEIYSAO567HLmnsyPoWUmc4ykDqsKgKQpUxR25K0LcG4NOZNkc0fN67vsGpwEAz+wl620uTMZtcafInZgnxUTulwD4KIArGWOv2//eBeCLAK5hjO0DcI19H5zznQAeBLALwKMAbueckym7iIhIvDWSbct4l9fzPh5PGdBVJVPnrrhtGSBT6w4ALWE9ry1zYNgS91+SuM+JyZhty1ClDDFPCl7zcc5/BX8fHQCuyvGcuwHcvYBxEQtA7i0T0hVMJ9L4u0f3YMuK1izPHXAnXBNpE5oduct17kBG6Je2hJ1tLREdg5PxnGMR4r69bxz3vXgEHzp/BXU4LAKK3ImFQoZeHSIi94CqoCGoYyqexvdeOgoAeP+W3qz9vZG7pmQ8d5ctY9/ubY0425rDAcTT/pOYYkkDx8dj+OD5y/H6sQl87sc7sGVFKzaegh365spkLAVdZafMqkFE6aFfTh2StKNvRWFoCKp5SyEBYCaZeTyWMqCpCjSFQc1KqFr/97ZmIveOhgCSadPXdz80MgPOgUvWdeCv33MGAOSN8okMk/EUmkI6XeUQ84bEvQ5Jpk1n6byGkIbR6YTzmDXT1L2/WKcTsFZX0lSGZa1hLGkOOT47kFmVSRZ3cXt4ynqPkekEfvBKH4CMJbO2s8GZ2Sr2I/IzGUuT304sCLJl6pCkIYl7UMMJKVpOGRyRgOaK5r22TEtEx+d/czNMDrx6dMx5TETundKkmuVtlkUzNJXA8rYIPvGdbXjt6Dgu29CBI6MzAIDVHVGnydjwNIl7MViRO/15EvOHIvcagnOOH73aV3Dlo2TahK5mxH1wQhZ3M2uBYFno42kTmqJAUxUENMW3Wka2CoT/PjxlvcfOfmt+20zCwOHRWXQ3BRHSVYQDKhqCWtVF7t/+9SH8xUM7Kj2MLCZjKYrciQVB4l5DHBqZwacf3I7Hdw/m3S9pmAhI4j4jnQxSholIwC3ucuRunRiyk6hAps5dxmvLiGTuZCyFI6MzWNkedfbtbAxiZDqZ/yAXmSd2D+LRnScqPYwsJuNpqpQhFgSJew0hJhsVWvkomTadFd6jnl7ghcQdgBP1A/517jIdDUEozLJlZKbiaRwZncXKtoi0b8CJ8KuFockETs4kq649ghW5ky1DzB/69dQQYkHqhE/Fi4w3oSpQmLWMXtgr7p6Thfy4HKzLkftjd16GV4+OQVUY2huCGJpMuCpxBifjGJpKYFWHO3J/68RUocNcVAYn40gZHJPxNJqryAYR1TIEMV8ocq8h4ilLPFM56soFckK1UYrcA5qCVLpw5B7UZHHP7i0DAKctacQHz18BwGpHMDydQP94zHl8R/8EAGCFFLl3NgTL5rn/xUM78OPX+ub0nHjKcNoonJypHrsokTYQT5nkuRMLgsS9hhC15N7mXynDxJVffhqP7rC842Q647nLtkxAVZA2TYR19wXbbNJwbBwArokzhWwZwIrIh6birk6RO45b4r5K8tw7GqyeNOXo/37v80fwBw9sn9NzhiYzJ5rRKqrimXI6QtKFNTF/SNxrCLEEnneR64lYCgeHZ/Dm8XHncblaRhDQVKR8bBnAbcXIy7p5+7n70dVoReTHTmYi9z0Dlv0i18R32rXuIyUWUtkvN8383vm3f30IX/zZHnDOMSj5/6NVFLlnOkJS5E7MHwoNaggR8XpnmYqWvmOzKedxEbE3StGfqgCJlIGQpkBhgKyDEV3FOKzny6WSsi2j5RT3EEamkzgwPI2ApkBXGKYSaQQ0BS2RjECJVZxOTMRdLQwWitzV8vDoDNZ0NuTc9ws/2QXASu72NGdOPH/937vwvZeO4j9+p/KLhlEvd6IUUOReZZycSeL3v/capuzGUTK5IndRpz5hi3tCSqjKtkzK4IinrTp30WJA4IrcA3OL3Fe2R2CYHM/sHcaq9oiTmOxqDLpq4kVy9fDorO/rzBe5fcIbfRN59z19SSMA4Aev9LlaIRw7GcPTbw1XRR0+9XInSgGJe5Xx+rEx/GR7vzMZSCaX5y7EfWw26Twuz1AVpOweMEE7upYraeT9QpL/rrLCnvuGbksw9w9NY3VH1LETRKQu6G0NQ1UYDo1YbQlSholv/uqQUwU0X2ak9gnyjFoA+MVbQ3jBXmgayHxW47MpDE7FoavMZUO9eGgUiwXnHF/82R5c/LdP4qdvZNazoY6QRCkgca8yRFTu1yNdVMt4I/eZhNuWSaZNBH0894RhibuI3OXHZCEOyaWQrpa//mNe15WxQVZ3NDhWkOgnI9BVBSvaIjg0YrUleP7AKP7qp7vwwsGT/i9cJHK1z6M7TsCQ/Kbf+fbLuPmeFzBk++ti35OzSQxNJtDVGHJ91vKJoJy8fmwcB0dm8K/PHMDITBJ//l87nIod6uVOlAIS9ypDWC9+LQZEhJvLlhkXkXuOOvdk2oTJrWoYXWUucZd7tIe0HLZMjsg9GtSwzH7+6o4IGkP+kbv1eBSHRixb5oTdFsHPgpoL4vg/fMEKDE0l8PwBS6Blkf/Hx/cBsKJ8hVmfxaGRGXQ1yX1ywnjuQPnF/eDwNN7ztV/jm786BAD442tPw9hsCv/12nEAFLkTpYHEvcrIJ+5OnbvHlpmKZ2yZvYNTmIqnfatlBCFdhaYornp3WYhdnnuOOncvG7qt6N0VuTcFs/Zb3RHF4ZEZcM4xYIu7vMbrfJi1Pfcbzl6KhqCGR3cOAICrYdobfeNIpA0kDdOpvd87OIXuxsxx33LRKhwcnnGuLMrFbruSSFwlXH5aJ8K6iuP2PAHq5U6UAvr1VBn5bBkncjf8bZl4ysQ7/vGXiKWMTEI1kC3uQbshmDxZqadZsmUkUVGUYsXd8t1XdUSciFMWTsGqjihiKQODkwmcmLTEbDqxMHEXLYvbGwJY0RbBiYk4Pv/wTtz1ozcBAGs6ougbiznevOhkOZs00N0UxD9/6FzcefV6XHfGEgDA47vK02tmZDqBG7/2a8dfPzhsnUS6m0LoaQ45VzLUy50oBZSOrzJE5O430Sfjubtruf3E8fiYJZyKwhANqK4WA0Fdha4y5wQAuKNsly3jsxKTHx++YCV6mkPoagzljdxFr5ljY7OZyH2B4i5ObtGghrZoAKMzSTyxe8h5/KK17bjvxaMYmLA+k+XSrNmuphDefdZS5/7mpU14bOcgbrts7YLG5MfP3hzA9mPj2H4ssy0aUNEY0rGkOeSMj3q5E6WAIvcqI5nPc89RLTPlsTVO627Ehy9c4dz/4PkrcNmGTud+SFeh2y19BXKPdndCtXCdOwCsaI/gty9ZDQB5PfcO+31GpxNOpLpQWyZL3KXOkwoDzl/dBgBOX5vlUo29N+l7xWldeP3YeFZLhlLw813Z3TzFZ7SkOYTj4zF8/N5teHTHCZqdSiwYEvcqI78tIyJ392NeIfrpp96OS9dnxPzP3r0J15+5xLkf0hTcefUGfMwWYyAze9R6fG517l429jRiaXPINTtV0NEQAACMTCcdT1yuU58PIvKP6CraogH0jWXq6Je2hJ0WCI64t2XG1eU5AZ23qhWGybG9b3zO43ijbxzn/uXPfZcSnIil8PyBUWcOgLgIEuLe0xzC4GQCT+weRNKgvjLEwiFxrzKEr+5fCmltm0kY+Pi927DTbs41nUi7ujfqPjWL8ragruL6s3pw0dp2Z5vcETFnQrVID/jy07rw3F1XIeLj97dGLXHvG4th3C7d9F55zJXZpIGQbi0w0hYNuGbermiLOCeZPba49zSHnZNWt8c62rK8FQDw6hF3vXwxbO+bwNhsCrsGrDkKw1MJ/Pa3X8LwVAL/9dpxpE2OP7t+IxgDLlxtffZL7FzHEs9JhipliIVC4l5l5LVl7Mf2nJjEE7sH8anvvQbAEnevOHiRLRh5kpJATt65E6qZffIlVItFV62WBOLEBMzdc+ec491ffRY/tNdqnU6knaqgNvvkAQDv39KLT1y6Bm3RAMK6ij0nLNFtDGlojVj7eZO+zREdG7ob8EoR4j48lcC/PnPA6WcjVrzqs/Mdzx8cxdNvDePlwyfx3ReO4OzeZty0dTme+PRv4GNvt66aRF5iSbP7KsdrvRHEXCFxXyTe7JvAn/zgjYKNrRJ5JzGJ3jLWa4iuhtOJNHrtJOE1m7p9X1eO3L3L7HlxNQ4rsBLTfGiPBpyukbrK5uxvT8RS2HF80pmNOpNIO20WZHH/5OVrcMXpXWCMobc1jEH782oIamiL6gio7t43gq2r2vDioZMY8rFXZB7e3o8v/mwP9g5ZVwTCjhG20GG7pPKJXYPYPzSND1+wEoC1YLiYF7BEsmUAOOM5MDQ9p8+EILyQuC8Sz+4fxgPbjjkTVHKRL3KPe7z2KVsUp+NptEcDePIPfwNf/eC5vq/ritwl8W6LBrC2M+raN5SjK2Sxtkwh2huCzmzajT1Nc7ZlRJXN8FQCH7znBTz0er9T8imLe4eUJF7ZnkmiRoNW5N7p6X0j+MSla5A2Of7yp7vyjkP0rxcljScm3ZG7qJd/dv8IAODcFS3Oczd0N+B3LlnlnIyFuP+PLb0AgPeeuyzvexNEISglv0ik7PLFmaSBljwNEcXluG9CNeV/qS5sibV5uiEGZM9dEvptn7sa3msJOaHKXJF77nHPBZFUjQZUrO1swMuH59Z+QFTZHD056/jowpZpt8VdV5krj7C2s8Epj4wGVFx+WlfOJmGrO6L4nYtX4Z5nD+LuWCrnCk2idFFE2YMecT9oi/vwVAKMuUswNVXBX/zmZud+e0MQ/3TzObhkXQf+5J2n561MIohiIHFfJMSs0tkCFoRIqPrWufs02BqfTWJasiVykcuW8bNaQgG3iqsKg2Hy0kXuUSuiXtvVYC3gPUdbRkTu+yTrImjnCUTCtj3qjsrXSv1vNFXBJy/PX8d+xeld+MYvD+LlQydxdQ6rq3/cGscD247hJ2/0Y++gNZ49A5O478Uj2DOQaf62pClU0A678RyK1onSQbbMIuGIexGLWwPFR+4HR2YwnUi7+rb7Ecix0pLvvp5qGyHqpUioAhm7ZG1nAxpCGqYT6TktUH3Cjpjl3jEiWm6NBMAY0NEYcD0n31WNH+csb0FAU/D4rkEcHPb3v0Xk3jcWc4S9MaghkTbxuR/vQCJtOiWPsi1EEIsBifsiIeyWQjXd+XvLZG/b1T8JzlFE5C5Xw+SPIL0+tLBjSiXu7bYts7YzioaghpTBneMuhhM+iU7hb6sKQ0tYd/nt4r3mQkhXsbGnCQ9sO4Yrv/xM1mefMkwM+dg6G5c2ue6fvsS6v7Jtbu9PEAuFxH2REBH5bCJ/5J5X3H0EUESshcRd9tm9kXkhSh+5C3FvcLzyuVgzwpYRtEUDuOudpzv3L17X4cxKFbRE3JF8MXz4/Mws3/2e6pXByTg4z65Pv+WiVfjMOzbgxT+9Cp95xwbc/LblAICVHRS5E4sLee6LhGPL5Fkc2jR5TluGc57V6heAkxSM+qyLKiM894CmzLmkUeyfr7fMXDhvZRuu2dSNC9e046k9VpJzOpFGe0N2Lxo/TkzE0RjUMJVIo6MhiG1/drXr8a99aEvO5/p1yczFB962HFtXteLKLz+DXQOTOGNZs/OYOMH85Y2bEQ6o2Ds4jb/66S5s7GnE9Wf1AAB+78r1eGbvMACK3InFp+AvnTH2LQDvBjDEOT/D3tYG4AEAqwAcBvABzvmY/dhdAG4FYAD4FOf8sbKMvMYQtem5EqoPvX4cd9z/ulPK5xV3EdELUQvrKtKm6Sw27TcbVEZ47n4TmAohIvZSRe6djUH8229tBZC54sg3kWnv4BTGZpI4d0Ur9g5O4fh4DGf3tuD5g6NY2pJ/8pbMji9cO+exrmyPIqyr2OVZGUuUQa7uiGJ9dyPevq4D15/Z48w4FVy4pg2fumo9rji9EwSxmBTzl/4fAK7zbPssgCc55+sBPGnfB2NsE4CbAWy2n/N1xlj+kPIUIZknoco5xx33vw4Azmo88ZTpmvAkkqkicRoJqIgEtEzkHiwuci/kt/tRaltGRhxPruZhQ1NxvPOfnsX/vOcF/M0ju/H+f3kODMD7z7PqwXuaixf3hqA2p8gdsI759J5G7B5wi/tDr/ejJaI75Y2MsSxhB4CgpuLT12woePIliFJTUNw5578E4C1EvhHAvfbtewG8R9p+P+c8wTk/BGA/gMovJ28zOp3IWuhisUgJz90nofp8jqXd5CSjKIMUHRcjQRXRgCpF7sWJe3AeC0CU2paRKRS5v3pkDIbJ0RYN4D+eO4xE2sQPf/divO/cZQhoCnpby+9lb+xpwq6BSaei542+cTy1ZwifuHTNvE6WBLEYzDeh2s05HwAA+/8ue/syAFK3avTZ2yqOYXJc+eVn8L2Xjlbk/VNOtUx25O7XRRBwWzOiWsOJ3HUNkaCGUTvSLxQZBh1bJrcYPXrnpXjkU5dmbS9n5C5yBblKRLcdHkNAU/AHV68HYAnt6UuaoCgM37rlbbjtsjUlH5OXTfYsWrFS0ov2mq8iWUoQ1Uipq2X8/vp9C5gZY7cxxrYxxrYNDw+XeBjZJNIGJmIpZ3bjYiM8d78qmFjSfTUhervIUb6I4kUr2HDAitxFrbffiksyxdgypy9pwiZPKR8gee5ljNz9rmgA4JWjYzi7txk3nLMMrREdH7ogU8Hy9vUdvj3jS83GHuszEb67KGedTwUOQSwW8xX3QcZYDwDY/4tlb/oAyOFML4B+vxfgnN/DOd/KOd/a2Vn+ZJNYxchvctBi4NS5+9gP3jGJ6e7xPJF7NKi6ovVIAc9dVRhUZX7rcoo691K1H5ARJ6UZnxLReMrAjuMT2LKyFc1hHS997mp8RBL3xeL0JY1gLLP2qWgxXI4rGYIoFfP9c30YwC327VsAPCRtv5kxFmSMrQawHsBLCxtiacg3rX8xcOrc80xOEmIhOgPKEb1TLWOLe1jXXEnUQpE7YE1kCuaxZXIhInatDOouesf7nfSGpxJIGRxrO6zZpbqqVGRd0WhQw6r2KP7xib145z89i8lYas6JWYJYbAr+tTLGvgfgeQCnMcb6GGO3AvgigGsYY/sAXGPfB+d8J4AHAewC8CiA2znnlVFTDyJyj+dovlVuMu0HskUsnjLAGNBqi7qI3OXZrJnI3XpMjtwZK9xSALAmL80vcrcTqmWI3AOagoCq+OYiRAfNpnDlhVSUqO4emET/RJyqX4iqp+AvlHP+wRwPXZVj/7sB3L2QQZUDZ4WjAr1dykW+hGo8ZSCsq05ULZa8m4hl2gP7lUIKogGtqIg2oCkILqQUskxRcySo+p70RCvgxipYleidZyxxFvAYnIgXrE4iiEpzyoQflfLcd/VP4tEdA3n7tMdSBkK66pQpiiTh+GxmoWdRCimWX4sENIheW8UKTWNIz9m+Nh+lnsTkJRrQfD33jLhX/mf6sUtWozUSwB9+fztOTMaxvmtujcgIYrGp/F/NIiFsjcUW95/tGMBXn9rvNLLyaxwWT5muyF30KxELWoh9AP/IvVhx/8ZHz/NdeagQor69VCsxeYkEsiN3zjkm7SuXalhPVFEYltqrJ03EUoiQ505UOafML1QkJBOLLO4iIp2y/ePZhIG0YaJvLIZVHVa/kVjKQFBXnFr05oi1BNy4JO4Jn8hdUKz/u6G7cV7HUM5SSACIBDWXXXVyJonL/v4XzgLe1RC5A+5xFOrlQxCV5pTpClmpyF1EpAlphur/fWo/Lv/S0zgyarWpTdieu0h2BjVrbU+XLeMTuYtqmUKtBxaKUnZbRnX13NkzMInpRBov2jN3q8FzB9xXEIW6cBJEpTllxD3fwtPlxJtAnU0a2H5sHACcBR5inoRqUFPQGglgTBJ3EbmLqo2msOZE7OWu3BCaXo72A4A1fvlzEsvTTcbTCOmKa6GRSkKRO1FLVMdfzSIgIvfFLoX0doFMm9zxvcWiE/GUaSVUbRHj3LJm/Dz31R1R/NtvbcU7z+hxBKbckXumzr084t7gqZYRC28A1RO1A0BDSJ40RpE7Ud2cMr9QEbnHF7kUUk6gNoY0TMXTTgR8xBaxWNJAayTglCkm0iZaI7pL5BIpA0HNmsRzjb2mpxCYskfuSpkTqkGrWubHr/Xhbx7Z46ooqha/HbAmUYV0BfGUSZOYiKrnlIncExXz3DPv125bKqK/jRDveMqazi4i90TasG0ZOaFqulZTAjLWQLktApWxsk61F90t/+CB7RieSrg6RFZT5A5IXTnJliGqnJoX95HpBH7jH36BvYNTefcTkXva5Atq+/v3j+7BH31/e9H7y9PqxUpDYmFlWdzDuorrz7RW8DmrtwUtkQDGZ5NOm9m4XQsv43juZY4iVYWVrVIGcF95nNXb7HqsqYoid0Dq7UMzVIkqp+bFff/QNI6MzjpJylz4NeHKx8HhaUeEZZ7dN4L/fnPA6cZYCDlyF8lQsUTb0ZOzSBumM4npitO7cOBv3oWNPU1ojehIGdx5fiJtZvVid6plyhxFKgorS+sBgZwz+N3L1wLIdMashhp3mUz7BxJ3orqpeXEXU/TFCka5kBe+KGTNJNMmrvzyM7jhn3+d9djARAyzSSNrweRcyOIuFoaWryL2nJiyJjHZAi3sj1a7nayomImnjKxe7EJgyi00KitfjTvgjtyvO6MHD91+CT59zQYA1eW5A5kriUJdOAmi0tS+uM8WJ+6uyN3TP/1fnzmAX+0bce4/tvMEgMzi04JE2sDItPU+ha4UBHIViIjcAeDcFS1gDPj5rkErcvf46aKiRkxk8rNlOhqC+Pv3n4XfPHtpUWOZL6rCypZMBTKRu/h8zl7egm57ybpqE3eyZYhaoebFfTxmia0Q3Vy4xD3tjty/9ov9+OGrfc79+1/OrNYknzTkhT629427XuPg8LSz0r0gmTadRToAoC0adG6vbItg68pW/PQNq919yGOttNtR/tCU9Z5+CVUA+MDbljutDcqFUuaEqojcxZWNfLvqbJlgpisnQVQztS/uTuSeyLufy5axrZJ/++VBvHz4JKbiaZeIHxqeQXeTJZh7TmQWRu4ft4Q2pCvY0e9eMPlrvziA3//PV13bvP1S2qXIPRrUcM2mbhwctpKqYU9UvsbuYX5gSK6oqYyglDuhqqvWa4tumADQaZ+wKHIniPlR++JepOcuR+6xlAHOOf7hsbfwrV8dAgDXbNCx2RQuWmP1NXnrRKYKRyRYN/U0YcRj2fSPxzAZTzs9yIHs2akhXXVK6BqCGs5Y1ux6TKY1GkBHQ9CpAsoVuS8GilLeyF10zFzZHnW2rWyP4jfPXopL1nWU7X3ng7xAOUFUMzUv7iKhWsiW8SZU4ykTScN0yhHFySGeMhBLGVjf3Yj2aADfef4IfvbmAABLwAFrTU251zqQmW0q9vnuC0dwyRefcu0T0Ji0TJ7mEjNv5A4AG7obsNdO3FY0ci+zLXPVxm7cefV6/Om7NjrbApqCr37wXKyfZ7OzcnHGsias6YyiJUzrpxLVTe2L+xwSqkJYEynDEefDdvOuMfv5wuZpiei4+fzlODQyg2//+jAAoH8ijrZoAEuaQphOpJ2Ik3PuRPXHx6z/Xzs6ljUGXVVcpXQ90uLOfiskbehuxP7BKXDOEU9ll0IuFprCytZXBrA+lzuv3lATsz6v2tiNp/7w8qrpd0MQuaj+v6YCiIRqLGVgNpl2ldWdnEliy189jq99aIs9pT+AqXgaMUncRc+WmaSBB14+6iRAWyMBfPiClTgwNIMDw1b0fGIijiVNIaeSZSKWQmdjEJOxtPM6InIXIi+jq4pTStcQVF0VKH5R+bquBswkDfRPxG1bpjKR+4cuWIG3r68ue4QgiPzUvrjPpqAwwOTA6HQSkbbMIT1i2ynff+UY4ikTLREdR09aC097bRUAuOtHbzrRoxDwloju+Poj0wl0NgbRbNegT8SS6GwMYmAyI+R9trj3SxOgxPjkyL0hmJnGPpv0t1xE//V9g1NI2C0KKsHWVW3YuqqtIu9NEMT8qPlry4nZFHpbIwCyrZmfbLfKDHuaw0ikDWeJubgUucuY3GozC2QmEcltAEank2hvCKDFfh3R+2VAKpH8xjMHceM//woD45lton47oCqS526JeZddIeIn7ms6LU/+0MhMRSN3giBqj5oW97RhYiqRdkRwZNqqYImnDHzo317Ai4dOOtsTKRMttmDHUoazhFsuhLiLNgAzSQOjMwm0RwPOY8KfF/XvwnLZ3jeBtCnXt9virsmRu7WvODH5tURojwbQGNRwYHgaScOsWOROEETtUdNqIaLs81a0IqAq+NV+a5bp47sG8dyBUdxx1XpcsLrNEve0lVAN6QqGpxK+kbuMsGWEkPePxxBPmWiLBqXZo9aVwsBEHIwBS5pDvq/Vbk9e0lXmnABEy4Av3XQ2PnLhCpy3sjXreYwxrO6MYs+AVQ5ZqWoZgiBqj5oWdyGuy9siuPL0Lvxkez/ShomHXj+OJU0h3HHVevQ0hzAynbAWxNBUnLeyFS8dOukSd2/lQ0hXHCEVQn7ALklsbwigWUqoApZt0t0Ywtc/vAX//KFzsxp5tdmzLXVVQVPYHbkvaQ7hr99zJnTV/6tY1R7FHrvWvlJ17gRB1B41rRYi0dkc0fGec5dhZDqJR3eewNNvDeOGc5ZCURg6GoIYmUo6PdMvXN2O3ScmcezkrPM6y1vDrtcV0ToAx8oRFTPCKlEVhrHZJGaTaTy1exCXn9aJdV2NePdZS3HuCisK/+6t5+N/XbbGidYDmoJV7VE0BDVXn5l8rO6IOv3NKXInCKJYarpaZlNPE5749G9gSXMIKmPQFIav/eIA0ibHFad1AbB6qIsukEFNxYVr28EfB556awiqwmCYHO0NQYxMJ9HTHMKeE1OOoAOW5w4AB+w2AW3RABhjaAnrGJ9N4dEdJzCTNHDjOcuc59xw9lIYJsel6ztx6fpOfOEnOwFYkfu7zlyCy0/rLLqTo8gnWOOv6XMxQRCLSE2Le0hXsa6rwbm/aWkT3uibgKownLO8BYC7GVVPSwhn9TYjpCsYn01hVXsEh0dn0RzW8dsXr8KG7kZ86v7XHEEHsiN30aSrOaLjvheP4r4Xj2JlewQXrM6UCn7gbcvxgbctd+6L2ae6ysAYm1OL3s1LMy0K/GaxEgRB+FFXoeAW2w7ZvLTJ6Y/eITWjumhNO4K27w5YJZIBTUFzWMcfXLMB15/Vg97WsKuBlddzF3ZKwp60dPXGLnz/f12UtyVuc1iHprB5zWpc19WAH/3uxfjUVetpIhFBEEVT05G7l62rWvEfzx12VZ50Su1we21v/cLV7fj1/lE0h3V89MKVuNBuEgYA//Lh89AUznwsuqqgMahhKpFGUFOcxl+il8z/efdmdDX5V8kIbj5/BbasbJ13nfqWFa3OiYsgCKIY6krcL1jdjvZoANds7Ha2yb3Omd0f5cK17cDjQFNYw5+/e5PrNTYtbcp63eaIjqlEGh0NQec1vvux89E/EceK9kjBcTWHdbyNZngSBLGI1JW4dzYG8cqfX5O17R2buvE7l6x2tp3V24ymkIalLWHvS/jSGgmgbyzmqnC5uMpa0RIEQcjUlbj7oSoM9/zWVte2oKbisT+4rOi2rR+/dDUe23kC125eUo4hEgRBlJy6F/dc9DQXF7UDwI3nLHOVOhIEQVQ7ZauWYYxdxxh7izG2nzH22XK9D0EQBJFNWcSdMaYC+BqAdwLYBOCDjLFN+Z9FEARBlIpyRe7nA9jPOT/IOU8CuB/AjWV6L4IgCMJDucR9GYBj0v0+e5sDY+w2xtg2xti24eHhMg2DIAji1KRc4u43XZO77nB+D+d8K+d8a2dnZ5mGQRAEcWpSLnHvA7Bcut8LoL9M70UQBEF4KJe4vwxgPWNsNWMsAOBmAA+X6b0IgiAID2Wpc+ecpxljvwfgMQAqgG9xzneW470IgiCIbBjnvPBe5R4EY8MAjizgJToAjJRoOJWkXo4DoGOpVuhYqpP5HstKzrlv0rIqxH2hMMa2cc63Ft6zuqmX4wDoWKoVOpbqpBzHUlf93AmCIAgLEneCIIg6pF7E/Z5KD6BE1MtxAHQs1QodS3VS8mOpC8+dIAiCcFMvkTtBEAQhQeJOEARRh9S0uNd6z3jG2GHG2JuMsdcZY9vsbW2MsccZY/vs/6tyZWzG2LcYY0OMsR3StpxjZ4zdZX9PbzHGrq3MqP3JcSyfZ4wdt7+b1xlj75Ieq8pjYYwtZ4z9gjG2mzG2kzF2h7295r6XPMdSi99LiDH2EmNsu30sX7C3l/d74ZzX5D9YM18PAFgDIABgO4BNlR7XHI/hMIAOz7a/B/BZ+/ZnAfxdpceZY+yXAdgCYEehscPq6b8dQBDAavt7Uyt9DAWO5fMAPuOzb9UeC4AeAFvs240A9trjrbnvJc+x1OL3wgA02Ld1AC8CuLDc30stR+712jP+RgD32rfvBfCeyg0lN5zzXwI46dmca+w3Arifc57gnB8CsB/W91cV5DiWXFTtsXDOBzjnr9q3pwDshtVqu+a+lzzHkotqPhbOOZ+27+r2P44yfy+1LO4Fe8bXABzAzxljrzDGbrO3dXPOBwDrBw6gq2Kjmzu5xl6r39XvMcbesG0bcclcE8fCGFsF4FxYUWJNfy+eYwFq8HthjKmMsdcBDAF4nHNe9u+llsW9YM/4GuASzvkWWMsR3s4Yu6zSAyoTtfhd/QuAtQDOATAA4Mv29qo/FsZYA4AfAriTcz6Zb1efbdV+LDX5vXDODc75ObDan5/PGDsjz+4lOZZaFvea7xnPOe+3/x8C8GNYl16DjLEeALD/H6rcCOdMrrHX3HfFOR+0/yBNAP+GzGVxVR8LY0yHJYb3cc5/ZG+uye/F71hq9XsRcM7HATwN4DqU+XupZXGv6Z7xjLEoY6xR3AbwDgA7YB3DLfZutwB4qDIjnBe5xv4wgJsZY0HG2GoA6wG8VIHxFY34o7N5L6zvBqjiY2GMMQDfBLCbc/4V6aGa+15yHUuNfi+djLEW+3YYwNUA9qDc30ulM8kLzEK/C1YW/QCAz1V6PHMc+xpYGfHtAHaK8QNoB/AkgH32/22VHmuO8X8P1mVxClakcWu+sQP4nP09vQXgnZUefxHH8l0AbwJ4w/5j66n2YwHwdliX728AeN3+965a/F7yHEstfi9nAXjNHvMOAP/H3l7W74XaDxAEQdQhtWzLEARBEDkgcScIgqhDSNwJgiDqEBJ3giCIOoTEnSAIog4hcScIgqhDSNwJgiDqkP8fSGijqAmoc1sAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "alpha = 1e-4\n",
    "\n",
    "history = []\n",
    "for epoch in range(300):\n",
    "    states, actions, probs, rewards = run_episode()\n",
    "    one_hot_actions = np.eye(2)[actions.T][0]\n",
    "    gradients = one_hot_actions-probs\n",
    "    dr = discounted_rewards(rewards)\n",
    "    gradients *= dr\n",
    "    target = alpha*np.vstack([gradients])+probs\n",
    "    model.train_on_batch(states,target)\n",
    "    history.append(np.sum(rewards))\n",
    "    if epoch%100==0:\n",
    "        print(f\"{epoch} -> {np.sum(rewards)}\")\n",
    "\n",
    "plt.plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们运行带有渲染的剧集来查看结果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "error",
     "evalue": "display Surface quit",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31merror\u001b[0m                                     Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_44248/1459719159.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_episode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/tmp/ipykernel_44248/3855001447.py\u001b[0m in \u001b[0;36mrun_episode\u001b[0;34m(max_steps_per_episode, render)\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmax_steps_per_episode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m             \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m         \u001b[0maction_probs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexpand_dims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0maction\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_actions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction_probs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    429\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    430\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 431\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    432\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    433\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/wrappers/order_enforcing.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     49\u001b[0m                 \u001b[0;34m\"set `disable_render_order_enforcing=True` on the OrderEnforcer wrapper.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m             )\n\u001b[0;32m---> 51\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    429\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    430\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 431\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    432\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    433\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/wrappers/env_checker.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0menv_render_passive_checker\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    216\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 217\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    218\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"human\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36m_render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    296\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 298\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    299\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"human\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    300\u001b[0m             \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31merror\u001b[0m: display Surface quit"
     ]
    }
   ],
   "source": [
    "_ = run_episode(render=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "希望你能看到，现在杆子已经能够很好地保持平衡了！\n",
    "\n",
    "## Actor-Critic 模型\n",
    "\n",
    "Actor-Critic 模型是策略梯度的进一步发展，在这种模型中，我们构建一个神经网络来同时学习策略和奖励的估计。这个网络将有两个输出（或者你也可以将其视为两个独立的网络）：\n",
    "* **Actor** 将通过给出状态的概率分布来推荐采取的动作，就像在策略梯度模型中一样。\n",
    "* **Critic** 将估计这些动作可能带来的奖励。它会返回在给定状态下未来的总估计奖励。\n",
    "\n",
    "让我们定义这样一个模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_inputs = 4\n",
    "num_actions = 2\n",
    "num_hidden = 128\n",
    "\n",
    "inputs = keras.layers.Input(shape=(num_inputs,))\n",
    "common = keras.layers.Dense(num_hidden, activation=\"relu\")(inputs)\n",
    "action = keras.layers.Dense(num_actions, activation=\"softmax\")(common)\n",
    "critic = keras.layers.Dense(1)(common)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=[action, critic])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要稍微修改我们的 `run_episode` 函数以同时返回评论结果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_episode(max_steps_per_episode = 10000,render=False):    \n",
    "    states, actions, probs, rewards, critic = [],[],[],[],[]\n",
    "    state = env.reset()\n",
    "    for _ in range(max_steps_per_episode):\n",
    "        if render:\n",
    "            env.render()\n",
    "        action_probs, est_rew = model(np.expand_dims(state,0))\n",
    "        action = np.random.choice(num_actions, p=np.squeeze(action_probs[0]))\n",
    "        nstate, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "        states.append(state)\n",
    "        actions.append(action)\n",
    "        probs.append(tf.math.log(action_probs[0,action]))\n",
    "        rewards.append(reward)\n",
    "        critic.append(est_rew[0,0])\n",
    "        state = nstate\n",
    "    return states, actions, probs, rewards, critic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们将运行主要训练循环。我们将通过计算适当的损失函数并更新网络参数来使用手动网络训练过程：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running reward: 5.82 at episode 10\n",
      "running reward: 9.43 at episode 20\n",
      "running reward: 10.30 at episode 30\n",
      "running reward: 10.28 at episode 40\n",
      "running reward: 11.00 at episode 50\n",
      "running reward: 13.01 at episode 60\n",
      "running reward: 21.78 at episode 70\n",
      "running reward: 40.54 at episode 80\n",
      "running reward: 73.70 at episode 90\n",
      "running reward: 100.19 at episode 100\n",
      "running reward: 159.20 at episode 110\n",
      "Solved at episode 114!\n"
     ]
    }
   ],
   "source": [
    "optimizer = keras.optimizers.Adam(learning_rate=0.01)\n",
    "huber_loss = keras.losses.Huber()\n",
    "episode_count = 0\n",
    "running_reward = 0\n",
    "\n",
    "while True:  # Run until solved\n",
    "    state = env.reset()\n",
    "    episode_reward = 0\n",
    "    with tf.GradientTape() as tape:\n",
    "        _,_,action_probs, rewards, critic_values = run_episode()\n",
    "        episode_reward = np.sum(rewards)\n",
    "        \n",
    "        # Update running reward to check condition for solving\n",
    "        running_reward = 0.05 * episode_reward + (1 - 0.05) * running_reward\n",
    "\n",
    "        # Calculate discounted rewards that will be labels for our critic\n",
    "        dr = discounted_rewards(rewards)\n",
    "\n",
    "        # Calculating loss values to update our network\n",
    "        actor_losses = []\n",
    "        critic_losses = []\n",
    "        for log_prob, value, rew in zip(action_probs, critic_values, dr):\n",
    "            # When we took the action with probability `log_prob`, we received discounted reward of `rew`,\n",
    "            # while critic predicted it to be `value` \n",
    "            # First we calculate actor loss, to make actor predict actions that lead to higher rewards\n",
    "            diff = rew - value\n",
    "            actor_losses.append(-log_prob * diff)\n",
    "\n",
    "            # The critic loss is to minimize the difference between predicted reward `value` and actual\n",
    "            # discounted reward `rew`\n",
    "            critic_losses.append(\n",
    "                huber_loss(tf.expand_dims(value, 0), tf.expand_dims(rew, 0))\n",
    "            )\n",
    "\n",
    "        # Backpropagation\n",
    "        loss_value = sum(actor_losses) + sum(critic_losses)\n",
    "        grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "    # Log details\n",
    "    episode_count += 1\n",
    "    if episode_count % 10 == 0:\n",
    "        template = \"running reward: {:.2f} at episode {}\"\n",
    "        print(template.format(running_reward, episode_count))\n",
    "\n",
    "    if running_reward > 195:  # Condition to consider the task solved\n",
    "        print(\"Solved at episode {}!\".format(episode_count))\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们运行这一集，看看我们的模型有多好：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = run_episode(render=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 要点\n",
    "\n",
    "在这个演示中，我们看到了两种强化学习算法：简单的策略梯度和更复杂的演员-评论者方法。可以看到，这些算法基于状态、动作和奖励的抽象概念进行操作，因此它们可以应用于非常不同的环境。\n",
    "\n",
    "强化学习使我们能够仅通过观察最终奖励来学习解决问题的最佳策略。不需要标注数据集这一事实使我们可以多次重复模拟以优化模型。然而，强化学习仍然面临许多挑战，如果你决定深入研究这个有趣的人工智能领域，你将会了解更多。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n---\n\n**免责声明**：  \n本文档使用AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。原始语言的文档应被视为权威来源。对于关键信息，建议使用专业人工翻译。我们不对因使用此翻译而产生的任何误解或误读承担责任。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  },
  "coopTranslator": {
   "original_hash": "4acab0ddd5fd306aaecd56a7b1e0d069",
   "translation_date": "2025-08-31T09:28:01+00:00",
   "source_file": "lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb",
   "language_code": "zh"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}