{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "i_f5u2x9nn6I", "slideshow": { "slide_type": "slide" } }, "source": [ "# **Lecture 12: Support Vector Machines**\n", "In this lecture, we are going to cover support vector machines (SVMs), one the most successful classification algorithms in machine learning." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 12.1. Classification Margins\n", "\n", "We start the presentation of SVMs by defining the classification *margin*." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.1.1. Review and Motivation \n", "\n", "### 12.1.1.1. Review of Binary Classification\n", "\n", "Consider a training dataset $\\mathcal{D} = \\{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \\ldots, (x^{(n)}, y^{(n)})\\}$.\n", "Recall that we distinguish between two types of supervised learning problems depending on the targets $y^{(i)}$. \n", "\n", "1. __Regression__: The target variable $y \\in \\mathcal{Y}$ is continuous: $\\mathcal{Y} \\subseteq \\mathbb{R}$.\n", "\n", "2. __Binary Classification__: The target variable $y$ is discrete and takes on one of $K=2$ possible values.\n", "\n", "In this lecture, we focus on binary classification and assume $\\mathcal{Y} = \\{-1, +1\\}$.\n", "\n", "#### Linear Model Family\n", "\n", "In this lecture, we will work with linear models of the form:\n", "\n", "$$\n", "\\begin{align*}\n", "f_\\theta(x) & = \\theta_0 + \\theta_1 \\cdot x_1 + \\theta_2 \\cdot x_2 + ... + \\theta_d \\cdot x_d\n", "\\end{align*}\n", "$$\n", "\n", "where $x \\in \\mathbb{R}^d$ is a vector of features and $y \\in \\{-1, 1\\}$ is the target. The $\\theta_j$ are the *parameters* of the model.\n", "We can represent the model in a vectorized form as\n", "\n", "$$\n", "\\begin{align*}\n", "f_\\theta(x) = \\theta^\\top x + \\theta_0.\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "#### 12.1.1.2. Binary Classification Problem and The Iris Dataset\n", "\n", "In this lecture, we will again use the Iris flower dataset. We will transform this problem into a binary classification task by merging the two non-Setosa flowers into one class.\n", "We use $\\mathcal{Y} =\\{-1,1\\}$ as the label space.\n", "\n", "The resulting dataset is partly shown below." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)target
05.13.51.40.2-1
45.03.61.40.2-1
84.42.91.40.2-1
124.83.01.40.1-1
165.43.91.30.4-1
\n", "
" ], "text/plain": [ " sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) \\\n", "0 5.1 3.5 1.4 0.2 \n", "4 5.0 3.6 1.4 0.2 \n", "8 4.4 2.9 1.4 0.2 \n", "12 4.8 3.0 1.4 0.1 \n", "16 5.4 3.9 1.3 0.4 \n", "\n", " target \n", "0 -1 \n", "4 -1 \n", "8 -1 \n", "12 -1 \n", "16 -1 " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn import datasets\n", "\n", "# Load the Iris dataset\n", "iris = datasets.load_iris(as_frame=True)\n", "iris_X, iris_y = iris.data, iris.target\n", "\n", "# subsample to a third of the data points\n", "iris_X = iris_X.loc[::4]\n", "iris_y = iris_y.loc[::4]\n", "\n", "# create a binary classification dataset with labels +/- 1\n", "iris_y2 = iris_y.copy()\n", "iris_y2[iris_y2==2] = 1\n", "iris_y2[iris_y2==0] = -1\n", "\n", "# print part of the dataset\n", "pd.concat([iris_X, iris_y2], axis=1).head()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "As in earlier lectures, we visualize this dataset using `matplotlib`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtIAAAEGCAYAAABILXIOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA62klEQVR4nO3deXycZbn/8c81SzLZu6V0pQuF0pZuUNaWRZCygwoobmx6OAKyycEfRzkc5RzPUUFQQFmOIKgoYEUEVATZZClgC22hlLZQWro3Tds02ySzXL8/Mq1dkmk6ycxk+b5fr3l18txPnvk2Tye98uR+rtvcHRERERER2TuBfAcQEREREemOVEiLiIiIiGRAhbSIiIiISAZUSIuIiIiIZECFtIiIiIhIBkL5DrC3BgwY4CNHjsx3DBERERHp4ebOnbvR3SvbGu92hfTIkSOZM2dOvmOIiIiISA9nZivSjWtqh4iIiIhIBlRIi4iIiIhkQIW0iIiIiEgGVEiLSLvFEknqm+Mkkp7vKCIiInnX7W42FJHca2iOM3fVFtZsjWJm4DCiXzEHD60gHNTP4yIi0jupkBaRtBpjCZ5evIGmeLJlg7dcjV6+qZ7q+iZOGrsPwYDlMaGIiEh+6FKSiKS1aH0tsW1F9A6SDnVNCZZvbshDKhERkfxTIS0iaS3f1MDuZXSLhDsfbKzLaR4REZGuQoW0iKQV9/Q3FsYSuvFQRER6JxXSIpJWn0jbt1IYMKCkIHdhREREuhAV0iKS1oRB5W3eTBgwY9zAshwnEhER6RpUSItIWkMrihg/sIyAtVyBBggYBA0O27cPFUXhvOYTERHJF7W/E5E9OmhwOSP7FfNhdT31zXEqImFG9y+hKBzMdzQREZG8USEtIu1SWhhi8pCKfMcQERHpMrI+tcPMgmb2tpk91cpYoZk9YmYfmNkbZjYy23lERERERDpDLuZIXwUsamPsK8Bmdx8D3Ab8IAd5REREREQ6LKuFtJkNA04Dft7GLmcBD6aezwJOMDOtNSwiIiIiXV62r0j/GPgmtLkw2lBgJYC7x4EaoH+WM4mIiIiIdFjWCmkzOx3Y4O5zO+FYl5jZHDObU1VV1QnpREREREQ6JptXpKcDZ5rZcuBh4Hgz+/Uu+6wGhgOYWQioAKp3PZC73+vu09x9WmVlZRYji4iIiIi0T9YKaXf/d3cf5u4jgfOA5939S7vs9gRwQer5Oal9PFuZREREREQ6S877SJvZTcAcd38CuA/4lZl9AGyipeAWEREREenyclJIu/uLwIup5zfusD0KnJuLDCIiiaSzsb6JpEO/4jCFIa3MKCIimdPKhiLSKyzeUMuCtVvZ1l8z4c6IvsUcOrwvwYC6boqIyN5TIS0iPd7Sqjrmr9lKYpdbMD7e3EBzIskxowfkKZmIiHRnuVjZUEQkb5LuLFhbs1sRDZBwWLc1Sk00lodkIiLS3amQFpEebXNDjGSaXkDusKYmmrtAIiLSY6iQFpEezXHSzYB2Wq5ai4iI7C0V0iLSo/UpKiBdmRwIGPuUFeYsj4iI9BwqpEWkRwsFjLGVpQRt9+vSAaAiEqJ/cUHug4mISLenrh0i0uNNHFxOcyLJh9X1mBnujmH0LQ5zzOgBWCtFtoiIyJ6okBaRHs/MmDa8LxMGlbOmJkrCncqSAvrqSrSIiHSACmmRHqC+Oc7762v5eEsjDgwqK2T8PuX0KQrnO1qXUhQOst+AknzHEBGRHkKFtEg3t6Uxxt+WbCCe9O031X28uZFVW6LMGNWPIRVFec0nIiLSU+lmQ5Fu7rXl1cR2KKKhpaVbwp1Xl28ika6JsoiIiGRMhbRIN1bTGKOuKZF2nzVbG3OURkREpHdRIS3SjdXHEgTSNJxIulPfnL7QFhERkcyokBbpxkoLgmmXvw6YUVKgWyFERESyQYW0SDdWHglTVth2oWwGQ8ojOUwkIiLSe6iQFunmpo/qRzhoO03xMCAYMGaM6k8w3dwPERERyZh+5yvSzZVHwpw2bhBLqupYsbkBdxhcHuHAgaWUR9RHWkREJFtUSIv0AEXhIJOHVDB5SEW+o2TM3dlQ10R1Q4xwwBjWp4iicDDfsURERNqkQlpE8q6uKc4LH24kGkuQSDoBM95avYUDKkuZMqQCM01PERGRrkdzpEUkr5LuPLe0ivqm+PbVGRPuJB2WbqxnSVVdviOKiIi0SoW0iOTV6ppGmhNJWuvil0g6C9fVknStzigiIl2PCmkRyau1W5uIp2mGHU86DVpURkREuiAV0iKSV8E9TH92nIBa+ImISBekQlpE8mp432JCaQrlkoIQxereISIiXZAKaRHJq8qSAvoWhWmtlg6acfDQ7tvST0REejYV0iKSV2bGcfsNYHifIgIG4YARChiRUIAjR/RlSEVRviOKiIi0Sn2kRXLE3YknnWDACKgv8k5CwQBHjexPUzxBTWOcUNDoWxRW/2gREenSVEiLZJm7s7iqjkXra2mKJ8FgaHmEKUP7UFaot+COCkNBBpZpPrSIiHQPmtohkmWvr9jEgrVbicZbeiW7w6qaKH99fz1bo7F8xxMREZEMqZAWyaLqhmZWbomSaKVPcizpvL26Jg+pREREpDOokBbJomXV9WlX5VtXGyWeTOYwkYiIiHQWFdIiWdQUT7S69PWO4gktfy0iItIdqZAWyaJ+xQVpV+4LBoyCkN6GIiIi3ZH+BxfJov36l0AbLdyCBgdUlqoVnoiISDelQlokiwpDQY4e1T/VO/qf20MBY2BZIQcNKs9fOBEREekQNbEVybLB5RHOGD+IDzbWU13fREEowH79SxhYWqgFR0RERLoxFdIiORAwCAZa5kQHA4FOn86RdGdNTZSPtzSQdGdIeRH79i0mFFChnktbozE+3FhPbXOc8sIQYwaUUqpFd0REeix9hxfJsvW1Uf6+rBp3SKRa4a3Y3MDgskKmj+rf4aK6OZ7kb0s3UN+cIJ7qV712axPz19TwyQMGavXEHFm4bisL120l6eDAGmBJVR2Th1QwdmBZvuOJiEgWZG2OtJlFzOxNM5tvZgvN7Lut7HOhmVWZ2bzU46vZyiOSD03xJH9fVk086duLaIBE0lm7tYn31tV2+DVmr9hEbVN8exENEE860XiSFz+owtP0sZbOsXZrlIXra0mkimho+TPhMH/NVjbWN+UznoiIZEk2bzZsAo5398nAFOBkMzuilf0ecfcpqcfPs5hHJOeWVdfTVh2bcGdxVW3aBVv2pKE5wbraKK0snAhANJ5kQ52KuGx7b/3WVlevhJbz/N76jv/AJCIiXU/WCmlvUZf6MJx66NKY9CrVDU07XYneVTzpNMczX9mwJhojmGZqSNKdzY2xjI8v7bOlMZ52fHODzoGISE+U1fZ3ZhY0s3nABuBZd3+jld3ONrMFZjbLzIa3cZxLzGyOmc2pqqrKZmSRTlUQDO5xn47cEBgOWtqfTgNmFATV5TLbwulW3WnHuIiIdE9Z/R/W3RPuPgUYBhxmZgftssuTwEh3nwQ8CzzYxnHudfdp7j6tsrIym5FFOtXo/sUE0xTKA0sLCXWg0O1fXJC2EHd3hlUUZXx8aZ/R/UvaXMEyaKmFeUREpMfJyaUqd98CvACcvMv2anffNoHz58Ahucgjkiv9iwsYUh5ptZgOBYyDh/Xp0PHNjMP27dvq9I6gGRMHV2gJ8hw4oLKUSDi42zfUgEFJQUiFtIhID5XNrh2VZtYn9bwIOBF4f5d9Bu/w4ZnAomzlEckHM+Ookf2YsE8ZBcEAAWtZMXxweYSTxg6kIhLu8GsMrSji2P3607c4jLGteAty2L59GLeP2q7lQkEwwEljBzKyXzFB29Y33Bjdv4QTxw7s0G8dRESk68pmg9nBwINmFqSlYH/U3Z8ys5uAOe7+BHClmZ0JxIFNwIVZzCPd2Pab5hz6FIXTTpfoagJmTBhUzvh9ymhOJAkGjFCgcwurfcoinDw2QiyRxL1lTm42Vk2sa4rTGEtQWhiiKLzn+d9djbuzpTFGwp0+kXCnFriFoSCHj+jHtOF9iSWSFIQ6f+Edkc6SaI6y9ePFWDBExYgDsUDnvp/dnbrVHxJrqKV0yGgKSis69fgiXYV1tx6z06ZN8zlz5uQ7huSIu7Okqo531m3dqY3c+H3KGL9PmZbYzpGaxhizV2xiazRGwIyEO5WlhRwxoh/F3aSgXr6pnrdW15BIOgYkHfavLGHykAoVvNJreDLJ+7PuYNlffokFAuBOIBRm3Oe/wYhPnNspr1G18HXm/9+NNNVsxAJBkvEYgw+fyeSL/5NQRNOcpHsxs7nuPq2tcS15Jl3aog11vLtu9x69C9fXEkskmTK0T36C9SJ1TXGeXbKBWOocbGvnt6G2iWcWr+e0cYMId/GpCx9tqucfH2/ZrRXh0qp6orEER47sn6dkIrm14IH/YtXfHyfRHN1p+7sP/g/JeJxRJ36+Q8evfn8ub9x8Kcldjr/2jWeoX7eCo7/z25YCXqSH0L9m6bLiyWSrRTS0rAy4uKqOpg70YJb2Wbhu606rJm7jtCxPvqy6Pveh9kLSnbdX17TazzvhzsdbGqltSt8HWqQnaNy0npUvPbZbEQ0tUz0WPXwryXjHep4v/PUPdiuiAZKxJmpXfUDVwtkdOr5IV6NCWrqsDbVNaf+BBsxYu3X3b9jSuVbWNLbZqzrhsGxTQ07z7K0tjbE2Vx3cZtWWxhylEcmf9W+/CJb+v/3NS+dlfPxYQy01K9ruGZCINrD61T9lfHyRrkiFtHRZCd/zUpjpVg2UzrGHGpTknnbIs6S3zIlue5wOLdMu0l0kY83gaX6LZ0Yi3pz58eOxPd63kmjWD63Ss6iQli6rf0lB2gLH3aksKchhot5pQHHbX2MD9ikrzF2YDFREwml/GAgFjMrSrv13EOkM/cYeTEsjrdYlY830GTUh4+MXlPWloKxfm+PBwmIqJ07P+PgiXZEKaemyisNBhlYU0Vqnu4DBgNJCyjuhD7Okd9Dg8lYXfAEIBIwDB3btXtXhYID9BpS0+ncwWnpu6wcy6Q36jJpA2fD9seDufQYCBYUMm346BaV9Mj6+mXHAZy4jWNDaaqpGsKCQoUedlvHxRboiFdLSpR0xoi8DSwsJmmG0FD7BgNGvuIAZo9RpIRcGlhZyyPAKgqlFRqDlKm4oYBwzuj+lhV2/+c/UoRUMrYi0LJaS2hYKGGWREJ8YU6k2itJrHH7d3ZQPP4BgYRFYAAuGCBQUMnDidCZedGOHjz/i+HMZdfKXCYQLCIQLASMYKaGwzwCm/8cvCRW2VmSLdF/qIy3dwubGZtbUtNxYOKgsQn9dQcy55niSj7c0EI0lKS0MMaxPEaFutDAOQG00xqqaKAl3BpYUUllaoCJaeh13Z/PSeWx87w0sEGTQwZ+gbNiYTn2N6OYNrHn9aWKNtVSMGM8+U4/p9EVfRHJBfaSlR+hbVEDfIhXP+VQQCjBmQGm+Y3RIWSTMOE0Hkl7OzOh3wFT6HTA1K8dvrtvCu7/+Ievm/A1PxCnsU8m4hmsYfvSZWXk9ad3yvz3C4t/fSXPtZiwUZuhRpzHx/H/XojidTFekRUREpFM0VK/j+WtmttqPeuj0Mzjk8h/mIVXvM/sHl1A1/+XdtgcLizjxjhcpKC3Pfahuak9XpPc4R9rMPmNmS82sxsy2mlmtmW3t3JgiIiLS3b3+/a+2uajL6lefZPMH83OcqPdZO+e5VotogERTI2/+6LIcJ+rZ2nOz4Q+BM929wt3L3b3M3fWjjIiIiGwXj9ZTt/rDtPu8++ubc5Sm91r0yG1pxzctnpujJL1Dewrp9e7e9lJFIiIi0uvVV63e4z4NVStzkKR3i25av8d94tH6HCTpHdq82dDMPpN6OsfMHgEeB5q2jbv7Y9mNJiIiIt1FUf8he9wn0ndgDpL0bgVlfYk31qXdJ9Bqr2/JRLor0mekHuVAAzBzh22nZz+aiIiIdBcFxaUUDRiadp9xn70yR2l6rwM+fWna8fIRBxIIaBmRztLmFWl3vwjAzKa7+6s7jpmZ1vgUERGRnRx+3d28eP1Z4MndxvqPP4yBk47OQ6reZd9jP82yp3/J1hXv7zZmwRCHXfuzPKTqudrzI8kd7dwmIiIivVj58DEcf+uf6Tf2ELCWEiMYKWHsOVcw/YYH85yu9zjme79nzBlfaVnBErBAgAEHHcmJP3mO4gGD85yuZ2mzj7SZHQkcBVwN7HgLaDnwaXefnPV0rVAfaREREZGeL9HcxIb5f6dp62bKho6m39hDcr4abUdWNiwASlP7lO2wfStwTufEExERERHZ2erZf2b+/90IgCcTEAhQWNaPw795N2VD98tzun/a48qGZjbC3VfkKM8e6Yq0iIiISM+18b03eeOH/0qiObrLiBEuKeeE2/5KQWlFTrJkfEXazJ4EPPV8t3F3P7MzAoqIiIiIbPP+oz9ppYgGcJKxJlb+/Q/sd+qFuY7VqnQ3G94C/Aj4CGgE/i/1qAPSL10kIiIiIrKX3J1NH8xrczzRHGXtm8/mLtAepGt/9xKAmf1ol0vaT5qZ5laIiIiISKczjHQTj60L9cFuT5ISMxu97QMzGwWUZC+SiIiIiPRGZkb/8Ye3OR4sLGLIkafmMFF67SmkrwFeNLMXzewl4AVaWuKJiIiIiHSqcZ+9imBBZLftFggQKipl+NFd5za9dO3vAHD3p81sf+DA1Kb33b0pu7FEREREpDfqO2YSh37jTt6+63oSTY1ghifilA4bw6FX/4RQpOtMjEjXteN4d3/ezD6zy9B+Zoa7P5blbCIiIiLSCw2cNJ2ZP32JTUveorl2M6VDRnep/tHbpLsifSzwPHBGK2MOqJAWERER6WI8mWTFC7/jgyd+TsPG1YQixQybcRZjz76cwvJ++Y7XLs21m1ny+N18/OJjxKP1FPUfxJjTvsLIE8/DAsF8x9su3RLhU4D5vqcVW3JMC7KIiIiItM7deeun32Td3OdapkWkWDBMYXlfjv2fxyis6J/HhHvWXLuZl751Nk01G0nGY9u3BwuLGDhpBtOu/knOlgrf04Is6W42/DlQbWbPmtl3zWymmZWl2V9ERERE8mjT4rm7FdEAnojRtHUTi373kzwla7/Fj91FdJciGiDR1MiGBa9Q9e5reUq2uzYL6VT1PQz4HtAEXAl8YGbzzexnOconIiIiIu20/G8Pk2hqbVVA8EScVa88QRebbLCblS89hu9SRG+TaGpk+bMP5zhR29J27XD3Blpa3/0DeAOYDpwPnJyDbCIiIiKyF6Kb1kOa5UySsRieiGGhgtyF2gvuTjxan3af6JYNOUqzZ+m6dnwBOAqYQssV6W3F9Ax3X5eTdCIiIiLSbhUjx7Np6Tw8EW91vKCsD4EuWkRDy4IskX77pH4gaGU8EKRi3wNbHcuHdHOk7wGOAB4ALnX36939DyqiRURERLqmkTO/gAVbv04aKIiw36kX5jZQBvY77aJWF2QBsFCY0Sd/OceJ2paukO4DXAJEgO+Y2Vwze8rMvm1mx+cknYiIiIi0W+mgEUy84NsECiI7tYkLFhbT/8Bp7HfaRXlM1z6jT/oSAyYeRbCw+J8bA0ECBREmfOE6yoaNyV+4XbTZ/m63Hc32Ac6lZXnwUe6elyZ+an8nIiIikl7t6mUse/qX1Hy0kIKK/oz65HkMnHwMFkh3DbXr8GSSqnde5aNnf0vTlirKR45j9MlfpnzY/jnNsaf2d+n6SE+iZY70tkcB8BowG3jV3fNSzaqQ7n3cnfV1Taze0ogDQ8ojDC6P5KyHpPQM7s7mpfNY8+YzJJqjVB50JIMO/gSBUDjf0dotGW9mzZvPUr3oH4QiJQw98lT6jJ6Q71giPU4y3szSP97LmjefAYzhx5zF6FMuJNBNitBc8GSSqoWzWT/3BRxn0NTjqJw4vdsU6u3VkUL6LeAV/lk4f5ydiHtHhXTv0hxP8vwHVdQ2xYknW/6thgJGUTjICftXUhTuOqsbSdeVaI7yxi2XsnnpfBLNTeBJgpESwsWlTP+PX1Gyz/B8R9yj2tXLeO2/zyfeFCURrQcLEAwXMuCgIzj06h936ZuHRLqTmpVL+Pu3z9mt/VqgIMLxNz9FceXQPCXrOprrtvDaf19I/YaVJKINAAQjxRQPGMJRNzzYbVZPbI+MF2Rx94Pd/Up3/20mRbSZRczszVTf6YVm9t1W9ik0s0fM7AMze8PMRu7t60jP9tryamqise1FNEA86dQ1xXl5WXUek0l3suAX/82mxW+3LFDgSQAS0Xqim6uY/T8X48lknhOml4zHeO17F9JUs6mliAbwJInmRqrenc17v/lRfgOK9CCv3Hheqz2Mk81RXvr2uXlI1PXM+ck11K5etr2IBkhEG6hbu5x/3HpFHpPlXjavvzcBx7v7ZFpa6J1sZkfsss9XgM3uPga4DfhBFvNIN1PXFGd9XRPJVn5p4sCWxhibG5tznku6l1j9Vla/9hTJWNPug56kqXYTVe/Ozn2wvbDurRdS/2Ht/mZINkdZ8fzviO+yipmI7L3Vr/9ltxUBdxSr20z1+3NzmKjrqVu3gk1L3sYTu/+w4Yk4W5YvpHbVB3lIlh9ZK6S9RV3qw3Dqsev/AmcBD6aezwJOME18lZTNjc0E9vDPYVND6ysfiWyzddUHaedBJ5qibFn2Tg4T7b3NS+elXaDAggHq13eJ2Xci3dr6t1/a4z7r3no+B0m6rpqPFrbZXg/ALMiWZe/mMFF+ZXVGuJkFzWwesAF41t3f2GWXocBKAHePAzVA/1aOc4mZzTGzOVVVVdmMLF1IaA83LJi1zJcWSScUKU47dSMQChPascVSFxQqKk37H1cyEe/yfweR7iBcVLrnfUoqcpCk6wpFikl3jcvMCEZ6z/ejNisVM3vSzJ5o69Geg7t7wt2nAMOAw8zsoExCuvu97j7N3adVVlZmcgjphgaWFqYdd2/p4CGSTvm+YwkXl6XdZ/BhM3OUJjNDjzhlp36wuyrqP6Rb3DAp0tXtd8ZX97jPyBO/kIMkXdeACUfgrc25TPFkgoGTZuQwUX6lu+R3C/CjNI92c/ctwAvAybsMrQaGA5hZCKgAdAeZABAMGFOHVhBs5UffYMA4aHAZ4WDParMjnc/MmHTxf7a6SlawoIgRx59LUf9BeUjWfqVDRjH0yFMJFu7+dwgURJh00X/kIZVIz1PcfxCVk49uc3zY0WdSULznq9Y9WbAgwvjPX9vm99QDP3sVoV50RbrdC7Ls9YHNKoGYu28xsyLgGeAH7v7UDvtcDkx096+Z2XnAZ9z9s+mOq/Z3vc+KTfXMW7OVpngCMMJBY+LgcsYM6N3fzGTvbJj/Mu/88n9p3LgGCwQIhAvZ/8x/Yb/TLuoWPck9mWTJ4/fw4Z9/gSfieDJByaARTDz/2wyYcHi+44n0KG/d9e+seuWJ7V1+LBBk5Cc/z8QLv53nZF3HyleeYNHDt9JctwXDCJeUc+Bnr2LfYz+T72idKuM+0jscYH/gf4HxtCwXDoC7j97D502i5UbCIC1Xvh9195vM7CZgjrs/YWYR4FfAVGATcJ67L0t33HwU0rFEkoZYgsJggIj6FueFu9MQS+AOJQXBblH4SNfUWL2OZLyZogFDCKSZd9xVxaP1bFoyj3BJGX33m5SV12jYuIZkcxPFA4eqP7X0Wsl4nE1L38YCQfruP0WLsbTC3WncuAZ3p3jAkB63GAt0TiH9CvCftLSnOwO4CAi4+42dGbS9cllIxxJJ5q7awsebGzAzku70Ly7g0OF9qSjqPquhiUj358kEi3//M5Y9/SDujicSFA8YwsQLb6By4lGd8hpV77zGOw/8Nw0b12DBIGYBRp98PmPPviztHG0RkZ4q4wVZdlDk7s/RUnSvcPfvAKd1VsCuKpF0/rakihWbG0h4yyIgSYeq+maeXbKB2qZ4viOKSC/y9j3f5sM/3U+8sZ5EtIFkrIm6tR/xxo8uZ8M7r3b4+BveeZU3fnQ5dWs/IhlrIhFtIN5Yx4d/up+379Gvs0VEWtOeQrrJzALAUjP7upl9Gujxk1NX1TRS1xxvdTGQeNJZsKYm96FEpFeqXf0ha954mkRzdLexZHOUd+6/iY7c7+LuLLj/JpKtHD/RHGXNG09TuzrtrDsRkV6pPYX0VUAxcCVwCPAl4PxshuoKllXX77Qs9Y6clkJbRCQX1rz+NJ5o+7dgjZs30LBhVcbHb9iwkujmDW2OeyLOmjeezvj4IiI9VXsK6ZHuXufuq9z9Inc/G9g328HyLZZoewEHgKTToStAIiLtFW9qwBOJNscDgSDxpoYOHL+RQJo50J5IpF1ZUUSkt2pPIf3v7dzWowwsi5Bu0byKSEidI0QkJ/rtPyXtSmHuSUoHjcj4+KWDRuDe9sWDYKSEfvtPzfj4IiI9VZu9n8zsFOBUYKiZ3b7DUDnQ4++0O2BACUuq6lqWz9tFMGAcNKg8D6lEpDfa5+BPEC4qJdHUuNv3pEBBhJEnnNfq4gjtFSyIMOKEz7H8bw/vPk/ajHBRCfscfFzGxxcR6anSXZFeA8wBosDcHR5PACdlP1p+FReEOGZ0f0IBI5i6NB2wlseBA0vZt2/vWbVHRPIrEAxx1A0PUFgxgGCkJLUxQLAgwsBJMxh33jUdfo3x532DgZNmtBTkqV6wwUgJhRUDOOqGB7plz20RkWxrTx/pMC1Xrvd198U5SZVGrhdkiSWSrNjcwJbGGJFwkFH9iikp0H8oIpJ7yXiMdXOeo/r9OYQixQw54hQqRo7r1NeoWb6INa//hXi0gf4HTmPQtBMIhNQ3X0R6p85YkOUM4BagwN1HmdkU4CZ3P7NTk7aTlggXERHp3Va99hTvP3o70c0bCEWKGXni5zngU5cSCOlCV64019Xw8UuPseb1pwFn8KEnMuL4cyko7ZPvaJ2qMwrpucDxwIvuPjW17R13n9ipSdtJhbSIiEjv9fotl7LhrRd3215QMYCZdzynZe1zoG7tcl75zueJN0W331cRKIgQDBcy4zsPUTZ0vzwn7DydsbJhzN13XX1Efd9EREQkp1a/9udWi2iA5pqNzLnzutwG6qXevPXrNNfV7HRzcrI5Sqx+K2/eclmvag/cnkJ6oZl9AQia2f5mdgfwWpZziYiIiOxk0aM/Tju+fs5zuQnSi235aCGNG9e02tUMnOiWjWz+YH7Oc+VLewrpK4AJQBPwG6AGuDqLmURERER2E91SlXbck20vXCSdo37tcszSl491a5fnJkwXkK6PdAT4GjAGeAc40t17fP9oERER6ZpCkRKad+11vhMtlJZtBRX9087vtYBRWN4vZ3nyLd2PFA8C02gpok+hpXOHiIiISF6MPumLaccr9jsoR0l6rwHjDiUYbvuGTgsEqTzoyBwmyq90hfR4d/+Su98DnAMck6NMIiIiIrsZc9a/Uth3YKtjFghy6JW35ThR72OBIAdf9sPUaqo7/wYgWBBh6qXf71W959MV0rFtTzSlQ0RERPItEAhw4k+eZcgRp2KBYGqr0WfMJE647a8UVw7Na77eYuDkGUy/8dcMnHIMgVABgVCYyknTOeqGBxh08CfyHS+n2uwjbWYJoH7bh0AR0JB67u5enpOEu1AfaRERERHJhT31kW7zZkN3D7Y1JiIiIiLS27Wn/Z2IiIiIiOxChbSIiIiISAZUSIuIiIiIZECFtIiIiIhIBlRIi4iIiIhkQIW0iIiIiEgGVEiLiIiIiGRAhbSIiIiISAbaXJBFRESkMyUTcVa8MItlf36A6OYNFFYMYPTJX2bkJz9HIFSQ73giIntNhbSIiGSdJxO8cfOlbFo8l0RTIwANG1by3sO3subNZzjqW/epmBaRbkdTO0REJOtWvfannYrobZLNUWqWvcuKFx/LUzIRkcypkBYRkaxb9vSvdiuit0k0R/nor7/OcSIRkY5TIS0iIlnXtKVqD+Mbc5RERKTzqJAWEZGsKx44PP145dAcJRER6TwqpEVEJOvGnHYxwcKiVseChUXsd/rFOU4kItJxKqRFRCTr9jn4OIZOP323YjpYWMSgQ45n6JGn5imZiEjm1P5ORESyzsyY/JXvMuTQmXz4lwdoWL+SogFDGH3K+ewz9TjMLN8RRUT2mgppERHJCTNj4OQZDJw8I99RJMsaq9ey5vW/EmvYSsXIcexz8CcIBFVy5FIi1sy6OX9j66oPKCztw5AjTyHSpzLfsXoc/asWERGRTuHuvPfbW1raGTok482EIiUECyMc+a37KR9+QL4j9gqbP1jA6z+4hGQiTiJaTyBcyHsP/4gDPn0pB3zqa/mO16NkbY60mQ03sxfM7D0zW2hmV7Wyz3FmVmNm81KPG7OVR0RERLJrxXOPsPzZ35KMNZOMNwMQj9bTVFPNq/91AfFoQ54T9nzNtZuZ/b9fIVZfQyJaD0Ay1kQy1szSP97LmtefznPCniWbNxvGgWvdfTxwBHC5mY1vZb+X3X1K6nFTFvOIiIhIlrg7ix/7WZsL7yRjzaye/accp+p9Vrwwi2Qi3upYoqmR92fdkeNEPVvWCml3X+vub6We1wKLADUKFRER6YGaazcTq6tpczzR1EDVgldzmKh32rDgFZLN0TbH69Z+RDIey2Gini0n7e/MbCQwFXijleEjzWy+mf3FzCa08fmXmNkcM5tTVZV+dSwRERHJvUCoAPdk2n2ChcU5StN7tdWvfRuzABYI5ihNz5f1QtrMSoHfA1e7+9Zdht8CRrj7ZOAO4PHWjuHu97r7NHefVlmpO05FRES6mnBxKRWjWpvB2SIYKWbY9NNzmKh32vfoTxGMtPUDi1E5aQYW0DIinSWrX0kzC9NSRD/k7o/tOu7uW929LvX8z0DYzAZkM5OIiIhkx0Ffup5gQWS37YFwIeXDD2DAQUfmIVXvMmjaCRRXDiUQKthtLFgYYdx51+QhVc+Vza4dBtwHLHL3W9vYZ1BqP8zssFSe6mxlEhERkezpd8BUDv9/91AyaATBggiholIC4UKGHnkqR37rfi28kwOBUJgZ//kQg6YdTyBc2HIOCiKUDdufo779ABX7js13xB7F3D07BzabAbwMvANsmzT1LWBfAHe/28y+DlxKS4ePRuAb7v5auuNOmzbN58yZk5XMIiIdlWhuIlZfQ7i0D8Hw7leERHoDd6d+3XJiDXWUDhpBuKQ835F6pea6LdSvX0lBaR9K9hme7zjdkpnNdfdpbY1nbUEWd38FSPujp7vfCdyZrQwiIrnStHUzCx/6AWte/wtgYDDsqNOZ8MVvqoiQXsfMKB08Kt8xer2C0j4UlPbJd4weTSsbioh0UKx+K3+/4RyimzfgO/RvXfnyE1Qvnsux3/s9oTZv/hERke5Kt22KiHTQsr/+mqaa6p2KaABPxIhWr+PjF3+fp2QiIpJNKqRFRDro4xd/TzLW1OpYojnKiucfzXEiERHJBRXSIiIdFI82pB2PNdTlKImIiOSSCmkRkQ6qGHFg24Nm9Bnd6qKtIiLSzamQFhHpoP3PuqTVRSgAguFCxpzx1RwnEhGRXFAhLSLSQZUHHckBZ3+dQLgQC7U0QwqECgiECxn/hevot/+U/AYUEZGsUPs7EZFOsP8ZX2HIYTNZ8fyj1K//mNIhoxlx/LkUDxiS72giIpIlKqRFRDpJyT7DGf/5a/MdQ0REckRTO0REREREMqBCWkREREQkAyqkRUREREQyoEJaRERERCQDutlQREREpAuLxWKsWrWKaDSa7yg9ViQSYdiwYYTD4b36PBXSIiIiIl3YqlWrKCsrY+TIkZhZvuP0OO5OdXU1q1atYtSoUXv1uZraISIiItKFRaNR+vfvryI6S8yM/v37Z3TFX4W0iIiISBenIjq7Mv36amqHiIhILxNrqGPLR+8SCATpM2YywXBBviOJdEsqpEVERHqJZCLOwl//kBXPP0oglLqpyp2x53yd0adcoKuePcTyTfXMX7OVhliC4nCQyUPKGdmvpEPH/N73vsdvfvMbgsEggUCAe+65h8MPP7zVfR944AFmzpzJkCFDOvSa3YEKaRERkV5i/v/dyOrX/0Iy1kQy1rR9+/u/ux0Hxpx6Yd6ySedYvqmeNz/eQsIdgIZYgjc/3gKQcTE9e/ZsnnrqKd566y0KCwvZuHEjzc3Nbe7/wAMPcNBBB/WKQlpzpEVERHqBho1rWD37TySbd7+hKtHUyJJZd5KItV0cSfcwf83W7UX0Ngl35q/ZmvEx165dy4ABAygsLARgwIABDBkyhLlz53LsscdyyCGHcNJJJ7F27VpmzZrFnDlz+OIXv8iUKVNobGzkueeeY+rUqUycOJGLL76YpqaWH+Kuv/56xo8fz6RJk/i3f/s3AJ588kkOP/xwpk6dyic/+UnWr1+fce5cUCEtIiLSC6x/+yXM0vy3b8bmpfNylkeyoyGW2Kvt7TFz5kxWrlzJAQccwGWXXcZLL71ELBbjiiuuYNasWcydO5eLL76Yb3/725xzzjlMmzaNhx56iHnz5mFmXHjhhTzyyCO88847xONx7rrrLqqrq/nDH/7AwoULWbBgATfccAMAM2bM4PXXX+ftt9/mvPPO44c//GHGuXNBUztERER6AU/E8WQy7T7JRCxHaSRbisPBVovm4nAw42OWlpYyd+5cXn75ZV544QU+97nPccMNN/Duu+9y4oknApBIJBg8ePBun7t48WJGjRrFAQccAMAFF1zAT3/6U77+9a8TiUT4yle+wumnn87pp58OtPTM/tznPsfatWtpbm7e677OuaZCWkREpBfoP/4wLND2FelkPEbf0RNzmEiyYfKQ8p3mSAMEzZg8pLxDxw0Ggxx33HEcd9xxTJw4kZ/+9KdMmDCB2bNnZ3S8UCjEm2++yXPPPcesWbO48847ef7557niiiv4xje+wZlnnsmLL77Id77znQ7lzjZN7RAREekFKvYdS5/RB/2zW8cOggUR9j3ubMIlHSu2JP9G9ivhsH37bL8CXRwOcti+fTrUtWPx4sUsXbp0+8fz5s1j3LhxVFVVbS+kY7EYCxcuBKCsrIza2loAxo4dy/Lly/nggw8A+NWvfsWxxx5LXV0dNTU1nHrqqdx2223Mnz8fgJqaGoYOHQrAgw8+mHHmXNEVaRERkV7isGt/yhu3XEbNRwtbpnmk2t0NOvSTHPTl6/OcTjrLyH4lHW53t6O6ujquuOIKtmzZQigUYsyYMdx7771ccsklXHnlldTU1BCPx7n66quZMGECF154IV/72tcoKipi9uzZ/OIXv+Dcc88lHo9z6KGH8rWvfY1NmzZx1llnEY1GcXduvfVWAL7zne9w7rnn0rdvX44//ng++uijTvt7ZIP5Lnd2dnXTpk3zOXPm5DuGiIhIt1WzfBEbF71JIBhi4JRjKRk4LN+RJI1FixYxbty4fMfo8Vr7OpvZXHef1tbn6Iq0iIhIL1MxchwVI1WYiXSU5kiLiIiIiGRAhbSIiIiISAZUSIuIiIiIZECFtIiIiIhIBlRIi4iIiIhkQIW0iIiISA+y6pUnefbKE3jiC+N59soTWPXKkx0+pplx7bXXbv/4lltu2eOqg48//jjvvfdeq2OLFy/muOOOY8qUKYwbN45LLrkk7bGWL1/Ob37zm73OnW0qpEVERER6iFWvPMn8n99I48Y1gNO4cQ3zf35jh4vpwsJCHnvsMTZu3Njuz0lXSF955ZVcc801zJs3j0WLFnHFFVekPZYKaRERERHJqkWP/phEc3SnbYnmKIse/XGHjhsKhbjkkku47bbbdhtbvnw5xx9/PJMmTeKEE07g448/5rXXXuOJJ57guuuuY8qUKXz44Yc7fc7atWsZNuyfCwFNnDixJWsiwXXXXcehhx7KpEmTuOeeewC4/vrrefnll5kyZQq33XYb0WiUiy66iIkTJzJ16lReeOEFABYuXMhhhx3GlClTmDRp0valzT/1qU9xyCGHMGHCBO69994OfS12+rp02pFEpE3uTvV7b1KzYhHh4jIGTTuBgtI++Y4lIiI9TOPGta1vr259+964/PLLmTRpEt/85jd32n7FFVdwwQUXcMEFF3D//fdz5ZVX8vjjj3PmmWdy+umnc8455+x2rGuuuYbjjz+eo446ipkzZ3LRRRfRp08f7rvvPioqKvjHP/5BU1MT06dPZ+bMmXz/+9/nlltu4amnngLgRz/6EWbGO++8w/vvv8/MmTNZsmQJd999N1dddRVf/OIXaW5uJpFIAHD//ffTr18/GhsbOfTQQzn77LPp379/h78mKqRFsqx+/cfM/v5XaaqpJhmPEQiGWfCL/2Ls2Zez/5n/ku94IiLSgxQNGJya1rHL9v6DO3zs8vJyzj//fG6//XaKioq2b589ezaPPfYYAF/+8pd3K7Rbc9FFF3HSSSfx9NNP88c//pF77rmH+fPn88wzz7BgwQJmzZoFQE1NDUuXLqWgoGCnz3/llVe2Twc58MADGTFiBEuWLOHII4/ke9/7HqtWreIzn/kM+++/PwC33347f/jDHwBYuXIlS5cu7ZRCOmtTO8xsuJm9YGbvmdlCM7uqlX3MzG43sw/MbIGZHZytPCL5kGhu4pXvfpGGDatJRBvweIxEUwPJWBNLHvsZK195It8RRUSkBxn32asJFkR22hYsiDDus1d3yvGvvvpq7rvvPurr6zt8rCFDhnDxxRfzxz/+kVAoxLvvvou7c8cddzBv3jzmzZvHRx99xMyZM9t9zC984Qs88cQTFBUVceqpp/L888/z4osv8re//Y3Zs2czf/58pk6dSjQa3fPB2iGbc6TjwLXuPh44ArjczMbvss8pwP6pxyXAXVnMI5Jza17/C/FoA3hyt7FEc5T3H/0J7p6HZCIi0hMNm3EGk796E0UDhoAZRQOGMPmrNzFsxhmdcvx+/frx2c9+lvvuu2/7tqOOOoqHH34YgIceeoijjz4agLKyMmpra1s9ztNPP00sFgNg3bp1VFdXM3ToUE466STuuuuu7WNLliyhvr5+t2MdffTRPPTQQ9v3+fjjjxk7dizLli1j9OjRXHnllZx11lksWLCAmpoa+vbtS3FxMe+//z6vv/56p3wtIItTO9x9LbA29bzWzBYBQ4Edb988C/ilt1QSr5tZHzMbnPpckW5v/by/k4g2tDnetKWK5totFJb3zWEqERHpyYbNOKPTCufWXHvttdx5553bP77jjju46KKLuPnmm6msrOQXv/gFAOeddx7/8i//wu23386sWbPYb7/9tn/OM888w1VXXUUk0nL1/Oabb2bQoEF89atfZfny5Rx88MG4O5WVlTz++ONMmjSJYDDI5MmTufDCC7nsssu49NJLmThxIqFQiAceeIDCwkIeffRRfvWrXxEOhxk0aBDf+ta3KCkp4e6772bcuHGMHTuWI444otO+FpaLq2FmNhL4O3CQu2/dYftTwPfd/ZXUx88B/8/d5+zy+ZfQcsWafffd95AVK1ZkPbNIZ3jrZ/+PVWmmb1gwxMn3vEa4uCyHqUREpDtZtGgR48aNy3eMHq+1r7OZzXX3aW19Ttbb35lZKfB74Oodi+i94e73uvs0d59WWVnZuQFFsmjIEScTjBS3OV42bH8V0SIiIt1UVgtpMwvTUkQ/5O6PtbLLamD4Dh8PS20T6RH2mXIMJQOHEwiFdxsLFESY8KU939ksIiIiXVM2u3YYcB+wyN1vbWO3J4DzU907jgBqND9aehILBJn+H7+kctIMAuECQsVlhCIlFPapZNqVt1I5ofPmaYmISM+lG9OzK9Ovbzb7SE8Hvgy8Y2bzUtu+BewL4O53A38GTgU+ABqAi7KYRyQvwiXlHP5vPyO6eQNbVy4lXFxKn9ETsYAWFhURkT2LRCJUV1fTv39/Wq5TSmdyd6qrq7ff+Lg3stm14xUg7dlOdeu4PFsZRLqSSN+BRPoOzHcMERHpZoYNG8aqVauoqqrKd5QeKxKJ7LRkeXtpZUMRERGRLiwcDjNq1Kh8x5BW6HfLIiIiIiIZUCEtIiIiIpIBFdIiIiIiIhnIycqGncnMqoCeurThAGBjvkNI1uk893w6x72DznPvoPPcO7R1nke4e5urAXa7QronM7M56ZahlJ5B57nn0znuHXSeewed594h0/OsqR0iIiIiIhlQIS0iIiIikgEV0l3LvfkOIDmh89zz6Rz3DjrPvYPOc++Q0XnWHGkRERERkQzoirSIiIiISAZUSIuIiIiIZECFdB6YWdDM3jazp1oZu9DMqsxsXurx1XxklI4zs+Vm9k7qPM5pZdzM7HYz+8DMFpjZwfnIKZlrxzk+zsxqdng/35iPnNIxZtbHzGaZ2ftmtsjMjtxlXO/lHqAd51nv527OzMbucP7mmdlWM7t6l3326v0cympiactVwCKgvI3xR9z96znMI9nzCXdvq5H/KcD+qcfhwF2pP6V7SXeOAV5299Nzlkay4SfA0+5+jpkVAMW7jOu93DPs6TyD3s/dmrsvBqZAy0VNYDXwh11226v3s65I55iZDQNOA36e7yySd2cBv/QWrwN9zGxwvkOJyD+ZWQVwDHAfgLs3u/uWXXbTe7mba+d5lp7lBOBDd991tey9ej+rkM69HwPfBJJp9jk79euEWWY2PDexJAsceMbM5prZJa2MDwVW7vDxqtQ26T72dI4BjjSz+Wb2FzObkMtw0ilGAVXAL1JT8n5uZiW77KP3cvfXnvMMej/3JOcBv21l+169n1VI55CZnQ5scPe5aXZ7Ehjp7pOAZ4EHcxJOsmGGux9My6+JLjezY/IdSDrdns7xW8AId58M3AE8nuN80nEh4GDgLnefCtQD1+c3kmRBe86z3s89RGrqzpnA7zp6LBXSuTUdONPMlgMPA8eb2a933MHdq929KfXhz4FDchtROou7r079uYGWOViH7bLLamDH3zgMS22TbmJP59jdt7p7Xer5n4GwmQ3IeVDpiFXAKnd/I/XxLFoKrh3pvdz97fE86/3co5wCvOXu61sZ26v3swrpHHL3f3f3Ye4+kpZfKTzv7l/acZ9d5uGcSctNidLNmFmJmZVtew7MBN7dZbcngPNTdwgfAdS4+9ocR5UMteccm9kgM7PU88No+Z5bneuskjl3XwesNLOxqU0nAO/tspvey91ce86z3s89yudpfVoH7OX7WV07ugAzuwmY4+5PAFea2ZlAHNgEXJjPbJKxfYA/pL7nhoDfuPvTZvY1AHe/G/gzcCrwAdAAXJSnrJKZ9pzjc4BLzSwONALnuZaT7Y6uAB5K/Tp4GXCR3ss90p7Os97PPUDqwseJwL/usC3j97OWCBcRERERyYCmdoiIiIiIZECFtIiIiIhIBlRIi4iIiIhkQIW0iIiIiEgGVEiLiIiIiGRAhbSISA6ZWcLM5pnZu2b2OzMrTrPvFDM7tR3HPM7Mnmrv9s5iZn3M7LJcvZ6ISFejQlpEJLca3X2Kux8ENANfS7PvFFr6mXZVfYDL9rSTiEhPpUJaRCR/XgbGpFZJvN/M3jSzt83srNSiEDcBn0tdwf6cmR1mZrNT+7y2wypse8XMZqaO81bqqnhpavtyM/tuavs7ZnZganulmT1rZgvN7OdmtiK1NPL3gf1S+W5OHb7UzGaZ2ftm9tC2leBERHoiFdIiInlgZiHgFOAd4NvA8+5+GPAJ4GYgDNwIPJK6gv0I8D5wtLtPTY39TwavOwC4Afikux8MzAG+scMuG1Pb7wL+LbXtP1P5JgCzgH1T268HPkzluy61bSpwNTAeGA1M39uMIiLdhZYIFxHJrSIzm5d6/jJwH/AacKaZbStcI/yzWN1RBfCgme0POC3F9t46gpYi99XUxeICYPYO44+l/pwLfCb1fAbwaYDUMuib0xz/TXdfBZD6e44EXskgp4hIl6dCWkQktxrdfcqOG1LTH85298W7bD98l8/9L+AFd/+0mY0EXszg9Q141t0/38Z4U+rPBJn9H9G0w/NMjyEi0i1oaoeISP79Fbhi23xiM5ua2l4LlO2wXwWwOvX8wgxf63VgupmNSb1WiZkdsIfPeRX4bGr/mUDfNvKJiPQqKqRFRPLvv2iZprHAzBamPgZ4ARi/7WZD4IfA/5rZ27T/Su8JZrZq2wMYQ0sR/lszW0DLtI4D93CM7wIzzexd4FxgHVDr7tW0TBF5d4ebDUVEeg1z93xnEBGRLszMCoGEu8fN7Ejgrl2np4iI9EaauyYiInuyL/ComQVo6X39L3nOIyLSJeiKtIiIiIhIBjRHWkREREQkAyqkRUREREQyoEJaRERERCQDKqRFRERERDKgQlpEREREJAP/H4Djw3TKQm7hAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# https://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.rcParams['figure.figsize'] = [12, 4]\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "# create 2d version of dataset and subsample it\n", "X = iris_X.to_numpy()[:,:2]\n", "x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n", "y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, .02), np.arange(y_min, y_max, .02))\n", "\n", "# Plot also the training points\n", "p1 = plt.scatter(X[:, 0], X[:, 1], c=iris_y2, s=60, cmap=plt.cm.Paired)\n", "plt.xlabel('Petal Length')\n", "plt.ylabel('Petal Width')\n", "plt.legend(handles=p1.legend_elements()[0], labels=['Setosa', 'Not Setosa'], loc='lower right')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.1.2. Comparing Classification Algorithms\n", "\n", "We have seen different types approaches to classification. When fitting a model, there may be many valid decision boundaries. How do we select one of them?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Consider the following three classification algorithms from `sklearn`. Each of them outputs a different classification boundary." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import LogisticRegression, Perceptron, RidgeClassifier\n", "models = [LogisticRegression(), Perceptron(), RidgeClassifier()]\n", "\n", "def fit_and_create_boundary(model):\n", " model.fit(X, iris_y2)\n", " Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " return Z\n", "\n", "plt.figure(figsize=(12,3))\n", "for i, model in enumerate(models):\n", " plt.subplot('13%d' % (i+1))\n", " Z = fit_and_create_boundary(model)\n", " plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired) \n", "\n", " # Plot also the training points\n", " plt.scatter(X[:, 0], X[:, 1], c=iris_y2, edgecolors='k', cmap=plt.cm.Paired)\n", " plt.title('Algorithm %d' % (i+1))\n", " plt.xlabel('Sepal length')\n", " plt.ylabel('Sepal width')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### 12.1.2.1. Classification Scores\n", "\n", "Most classification algorithms output not just a class label but a score.\n", "For example, logistic regression returns the class probability\n", "\n", "$$ \n", "p(y=1|\\mid x) = \\sigma(\\theta^\\top x) \\in [0,1] \n", "$$\n", "\n", "If the class probability is $>0.5$, the model outputs class $1$. \n", "The score is an estimate of confidence; it also represents how far we are from the decision boundary." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### 12.1.2.2. The Max-Margin Principle\n", "\n", "Intuitively, we want to select boundaries with high *margin*. \n", "This means that we are as confident as possible for every point and we are as far as possible from the decision boundary." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Several of the separating boundaries in our previous example had low margin: they came too close to the boundary." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAADgCAYAAADFcDomAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABra0lEQVR4nO3ddXxT1//H8ddJ3Z0aLVDcGe422NhgMGfynbu7uwvf+b5z+83dYDAFhjPcXUpbSt0tSXN+fyR0LS3VpEnaz/Px6GPt7c3NCev7nnPvPaK01gghhBBCCCEaz+DsAgghhBBCCOFupBEthBBCCCFEE0kjWgghhBBCiCaSRrQQQgghhBBNJI1oIYQQQgghmkga0UIIIYQQQjSRNKJFLUqpg0qpKc4uR3VKqfuVUu85uxxCuALJqBDuQ/Ladkkj2k0opc5TSq1WSpUopTJt31+vlFKtXI6PlFJaKTXrmO0v2bZf6oj31Vo/rbW+0hHHFsLRbJVomVKqWCmVp5T6RSmV4KD3kowK0QKSV9FY0oh2A0qpO4BXgDlADBANXAuMAbydUKTdwMXVyucJnAvsa87BbK8Xoq07TWsdCMQCGcBrDnwvyagQLSN5FQ2SRrSLU0qFAI8D12utv9VaF2mrDVrrC7XWFUf3U0p9rJTKUkolK6UeVEoZbL8z2H5Ott3F/th23KPvcZHtdzlKqQcaUay5wFilVJjt52nAZuBItWN2VUottB0zWyn1mVIqtNrvDyql7lFKbQZKlFKeSqmLq5XjoeqPwJRSjyqlPrV939l2hX6JUuqQ7fiNKbcQTqe1Lge+Bfoc3aaUmq6U2qCUKlRKpSilHq3+GsmoEM4heRX1kUa06xsF+AA/NbDfa0AIkARMwHpVe5ntd5favibZfh8IvA6glOoDvAlcBMQBEUDHBt6r3Fae82w/Xwx8fMw+CnjGdszeQALw6DH7nA9MB0KBHsAbwIVYr/xDgPgGyjEW6AmcCDyslOrdwP5COJ1Syh+YDayqtrkEa45CsWbiOqXU6bb9JaNCOInkVdRHGtGuLxLI1lqbj25QSq1QSuUra5+t8UopD6zhu892p/og8ALWEIM1RC9qrfdrrYuB+4DzlPWRz9nAPK31Ettd7YcASyPK9TFwse1KeALwY/Vfaq33aq3/0FpXaK2zgBdt+1X3qtY6RWtdZivHXK31Mq21EXgY0A2U4TGtdZnWehOwCRjYiHIL4Sw/KqXygQJgKtbuWQBorRdrrbdorS1a683AF/ybF8moEK1P8ip5bZA0ol1fDhCpqvVx0lqP1lqH2n5nwNrQ9gKSq70umX+vOuPq+J0n1r7VcUBKtWOX2I5bL631MiAKeADrCaOs+u+VUtFKqS+VUmlKqULgU1s5q0up9v2x5ShtRDmOVPu+FOsddiFc1em23PoCNwJ/K6ViAJRSI5RSi5S1O1YB1jEPR/MiGRWi9UleJa8Nkka061sJVACz6tknGzABnaptSwTSbN8fruN3ZqyDJdKxPhYCqh5dRTSybJ8Cd1D7sRPA01ivevtrrYOB/2B9HFVd9avidKo98lJK+TWhHEK4Da11pdb6e6AS6+NTgM+Bn4EErXUI8Bb/5kUyKoSTSF5FfaQR7eK01vnAY8AbSqmzlVJByjpQcBAQYNunEvgaeMr2+07A7VgDCdZHTbcppboopQKxhvErWxeRb4EZSqmxSilvrIMYG/t38SrWx1xL6vhdEFAMFCil4oG7GjjWt8BpSqnRtnI8Su0ThBBuT1nNAsKAHbbNQUCu1rpcKTUcuKDaSySjQjiJ5FXURxrRbkBr/TzWRvHdWO8eZwBvA/cAK2y73YR1sMN+YBnWK+UPbL/7APgEazAPYB3EcJPt2NuAG2z7pwN5QGojy5Wrtf5La11XP6vHgMFY+5P9AnzfwLG22cr0pa0cxUAm1rvwQrQFc5VSxUAh8BRwie3vHuB64HGlVBHWvotfH32RZFQIp5C8igapuv9fCeFctjvm+UB3rfUBJxdHCHEMyagQ7kPy6hhyJ1q4DKXUaUopf6VUAPBfYAtw0LmlEkIcJRkVwn1IXh1PGtHClczCOgjyMNAdOO84j7WEEM4hGRXCfUheHUy6cwghhBBCCNFEcidaCCGEEEKIJpJGtBBCCCGEEE3k2fAuLWNbknotkKa1nnHM7y7FupTm0UVBXtdav1ff8YJCw3VUXEPL0AvRfhzYsSVbax1lj2PZO68gmRXtjyFjz3F/ty+3XPIqhAPUl7uWqC+zDm9EA7dgnaA8+Di//0prfWNjDxYV15GnPptvl4IJ0RZcMDghueG9Gs2ueQXJrGhfAuZMBTof9/ezvtgpeRXCzhrKXUvUl1mHdudQSnUEpgMNXv0KIZxL8ipE8wXMmWqryFuH5FUIq9bM3bEcfSf6Zayr7AXVs89ZSqnxwG7gNq11ioPLJESbYeeTx8tIXoVoEidW4C8jeRXtmDMbz0c57E60UmoGkKm1XlfPbnOBzlrrAcAfwP8d51hXK6XWKqXWFuXlOqC0QrgXe9/1smdebceTzIo2rbXvPFcneRXtnSs0oMGx3TnGADOVUgexrt0+WSn1afUdtNY5Wuuj67i/Bwyp60Ba63e01kO11kODwsIdWGQh2i275dW2r2RWCMeRvArhAhzWnUNrfR9wH4BSaiJwp9b6P9X3UUrFaq3TbT/OxDpAQghxHI66+pa8CtE4rnAHTPIq2jNXyOBRrTE7Rw1KqceBtVrrn4GblVIzATOQC1za2uURwl0448QheRXiX65UeddF8iraMlfMX6s0orXWi4HFtu8frra96mpaCFG31j5xSF6FqMkVK++jJK+iPXDVDMqKhUK4MFc9cQjRXkgGhXAuV85gq3fnEELUz5VPGEK0B5JBIZzPHXIod6KFcCHucNIQoi2TDArhfO6SQ7kTLYQLcJcThhBtlWRQCNfgTlmURrQQTuZOJwwh2hrJnxCuw93yKI1oIZzE3U4WQrQ1kkEhXIc75lEa0UK0Mnc8UQjRVkj+hHA97ppLaUQL0Urc9SQhRFshGRTC9bhzLqURLUQrcOeThBDuTvInhGty92xKI1oIB3L3E4QQ7k4yKITraSu5lEa0EA7SVk4SQrgbyZ4Qrqst5VMa0ULYUVs6OQjhjiSDQriutpZPaUS3MbmZ6ezc8A+BwaH0HTYGD0/5X9xa2trJQTiesaKcLauWYjJW0G/4GAJDwpxdJLcl+ROOprVm39aNZKQepFOPPnTs2tPZRXIrbTGj0sJqI7TWfPfWi/z21Yf0GTqa3Mx0Pnjmfu58+UM6JvVwdvHatLZ4YhCOt2PdKl655zo6JvXA19+f9564hwtve5BJZ5zv7KK5FcmfaA3FBXm8cNsVFORm07lXPz576Ul6njCcG558BU8vb2cXz+W11ZxKI7qN2LhsISv/mMsLP/xNcFgEAIt+/JJX77mO577+E6WUk0sohDiqoqyMV+6+lhuefo3+I8YBcOTQAR69/Ey69T+BhG69nFxCIUR1H//3MRJ79OGSux/HYDBgMlbw8p3X8PNHb3DmVbc6u3jCSQzOLoCwj6XzvmXGRddUNaABJs6aTaXZzMGdW51YsratrV5dC8fauHwhnXv1q2pAA8QkdmHirNksX/Cj8wrmZiR/ojWYjBWsWbiAc66/E4PB2mzy8vbh7OvuYOm875xcOtfXlnMqd6LbiIqyMgKCQ2psU0rhHxRMeWmJk0rVdrXlk4JwvIqy0lp5BQgIDiU347ATSuQ+JHuitVWazVgslfj6BdTYHhAcQkVZqZNK5R7ael7lTnQbMWjsJBb98AUWi6Vq28GdW8lMPUTXfoOcV7A2JmDO1DZ/UhCO12/EWDav/Ju8rCNV24wV5Sz75VsGjZnkxJK5NsmecAZf/wC69h3Esvnf19i+8PvPGTR2spNK5fraQ14dfidaKeUBrAXStNYzjvmdD/AxMATIAWZrrQ86ukxt0YRZs1n1xzyevPpcRk+bRW5GOgu//5xL73kCbx9fZxfP7bWHkwFIXltLeIdYZl1+Iw9dPJMpZ1+EX0Agi3/8koTuvek/aoKzi+dy2kv+mkMy2zouvvNRnr3xIvZv30SX3gPYunope7ds4OH3pTvHsdpTXlvjTvQtwI7j/O4KIE9r3Q14CXiuFcrTJnn7+HLfG58x+YwL2L9tE2aTiQff/ZpRJ890dtHcXns6ISB5bTUzLr6WW59/m4KcLA7t2cnZ193B9U+8UtXnUli1s/w1h2S2FXTu1Y9nvviVsKhodm9cQ7d+J/D0FwuIiI51dtFcSnvLq0PvRCulOgLTgaeA2+vYZRbwqO37b4HXlVJKa60dWa62ytPLm7HTz2Ts9DOdXZQ2ob2dDCSvra9b/xPo1v8EZxfDJbW3/DWHZLZ1hUVFc8aVtzi7GC6rPWbW0bc8XgbuBizH+X08kAKgtTYDBUDEsTsppa5WSq1VSq0tyst1UFGF+Fd7PBlgp7yCZFa0TDvNX3O8jNSxwgW018w67E60UmoGkKm1XqeUmtiSY2mt3wHeAUjqM0CuoIVDtNeTANg3ryCZFU3XnvPXHFLHClfQ3nPryO4cY4CZSqlTAV8gWCn1qdb6P9X2SQMSgFSllCcQgnXwg3Awi8XC8vk/sOLXHzEZjQweP4UpZ/8Hb18/ZxfNKdr7iQDJq8vbv30zv335IZmpyXTq2ZdTLryS6I6dnF2sFpPsNZtk1oUV5uXy25cfsGPtSoLCIph85gUMHD3R2cWyK8muA7tzaK3v01p31Fp3Bs4DFh4TboCfgUts359t20euglvBh88+wG9ffsCEWbOZfvE1bPtnOc/ddDFmk8nZRWtVMmWdleTVtW1avojnb76ExB69OfeGu/ALCOSRS08nbf8eZxetRSR7zSeZdV1F+Xk8cuks8rMzOfPq2zhh7GQ+eOYBfvvyQ2cXzS6k3vxXqy+2opR6HFirtf4ZeB/4RCm1F8jFeiIQDpa2fw9rF/3GSz8txdffOnn8wNETeezyM1n39++MmDLdySVsHXISaJjk1fm01nz+ytNc+9iLVXNI9x4yisCQUL575yVufvYNJ5ew6SR7jiOZdb7fv/qI3oNHcNVDz1dt6z1kJA9dfBoTZp5bVe+6I8luTa3SiNZaLwYW275/uNr2cuCc1iiD+NeujWsYOHpijSAbDAaGTzmVnetXt/lGtJwE6id5dS0lRQVkp6fWehQ8cupp/PLJO84pVDNJ9hxDMutadm74hxkXXV1jW3RCZ6ITOpO8ezs9Bw1zUslaRvJbmyz73Q4Fh0eQmXao1vbM1EOERXZwQolah5wAhDvy9fNHKQMFOVmEVstnZloyIeGRTixZ40n2RHsSUkcdazaZyDly2G0yeyzJcN1kVn83VZCTxR9f/x+/fPI2aQf2Num1g8ZMIjs9rcYy4dv+Wc6q3+cybsbZjiiuU0n/LeFsZpOR1X/+ws8f/o+Nyxdhqaxs9Gs9vbwZN+MsPnz2QcpLSwDIz87ks5eeZMrZFzmqyHYj2RPuaP/2Tcz96E0W/fglpUWFTXrtiWddyM8fvUHq/t2ANf/fvDGH+KQexCR2cURxHUoyfHxyJ9oNrVm4gHcev4vB46fi6x/Ak1efy+QzL+Cc6+5s1Os9vby5+7X/438P3MwP776Ct68fxvIybnz6NSJi4hxc+tYl4RfOlpORztPXnkdYVDRJfQbyzRtz+P6dl7n39U/wDwpu1DHOv+V+Pnj6Pm46dQRRcYlkph3ipHMvZvJZFzq49M0n2RPuyGKx8O7jd7FtzQqGTTqZfds38dVrz3LbC+82uhtG7yGjOPPq23jiynMI7xBDXnYmnXr04canX3Nw6e1Pclw/aUS7mdKiQt55/C4eeOtLOvfqB8BZ19zOg/+ZzsDRE+kxcGijjtMxqQdPf76Awwf2YjJWkNi9NwYPD0cWvVVJ8IWr+Pj5hxl18izOvta6oJzWmnceu5Pv3n6Ri+58tFHH8Pbx5drHXiI/O5OcjMPEJHYhICjEgaVuGcmfcFerfv+ZlL07mfPtQnz8rFO+blj6F/974GZe/nlZo+vJSaefx5hps0jdv5vAkDA6xCc6stgOITlumHTncDObViymx8BhVQ1ogOCwcCafcT6rfp/XpGMppYhP6k7nXv2kAS2EA5iMFWxcsYgZF19TtU0pxWmXXs/qP39p8vFCIzvQte8gl2xAH+02JfkT7mzVH/OYdsGVVQ1ogBPGnYhfQCB7t25s0rG8ff1I6jNQGtBtmDSi3YzWFgwGVWu7wcMDjUz/KYTL0YCqmVmDwYBM1yuE69FaH7eORepYcQxpRLuZAaMnsmP9P6Tu21W1raSogIU/fM7wydOO+7riwnxS9u6kvKy0NYrZ6uQumHBFXt4+9B85jt+++KBqm9aaXz55h2GTTznu68wmE6n7d5OXldEaxWwRyZ1oS4ZNPoVfv/gQk7GiatuW1Uspys+la99Bx31d1uEUDh/c5/YXx5LnppE+0W4mMDiUy+97ksevPJsRU2bgFxDIit9+YtRJp9Fr8Mha+5tNRj7572MsX/AjYVHR5Odkcup/ruL0K25GqdpX2+5IAi9c2cV3PcZT157Pro1rSOozkG3/LKO8rJQH3vqizv1X/PYTn734BD5+/hTl59LrhOFc/cgLBIWGtXLJGybZE23N2FPOYOOyhdw7+ySGn3gqORnpbFy+kFuefRMPz9pNpiOHDvDmw7eRkZqMt7cPXj4+XPngc/QeUrs+dnWS56aTRrQbGj3tdHoOGs7K33/GWFHOnS99UKOPdHVf/28OmYdTeHnuMgJDwshOT+OF268gNKIDk844v5VLbl8SeOEOOsQnMuebP1n953yOpBzglAuvZPD4qXh6edXad8/m9Xz6wuPc+fIHJPUZiLG8jC9fe47X77+R+974zAmlr5tkT7RVBg8Pbnrmf+xcv5pta1bQte9A/nP7QwSHRdTa12wy8dyNF3Hy+Zdz0rmXoAwGNiz9i5fvuoZnv/qNsKgYJ3yC5pFMN480ot1UREwcMy6+tt59Ks1mFv34Bc988RuBIda7WJGx8Vx0xyN88t9H3boRLYEX7sTb149xM85qcL+/vvuE0y65jqQ+A6ted8GtD3Dz9JGkHzpArAvMMSvZE22dUoreQ0Y2eDd588q/CYnswLTzL6/aNnj8FIafeCpL5n7LrMtvdHRR7UIy3XzSiG7DKsrLMJtMteZ+ju3UxS36WtZFwi7asrysDEaeNLPGNk8vLzrEJ5KfleG0RrTkToja8rMz68xkbKcksg6nOKFETSfZbhkZWNiG+QUEEhXXkS2rltTYvmbhr/QY1Lj5pF2FDHYQ7UH3AUNZs/DXGttyMtJJ3b+HTj36OKVMkjsh6tZ9wGA2rVhMRVlZ1TatNWsX/0b3gUOcWLLGkWy3nNyJdnGbVixm3sdvk3HoAAndejHz8hsavWqSUorzbrqPNx++jTOvvpWkPgPYunoZ8z99l/ve/NzBJbcPCblwJ1mHU/jh3VfYtmYFAcGhTDrjPE486z8YDI27X3HS7Et46OKZfPTcQ4yedjrZR1L57u2XmHnpdY1e3dBeJHuirTObjMz/9F2WL/gBY3k5g8adyBlX3lxn/+e6JHTrxYBRE3jmhguZddkN+Pj58/vX/4fZZGJ4PbPvuALJt33InWgX9s9f83n38buYfMb5PPju1wybPI2X7riKHetWNfoYg8dP4bb/vsP2tSv54On7OXLoAA+99w2de/Z1YMntQ0Iu3El+diaPXXEWYR1iuPd/n3LhbQ+ydO63fPHKU40+RnBYBI99+ANe3j58POdhlvz8DbNvuIeZl93gwJLXJtkT7cEbD97C9rUrufLB57njpfexVFby+BVnN2kq2KsfnsO46Wcx7+O3+PLVZ+jcsy/3v/k5nl7eDix588lTXfuSO9EuSmvNN2/8l+sef5m+w8cA1lH+nt4+fP/Oyzzw9peNPlaPgUMbvRy4K5CAC3f0xzcfM3j8FM657k7A2i/y7tf+j1tnjmPGxdcSEhHVqOOERnbgwtsedGRRj0uyJ9qLQ7t3sHvTWl76eRle3j4AXHbvk8y55TKWz/+BE8+6sFHHMXh4cOJZFzZ6f2eSfNufNKJdlKminIzUZPoMG11j+8DRE/nouYecVCrHk5ALd3Vgx5ZaFWlgSBide/bl0N6d9G9kI7q1SeZEe3Rg5xb6DB1d1YA+atCYiRzcudVJpXIcybljSCPaRXn5+BIQHMLhA3uJT+petf3Q7u1ExcY7sWT2J+EWbUFUbEeSd29nyISTqraZTUbSDuwhMrajE0t2fJI90V5FxSVwaM8OtNY1Fh5L3r2DqDjXzGtzSc4dx2GNaKWUL7AE8LG9z7da60eO2edSYA6QZtv0utb6PUeVyZ0opTjlgit446FbGTB6AqWFBQSGhrHq93nMuvwGDu3ewfJff8RYUc7gcSfSb8Q4u6xAWGk2s3bxb2xfu5LgsAjGzTiLDvGJdvhEdZNwuwbJa8tNOecinrx6NqYKI2UlRXj7+pKZeoikPgMJDA7ll0/fITMlmcTuvRlz6hn4+gfY5X0P7tzKyt9+xmQ0MmTCVPoMG93guUBy5/4ksy3Ta/AIDB4evPP4XfgHBVNpMuHj68eahQt4+vNf+eevBWxfu4LA4FDGzTiL6ITOdnnfwrwclsz9hqy0FDr36seoabPw9fO3y7HrIll3rAYHFiqlfJRSFyil7ldKPXz0qxHHrgAma60HAoOAaUqpumYu/0prPcj2JeGu5oTxU8hMO0T6wX3EdenGvq0bKS0uJDfrCM/ccCEeHh6ERUXz8ZxHefvRO9Bat+j9jBXlPHvDf5j/6btEd+xEaVEBD110GhuW/mWnT/QvGdzgOM3MrOS1hWI7JRGd0Jl1i38jKj4BY3k5W1Ytoeegodxz7hQO7d5BXJdubFq5mPvOn0Ze1pEWv+f8T9/l+ZsvwdPbm9DIKD545n4+eOb+es8FkjvXInWscxgMBkZOncHahb/i6elFWFQMq/6YS7f+J/D2Y3cw9//eoEN8ImWlxTx8ySzWLPq14YM2IHn3du45dypp+/cQ16Ubaxf/zoMXTqcgN9sOn6g2ybrjNeZO9E9AAbAOa2gbRVvP4sW2H71sXy1r5bUzn734BGddc1vVakgnn3cZX7z6DHM/fINnv/qdqLgE6/bZl/HgRTPYvPJvBo6e2Oz3W/TDl3h6eXHfm59XTck17MRTee3e63ll3so6lyluDgm2wzU5s5LXllv6y/d4+/jwyPvf4eFpPbWOP+1snrjqXM65/k5OvfAqwJrjz19+im/fepGrHnq+2e+Xk5HOD++9wrNf/UFEdCwAJ82+lPvOn8bO9atrrbYmuXNZUsc6QUFuNj9/9CbPfLGgqi6ddv7l3Hf+yfj4+vPUZ79g8PAAYNRJM3nh9isYNGZSrT7UTfHRsw8y+4a7mXj6eYD1XPDRcw/xw7uvcOk9T7T8Q9lI1ltPYxrRHbXW05pzcKWUB9YTQzfgf1rr1XXsdpZSajywG7hNa+0ey/w4mNlkZPvaldz+Ys0bByfNvpQ/vv6/qtAD+Pj5MWHmOaxf8meLGtEblv7JlHMuqjGnba8ThhMUGs7BnVvp1v+EZh8bJNitqFmZlby2zPolf3DiWf+pakADJPUZSERMHB2TetTYd+q5l/DwJTOPPUSTbF6xmBPGnljVgAbw9Q9g/Glns37Jn1WNaMmdy5M61gm2rFpKv+FjatWlk884n21rV1Y1oAG69T+BiOhY9m3dSK/BI5r1fiVFBSTv3s64GWfX2D713Et47saL7NKIlqy3vsbME71CKdW/OQfXWldqrQcBHYHhSql+x+wyF+istR4A/AH8X13HUUpdrZRaq5RaW5SX25yiuB2lDBg8DBjLy2psLy8prnPhhvLSUrx9mn+FDODl40N5ac35MbXWVJSV4tWCY0vXjVbXrMzaK6/QPjPr5e1Ta37Zo/lRx2S2vLS4RXe0/n2/klrby0tK8PLxkdy5D6ljncDbx4fy0tr5KS0prrVNa22tY319m/1+Hh6eaK0xGWs+bKgoK2lR/XqUZN05jtuIVkptUUptBsYC65VSu5RSm6ttbzStdT6wCJh2zPYcrfXRv6j3gDrXydRav6O1Hqq1HhoUFt6Ut3ZJlspKdm1cw+aVf9cZYgAPT0+Gnzid795+qap/o6Wykm/ffhGtNTvWrazaN+fIYRZ+/xmjp51OcWE+m5YvYt+2jQ32ka40m9mxbhVbVi2hoqyMMdNO55eP36K0qLBqnyVzv8HLx4fE7r2b/DkdVYkn51ew9nAxuWXmqm1mi2ZLRgmbj5RgqrTY/T3dgb0y29K82vZtU5lNP3SADcsWknX4+DfxxpxyBvM/eYfigryqbSt//xljeTkblvxZlcdKs5nv3nmZMaecjqWykp3rV1dlsCFpB/ayYdlCco4cZvD4Kexcv5rdm9ZV/T4z7RB/z/2GEw/PbcGnta/8cjPrDhezP6+8xjnpYH456w4Xk1ctx+2J1LGOU1JUwKbli9i7ZcNx68EBoydyYMeWOurSz0lP3kdxYX7V9hW//ohG07lXfw4f3FeVwYYUF+Sxcfki9m/fhI+fPwNGTeDH91+rKpPZZOK7t63ngpawZz17vLr0aI4PHJPj9q6+7hwzWnJgpVQUYNJa5yul/ICpwHPH7BOrtU63/TgT2NGS93QHybu38/KdV+Pj549/UDCp+3Zx0R2PMm7GWbX2veiOR5hz8yXcO3sqSX0GsmP9KmISunDj06/z0l3X0LXvIPwCAtm8cglnXX0rO9at4qlrz6dLr37kZqbj7ePLbS+8W+fsGnu3buDVe64nMCQUH18/0g7u5bJ7n6L3kFHcfvp4BoyeSHZ6KtmHU7nr1Y/sMvNHSxVWVDJneRqHi4x0DPZhT04ZU5JCGBIXwEurjhDp74kByCgxccuIWAbHBTq7yK2t2ZmVvNatoqyMNx66hd2b1tKpRx/279jM4HEncuWDz9caIzB4/BR2b1rL7WdMYNDoSeRmpnPk0AFuevYNvnztGe6dfRJJfQawfd0q4jp3ZdCYSdw2axz+QcH4+QeQemAPl93zJKNOrt3No6ykmNfuu4GDO7eS2L03+7dvYsSUGVz32Ev899ZL6d5/CN5+fmxZtZTzbrqHxAMft9Y/0XFprfl8Sza/7M6je4Qv6UUmwvw8uWF4NG+vzSSj2Ei8LccndQ3lkkFRLnGeaUVSxzrAb19+yDdvvkCXXv3Iy8rAw9OD2/77LjGJXWrs5+vnz03P/K9WXXrm1beQl5nB7aePZ9DoSeRkHCYjNZmbn32DF267ggM7NldlcNjkU7n8vqdqdOE6au5Hb/LTh/+jS+/+5BxJw9c/kCseeIb3nryHTSsW06lHH7avWUHn3v047ZLrWuufp16bM0p4cWV6jbr05hGx7MguY74tx4eLTET4eXLv2HhC/WSWZNXQFYVS6hOt9UUNbavjdQOwPjrywHrH+2ut9eNKqceBtVrrn5VSz2ANthnIBa7TWu+s77hJfQbopz6b39Dncklmk4nbZ43j3BvvYeypZwCQum8XT15zHve/+Xmdd3u11uxcv4ojhw6S0K0XXfsNQilFeWkJG5ctpKK8jAGjJpC2fw/vPnE3D737DZGx8Witmf/pOyxf8BNPffZLjcrJWF7GrTPHctm9TzJs8imAdZqsZ66/kMf/7ycsWrNr/WqCwsIZOHpis5YvdcQd6OeWpRHm58kVJ3TAw6Aoqqjk0cUppBRUcP/4jgyKsU4Ztj2rlKeXpPL6qUntIuSzvti5TmtdtSRlczLrqLyCe2f2/+Y8QkFOFtc9/lJVd42X77yaHgOHcubVt9b5mozUZHasXUlASGjVQCTr06NVZKQcJKF7LxK69eK2mWO59J4nGH7iqYD1Avvpa8/n0Q9/ILZTUo1jvvP4XVgqK7nywefw9PKirKSY/956GYMnTGXyGRewYdlfmI1GRm14w2X+5pcdKuSrrdk8MTmRUF9PLFrz2eZsFh8sYFRCEJcNsua40JbjU7uHMiUp1NnFdjh75NW2j9Sxx9ixbhVvPHQLD737DR3iE9Fa8/tXH7Hohy945svf6rxIO7YuDYuKBiAj5SA71q0iMDTMtsjZw5hNRq566PmqDL5w2xUMHDOJ0y65tsYxNy1fxEfPPcSD735DRHQsWmvmfvQGaxf/zqMf/sD2NSvISk+lU48+JPUZ0KLPbK+6tthYybVz93PnmLgadekTf6cS7uvJ01MSCbHl+JNNWSTnV/DwxIQGjto2HJvZ6hpztu1b/QfbQIbjPsY9Smu9Gag1Ek1r/XC17+8D7mtEGdqEbf8sIzw6tqoBDdCxa09OPOtCls77lgtvq70SoVKK3kNG0XvIqBrbff0DGHnSaVU/f/byk8y45FoibQuxKKU45cKr+P3rjzm0ZwedevSp2nfDsoUkdOtZ1YAG6NyrH2Onn8nSX77n7GtvJ/aYq/bGclS/rGJjJRuPlPDBrG54GKwnwiAfDy7sH8nr/xypCj1Anyh/hsUHsfRQIaf1dP9Hk83Q5MxKXmuzWCwsmfsNc779q6r/sq+fPxfc8gD/ve3y4zaiozt2IrpjpxrblFL0GTqKPkOtOV6zcAHxXbpXNaABOvXow/jTzmHpL99x7vV3VW03VpSz6ve5NWbI8QsIZPaN9/LuE3dzbvo3VKXORRrQAH/sK+C8fpGE+lrLZFCK8/pFMG93Lqd2C63KcbCPBxf0i+S7HTntohFdB6lj7eTvn79m+kXXVD19VUpx0uxL+f2rjziwYzNJfQbWes2xdelR0Qmdq+aGNpuMrPztJ16eu7xGBs+76V7eeuT2Wo3oRT99xczLb6wa9KuUYsbF1/LHNx+TfnAf/UaMtcvntWd9u+JQEQNi/GvVpUNiA/D2MBBSLcfn94/k8h/3kltmJtyFzjnOUF+f6PuUUkXAAKVUoe2rCMjEOiWPaKKSogJC6lj6NzSyAyWFBS06dmlRIaGRHWpsMxgMhEZE1Tp2aVEhIRE197WWI5qSouaXw5EDG8pMFrw9FL6eNe8khPlZr4yPFe7nSbGx0mHlcUWSWfuyVFZSUVZKcFhEje2hUR1alBOAkqJCQiLrOBdE1T4XmIwVaK0JCAqusT0sqgOlWWm4qhJjJWHHVLBeHgb8vQwYK2tmNkzyKnm1g5KiAkKPqWOVUoTUUQ82hcloxGKpJDA4tMb20Mi6zwWlRYW1ymHw8CA4LKLF546j7F3fFpsqCfOt3SCOCvDi2Bv43h4GArw9KGlnma3LcRvRWutntNZBwBytdbDtK0hrHWG7uhVN1HvIKLavXUF+dmbVNktlJcvn/0D/keNbdOx+w8eydN63NTr8px3YS3ryPpL61rz67jNsNBuXL6Qo/98BUGaTiRW//kj/EeOa9L5HBw86emRwpL8nAV4ebDxSc/aDhQcKMFk0ZaZ/B0BUmC0sO1RY44q6pSwW1x+sKJm1L08vL3oOGsbyBT/W2L5k7jdNzsmx+g4bzeYViymsNhOC2WRi+fwf6T+y5rEDgkKI79Kt1mIPS37+ikExjlvprKUGxgSw8EDNBsPO7DJKTZYag4IBFh0oYKCd8+rqmZW82l//EeNYOu+7GvXgkUMHSNm7k679mj9Fq19AIInderP6z19qbF/6y7d1ngv6jahdH6fs3Ul2ehqdex07gUrTOaK+HRQdwMrUolp16ZLkQgorajaWt2eVUmnRxAU1vavn8bh6Xo/nuPfhlVKDbd9+U+37Klrr9Q4rVRsVFhXN9Iuu4dHLz+CUC64iICiYxT99iY+fX42uFc0x+cwLWL7gB1647XLGnHIGORmHWfDZe5x/ywO1lhSN7tiJyWdcwKOXnc60C67Ax8+fhd9/TmRMHIPGTm70e7bmlDpKKa4c0oGXVh7mlO6hJIb4sPZwCVszS+kd6cdtvx7gzD4RGBT8sCOXhGBvekX6tfh9v9+ew487cymsqMTHU9G/gz/3jovHs45pBp1NMmt/F972IM/fdAlp+/fQrf9gtq9dwT9/zeeBt79q0XGj4hKYcs7FPHrZ6ZxywRX4+gey8PvPCIuKZvC4KbX2/88dj/DSHVdxcOdWuvTuz+alf7Jx0XyenRjdonI40qxe4dz7RzL/XZ7GqIQg0oqMzNuVx/QeYcxZcZgZPcLoHOrD8kNFbMsq5eVpzetCVt3+vHLmLE8jo9gEQGygF3ePiadTWPOnJnMUyav9TZg5m2W/fM+cWy5l7PQzycvKYMFn7zH7xnvwDwxq0bEvvP0hXrz9SpJ3byepzwC2rF7K+r//4OH3vq2175SzL+KJK8/mpTuvYtRJM8lOT2PB5+9x4e0P4e3T/L9FR9a5SeG+DIsL5N4/k5neIwyDgvm78+gV4ce2rBKeWZrK+E7BHCqo4OddedwwLKaqS1ZzGc0WnlmWxvasUirMmhBfD87uE+FW3TCPO7BQKbXI9q0vMBTYBChgANZBC6PqfKGDufOgh6O2rl7GsvnfUV5ayqCxkxl76hnNGrx3rPKyUv7+6Wu2r1lOYGg4k04/77gLpGit2bxiMct//QmTsYIh46cy6uSZdY4yPpYz56M8VFDBb3vzySo10SPcj0mdg7lx/n76dPDDy2BAAxYNm4+U8OHpXfH3bn5/rd/35vPBhkxuGRnLsPhAUgoqeGVVOv5eBp6e0qnhA7SSo4MeJLOOkXU4hT+/+YT0Q/vp2LUnU8+5iLCoGLsce9OKxSxf8CPGinIGj5/C6JNnHXdl0MKHJzB/fzEZ5YougZpTkoJdZhDh8ZQYK/l9Xz47sssI9/Pk5K6hvL8+g5wyM93CfSk3a/w9DaxKLeLywdGc3C202e9VajRzxc/7OalrKGf1iUBrzTfbclh4oIAPTu+Gr6drXPhKXh2roqyMJfO+YevqZQQEhzBx1mx6DKxzTFiTZaQc5M9vPyEjNZnE7n2YcvZ/anWjPKq8tITFP33FjrUrCQqLYNIZ59G176Bmv3dr1Ltaa1anFbP8UBEWrRmdEESF2cJba48wsmMwZWYLvp6KA3kVBHgZeO6kzi16v3v+OIjZAjePiCUuyJvVaUW8tjqda4fGMKlLiH0+lB3UN7CwMbNzfA88orXeYvu5H/Co1vrsel/oIO4ecHfnahO6f7Ipk3WHS3hpWucaI6/v+SOZLqE+XDus+Y2dq+fu4/SeYZza49+r4uxSE9fO2897M7tWDZhytjpG+0tm2xhXy11zFZabufynfbw5I4mogH8vFn7dk8f3O3J5Z2bXZh/7nXUZ7M0p4/lqFbvWmtt/O8jA6AAuPaHuxk5rk7yKpnJm/q+bt49Tu4fVuDucV2bmqp/38dZpXYj0b94NwOxSI9fOPWCtS6vdDPh5Zy4L9ubz5oykel7dulo6O0fPo+EG0FpvVUo1feUN4dZctRLfn1dB/2j/WlMXDYrxZ3NG6XFe1Tilxkr6RNXsChPp70WAl4GDeRUMinWNRnQdJLNtgKtmriUO5lfg72Wo0YAG6NPBn082Z7Xs2HnltfpVK6UYGB3AgfyK47zKJUhexXE5+zxQbLTQ+5h6MMzPkxBfD/blVjS7Eb0np4IQX49aT9N6R/nxzfacZpe3tTXm+dZmpdR7SqmJtq93gSatpiTcm7NDXJ+kMB82HSmttYLShiOldA5t2VKqAV4ebMuq2RDPKjFRYrSQFNbyZVodSDLr5lw5cy3RJcyHMrOFrBJTje3bMksJ8GpZd4ukcF82Hqm5AqzWmo1HSugqeRVuyBXOA4HeHmzPrFkP5paZKSivpFt48/t394jwpaC8staKpduzSgn0do2uV43RmFtplwHXAbfYfl4CvOmwEgmX4ArhbYzZfSP5dW8+b6/N4Nx+kRgUfL8jh5SCCh6dEN+iY5/TL4J312UQ7OPJ8PhAUguNvLo6nV6RvgTX0ZWj0qJZfqiIlalFKAVjEoIYlRCEofVXYZPMuil3yV1zBfl40jvSjyf+TuHmkXEkhnjzT1oxH23M5MrBLRskeUG/KK74eS/vrcvg7D4RaODrbdlklpg4r19kna/Zm1vO73vzyS030yvSj2ndQgn09mhROZpB8ipqcZVzwfn9IvjfmiOE+HowsqN1gPBrq9PpHu5LhH/d4zcaI8Lfi27hPjz+dwo3jYglPsiblalFfL4lmxuO0w2zoNzMr3vz2ZNTRmSAF6d0C6NTC2+WtVSDfaJdjfTXcjxXCW9jpRVW8OzSNA4XmwBNTIAXd46Jp4sdRuT/tDOHH3bkkl9Ria+HgQHR/tw9Nq7W7Bxaa15cmU56kZFTe4Rh0Zp5u/LoEeHH9cPtMwjteOrrr+UKJLMNc7fMtYTFYuG5ZYfZlFFKeaWFUB8PTu8dzum9Ihp+cQOS88p5fkUaR2yzc8QFeXP3mHgSQmpXtMsPFfLOugxm9gwnLsibValF7M4p47mpnQj2cVxXLcmraIirnQ/m7crl2+055JdX4uup6GenWarMFgvPLE1jW2Yp5WZNqG12jhl1zM6RXWri3j+SGRgTwLD4QA7lVzBvdx63jIxlSFxgi8rRkGb1iVZKfa21PlcptQWo1dLWWrdsrUrhclwtuI0VH+zDa9MdMwhhVq8IZjWict+eVcbe3DJeOaUL3h7WE8uYhGBumL+f/bnlJLXgsVdjSWbdj7tmriUMBgP3je/okGN3CvPlf9MbHpxotmjeW5/JA+M70iPCOhXmqIQg/vdPOj/tzOOigbUXwrE3yauoiyueE2b0DK+zYdtSngYDD01o3NLh327PYWxicNUA4ZEdg+gR4cfb6zIYHBtQ55LuraG+y+2jj5ZmtEZBhHO5YnDdyaaMEkYnBFc1oAH8vAyM6hjExoySVmlEI5l1G5I350orNOLraahqQB81sXMIH27M5CIc34hG8iqqkXNC/TYfKeXusXE1tg2M8afUVEl2qbnWYOXWctxGtNY63fbtFGCJ1npP6xRJtBYJrf0EeXuwP6/2DAC5ZeYWD3BsLMmse5DcOV+At4GiikpMlRa8ql345paZCWqlPtGSV3GUnBMaFujjYatP/91WZrZgrNT4tXBQcks0puNXIvC2UqozsA7roIelWuuNDiyXcCAJbNPszytnT045EX6enBAbUOcqTeMSg/lq6wG2ZpbSr4N1OqD16cVszSzlRgf3ia6DZNYFSe5aR0G5mXWHS0DB0LhAgn1qN4oj/b3oHuHLF1uyuXBAFB4GRV6Zma+2ZnN+/7oHITqQ5LUda+/nBbNFsz69mLyySnpG+tI5tO6ntlOTQvh0czbdwv0I9vGg0qL5eGMWg2MDnDEYuEqDjWit9SMASik/4CrgLuBlwHmlFs3W3gPbFJUWzcur0tmeVcqgmABSCo28vyGTRyd2JDqw5tyYoX6eTO4SzGOLU4gJ9KJSQ3aJiTN7hxPQygGXzLoeyV3rWHiggPfWZTAoJgAL8N66DK4dFsP4TsG19j23bwRPLknlz/0FRAd6cTC/goRgH0bEO3aQ0rEkr+1Xez8vpBcZeXRxCuF+nsQFefPV1mwGRPtz04jYWjerJnQK5ueduVzx0166hPpwuMiIUoonJjWuT7WjNNiIVko9CIwBAoENwJ3AUgeXS9hZew9rc8zfk0demZm3ZiRVPfL9YUcOr64+wlMnJtbYN6PYyMIDhcw5qRPFRgsK8PVUPLwohZO7hRHWisszS2Zdg2SudWWWmPhgfQbPn9SJjsHWLlTJ+RXc/1cyfaP8ak3H9cGGTC7oH0W/Dv7klplJCPZmzorDLDxQyEktWH68qSSv7ZOcH+DlVenM6PHvaogVZguPLk7h9335nNI9rMa+v+zJI9zfi4cnJpBaaCQqwJOVKUV8uDGLx5zYkG5MR5IzgQjgT+B74KdqfbmEaLOWJBdyTt+IGn0mT+sZTnJ+ObnHTBC/7FARYzsF0TnUl34d/OnbwZ+u4X4MiwtkZWpRaxddMivaneWHChmTGFzVgAboFOrDiPigWhk8Umwkq8TEtG6hdAr14YTYACIDvDirTzh/Jxe2dtElr6LdySg2WqeErdZY9vE0cFafCJbWkcElyYWc2zeCqAAvTogNoGOwD6f3Cmd3ThmFFeZa+7eWBhvRWuvBWAc+/ANMBbYopZY5umDCfuSKt3lMlRofz5oRMSjwNChMlZaa+1o0Ph614+TjacBU2bpzsUtmnU8y1/qsGaw9XsHHU9XKoKlS4+Vh4NjhDT4eBkwWyatwLDk/gNkCXh6qjgwqjHVksO76WOGhaue7NTXYiFZK9QMuBC4BZgNpwEIHl6vNKykqID87s9Zy1fYSMGdq1Vd9iioqyS9r3FVchdlCdqmJylauZByhsNzM3pwyjGbLcfcZ3jGQX3bn1fh/tDKliBBfTzoEeFFutpBj+/cYER/IkuRCiioqq/bNLzOzIqWIYa3cx1Iya38mYwU5GemYTcZ692tM5lrCWGnNoLkRGbRoTU6piTLT8f/G3YXRbGFvThn55cc/Vw2PD2TJoSIKq2ew3MzyQ9YMVlo02aUmKswW4oO98TRgHYBoY9Ga+XvyGC55dXtaa/KyjlBa3OpPAWuwWCzkZqZTVlIMNHx+KKqorPdv3J2kFVaQVlh7xqqj4oK88PU08E9acdU2rTXz9+RXZbCg3FyV5+HxtevjZYcK6RDo1aKVE1uqMR01n8U6WvhVYI3W2tSYAyulfG2v87G9z7dHB1BU28cH+BgYAuQAs7XWBxtdejeUn53J+0/fx/Y1K/Dw9CQytiOX3vMEPQbabwGrxlTi2aUm3lhzhO2ZZXgYIDbIm2uHxtCtjvmMKy2aTzdn8fu+/KquDbP7RtTqs+QOSo1mHliYQkqBES8PhdaaCZ2CuW54bK19T+8VzkMLU3hwYQrD4wNJKazgn9Ri7h4bx9trM1iSXIiXh8Lbw8BFA6OY3CWE2349wJSkUCq15q/9BZzWM4y4IO86SuJQklk7sVgs/PDuK/z21Yd4enphsVg47dLrOPXCq2pM7u/oO0uVFs1nm7P4rVoGz+0bUeNRaHWrU4v4cEMmpSYLJotmVMcgrhoS7dSpoJrrrTVHWHywAGW745QQ4s0TkxMI9K5ZfXUO9WVK0tEMhqCBP/cXML1HGLtzynlkUQqVFo2xUjOpSwjXDI3hhRWHGZUQRFygF6vTirFomNGj1c9rklc72rzybz7572MU5uVgNhk5YdyJXHbfUwQEhbRqOdb9/QefP/8AJYX5mI0VjOwYyFVDovH3qj1eNLPExBv/HGFndhkGBR2Dvbl2WAxJdlh1t7WtO1zEa6uPUGK7eA/wMnDTiBiGxAXV2E8pxfXDYnh2WRrrDpcQH2xdNbTSojmzdzj3/ZlMcn4FGugW7sslg6L43z9HeHhRCkPjAknOr2Dt4WIemuCYhZsay2HLfitrDROgtS5WSnkBy4BbtNarqu1zPTBAa32tUuo84Ayt9ez6juvOS5JqrXno4tPoO2wMZ1x1C94+vvzz13w+evZBnvpsPhExcQ0fpB6NrcgrLZpbfz3AmMRgzugVjqdBsTS5kA82ZPLqqV0I9a1ZOX2yKYvdOWXcMjKWSH8vDuaV8+yyNP4zMIqxibVHvbuyWxccINTXk1tHxRLq68nunDKe+DuV03uGc1bf2isTmio1K1OK2JVTRoS/J5M7h/DJ5ixKjJVcNyyGEF9PdmaX8fyyNG4dFYuPh4GVKUUoBWMSg+u8KLE3ey0jLJmt7eeP3mDtot+46ZnXiYpL4PDBfbxy9zVMu+BKJp1+HtA6j2Y/25zF9qwybhtlzWByfgXPLkvj/P6RtWae2JNTxpNLUrlzdBz9OvhTbLTw3voMTBbN3WPiHV5We/puew4/7MjhoQkJ9Iz0I7/czKur0skpM/PKKV3qfM2u7LKqDI5OCKKwopI31xzhnrHxdI/wI6/MzP/+SadDoDdn9Q5n0cFC8srM9I7yY2THIDzrmMLSniSvjpO6fzdPXnUu1z7+EgNHT6SspJgvXn2a7MOp3PP6J61Wjv3bNzPn2nO4c1gE/aP9KTFZeH99BmVmzb1ja2aw0qK5cf4BJncJZlavcDyUYvHBAj7ZlMVrpyYRVMcUja6qsNzMVXP3cdGAKKbZLvAX7Mnj081ZvHtaV4J9a9+3zS0zs+hAAbllZnpF+jEwxp9bFhzk3L4RTO0aitbWQf7zdufxyrRO/JNWwp7ccqL8PZnYJaRWe8UR6susw25LaKuj9+m9bF/HtthnAf9n+/5b4ETlrLUbW8GujWuoKCvjvJvuxdfPH4PBwMipMxh18kwW/fhli47dlIp8U0YJ3h4GzusXiY+nAQ+DYmKXEIbGBbLoQEGNfU2Vmt/25nHTCGvlDdA5zJcrBkfz8668FpW5tWWVGEktNHL76Liq4PWI8OPyEzqwYG/dn8XLQzG+czBXDYnmzN4ReBgUK1OKuHFELCG2Y/SK9OP8/pHM251Hz0g/Lj2hA5cM6tAqDWh7kszWpLXm18/f5+pH5hAVZx39Hde5K5fe8yS/vvmUw7tuHGW2aBbsyefmETFVGewU6sOVgzswd1durf1/2Z3Hmb0j6B9tXQo3yMeD64fFsPlIKdmljbrJ6TJ+3ZvH5SdE0zPSurJgqK8nt42KI63QSEZx3V1rqmewe4Qfc3dZl/HubludMMzPkxtHxLL4QAF+XgbO7hPBVUOiGZsY7PAGtD1JXmv785tPmHruJQwaMwmlFP6BQVxy1+Mc2rODtAN7W60cf3z+Dmd0D2JAjDWDgd4eXDs0hm2ZpWSV1MzguvRign08OKdvJN4e1vr4xKRQBkQHsPhgwXHewTV9sjmLHhF+zOhpvTnnaVCc1jOc7uF+fLo5q87XhPt5cpYtg+M6BbMqtZgeEb6c0j0MT4PCy0Mxq1c4cUFerEsvZWKXEK4aEs3pvSNapQHdEIc+21NKeSilNgKZwB9a69XH7BIPpABorc1AAdZRysce52ql1Fql1NqivNqVhrvITk8joVvPWmu8J3bvTXZ6arOO2ZyKPKuk7lX0Oof61Ap4qakSDXQ4ZknNuvZ1dSkFRgK8DbUWX+gc6kN5PX2jq8srNxPq61lrcnd3/Peoi2T2X5VmE4V5OcR36V5je2KP3mQVlrZaOcpMFiq1rjODmXX8zWWVmmrl28fTQEygF9ml7tXfstyk6XTMZwny8SDQ20BKPf0tq7P+e9S8oA319cTfy1Cj/7Q7krzWlJ2eSmL3XjW2eXp5Edele7Pr2GaVI/UgnUNqduPz8TQQG+hF1jEXsk2pj11depGJ7nXcPOoe4Ut6ceM+S1ZJ7fMXWLtr1XW+czaHNqK11pVa60FAR2C4bQBFc47zjtZ6qNZ6aFBYuF3L2JqS+gxgx7pVGMvLamzfuGwhXfsObNKxWnIXrFu4L5uOlNQY0aq1Zl16Md1sd2uOCvLxIMDLg13ZNcu89nBxnWFxZb0ifSkzWUg9pvJdm1ZcdVe5IdEBXhRWVJJeVPMu2LrDJXQP9zvOq9yHZPZfnl7exHfpzpZVS2ps37hsId2igo7zKvsLtF347Twmg9a/udoZ7Bbux7rDxTW25ZSaOFxkpGNwq/fPb5FQPw/WHvNZUgsrKDFZ6BXZuLx1C/et9e9xqKACk0UT4ee8AUn2IHmtKanvQDYuX1RjW3FBHgd2bKFTjz6tVo6ephTWpdf8m8srM5NaaCQxpGYDsVu4LxuPlNQYLGytj0tq1ceubkC0P6tSi7FU6yZs0ZpVqUX0j/Zv1DG6hfuyLr2kxjEqbasaumKb47gtB6XUXGo/GqqitZ7Z2DfRWucrpRYB04Ct1X6VBiQAqUopTyAE6+AHp9u/fRPzPn6b9IP7iOvSjekXXUNSnwEtOmZc564MHD2B52++lLOuuZWAoFD++v4zUg/s4drHX2rUMezx+LhruC/dIvx4emkq5/aNwMfTwPw9eeSXVzI2sWbjwKAUFwyI5LllaVw+uANdQn3ZkF7Ml1uzeXB8R+btzmVpsnUwwPD4QGb2CsfX0zUHL/l7ezI0LpCHF6Zw9dBoOgZ7szq1mG+253D32Mb1R/fxNHCObaWziwdGEW87xvw9eTw7tZODP0H92nNmC/Nymf/pO2xeuQT/wEDGn3YO42acXeupT1Odc/2dvPXI7VxwywN07X8CO9au5OtXnuCe4a3X0FBKcWH/KP674jCXDupAUpi10v1ya3adg2pm9Ajjzt8P4utpYFynYDJLTHy0MZOZPcM5kFfO3F15ZJaYSArz5cw+4TXmVXY1lw6K4rllh/E0wIiOQaQVGnlnXQZD4wJqDSw8nrP6RPDgX4fwMCiGxgVyqKCC/9uYyXn9IvGqY0q81tKe82qxWFgy9xuWzvuWspISBo6ewPSLriYwpGWDOqecfREPXjidL159hnHTzyQvO5Ov//c8E2edS2hkBzuVvn4Bc6baMpiMv5eBcYnBZJWY+HhTFtN7hNV6itkjwpeOwd48+XcKs/tF4uNh4KdduZQYK+kR7ss76zLYnllKiK8np3QPZWTH1ruAb6oz+4Tzy+48nl2Wxjl9rA88vtmWQ6nJwpm9G3fOHBoXyPc7cnlpZToze4Zh0fD9jhzCfD0b3RBvTccdWKiUmlDfC7XWf9d7YKWiAJMt3H7A78BzWut51fa5AehfbdDDmVrrc+s7bmsMeti+diWv3nMdZ1x1Cz0GDmX3prX88O4r3PL8W/QeMrJFx640m/ntyw9ZNv97KspKGTRmEjMvv5GQ8MgGX2vP/pemSs3cXbksSS7EWGltAJ/VJ6LOQQwrUgr53z9HiPTzosRUSZCPB5nFJnpG+mGs1JzVJwJvD8W83Xnklpl56sREl+5b+MH6jKrPHeLrwaWDOjCiiSempcmFzN+TVzUY4py+EU5rjBwd9NBeM1taXMTDF8+k1+ARjD/tHIryc/ju7ZfpO2wUF972UIuOHTBnKpuOlPDD/nIOF1aQGOLD2d39G30X1J7WHS7mp525ZJSY6BLqw9l9I+vsd59bZub2X60DaIuNlfh6Gig2VtI/2p+tmWVcOCCSLqG+rE8vZu6uPJ6YnFiry4QrWZNWxAcbMskvr8THQzGuUzBXDI5u0jGS8yv4dnsOu3PKiPT3ZEaPcEYlOKcx0t7zCvB/cx5hz+Z1nHHlLQSFhrP4py/Zu2U9j//fz/j6B7To2DlHDvPjB6+xdfUyAoJDmHT6+Uw643wMBsff3KleR2cUG/l6Ww7bMksJ9vHg5G6hTO4SUueF/XPLUjmQZ31CatHg7WHtT1xYUcn4TsGMSQziSLGJz7dkM61bKLN6ue7TgvxyM3OWp3EwvwKFdfzGXWPim9R/udRUyQ87clmRUoTBNkj/jF7hteaJbi31DSx05OwcA7AOaPDA2m3ka63140qpx4G1WuufbVP0fAKcAOQC52mt99d33NYI+KOXn8m08y9n5NQZVdtW/PYTf379MQ+//51D37suzpyYXWvN9b/s54bhsfTr8O9V4IcbMlh8sJD3ZnatmnLLojX3/XmImT3DGONms3a4MzuO9nfLzC747D12b17HLc+9WbWtuCCP22aN47mv/yC8Q+3pCxvizoshfLwxk3KzhauHxlRtO1Js5JYFB3hkYgJ9ov7N8U87c9mdU8ZdbjZrhztr73nNTk/j/gum8fLPy/EP+reeePGOK+k3fCwnzb7UYe/tKC05X+zNLee5Zam8MT2pRl169c/7GB4fxNVD/71gzCg2cvtvB3lvZje3nK7SXdWX2QYvDZRS3YFngD5A1W0PrXVSfa/TWm/GGtxjtz9c7fty4JyGytCatNbs2byOB9+uOVvG0Ikn8+ZDt7Z6eZxdmRcZKykor6RvVM07b/5eHgyODaixJLZBKYbHB7Iru0wa0U7U3jK7Z8t6hkw4qca2wJAwegwYyv7tm5vUiHZ23uxhV0455xwzXePRLla9j7mDPqJjYJ0zfIjW097yum/bRnoNHlGjAQ0wZMJJbPtnuds1olt6ztiVXcYJsYG16lJvDwMjE2ou/BMd6E2HAC9SCyuqZpsRztWY++sfAo8ALwGTgMtw8IBEZ1JKERYVzeED+0js0btq++GD+wiLiqnnlfbjShW5n63yzS0z11gVyGzRHCqoPcVUaqGxzpG1olW1q8yGRUWTnryvxjaLxcLh5H2Ed2hcZl0pcy0V7udJWqGRQTH/Phb3NEClBfLKKwn3+/e0n1poJNyJq30JoD3m9eB+tNY1ujZY69imddNxNnucN8L9PFmRUljn71ILKhgQ/W+OK8wWskrMLjG1m7BqTFD9tNZ/Ye36kay1fhSY7thiOdfJsy/jg2fuJy/rCAB5WUf48JkHOGn2JQ5/b1erzL08DExJCuGNNUeqpoNKLzKyMqWQnFIT83bnUmnRaK1ZfqiQdYeLmdDZNe5CG80Wvticxb1/JDNnedpx55VtjmJjJfNtk8ivTSt2taXQ21VmJ59xAX99+ynb1qxAa42xopxv3/wvQSFhdOnd8GBgV8tcS53aPZRvt+ewJ8c6m0e52cInm7LpEODJG2uOVC1Nn15k5MMNmUzvHurE0ta0Jq2IhxYe4sG/DrE61X5LNldaNKtTi/hkUxYL9uRRYnSpqe3aVV67DxiCl48v3739EiZjBVprtqxeyt8/fcXEM853dvEazV7njaFxgWSVmGvVpfnlZr7Znsu+3HLAOtXle+sz6NvBj6gA17jwzSg28t/ladz7RzKfbs7E2MipYhsjOb+Cr7Zm8/W27FozarmSxlzOVCilDMAepdSNWEf7BjbwGrc2/eJrKCkq4K6zpxASHkFBbg5Tz7mIU/9ztcPe05Ur8osGduD9DRlcO3cfIb4eFFVYOLNPOCPiA3l19RG+3pqDh0ER6G3ggfEdXeIquajCzE3zDxDg5cG4TsEczC/nxvkHuHZoNCcmhbbo2HtsqxwOjAkgPsibz7dk8dOuXB4c39FpAx+O0a4yG5/UnWsee5F3Hr8LbbFQXlpM176DuO2Fd+udncOVM9cSvaP8uXRQFM8sTcPLQ1FUUcmgmACeOjGRL7fmcE21HJ/TN4IJnVt3OeTjeervFLZkljElKQSl4OVV6fSOzOfhiQktOm6ZycLjf6dgsmiGxgWyJbOUr7Zm88jEBLq4xrLK7SqvSinufPl93n70Dm44eSi+fgF4eHpy3RMvE5tY9yqUrsae5w4vD8UjExN4dXV6VV0a5O3BIxMTSC8y8uSSVLyP5jg2gJtHNH2MhyMsOVjA6/8cYVBMAINiA1iWXMgfewt47dQuda5M2BTfbMtm3u48JnYOoVJr7v/zEOf0jeC0nq43oLLBgYVKqWHADiAUeALrFDnPV19atDW15pKk5aUl5Bw5TERMXItHDNfHXSrzYmMl+eVmovy9ajQWs0pMmC2amECvFk8pZi+PL07BojUPTUjAwzZTyJLkQt5ee4RPzujW7JHaWmtuWXCQc/pGMM623HKlRfPMsjT6RPlxZu/aS4c72rGDHtprZi0WCxkpB/H1D6j3sbC75K2lKi2a9GIjQd4eNeZBP16OnWnTkRKeXmodXHW021humZnr5+3nzjGxDI1r/kwaX27N5lB+BXeOicNgOz/9tT+fBXvz+e9Jne1R/CaRvP4rL+sI5aWlRCd0bpXZM+zBkeePuupSs0VzpI4cO5PFYuHiH/ZyxeBoJnWxXoRbtOapJamYLZrHJiU2+9iHCip4aOEhXp7WhTBb17OsEhO3/XqQF07uRHRg689136KBhVrrNQC2K+Wbtdb2e8bm4nz9A4hP6t7wjs3kbpV5oLdHrTkuAZd5tFTdvrxy7hgVV9WABhibGMTba4+wJbOMgTHNuyg6Umyi2FhzPm0Pg2JmjzA+3ZzllEb0sdprZg0GA7Gdjj8Wy93y1lIeBlXntIvHy7Ezzd2Vy7hOwTXGXYT7eTKxSzDzduW3qBG9KqWIa4fFVDWgASZ2DuHDDVnklJpqvKcztNe8Aq02zsheHH0Oqasu9TxOjp1pV045lZoaXTcNSnFm7wieXZbWomOvSiliQqfgqgY0WP9dRicEsTq1mJkuNr1fg5d+SqmhSqktwGZgi1Jqk1JqiOOL1ra1twq9tSmg8piHLFpbv1py882grFfcxz6/MWtdo5J2JslsbZI312ZQqs5xBZUWTUtvUBoUtY6tAY1rZFby6h7kHPIvD4Oy1oPHRLZSa1qaKINStepusN6Rd8WHFY0p0gfA9VrrzlrrzsANWEcTC+Gyekb68fXW7BpLm/+1Px8Pg6Jvh+Z3zekQ4EWEvxd/7S+o2maq1PywI5exnVxmJSnJrHArZ/YOZ/mhItKL/h38m1FsZMnBQs5o4Z2nMYnBfL8jp0ZD+te9eSSG+NS42+VEklfhVnpE+OHjYeC3vXlV28wWzddbc+gR0bJxBqMTg1iSXFhjIoC0QiOr04oY5YKrNTbmDFKptV569Aet9TKllNmBZWqz5Eq29dw2Ko6b5u/n6rn7GNkxkOT8CvbmlnPH6MYt7308SiluGRnLo4tTWJFSRHyQN2sPF9Mp1Idp3Vq2ZK0dSWZtJHPuoVeUP6MTgrh5wQFGdgzEoBQrU4oYER9I/+iWjUeZ2TOMbVml3DT/AIPjAkgpqOBwkZFHJza/36adSV5dmJxD6nbzyBjmLD/M0kNFdA71YXVqMQYFr55S7/TmDYoL8ua8fpHc8dtBRnQMwqI1/6QVc/kJHZze9aoujWlE/62Uehv4AutTsNnAYqXUYACt9XoHlq/NkCC2Ll9PA2/PSOL3fQVVjdz7xsUT5NPyO0+JIT68OT2J1anF5JaZuGVkLL0i/VxmUCWSWUAy525uHRXHjB5lfL8jFzOaJyYn0tMOy6t7eRh4aHxHtmeVsSenjJ4RfozoGIi3h8s8G5a8uig5hxzf0Lgg3p/Vlc+35JBZbOTsvuFM6xpqlwGi03uEMTw+kNVpRSgU/xkQ5ZINaGhcI3qg7b+PHLP9BKyBn2zXErUxEkLnMRgMTOsexrTu9r9D7ONpYLyLzIddh3abWcmbe+sW4cfdY+2/BLlSir4d/Onbwb/hnVtfu82rK5NzScMCvT25eohjFsiJCvBiRg/XGkRYl8bMzjGpNQrSFkkIhTO0x8xK1oS7ao95dXVyPhGN1WAjWikVDTwNxGmtT1FK9QFGaa3fd3jp3JCETzhbe8usZE64s/aWV1cn5xPRFI3pzvER1pHCD9h+3g18BUjAjyHhszqYV87nW7LZnlVKiK8n07qFMr1HmEtMJ9VOfEQ7yKzkzT5KTZV8tTWHpYcKqbRoRnQM4oL+kS6x8mg78RHtIK/uwF3OKQsPFPDzzlyySk10DfPlvP6R9Ilyya5KbV5jeoBHaq2/BiwAWmszUOnQUrmZgDlT3SZ8jna4yMjDi1IYEOPP66cmcdOIWJYmF/LJpixnF609afOZlbzZh7atMpZXZubxSQk8P7UT3gbFg38dwlhpcXbx2os2n1d34C7nlF925/Hd9hwuO6EDb0xPYmKXEJ5dmsau7DJnF61dasythhKlVATWAQ4opUYCBfW/pP1wl+C1lp935TKte2jVgIBQP0/uH9+R6+ft56w+ES63Ulob1SYzK1mzv62ZpRRWVPLE5MSqJ0VXDonmkUWHWHGoiIm2JX2FQ7XJvLoDdzunmC2ab7Zl8/jkRBJDrKsYTu4SgqlS8932HO4f39HJJWx/GtOIvh34GeiqlFoORAFnO7RUbsDdwtdakvMruHBAVI1tob6exAR5c7jISI+Ilk9ZJRrU5jIreXOMg/kV9O/gX6ur1YDoAA7kVzDROcVqb9pcXt2BO55TCsrNaKhqQB81INqfb7fnOKdQ7VxjZudYr5SaAPTEupryLq21yeElc2HuGL7WEhPozZ6cMvpVm0qq1FRJRrGRDi46z2Nb05YyK1lzrNggb5YkF9bavie3nBNiWrbIiWictpRXd+Gu55VgHw8qLZrMEhMdAv6tT/fklhMXJPWrMxy3Ea2UGgakaK2PaK3NSqkhwFlAslLqUa11bn0HVkolAB8D0VgfU72jtX7lmH0mAj8BB2ybvtdaP97cD+NI7hq61nZazzAeXZRCbJA3w+ICyS0z8/a6DEZ2DCLUQUvsaq3ZnFHK8kNFWNCM6hjE4NgAuyx+UmaysPBAAbtzyoj092Jq1xBiAr3tUGr7a0lmXTGvkjnHOyEmgI83ZvF/GzM5u08EngbFgj157Mkp4+YRMQ573yPFRv7YV0B2qYkeEX5M6hKMv1fLu3pprdmQXsKK1CIUMCYhmIEx/q60EFIVqWOdw53PK14eBqb3COPFFYe5cUQM8UHebM0s5aMNmdzowLyWmy0sOlDAzuwywv08OalrKLFB9qkHDxcZ+WNfPrllZnpF+jGpSwi+ni6zEFKD6ivp24ARQCk1HngWa2ALgHcacWwzcIfWug8wErjBNnXPsZZqrQfZvlwy3O4cutaWFObLHaPj+HpbDrO/2c0tCw4QH+TNtUMdMyE7wEcbs3hr7RHig73pFOLDBxsyeXttRouPW1hRyV2/H2RzRgn9o/0xWzR3/Z7MpiMldii1Q7Qksy6R16ODdCVzrcPDoHhsUgIZJSYu/XEvF363h82ZpTw5OdEujdq6bM4o4a7fkzFbNP2j/at+Lqxo+UrX76zL4P0NmXQK8SEh2Ie312Xw4YZMO5TaIaSObWVt4bwyu18kg+MCuP/PQ5zz9W7eWpvBFYM7MDg20CHvV2ys5O7fk1mXXkJ/2xPmu/9IZn16cYuPvT69mHv+SAagf7Q/69JLuPv3ZIqN7jOutr5bgx7VroRnY73K/Q74Tim1saEDa63TgXTb90VKqR1APLC9ZUVuPW0hcM4wMCaAF2MCqDBb8DQoPAyOuwuUnF/B3wcLeH16UtWgxROTQrhlwQH25JTRvQV9sH/YkUPvKD9uGB5btW1AtD9vr83gf9O7uOLdrWZn1tl5law5T5ifJ3ePicds0Wit8XLgcthaa95em8EtI2IZGm+t9KckhfLGmiP8sCOXSwZ1aPax9+SUsSatmFdP7VJ1ATA5KYSb5h9gclIInUN97fIZ7Kjd17Gtqa2cYwxKcW7fSM7uE4GxUuPjoRxaF/20M5eu4T7cPCK26n0GxgTw5pojvDkjoNlT11q05q01Gdw5Oo6Btq5jJ3YJ4dXV6fy8K5cL+kc1cATXUN/Z0kMpdbSRfSKwsNrvmvRcXinVGesSpqvr+PUopdQmpdQCpVTfphzXkdpK4JzJx9Pg0AY0WK9kRycE1Zj1w9/Lg7GJwaw73LI7xuvTS5jaNbTGtsGxAZSZLWSUuGSXRbtktrXzKllzDZ4G5dAGNEBmiYkSk4UhcTX7W09NCrFLXscm1uwWEujtweiEINa38NgO0q7r2NbUFs8xBqXw9TQ4/GbO+vQSTuoaWuN9Bkb7Y9HWrhjNdfS1A6L/HT+llGJq11BXzWud6gvqF8DfSqlsoAxYCqCU6kYTpt9RSgUC3wG3aq2PHcGyHuiktS5WSp0K/Ah0r+MYVwNXA0TGxDf2rZusLQatrfP1NFBkrD2fbZGxktDglvXB9vU0UFxR87GSyaKpMFvwdXBjo5lanFl75NV2nAYzK3lrf3w9DRjNFusdNM9/K+ViowVfr5Zlys/TQGYdF7fFxko6BrvkOIZ2V8e2NjnHtJyvp4HCY+pBs8U6XqglfZd9PQ2UmS1Uaqh2KqCoorJt9InWWj8F3IF1NaWxWmtd7TU3NebgSikvrOH+TGv9fR3vUai1LrZ9Px/wUkpF1rHfO1rroVrroUFh4Y156yaTsP0rp9TEskOFbM4oodKiG36BE41JDGJDejE7q000vy+3nFUpRYxNDGrRsSd3CeHLrdmUmqwnEK0132zLoWekn8MGSbZESzNrr7zafl9vZiVv9mOstLAmrZiVKUUu35cwxNeTXlF+fL0th6N/nqWmSr7Yms2kzsEtOvaYxCBWpRaxJ+ffc8Gu7DLWHbY+rXI17a2ObW2ueo7RWrM7p4ylyYWkFlY4uzgNmtwlmK+35VBi/Lce/H5HDl3CfIhswYxbkf5edA714fvt/54LSoyVfLUth8luND99vS0BrfWqOrbtbsyBlfXe//vADq31i8fZJwbI0FprpdRwrCePVp3s0FWD5gxaaz7fks38PXn07eBPVomJCrPmwQkdibPTSFx7C/bx5LZRcTz5dypdwnwwKGsj+obhsUS0cEq9qV1DOJBfztU/76dfBz9SCo34eBp4YJzr3qlpbmZbI6+SNfvbklHCnOWHSQjxxsfDwGur07lqSDSTXLgSunF4LE8tSWVVahEJIT5syyxldEIQJ3cLbdFxI/y9uGlELI8tTiEpzBcNHMir4NaRcYS46BLm7aGOdQZXPdcUVlTyzFLrCqFdwnx5d10GJ8QGcNOIWDwd3PWxuSZ3CWF/XgXXzN1Hvw7+pBUZ8VCKBye0fGGXW0bG8uTfqSw7VERckBdbM0uZ2DmESV1adkHdmhx5ZhkDXARsqTZI4n4gEUBr/RbWCeWvU0qZsT7OOq/a1bhoZf+kFbMipYg3ZyQR7GP90/hldx5zlqfx4smdXXEgHQBD4gJ5f1ZXNmWUoLV1oQi/Fj4aBmufs2uHxnBGr3D25JYT4edJr0g/l/13aCHJq5spM1l4fvlh7hodxwDbwJyUggru/+sQPSL8iHfNLgyE+3ny35M6sTO7jJwyM5cOirLbtJEjOwYxMDqAzRnWPpWDYgLwcaNHw00kmXUz767LoEuYL0+d2AGDUlSYLTy1JJWfd+ZyZp8IZxevTkoprhoSzaxe4ezOsU5x19tO9WCkvxcvTevMjuwycsvMXD44usb81+7AYY1orfUyrBPH17fP68DrjipDQ1z1atVZFh4o4KzeEVUNaIBTuofy485cDhUY6RTqU8+rncvH08DweMc8so0O9CbaReeGthdH51WyZn9r0orpGeFb1YAGSAjxYXKXEJYkF3C+C49uV0rRO8q/4R2bwc/LwIiOrtd9w97coY51Blc915SbLfyTVswHs7pWzWjh42ngwgFRvLY63WUb0Ud1CPBySANXKUUfB50LWoNrPuNyMFcNmbOVmy0E+tS8a2NQikBv6wAAIZrDkLEH6OzsYrQ55WZLjVlpjgr0NlBU4dp9o4VwBFeu202VGgW1Bs0F+XhQLvWr22qzz7mOx5VD5myDYwP5c18B1Z/27cmxPmbpGua6d6GFaI8Gxviz9nAx+eX/LlJSYbaw+GAhQ+Ics/CCEK7K1ev2IB8PEkK8WX6oqMb23/fmM1jy6rbaxZ1oVw+Xq5jWLZTlhwp5ZFEKYzsFk1Vi4re9+VwzNNrh88cKIZomOtCb03qGc9fvBzmlWxg+ngb+2JdPt3DfGnOvCtHWuUsdf9WQaJ78O5VdOWV0CfNhY3oJe3LLefrERGcXTTRTm29Eu0u4XIGPp4EnT0xkSXIh2zJLCfbx5MkTE0kMkbvQQrii2f0i6dfBn6XJhZgsmvP7RzIsPrCtDn4VohZ3quN7RPjxwsmd+WN/PlszSuke4ce1w2Lq7JYl3EObbUS7U7BcibeHgSlJoUxJCnV2UYQQjdC3gz99O8idZ9G+uGsdHxXg5TZLWouGtcln9O4aLiGEEELUT+p44SrazJ1oCZUQQgjRtkldL1xJm2hES6jaj13ZZczfk0d2qYlu4X6c1jOsRUuPCiEcJ7/MzNzdeezMLiPM14NTuodJ1xPRbFLXO1alRbPoQAHLU4qwaBiVEMiUpFCXXU3RFbh1d46AOVMlVO3IypQinlmaSrdwX87tG0ml1tz1ezIZxUZnF00IcYzcMjN3/XGQEmMl5/SNoE8Hf15YcZiFBwqcXTThhqSudyytNS+vSueP/QVM7RrCtG6hLE0u4vnlacgil8fntneiJVDti0VrPtyQyd1j46tWNxoYE4Cfp4Hvtudy/fAYJ5dQCFHdTztzGR4fxFVDogHrEty9Iv14fHEK4xKD8fKQu1uicaS+d7zdOeXszinjtVO74G2b0nZoXCC3/nqALRmlNVZGFf9yuzvRhow9Eqh2KLvUjMmiay0POiYxiK2ZpU4qlRDieLZmljI2seby20lhvvh7eZBWVOGkUgl3I/V969iWWcrw+MCqBjSAl4diVEIQW7Okjj0et70TLdqXAC8D5WYLJcZKAqrNqZlZYiLEV+bYFMLVhPh6kFlione12bwqzBYKK8wE+0jVI+onjefWFeLrwa6cslrbM4tN9Iz0c0KJ3IPb3YkW7VOAtwfD4wN5b30mFWYLANmlJj7dlM20bqHOLZwQopZp3UL5Ykt21ZgFU6WFjzZm0qeDP+F+0ogWdZOxTs4xKiGIndllLD9UiNYarTVr0orZcKSEcZ2CnV08lyVnMuE2rhkazaurj3DFT/uICfTicJGR03uFM14CLoTLGR4fRHqRidt/O0h0oDfZJSa6R/hy68g4ZxdNuChpPDuPv5cHD47vyIsr0/l0cxYKRaXW3DcunmAfedp7PNKIFm7D38uDe8fGk1ViIrfMTMdg7xpdO4QQrmVWr3Cmdg0hpcBIqK8H0YHezi6ScFHSgHa+7hF+/G96Fw7mV6A1dAnzwaBkAHB9pBEt3E5UgBdRATI3tBDuwN/LQ/pUinpJA9p1GJQiKczX2cVwG9InWgghhBBCiCaSRrQbKDVVklJQQbltQJ0QwnVVWjSphRXkl5mdXRQhXJ4r3IXOLjVxuMgoi4qIJnNYdw6lVALwMRANaOAdrfUrx+yjgFeAU4FS4FKt9XpHlcndVFo0H23M5K/9BYT4elJYYWZmz3DO7RuBkn5Kwo4kr/axMqWI99Zn4KEUJaZK+kT5c9OIGJnSTdhdW8issxvQGcVGXl2dzqECI14eCl8PA9cNi6Z/tCwsIhrHkWd2M3CH1nq9UioIWKeU+kNrvb3aPqcA3W1fI4A3bf8VwJdbszmYX8EbM5II9fUkq8TEM0tTCfbx4JTuYc4unmhbJK8ttC+3nLfWHuHesfH0jvKnwmzh081ZzFl+mCcmJzq7eKLtcdvMOrvxDNabVI//ncqJSSE8NikcDwXrDpfw/PLDvHhyZxl3IxrFYd05tNbpR694tdZFwA4g/pjdZgEfa6tVQKhSKtZRZXInFq1ZsCef64fFEOprvdaJCvDiqiHR/LI7z8mlE22N5LXlft2bx6ye4fS2rarp42ng0kEdSCs0cqhAVugT9uWumXWFBjTAloxSfD0NnNk7Ak+DQinF0PhAxiUGsfBAgbOLJ9xEq/SJVkp1Bk4AVh/zq3ggpdrPqdQ+CaCUuloptVYptbawvH30MzRbNGVmC9GBNa+G44O9yZW+lsKBWppX2zHaXWZzy8zEB9ecws3DoIgN8pLMCoeyZx1blJfrsHK6SgMabHkNqj3lYnywDzmSV9FIDm9EK6UCge+AW7XWhc05htb6Ha31UK310GDf9tG30NvDQEKwN+vTS2psX51aLNNFCYexR16hfWa2Z6Qfq1OLa2zLKzNzIK9CpowSDmPvOjYoLNy+BbRxpQY0QI9IXzZllFStgAugtWZ1ahG9IqSOFY3j0NpNKeWFNdyfaa2/r2OXNCCh2s8dbdsEcNHAKF5dnc55/SLpEeHH5owSvt+Ry0MTOjq7aKINkry2zLRuYdz1+0HeXnuEiZ1DyCkz8eWWHGb2DJcVv4RDuEtmXa0BDdAx2IehcYE8vCiFc/pG4O9lYMGefIqNlYztFOTs4gk34bA70bZRwe8DO7TWLx5nt5+Bi5XVSKBAa53uqDK5myFxgdw3riObM0p5bXU6B/MreGJSAj3kKlnYmeS15YJ9PHhuSie8PQy8tfYIC/bkc3bfCGb3i3B20UQb5C6ZdcUG9FHXD4thUpdgvtuew3vrM4kL8uKJyYl4e8jsv6JxHHknegxwEbBFKbXRtu1+IBFAa/0WMB/r1Dt7sU6/c5kDy+OWekX60Wtsnd1OhbAnyasdhPp5ctkJHZxdDNE+uHxmXbkBDdYxC9O6hTGtm8x2JZrHYY1orfUyoN7JjLV1ZvMbHFUGIUTjSF6FcC+unFlXbzwLYS/yzEIIIYQQdiENaNGeSCNaCCGEEC0mDWjR3kgjWgghhBAtIg1o0R5JI1oIIYQQzSYNaNFeSSNaCCGEEM0iDWjRnkkjWgghhBBNJg1o0d5JI1oIIYQQTSINaCGkES2EEEIIIUSTOXLFQiGEEEK0IXIHWoh/yZ1oIYQQQjRIGtBC1CR3ooUQQghxXNJ4FqJucidaCCGEEHWSBrQQxyeNaCGEEELUIg1oIeonjWghhBBC1CANaCEaJo1oIYQQQgBgyNgjDWghGkka0UIIIYQQQjSRNKKFEEIIIYRoIoc1opVSHyilMpVSW4/z+4lKqQKl1Ebb18OOKosQomGSWSHch+RVCOdz5DzRHwGvAx/Xs89SrfUMB5ZBCNF4HyGZFcJdfITkVQinctidaK31EiDXUccXQtiXZFYI9yF5FcL5nN0nepRSapNSaoFSqq+TyyKEaJhkVgj3IXkVwoGU1tpxB1eqMzBPa92vjt8FAxatdbFS6lTgFa119+Mc52rgatuPPYFddewWCWTbpeCuqa1/PpDP2FydtNZR9jhQK2ZW/l+7v7b++UDyWl1b///d1j8ftP3P6KjPd9zMOq0RXce+B4GhWutm/QMopdZqrYc257XuoK1/PpDP6ApaK7Ou/u9gD239M7b1zweu/xmljrWftv75oO1/Rmd8Pqd151BKxSillO374bay5DirPEKI+klmhXAfklchHM9hs3Mopb4AJgKRSqlU4BHAC0Br/RZwNnCdUsoMlAHnaUfeFhdC1EsyK4T7kLwK4XwOa0Rrrc9v4PevY52ex17eseOxXFFb/3wgn9GpWjmzLvvvYEdt/TO29c8HLvwZpY61u7b++aDtf8ZW/3wO7RMthBBCCCFEW+TsKe6EEEIIIYRwO22iEa2U8lBKbVBKzXN2WRxBKXVQKbXFtnTrWmeXxxGUUqFKqW+VUjuVUjuUUqOcXSZ7UUr1rLb07kalVKFS6lZnl8uZ2nJmJa/uTzJbU1vOK7T9zEpeHceRy363pluAHUCwswviQJOaOzWRm3gF+FVrfbZSyhvwd3aB7EVrvQsYBNbKCEgDfnBmmVxAW8+s5NWNSWZraet5hbadWcmrg7j9nWilVEdgOvCes8simkcpFQKMB94H0Fobtdb5Ti2U45wI7NNaJzu7IM4imXVv7Syv0M4zK3l1b5JXx3L7RjTwMnA3YHFyORxJA78rpdbZVpZqa7oAWcCHtkeG7ymlApxdKAc5D/jC2YVwspdp25mVvLYt7T2zL9O28wptO7OSVwdy60a0UmoGkKm1XufssjjYWK31YOAU4Aal1HhnF8jOPIHBwJta6xOAEuBe5xbJ/myP0WYC3zi7LM7STjIreW0j2ntm20leoW1nVvLqQG7diAbGADNty5l+CUxWSn3q3CLZn9Y6zfbfTKz9fIY7t0R2lwqkaq1X237+Fmvo25pTgPVa6wxnF8SJ2nxmJa9tSnvPbJvPK7T5zEpeHcitG9Fa6/u01h211p2x3sJfqLX+j5OLZVdKqQClVNDR74GTgK3OLZV9aa2PAClKqZ62TScC251YJEc5n/b9WLjNZ1by2ua068y29bxC28+s5NWx2srsHG1ZNPCDUgqs/78+11r/6twiOcRNwGe2xzH7gcucXB67sp2cpwLXOLsswqEkr22EZLbdaA+Zlbw66n1lxUIhhBBCCCGaxq27cwghhBBCCOEM0ogWQgghhBCiiaQRLYQQQgghRBNJI1oIIYQQQogmkka0EEIIIYQQTSSNaDenlHpAKbVNKbVZKbVRKTXCzsefqJSa19jtdni/05VSfar9vFgpNdTe7yOEs0hmhXAfkldRH5kn2o0ppUYBM4DBWusKpVQk4O3kYrXU6cA82u5k8KIdk8wK4T4kr6IhcifavcUC2VrrCgCtdbbW+jCAUmqIUupvpdQ6pdRvSqlY2/bFSqlXbFfUW5VSw23bhyulViqlNiilVlRb3ahBthWfPlBK/WN7/Szb9kuVUt8rpX5VSu1RSj1f7TVXKKV2217zrlLqdaXUaKzr3s+xla+rbfdzbPvtVkqNs8c/nBBOIpkVwn1IXkX9tNby5aZfQCCwEdgNvAFMsG33AlYAUbafZwMf2L5fDLxr+348sNX2fTDgaft+CvCd7fuJwLw63rtqO/A08B/b96G28gQAl2JdHSkE8AWSgQQgDjgIhNvKuhR43fb6j4Czq73PYuAF2/enAn86+99dvuSruV+SWfmSL/f5krzKV0Nf0p3DjWmti5VSQ4BxwCTgK6XUvcBaoB/wh7IuZeoBpFd76Re21y9RSgUrpUKBIOD/lFLdAY01eI11EjBTKXWn7WdfINH2/V9a6wIApdR2oBMQCfyttc61bf8G6FHP8b+3/Xcd0LkJ5RLCpUhmhXAfklfREGlEuzmtdSXWK8nFSqktwCVYg7BNaz3qeC+r4+cngEVa6zOUUp1tx2wsBZyltd5VY6N1AEZFtU2VNO9v7ugxmvt6IVyGZFYI9yF5FfWRPtFuTCnV03ZVe9QgrI9zdgFRyjooAqWUl1Kqb7X9Ztu2jwUKbFexIUCa7feXNrEovwE3KdsluVLqhAb2XwNMUEqFKaU8gbOq/a4I6xW7EG2OZFYI9yF5FQ2RRrR7C8T6eGi7Umoz0Ad4VGttBM4GnlNKbcLap2t0tdeVK6U2AG8BV9i2PQ88Y9ve1CvRJ7A+mtqslNpm+/m4tNZpWPt4/QMsx9p3q8D26y+Bu2yDJ7rWfQQh3JZkVgj3IXkV9VK2zuSinVBKLQbu1FqvdXI5Am39zTyBH7AOyvjBmWUSwhVJZoVwH5LX9kXuRAtneVQptRHYChwAfnRqaYQQDZHMCuE+JK+tQO5ECyGEEEII0URyJ1oIIYQQQogmkka0EEIIIYQQTSSNaCGEEEIIIZpIGtFCCCGEEEI0kTSihRBCCCGEaCJpRAshhBBCCNFE/w+rJL4JcdnJQAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.linear_model import Perceptron, RidgeClassifier\n", "from sklearn.svm import SVC\n", "models = [SVC(kernel='linear', C=10000), Perceptron(), RidgeClassifier()]\n", "\n", "def fit_and_create_boundary(model):\n", " model.fit(X, iris_y2)\n", " Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " return Z\n", "\n", "plt.figure(figsize=(12,3))\n", "for i, model in enumerate(models):\n", " plt.subplot('13%d' % (i+1))\n", " Z = fit_and_create_boundary(model)\n", " plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired) \n", "\n", " # Plot also the training points\n", " plt.scatter(X[:, 0], X[:, 1], c=iris_y2, edgecolors='k', cmap=plt.cm.Paired)\n", " if i == 0:\n", " plt.title('Good Margin')\n", " else:\n", " plt.title('Bad Margin')\n", " plt.xlabel('Sepal length')\n", " plt.ylabel('Sepal width')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Below, we plot a decision boundary between the two classes (solid line) that has a high margin. The two dashed lines that lie at the margin.\n", "\n", "Points that are the margin are highlighted in black. A good decision boundary is as far away as possible from the points at the margin." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "(2.25, 4.0)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEzCAYAAABJzXq/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABFrUlEQVR4nO3de1zU5533/9c1w3A+KQKCioCieBYEj6iARqMxGmPOJk3bdNM23TbZ3vdu2/3dd7vb/u7fbne729xp0yb2sG22e0pt0pyapA0zoHg+n09EEJHz+cwwM5/fHwyzhJiIwjDDl+v5ePgIzFx8eUeZD9+5vt/rcykRQdM0TQOTrwNomqb5C10QNU3T3HRB1DRNc9MFUdM0zU0XRE3TNDddEDVN09yGXBCVUmal1Aml1Ns3eS5IKfVfSqkSpdQhpVTyiKbUNE0bBbdzhvgscOETnnsKaBKRmcAPge8PN5imadpoG1JBVEpNBe4Bfv4JQ7YBv3Z/vBtYp5RSw4+naZo2eoZ6hvg88FeA6xOenwJcBxARB9ACxAw3nKZp2mgKuNUApdQWoFZEjimlcofzzZRSTwNPA4SFhS1JT08fzuE0TdM+5tixY/UiEnsnX3vLggisArYqpTYDwUCkUuo3IvL4gDE3gGlAhVIqAIgCGgYfSER2AbsAsrKy5OjRo3eSWdM07RMppa7d6dfe8i2ziHxLRKaKSDLwCGAdVAwB3gSedH/8gHuM7hqhadqYMpQzxJtSSn0XOCoibwK/AP5VKVUCNNJXODVN08aU2yqIIlIIFLo//vaAx7uBB0cymKZp2mjTK1U0TdPcdEHUNE1z0wVR0zTNTRdETdM0N10QNU3T3HRB1DRNc9MFUdM0ze2Ob8zWtLHG7nRxpa6dqtZuooItpMdHEBGkXwLaf9M/Ddq40Ot08d7FGrp6nbgE6jvslDV1sn5WLBNCAn0dT/MT+i2zNi6UNnbQ7S6GAAI4XMKpylaf5tL8iy6I2rjQ0NGL8ybtRpo67aMfRvNbuiBq48KEUAvmmzRxjwq2+CCN5q90QdTGhdSYMAIDFKYBNdGsFIsSo3wXSvM7+qKKNi4Emk3cPTue8zVt1LR1ExFsYV58BBNC9QUV7b/pgqiNG8EWM5lTo30dQ/Nj+i2zpmmamy6ImqZpbrogapqmuemCqGma5qYLoqZpmpsuiJqmaW66IGqaprndsiAqpYKVUoeVUqeUUueUUn97kzFJSimbUuqEUuq0Umqzd+JqI01EaO3upavX6esomuZzQ7kxuwfIF5F2pZQFKFZKvSsiBweM+V/AqyLyU6XUXOAPQPLIx9VGUn1HD8WlDdgdgiDEhgWRkxJDYIB+46CNT7f8yZc+7e5PLe4/g/uGCBDp/jgKqByxhJpXOJwuCkvq6ep14RTBJVDX0cPB8kZfR9M0nxnSqYBSyqyUOgnUAn8SkUODhvwN8LhSqoK+s8OvjmRIbeRVtnZ/7LeaS/oed7hu0idL0/ycw+Hg8OHDwzrGkNYyi4gTWKyUigZeV0rNF5GzA4Y8CvxKRP5JKbUC+Ff3GNfA4yilngaeBkhKShpWcG14PrHmSd+8Iny8VZam+SOXy8WpU6coLCykpaVlWMe6rckiEWkGbMDdg556CnjVPeYAEAxMusnX7xKRLBHJio2NvaPA2shIiAxGBhVFBUwKC8Ri1nOI2thQUlLCiy++yBtvvEF4eDhPPPHEsI53yzNEpVQs0CsizUqpEOAu4PuDhpUD64BfKaXm0FcQ64aVTPOqoAATK1MmcqCsEUXfJHCwxcyK5Im+jqZpn0pEcLlcmM1m7HY7ZrOZRx55hNmzZ6Nu0gT4digZfJoweIBSC4FfA2b6zihfFZHvKqW+CxwVkTfdV5Z/BoTT99r6KxH546cdNysrS44ePTqs8NrwOZwu6jrsWMyKmNDAYf9AaZq3iAilpaVYrVbS0tJYu3YtIoKIYDL997sapdQxEcm6k+9xyzNEETkNZNzk8W8P+Pg8sOpOAmi+FWA2kRAZ7OsYmvaprl+/TkFBAWVlZURFRTFhwgQAlFIj+ktcN4jVNM2vFRYWUlhYSHh4OJs2bWLJkiUEBHindOmCqGma36mrqyMoKIjIyEhmzZpFQEAAS5cuJTDQu1s+6IKoaZrfaGpqoqioiFOnTrFkyRK2bNlCYmIiiYmJo/L9dUHUNM3nWltb2bNnD8ePH8dkMrFixQpWrRr9yxK6IGqa5nNFRUWcPHmSJUuWsHr1aiIjI2/9RV6gC6KmaaOuu7ub/fv3M3v2bKZMmUJubi45OTmeq8e+oguipmmjxm63c/jwYfbt20dXVxcWi4UpU6YQERHh62iALoiaH3O6hGtNndxo6SI8MIC02HDCg/SP7Fh1/PhxCgoK6OjoYNasWeTl5ZGQkODrWB+hf7o0v+QSwVpSR1NXL06XoICShg7WpcUyMdS7t15oI8fpdGIymVBK0dbWRlxcHPn5+UybNs3X0W5Kr+LX/NKNlm6a3cUQ+taDOlzCsYpmn+bShsblcnH69GlefPFFLly4AMDq1at58skn/bYYgj5D1PxUQ0fPTfsyNnX1+iCNNlQiwsWLF7HZbNTW1hIfH09ISAjAR9Yb+ytdEDW/FBFswWxSnjPEfmGBZh8l0oZi9+7dnDt3jpiYGB544AHmzZs3phqG6IKo+aXp0SGcqWqh2yWezt5mpVicGOXTXNrHlZeXk5CQgMViYcGCBcycOZNFixaNiTPCwXRB1PxSgNnExtnxnKlqpbqtm1CLmXmTI3VnHj9SWVmJ1WqlpKSETZs2sWzZMtLT030da1h0QdT8VojFzNIk396oq31cbW0tNpuNCxcuEBISwl133UVmZqavY40IXRA1Tbst77zzDtXV1eTm5rJixQqCgoJ8HWnE6IKoadqnamlpobi4mLVr1xIeHs7WrVsJCQkhNDTU19FGnC6ImqbdVHt7O8XFxRw9ehQRISUlhblz5xITE+PraF6jC6KmaR8hIthsNg4cOIDT6WTRokWsXbuW6OhoX0fzOl0QNU0D+pbZmc1mlFI0Nzcze/Zs8vLyDH1GOJguiJo2zvX29nL06FH27dvHE088QXx8PPfdd9+YvI9wuHRB1LRxyul0cuLECfbs2UNrayupqameVSXjsRjC0DaqDwb2AEHu8btF5Ds3GfcQ8Df0rcM/JSKPjWxUrV+Pw4lSikDz+Pyh1YbP5XKxa9cuampqmDZtGtu3byclJcXXsXxuKGeIPUC+iLQrpSxAsVLqXRE52D9AKZUGfAtYJSJNSqk4L+Ud19p6HOwrbaClu6/BQXx4ECuTYwgM0IVRuzUR4erVq6SmpmIymcjOziYqKoqZM2eOqfXG3jSUjeoFaHd/anH/GdyG5M+AF0Wkyf01tSMZUuvrD1hwpY6uXqfnsZr2HorLGsifGevDZJq/ExFKSkqwWq1UVVXxmc98htTUVLKysnwdze8MaQ5RKWUGjgEz6St8hwYNmeUetw8wA38jIu+NZNDxrr7dTq/T9ZHHXAJ17T109zoJtuguMNrHlZWVYbVaKS8vZ8KECWzfvp3k5GRfx/IKl8vF2bNnh3WMIRVEEXECi5VS0cDrSqn5IjLwOwcAaUAuMBXYo5RaICLNA4+jlHoaeBogKSlpWMHHm16X66aPKxRO+XjfQE1zOBz87ne/A2DLli1kZGRgNhvvF+fgHozDcVuTT+4CZwPuHvRUBfCmiPSKSClwmb4COfjrd4lIlohkxcbqt3m3Iy486GPzFAAhFhOh+uxQc6upqeGtt97C6XQSEBDAzp07+drXvkZWVpbhimH/VMDPfvYz/uu//guXy8WDDz44rGMO5SpzLNArIs1KqRDgLuD7g4b9HngU+Bel1CT63kJfHVYy7SMsZhOrkieyr7SR/vlvk4Kc1Bg9Ia7R0NBAYWEhZ8+eJTAwkCVLlpCYmMjkyZN9Hc0rrl27RkFBAeXl5URHR3PfffexcOHCYd8uNJS3zAnAr93ziCbgVRF5Wyn1XeCoiLwJvA9sUEqdB5zAX4pIw7CSaR8zJSqE+xYkUN3ajdmkmBwRjNmki+F4Zrfbee+99zh58iRms5mcnBxWrlzpadtvNAN7MEZERHDPPfeQmZk5Yme/Q7nKfBrIuMnj3x7wsQBfd//RvCjQbCJpgvG6jGi3x+FwEBAQgMVioba2lqVLl5KTk0N4eLivo3nFwB6MoaGhbNiwgezsbCwWy4h+H71SRdPGkM7OTvbv38+pU6d45plnCAkJ4fOf/7xhV5Y0NjZis9k8UwF5eXksX77caz0YdUHUtDGgp6eHgwcPsn//fux2OwsWLMDhcADGXGbX0tJCUVGRZypg1apVrFy50us9GHVB1DQ/19nZyY9//GM6OzuZM2cOeXl5xMUZczFYe3s7e/fu5ejRowBkZ2ezevXqUZsK0AVR0/yQ0+mkvLyclJQUQkNDWb58OTNnziQxMdHX0byiq6uL/fv3c/DgQZxOJ4sXL2bt2rVERY3uLou6IGqaH3G5XJw+fZrCwkJaW1v52te+RnR0NGvWrPF1NK/onwo4cOAAPT09zJ8/n9zcXJ/1YNQFUdP8gIhw/vx5bDYb9fX1JCYmsmXLllE/Qxotvb29HDlyhOLiYjo7O0lPTycvL4/4+Hif5tIFUdP8QFtbG6+99hoxMTE8/PDDpKenG/KG+/4ejEVFRbS1tTFjxgzy8/OZMmWKr6MBuiCOOfUdPVxr7MRsMpEaE0pk8J3fhyUi1Lb3cL25i0CzidSYMMKD9I/EUDldwrWmTuo77ESHBJAyMQzLbfSoLC0t5cqVK2zYsIHIyEg+//nPk5CQYMirxv1TAUVFRTQ1NZGUlMSOHTv8rtGE/ukfQ85UtXChth2nS1DA5bp2lk+fcMc3ah+raOZqY6fneBfr2lmTEsPkyOARzW1EDqeL9y/X0ml34nAJZgXna9rYODuekFusLb9+/TpWq5XS0lIiIyNZtWoVYWFhfnOWNJIGTwUkJCSwc+dOv+3BqAviGNHV6+R8TRsud4cHAZwiHLnezNToEEy3+cPV2t3L1YYOnAOP5xIOlTexdd5kv/xh9SclDR109Dg8f39Oge5eF+eqW8maNuGmX9Pa2srbb7/N5cuXCQsL4+677yYrK4uAAOO9DEWEK1euYLVaqa6uJjY2loceeog5c+b49c+W8f4lDKqhw45JKVyDWn05ReiwO4m4zbe69R32vh/MQcfrdjjpdQqBAf77Q+sPqlu7PcWwnwA1bT0fG9u/zC4oKIiGhgbWrVvHsmXLCAwMHJ2wo6ysrIyCggKuX7/u6cG4YMGCMTEVoAviGBEWaB5cu4C+38TBd7CFQGjgzd/WmZQiwKyL4a2EBwWg2no+1pJt4BxsU1MTRUVFVFZW8qUvfYmgoCC+8pWvjInCcCcqKiqwWq1cvXqVyMjIMdmDURfEMWJCaCATQi00dto9b5vNClJjbm8iv198eBChgWbaehyeQmtWkB4Xfttvv8ej2XERlDZ24nD9d0k0K8W8yRG0tbWxZ88ejh8/jlKK7OxsHA4HgYGBhiyGNTU1WK1WLl26RFhYGBs3biQ7O3tMTgWMvcTjWO6MSZyuaqW8qROTUqTFhpEeF3FHx1JKcVdaHCcrW6ho6cJiUsyOCydtkjG7pYy0iKAA1qfFcuJGC01ddiKCAlg8JRp7SwMv/fKXuFwuMjMzWbNmDZGRkb6O6xX19fWeHozBwcHk5+ezfPnyMT0VoMRH7eezsrKkf72ipo1l3d3d1NXVMW3aNFwuFwUFBWRlZTFhws0vrox1zc3NnsYLFouFZcuW+VUPRqXUMRG5ox209Bmipt0hu93O4cOH2bdvHyaTib/4i78gICCAu+66y9fRvKKtrY29e/dy7NgxlFIsW7aM1atXExYW5utoI0YXRE27TQ6Hg2PHjrF3717a29tJS0sjPz9/TM6ZDUVnZyf79u3j8OHDOJ1OMjIyWLNmjSGXFRrzX1DTvKiiooJ3332X5ORkHnroIcPuINnT08OBAwc4cOCApwdjbm4uEydO9HU0r9EFUdNuQUQ4e/Ysra2trFq1iuTkZP7sz/6MxMREv77J+E719vZy+PBhiouL6erqMnwPxoF0QdS0TyAiXLp0CZvNRk1NDYmJiaxYsQKTyWTIZXYOh4Pjx4+zZ88e2tvbmTlzJvn5+YbtwXgzuiBq44aIcPLkSW7cuEFkZOSn3iJSVVXF22+/zY0bN4iJiWHHjh3Mnz/fkGeELpeLU6dOUVRURHNzM9OnTzf0VMCn0QVRGxd2797N9773PTo6Opg1axb19fVUVFTwzDPP8M1vftNzQWTgbnadnZ1s27aNRYsWGfKGahHh3Llz2Gw2GhoaSExM5N577yU1NdWQhX8obnkfolIqGNgDBNFXQHeLyHc+YewOYDeQLSKfepOhP9+H6BJBBL3n8R1yugSl8JsVLy+88ALPP/88L7/8MuvXr8cpfatyLly4wHPPPUdkZCTPP/88RUVFmM1mHnnkEaCvYBixMIgIly9fxmq1UlNTQ1xcHPn5+cyePdsQ/7/DuQ9xKAVRAWEi0q6UsgDFwLMicnDQuAjgHSAQ+POxWBB7nS4OlTdxo6ULEYiPCGL59Im3bOek9Wnu6uXgtUaau3oxKcXMSWEsnhLl08J4+fJlcnJyOHLkCEET4zlc3kSn3UmAWTE/PpLw3hbWr19PWloaOTk5no3ejVAYbubq1atYrVYqKiqYOHEieXl5zJs3z1BnwF69Mdu9CX27+1OL+8/Nquj3gO8Df3knQfxBcWkDte09nrXCNW09WK/UsXlOvGFfICOl1+nigyu19LpbwDhFKKnvwKRg8ZRon+X66U9/yhe+8AUmTp7CuxdrcLr/cXudwtvFh6k4bGXlypUUFRXx6quv+s1qi5E2sAdjVFQUW7duZdGiRWOq8cJoGNIcolLKDBwDZgIvisihQc9nAtNE5B2l1JgsiF29zo8UQ+ir+p29Thq7eokJHbvrM0fD9eauj3XjcYpwpb7DpwXxT3/6E6+88gol9e24XEJnexu9Pd1ExcQyKSGJnlmL+H//97eYNm0a7e3thiuIVVVV2Gw2Tw/GTZs2sWTJEkPeRC4iVFZWDusYQ/pbEREnsFgpFQ28rpSaLyJnAZRSJuCfgc/e6jhKqaeBpwG/u4LV63TdtD+gAuwOl29CjSF2p+tjvRoBHC7x6Vyc3W4nLCyMytY2Tu4r5OqF00THxJG3/VGCQkKYtyyH8PBwQkJC6On5eC/Dsaqurg6bzcb58+cJDg5m/fr1LF26dEw3Xvg0paWlWK1Wrl+/Pqzj3NavCRFpVkrZgLuBs+6HI4D5QKH7h34y8KZSauvgeUQR2QXsgr45xGElH2ERQQFYTMrzlqqfC4gNM+YP0UhKiAzmdGXLxx6PCw/06XTDzJkz+eUvf4kKCuVqXSvTZs1lzpLlACgFU6NCKCsrw263G+LG4/4ejKdOncJisbB27VpWrFhBcLAxt4WoqKigoKDAsx3Dvffey9/+7d/e8fFuWRCVUrFAr7sYhgB30TdXCICItACTBowvBP7nrS6q+BulFKtSYij6sB5wT5KKsDxpAgF30G9wvIkKtjA3PoLzNW1A399ngEmxLMm3y7zWr1/PD37wA374wx+yNi2DNlMY0HfGGmoxszAhir/+5v/hM5/5zJg+e2ptbfX0YDSZTKxYscKzV4tR2e12fvOb32A2m0dsO4ahfHUC8Gv3PKIJeFVE3lZKfRc4KiJvDiuBH4kLD2Lb/ARutHThEpgSGUywvsI8ZPMTopg+MYzqtm6CA8wkRgaP+q1LDoeDo0ePEhoaysKFC/nzP/9zdu/ezQcffMBPd+yg1e6iocNOWFAAkyOC+NdXXuE//uM/OHTo0K0P7oc6OzspLi7m8OHD46YH44kTJ1i/fj2BgYE89thjTJ48ecR+mQ3lKvNpIOMmj3/7E8bnDj+W7wSaTaRMNO5vVW+LCAogImj0m8w6nU5OnjxJUVERra2tLFy4kIULFxIYGMh7773HY489RmpqKl/4wheYM2cOtbW1vPLKKzQ1NfGnP/2JqVOnjnrm4eju7vY0Xujt7WXhwoXk5uYaugdjYWGhZypg0aJFxMXFjfi1CN0gVhvzrly5wrvvvktjYyNTp05l3bp1pKSkfGzc8ePH+fWvf82NGzeIiIhgx44dbNq0aUzdejKwB2NXVxfz5s0jNzeX2NhYX0fzip6eHj744IOPbMeQk5PzqVMBukGsNu6ICE6nk4CAAETE8/YpLS3tEy/iZGZmkpmZOcpJR0b/VMDevXs9yw/z8vJISEjwdTSvcLlcmEwmAgICKC0t9fRg9PZUgD5D1MYUEeHDDz/EarUyY8YM1q1bR//PsBFvnnc6nZ7GCy0tLSQnJ7Nu3TqmTZvm62he0T8VcO7cOb74xS9isVg868uHSp8hauPCtWvXKCgooLy8nOjoaM9tMkYshP09GG02m2cqYNu2baSkpBjy/3dwD8a5c+fS09ODxWIZ1ZvIdUHUxoTCwkIKCwuJiIjgnnvuITMzc0zN/Q1Vfw9Gq9VKbW0t8fHxPProo8yaNcuQhRD6bhnatWuXZzuGvLw8n/Vg1AVR81u1tbUEBgYSHR3NnDlzCAwMJDs7G4vF4utoI05EPI0X+nswPvDAA8ybN8+QhdDlclFdXU1iYiIRERHMnz+fuXPn+nwFmy6Imt9pbGzEZrNx9uxZFi9ezLZt24iPjyc+Pt7X0byivLycgoICrl27RnR09Ljpwdja2spzzz1HWFgYd999t6+jAbogan6kpaXFs9+v2Wxm1apVrFq1ytexvKayshKr1UpJSQnh4eFs3ryZzMxMwzZeGNiDMT4+ngceeIDQ0FBfR/sI4/3Na2PW/v37OXXqFNnZ2axevZrw8NG/wXs01NbWYrPZuHDhAiEhIdx1110sXbrUkFMB/RoaGvjP//xPJk6c6NfbMejbbjSf6erqYt++fcyaNYukpCQ6OjpwOByG3O8X+qYCCgsLOXPmDIGBgaxYsYLly5cbtvHC9evXKSsrY/Xq1UBfR5rp06d7fSpA33ajjSk9PT0cPHiQ/fv3Y7fbCQoKIikpybCNCFpbWykqKuLEiROYzWZWrlzJqlWr/O7t4kipqqrCarVy5coVIiIiWLp0KUFBQTddPeRvdEHURtWxY8coKCigs7OT9PR08vLyDHuxpKOjg71793L06FFEhKysLFavXk1ERISvo3lFS0sL77//PufPnyckJGRM9mDUBVHzOqfTiVIKk8lEd3c3CQkJ5OfnG3JvY+ibCti/fz+HDh2it7eXxYsXs3btWqKjo30dzSv6l9mZzWbKy8vHdA9GPYeoeY3L5eL06dMUFRWRl5fHwoULDbuTHfRNBRw6dIj9+/fT3d3N/Pnzyc3NZdKkSbf+4jGovwdjfX09Tz75JEopnE6nz2+Y13OIml8REc6fP4/NZqO+vp6EhATPonwjFsPe3l5P44XOzk5mz55NXl4ekydP9nU0r+jo6KC4uJgjR44gImRmZnoabfi6GA6XLojaiNu9ezfnzp0jNjaWhx56iDlz5hiyEDqdTk6cOMGePXtobW0lNTWV/Pz8Mddb8XaUl5fzm9/8ht7eXhYtWkRubq6hpgJ0QdRGRFlZGYmJiQQGBrJ48WJmzZrFggULDLnawuVycebMGQoLC2lqamLatGncf//9JCcn+zqaV9jtdpqamoiPjychIYEFCxawYsUKQ04F6IKoDUtFRQVWq5WrV6+yYcMGVq5cSVpamq9jeYWIcOHCBWw2G3V1dUyePJmdO3cyc+ZMQ54BD+zBGBgYyFe/+lUsFgv33nuvr6N5jS6I2h2pqanBarVy6dIlwsLC2LhxI9nZ2b6O5RUiQklJCVarlaqqKiZNmmT4qYCB2zGkpKSwbt06Q57tD6YLonZH3n//fSorK8nPz2f58uVj6l6z21FWVobVaqW8vJwJEyawfft2w04F9CspKeGtt95i6tSpbN++fUzcUD1S9G032pA0Nzezd+9e1qxZQ1RUFE1NTQQHBxMSEuLraF5x48YNrFYrH374IREREaxdu5aMjIwxfxX1ZkSEixcv0tXVRWZmJiJCaWnpmG1Gq2+70bymra2NvXv3cuzYMZRSpKamEhUVZdjd3WpqarDZbFy8eJHQ0FA2btxIVlaWIRsvDNyOobKykilTppCRkeH5dx6PhrJRfTCwBwhyj98tIt8ZNObrwBcAB1AHfF5Ero18XG20iAhWq5WDBw/idDo9m/wYtfFCQ0MDhYWFnD17lsDAQPLy8li+fDlBQUG+juYVVVVVvPfee54ejPfddx8LFy4ck2eEI2koZ4g9QL6ItCulLECxUupdETk4YMwJIEtEOpVSXwb+AXjYC3k1L+vt7cVisaCUoqOjgzlz5pCbm8vEiRN9Hc0rPqkHo1GnAvqX2blcLhobGw29HcOdGMpG9QK0uz+1uP/IoDG2AZ8eBB4fqYDa6Ojf5Gffvn08/vjjJCYmcu+99xr2jKG9vd3TeAFg6dKl5OTkGLoHo9VqJTQ0lK1btzJlyhSee+45XQgHGdIcolLKDBwDZgIvisihTxn+FPDuCGTTRoHD4eD48ePs2bOH9vZ2Zs6c6ZkvM2Ix7O/BeOjQoXExFTC4B2NOTo7nOV0MP25IBVFEnMBipVQ08LpSar6InB08Tin1OJAFrL3ZcZRSTwNPAz7fTEbrmyf82c9+Rk1NDdOnT+fBBx9k+vTpvo7lFYN7MC5YsMDQUwEAp06d4o033vBMBaxcudKwPRhHym1dZRaRZqWUDbgb+EhBVEqtB/4fYK2I9HzC1+8CdkHfbTd3lFgblv6bjPtXVyxfvpzIyEhSU1MNeUbY29vLkSNHKC4uprOzkzlz5pCXl+fZ09lo2tvbsdvtTJw4keTkZMNvxzDShnKVORbodRfDEOAu4PuDxmQALwN3i0itV5JqwzJ4k5+dO3eSlpZGRkaGr6N5hdPp9EwFtLW1MXPmTPLz832236+39fdgPHjwINOnT+fxxx8nKiqKTZs2+TramDKUM8QE4NfueUQT8KqIvK2U+i5wVETeBP4RCAd+6z7LKBeRrd4Krd2e/v1+KyoqPJv8zJgxw9exvKK/B2NhYSHNzc0kJSXxwAMPGHoqYGAPxv6pAO3ODOUq82ngY6cRIvLtAR+vH+Fc2ghxOp288cYbAGzdupVFixYZcjJ9cA/GxMREtmzZwowZMww5FdDvyJEjWK1Ww2/HMFr00j0Dqqqq4tChQ2zZsoWAgADq6uqYMGGCYff7vXLlClarlerqamJjY8nPzyc9Pd2QhbC/B2NUVBRpaWn09PRQX19v2O0Y7oReuqcBUFdXh81m82zyk52dzZQpU4iNjfV1NK8oLS2loKDAMxVw//33M3/+fEM2Xhi4HUNTUxOLFy8mLS2NoKAgXQxHkC6IBtDb28s777zDqVOnsFgsY3qTn6GoqKigoKCA0tJSIiMjuffee1m8eLEhpwKgr/vM+++/T11dHQkJCZ4ejNrI0wVxDOtfZhcQEEBzczMrVqxg1apVht3fuLq6GpvN5unBePfdd5OVlWXYqQARwWQy0d7et1DMyD0Y/YWeQxyDOjs7KS4u5uTJk3zlK18hLCzM0LvZ1dfXY7PZOHfuHMHBwaxatYply5YZugdjQUEBc+fOZcWKFbhcLgBDTgV4g55DHCe6u7s5cOAABw4c8Gzy0/9iMWIxbG5uprCw0DMVsGbNGlauXGnoqYD+7RgiIiI8Z/q6EI4eXRDHiK6uLl544QW6urqYN28eubm5hr1Y0tbWxp49ezh+/LhnNU1OTo5hpwIArFYre/bsMXwPRn+nC6IfczgclJWVMXPmTEJCQsjJySE1NZWEhARfR/OK/qmAw4cP43K5yMzMZM2aNZ49nY2mvr6ekJAQwsLCmDFjBgEBASxbtsywPRjHAl0Q/ZDT6eTUqVOeTX6+9rWvMWHCBFatWuXraF7RPxVw8OBB7HY7CxcuZO3atYZtvNDc3Ozpwbh8+XI2btzI9OnTDbuaZizRBdGPiAhnz57FZrPR2NjI1KlT2bZtm2Hb9dvtdk8Pxq6uLubOnUteXp6hpwIGbsewbNmyj7Tj0nxPF0Q/0t7ezhtvvEFMTAyPPvoos2bNMuTFEofDwbFjx9i7dy/t7e2kpaWRn59v2KmAfh988AFnzpwxfA/GsUzfduNDIsLVq1e5dOkSmzZtQilFdXU18fHxhiyELpfLs99vS0sLycnJ5OfnG7Y3Zk9PDwcOHGDu3LnExcXR3NyMy+Uy7FSAv9C33YxB165dw2q1cu3aNaKiolizZg3h4eFMnjzZ19FGXP9UQGFhIQ0NDUyZMoWtW7caugdj/1RAZ2cnFouFuLg4oqOjfR1NuwVdEEdZa2srb775JiUlJYSHh7N582YyMzMNu9ri0qVL2Gw2ampqiI+P55FHHmH27NmGLIQAx48fx2azjYsejEZkvFehn+pfZhccHExrayt33XUXS5cuNeS9Zv1TAVarlRs3bhATE8OOHTuYP3++IQuhy+VCKYVSioaGBiZOnGjoHoxGpucQvax/k58bN27wzDPPYDabDb3Mrry8HKvVSllZGVFRUaxdu5bFixcbcrWFiHDu3DkKCwvZuHEjaWlpOJ1OTCaTYf99xwI9h+iHWltbKSoq4sSJE5jNZpYuXYrT6cRsNhvyxVJVVYXVauXKlSuEh4ezadMmlixZYtipgMuXL2Oz2aiuriYuLs7z/2nUjjvjhfF+Wv1AVVUVv/jFLxARsrKyWL16NREREb6O5RWDezCuX7+epUuXGrbxAsCrr77KhQsXDN+DcTzSBXGEdHV1UVNTQ3JyMpMnT2blypVkZmYa9spiU1MThYWFnD59etz0YExISMBsNjN37lxmzpxp6B6M45UuiMNkt9s9+/0qpfj617+OxWIhPz/f19G8orW11dN4wWQysWLFCnJycgy73291dTVWq5XLly+zdetWMjMzWbBgga9jaV6iC+Igdrud0tJSXC4XycnJhISE3HScw+Hw7Pfb0dHB7Nmzyc/PN+RVY4COjg6Ki4s5cuQIIsKSJUtYs2bNmJsKqK+vp7q6msjISKZNm/aJ87kDezD2TwXMnz9/lNNqo00XRLfW1lb+4R/+gZ///OeEhYUREBBAfX09TzzxBN/61rc+tptZVVUV77//PqmpqeTn5zN16lQfJfeu7u5uz36//T0Y165dO+bWVx84cIC///u/p6ioiMTERBoaGpg2bRrPPfccO3fu/EhhFBFee+016uvrDT8VoA3S36p8tP8sWbJE/EVjY6MsWrRIdu7cKefPn/c8XlpaKl/72tckOTlZSktL5eTJk1JYWOh5vqqqyhdxR0VPT4/s2bNH/u7v/k6+853vyKuvvip1dXW+jnVHfvvb30pcXJy8/PLL0t7eLiIiDodD3nnnHVmwYIE8++yz0tLSIu+++650dnaKiEhNTY1nrDa20Ldf/B3VpVueISqlgoE9QBB9Z5S7ReQ7g8YEAa8AS4AG4GERKRvp4u0tX/nKV1izZg0/fP55zla38dqZSkSEpAlR/OM//TMWi4V169bxxBNPMGXKFFavXo3JZDLkMjuHw8HRo0fZu3cvHR0dzJo1i/z8/DH7/1pRUcEXv/hFCgoKSI0J5dTzf07zh2cIjplMxsN/wfvvv8+KFSuorq5m7ty5JCcnk56eTlxc3Khnrb9wmHO/+Qc6qsoInzqDeTu/QczszFHPMZ7d8sZs1fdeIkxE2pVSFqAYeFZEDg4Y8wywUES+pJR6BNguIg9/2nH95cbsyspK5s2bR1lZGeebnFxv7sTp/itpqa/h4gEbkY5WfvSjH/Hzn/+c+++/35D3ETqdTk/jhdbWVlJSUli3bt2Ynwr4zne+Q0NDAz/43rex/s/NOLo6APdN1Y0O2tPy+LCulTNnzrBv3z6fTQU0l55j398+jtPe7XnMHBjM6u/+F5FJs3ySaazy6o3Z7lPQdvenFvefwVV0G/A37o93Az9WSim5VbX1A++99x6bNm0iOCyc8tIqXAKO3l4CLBbMgUG0dnbzwPatdHd3c/bsWXbs2OHryCPK5XJ5Gi/092Dcvn07KSkpvo42It544w1eeuklrllfxdXbi0sEk3uZXWNbF+GVZ3j+xdeZPXs2vvxxvfLGz3D29nzkMWevnZJ3fknml//eR6nGnyFdVFFKmYFjwEzgRRE5NGjIFOA6gIg4lFItQAxQP+g4TwNPA37T8qmjo4OJEyfS3euiubaaU4f2oUwmcjZvJzwymk2PfJYZqZOYOPEDGhsbfR13xIgIFy9exGazUVtby+TJk3nsscdIS0sz1Blw/79v89FyLlS3cLG+i/yUKCKCzKxKiiB0YiAJCQlERER4xvpCZ10FDC7I4qKz5rpP8oxXQyqIIuIEFiulooHXlVLzReTs7X4zEdkF7IK+t8y3+/XekJSUxL//+7/zzu93U7DnCAGBIczOyPasN3YBE0MDOXPmDCtXrvR13GETET788EOsViuVlZVMmjSJBx98kLlz5xqqEPZLSkrijTfeoP16E1dqe4gNNuFyFx5zgJlJ85ZTU1NDR0cHkyZN8lnO+MVraKu4gqvX7nnMFBhEXMZan2Uaj27rthsRaVZK2YC7gYEF8QYwDahQSgUAUfRdXPF706dP5+zZsxw+dIgH7tlEb1waZksgApiVYmFiJC2N9bzzzju88MILvo47LAN7MEZHR3PfffexcOFCwy47czqdJCUl8ZOf/IRvfuOvmGeqI7S5HJe9C3NgCObgEOY+8hf8009/xoMPPviJ95yOhhmbP8uN/X+gu7kWZ08X5qBQQmImk7php88yjUdDucocC/S6i2EIcBfw/UHD3gSeBA4ADwBWf54/bGlpoaOjg8TERNLT0/nc5z7Hn/70J775zW8SFh1DWVMnLoGk6BACnD1s2fIgX/rSl4iJifF19DtSWVmJ1WqlpKSEiIgI7rnnHjIzMw257ExEKC8vZ/r06ZjNZp588kn2799PTW0dX/jBb6k9UUjDpeOET57OlJVbKNp/kBdeeIE9e/b4NLclLJLc7/+eGwffo/XaRaJS5pK47G7MFuOuCfdHQ7nKvBD4NWAGTMCrIvJdpdR36bvf5033rTn/CmQAjcAjInL1047ri6vM7e3t7N27l6NHjxIfH8/TTz8N9L2Ivve97/GjH/2Ip556ii1bthAQEMAHH3zAyy+/zD333MOLL7445gpIbW0tVquVixcvEhoaSk5ODtnZ2YZcTTN4KuDzn/+8Z566srKSzZs3Exoaype+9CXmzJlDbW0tr7zyCjabjd27d7NmzRof/x9oI2U4V5nHRT/Erq4u9u3bx6FDh3A6nSxevJi1a9d+bJOfy5cv89JLL3HgwAFcLheLFy/my1/+MosXLx6VnCOlsbERm83G2bNnCQwMZOXKlSxfvtyw+/2Wl5dTUFDg2Y4hNzeXRYsWfWQqwOFw8NZbb/GrX/2KGzduEBERwY4dO3jiiSf0Zk8GowviLRw/fpy33nqL+fPnk5ubO2bf+t5KS0uLZ79fs9nMsmXLWLlypWEbL0BfJ/If/vCHmEwm1qxZY9jtGLSh0w1iB+nt7eXIkSOEhISQkZHB4sWLmTp1qk9WH4yG9vZ2T+MFgOzsbFavXk14eLiPk3lHXV0dx44dY8OGDVgsFnbu3ElcXJwhpwK00WWoguh0Ojl+/Dh79uyhra2NRYsWkZGRgclkMmQx7Orq8jRe+LSpAKNobGykqKiI06dPExgYSEZGBvHx8UyZMsXX0TSDMExBvHLlCu+88w7Nzc1Mnz7d0Jv89PT0cOjQIfbv309PT4/hpwJ6enr44x//6NmOYeXKlaxatcrQUwGab4zpgigiOBwOLBYLJpOJ0NBQtmzZwowZMwx5k3Fvb6+n8UJnZyfp6enk5eV9rDWZUbhcLkwmExaLhYqKCsNvx6D53pi8qCIiXLlyBavVSkpKChs3bvSsQzViIXQ6nZw4cYKioiLa2tqYMWMGeXl5Y77xwifpnwo4e/YsX/7ylwkMDPRs0KVptzKuLqqUlpZitVq5fv06EyZM8MwfGbEQulwuzpw5Q2FhIU1NTUybNo0dO3aQnJzs62heMXA7hu7ububPn4/dbicwMHBcFEMRofnDM7RVlBCZNJvo1HnDPl7jpeN0VF8jOnW+7pozBGOqIBYVFWGz2YiMjOTee+817CY/IsKFCxew2WzU1dWRkJDAzp07mTlzpiELP0BbWxsvvfSSZzuGvLy8MduD8U64HHYO/sOXaLpysu8BEWLmLmPp13+EKeD2r547ujvZ//99jrbrJUBf89OE7LvI/PLfowy6VHMk+H1BrK6uJjAwkIkTJzJ37lyCgoLIysoy5L1mIkJJSQlWq5WqqipiY2N56KGHmDNnjiELodPppLKykmnTphEREUFGRgbp6emGnQr4NGUf/BeNl0/gGtAPseHCYcqLXid53UO3fbwrb/6M1msXP9IsovroB1Qd/YDEpRtGJLMR+W1VGbjJz6JFi9i+fTuxsbHExsb6OppXlJWVYbVaKS8vZ8KECWzfvp0FCxYYsvFCfw9Gm81Ga2srzz33HBEREaxfv97X0XymYt/bHymGAM6eLm7sf/uOCmLlwXc/Ugz/+3h/0AXxU/hdQWxubqawsJBTp05hsVhYs2aNIdpufZKKigqsVitXr14lMjKSLVu2kJGRYdipgME9GB9++GHD3kB+OyxhkZ/w+J3dUxoQ8vG/U2UyERhhzHtUR4rfFcTDhw9z9uxZli9fTk5ODmFhYb6O5BU1NTVYrVYuXbpEWFgYGzduJDs725BTAf2ampp49dVXiYmJMXQPxjsxY/Nnabx4DKe9y/OYOTCE1E1P3NHxZm75PCd3/S+cPf99PFNAIMnrHx12ViPz+W03nZ2dFBcXM3PmTFJTU+nq6qK3t5fIyJv/xhzrGhoaPI0XgoODPY0XAgON2eaprKyMsrIycnNzgb5GDFOnTjXkVMBwlVl/y4X/+EFfP8TgMOY//g2mrbnvjo9X8s6/cPm1F3H22gkMi2LB579NYvZdIxfYT43J5g5LliyRH/zgBxw8eBC73U5eXp6hWzA1NzdTVFTEqVOnMJvNLF++nJUrV/q0Kak33bhxA6vVyocffkhERATPPPOMYf9fR5K4nPR2tmEJjRyRq8EupwNHV/uIHW8sGJP3IdbU1FBUVMTcuXPJy8sz7MWS9vZ29uzZw7Fjx1BKsXTpUlavXm3YqYCWlhbeffddTw/GDRs2GLYHozcok5nA8OgRO57JHDCixzM6nxXEwMBAvvjFL5KQkOCrCF7V2dnJvn37OHz4ME6nk4yMDNasWWPYxgv9K0ksFguVlZXk5eUZugejZkw+n0M0mp6eHs9qC7vdzoIFC8jNzfXZbm7e1t+Dsba2lqeeegqllF5mp/nUmHzLbDS9vb0cPnyYffv20dnZyZw5c8jLyzNk2zH46HYMAFlZWZ5GG7oYamOVLojD5HQ6OXbsGHv37qWtrY2ZM2eSn59PYmKir6N5zfXr13nllVfGRQ9GbXzRBfEOuVwuTp06RVFR0bjpwdjQ0EBiYiIJCQlkZGSwbNkyw/Zg1MYnXRBvk4hw7tw5bDabp0Dce++9pKamGvIm4/7tGIqLiwkICODZZ58lICCAzZs3+zqapo04XRCHSES4fPkyNpuN6upq4uLieOSRR5g9e7YhC+Hg7RhmzJhBfn6+nh/UDG0oG9VPA14B4gEBdonI/x00Jgr4DZDkPuYPRORfRj6ub1y9ehWr1UpFRQUTJ05kx44dzJs3z9CrLa5evco777xDUlKSoacCNG2goWxUnwAkiMhxpVQEcAy4T0TODxjz10CUiHxDKRULXAImi4j95kcdG7fdXL9+HavVSmlpKVFRUaxdu5ZFixYZ8ixJRDh//jydnZ1kZ2cjIpSXl5OUlGTIM2DNuLx6242IVAFV7o/blFIXgCnA+YHDgAjV98oJBxoBx50E8gfV1dVYrVYuX75MWFgYmzZtYsmSJYZsvDBwO4bq6mqmTJlCVlYWSil9VqiNO7f1CldKJQMZwKFBT/0YeBOoBCKAh0XEdZOvfxp4GiApKekO4nrXwB6MwcHBrFu3jmXLlhm28UJVVRV/+MMfPNsx3H///cyfP1+fEWrj1pALolIqHPgd8JyItA56eiNwEsgHZgB/UkrtHTxORHYBu6DvLfMwco+opqYmT+OFgT0Yg4ODfR3NK/pXkiilaG1tNfR2DJp2O4ZUEJVSFvqK4b+JyGs3GfI54O+lb0KyRClVCqQDh0csqRe0tbWxZ88ejh8/jlLK8D0Yq6ursdlsBAUFcf/99zN58mSeffZZQ18c0rTbMZSrzAr4BXBBRP75E4aVA+uAvUqpeGA2cHXEUo6w/h6Mhw8fxuVykZmZyZo1awzbg7G+vp7CwkJPD8acnBxEBKWULoaaNsBQzhBXAU8AZ5RSJ92P/TV9t9ggIi8B3wN+pZQ6AyjgGyJSP/Jxh6e7u5sDBw5w4MABent7WbhwIbm5uUyYMMHX0bzm9OnTvP76656pgBUrVui+hJr2CYZylbmYviL3aWMqAb/ducZut3saL3R1dRm+B2NbWxs9PT1MmjSJlJQUw08FaNpIMd59JAM4HA5P44X29nbS0tLIz88fFz0Yp06dypNPPklERAQbN270dTRNGxMMWRCdTqen8UJLSwvJyck8/PDDTJs2zdfRvKKnp8czFWC321m4cCFr1671dSxNG3MMVRBFxLPfb2NjI1OnTmXbtm2kpKQY+t66Y8eOUVhYaPipAE3zNkMURBHh0qVLWK1WamtriY+P59FHH2XWrFmGLIQOh4Pjx48TGRlJeno62dnZpKSkGHYqQNNGy5guiCLiabxw48YNYmJieOCBB5g3b54hC2F/D8bCwkJaWlpYvHgx6enpWCwWXQw1bQSM2YJYXl5OQUEB165dIyoqim3btrFo0SLD3ld35coV3nvvPRoaGpgyZQpbt24lNTXV17E0zVDGXEGsrKzEarVSUlJCeHg4mzdvJjMz07CNF1wuF2azmZ6eHgICAgzdg1HTfG3MVJG6ujqsVisXLlwgJCSEu+66i6VLlxpyv18RobS0lIKCAtLT01m9ejXz5s0z7FSApvkLvy+IjY2NFBUVcfr0aQIDA8nNzWX58uWGbbxw/fp1CgoKKCsrIyoqiujoaABdCDVtFPhtQWxtbaWoqIgTJ05gNptZuXIlq1atIjQ01NfRvMZms1FUVER4eLihezBqmr/yu1dbR0eHZ79fESErK4vVq1cTERHh62heUVdXR3BwMBEREaSlpWGxWFi6dKlhezBqmj/zm4LY1dXF/v37OXToEL29vZ79fvvfMhpNU1MThYWFnD59muzsbDZv3szUqVOZOnWqr6Np2rjl84Jot9s5ePAg+/fvp7u7m/nz55Obm8ukSZN8Hc0rWltbPT0YTSYTK1asICcnx9exNE3DhwVRRDhw4ADFxcV0dHQwe/Zs8vLymDx5sq8ijYrCwkJOnTrFkiVLWLNmjWGnAjRtLLrlrnveMm3aNHnqqadITU0lPz/fsG8Vu7u72b9/P3PmzCEhIYHW1lacTqehezBqmi95ddc9bzGbzTz55JOkpKT4KoJX2e12Dh06xL59++ju7iYoKIiEhATDduXWNCPwWUHsb15qRMeOHcNqtdLR0cGsWbPIz883/FSAphmBzy+qGIXT6cRkMnl2souLiyM/P9+wPRg1zYh0QRwml8vF2bNnKSwsZMOGDaSnp7N27VrDNpnQNCPTBfEOiQgXL17EZrN5ejD2LyfUxVDTxiZdEO/Qb3/7W86fP2/4HoyaNp7ogngbysvLSUxMJCAggPnz55OWlmboHoyaNt4MZaP6acArQDwgwC4R+b83GZcLPA9YgHoRMcwuRwN7MN5zzz1kZ2czd+5cX8fStFHVeOUkl1/7CZ11N5g0bzmz7v8ywVHGWlE2lDNEB/A/ROS4UioCOKaU+pOInO8foJSKBn4C3C0i5UqpOO/EHV21tbXYbDYuXLhAaGgoGzZsYPHixb6OpWmjrv7CYQ59/4s47d0AdNSUU330A/L+8R0soeE+TjdyhrJRfRVQ5f64TSl1AZgCnB8w7DHgNREpd4+r9ULWUffWW29RW1tLbm4uK1asICgoyNeRNM0nzv/HP3mKIYA4HfR2tFJR/CYpGx7zYbKRdVtziEqpZCADODToqVmARSlVCEQA/1dEXrnJ1z8NPA2QlJR0B3G9q6WlheLiYnJzcwkLC2Pbtm2EhoYaugejpg1FR/W1jz3mtHfTev2SD9J4z5ALolIqHPgd8JyItN7kOEuAdUAIcEApdVBELg8cJCK7gF0AWVlZvllEfRPt7e0UFxd7ejCmpqYyZ84cw3bc0bTbFZU8l/qzBz7ymDkohAlpGT5K5B1DKohKKQt9xfDfROS1mwypABpEpAPoUErtARYBl28y1m+ICDabjQMHDuB0Olm0aJGhezBq2p2at/OvKP7bnbh67YjTgTkwmNDYKUxZvsnX0UbUUK4yK+AXwAUR+edPGPYG8GOlVAAQCCwDfjhiKUeY0+nEbDajlKKxsdHTeiwmJsbX0TTNL0VNTyf3737P1ff+lY7qMmIX5jA97wHMgcaaV79l+y+lVA6wFzgDuNwP/zWQBCAiL7nH/SXwOfeYn4vI85923KysLDl69Ohwst+23t5ejh49yr59+3jyySeJjY3F5XLp+wg1zUC82v5LRIqBWy7BEJF/BP7xTkJ4m9Pp5MSJE+zZs4fW1lZmzJjheU4XQ03T+hl+pYrL5eLll1+mtraWpKQk7r//fpKTk30dS9M0P2TIgti/0XtKSgomk4msrCwmTJjAzJkz9XpjTdM+kaEKoohQUlKC1WqlqqqKz372syQnJ7N06VJfR9M0bQwwTEEsKyujoKCA69evM2HCBLZv3+6XN39rmua/DFEQHQ4Hu3fvRinFli1byMjIwGw2+zqWpmljzJgtiDU1NRw5coRNmzYREBDAzp07mTRpEhaLxdfRNE0bo8ZcQWxoaMBms3Hu3DmCgoLIyspi8uTJJCQk+Dqapmlj3JgpiHa7nffee4+TJ09iNpvJyclh5cqVhISE+DqapmkG4fcFsX+ZncVioaamhqVLl5KTk0N4uHF6sGma5h/8tiB2dnayb98+zpw5wzPPPENwcDBPPfWUXlmiaZrX+F1B7Onp4eDBg+zfvx+73c6CBQtwOByAXmanaZp3+VVB7Ojo4MUXX6Szs5M5c+aQl5dHXJwhdiPQNG0M8HlBdDqdXL9+neTkZMLCwli6dCmzZs0iMTHR19E0TRtnfFoQT5w4QVFREa2trTz33HNERkaSm5vry0iapo1jPpuUq62t5Y033iA0NJTHHnuMiIgIX0XRNE0DfHiGqJTi4YcfJj09XXeg0TTNL/isIMbGxjJnzhxffXtN07SP0fexaJqmuemCqGma5qYLoqZpmpsuiJqmaW66IGqaprndsiAqpaYppWxKqfNKqXNKqWc/ZWy2UsqhlHpgZGNqmv/pbq7j7G++z57//TAnf/5tOmqu+zqSNkxDue3GAfwPETmulIoAjiml/iQi5wcOUkqZge8Df/RCTk3zKz0tDRR+8z56O9sQRy8tpeeo3P8H1vyf3xKekOLreNoduuUZoohUichx98dtwAVgyk2GfhX4HVA7ogk1zQ+V/vHfcHS2I45eAMTlxGnv4tJrP/FxMm04bmsOUSmVDGQAhwY9PgXYDvx0xJJpmh9rKjmNy2H/yGPictFy9ZyPEmkjYcgFUSkVTt8Z4HMi0jro6eeBb4iI6xbHeFopdVQpdbSuru62w2qav4iesQBTQOBHH1QmolLm+iaQNiKGVBCVUhb6iuG/ichrNxmSBfynUqoMeAD4iVLqvsGDRGSXiGSJSFZsbOydp9Y0H0vd+DgBoeEos3uXR5OZgKAQZu/4im+DacNyy4sqqq/zwi+ACyLyzzcbIyIpA8b/CnhbRH4/Qhk1ze8ERcWQ+3evU/L2L2m8dJzI6bNJ2/pnhMUn+TqaNgxDucq8CngCOKOUOul+7K+BJAAReck70TTNvwVPiGP+E9/0dQxtBN2yIIpIMTDk/lwi8tnhBNI0TfMVvVJF0zTNTRdETdM0N10QNU3T3HRB1DRNc9MFUdM0zU0XRE3TNDddEDVN09x0QdQ0TXPTBVHTNM1NF0RN0zQ3XRA1TdPcdEHUNE1z0wVR0zTNTRdETdM0N10QNU3T3HRB1DRNc9MFUdM0zU0XRE3TNDclIr75xkq1AZd88s2HZhJQ7+sQn0LnGx5/zufP2cD/880WkYg7+cKhbDLlLZdEJMuH3/9TKaWO6nx3Tue7c/6cDcZGvjv9Wv2WWdM0zU0XRE3TNDdfFsRdPvzeQ6HzDY/Od+f8ORsYOJ/PLqpomqb5G/2WWdM0zW1UCqJSyqyUOqGUevsTnn9IKXVeKXVOKfXvo5FpqPmUUklKKZv7+dNKqc2jnK1MKXVGKXXyZlfPVJ8XlFIl7nyZfpZvpzvXGaXUfqXUIn/KN2BctlLKoZR6wJ+yKaVy3c+fU0oVjVa2oeRTSkUppd5SSp1y5/vcKOeLVkrtVkpdVEpdUEqtGPT87b82RMTrf4CvA/8OvH2T59KAE8AE9+dxo5HpNvLtAr7s/nguUDbK2cqASZ/y/GbgXUABy4FDfpZv5YB/203+ls89xgxYgT8AD/hLNiAaOA8kuT8f1dfGEPL9NfB998exQCMQOIr5fg18wf1xIBA96Pnbfm14/QxRKTUVuAf4+ScM+TPgRRFpAhCRWm9nGmgI+QSIdH8cBVSORq7bsA14RfocBKKVUgm+DtVPRPb3/9sCB4GpvszzCb4K/A4Y1Z+9IXgMeE1EymH0XxtDIECEUkoB4fQVRMdofGOlVBSwBvgFgIjYRaR50LDbfm2Mxlvm54G/Alyf8PwsYJZSap9S6qBS6u5RyDTQ83x6vr8BHldKVdB3BvHV0YnlIcAflVLHlFJP3+T5KcD1AZ9XuB8bLbfKN9BT9P3GHk2fmk8pNQXYDvx0lHPBrf/uZgETlFKF7jGf8bN8Pwbm0HeScAZ4VkQ+6XU00lKAOuBf3NNZP1dKhQ0ac9uvDa8WRKXUFqBWRI59yrAA+t425wKPAj9TSkV7M1e/IeZ7FPiViEyl7xT8X5VSo3kxKkdEMul7u/kVpdSaUfzeQzGkfEqpPPoK4jdGMxy3zvc88I1RfCEPdKtsAcAS+t7BbAT+t1Jqlh/l2wicBBKBxcCPlVKRjI4AIBP4qYhkAB3AN4d7UG+/sFcBW5VSZcB/AvlKqd8MGlMBvCkivSJSClymr0COhqHkewp4FUBEDgDB9K3lHBUicsP931rgdWDpoCE3gGkDPp/qfmxUDCEfSqmF9E1JbBORhtHKNsR8WcB/un8GHgB+opS6z0+yVQDvi0iHiNQDe4BRuyg1hHyfo+8tvYhICVAKpI9SvAqgQkQOuT/fTV+BHOi2XxteLYgi8i0RmSoiycAjgFVEHh807Pf0nR2ilJpE39uEq97MdZv5yoF17nxz6CuIdaORTykVppSK6P8Y2ACcHTTsTeAz7itqy4EWEanyl3xKqSTgNeAJEbk8GrluJ5+IpIhIsvtnYDfwjIj83h+yAW8AOUqpAKVUKLAMuODtbLeRb+BrIx6Yzei9dquB60qp2e6H1tF3AWqg235t+KS5g1Lqu8BREXkTeB/YoJQ6DziBvxzts4hb5Psf9L2N/wv65lQ+K+5LWKMgHni9b86aAODfReQ9pdSXAETkJfrmNTcDJUAnfb+1R8tQ8n0biKHvzAvAIaPXGGAo+XzlltlE5IJS6j3gNH1z3D8XkcFFyWf5gO8Bv1JKnaHvSu433Geyo+WrwL8ppQLpK8SfG+5rQ69U0TRNc9MrVTRN09x0QdQ0TXPTBVHTNM1NF0RN0zQ3XRA1TdPcdEHUNE1z0wVR0zTNTRdETdM0t/8fUdE7XUG+brcAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane.html\n", "from sklearn import svm\n", "\n", "# fit the model, don't regularize for illustration purposes\n", "clf = svm.SVC(kernel='linear', C=1000) # we'll explain this algorithm shortly\n", "clf.fit(X, iris_y2)\n", "\n", "plt.figure(figsize=(5,5))\n", "plt.scatter(X[:, 0], X[:, 1], c=iris_y2, s=30, cmap=plt.cm.Paired)\n", "Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", "\n", "# plot decision boundary and margins\n", "plt.contour(xx, yy, Z, colors='k', levels=[-1, 0, 1], alpha=0.5,\n", " linestyles=['--', '-', '--'])\n", "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=100,\n", " linewidth=1, facecolors='none', edgecolors='k')\n", "plt.xlim([4.6, 6])\n", "plt.ylim([2.25, 4])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## 12.1.3. The Functional Classification Margin\n", "\n", "How can we define the concept of margin more formally?\n", "\n", "We can try to define the margin $\\tilde \\gamma^{(i)}$ with respect to a training example $(x^{(i)}, y^{(i)})$ as\n", "\n", "$$ \n", "\\tilde \\gamma^{(i)} = y^{(i)} \\cdot f(x^{(i)}) = y^{(i)} \\cdot \\left( \\theta^\\top x^{(i)} + \\theta_0 \\right). \n", "$$\n", "\n", "We call this the *functional* margin. Let's analyze it." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "We defined the functional margin as\n", "\n", "$$ \n", "\\tilde\\gamma^{(i)} = y^{(i)} \\cdot \\left( \\theta^\\top x^{(i)} + \\theta_0 \\right).\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* If $y^{(i)}=1$, then the margin $\\tilde\\gamma^{(i)}$ is large when the model score $f(x^{(i)}) = \\theta^\\top x^{(i)} + \\theta_0$ is positive and large." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* Thus, we are classifying $x^{(i)}$ correctly and with high confidence." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* If $y^{(i)}=-1$, then the margin $\\tilde\\gamma^{(i)}$ is large when the model score $f(x^{(i)}) = \\theta^\\top x^{(i)} + \\theta_0$ is negative and large in absolute value." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* We are again classifying $x^{(i)}$ correctly and with high confidence." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "Thus higher margin means higher confidence at each input point. However, we have a problem. \n", "\n", "* If we rescale the parameters $\\theta, \\theta_0$ by a scalar $\\alpha > 0$, we get new parameters $\\alpha \\theta, \\alpha \\theta_0$ .\n", "\n", "* The $\\alpha \\theta, \\alpha \\theta_0$ doesn't change the classification of points.\n", "\n", "* However, the margin $\\left( \\alpha \\theta^\\top x^{(i)} + \\alpha \\theta_0 \\right) = \\alpha \\left( \\theta^\\top x^{(i)} + \\theta_0 \\right)$ is now scaled by $\\alpha$!\n", "\n", "It doesn't make sense that the same classification boundary can have different margins when we rescale it." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.1.4. The Geometric Classification Margin\n", "\n", "We define the *geometric* margin $\\gamma^{(i)}$ with respect to a training example $(x^{(i)}, y^{(i)})$ as\n", "\n", "$$ \n", "\\gamma^{(i)} = y^{(i)}\\left( \\frac{\\theta^\\top x^{(i)} + \\theta_0}{||\\theta||} \\right). \n", "$$\n", "\n", "We call it geometric because $\\gamma^{(i)}$ equals the distance between $x^{(i)}$ and the hyperplane." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* We normalize the functional margin by $||\\theta||$ \n", "\n", "* Rescaling the weights does not make the margin arbitrarily large." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "Let's make sure our intuition about the margin holds.\n", "\n", "$$ \n", "\\gamma^{(i)} = y^{(i)}\\left( \\frac{\\theta^\\top x^{(i)} + \\theta_0}{||\\theta||} \\right). \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* If $y^{(i)}=1$, then the margin $\\gamma^{(i)}$ is large when the model score $f(x^{(i)}) = \\theta^\\top x^{(i)} + \\theta_0$ is positive and large." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* Thus, we are classifying $x^{(i)}$ correctly and with high confidence." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* The same holds when $y^{(i)}=-1$. We again capture our intuition that increasing margin means increasing the confidence of each input point." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### 12.1.4.1. Geometric Intuitions\n", "\n", "The margin $\\gamma^{(i)}$ is called geometric because it corresponds to the distance from $x^{(i)}$ to the separating hyperplane $\\theta^\\top x + \\theta_0 = 0$ (dashed line below).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Suppose that $y^{(i)}=1$ ($x^{(i)}$ lies on positive side of boundary). Then:\n", "1. The points $x$ that lie on the decision boundary are those for which $\\theta^\\top x + \\theta_0 = 0$ (score is precisely zero, and between 1 and -1)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "2. The vector $\\frac{\\theta}{||\\theta||}$ is perpendicular to the hyperplane $\\theta^\\top x + \\theta_0$ and has unit norm (fact from calculus)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "\n", "3. Let $x_0$ be the point on the boundary closest to $x^{(i)}$. Then by definition of the margin\n", "$x^{(i)} = x_0 + \\gamma^{(i)} \\frac{\\theta}{||\\theta||}$ or\n", "\n", "$$ \n", "x_0 = x^{(i)} - \\gamma^{(i)} \\frac{\\theta}{||\\theta||}. \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "4. Since $x_0$ is on the hyperplane, $\\theta^\\top x_0 + \\theta_0 = 0$, or\n", "\n", "$$\n", "\\theta^\\top \\left(x^{(i)} - \\gamma^{(i)} \\frac{\\theta}{||\\theta||} \\right) + \\theta_0 = 0.\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "5. Solving for $\\gamma^{(i)}$ and using the fact that $\\theta^\\top \\theta = ||\\theta||^2$, we obtain\n", "\n", "$$ \n", "\\gamma^{(i)} = \\frac{\\theta^\\top x^{(i)} + \\theta_0}{||\\theta||}. \n", "$$\n", "\n", "Which is our geometric margin. The case of $y^{(i)}=-1$ can also be proven in a similar way." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "We can use our formula for $\\gamma$ to precisely plot the margins on our earlier plot." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAATwAAAEzCAYAAACsU7/TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABJAElEQVR4nO3deVyU573//9c17PuqiAgqLrjiAm6AiGwuSU3UxJiYvY1t2iZpT79t2uZ0OT09vzSnPTk9XbJ1b7MnTdIsTRh2RAXFDfcNFVwQZN+Z5fr9MUgJooLMcM8M1/PxyCMwc3HPRxjeXMt9X7eQUqIoijIa6LQuQFEUZaSowFMUZdRQgacoyqihAk9RlFFDBZ6iKKOGCjxFUUaNQQeeEMJFCLFPCPHxAM95CCHeEkKcEkKUCiEmWbVKRVEUKxhKD+8p4Oh1nvsi0CClnAr8L/DccAtTFEWxtkEFnhBiAnAb8PvrNLkD+EvPx+8CaUIIMfzyFEVRrGewPbxfAt8BzNd5PgKoApBSGoEmIGS4xSmKoliT680aCCFuB2qklHuEECnDeTEhxFZgK4CPj0/cjBkzhnM4RVGUa+zZs+eKlHLMQM/dNPCARGCdEGIt4An4CyFelVLe36fNBSASOC+EcAUCgLr+B5JSvgK8AhAfHy/LysqG9i9RFEW5CSHEues9d9MhrZTye1LKCVLKScBmIK9f2AF8CDzU8/FdPW3UrgSKotiVwfTwBiSE+AlQJqX8EPgD8DchxCmgHkswKoqi2JUhBZ6UsgAo6Pn4h30e7wTutmZhiqIo1qautFAUZdRQgacoyqihAk9RlFFDBZ6iKKOGCjxFUUYNFXiKoowaKvAURRk1bvnEY0WxB1JKLjR3cvpKGwBTQn2I8PdEbdajDEQFnuLQyi81c7y2FZPZciXj5dYuYsb4Mm98gMaVKfZIDWkVh9VlNHOspqU37ABMZsmxmha6jNfbyUwZzVTgKQ6rudOAywBDVxchaO40aFCRYu9U4CkOy8/DFdMAm/KYpMTPQ83WKNdSgac4LE83F6aE+OCi+1cvz0UnmBLig6ebi4aVKfZK/RlUHFrchEBCvN052bNKOy3Uh0nB3hpXpdgrFXiKQxNCMDnEh8khPlqXotiBm+07rIa0iqI4jYqKihs+rwJPURSH1tjYyPHjxwGIjo6+YVs1pFUUxSF1dnaybds2SktL8fT0ZMqUKbi63jjSVOApiuJQTCYTe/bsoaCggI6ODmJjY0lLS7tp2IEKPEVRHMzly5f55z//yeTJk8nMzCQ8PHzQX6sCT1EUu3fx4kUqKytZunQp48ePZ+vWrYSHhw95kwgVeIqi2K2mpiZyc3MpLy/H19eXBQsW4OHhwfjx42/peDcNPCGEJ1AEePS0f1dK+aN+baKAvwCBgAvwXSnlP2+pIsUmOgwmDCYzfh6uauskxe51dXVRXFzMzp07AUhKSmL58uV4eHgM67iD6eF1AalSylYhhBtQLIT4VEpZ0qfNvwNvSylfFELMAv4JTBpWZYpVGExmtp+t53JLJ0II3HSCxMkhjPUd3htHUWyps7OTkpISZs2aRVpaGgEB1tnu66aBJy2nLrf2fOrW81//05kl4N/zcQBw0SrVKcO2u6qByy2dmCUgJSazpPD0Fe6YHY67qzoNU7EPUkpOnDjByZMnue222wgICOCpp57C19fXqq8zqDk8IYQLsAeYCvxWSlnar8mPAb0Q4gnAB0i3ZpHKrTFLSVVjB+YBrrY539RBtLocS7EDly5dQq/Xc+bMGUJCQujo6MDb29vqYQeDDDwppQmYL4QIBN4XQsyRUh7q0+Re4M9Syv8RQiwD/tbT5nO7MAohtgJbAaKioqzyD1BubKBLC6W0hKGiaKm9vZ2srCzKy8vx8vJi7dq1xMXF4eJiu51uhrRKK6VsFELkA6uBvoH3xZ7HkFLu7FnoCAVq+n39K8ArAPHx8eo3zsZ0QjDOz4Pqlq5+cxCSiAAvjapSFAsXFxfOnTtHQkICy5cvx9PT0+avedNJHCHEmJ6eHUIILyADONavWSWQ1tNmJuAJ1Fq1UuWWLJkYjJ+nK646gatO4CIES6KC8FL7xSkjzGw2U1ZWxp///GdMJhMeHh58/etfJyMjY0TCDgbXwwsH/tIzj6fDshr7sRDiJ0CZlPJD4FvA74QQ38SygPGwvNk+LcqI8HJzYe2MMOo7DHQbzYT6uOPmohYrlJEjpeTUqVPo9Xpqa2uJioqivb0dPz+/QV0OZk2DWaUtBxYM8PgP+3x8BEi0bmmKtQghCPF217oMZRRqb2/n3XffpaKiguDgYO655x5mzJih2bmg6koLRVGszmQy4eLigqenJ1JK1qxZQ3x8vE0XJAZDBZ6iKFbT3d3N9u3b2b9/P48//jienp48+OCDdnN1jwo8RVGGzWw2s3//fvLy8mhtbWXOnDkYjUYAuwk7UIGnKMowdXZ28sc//pGamhoiIyPZvHkzEyZM0LqsAanAUxTllrS1teHj44OnpyeTJk0iJSWFmTNn2lWPrj8VeIqiDElLSwv5+fmUl5fz1a9+leDgYNauXat1WYOiAk9RlEHp7u5mx44dbN++HbPZzOLFi/HycqwrdlTgKZroMJg4WdtKU6eBMF8PokN8cFUnRNsto9HICy+8QGNjI7NmzSI9PZ3g4GCtyxoyFXjKiGvtMvLZ8cuYzBKzhEvNnZy40sbqmLEq9OxMdXU148aNw9XVlWXLlhEeHu7QG3+od5cy4g5cbMJokr3bVpkktHcbqahv07YwpVdNTQ2vvfYaL730EpWVlQAsWbLEocMOVA9P0cCVtu5rdpA1Sahp7WL6GD9NalIsWltbyc/PZ+/evXh4eJCZmXnL94+wRyrwlBHn5+FKu8H0ucd0AgI83TSqSAHLycO/+93vaGlpYcmSJSQnJ+Pt7a11WValAk8ZcbHj/ck7eQVTnw11XHSCqaHW3+FWuTEpJceOHWPGjBnodDpuu+02QkJCCAkJ0bo0m1CBp4y4UB8PUqeFUn6pmZZOI2N83YkND1B79I2wM2fOkJWVRXV1NZs3b2bGjBlMnz5d67JsSgWeoolQHw9Sp47RuoxRqba2luzsbE6cOEFgYCAbN24kJiZG67JGhAo8RRlFpJS8/fbbtLS0kJGRwZIlS0Z8E05buRrkN+Ic/1JFUa7LYDBQVlZGXFwc7u7ubNiwgYCAAKdZkGhra6OgoIA9e/bg5nbjhS8VeIripKSUlJeXk5eXR1NTE76+vsydO5fw8HCtS7MKg8FASUkJxcXFGAwG4uPjWbFiBd///vev+zUq8BTFCZ09exa9Xs/FixcZP34869evZ9KkSVqXZRVSSg4ePEhubi5NTU3ExMSQkZFBaGjoTb9WBZ6iOKHCwkLa2trYsGEDc+fOtestm4aib5CHh4cPOchV4CmKE2hra6OoqIjExET8/f1Zv349Xl5eN53TchRXrlwhOzub48ePExAQwPr164mNjR1ykKvAUxQHZjQaKSkpYdu2bRgMBsaPH8+8efPw9/fXujSraGtro7CwkLKyMtzc3EhLS2Pp0qW3HOQ3DTwhhCdQBHj0tH9XSvmjAdptAn6M5b60B6SU991SRUovs5R0Gc24u+hw0TnHkESxnkOHDpGdnU1TUxPTp08nIyODMWOc49zG/kEeFxdHSkoKPj4+wzruYHp4XUCqlLJVCOEGFAshPpVSllxtIISYBnwPSJRSNgghxg6rKoXKhnZ2VzViMptBCGaO9WXOOH+nmYtRhu/06dN4e3tz5513MnnyZK3LsQopJYcOHSI3N5fGxkarB/lgbsQtgdaeT916/uu/2cVjwG+llA09X1NjlepGqYb2bkrONfzrWlMpOVrTio+7C9Eh6nrT0aquro6cnBySkpKIiIhgzZo1uLm5Oc0fwXPnzqHX67lw4QLjxo3joYcesnqQD2oOTwjhAuwBpmIJttJ+Tab3tNsOuAA/llJ+Zs1CR5NTdW2Y5ef/ppjMkuM1rSrwRqH29nYKCwvZvXs3rq6uzJo1i4iICNzd3bUuzSquBvnRo0d7F1xuZUFiMAYVeFJKEzBfCBEIvC+EmCOlPNTvONOAFGACUCSEmCulbOx7HCHEVmAr4PAbCdqSwWS+pgsNYDQP9KjizHbt2kVeXh5dXV2981i+vs7xR69/kKemprJs2TKbriwPaZVWStkohMgHVgN9A+88UCqlNABnhBAnsATg7n5f/wrwCkB8fLz67b2OiUHenG/qxNQn4HQCooKc41Ig5cZkT+9eCEFHRweRkZFkZmY61YLErl27KCoqGvEgH8wq7RjA0BN2XkAG8Fy/Zh8A9wJ/EkKEYhniVli51lFjvL8nk4K8OVvf1tutD/RyY/Y4tRuws6usrESv15OQkMCsWbNITk52mjk6KSWHDx8mJyeHxsZGpk2bRkZGBmPHjtwa52B6eOHAX3rm8XTA21LKj4UQPwHKpJQfAllAphDiCGACvi2lrLNZ1U5OCMHiqCBmjvWlrt2An4crwd7OMzmtXKu+vp6cnByOHDmCn59f78/aWX7mVVVVZGVlcf78ecLCwnjwwQeJjo4e8TqElNqMLOPj42VZWZkmr60o9qS4uJj8/HxcXFxITExk2bJlTrMg0T/IU1NTmTdvHjqd7e4fJoTYI6WMH+g5daWFomjAaDQihMDFxYWAgADmzZvHypUr8fNzjmmLjo4OioqK2LVrFy4uLqxcudIuglwFnqKMICklR44cIScnh0WLFpGQkMDcuXOZO3eu1qVZhdFoZPfu3RQWFtLV1cWCBQvsKshV4CnKCKmqqkKv11NVVUVYWBjjxo3TuiSr6RvkDQ0NTJkyhczMTMLCwrQu7XNU4CnKCCgoKKCgoABfX1/WrVvH/PnzbTqPNZLOnz9PVlYWVVVVjB07lvvvv5+pU6dqXdaAVOApio10dHQA4OXlRXR0NFJKEhMTNZ/HspaGhgZycnI4fPiwwwS5CjxFsTKTydQ7jzVnzhxuu+02oqKinObqoo6ODrZt20ZpaSk6nY4VK1Y4TJCrwFMUK7l6U+vs7Gzq6+uJjo4mLi5O67Kspm+Qd3Z2Mn/+fFauXOlQe++pwLNTJrOkqrGDmtYuAjxdmRzsg7vrrQ0V2rqNVNS10Wk0MyHAi3F+Hk5zQqs1Dfd7XlBQQGFhIWPGjGHLli1MnTrVKb7PAwV5ZmamQy66qMCzQyazJPtEDc1dRkxmiYsQHL7cwuqYsXi7D+1HVtvaRf7pK0gpMUs4U99OVKAXSycG26h6x3Sr3/PGxkaklAQFBTF//nz8/f1ZsGCBXc9jDcWFCxfIysqisrLSKYJcBZ4dOlvfTnOnsXc/PJOUmI2S8kvNQw6qXZUNn9uEwGSWVDZ0EDOmmyBv+59zGSlD/Z53dnaybds2SkpKmDZtGps3byYoKMhphrCNjY3k5ORw6NAhfH19+cIXvuAUQa4Czw5daun81+afPSRwuaVrSMcxmSXNXcZrHpdIattU4PU12O+5yWSirKyMwsJCOjo6mDdvHqmpqSNYqW31DXKdTkdycjKJiYl4eHhoXZpVqMCzQ34erugE9N/+zsfdZUjH0QlwcxEYTLLf42LIx3J2g/2e79y5k5ycHCZPnkxmZqbT3NTaZDKxZ88eCgoKPhfkjrQgMRgq8OzQtFAfTtS2fm7XYxchmBs+tDefEILZYf4crG7uHdYKwNNVR7i/pzVLdng3+p5fvHgRk8lEZGQk8fHxjB07lmnTpjnsPFZfUkqOHz9OdnY2dXV1Thfk/andUuxUU4eBfRcaqWvvxsfDlfnhAYy7hZCSUlJR18aRmlYMRjPjAzyZPz4ATzfVw+uv//c82hsO7d5OeXk5kydP5qGHHtK6RKu6ePEier2es2fPMmbMGDIyMpwiyG+0W4oKPEXpp7Ozk+LiYkpKLDfmW7ZsGUlJSU4zj9XU1ERubi7l5eX4+PiQkpJCXFycwy9IXKW2h1KUITh8+DDFxcW981gBAQFal2QVXV1dFBcXs3PnTgCWL1/ulEF+IyrwlFFPSsmJEycwGo3Mnj2bBQsWEBER4ZAn1g7EZDKxd+9eCgoKaGtrIzY2lrS0NKcJcrPZTH5+fm+Q34gKPGVUu3TpEnq9njNnzjBx4kRmzZqFTqdzirC7GuTZ2dlcuXKFiRMnsmXLFsaPH691aVYhpUQIgRCCixcvMmvWLNLS0vjBD35w3a9RgaeMSs3NzeTm5nLgwAG8vb1Zu3YtcXFxDj9hf1XfIA8JCWHz5s3ExMQ4xb9PSsnJkyfJz89n8+bNBAQEcN999+HicvOFOBV4yqhUU1PD4cOHSUpKIikpCU9P5zhNp6mpiby8PMrLy/Hy8uoN8sGEgSPoH+Stra0EBAQM+t+nVmkVh1ddXc3+/fsBmD9//oDDUbPZzN69e+ns7CQpKQmA1tZWp7mpdf8FiSVLlrB8+XKnCXIpJR9++CH79+/Hy8uLFStWEB8fP2DQqVVaxSlVVFTw3e9+l+zsbBYtWgRAWVkZGRkZPPvss72bbp46dQq9Xk9tbS1TpkwhMTERIYRThN3VIM/Pz6etrY25c+eSlpZGYGCg1qVZhdFoxNXVFSEE7u7uJCQkDCvIB3Mjbk+gCPDoaf+ulPJH12m7EXgXWCSldKjum5QSkwQX4Tz3ArUVk1kihOUSNa2cOHGClJQUnnjiCX7/+9/j23OTmNaWFn7729+SlJTE+++/z/Hjx6moqCA4OJh77rmHGTNmOMXPt3+QR0VFcd999xEREaF1aVZhNpvZt29f7zzdhAkTWLNmzbCPO5geXheQKqVsFUK4AcVCiE+llCV9Gwkh/ICngNJhVzXCjte0cLC6GaNJ4u3uwqLIIHXp1QCaOg2UnKunod2ATsCUEB8WTAjUJPgeeeQR/v3f/50vbv0ypZUNXDx9AYDxAZ7827e/Q0BAAE8++SR33XUXa9asue7wxxFVV1ej1+udOsizs7OpqakhKioKV1frDURveiRpmeRr7fnUree/gSb+/hN4Dvi21aobAZUN7Ry42Ny7U0Zbt4ltFXWsmjGWAE83jauzH0azmZwTNXSbrm6fBKfr2kEI4iYEjmgt+/bto6qqiq1bt5J3uo6G9m4kYDR0k5OzgzxDO//+5a08++yzrFy5kvj4AadzHE5zczP5+fns378fT09PpwtygHfeeYcjR47YLMgHFZ1CCBdgDzAV+K2UsrTf8wuBSCnlJ0IIhwq8I5dbrtkWyCwlJ2tbiY8M0qgq+3O+sfOanURMUnL6SisLIwJGtHeRk5PD+vXraTdKGjsMGM1m3nulkfOnK5gwpYSJU2No7TKyYcMGCgoKHD7wuru72b59Ozt27MBsNrNs2TKSk5OdZkGitbUVHx8fhBBMmjSJqKgoFi1aZJMgH1TgSSlNwHwhRCDwvhBijpTyEIAQQgc8Dzx8s+MIIbYCWwG7uaFJt8l8zWMS6DJe+/hoZjCZGWhF3yQt36+RHEx1d3fj4+NDt8lMU30tO/SfsOOzO7l8/gskrI5hWYY/RgTe3t50dQ1tD0F70nceq7W1lTlz5pCWlkZQkHP8Ie4b5HfccQdz5sxh8eLFNn3NIV0tLKVsBPKB1X0e9gPmAAVCiLPAUuBDIcQ1f1allK9IKeOllPFjxoy55aKtKTLAC12/31ZXnSAy0EubguzUOH/PAecxQrzdR3wOLyYmhp07dxLk5Y6Xtw8uLi48/p91jI0w86efTae+RkeQlzulpaXExMSMaG3WcurUKV566SU++ugjgoKC+NKXvsRdd93lFGF3dWX5V7/6FYWFhcTExIzYYstgVmnHAAYpZaMQwgvIwDJXB4CUsgkI7dO+APh/jrJKOyfcn0stnbR1m3p6MIJwf08VeP34ebgyNzyAQ5eaAIEQlv3ilk4c2V/AlpYWhBDs2bOHQwfLSZsVg5vH/Qid4Nv/W88zD4zhd8+EERd0iKNHj7Ju3boRrW+4Ll++jF6v5/Tp0wQFBbFp0yZmzpzpFAsSV7355pucOHGCyMjI3hXYkXLTE4+FELHAXwAXLD3Ct6WUPxFC/AQok1J+2K99AYMIPHs68VhKSXVLF61dRkJ83AlWW59fV2uXkeqWTtxddEQEeOHSv3tsI93d3ezYsYPt27djNptpaGjgH//4B9nZ2URETeJCk+Wm17tyvbj3Hh1+fm/yy1+28+ijj45IfcPV0tJCfn4++/btw9PTk+TkZBYtWmTVFUot1dTUEBQUhJubGydOnMBgMDBr1iybBPmwTjyWUpYDCwZ4/IfXaZ8y1AK1JoRQp6EMkq+HK1M9RvaE3cuXL/Pqq6/S0tLC7NmzSU9PJygoiOjoaBYuXMi6detYvdoyy/LZZ5/h4RFLS8v/w2AY0TJvSf8gX7p0KcnJyXh5OccIo2+Qp6enk5iYyPTp0zWrR11aptittrY2fHx8MBqNvP/++yxdupTIyMjPtamrq+NPf/oTV99LixYt4oEHHubhh0PIyYGCAkhI0KD4mzCbzRw4cIC8vLzeIE9LSyM42Dlun9nd3c3OnTvZvn07JpOJxYsXj1iQqx2PFYdSU1PTewXB17/+ddzchn4+ZEMDLFoEbW2wZw/Y045Ip0+fRq/Xc/nyZSZMmMCqVauuCXJH99Zbb3H06FFmzZpFenr6iAa5upZWcQitra3k5+ezd+9ePDw8WLFixS1vOx4UBB98AEuXwl13QX4+aL2xb01NDdnZ2Zw8eZKgoCDuvvtum81jaaGiooKxY8fi6+vLihUrWLZsmd2cfnaVCjzFLtTV1fHyyy9jMplYsmQJycnJeHt7D+uYc+bAn/4EmzbBU0/BSy9Zqdgh6h/kmZmZLF682KkWJK4GeVJSEunp6Xa7gapzfMcVh2Q2m6mtrSUsLIzg4GCWLl3KvHnzCAkJsdpr3H03fPe78LOfQVwcPPaY1Q59UwaDgZ07d1JcXIzRaLRakNuL6wW5PVOBp2iioqICvV5PfX09Tz31FD4+PqSmptrktX76U9i3D77+dZg71zLMtSUpJQcOHCA3N5eWlhZmzpxJenq6VYPcHlzdMXrx4sWsWLHCIYJcLVooI6q2tpbs7GxOnDhBYGAgaWlpzJkzx+bzWPX1lkWMjg7LIoat7jN95swZsrKyqK6uJiIigszMTCZOnGibFxthUkrKy8sJDw9n7NixNDc3YzAY7C7I1aKFYheam5t58cUXcXd3JyMjgyVLlozYPFZwMLz/PixbZhnm5uWBuxXPL+8f5Bs3bhyRIB8pZ86cQa/Xc+nSJZYuXcrq1avx9/fXuqwhU4Gn2JTBYKCiooKYmBj8/f254447mDZtmibDn9hYyyLGPffAN74BL7ww/GO2tbX1zmO5ubmNeJDbWt8gDwgI6A1yR+UcPxXF7lwd/lydx3ryyScJCgpi3rx5mta1aZNlSPvf/21ZxPjiF2/tOAaDgZKSEoqLizEYDMTHx5OSkuIQ81hDUV5ezrlz55wmyNUcnmJ1fYc/48ePJzMzk0mTJmldVi+TCdasgcJCKCqCJUsG/7V9g7y5uZkZM2aQnp5OaGjozb/YAVwN8vDwcKZOnUpXVxdGoxEfHx+tSxs0NYenjJj29nZef/11vL297XYey8UF3nwT4uNh40YoK4PBnDZ29uxZsrKyeoN8w4YNdhXkwyGl5ODBg+Tm5tLU1MSyZcuYOnUqHh4eeGh9xrYVqR6eMmxtbW2Ul5ezdOlShBCcO3eO8ePH39IlYSPpwAHLIkZcHOTmXn8R48qVK2RnZ3P8+HECAgJIS0tj7ty5dhfkt6qyspLPPvuMixcv2mWPfKhUD0+xCYPBQGlpKdu2bcNgMBAdHU1YWJjDnIYxbx788Y9w773wb/8Gv/nN559va2ujsLCQsrIy3NzcSE9PZ8mSJXYf5EN15coV2tra2LBhg1MF+UBUD08Zsv7Dn5iYGDIyMhx2Huvb34Zf/AJCQizn60VGGnnooRK8vCxBHhcXR0pKikPNY93I1SAfO3Ys8fHxmM1mTCaT0wS56uEpVmUwGMjOzsbX15c777yTyZMna13SsMydCzod1NVJ4DSVlR/z7LONbN0aw3/8h+MGeX9Go5HS0lKKioowGAwkJiYCoNPpbnmTBkejenjKoNTV1VFSUsLq1atxcXGhvr6eoKAgpxj+TJoE585VAYGAGfg7sJKJEydz9qyGhVnRqVOn+Pjjj2lsbGT69OlkZGRgL/eVsTbVw1NuWXt7O4WFhezevRtXV1fmz59PRESE02xUWVdXx7lz2cAxYAqQAjwCCCortazMOsxmc28PztPTkwcffJDo6Gity9KMCjxlQCaTiZKSErZt20ZXV1fvPJav78hu724rfYM8MNCVxsZUYBmW+8xb2NlWbkNSV1dHTk4OgYGBrFq1iujoaL785S87RY98OFTgKQPS6XQcOnSIqKgopxr+GI1Gdu3aRVFRUW+Q/+IXKTz5pC/t7f9q5+0N//Vf2tV5q/r3yFesWNH73GgPO1CBp/RRWVlJUVERGzduxMvLi4cffthpTjqVUnL48GFycnJobGxk2rRpZGRkMHbsWAA8PeGZZ6Cy0tKz+6//gi1bNC56iI4dO8YHH3xAV1cXCxcuZOXKlU7TI7cWFXgK9fX1ZGdnc/ToUfz8/Kirq2PChAlOE3aVlZXo9XrOnz/PuHHjBpzH2rLF8QIOLEHe3d2Nh4cHoaGhREVFkZ6e3hvkyucN5kbcnkAR4NHT/l0p5Y/6tfk34EuAEagFHpVSnrN+uYo1mc1m9Ho9u3btwtXVlZUrV5KQkOA052P1D/I777yT2NhYpzkFo6qqiqysLPz8/LjnnnsIDQ3lvvvu07osuzaYHl4XkCqlbBVCuAHFQohPpZQlfdrsA+KllO1CiMeB/wbusUG9ihVIKRFCoNPpaG5uZv78+axcuRI/Pz+tS7OKjo6O3nksFxcXVq5cybJly3C35gZ4GqqvrycnJ4cjR47g5+dHXFxc789UubHB3IhbAq09n7r1/Cf7tcnv82kJcL+1ClSsR0rJkSNHyM/P59577yUkJIS7777baX5RjEYju3fvprCwkK6uLhYsWOBUQQ6Webp33nnHKYN8JAxqDk8I4QLsAaYCv5VSlt6g+ReBT61Qm2JFVVVV6PV6qqqqCAsLo6urC3COlTspJUePHiU7O5uGhgamTp1KZmam08xjGY1GWltbCQwMJCoqiri4OJYvX+5UQT5ShnSlhRAiEHgfeEJKeWiA5+8Hvg6skFJ2DfD8VmArQFRUVNy5c2qaz9aklLz//vuUl5fj6+tLamoq8+fPd6p5rL5BnpmZyZQpU7Quyyr6Brmnpydbt251ij9Qtma1Ky2klI1CiHxgNfC5wBNCpAPPcJ2w6/n6V4BXwHJp2VBeWxma7u5u3N3dEUIQGBhISkoKCQkJTjP8aWhoICcnh8OHD+Pr68u6deucKsjPnz9PVlZWb5CnpaWpsLOCwazSjgEMPWHnBWQAz/VrswB4GVgtpayxSaXKoJhMpt55rE2bNjF58mSb3f5QCx0dHWzbto3S0lJ0Op3TBTnAiRMneP31150yyLU2mB5eOPCXnnk8HfC2lPJjIcRPgDIp5YfAzwFf4J2ev0KVUsp1tipaudbV4U9OTg719fVMmTLFqU467RvknZ2dzJ8/n9TUVKeZx+ro6KC+vp6IiAimTJlCZmYm8fHxThXk9kDtluIk3nzzTY4dO8bYsWPJzMxk6tSpWpdkFVJKjh07RnZ2dm+QZ2ZmEhYWpnVpVtE3yN3d3XnyySdxcXHRuiyHpnZLcVKNjY34+/uj0+mYMWMG06ZNY8GCBU4z/Llw4QJZWVlUVlYyduxYtmzZwrRp07Quyyr6B3l0dDSZmZkq7GxMBZ4D6uzspKioiNLSUm6//XYWLFjA/PnztS7LahoaGsjNzeXQoUP4+vryhS98wamCHODcuXO89dZbjBkzhi1btjB16lS1KDECVOA5EJPJRFlZGYWFhXR0dDBv3jyHPwXjtdf+ddF+ZGQn9967DR+fEnQ6HcnJySQmJjrNNb2NjY1cvHiRWbNmMXHiRO655x5iYmKcKsjtnQo8B/L2229z/Pjx3uHPuMHcW9COvfYabN0K7e0moIzKykKef76Dr31tHv/xH6n4+/trXaJVdHZ2sm3bNkpKSvDw8GDatGm4ubkxc+ZMrUsbddSihZ27cOECoaGheHh4cObMGQwGA9OmTXOK4c/EiZLKypNAKHAUOAVkMnFiuFNsrW4ymdizZw8FBQV0dHQQGxtLWlqa0wS5vVKLFg6osbGR3NxcDh48yMqVK1mxYoXD3yynrwsXLlBZqQfOARuARCABZ9laHSy3P/z000+ZNGkSmZmZhIeHa13SqKcCz85cHf6UllouV16+fDlLly7VuCrr6RvkgYE+NDbeBszpedbSa3XkrdUvXrzImTNnSExMJCwsjC9/+cuEhYU5RY/cGajAszMfffQRhw8fZt68eaSmphIQEKB1SVbR2dlJcXExJSWWXcWWL1/OpElJfO1rHp/bWl0I+O53NSpyGJqamsjNze29ZjkuLg5PT0+Hn2d1NirwNCal5MSJE4SFhREYGMjKlStJTExk/PjxWpdmFSaTib1795Kfn097e/s1Qe7m9q9V2rAwuHIF3nsPvvQlcHWAd2dXV1fvggRYgjwpKclpVpadjQO8pZzXxYsX0ev1nD17loSEBDIzM53mps9Xgzw7O5srV64wadIkVq1adc08Vv+t1f/wB0vYPfMMPPccdq+7u5tdu3Yxe/Zsp+qROyu1SquBvsMfb29vUlJSiIuLc5qz7C9duoRer+fMmTOEhoaSkZHB9OnTBz2P9dWvwosvwltvwaZNNi52iK4G+bFjx1i3bh1CCNra2vDx8dG6NKWHWqW1M8XFxRw5coSkpCSSkpLw9PTUuiSraGpqIi8vj/Lycry8vFi7du0tBfkvfwkHDsAjj8CMGRAba5t6h6pvkIeEhNDW1oavr68KOweiengjwGw2s3fvXsLDw4mIiKCtrQ2DwUBgYKDWpVlFV1cXxcXF7Ny5E4ClS5cOO8gvXYL4eMvtE3fvhuBga1U7dO3t7WRlZfUGubP1yJ2N6uFpRErJyZMnyc7Opra2liVLlhAREeE0PYKrQZ6fn09bWxuxsbGkpqZaJcjDw+Hvf4fkZLjvPvjkExjpfLl6Yxw3NzeqqqpITEx0qh75aKR6eDZSXV1NVlZW7/AnIyODmJgYpzgfq3+QT5w4kczMTCIiIqz+Wr/7neXys+9+F5591uqHH9DVID9w4AAPPfQQrq6umEwm1aNzEKqHp4GTJ09y+fJl1qxZQ3x8vNP8slRXV6PX66moqCAkJITNmzfbNMgfewz27IGf/QwWLoS777bJywCWID916hTZ2dnU1NQQFRVFe3s7/v7+TvPzG+1UD89Kurq62L59O+Hh4cycORODwYDJZHKa4U9zczN5eXkcOHAALy8vVqxYMWJB3tUFK1dCeTmUlMCcOTf/mqHq6OjgnXfeoaKiguDgYDIyMpgxY4ZT9MhHG9XDsyGz2cy+ffvIz8+ntbWVhIQEZs6ciZubG25ublqXN2xdXV3s2LGDHTt2YDabWbZsGcnJySMa5B4e8O67EBcHd95pWcQICrLOsY1GI66urnh6eqLT6ZyuR658nurhDcOZM2f49NNPqampITIyklWrVjFhwgSty7KK/kE+Z84c0tLSCLJW0tyCHTsgJQXS0+Gjj4a3iNHd3c327dvZu3cvjz/+ON7e3larU9GW6uHZSFtbG0ajkU2bNjFz5kynGP5IKTl9+jR6vb43yDdv3mwXQZ6QAL/+NXzlK/DDH8J//dfQj2E2m9m/fz95eXm9QW4ymaxfrGKXVA+vR1VVFU1NTYSFhTFmzJgB27S0tJCXl0doaCiJiYlIKTGbzU4z/KmuriY7O5vTp08THBxMenq6XQb51q2W1dt334WNGy3D7jNnzgAwefLk617H2t3dzR/+8AcuX77sdD1y5V9UD+86pJS89dZb/O///m/vZVAXL14kKSmJp59+muXLlwOWX5QdO3awfft2zGZz7+NCCKcIu6tBvn//fjw9PVm9ejWLFi2y23/br39tWcB46CHJJ588z0cf/az33L+mpiYeeeQRnn76aYJ7zlZubW3F19cXd3d3oqOjWbFihV0GuWJ7o7qH9/TTT/Pxxx/z3HPPsWbNGlxcXGhvb+eNN97gmWee4fnnnyc+Pp6PPvqIlpYWZs+eTXp6uqbzWNbUP8gXL15McnIyXl5eWpd2U/v31xIfL/HxkeTnt7FwYTRgOR3o5z//OUVFRXzyySccOnSI8vJyvvKVr1y35644lxv18G4aeEIIT6AI8MDSI3xXSvmjfm08gL8CcUAdcI+U8uyNjqt14H344Yd8+9vfpqSkhHadJwcuNtHWbSLY242FEYFUnTpGamoqH374IUePHiUzM5PIyEjN6rWmq/NY+fn5tLS0MGvWLNLT03t7RI7g9ttvJzT0Dl577Ussm13Bdxc/iE5IIpPvJPqOx9n6+FfZtWsX999//4gHuZSSM/rXOP3xHzF0tDJ23nLm3P80nkFjR+T1R7vhBp4AfKSUrUIIN6AYeEpKWdKnzVeBWCnlV4QQm4H1Usp7bnRcrQMvPT2dRx99lNTbN1BUUYep5/vQVH+Fw6VFJM+IYmdxEe7u7jz77LNOM/y5uiBxdR7LEYP81KlTJCQkUFlZyQ/ve5+fv38vm2a/yJbYXyFd3Cls9sN7bgq/+tWvyM/PJy4ubkTrO/HBy5z8x8uYujoAEDoXPIPDSHv+M3Sujn+qkr0b1hyetCRia8+nbj3/9U/JO4Af93z8LvAbIYSQWo2Xb6KtrY3t27fzySefUHSuCZOUXKnu4M1fNzMu8g08PNzomDOLBx54gPvuu4+f/exnWpc8bDU1Nej1ek6dOkVQUBB33303s2bNcsgg/+ijj9i4cSPmljqW+z7Hvmg33j78ONFBR1gWmUOksYbb7r6Duro6ioqKRjTwpJSc+uj3vWEHIM0mDK1NXN5fRHh82ojVolxrUIsWQggXYA8wFfitlLK0X5MIoApASmkUQjQBIcCVfsfZCmwFiNLwxgXt7e34+Pjg4eFBe7eJC2dO8c4Ll9mVew8Jq9147AdeRIcHEyRaaGtr06xOa2htbSU/P5+9e/fi4eFBZmYmixcvxtURthO+jra2NoKDg+lsqKHFCDPHfJu9lybz1uFHWDohh5hxAYR6uRIUFDTiPz9pMmDsbL/mcbPJSGdd9YjWolxrUO96KaUJmC+ECATeF0LMkVIeGuqLSSlfAV4By5B2qF9vLYGBgRgMBk6ePMk4vxCuhI5haeZx3Nzr2f5pLAmZdSQ/6MnBHTuYOHGiVmUOS3d3Nzt37mT79u2YTCaWLFnCihUrHGJB4mYmTpzIG2+8QdHCBXxy5DIumHg8/svMGmNGCEuPyj8qhoMHD7JphHcQ1bm64xcRTcv5U597XAhB8MwBR1nKCBryKq0Q4odAu5TyF30eywJ+LKXcKYRwBaqBMTca0mo5h1dRUcEjjzxCYGAgb7zzd7KO12AwmenohJ8+NpaLZ1wpLRV8+/+tZdOmTTzyyCOa1HkrzGYz5eXl5ObmOuyCxM20tLQwbtw4vvzlLzN/nA/e5f/AAzMAwtWV2Id/iHnSQuLi4qisrBzx7bgaTh1gx//3KNJkxGzoxsXDiwlJdzDviz+6+RcrwzbcRYsxgEFK2SiE8AL0wHNSyo/7tPkaMLfPosUGKeUN/7RqEXi1tbXo9XpOnjyJ0WjkhRde4NVXXyVj1WrONbTT3GnE0OjBujRPzOYGgoJWceBAkcP0iioqKtDr9VRXVxMREcGqVas0nTqwJiklR44cYebMmeh0Or71rW+Rn59PTk4Obu31nN/5TwAmLFuLwTuYVatWsWHDBr73ve9pUm9nQw1V2z6ku6WBcQtTCJ4R75DzpY5ouCcehwN/6ZnH0wFvSyk/FkL8BCiTUn4I/AH4mxDiFFAPbLZS7VZz7Ngx3nrrLTw8PMjIyGDJkiWsXbuWDRs2kJSUxMMPP0xYWBinTpxg3LgSysufZ/78Qjw87D/samtryc7O5sSJEwQGBrJx40bmzJnjNL9gZ86cISsri+rqau6++25mz57NL37xC77zne8we/ZsHnvsMVavXg3AG6/8ld/97nfcf//9fFfD+z16Bo1l2rovafb6ysCc+sRjg8FAU1MToaGhvbfTS0hI+NyF4i0tLbz66qu8++67NDU1ER4ezkMPPcSFC+v5xjdc+NGP4Mc/tmmZt6y1tZWCggL27t2Lu7s7y5cvZ8mSJQ69INFX/yBPS0u7JsgPHz7Miy++yNX30qJFi/jKV77C7NmztSpb0diwhrS2YsvAk1Jy4MAB8vLy8PDw4PHHH0en0w3xGPDoo/DnP8M//gHr1tmk1FtiMBjYuXMnxcXFGI1G4uPjSUlJcaodP6SUvPzyyzQ0NJCcnOxUQa7Y1qi6lrbv8CciIoLMzMwhhx2AEJZbBR46BPffD7t2We6gpSUpZe+CRHNzMzNmzCA9Pd1p7mVrMBjYtWsXcXFxeHp6sn79enVXMMWqnCrwTp06xauvvkpAQIBV5rE8PeG99ywbT65fD6Wl4O9vxYKH4OzZs2RlZXHp0iXGjx/Phg0bmDRpkjbFWFn/IPfx8WH+/PmEhYVpXZriZBw+8Nra2qitrWXSpElER0dz++23M2/ePKvtNhwZCe+8A2lp8OCDlgC8hQ7jLbty5QrZ2dkcP36cgIAANmzYwNy5c51mQcKZg1yxPw4beAaDgZKSEoqLi3Fzc+Ob3/wmLi4uxMdb/+TOFSvg+efhqacsm07+4AdWf4lrtLW1UVBQwJ49e3BzcyM9PZ0lS5Y4xbbxfRUXF9Pe3u50Qa7YJ4cLPCklBw8eJDc3l6amJmJiYsjIyLD53m1PPGG5e9aPfgQLFsDtt9vmdQwGA6WlpWzbtg2DwUBcXBwpKSlOM4/V1tZGYWEhy5YtIygoiDvuuANPT0+nC3LFPjlc4J0/f5733nuP8PBw1q9fP2LDHyHgpZcsixhbtlhuJDN9uvWOL6Xk0KFD5OTkfC7InWVBwmg0UlJS0hvk4eHhBAUF4efnp3VpNmE2dlNTvh1DaxOhc5bhFXzr85Ed9Ze5cmgnbr4BjI1NUjuuDINDnJZSV1fH+fPnmTdvHmC5omDy5MmaDH8qKy2LGGPGWBYxrPH7eu7cOfR6PRcuXCA8PJzMzEwmT548/APbiUOHDpGdnd0b5Onp6U69GWfb5SqK/2MLpq52pFkizSZm3fstolc/MORjVXz2N4688T8InQtCJ3Dx8CbpR6/hE+ZYW3qNJIc9LaW9vZ2CggLKysrw9PRk5syZvdt0ayUqCt5+GzIy4KGHLPdVuNVFjLq6OrKzszl27Bj+/v6sX7+e2NhYp5vHOnfuHN7e3tx5551OFeTXs++l79HVVAfS3PvYkTf+h3FxqXiPiRj0cdprL3Dkjf/BbOjqfczY2cG+l79P0g//ZtWaRwu7DDyj0UhpaSlFRUV0d3f3zmO5u7trXRpguSn0L34B3/wmPPssPPPM0L6+vb2dwsJCdu/ejaurK2lpaSxdutRp5rHq6urIyclh2bJlREVFkZGRgZubm9MF+UDMRgP1J/Z9LuzAslvK5f1FTM64d9DHury/6NrvmTRTf3wvZqNBDW1vgV0GXmNjI7m5uUyZMoXMzEy7HP489RSUlVlWbBcsgLVrb/41V4N827ZtdHV19Qa5r6+v7QseAf2DfObMmURFRdnNH6qRIHQu6FzdPtcrA0Dngpv30OY/3Lx8Bxw+6FzdEDr7vMGSvbObwKusrOTUqVOkpqYSGhrK1772NUJCQrQu67qEgFdegSNH4L77LIsY06YN3FZKyeHDh8nJyaGxsZHp06eTkZFhl0F+q3bv3k1ubq5TBvlQCJ2OyBXrqSp6H3P3v0JPp3NhXFzqkI41Lj6Vg3/+KX3vmqtz8yByxXrESJ4M6kQ0D7yrw5+jR4/i7+/P0qVL8fb2tuuwu8rb23Iicnw83HknlJRcu4hRWVmJXq/n/PnzjBs3jgcffFDTOUhrurrgJYSgu7u79x4ZzhTkt2LOA99Dmoyc3/YhUprxHT+ZhV/9b1w9h3ats6unDwk/+At7f/sdWi+dQQgdE5K+wJwHtNnyyhlotkobFxcnf/rTn/YOf5KSkli2bJlDzmPl5kJmpmX19vJlqKqCiIh67rwzm5AQS5CnpqYSGxt7S9f12qOrQb548WJiY2ORUo6KObqhMBm6MRu6hjyUHYihvQWdmwcubqNneuBW2eUq7dUTiBcsWMDKlSsdeviTlgabN8PrrwN0A3mcP7+bl1924emnU3niCccM8oHU19eTk5PDkSNH8PPz693BRIXdtVzc3K0WUNYITUXj8/C2b9+Oh4eHJq9vbVFRRqqqGrHcu+g1IABYycSJvpw9q2Vl1rN9+3by8vJwcXEhMTGRZcuWjaoFCcUx2GUPD3CKsLu69XhVVQ6Wu1luBlYDliskKis1LM4KjEYjAK6urgQHBzNv3jxWrlzptFdIKM5N80ULR1ZVVYVer6eqqorg4DDq6zcDUz7XxlFvKXE1yHNycliwYAHJycnMnDmTmTNnal2aotwyFXi3oL6+ntzcXA4fPoyvry/r1q1jypT5fOUrOtr73JLU29uyu4qj6RvkYWFhTJgwQeuSFMUqVOANQUdHB0VFRezatQudTkdKSgoJCQm4u7uzcKHlHNFnnrEMY6OiLGG3ZYvWVQ9NUVEReXl5vUE+f/58p1lZVhQVeINgMpnYvXs3hYWFdHZ29q4s95/H2rLF8QIOLEEupcTb25upU6diNpt7g1xRnIkKvBuQUnL06FFycnKor6/vvdTNWbYe7xvkM2bM4I477mD8+PGMHz9e69IUxSZuGnhCiEjgr0AYIIFXpJT/169NAPAqENVzzF9IKf9k/XJHzvnz59Hr9VRWVjJ27Fjuv/9+pk6dqnVZVjFQkC9dulTrshTF5gbTwzMC35JS7hVC+AF7hBDZUsojfdp8DTgipfyCEGIMcFwI8ZqUstsWRdtSQ0MDubm5HDp0CF9fX77whS+wYMECp5rH2rZtG3l5eb1BPmXKFHXisDIq3DTwpJSXgEs9H7cIIY4CEUDfwJOAn7D81vgC9ViC0mF0dnaybds2SkpK0Ol0rFixgoSEBKc4VxAsO9CYTCZCQkKYN28ePj4+ThfkinIzQ5rDE0JMAhYApf2e+g3wIXAR8APukbLfhmCWr98KbAWIspMT1EwmE2VlZRQWFtLR0cG8efNITU3FX6v7MVpZ3yCPjo5my5YtBAQEEBcXp3VpijLiBh14Qghf4O/AN6SUzf2eXgXsB1KxnHmbLYTY1r+dlPIV4BWwXFo2jLqHTUrJsWPHyMnJoa6ujujoaDIzMxk3bpyWZVnN9YJcUUazQQWeEMINS9i9JqV8b4AmjwA/k5YLc08JIc4AM4BdVqvUii5cuIBer+fcuXOMGTOGLVu2MHXqVKeax9q1axdZWVlOF+SKMhyDWaUVwB+Ao1LK56/TrBJIA7YJIcKAGKDCalVaydWdlA8ePIiPjw+33347CxcudJp5rIsXL2IwGJg4cSJxcXGEhoY6XZArynAMpoeXCDwAHBRC7O957PtYTkFBSvkS8J/An4UQBwEBPC2lvGL9cm/N1Xms0lLL1GNycjKJiYlOtSCRl5dHeXk5UVFRPProo7i7uzPtelswK8ooNZhV2mIsIXajNheBTGsVZS0mk4k9e/ZQUFBAe3t77zxWQECA1qVZRWdnJ8XFxZSUlACwfPlykpKSNK5KUeyXU15pIaXkxIkTZGdnc+XKFSZNmsSqVasIDw/XujSrOn78OMXFxcTGxpKWluY0Qa4otuJ0gXfx4kX0ej1nz54lNDSUe++9l+nTpzvFPNbVIO/q6iI2NpbY2FjGjRvnNJe6KYqtOU3gNTU1kZeXx4EDB/D29ua2225j4cKFuLg4x+3sLl26RFZWFmfPniUyMpK5c+cihFBhpyhD4PCB19XVRXFxMTt37gQgKSmJpKQkPD09Na7MOpqbm8nNze0N8rVr1xIXF+cUPVZFGWkOG3hms5m9e/eSn59PW1sbsbGxpKamEhgYqHVpVlVfX8/hw4dJTExk+fLlThPkiqIFTW/iU1ZWNuSvk1Jy8uRJ9Ho9V65cYeLEiWRmZhIREWGDKkfe1SBvbW0lJSUFgLa2Nnx8fLQtTFEchN3exGeoqqur0ev1VFRUEBISwubNm4mJiXGK4Z2UklOnTqHX66mtrWXy5MmYzWZ0Op0KO0WxEocIvObm5t4FCS8vL9asWUN8fLzTLEjU1dXxySef9Ab5Pffcw4wZM5wiyBXFnth14HV1dbF9+3Z27tzZu+24M81jSSkRQiCEoKamxumCXFHsjV0GntlsZt++feTn59Pa2sqcOXNIT093mgWJrq4uduzYQV1dHXfddRfBwcF885vfVEGnKDZmV4F3dR4rOzubmpoaoqKi2Lx5s9PcJnCgIDeZTLi4uKiwU5QRYDeB13dBIjg4mE2bNjFz5kynmceqqanh3XffdcogVxRHoXngtbS0kJeXx/79+/H09GT16tUsWrTIaXo8RqMRV1dX/Pz8cHNzc7ogVxRHolngSSkpKChg+/btmM1mli5dSnJyMl5eXlqVZFVXg7y6uprHHnsMLy8vHnvsMa3LUpRRTbPAq6mpoaCggNmzZ5Oenk5QUJBWpVhVd3c3O3bs6A3yxYsXYzKZnGaTUUVxZJoFnouLC1/84heJjIzUqgSrq62t5a9//SstLS3MmjWL9PR0goODtS5LUZQemgVeaGio04Rda2srvr6+BAcHM3nyZBYtWuQ0/zZFcSaaL1o4spqaGvR6PdXV1Tz55JO4u7uzYcMGrctSFOU6VODdgtbWVvLz89m7dy8eHh6sWLFCzdEpigNQgTdEDQ0NvPjiixiNRpYsWUJycjLe3t5al6UoyiCowBsEKSXV1dWEh4cTGBhIYmIic+fOVQsSiuJgVODdxJkzZ8jKyuLKlSs8+eST+Pv7s2LFCq3LUhTlFgzmRtyRwF+BMEACr0gp/2+AdinALwE34IqU0qFToba2luzsbE6cOEFgYCB33HEHfn5+WpelKLes6ewRjv/9BVouVhAyI46YDV/DK2Sc1mWNqMH08IzAt6SUe4UQfsAeIUS2lPLI1QZCiEDgBWC1lLJSCDHWNuWOjNbWVl566SXc3NzIyMhgyZIluLqqzrDiuBorDrH9Jw9g6u4CJO2XK6kuy2Hlzz/Bw3/0TM0M5kbcl4BLPR+3CCGOAhHAkT7N7gPek1JW9rSrsUGtNmUwGDh9+jQzZszA19eX9evXEx0drRYkFKdw9O3/w9Td2fu5NJswdnZwLvctpq9/XMPKRtaQzqUQQkwCFgCl/Z6aDgQJIQqEEHuEEA9e5+u3CiHKhBBltbW1t1SwtUkpOXDgAL/+9a958803qaurA2DOnDkq7BSn0Xrh9DWPmQ1dNJ07pkE12hn0OE0I4Qv8HfiGlLJ5gOPEAWmAF7BTCFEipTzRt5GU8hXgFbDcxGc4hVvD2bNnycrK4tKlS4wfP56NGzcSEhKidVmKYnWBU2PpqL8M0tz7mM7dk+CYOA2rGnmDCjwhhBuWsHtNSvneAE3OA3VSyjagTQhRBMwDTgzQ1i50dnby+uuv4+XlxcaNG5kzZ47asklxWjM3PUVt+XZM3V1IkwGduweegWOYmLJR69JG1E1v0ygsKfAXoF5K+Y3rtJkJ/AZYBbgDu4DNUspD1zvurd6mcTja2trYv38/CQkJCCGoqqpi3LhxuLm5jWgdiqKFjrpLVHz2N5qrThI6awmT0jfj5u2rdVlWN9zbNCYCDwAHhRD7ex77PhAFIKV8SUp5VAjxGVAOmIHf3yjsRprBYKC0tJRt27ZhMBiIjo4mPDxcXeCvjCpeIeHM3vIdrcvQ1GBWaYuBm471pJQ/B35ujaKsRUrJwYMHyc3NpampiZiYGDIyMggNDdW6NEVRNODUJ5eZTCZyc3Px9vbmzjvvZPLkyVqXpCiKhpwu8Orq6tixYwdr1qzB1dWVRx55hICAALUgoSiK8wRee3s7hYWF7N69G1dXVxYsWMCECROc5l62iqIMn8MHntlsZufOnWzbto2uri7i4uJISUnB19f5Vp8URRkehw88IQRHjx4lMjKSzMxMxowZo3VJiqLYKYcMvMrKSgoKCtiwYQO+vr48+OCDuLu7a12Woih2zqECr76+nuzsbI4ePYqfnx8NDQ34+vqqsFMUZVAcIvCklOj1enbt2oWLiwsrV64kISFBXSGhKMqQ2HXgSSkRQiCEoK2tjfnz57Ny5Uq1IKEoyi2xy8CTUnLkyBHy8vLYtGkTYWFhrF+/Xp1LpyjKsNhd4FVVVaHX66mqqiIsLAyj0Qigwk5RlGGzm8CTUvLBBx9w4MAB/Pz8WLduHfPnz1f3e1UUxWo0D7yuri7c3d0RQhAcHExKSgoJCQlq5VVRFKvTNPBKSkooLCxkw4YNTJs2Td3+UFEUm9JsvFhTU8Nnn33G+PHjCQgI0KoMRVFGEc16eEII7r//fqZMmaIWJBRFGRGaBd6YMWOYOnWqVi+vKMoopJZAFUUZNVTgKYoyaqjAUxRl1FCBpyjKqKECT1GUUeOmgSeEiBRC5AshjgghDgshnrpB20VCCKMQ4i7rlqkottfVXM+RN/6Hoh/cw/6Xn6H10hmtS1KsbDCnpRiBb0kp9woh/IA9QohsKeWRvo2EEC7Ac4DeBnUqik11tzZR+L31dLU0II0Gms4c5kLJZyz/jzfwj5qudXmKldy0hyelvCSl3NvzcQtwFIgYoOkTwN+BGqtWqCgj4FzeO3S3NSGNBgCk2YSpq4Oj7/yfxpUp1jSkOTwhxCRgAVDa7/EIYD3wotUqU5QR1FhxEHN3V79HJc1nj2pSj2Ibgw48IYQvlh7cN6SUzf2e/iXwtJTSfJNjbBVClAkhympra4dcrKLYSmD0XHTuHv0eFfhPnKFJPYptDCrwhBBuWMLuNSnlewM0iQfeFEKcBe4CXhBC3Nm/kZTyFSllvJQyXt1OUbEnE1Pvxt0nAJ2r5T4pQueCi4cXMzd9Q9vCFKu66aKFsFzZ/wfgqJTy+YHaSCkn92n/Z+BjKeUHVqpRUWzO3TeAFc++z6mP/0jdkV34TZjKtHWP4Tt+8s2/WHEYg1mlTQQeAA4KIfb3PPZ9IApASvmSbUpTlJHl4R/M7Pv+n9ZlKDZ008CTUhYDg96/SUr58HAKUhRFsRV1pYWiKKOGCjxFUUYNFXiKoowaKvAURRk1VOApijJqqMBTFGXUUIGnKMqooQJPUZRRQwWeoiijhgo8RVFGDRV4iqKMGirwFEUZNVTgKYoyaqjAUxRl1FCBpyjKqKECT1GUUUMFnqIoo4YKPEVRRg0hpdTmhYVoAY5r8uLXFwpc0bqIAdhjXaqmwVE1DZ616poopRzwtoiDuYmPrRyXUsZr+PrXEEKU2VtNYJ91qZoGR9U0eCNRlxrSKooyaqjAUxRl1NAy8F7R8LWvxx5rAvusS9U0OKqmwbN5XZotWiiKoow0NaRVFGXUGJHAE0K4CCH2CSE+vs7zm4QQR4QQh4UQr2tdkxAiSgiR3/N8uRBi7QjUc1YIcVAIsV8IUTbA80II8SshxKmemhbauqZB1rWlp56DQogdQoh5WtfUp90iIYRRCHGXPdQkhEjpef6wEKJQ65qEEAFCiI+EEAd6anpkBGoKFEK8K4Q4JoQ4KoRY1u95277PpZQ2/w/4N+B14OMBnpsG7AOCej4fawc1vQI83vPxLODsCNRzFgi9wfNrgU8BASwFSkfo+3SzuhL6/OzWjERdN6upp40LkAf8E7hL65qAQOAIENXzuc3f54Oo6fvAcz0fjwHqAXcb1/QX4Es9H7sDgf2et+n73OY9PCHEBOA24PfXafIY8FspZQOAlLLGDmqSgH/PxwHARVvXNAh3AH+VFiVAoBAiXOuipJQ7rv7sgBJggpb19PEE8HfA5u+nQboPeE9KWQkj8z4fBAn4CSEE4Isl8Iy2ejEhRACQDPwBQErZLaVs7NfMpu/zkRjS/hL4DmC+zvPTgelCiO1CiBIhxGo7qOnHwP1CiPNYeghPjEBNEtALIfYIIbYO8HwEUNXn8/M9j2ldV19fxPLXWdOahBARwHrgxRGoZVA1YXmfBwkhCnraPGgHNf0GmInlD/pB4Ckp5fV+J6xhMlAL/Klnuuj3Qgiffm1s+j63aeAJIW4HaqSUe27QzBXLsDYFuBf4nRAiUOOa7gX+LKWcgKWL/TchhK3/OCRJKRdiGRZ+TQiRbOPXG6xB1SWEWIkl8J62g5p+CTxt41/eodbkCsRhGVmsAn4ghJiucU2rgP3AeGA+8BshhD+24wosBF6UUi4A2oDv2vD1rmHrX+JEYJ0Q4izwJpAqhHi1X5vzwIdSSoOU8gxwAksAalnTF4G3AaSUOwFPLNf52YyU8kLP/2uA94HF/ZpcACL7fD6h5zGbGkRdCCFisUwP3CGlrLODmuKBN3t+xncBLwgh7tS4pvNAlpSyTUp5BSgCbLrAM4iaHsEyzJZSylPAGWCGDUs6D5yXUpb2fP4ulgDsy6bvc5sGnpTye1LKCVLKScBmIE9KeX+/Zh9g6d0hhAjF0vWv0LimSiCtp6aZWAKv1lY1CSF8hBB+Vz8GMoFD/Zp9CDzYs4q1FGiSUl6yVU2DrUsIEQW8BzwgpTxhy3oGW5OUcrKUclLPz/hd4KtSyg+0rAn4B5AkhHAVQngDS4CjGtfU930eBsRg29+9aqBKCBHT81AaloWcvmz6Ptdk8wAhxE+AMinlh0AWkCmEOAKYgG+PRC/hJjV9C8vQ+ptY5kEelj1LSDYSBrxvmTvGFXhdSvmZEOIrAFLKl7DMJa4FTgHtWP4629pg6vohEIKlFwVglLa9AHwwNY20m9YkpTwqhPgMKMcyd/x7KWX/ABrRmoD/BP4shDiIZVX06Z7epy09AbwmhHDHEq6PjOT7XF1poSjKqKGutFAUZdRQgacoyqihAk9RlFFDBZ6iKKOGCjxFUUYNFXiKoowaKvAURRk1VOApijJq/P+nrSnDYYrh/gAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# plot decision boundary and margins\n", "plt.figure(figsize=(5,5))\n", "plt.scatter(X[:, 0], X[:, 1], c=iris_y2, s=30, cmap=plt.cm.Paired)\n", "plt.contour(xx, yy, Z, colors='k', levels=[-1, 0, 1], alpha=0.5,\n", " linestyles=['--', '-', '--'])\n", "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=100,\n", " linewidth=1, facecolors='none', edgecolors='k')\n", "plt.xlim([4.6, 6.1])\n", "plt.ylim([2.25, 4])\n", "\n", "# plot margin vectors\n", "theta = clf.coef_[0]\n", "theta0 = clf.intercept_\n", "for idx in clf.support_[:3]:\n", " x0 = X[idx]\n", " y0 = iris_y2.iloc[idx]\n", " margin_x0 = (theta.dot(x0) + theta0)[0] / np.linalg.norm(theta)\n", " w = theta / np.linalg.norm(theta)\n", " plt.plot([x0[0], x0[0]-w[0]*margin_x0], [x0[1], x0[1]-w[1]*margin_x0], color='blue')\n", " plt.scatter([x0[0]-w[0]*margin_x0], [x0[1]-w[1]*margin_x0], color='blue')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 12.2. The Max-Margin Classifier\n", "\n", "We have seen a way to measure the confidence level of a classifier at a data point using the notion of a *margin*.\n", "Next, we are going to see how to maximize the margin of linear classifiers." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.2.1. Maximizing the Margin\n", "\n", "We want to define an objective that will result in maximizing the margin. As a first attempt, consider the following optimization problem.\n", "\n", "$$\n", "\\begin{align*}\n", "\\max_{\\theta,\\theta_0,\\gamma} \\gamma \\; & \\\\\n", "\\text{subject to } \\; & y^{(i)}\\frac{(x^{(i)})^\\top\\theta+\\theta_0}{||\\theta||}\\geq \\gamma \\; \\text{for all $i$} \n", "\\end{align*}\n", "$$\n", "\n", "This maximises the smallest margin over the $(x^{(i)}, y^{(i)})$. It guarantees each point has margin at least $\\gamma$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "This problem is difficult to optimize because of the division by $||\\theta||$ and we would like to simplify it. First, consider the equivalent problem:\n", "\n", "$$\n", "\\begin{align*}\n", "\\max_{\\theta,\\theta_0,\\gamma} \\gamma \\; & \\\\\n", "\\text{subject to } \\; & y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq \\gamma ||\\theta|| \\; \\text{for all $i$}\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Note that this problem has an extra degree of freedom: \n", "\n", "* Suppose we multiply $\\theta, \\theta_0$ by some constant $c >0$.\n", "\n", "* This yields another valid solution!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "To enforce uniqueness, we add another constraint that doesn't change the minimizer:\n", "\n", "$$ \n", "||\\theta|| \\cdot \\gamma = 1. \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This ensures we cannot rescale $\\theta$ and also asks our linear model to assign each $x^{(i)}$ a score of at least $\\pm 1$:\n", "\n", "$$ \n", "y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq 1 \\; \\text{for all $i$} \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "If we constraint $||\\theta|| \\cdot \\gamma = 1$ holds, then we know that $\\gamma = 1/||\\theta||$ and we can replace $\\gamma$ in the optimization problem to obtain:\n", "\n", "$$\n", "\\begin{align*}\n", "\\max_{\\theta,\\theta_0} \\frac{1}{||\\theta||} \\; & \\\\\n", "\\text{subject to } \\; & y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq 1 \\; \\text{for all $i$}\n", "\\end{align*}\n", "$$\n", "\n", "The solution of this problem is still the same." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Finally, instead of maximizing $1/||\\theta||$, we can minimize $||\\theta||$, or equivalently we can minimize $\\frac{1}{2}||\\theta||^2$.\n", "\n", "$$\n", "\\begin{align*}\n", "\\min_{\\theta,\\theta_0} \\frac{1}{2}||\\theta||^2 \\; & \\\\\n", "\\text{subject to } \\; & y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq 1 \\; \\text{for all $i$}\n", "\\end{align*}\n", "$$\n", "\n", "This is now a quadratic program that can be solved using off-the-shelf optimization algorithms!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 12.2.2. Algorithm: Linear Support Vector Machine Classification\n", "The above procedure describes the closed solution for Support Vector Machine. We can succinctly define the algorithm components.\n", "\n", "* __Type__: Supervised learning (binary classification).\n", "\n", "* __Model family__: Linear decision boundaries.\n", "\n", "* __Objective function__: Max-margin optimization.\n", "\n", "* __Optimizer__: Quadratic optimization algorithms.\n", "\n", "* __Probabilistic interpretation__: No simple interpretation!\n", "\n", "Later, we will see several other versions of this algorithm." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 12.3. Soft Margins and the Hinge Loss\n", "\n", "Let's continue looking at how we can maximize the margin." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.3.1. Non-Separable Problems\n", "\n", "So far, we have assume that a linear hyperplane exists. However, what if the classes are non-separable? Then our optimization problem does not have a solution and we need to modify it." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our solution is going to be to make each constraint \"soft\", by introducing \"slack\" variables, which allow the constraint to be violated.\n", "\n", "$$\n", "y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq 1 - \\xi_i.\n", "$$\n", "\n", "* If we can classify each point with a perfect score of $\\geq 1$, the $\\xi_i=0$.\n", "\n", "* If we cannot assign a perfect score, we assign a score of $1-\\xi_i$.\n", "\n", "* We define optimization such that the $\\xi_i$ are chosen to be as small as possible." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "In the optimization problem, we assign a penalty $C$ to these slack variables to obtain:\n", "\n", "$$\n", "\\begin{align*}\n", "\\min_{\\theta,\\theta_0, \\xi}\\; & \\frac{1}{2}||\\theta||^2 + C \\sum_{i=1}^n \\xi_i \\; \\\\\n", "\\text{subject to } \\; & y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\geq 1 - \\xi_i \\; \\text{for all $i$} \\\\\n", "& \\xi_i \\geq 0\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.3.2. Towards an Unconstrained Objective\n", "\n", "Let's further modify things. Moving around terms in the inequality we get:\n", "\n", "$$\n", "\\begin{align*}\n", "\\min_{\\theta,\\theta_0, \\xi}\\; & \\frac{1}{2}||\\theta||^2 + C \\sum_{i=1}^n \\xi_i \\; \\\\\n", "\\text{subject to } \\; & \\xi_i \\geq 1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right) \\; \\xi_i \\geq 0 \\; \\text{for all $i$} \n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "If $0 \\geq 1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)$, we classified $x^{(i)}$ perfectly and $\\xi_i = 0$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "If $0 < 1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)$, then $\\xi_i = 1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Thus, $\\xi_i = \\max\\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right), 0 \\right)$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We simplify notation a bit by using the notation $(x)^+ = \\max(x,0)$.\n", "\n", "This yields:\n", "\n", "$$\n", "\\xi_i = \\max\\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right), 0 \\right) := \\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Since $\\xi_i = \\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+$, we can take\n", "\n", "$$\n", "\\begin{align*}\n", "\\min_{\\theta,\\theta_0, \\xi}\\; & \\frac{1}{2}||\\theta||^2 + C \\sum_{i=1}^n \\xi_i \\; \\\\\n", "\\text{subject to } \\; & \\xi_i \\geq 1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right) \\; \\xi_i \\geq 0 \\; \\text{for all $i$} \n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "And we turn it into the following by plugging in the definition of $\\xi_i$:\n", "\n", "$$ \n", "\\min_{\\theta,\\theta_0}\\; \\frac{1}{2}||\\theta||^2 + C \\sum_{i=1}^n \\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+ \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Since it doesn't matter which term we multiply by $C>0$, this is equivalent to\n", "\n", "$$ \\min_{\\theta,\\theta_0, \\xi}\\; \\sum_{i=1}^n \\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+ + \\frac{\\lambda}{2}||\\theta||^2 $$\n", "\n", "for some $\\lambda > 0$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "We have now turned our optimizatin problem into an unconstrained form:\n", "\n", "$$ \n", "\\min_{\\theta,\\theta_0}\\; \\sum_{i=1}^n \\underbrace{\\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+}_\\text{hinge loss} + \\underbrace{\\frac{\\lambda}{2}||\\theta||^2}_\\text{regularizer} \n", "$$\n", "\n", "* The hinge loss penalizes incorrect predictions.\n", "* The L2 regularizer ensures the weights are small and well-behaved." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.3.3. The Hinge Loss\n", "\n", "Consider again our new loss term for a label $y$ and a prediction $f$:\n", "\n", "$$ \n", "L(y, f) = \\max\\left(1 - y \\cdot f, 0\\right). \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's examine the behavior of this loss on different $y, f$:" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* If prediction $f$ has same sign as $y$, and $|f| \\geq 1$, the loss is zero. In other words, if the class is correct, no penalty is applied if the absolute value of the score $f$ is greater than 1." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* However, if the prediction $f$ is of the wrong sign, or $|f| \\leq 1$, the loss is $|y - f|$. Thus, we penalize incorrect predictions, or predictions that are too close to the midpoint between the two class labels (which is at zero, since the labels are $\\pm 1$)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Let's visualize a few losses $L(y=1,f)$, as a function of $f$, including hinge." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'L(y=1,f)')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAEGCAYAAABM2KIzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABtg0lEQVR4nO3dd3ic1Znw/++Zot67ZUmWLRdJtuVuDLZxJ4BpISTUAEmAJJtks5t3C9uSTd7kfZPdffPLZjcbAoRAIKEEQoKxAdu4AcbGkrslV9lWsa3e27Tz++MZg5Ble+bRjDQa3Z/rmsvS6Dlnjh6PZu45zzn3rbTWCCGEEEIIIXxjGekBCCGEEEIIMZpIAC2EEEIIIYQfJIAWQgghhBDCDxJACyGEEEII4QcJoIUQQgghhPCDbaQH4K+0tDSdn58/0sMQQgghhBBhrqysrFFrnT7w/lEXQOfn51NaWjrSwxBCCCGEEGFOKXV2sPtlCYcQQgghhBB+kADaH1J0RgghhBBi+IRo7CUB9FWcarrAP2/+DWcOvQmHXgWPZ6SHJIQQQggR/vo6ofTXVJ04yGtlNbjcoRODjbo10MPJ5Xbz6z2v092wm6er21gcm8WqmDQiJq8Y6aEJIYQQQoQvj4e+Q3+kqrKSxs5juBPnsjthNYunXLKfb0TIDPQVrNv3Ns66bUS7WnGj2dF1nv/e+SuOH3tvpIcmhBBCCBGWtNZUla3n0OFDNHY6AE1uWxkde16ivrl1pIcHBDGAVko9o5SqV0odvszPlVLq50qpk0qpg0qpucEaixnnTu1i7+FnsXn6PnV/g7uXZ3f+J8/sfI1uZ99lWgshhBBCCH919DrZ+t52ag9ux+n+9PrnxJ6znN34P7jb60ZodJ8I5gz0s8CNV/j5TcAU7+0x4JdBHIvvPB48xzex7sNf4NLOQQ/RuCg/8Qf+8Z1fUlZ7apgHKIQQQggRXrTWHKpp4w/b9xFx8p3LHufqaubc1l9B3ZFhHN2lghZAa613AM1XOOR24LfasAtIUkqNC9Z4fHZqC7sOv8kZZ8cVD7N7etEtB/nZrt/xXx/+mY6+nmEaoBBCCCFE+GjtdvBqWQ1bj1Qx8cI7WC8zgXnRuaZ2eg68Dg3Hh2mElxrJNdDjgep+39d477uEUuoxpVSpUqq0oaEhqINqSC3k7Y4LPh0b42omzlHPR+cO8vcbn+CDsxVBHZsQQgghRLjweDRlZ5t5YddZapq7KWjaTpSr7ertNBzqTESnFgzDKAc3KjYRaq2f1FrP11rPT08P7u7LjdUnORc1wefjk3priHR10OHo5InSV/n39/5AS3dnEEcohBBCCDG6NXT08XJpNTuON+J0a8a37yel54xPbR3WWD6KvZ5z7Y7gDvIKRjKArgVy+32f471vRN0783rumnMPfbGTfG6T2lOJ1WP8Jx6sP8rjm59g88kDeCRntBBCCCHEx1xuDztPNfL73VVcaOsFILGnhty2Up/aa2WhMf9m7lo0jfFJ0cEc6hWNZAD9BvCgNxvHIqBNa31+BMcDgMVi4capc/mntf9KflKRb220i9SeShRGwNzt7OG5A2/wf3a8RF3n1S9FCCGEEEKEu/NtPbz4URW7K5vxeCsMRro6mNL0LnD1ioMWBfEzbuKO6xeSlRgV5NFemdJBKpGolHoRWA6kAXXA9wA7gNb6CaWUAv4bI1NHN/AlrfVVP37Mnz9fl5b69illyFwOyt79N946f4CuqyxoB+i0p9Manfep+yJtEdw+dQVrp83HYhkVK2aEEEIIIQLG4fLwYWUT+6paPlWZ2+JxMb3+DWIdjVftIz7SRt6MRSTMuh2UCuJoP00pVaa1nn/J/cEKoINlWANogO5mOnc/xZsXjnOgt+mqhzdH59NtT73k/gmJOXx1/m3kJl36MyGEEEKIcFTd3M2m8jraegZMRGpNQfN20ruunEnDqiA3JYbM8ROxzH0ArPYgjvZSlwugZUr0amJSiJt5J/dkTebBtGkkWiKueHhyz1ki3N2X3H+2rYbvbXuSVw69j8vtDtZohRBCCCFGXK/TzabyOl4tq7k0eAYyOyuuGjwnRtspyUliXFoKlhmfHfbg+UokgPZF2hSYcB1FcUn8VU4JC6IzUJeZuFdoUntOYdGuS37mdLtYd3wr/7T5GU40jvhybyGEEEKIgDtZ38nzH57lcO3g+8Di+urIb9152fY2i6IgPZaicfFERdig+DaITgrSaM2RANpX+UshZRJRFit3Zk7kK5mFpFoHX8Bu9ThI6T7N5RbEn+u8wI/ee4bn9r6Lw3VpoC2EEEIIMdp0O1xsOHSedQfO0dk3eHxjd3cztXEzSg+eqSwlNoJZuUlkxEehUDDxekjxPTPacJEA2lcWi/EJKCoRgIKYRL49fgZLYrOwculi9ih3O4m95y7bndvjYfPpnTy+6SkOX6gK2rCFEEIIIYJJa03F+Xae23mWYxeuUMlZe5jS+C4R7q5LfmS3KqZmxjE1M44Iqzc8TZsCedcGadRDIwG0P+zRMONOsNiMby1W1qZP4LHMYrKsMZccHu+4QLSz9YpdNnQ38m87f8uvPtpAt7MvGKMWQgghhAiK9l4nf95/jrcPX6DXeeU9XhNad5PQd+kS1vS4SGblJJEaG2nMOgPEpEDRrcOaccMfEkD7Kz4Lpt34qbvyouP45vjprIofj33AbHRK72lsnt4rdqm15v3qMv5+468orTkZ8CELIYQQQgSS1poD1a08/+FZTjdeOqM8UGrXScZ1HPrUfZE2C4VZ8UzOiMNu7ReSWu0w43Ngiwz0sANGAmgzsmbC+LmfustqsbA6NYdvjJtBnj3u4/uV9pDafQqlr555o7W3jf/c/SI/3/ln2novzeQhhBBCCDHSWroc/KGshi1H63G4rl51OdrRzKTm9z51X1ZCJCU5iSTHDJLdrHAtxKYFarhBIQG0WZNXQ+L4S+7OjIzhq9lF3JyQR4T39No9vaT0nMWXKjsAe84f5B82/Yr3TpcHcsRCCCGEEKZ5PJrSM828sOsstS09PrWxunuZ1rgJq7cgXbTdwvTsBCamxWEbrMBc7kLI8K0S9EiSANosixWK74CIS9c+W5SFpSnj+MvsmRTYEwCIdrWQ0HfB5+47HJ08ufc1/m3HKzR1X2FBvhBCCCFEkNV39PLinireO9GIy+NjET7tYWrTFqJcbShgfFI0M3OSSIi6TD7n5AkwaUXAxhxMEkAPRVQCTP8sqMFPY2pEFF/OnsadSROJVlYS+s5ddVPhQIcajvEPm5/gnRP78HiufplECCGEECJQXG4PO0828uLuaurb/Ut2MKF1N4m9NcRGWJkxPpG8lBisl9sUGBkPxbcbWc9GgdExylCWlAcFKy/7Y4uysCApg7/KLqEoMomUntPYB6lUeCU9zl5eOPgmP9z+ey50tAx1xEIIIYQQV3WutYfff1TF7tPNeLSPs85e6Z3HGN95iLyUGGaMTyQu0nb5gy1WI8tZROwQRzx8JIAOhJz5V12vk2CP4MFx07gvdRL5fTWDViq8mhPNp/mnd5/kT+W7ZDZaCCGEEEHhcHnYdqyeV0qraep0+N0+ru8Cs3o+ZOb4RMYnRWO5Wiq6KWsgIdvkaEfGFT4OCJ8pBdNuhq4G6Gq84qEl8alMjkngtY4uNnli0H5+hnG4HbxWsYk9teV8dcGt5CWlD2XkQgghhBAfO9vUxeaKetp7nKbax9HFzepDsrPiUL7kcB5XAuNmm3qskSQz0IFiizByFtoHL+/dX4zVzheTkvjb7HySvJUN/VXVXsv3tj7FSwd34HJfPUWeEEIIIcTl9DrdbDxygT/urTUdPOcn23kgrpTxMR7fgueEbJjymZAtlnIlEkAHUkzKFTcVDjSz9xz/NuMalubN9+2JNoDL42b9ie384+Zfc6zh8mXDhRBCCCEu52R9B7/98AxHzrWbah9pt3BDcQZ3RB8guvfKV+I/aRRvrHu2js7FEEr7uSh8pM2fP1+XlpaO9DCurKYMTmz07Vhlgdn3cqRP8eu9b9LQ7eMTbwCLsrB8wkLuLVlGlH2QpORCCCGEEP109bnYeqyeE3WdpvuYnBHHisIM4i58BJXbfWtkscGcByBhnOnHHS5KqTKt9fyB98sMdDCMnwvZs307Vnvg8B+ZnhDPj9c8yuqJ12E1kcLFoz1sObOLxzc/xcELZ/xuL4QQQoixQWvNkXNt/PbDs6aD59hIK7eUjOPWWdnEdZyG0zt8b1x486gInq9EZqCDxeOGAy9Ca7Vvx8elw5wHwRbBicbzPF32Juc6fS+80p9Simtz5vLgrFXERoZuHXkhhBBCDK+2HidbjtZxptG/lLr9FWcnsGxqOlF2K3Q2wN7nwO3juukJ18Kk5aYfe7hdbgZaAuhgcnRB2XPQ2+bb8elTYfqdoBQut5s/ln/I26few+n2P+UdQGJkAl8suZlr8qaYai+EEEKI8KC15kBNGx+cbMThMpcKNyHazqrCDPLTvPmaHd1G8NzT6lsHaVOMhAujaNOgBNAjpaMO9v0WfA2C85fAxKUff1vd2sSTpes40+bjTPYg5mbN4MvzPkNi1KVlx4UQQggR3pq7HGwur6O2tcdUe6VgVm4SiwvSiLB5l5l63HDwZWg561snsWkw90Gwja4r4xJAj6T6o3Dkdd+Pn/5ZyCj8+FuPx8P6Y2X8+fgW+lz+JzQHiIuI5e7pN7B80gxT7YUQQggxurg9mrKzLeyubMLlMRfvpcRGsLo4k/FJ0Z/+wfGNUFvmWyf2KJj7kJGtbJSRAHqknX4Pzrzv27FWm7EeOj7zU3fXdbbxVOl6jjWdMj2M4rSpPDLvJtLjEkz3IYQQQojQVt/ey8byOho6+ky1tyjF/PxkrpmYgs06ILnBuX1w7G3fOlIWKPkCpEw0NY6RJgH0SNPamIVuOObb8VEJxqe1yLhP3e3xeNhSeYg/lG+i22nuUkyULYrPFa3ihsmzsZjI+CGEEEKI0ORye9hV2UzZ2RY8JmO8jIRI1hRnkhE/SHG4lrNw4CUji5gvptwAOfNMjSMUSAAdClwO2Pc8dNb7dnzCOJh9P1jtl/yopbuTp8ve5mB9henhTE7O59H5a8lOGH2XVIQQQgjxabWtPWw6coGWbnOVBG0WxaKCVObmJWO1DLLRr6vJ2Nfl7PWtw+zZMPXGUbVpcCAJoENFT6uxY9XhY/qYfpk5BvPB2Qp+d/BtOhzm8jhGWO2snbKMO4qukdloIYQQYhTqc7nZebKJ/dWtpvsYnxTN6uJMUmIvU4zN0Q17fws9Lb51mJgDs+8Di9X0mEKBBNChpLUK9r/o++WPvGugYOVlf9zR18Nz+zazu3a/6SHlJmTz2LxbyU/JMN2HEEIIIYbXmcYuNlfU0dFrLuVthM3CkslplOQkoi43U+x2GbUt2mp86zQqAeY9DBGxpsYUSiSADjX+LMAHmHYjZM+54iFltad4bv8GWnpbTQ3JZrFyw6TFfH7GEmzW0f2JUQghhAhnvU432441UHG+3XQf+WkxrCzMJDH60qWiH9MaKt6AunLfOr1MIoTRSgLoUORPChhlgZLPQ8qkKx7W63Tw/P6tvF9disfXGe4BMmPTeWTeLRSm55hqL4QQQojgOVHXwZaj9XQ73KbaR9mtLJuaTtG4+MvPOl90egec+cD3zgek4h3tJIAORR43HPoDNJ/27XhbhPGpLi79qodW1Nfw9N511Hc1mhqaRVlYlreA+2YtJ8p+mfVQQgghhBg2nX0uth6t52S9uX1PAFMz41k+LZ3YSNvVDz5/EI6u973ziUuNgnBhRALoUOXsNTJz+BroXia93WAcLhevHH6Pd09/iMtj7lNqakwKD89ey+xx+abaCyGEEGJotNYcOdfOjhMN9DnNXV2OjbSysjCTyRlXjx8AI13dwZeNyT5fZM2AwltGdcaNwUgAHcp6Wo2drY4u346/Qnq7wVQ21/F06ZtUd5wzNTylFIvGz+ah2WuIjRxdJTiFEEKI0aytx8m7FXWcbfIxe9cgZoxPZOmUNKLsPu5v8jddXVIezLpn1GfcGMzlAuig5i1TSt2olDqmlDqplHp8kJ/nKaW2KqX2KaUOKqVuDuZ4QlZ0Esy8y1h474v288aCfh8//ExKyeSHq7/EHYWriPAx6O5Pa82HNfv4u02/ZFeVj4VghBBCCGGax6PZV9XCC7vOmg6eE6LtfG5uDmuKM30Pnh1dcOgV34PnmBSYcWdYBs9XErQZaKWUFTgOrAFqgD3AvVrr8n7HPAns01r/UilVDGzQWudfqd+wnIG+qOGYUa3Q1/+T3IUweZVfD1HT1sxTpeuobK0yMUDDnMzpfHneZ0iKHv3paYQQQohQ09TZx+aKOs61+hjEDqAUzM5N4rqCNCJsfsyVul1w4PfQVuvb8fZomPugEUSHqZGYgV4InNRaV2qtHcBLwO0DjtFAgvfrRMDcGoNwkT4NJq3w/fjqj6B2r18PkZOYwvdWfJF7ZtxElG2QEp0+2Fd3hMc3P8HWykOm2gshhBDiUm6PZndlE7/bXWU6eE6Ni+AL83NZPi3Dv+BZazj6pu/Bs8UKMz4X1sHzlfi4ZsCU8UB1v+9rgGsGHPOvwEal1LeAWGD1YB0ppR4DHgPIy8sL+EBDSu5C6GmGc/t9O/7EJohKhNQCnx/CYrGwdtp8Fo6fylNl66loPOn3MLsc3Tyz7098WH2YR+etJT0u4eqNhBBCCDGouvZeNpbX0djRZ6q9RSkWTExmYX4KNquJ+dHTO6C+wvfjC9dCUq7/jxMmRrp2873As1rrHOBm4Hml1CVj0lo/qbWer7Wen55+9RRuo5pSMOUGSJno2/HaA+V/gs4Gvx8qPS6Bf1x2L1+afTuxETF+tweoaDzJP777KzYcK8PjMbczWAghhBirnG4P751o4KWPqk0Hz5kJUdx3TR7XFaSZC57PH4SzO30/fuL1kDnd/8cJI8EMoGuB/h9Ncrz39fcV4BUArfWHQBSQFsQxjQ4WKxTfAbE+ngqXw1jw39dh6uFWFpTw49VfY3Zmsan2va5eXjy8gR9se4GatmZTfQghhBBjTU1LN7/bdZbSMy14TOxJs1kUS6ekcc+CXNLjTWbJajkDx/2ojJw1EyZcZ+6xwkgwA+g9wBSl1ESlVARwD/DGgGOqgFUASqkijADa/6nUcGSPgpmf972OfG87HHwFXOY+vSZFx/K/lnyObyz4AgmR8ab6ONVylu9tfZLXjuyU2WghhBDiMvpcbrYcreMPpTW0dDtN9TE+OZoHFk1gfn4KFovJ3Mud9XD4Nd9zPSflwbSbwi7XsxlBzQPtTUv3M8AKPKO1/pFS6gdAqdb6DW/mjaeAOIwNhX+ntd54pT7DOgvHYNrPwf7fGTtjfZGcbwTevqbEG0RXXx/P7tvI7nMHMPv8yI3P5pH5tzApJdP0OIQQQohwc7qxi3cr6ujo9fF9fYAIm4WlU9KYOT7x6mW4r6Sn1Sjk1udjVcOYFCPjhj3a/GOOQlJIZTTzN71dRhEU3z7kT4j7zp3m2f3rae5pMdXeZrGyauK1fGHGUiJswdyvKoQQQoS2Hoeb7cfrqThvbrklwMS0WFYWZZAQ5X9Nh09xdMO+F6C7ybfjx0C6usuRAHq0q9oNp7b4fnzOAiNH9BCD6F6ng98f2Mb2qj14tLllGZmx6Xxl7i0UZeQMaSxCCCHEaKO15kR9J1uP1tPt8HGpxADREVaWT0tnWmb80GadAdxOOPCif+nqZt07ZjNuSAA92mltLPL3Nb0dQMFKyBuYOdCcow01PF32JnVd5paoW5SF6/Pmc/+sFUTZIwIyJiGEECKUdfa52HK0nlP1Pi6TGMS0rHiWT0snJiIAV3I9HjjyR2g84Xub4tvGdMYNCaDDgccNh16F5krf2xTdClkzAvLwLreblw+/x+bKnbh83XAwQEp0Mg/PXsucbB/T9AkhhBCjjNaaI+fa2XGigT6nuau3cZE2VhZlUJAeF6hBwbG34PwB39tMvB7yFwfm8UcpCaDDhasP9v8eOi74dryyQMnnIWVSwIZQ2VzH06VvUt1hrnCkUoqF2bN4aM5q4iPH1mYEIYQQ4a2t28mmijqqm7tN9zFzfCJLpqQRZbcGbmCnd8CZD3w/PnsOTP3MmM+4IQF0OOnrNHbO9rT6drzVDrPvh4RxARuCx+PhTxW7WX9iOw63uRQ88RFxPFByE9dNKAzYuIQQQoiR4PFo9te0svNkI063udgqMdrOmuJMclPMFTe7rNq9cPwd349PmwLT7wTLSNfbG3kSQIeb7mYjiHb4+Ak3IgbmfDHgO2jPtTfzZOmbnGo5a7qPkowiHp1/E0nRPua8FkIIIUJIY2cfm8vrON/Wa6q9UjA3L5lrC1Kxm6kkeCX+ZvJKzIFZ9xiTb0IC6LDUft6bI9rHGeDoZJjzAEQGaD2Vl8fj4e0T+3j96BZ6XeZePGLs0dw9/QZWFpQEdGxCCCFEsLg9mj1nmvnodDNuj7l4Ki0ugtXFmYxLDMKSxtZqOPASeHzMOR2bZsQJYyzX85VIAB2umk4ZGwt9TTEXnwWz7wObyZKfV9DQ2c7TZW9R3njcdB+FqQU8Mn8tmXGJARyZEEIIEVgX2nrZVFFHY4e5CsBWi2JBfgoLJ6ZgNVtJ8Eo6G4wr1b5WKI6Mh7lfhCh5/+1PAuhwduEQVLzp+/EpE41qhZYAbk7oZ1vlYV468g5dvi4vGSDSFsHtU1eydto8LLL+SgghRAhxuj18eKqJvVUtPq+KGCgrMYo1xZmkxQV+MguA3jbY+zz0+Vi0xRZpLPOMSw/OeEYxCaDDXdUuOLXV9+Mzpxsp7oK0u7a1p4vf7N3I3guHTfcxKSmPR+bdQm5SagBHJoQQQphT3dzN5oo6WrvNbZ63WxXXFqQxJzcJSzBmnQGcPUaVwa5G34632GDW3ZCUF5zxjHISQIc7reHku1Czx/c2uQuNYitBTFGzu+oEvz24nnZfPwUPYLfauLngeu4oXoTNGpwZcyGEEOJKep1u3j/RyKHaNtN95KbEsLoog6SYIBYTczuNNc9tNb4drxRM/yykTwvemEY5CaDHAq2h/E9Qf9T3NsOQJL2rr4/n9m9iV+1+zD7fxsdn8ei8WylIzQrw6IQQQojLq2zoZMvRejp6fdyIN0CEzcL1U9KZMT5h6GW4r8RMsbWpN8D4ecEbUxiQAHqscLvg0CvgT1q5KWsg55LnRsDtP3+GZ/evp6m72VR7q8XCqvxruXvm9UTYAlDSVAghhLiMboeL7ccaOHrB3BVUgEnpsawszCA+Ksgp4TweYwKt4ZjvbSZcB5OWBW1I4UIC6LHE2Wukt+us971N4c0wblbwxuTV63Tw4sHtbDv7ER5fM4cMkB6Txlfm3sL0zNwAj04IIcRYp7XmWF0H24410ONwm+ojOsLKimkZTM2MC+6sMxhXn4+uNxIK+GpcCUy7ecxXGfSFBNBjTV+HsQO318f1WkpB8R2QMTxVAY81nOPpveu44E+Q349SiiW583hg9kpi7EHaxSyEEGJM6eh1suVoPZUNXab7KMyKZ9m0dGIihuFKqdZwcjPU+BEXpU6GGZ+TKoM+kgB6LOpqMnJAOnt8O95iNf6oUguCOy4vl9vNHw6/z8bKD3B5zH3KT4pK5OHZa5k3fnjGLIQQIvxorTlU28Z7JxpxuMxdHY2PsrGyMINJ6YEtVnZFp3fAmQ98Pz4hG2bdC7YgbmQMMxJAj1VttXDg98baaF9YbVAyvOlszjTX82TZOqrbz5nuY2F2CQ/PvYH4SKmeJIQQwnet3Q42lddR0+LjZNMgSnISWTIljUjbMGaLqtoNp7b4fnxMqlFlMCImeGMKQxJAj2XNp+HQH4wdur6wRcCs+yBhXHDH1Y/H4+GNox+x7vh2HG6HqT7iI+K4v+RGFk8oCvDohBBChBuPR7OvuoWdJ5twmSzDnRRjZ3VRJrkpwxyUntsHx972/fioRCN4jkoI3pjClATQY13DcTjyuu8lv+3Rxh9bbFpwxzXAufZmni7bwInm06b7KMko5JF5N5EcM4yX0YQQQowaDR19bCqvo66911R7pWDehGQWTUrFbh3mtcR1R6BiHT6XQYyINd7PY1KCO64wJQG0gAuH4eibvv/RRcYZf3TRycEd1wAej4dNpw7wWsVmepzmXtxi7NHcVbSaVQUlUg5cCCEEAC63h4/ONLPndAsek/FPWnwkNxRnkpkQFeDR+aDxBBz+ox+TYVEw+wEp0T0EEkALQ+1eOP6O78dHJ8Hs+0fksk9TdwdPlW7gSMNx031MSZnIVxfcSmZcYgBHJoQQYrQ539bDpvI6mjrNLRO0WhTXTExhfn4K1mCV4b6SljNw8A/g8XFPky3C2DCYkB3UYYU7CaDFJ85+CJXbfD8+Ng1m32dcBhoBO04f4cXD79DpMJdWKNIWwe1TV7B22nyZjRZCiDHG4fLwYWUT+6pafL4AO9C4xCjWFGeSGjdCaVPbauHAi0apbl9YbFDyBUieENxxjQESQItPq9xmBNK+is80NhbaR+CSFdDW281v9m6k7LwfieIHmJCYw1fn30ZuUmoARyaEECJUVTV1s7mijrYeHwPPAexWxXWT05idk4RlJGadwSiKtu8FcPX5dryywMy7hi0lbbiTAFp8mtZwYhPUlvneJjHHSHE3gvkj99Sc5LcHNtDqa4GYAexWG5+ZtITPTb8Om3UY0w0JIYQYNr1ON++daORwrbn3CoC8lBhWF2WSGBPkMtxX0tUE+18AR7dvxysFxbdDhmSjChQJoMWltDY2FV447HubpDyY+fkRDaK7nX08v+9dPqjZi9nnb3ZcFo/Mu4UpacOXqk8IIUTwnazvZOvRejr7fFwrPECk3cL1U9KZnp0Q/DLcV9LdDPt/B32dvrcpvBnGzQremMYgCaDF4DweKH/dSHPnq6Q8Y22VdQQ/lQOHL1TxzL51NHQ3m2pvtVhYMWER95YsI8I2DCVXhRBCBE1Xn4ttxxo4Xtdhuo+CjDhWFmYQFznC7wlmgufJqyF3QfDGNEZJAC0uz+2Cw68aBVd8lTzBmIke4SDa4XLx+wPb2Fa1G7fHXPnV9JgUvjznVmZkDV/1RSGEEIGhtebohQ62HWug1+ljwbABYiKsrCjMYEpG3MjOOoM3eP499PnxQWDiUshfErwxjWESQIsrczng4MvQVuN7m+R8Y6PCCAfRACcaz/NU2TrOd9aZaq+UYnHOXL44ZxUx9hHaZS2EEMIv7b1OtlTUc7rRXJYmgKJxCSybmk50RAjsizETPOcugIJVxvpnEXASQIurc/bCgd9Dhx9BaMpEmPG5kAiiXW43rx3ZyTuV7+N0m1v7lhSVyEOzbmZ+zuQAj04IIUSgaK05WNPG+ycbcbjMXX2Mj7KxqiiTiWkjk6L1Ej0tRvDc2+57m3GzYNpNEjwHkQTQwjeOLuMPuKvR9zYhFEQDVLU28GTpm5z1ZzZ9gHnjZvKluTeQGBUTwJEJIYQYqpYuB5sq6qht6THdx+zcJK6bnEqkLQRmncFc8JxRBEW3gdQ3CKrLBdBBPetKqRuVUseUUieVUo9f5pgvKKXKlVJHlFK/D+Z4hA8iYmHWPRCT4nub5tNGaVGTs76BlpeUzg9WPsRdRTcQYTWXLaTs/CH+YdOv2HH6SIBHJ4QQwgyPR7PnTDMv7DprOnhOjrHz+fk5rCjMCKHgudX/4Dl9GhTdKsHzCAraDLRSygocB9YANcAe4F6tdXm/Y6YArwArtdYtSqkMrXX9lfqVGehh0tdh/EH7k+EiZZJ3Jjp0Mlpc6GjhydL1nPBng+QAM9On8ZX5N5EaEx/AkQkhhPBVfUcvm8rrqG/3sZjIABalmDchmWsmpWC3hlDQ+XHw7Ee+6vSpUHwHWELkA0CYG4kZ6IXASa11pdbaAbwE3D7gmEeBX2itWwCuFjyLYRQZb5Tv9msmuhKOhM5MNEBWfDL/vOw+Hiy5lRh7tKk+DjUc4x82P8E7J/bhMZnpQwghhP9cbg87Tzby4u5q08Fzenwk9yzMZcmUtNAKnnvbJHgexYL5TBoPVPf7vsZ7X39TgalKqQ+UUruUUjcO1pFS6jGlVKlSqrShoSFIwxWXiIyHWfdCdLLvbZpOwZHXQyqItlgsrJkymx+v/holGYWm+uhx9vLCwTf54fbfc6GjJcAjFEIIMdC51h5+t7uK3aeb8Zi4Wm61KBZPTuPehXlkJkQFYYRDYCZ4TpsiwXMIGemPYjZgCrAcuBd4SimVNPAgrfWTWuv5Wuv56enpwzvCsS4qwZiJ9iuIPgnlfwKPuXycwZIcE8ffLv08j839HPERcab6ONF8mn9690n+VL5LZqOFECIIHC4PW4/V80ppNc1dDlN9ZCdF8cCiCSycmILVEmIZKnrbjeC5p9X3NmlTYPpnJXgOIcEMoGuB3H7f53jv668GeENr7dRan8ZYMz0liGMSZkQlwOx7ITrJ9zaNJ4yZ6BALogGWTizmJzd8jYXZJabaO9wOXqvYxL+8+yxnmmXVkRBCBMrZpi6e33WW/VWtmNmiFWGzsKIwgy/MzyUl1twm8qAyEzynTpaZ5xB01U2ESqkc4B5gKZAN9ACHgfXAW1rrQafhlFI2jIB4FUbgvAe4T2t9pN8xN2JsLHxIKZUG7ANma62bLjce2UQ4gi5ecvL3D3/6Z0NqY2F/pTUnee7ABlr9uYzWj81i5TMFS7hr+mJsVnlxE0IIM3qdbnYcb+DIOT8yUQwwITWGVUWZJEaHRkrVS/S0woEX/XwPLYDpd4bse+hYYCoPtFLqNxjrlt8ESoF6IApj7fIKYB7wuNZ6x2Xa3wz8DLACz2itf6SU+gFQqrV+Qxn1Mv8fcCPgBn6ktX7pSr+IBNAjzMyO4eQJMOMusIXgbADQ7ezjhf1beL+6DLNZabLiMnhk7q1MS88O8OiEECK8nazvYMvRerr6zF2xjLJbuX5qGsXjEka+DPflmKkwKMFzSDAbQM/QWh++ws8jgDyt9cnADPPqJIAOAWaC6MTxMPMLYA+xjRz9HKmr5td736Sh248iMv1YlIXlExZyb8kyouyh+WFBCCFCRVefi63H6jlR12m6jymZcayYlkFsZAgHmZ31cOAlo1CZr0IwLexYZTaAfldrvUop9ROt9d8HdYQ+kgA6RJipmhSfCSX3QEToVvdzuFy8fGgH7575ELfJTYKpMSl8ec5aSrLyAzs4IYQIA1prys+3s+N4I71Oc7POsZFWVhZmMDkjxPPzt5+Hgy+Bs9f3NikTjau2EjyHBLMBdDnwCPBr4D7gU9dGtNZ7AzzOq5IAOoSYCaJj04xKh5Gh/aJ3qukCT5au41znBVPtlVJcmzOXB2etIjYyMsCjE0KI0amtx8mWo3Wcaew23cf07ASun5pOlD3E9520VsOhV8DlRyaRlInemecQXcc9BpkNoO8CvgIswdgE2D+A1lrrlYEe6NVIAB1izKzrik42snpEJQZvXAHgcrv5Y/mHvH3qPZwm81onRibwxZKbuSZPkssIIcYurTUHatr44GQjDpe5q3sJ0XZWF2UwITU2wKMLguZKOPyafzURkvNh5l0SPIcYUwF0v8b/orX+30EZmZ8kgA5B3c3GzmJ/ZqKjEowiLf5UOhwh1a1NPFm6jjNt1Vc/+DLmZs3gy/M+Q2JU6C5fEUKIYGjucrC5vI7a1h5T7ZWCWblJLC5II8I20uUrfGAmjasEzyFrSAH0IJ2NA5q11ubqag6BBNAhqqfV2CTR40eVvohYI4iOC/3iOB6Ph/XHyvjz8S30+XM5rp+4iFjunn4DyyfNCPDohBAi9Lg9mrKzLeyqbMLtMZfhKCU2gjXFmWQnRQd4dEFSVw4V62DwDL+Du1hhUNY8h6RAB9CbgQLgNa313wRgfD6TADqE9XUYQXSXH1ks7NHGmuj4rOCNK4DqOtt4qnQ9x5pOme6jOG0qj8y7ifS4hACOTAghQkd9ey8by+to6DA3z2ZRigX5ySycmILNOgpmnQHOH4Bjb+FXBZiMIii6VYqkhLCABtDeDhVQ3L8wynCQADrEObqNHccddb63sUVAyd2QmBO8cQWQx+NhS+Uh/lC+iW6nuUuSUbYoPle0ihsmz8ZiGSVvDkIIcRVOt4fdlc2UnW3BYzK+yEiIZE1xJhnxoZv29BI1ZXBio39txpXA1JtA3gNCWjAC6DittfnkjSZJAD0KOHuNncdtAyu3X4HVbqz/Ss4P2rACraW7k6fL3uZgfYXpPiYn5/Po/LVkJ4T+WnAhhLiS2tYeNh25QEu301R7m0WxqCCVeXnJWCwhWhBlMGc/hMpt/rUZPw+mrDEWeIuQFowAukprnTfkkflJAuhRwuWAw69Cy1nf21hsRtnvtMnBG1cQ7Dx7lBcOvkWHw9znyQirnbVTlnFH0TUyGy2EGHX6XG52nmxif3Wr6T7GJ0ezpiiT5NhRVIRKazi9A87u9K9d3iKYtFyC51HCbBq771zuR8A/aa2HfdpMAuhRxO2EI3+CJj8KVSoLTLvJuLQ1inT09fDcvs3srt1vuo/chGwem3cr+SkZgRuYEEIE0ZnGLjZX1NHRay7VZ4TNwpLJaZTkJIZuGe7BeDzGko1z+/xrN/F6yF8cnDGJoDAbQPcC/w4M9pfx11rrpICN0EcSQI8yHjeU/xkajvnXbtJy41P6aHpBBcpqT/Hc/g209Laaam+zWFk96TrunrEUm1U2lQghQlOPw8324w1UnPcjfekA+WkxrCrKJCFqlKVuczuN97XGE/61m7wachcEZ0wiaMwG0DuBb2mtywb5WbXWOjeww7w6CaBHIY8Hjq2HC4f9a5ezACavGnVBdK/TwfP7t/J+dSkef1IZ9ZMZm84j826hMH10bKwUQowNWmtO1Hey9Wg93Q5zZbij7FaWTU2naFz86Jp1BnD2GAVSWv2oC6AUTP0MZM8J3rhE0JgNoKcBTVrrS/KSKaUytdZ+pFoIDAmgRymtjctdtX5Wf88ogsJbRmV+zIr6Gp7eu456f9L69WNRFpblLeC+WcuJso+idYFCiLDU2ediy9F6TtWbzx8wNTOeFYXpxESMvtd0etvh4Mv+pWpVFihcC1mS/3+0CvgmwpEiAfQopjWc2gLVH/nXLnkCzPgc2CKDM64gcrhcvHL4Pd49/SEuf6pS9ZMSncyX5tzC7HH5gR2cEEL4QGvNkXPt7DjRQJ/T3FW1uEgbKwozmJwRF+DRDZOuRiN49qfirsUKxbdD+rTgjUsE3ZADaKXU32mt/+3ivwEfoY8kgB7ltIYz7xs3f8RlGLmiI0fni29lcx1Plr5BbccFU+2VUlyTPYuH59xAbOTo+yAhhBid2rqdbK6oo6q523QfM8YnsnRKGlH2Ubqvo60GDv3BSNHqK4sNZtwJqQXBG5cYFoEIoPdqrede/DfgI/SRBNBhovojOPmuf22ik4wgOmZ05kz2eDy8XrGLDSd24HCby5OaEBnPF0tuYlGezGgIIYLH49Hsr2ll58lGnG5zV6oTo+2sLsokLzUmwKMbRo0njGxSHj+yjNgiYMZdxtVTMeoFMoDep7UesZXwEkCHkQuH4eh68GejnT0aSr4ACdnBG1eQ1bQ181TpOipbq0z3MTuzmK/Mu5Gk6NgAjkwIIaCps4/NFXWca/VjxrUfpWB2bhLXFaQRYRvFue3P7Yfjb/tXmjsyDmZ+AeIzgzYsMbwkgBahqbkSDv/RSAvkK6vdKLgyii+NeTwe3jqxlz8d3Uqvy9ybVGxEDHdPv4EVk2YGeHRCiLHI7dGUnmlm9+lm3B5zs86pcRGsKc5kXGJ0gEc3jLQ2iqOc3uFfu5hUY4InOikowxIjQwJoEbrazxulvx1+rLELk53NDZ3tPFW2nopGP4rNDFCUNplH560lPS4hgCMTQowlde29bCyvo7Gjz1R7i1IsmJjMwvwUbNZRPOvs8cDJTf5njErIhpmfh4hRvFxFDEoCaBHaupuNHc49rf61K1gJuQtHXa7ogbZWHuLlIxvp8udDRD9RtijuKFzBTVPmSjlwIYTPnG4PuyqbKDvb4tdKhf4yE6JYU5xJevwo3+DsdkHFG/4X/kotgOI7jLXPIuxcLoD2JxHjNu+/WwMyIiH6i0mBOV80ZqI7/EgvfmoL9LbC5DUwigPHFZNmMmfcJJ4pe4d9dUf8bt/r6uWlw2+xp7acR+bdQk7i6NxoKYQYPtXN3WyuqKO129ymZptFcW1BKnPzkrFYRvckBo4uo0BKW61/7bJmwrSbjJR1YkyRPNAitLj6jDXRLWf8a5cy0ZgBsEcFY1TDalfVMZ4/+BbtfR2m2kdY7dw85Xo+W7RIZqOFEJfoc7l5/0QjB2vaTPcxPjmaNUWZJMeGwaxrVyMcfAV6/TwfE66FictG/RVQcWVDWsKhlCoDfg28qLVuCcL4fCYB9BjgccPRN6Gu3L92sWkw8y6ITg7OuIZRV18fz+7byO5zBzD7ITc3PptH5t/CpBTZDS6EMJxu7OLdijo6ev1Iy9ZPhM3C0ilpzByfOPrKcA+muRKOvA4uh+9tlILJqyHnkphKhKGhBtCTgS8BdwOlwG+AjXoEpq8lgB4jtDbyRNfs8a9dRIxRtTAxJzjjGmb7zp3m2f3rae4x97nVZrGyauK1fGHGUiJso7B0rhAiIHocbrYfr6fivLkrWwAT02JZWZRBQpQ9gCMbQbVlcGKzf6lULVYouhUyioI3LhFSAlLKWyllAW4Bfgm4MQLp/9RaNwdqoFcjAfQYojVU74ZTfi67t9ig8GbInB6ccQ2zXqeD3x/YxvaqPXj8eaHvJyM2jUfm3kpRRnh8sBBC+EZrzfG6TrYdq6fb4TbVR3SEleXT0pmWGR8es84ej7F/xt8JGluEMUGTnB+UYYnQFIgsHCUYs9A3A+8AvwOWAF/UWs8O3FCvTALoMejCITi6wb9ZAoD8JcYtHF7wgaMNNTxd9iZ1XQ2m2luUhSW587l/9nJi7KN8t7wQ4qo6+1y8W1FHZUOX6T6mZcWzfFo6MRFhcgXL1Qflb0CTn6lDI2KNSrhSIGXMCcQa6FaMddCvaa37+v3sj1rrOwM41iuSAHqMajplrFPztwR2ZjFMWwvW8Hjxd7ndvHz4PTZX7sTlMTeblByVxJfm3MKc7IkBHp0QIhRorTlyrp0dJxroc5q7ahUXaWNlUQYF6XEBHt0I6m2DQ3+ATj8nIWJSvAVSRv/+GuG/oQbQk7TWlUEZmZ8kgB7DOuqMFz9/s1Mkjjcuu0WET9nryuY6ni59k+qOc6b7uGb8bB6as5r4yFFcMUwI8Smt3Q42V9RT3WwupzzAzPGJLJmSRpQ9jFKztZ+DQ68a6er8kTzBqHxrl9fJscpUAK2UegD4vdaDXztXShUA47TW7wdspFchAfQY19dhvAh2XPCvXXSSUSUqNi0owxoJHo+HP1XsZv2J7Tj8nZn3io+I4/6SG1k8QTbECDGaeTyafdWtfHiqEafb3P7+xGg7a4ozyU0Js2p69Ufh6DqjUIo/xpXA1Bslx/MYZzaA/jbwZaDMe2sAooDJwDKgEXhca30iGIMejATQArcTKtb5Xy3KFmHMJKRMCs64Rsi59maeLH2TUy1nTfdRklHEI/NuJDkmjC7XCjFGNHb2sam8jgttvabaKwVz85K5tiAV+2guwz2Q1lD1IVRu96+dUjBpOeReEzZ7aIR5ppdwKKWswEpgMTAO6AEqgLe01lVBGOsVSQAtAOOF8fR2OPuhf+2UBSavgvHzwuqF0ePx8PaJfbx+dAu9LnNvojH2aD5fvIaVk2ZKARYhRgG3R/PR6Wb2nGnG7TE365wWF8Ga4iyyEkd/EapPcTvh+Ntw4bB/7aw2KLoN0qcFZ1xi1AlIGrsBHf6V1vpnVznmRuA/ASvwtNb6x5c57nPAq8ACrfUVo2MJoMWnnD8Ax98xiq/4I2uGcWnOGib5TL0aOtt5uuwtyhuPm+5jWmoBj85fS2ZcYgBHJoQIpAttvWwqv0Bjpx8FQPqxWhQL8lNYODEF62gvwz1Qb5tR0dbfpX6RcTDjLkgYF5xxiVEpGAF0ldY67wo/twLHgTVADbAHuFdrXT7guHhgPRABfFMCaOG3lrNw5I/g9HPmNT4Tpt9prI8OM9sqD/PSkXfocpjbSBRpi+D2qStYO22+zEYLEUKcbg8fnmpib1ULZkuZZSVGsaY4k7S4MExn2XIGjvwJnD3+tYvLMPbJRCUEY1RiFLtcAD2Ud8arfWRdCJzUWldqrR3AS8Dtgxz3v4GfAOauOwuRPAHmPmSkGvJHRx2UPWu84IaZ5ZNm8OPVX2Nu1gxT7ftcDl4pf4fvbfkt1a1NAR6dEMKM6uZunv/wLGVnzQXPdqvi+qnp3D0/N/yCZ62h+iM48LL/wXPaFJjzRQmehV+GEkBf7c93PFDd7/sa730fU0rNBXK11uuv1JFS6jGlVKlSqrShwVwRCRHmYlJg7oOQdNmLIoNz9sCBl4wX3uGvTB9USdGx/PXiz/LNBfeQGGnujeFMWzXf2/Ykrxx6H5fbXN5pIcTQ9DrdbC6v49WyGtp6zGXcyU2J4YFFE5g3IRlLuC3ZcDuh4g04+a7/BbdyFxhXIm0RwRmbCFtXrC6hlOpg8EBZAUNKiugtC/5T4OGrHau1fhJ4EowlHEN5XBHG7NEw6x5j48j5g76309p44W0/B9NuDrsX0mvypjAj82s8t38Tu2r34++yLafbxbrjWyk7V8Fj82+lIDUrSCMVQgx0qqGTLRX1dPb5mYLNK8JmYdnUdKZnJ4RHGe6Belrg8Gv+F0dRFpiyBsbPDc64RNgzvQb6qh0rdS3wr1rrz3i//wcArfX/9X6fCJwCOr1NsoBm4LYrrYOWNdDiqrSGql1Quc3/tnHpRtGVMK04dfDCGZ7Zt56m7mZT7a0WCysmLOLekmVE2MKjuqMQoajb4WLbsQaOXfCzcFQ/k9JjWVmYQXxUeG2W/lhzJZT/2f/9L7ZImH5H2KU0FcER8E2EPjygDWMT4SqgFmMT4X1a6yOXOX4b8DeyiVAETMNx47Kev0VG7FFGGqPUguCMa4T1Oh28eHA7285+hMffy51e6TFpfGXuLUzPzA3w6IQY27TWHKvrYNuxBnoc5pZNxURYWT4tg6mZceE563xxkuT0dv+X3sWkGJMkYVRUSwTXsAfQ3ge9GfgZRhq7Z7TWP1JK/QAo1Vq/MeDYbUgALQKtq9FIZ9Tt50Y4pWDiMshbFFb5ovs71nCOp/eu40Jnvan2SimW5M7jgdkribGH2YYkIUZAe6+TrUfrqWzws9x0P0Xj4lk2NYPoiDCtnudywNE3/S+kBcZmwcJbjEkSIXw0IgF0MEgALfzm7DVecBtNFMxMnwaFa41LfmHI5Xbz6pEPeOfU+7j8zaXtlRSVyEOzbmZ+zuQAj06IsUFrzaHaNt470YjDZe6qUHyUjZWFGUxKD+Nqot3Nxnrnrkb/205cChMWh+2EiAgeCaDF2KY1nP0Azrxv4pJfqrFeLi4jKEMLBVWtDfxqzzqq2mtN97FgXAlfmncD8ZFD2l8sxJjS2u1gU3kdNS1+pl7rpyQnkSVT0oi0hemsM0D9UTi23piB9octAopuhzT5gC/MkQBaCIDGk8a6aFeff+0sNpiyGsbNDtsZDI/HwxtHP2Ld8e043Oaqm8VHxHF/yY0snlAU4NEJEV48Hs2+6hZ2nmzCZbIMd1KMndVFmeSmxAR4dCHE7YJT70LtXv/bxqYZ6539rREgRD8SQAtx0VAuA2YUwbSbwnZJB8CFjhaeLF3PiebTpvsoySjkkXk3kRwTxpeThTCpoaOPTeV11LWbqx+mFMybkMyiSanYrWFcKbS7GY68Dmb2aYT58jsxfCSAFqK/oWxEiU42lnTEh28+ZI/Hw6ZTB3itYjM9/qaI8oqxR3NX0WpWFZRIOXAhAJfbw0dnmtlzugWPyffetPhIbijOJDMhzDfCXThs5PT3N4vSGNgALoaXBNBCDDSUVEgWKxSshPHzwvpFuqm7g1+XvsUhMx80vKakTOSx+WvJig/P3NpC+OJ8Ww+byuto6jS3PMpqUVwzMYX5+SlYw62SYH8uB5zc5F8xrIvCPAWpGBkSQAtxOWaT8YM3LdJaowpiGNtx+ggvHd5Ih6Pz6gcPIsIawW1Tl3Nr4QKZjRZjisPlYeepRvZXt/r9Of2icYlRrCnOJDUuzJcjdDZA+Z/MLa8L8yJYYuRIAC3ElfS0GPmizay1i0qE4tshcXzgxxVC2nq7+c3ejZSdP2S6jwmJOTw2/xbyktIDODIhQlNVUzebK+po6/FzGYKX3aq4bnIas3OSsITzrLPWcP6AMfPsNlGyPLMYpt5kZNwQIsAkgBbiatxOOLHR3KVDZYFJyyD3mrBe0gGwp+Ykvz2wgdbeNlPt7VYbn5m0hM9Nvw6bNYzTbokxq9fp5r0TjRyuNfc3ApCXEsPqokwSY8K0DPdFrj5jrXNduf9tx8hSOjGyJIAWwlcXDsHxd/zfvALG2rvCWyAijNNKAd3OPp7f9y4f1OzF7GvIuLhMHp13K1PSxgV4dEKMnJP1nWw9Wk9nn4mZVCDSbuH6KelMz04IzzLc/XVcMJbPdTf73zY6CYrvgAR5/RDBJQG0EP7oaoLy1401ef6KjDPWRadMCvy4QszhC1U8s28dDWbeAAGrxcLyvGu4b9ZyImy2AI9OiOHT1edi27EGjtd1mO6jICOOlYUZxEWG+d+C1lCzByq3gZkKqBmFxpINKckthoEE0EL4y+2EU1vMJfAH47LipOVhvy7P4XLx+wPb2Fa1G7fHXBni9JgUvjznVmZk5QV4dEIEl9aaivMdbD/eQK/TRDAIxERYWVmYweSMuPCfde5phaProbXK/7ZjoKCVCD0SQAthVn0FHNvgfwlZMCpgFd4S9hsMAU40nuepsnWc76wz1V4pxeKcuXxxzipi7GGebUCEhfZeJ1sq6jnd2GW6j6JxCSybmk50RJjvB9DaWB53cpPJ19JUI/9+XEbAhybElUgALcRQ9LTAkT8Za/b8pRTkXQv5S4xNL2HM5Xbz2pGdvFP5Pk4zu+mBxMgEHpq9lgU5kwM8OiECQ2vNwZo23j/ZiMNl7qpLfJSN1UWZ5KfFBnh0IcjRBcfegsYT5tpnzYQpN4T91TwRmiSAFmKoPG6o3ArVe8y1j8+EwluNfKVhrqq1gSdL3+RsW43pPuaNm8mX5t5AYlR4b8gUo0tzl4PNFXXUtvSY7mN2bhLXTU4l0hbeH6gBaDgOx98CR7f/ba02mPIZGFcS+HEJ4SMJoIUIlMYTRhlwM4VXLDYj3V3OgrBfw+fxeFh/rJQ/HduKw22u+lpcRCz3zvgM10+cHuDRCeEfj0dTVtXCrlNNuDzm3jeTY+ysLs4kJ3kMfCh09sLJzcayDTPi0qH4sxCbGthxCeEnCaCFCKTeNiP9UlutufZJeUamjuikgA4rFF3oaOHJ0vWcaD5tuo/p6VN5dP7NpMbEB3BkQvimvqOXTeV11Lf3mWpvUYp5E5JZNCkFm3UMVOJsOWtMMvS2m2ufPQcmrwJrmOfAFqOCBNBCBJrHDWfeg6pdmKrRa4uAyashq2RMzEa/e+ogr1Zspttp7tJ3tD2KzxWtZk3BLCkHLoaFy+3ho9PN7DnTgsfke2V6fCQ3FGeSkTAGUq65XXB6m/llbrZImHqjUVlQiBAhAbQQwdJabcy29LSaa582xXjTiIwL6LBCUUt3J0+XvcXB+qOm+5iSMpFH5t1MdkJKAEcmxKeda+1hU3kdzV3mlh9ZLYpFk1KZNyEZaziX4b6o/bzxOtjVaK59cr5xVS4qIaDDEmKoJIAWIphcDiNn9Ll95trbIo2StONmhf1sNMB7p8t58fA7dDg6TbWPsEZw69Rl3Fa4UGajRUA5XB4+ONXIgepWUxeWALKTolhTnEVK7BjIGuFywJkdUFNq7kqc1QaTVsL4uWPitU+MPhJACzEcmk4ZRQIcJvPCJuXBtJuM/NFhrqOvh2f3buSjcwdN95GXMJ5H591CforkhhVDd7api80V9bT3OE21j7BZWDw5jVk5ieFfEAWM17vj7xh7QsxIGGdkJpKNgiKESQAtxHBxdMOJd8DsMgWLDSZcB3mLwj5vNEBpzUmeO7CBVpNvwjaLlRsmLebzM5Zgs4b/+RKB1+t0s/14A+XnTG56AyakxrCqKJPE6DGw8c3RBSffhboj5torC+QvhrzrQK4giRAnAbQQw0lrqC83Zmdc5nbuE5sG024eE1UMu519vLB/C+9Xl2H2NSkrLoNH5t7KtPTsAI9OhLMTdR1sPVZPV5+5MtxRdivXT02jeFxC+M86aw11h43g2eRmYGLTjOqsCeMCOzYhgkQCaCFGQm+7UQbcbAo3pWD8PJh4vbFOOswdqavm13vfpKHb3EYki7KwfMJC7i1ZRpR9DKw/FaZ19bnYeqyeE3Xm1uEDTMmMY8W0DGIjbQEcWYjqbjYmBFrOmO8jdwFMXCbp6cSoIgG0ECNFa6jdC5VbjDRPZkQlGBW50sK/vLXD5eLlQzt498yHuD3myiSnxqTw8Oy1zB6XH9jBiVFPa035+XZ2HG+k12lu1jk20srKwgwmZ4yBvOQeD9R8ZKTsHMrrV+FaI9OGEKOMBNBCjLTuZqh4w0j3ZFZGIUxeMyZS3p1qusBTZeuo7bhgqr1SikXjZ/PQ7DXERob/7L24urYeJ+9W1HG2yURZaa/p2QlcPzWdKPsYWG/fft4ow91RZ76PrJlGvnv7GMiDLcKSBNBChAKPB6p3w5n3wWNyNscWCQUrIGtW2G/Acbnd/Kl8FxtO7cBpcvYrITKeB0vWck3elACPTowWWmv2V7ey81QTDpe5qxoJ0XZWF2UwITU2wKMLQa4+Y8bZbGo6gMh4mPoZI8+9EKOYBNBChJLuZjj+tlHy1qz4TGM2Oik3cOMKUdWtTTxZuo4zbdWm+5iTOZ2vzL+RxKiYAI5MhLqmzj42V9RxrrXXVHulYFZuEosL0oiwhfcHVrSGC4egcpv5VJxKQfZcmLRsTOzbEOFPAmghQs3FN6tT74LT3Js7YJS9nbQi7Ct4eTwe1h8r48/Ht9DnMlcdLi4ilrun38DySTMCPDoRatweTdnZFnZVNuH2mHufS4mNYE1xJtlJ0QEeXQhqq4WTm4a2xCw2zchjn5gTuHEJMcIkgBYiVDm64ORmqCs334fVZuRUzb3G+DqM1XW28XTpeo42nTLdR3HaVB6ZdxPpceH9oWOsqm/vZWN5HQ0d5lJIWpRiQX4yCyemYLOG+axzX4cx43zhsPk+LFaYsHjM5K4XY4sE0EKEuqZTxrKOXvPFHIhOgoJVxrrDMM5J6/F42Hb6MC8f2Ui3yXy0UbYoPlu4khunzJFy4GHC6fawu7KZsrMteEy+t2UkRLKmOJOM+DDf9OZ2QW2psR/Dba7yImAsIZt6k1QTFGFrRAJopdSNwH8CVuBprfWPB/z8O8AjgAtoAL6stb7iotDBAmin00lNTQ29vUO4DC6uKCoqipycHOx2yd8ZVC4HnNkxtM07YKSLmrLGuKQaxlq6O3m67G0O1leY7mNycj6Pzl9LdkL4l08PZ7WtPWw6coGWbnPBoM2iWFSQyry8ZCyW8P3wCUDjSeOqV0+L+T4ubmYeNzusP6wLMewBtFLKChwH1gA1wB7gXq11eb9jVgC7tdbdSqmvA8u11ndfqd/BAujTp08THx9Pampq+FeCGgFaa5qamujo6GDixIkjPZyxof0cHHsLOuvN96EsRhGW/CVhn0Jq59mjvHDwLToc5opiRFjtrJ2yjDuKrpHZ6FGmz+Vm58km9le3mu5jfHI0a4oySY4N8+I7XU3GnoshLH8CIH2a8QE9cgzkwRZj3uUC6GAullwInNRaV3oH8BJwO/BxAK213trv+F3AA2YeqLe3l/z8fAmeg0QpRWpqKg0NDSM9lLEjIRvmPQzVH5lPeac9ULPHKL078XpjpihMg8PrJhQyM2sCz+3bzO7a/X63d7idvH50M6Xnynls3q3kp2QEfpAi4M40drG5oo6OXnMpDiNsFpZMTqMkJzG83z+cvXD2A++VLXNp/ABJTSdEP8EMoMcD/XNO1QDXXOH4rwBvDfYDpdRjwGMAeXl5gzYO6xe/ECDndwRYrDDhWmO259QWaDxhrh9nj1GCt6bUCKTTp4XlJdf4yGi+uehWrq0t5rn9G2jpbfW7j+r2c3x/+9OsnnQdd89Yis0qG6JCUY/DzfbjDVScN79fID8thlVFmSREhfGyNLfT+Luv+tDI7WyWskDOfONqlqSmEwIIbgDtM6XUA8B8YNlgP9daPwk8CcYSjoE/f2F/M2k1x4M6xoH+es3UYX08MYbFpMDMu4zLriffhe4mc/10N8GR1yE+y8jRmjwxLAPpeeMLmJ7xVZ7fv5X3q0vx+Dnj5vK4efvke+w7f5RH5t1CYbqk5AoVWmtO1Hey9Wg93Q5zZbij7FaWTU2naFx8+E4MeNxw/oAx69xnblnTx1ImGZUEZZOgEJ8SzOu5tUD/Cg853vs+RSm1Gvgn4Dat9RA+Io+suLhLSyvv2LGDuXPnYrPZePXVV/1qK8QlUgtgwVdg8iqwDWGtZscFOPAyHHjRWGsdhqLsETy64DM8vuQhMkxupKzrauD/vvccvy59h16nubzTInA6+1ysO3ie9QfPmw6ep2bG8+C1EyjOTgjP4FlrIx3mR08ZV52GEjxHJ8PMz0PJFyR4FmIQwZyB3gNMUUpNxAic7wHu63+AUmoO8CvgRq31EHZLhaa8vDyeffZZ/uM//mOkhyLChcUKuQshoxhO74ALB81n62g5C2XPQfpUmLgsLDN2FGXk8H9XP8orh9/j3dMf4vL4F3h5tIdtZz/iYP0JvjTnFmaPyw/OQMVlaa05cq6dHSca6HOaW78bF2ljRWEGkzPCdLJCa2iuNPI5D2XjMYDVbizVGD8/7HPKCzEUQfvr0Fq7lFLfBN7BSGP3jNb6iFLqB0Cp1voN4N+BOOAP3tmAKq31bcEa03DLz88HMLWrf//+/Xzta1+ju7ubgoICnnnmGZKTk/n5z3/OE088gc1mo7i4mJdeeont27fz7W9/GzDWKu/YsYP4eNkdHdYi46DwZsieY1QPa7vk4o7vGo4b66uzZhpvnFGJgRtnCIiw2Xhg9gquyyvmydI3qO244HcfzT0t/PTDF7gmexYPz7mB2EhZBzoc2rqdbK6oo6q523QfM8YnsnRKGlH2MF3P3lZjBM6t5svcfyxrBkxaLtk1hPBBUD9eaq03ABsG3Pfdfl+vDubjj2YPPvgg//Vf/8WyZcv47ne/y/e//31+9rOf8eMf/5jTp08TGRlJa2srAP/xH//BL37xCxYvXkxnZydRUeGdskz0kzAO5nwR6o5A5Vbzl2y1hvMHjX6y5xqbFyNiAzvWETYpJZP/s/orvF6xiw0nduDws3iE1ppdtfspbzzFF0tuYlHetCCNVHg8mv01rew82YjTbe4KS2K0ndVFmeSlxgR4dCGiswFObze/ubi/hHEweQ0kjh96X0KMEeGZ02qUa2tro7W1lWXLjD2VDz30EDt27ACgpKSE+++/nxdeeAGbzfj8s3jxYr7zne/w85//nNbW1o/vF2OEUsbM0cLHhl5K1+M2Ut/t+iWcGkJAHqIsFgufm34d31/xGAXJE0z10d7XwS/2vML/e/81Wnu6AjxC0dTZxyul1Ww/1mAqeFYK5k5I5oFFE8IzeO6sh/I/Q+mvhx48R8QaV7LmPiTBsxB+kgB6lFm/fj3f+MY32Lt3LwsWLMDlcvH444/z9NNP09PTw+LFizl69OhID1OMhIuVwRY8YqxrHgq3E6p2GYH08Y3Q0xqQIYaKnMQUvrv8Ae6ZcRNRNnNXbPbXlfP45ifYcupggEc3Nrk9ml2VTfxudxXn28xVlU2Ni+DuBbksm5pOhC3M3t7aauHQq7Dn18ZGwaEUQbPYIO8auOarMG5WWGbjESLYZKoyBCUmJpKcnMx7773H0qVLef7551m2bBkej4fq6mpWrFjBkiVLeOmll+js7KSpqYmZM2cyc+ZM9uzZw9GjRyksLBzpX0OMlJgUmPE54w339HZjs6BZHhfUlsG5fZA5HfKuDZsd+RaLhbXT5rNw/FSeKltPReNJv/vocnTzm/1/ZlfNER6dt5b0uIQgjDT81bX3srG8jsYOc4mYLEqxYGIyC/NTsFnDKHDWGlrOGHmch/J3fJGywLgSmLAYouS5KsRQBK2Ud7AMVsr7n178gLS09GEdx8A80BaLhezs7I+//853vsPSpUv57Gc/S0tLC1FRUWRlZXHkyJFL+hqs7cqVKz/eRDhp0iR+85vfEBcXx4oVK2hra0NrzQMPPMDjjz/Ot771LbZu3YrFYmH69Ok8++yzRAZhk1NFRQVFRUUB71cEkdbQchoqtxvp64ZKKUibChOuM/JJh5GtlYd4+chGuhzmNqxF2aK4o3AFN02ZK+XAfeR0e9hV2UTZ2RbTE6qZCVGsKc4kPT6MNnZqbSzPqNoJ7ecD02dGIeRfHzYfgIUYLpcr5R0WAbQEdsNDzvMopjU0HDNmpLubA9NnyiQjkE7Kvfqxo0RrTxfPlL3DvrpLP+j6alJSHo/Ov5WcxJQAjiz8VDd3s7mijtZu/zZzXmSzKK6bnMqc3GQsljBZguDxQH25MePc1RiYPlMmGmkqE8YFpj8hxpjLBdCyhEOIsUApYwYqbaqRO/rM+9DXMbQ+myuNW1KusbQjZdKoX0uZFB3Ld5bcya6q6Tx/8C3aTZyjytYqvrf1SW4qWMqd06+V2egB+lxu3j/RyMGaNtN95CRHs6Y4k6SYIRQUCiVul/F3Wb07cPsNEsYZKemS8wPTnxDiUySAFmIssVgge7axnrl2r3GJ2Gluw9bHWquNW2wajJ8HmTOGVikxBCzKm8bMzHye3beR3ecO4O+VOofbyZ+Pb6H0fDmPzruVgtTwWu5iVmVDJ1uO1tPR6zLVPsJmYemUNGaOTwyPSoK97XB+P5zbD44AZXSJTTNmnNOmjPoPtEKEMlnCIXwm5zkMOXuh5iOo/sjIvBEItkhjo1L2XGND4yi3//wZfrPvTZp7Wky1t1osrJ54HV+YsZSIMZpissfhZvvxeirOm7/qMSk9lpWFGcRH2QM4shGgtVH8pLbUKGKkzVVXvERUAuQvNT7AylUPIQJGlnAIIS5lj4KJ1xvBbs0eOLcXXI6h9enqg+o9UFMKKQUwfu6oXt4xe1w+P0l7jN8f2Mb2qj14/Ax43B4P75x6n30XjvLI3FspysgJ0khDj9aa43WdbD1WT4/DvzLqF0VHWFk+LZ1pmfGje9bZ7TQKFdWWDb3cdn9RiUZKuqxZUnpbiGEkf21CCKM0eMEKoxBLbZkRTA91aYfW0HTSuMWkfLK8wz76KmVG2SP48vwbuG5CMU+XvUldV4PffdR3NfLj959jSe587p+9nBh7GGWNGERHr5MtR+upbDC/NGFaVjzLp6UTEzGK36p6Wo0PpucPDP1vqr+YVKNiaEbx0IonCSFMCYslHA1/+A7pw5zGjhX/MLyPFwJkCccY4nIYazOrdwe2GqHVDlkzjWA6Ni1w/Q4jl9vNy4ffY3PlTlwec7OqyVFJPDT7ZuaNLwjw6Eae1prDte3sONGAw2VueUJ8lI0VhRkUpMcFeHTD5GL+5toy4wNkIN9n47OM7DdpU0ftVR0hRhNZwhFkcXFxdHZ+OtD46U9/ytNPP43NZiM9PZ1nnnmGCRMuLR88WFshRpQtAnIXGks76g4ZVQkDkR3A7TQ2L9buhaQ8I5hOLxxVmw5tViv3z1rO4txinixbR3X7Ob/7aOlt5We7fs/C7BIennsD8ZHRQRjp8GvtdrC5op7qZnO5tAFKchJZPDmNKPsonFXtbTeWaVw4BN1Nge07KdcInJMnSuAsRAiQADqI5syZQ2lpKTExMfzyl7/k7/7u73j55ZdHelhC+M5qg+w5xvrKhgo4uzNw+Wlbq4zbiY2QPs0IppMmjJrgID8lgx+u+hJ/qtjN+hPbcZjYhPnRuYNUNFZyf8mNLJ4weq/ueDyafdWtfHiqEafb3GxrUoyd1UWZ5KbEBHh0QeZ2QuNxI2huORPY2WaA1AIjTWQY5VsXIhxIAB1EK1as+PjrRYsW8cILL/jcdv/+/R9XIiwoKOCZZ54hOTmZn//85zzxxBPYbDaKi4t56aWX2L59O9/+9rcBUEqxY8cO4uPjA/77iDHMYjFS32UUG5ekz34QuAppbidcOGzcohKMddJZM0dFBg+LxcKd069lUe40nipdz8mWM3730eHo5InSV9lZVcQj824kOWZ0LVto7OxjU3kdF9rMre9VCubmJXNtQSr20VKGW2toqzaesw0VQ994O5BSxofKvOsgPjOwfQshAkLWQJs1YA301ZZhfPOb3yQrK4t//ud/vuRng7UtKSnhv/7rv1i2bBnf/e53aW9v52c/+xnZ2dmcPn2ayMhIWltbSUpK4tZbb+Xxxx9n8eLFdHZ2EhUVhS0I6bJkDbT4mNbQetZY49l4IvCzbgCJ441gOqN4VGw89Hg8bDy5n9cq3qXXZS6YjLFH8/niNaycNDPkC7C4PZqPTjez50wzbo+5//+0uAjWFGeRlRj6/78A9LQYQXPd4cAVPOnPaoesEsiZPyo+QAoxFsga6BH0wgsvUFpayvbt2306vq2tjdbWVpYtWwbAQw89xOc//3nACKzvv/9+7rjjDu644w4AFi9ezHe+8x3uv/9+7rzzTnJyxk6aLDFClDIqnCXne7MM7PNmGegJ3GO01Rq3k+9C2mTInGk8Xoim6rJYLNw4dS4Lcqbw5J4NlDce97uPbmcPzx14g101R3h0/loy4xKDMNKhu9DWy6byCzR2mpt5tVoUCyemsCA/BWuol+F29nyyRKO1OjiPEZNqbKzNmmHkURdChLzQfCcKI5s3b+ZHP/oR27dvJzJy6C+M69evZ8eOHaxbt44f/ehHHDp0iMcff5y1a9eyYcMGFi9ezDvvvENhYWEARi+ED6KTjBR4+UugvsIoENFRF7j+PS6oP2rcbBGQOtnIQJBSEJKbD1Nj4vmHZXezrfIwLx/ZSKeJCnPHmk7xT+8+we1TV7B22vyQmY12uj3sPNXEvqoW0xcdxiVGsbo4k7S4EA4U+zqMoLnhuLFOP1DFTvpTynguj58rGwOFGIUkgA6iffv28dWvfpW3336bjIwMn9slJiaSnJzMe++9x9KlS3n++edZtmwZHo+H6upqVqxYwZIlS3jppZfo7OykqamJmTNnMnPmTPbs2cPRo0clgBbDz2o3KhBmzYT2WmN5R/3RwAYfLgfUlRs3iw1SJhrBdOpkiAitzWfLJ81gTvYknil7h70XDvvdvs/l4JXyd/iotpzH5t9KblJqEEbpu+rmbjaV19HWY65ipd2quLYgjTm5SVhCcda5u9kImhuPG1c+gsUeZSzTGD8XopOD9zhCiKCSADpAuru7P7V04jvf+Q4bNmygs7Pz4+UXeXl5vPHGGz61fe655z7eRDhp0iR+85vf4Ha7eeCBB2hra0NrzV/+5V+SlJTEv/zLv7B161YsFgvTp0/npptuCv4vLMTlKAWJOcatYCWc228s8TAxE3tFHpex/rrxBCiLkaUgbRqkTTE2I4aAxKgY/nrxZ9ldNYPnD26gra/d7z7OtFXzvW1PcmPBUu4svhabdXjTu/U63bx/opFDtW2m+8hNiWFNUSaJMSFUhltroyJg4zEjaO70vziOX+LSYfx8YzOuNYTOgxDClLDYRCib24aHnGdhmscNDceMQLq1KviPlzDOCKZTJkFcRkhcHu/q6+O3B97lw5q9mH3dzY7L4rH5t1KQmhXg0Q3uVEMnWyrq6exzmWofYbOwbGo607MTQqMMt9sF7TXQdMoImoOxEbA/i834QDd+LiTmhsTzUAjhn8ttIpQAWvhMzrMIiJ5WI4vBhcNGVoNgs0d/suExOd9Ysz2CDl44wzP71tPU3WyqvdViYcWERdxbsoyIIGTbAeh2uNh2rIFjFzpM9zEpPZaVhRnER43gbKvW0Fln5GduOWOknnOb+zDgl8QcY0NgetGoyCAjhLg8CaDFkMl5FgGlNbTVGMF0fXngc+leTnQyJE8wgumkCSOydrrX6eClgzvYenY3HpNrxNNj0vjK3LVMz8wL2Li01hyr62DbsQZ6HObKlMdEWFlRmMGUjLiRmXXuafkkYG45G9jMMFcSlWgEzZkzJAWdEGFE0tgJIUKLUsa65aRcmLzaWMt84RC0nA5OXumLelqM27n9xhjiMj6ZnU7IGZbMHlH2CB6et5pr84p5eu86LnTW+91HQ3cjP/ngtyzJnccDs1cSYx9aVov2Xidbj9ZT2WB+rXrRuHiWTc0gOmIY12n3dXqrWp41guZgL8voz2o3StFnzTRK08sSDSHGDAmghRAjz2qHzGLj1tcBdUeMYDpQZcMvR2sj5V5HHVTtNgKg2DSIzzbWUcdnQ2y6UYkxCKalZ/N/Vz/Cq0c+4J1T7+Py+Dfrq7XmvapSDtWf4KFZNzM/Z7LfY9Bac6i2jfdONOJwmZsNj4+ysaook4lpsaba+8zthI7zRhXMjnPGv73mNzeaopRx5SJrhrHOPgRTKQohgk8CaCFEaImMh7xFkHuNsX614aiRj7e7KfiPrbWRjaGzwSgMA0bhlvhxxi0h2/g3KjFgs402q5V7Sq7nurwifrVnHVXt/qdQa+1t4z93v8iCqhK+NO8G4iOjfWrX0uVgc0UdNS3mlznMyk1k8eQ0Im0BnnX2eKC7EdrPeYPmc8YHqmDkZL4apSBhvFFeO32a8f8vhBjTwmIN9Pc3fZ/09OEt5f0Xs/9iWB8vFMgaaDGiuhqNTB6Nx6HjwsiOJSLWCKbjMiAmzagkF5My5PRkHo+HdUf38MbxbTjc5taEx0fEce2460iPysWiLj9z7tGaE3WduEyW4U6OsbO6OJOc5ACsIXf2GsFyd5Px/9xxwQia3eZyTgeExWrMNKdPhdQpEBk3cmMRQowYWQMdZHFxcXR2dn78/bPPPktpaSn//d//zRNPPEFMTAwPPvhg0B7/4Ycf5pZbbuGuu+4K2mMIMaJi04xb/mJjnWvjCSOHb1tNcNdMD8bR9UkO6ouUMmYmY1I/fYtNMzKB+MBisXB78TVckzuVJ0vXc6L5tN9D63B0svHsRmwqkiT7eJJsOcRZM7CowMwQW5Ri3oRkrpmUgt3qx9IWrY3lOd2NRtGSi8Fyd1Pgc4SbZbUZFS7Tpxn/SgYNIcRlSAA9DL72ta+N9BCECC/RSZC7wLh9HMweNzaR+bmOOGC0NgL7nlYjz3B/ETGfBNSR8RARZ/x78WaL+tSSkKz4ZP552X1sOnWA1yo20+Ps9Xs4Lt1Ho6OSRkclVmUn0TaeJNt4EmxZWJS5l/70+EjWFGeSmTBIYOnxgLPL2NTX1wGODuPf3nYjSO5uGtkZ5cuxR3nLw08zKltKkRMhhA8kgB4G//qv/0pcXBx/8zd/w/Lly7nmmmvYunUrra2t/PrXv2bp0qV0d3fz8MMPc/jwYaZNm8a5c+f4xS9+wfz589m4cSPf+9736Ovro6CggN/85jfExV39cmJvby9f//rXKS0txWaz8dOf/pQVK1Zw5MgRvvSlL+FwOPB4PLz22mtkZ2fzhS98gZqaGtxuN//yL//C3XffPQxnR4ghioiF7NnGzdUHzZWfpDEbzowMV+LoNm6t1YP/3GqDiIsBtRFcWyLi+UxiHAvm3cRz5Ts51HoGD1a0suJRFsD3Ndhu7aTZeYZm5xmsykaCLdsbTI/Dqi4TMGoPVo8Dm3Zix8n8nFhKxlmxdp+EVm+QfPHm6DQC55FYn+wvpSAu85PMK0l5xnINIYTwgwTQAdLT08Ps2bM//r65uZnbbrtt0GNdLhcfffQRGzZs4Pvf/z6bN2/mf/7nf0hOTqa8vJzDhw9/3FdjYyM//OEP2bx5M7GxsfzkJz/hpz/9Kd/97nevOqZf/OIXKKU4dOgQR48e5YYbbuD48eM88cQTfPvb3+b+++/H4XDgdrvZsGED2dnZrF+/HoC2tmHe2S5EINgiIaPIuIE3J/DZTwLq4coJ7C+365P0egOkAH9tg/0RDja0nqXDY8ziaix4lDegxuINrK34Flgfpx3oVIo04onWCot2Y9UuLNqFxeNCYQTD0XYLWQlRdFRb+eAy8X/Ii4gzgua4DONm86b86642bkKIUSUxMpFZ6bNGdAwSQAdIdHQ0+/fv//j7i2ugB3PnnXcCMG/ePM6cOQPA+++/z7e//W0AZsyYQUlJCQC7du2ivLycxYsXA+BwOLj22mt9GtP777/Pt771LQAKCwuZMGECx48f59prr+VHP/oRNTU13HnnnUyZMoWZM2fyv/7X/+Lv//7vueWWW1i6dKnf50CIkBOdbNyyZ3szbNT3q0pXNTxV6QJkdkIqU2ITeLPxLPt7mlB4sGoP6KEti+igmUvqDSqwAAnRdlyRNjo9vdA3pIcZXla7scwnynu7uJbZ1QatMjkgxGg3Pm58eAfQSqkbgf8ErMDTWusfD/h5JPBbYB7QBNyttT4TzDGFgshIY/bDarXicl35DVxrzZo1a3jxxRcD9vj33Xcf11xzDevXr+fmm2/mV7/6FStXrmTv3r1s2LCBf/7nf2bVqlU+zXILMWooBfGZxi3vGiN4bq/1BtM1I5/1wQexVjt3Z06mpDONN5pP0+oJTvXGKJuF5JgIbJZRUhjEGmEsf4lKNAJne4wUNRFCBFXQAmillBX4BbAGqAH2KKXe0FqX9zvsK0CL1nqyUuoe4CfAmFx4u3jxYl555RVWrFhBeXk5hw4dAmDRokV84xvf4OTJk0yePJmuri5qa2uZOnXqVftcunQpv/vd71i5ciXHjx+nqqqKadOmUVlZyaRJk/jLv/xLqqqqOHjwIIWFhaSkpPDAAw+QlJTE008/HexfWYiRZbV5S3pPML73eIyNbhcLdHScM/JBh+C63qK4JCbGlPBWQxV7uuvRAYoVLQqSou3ERNj8WGE9zCyWfuvFvWvGrZESMAshhlUwZ6AXAie11pUASqmXgNuB/gH07cC/er9+FfhvpZTSoy05dQD8xV/8BQ899BDFxcUUFhYyffp0EhMTSU9P59lnn+Xee++lr8+4hvrDH/5w0AD6q1/9Kn/1V38FQG5uLlu3buXrX/86M2fOxGaz8eyzzxIZGckrr7zC888/j91uJysri3/8x39kz549/O3f/i0WiwW73c4vf/nL4fz1hRh5FgvEpRu3cd5Lg27nJzmJLxb0CJGNiVEWK5/NnMis7lTeaDpLnbvbdF8KiLZbSYqxYw2lQFQpYza5f8YSmV0WQoSAoBVSUUrdBdyotX7E+/0XgWu01t/sd8xh7zE13u9PeY9pHNDXY8BjAHl5efPOnj37qccKhwIfbrcbp9NJVFQUp06dYvXq1Rw7doyIiNApExsO51mIIXN0GUF1V8Mn6dm6Go0MICPEoz2c7ungWG8rZ9ztdPmxLlopRYTVMvLLNWyRRr5se4zxb0SssflPMmQIIQYYHzee2yffPiyPNaoLqWitnwSeBKMS4QgPJyi6u7tZsWIFTqcTrTX/8z//E1LBsxDCKyIWUguM20VaG4F1d9OlhUL6LtmiF3AWZaEgJpGCmES01tS7ezjlbOO0q502dwjt/lPKyHn9cZAc88nXllHxdiSEEEBwA+haILff9zne+wY7pkYpZQMSMTYTjjnx8fGXzdohhAhxSnnzN8d9sqb6IlefEVD3NHvzJndCX7s3d3JHwPMnK6XItMWQaYvhWp1Fk6eXSmc7p5xttLj9L8jiF4vFWI9sjTBmlPv/a48xsmFcoby4EEKMFsEMoPcAU5RSEzEC5XuA+wYc8wbwEPAhcBewxez6Z601StbFBc0YXJYuRGDYIiFhnHEbzMXZ648LkrR/Us3P1WsE4O4+cDnA7TC+9jH9nlKKNGs0adZoFkZl0uw2gulKZxuNbh9yYlusxk3Z+n1tNdLEXQyM+wfJFpusTxZCjAlBC6C11i6l1DeBdzDS2D2jtT6ilPoBUKq1fgP4NfC8Uuok0IwRZPstKiqKpqYmUlNTJYgOAq01TU1NREUNUr5XCDE0/WevfeXxeIPqPmOjY/+vufyH3RTvbT7Q5uyksbeVbk8fWOxGUGy1f/rr0M3FIYQYw2IjYkd6CMHbRBgs8+fP1wOXOjidTmpqaujtDfLlyTEsKiqKnJwc7PbLlP0VQgghhAgzo3oT4dXY7XYmTpw40sMQQgghhBBjgOzmEEIIIYQQwg8SQAshhBBCCOEHCaCFEEIIIYTww6jbRKiUagDOXvXA4EgDGq96lLhIzpd/5Hz5R86Xf+R8+UfOl3/kfPlHzpd/RvJ8TdBapw+8c9QF0CNJKVU62E5MMTg5X/6R8+UfOV/+kfPlHzlf/pHz5R85X/4JxfMlSziEEEIIIYTwgwTQQgghhBBC+EECaP88OdIDGGXkfPlHzpd/5Hz5R86Xf+R8+UfOl3/kfPkn5M6XrIEWQgghhBDCDzIDLYQQQgghhB8kgBZCCCGEEMIPEkADSqkblVLHlFInlVKPD/LzSKXUy96f71ZK5ff72T947z+mlPrMsA58hPhwvr6jlCpXSh1USr2rlJrQ72dupdR+7+2N4R35yPDhfD2slGrod14e6fezh5RSJ7y3h4Z35CPDh/P1//U7V8eVUq39fjYWn1/PKKXqlVKHL/NzpZT6ufd8HlRKze33s7H4/Lra+brfe54OKaV2KqVm9fvZGe/9+5VSpcM36pHjw/larpRq6/d3991+P7vi33I48uF8/W2/c3XY+5qV4v3ZWHx+5SqltnpjhiNKqW8PckxovoZprcf0DbACp4BJQARwACgecMxfAE94v74HeNn7dbH3+Ehgorcf60j/TiFwvlYAMd6vv37xfHm/7xzp3yEEz9fDwH8P0jYFqPT+m+z9Onmkf6eRPl8Djv8W8Ey/78fU88v7O18PzAUOX+bnNwNvAQpYBOz23j/mnl8+nq/rLp4H4KaL58v7/RkgbaR/hxA7X8uBNwe536+/5XC5Xe18DTj2VmBLv+/H4vNrHDDX+3U8cHyQ98iQfA2TGWhYCJzUWldqrR3AS8DtA465HXjO+/WrwCqllPLe/5LWuk9rfRo46e0vnF31fGmtt2qtu73f7gJyhnmMocSX59flfAbYpLVu1lq3AJuAG4M0zlDh7/m6F3hxWEYWorTWO4DmKxxyO/BbbdgFJCmlxjE2n19XPV9a653e8wHy+uXL8+tyhvLaN2r5eb7k9Uvr81rrvd6vO4AKYPyAw0LyNUwCaOM/qrrf9zVc+p/38TFaaxfQBqT62Dbc+Ps7fwXjk+NFUUqpUqXULqXUHUEYX6jx9Xx9zntp6lWlVK6fbcOJz7+zd2nQRGBLv7vH2vPLF5c7p2Px+eWvga9fGtiolCpTSj02QmMKRdcqpQ4opd5SSk333ifPrytQSsVgBHuv9bt7TD+/lLE8dg6we8CPQvI1zDZcDyTGHqXUA8B8YFm/uydorWuVUpOALUqpQ1rrUyMzwpCxDnhRa92nlPoqxtWOlSM8ptHgHuBVrbW7333y/BIBoZRagRFAL+l39xLv8ysD2KSUOuqdcRzL9mL83XUqpW4G/gRMGdkhjQq3Ah9orfvPVo/Z55dSKg7jw8Rfaa3bR3o8vpAZaKgFcvt9n+O9b9BjlFI2IBFo8rFtuPHpd1ZKrQb+CbhNa9138X6tda3330pgG8anzXB21fOltW7qd46eBub52jYM+fM738OAy59j8Pnli8ud07H4/PKJUqoE42/xdq1108X7+z2/6oHXCf8le1eltW7XWnd6v94A2JVSacjz62qu9Po1pp5fSik7RvD8O631Hwc5JCRfwySAhj3AFKXURKVUBMaTeuDu/TeAi7s778JY9K+999+jjCwdEzE+dX80TOMeKVc9X0qpOcCvMILn+n73JyulIr1fpwGLgfJhG/nI8OV8jev37W0Ya8AA3gFu8J63ZOAG733hzJe/R5RShRibRj7sd99YfH754g3gQe9O9kVAm9b6PGPz+XVVSqk84I/AF7XWx/vdH6uUir/4Ncb5GjTTwliilMry7glCKbUQI65owse/5bFIKZWIcWX2z/3uG5PPL+9z59dAhdb6p5c5LCRfw8b8Eg6ttUsp9U2Mk27F2NF/RCn1A6BUa/0Gxn/u80qpkxibA+7xtj2ilHoF403aBXxjwOXksOPj+fp3IA74g/d1tUprfRtQBPxKKeXBeJH9sdY6rAMcH8/XXyqlbsN4DjVjZOVAa92slPrfGG9EAD8YcLkv7Ph4vsD4G3zJ+0H2ojH3/AJQSr2IkQkhTSlVA3wPsANorZ8ANmDsYj8JdANf8v5szD2/wKfz9V2MPS7/4339cmmt5wOZwOve+2zA77XWbw/7LzDMfDhfdwFfV0q5gB7gHu/f5aB/yyPwKwwrH84XwGeBjVrrrn5Nx+TzC2Oi44vAIaXUfu99/wjkQWi/hkkpbyGEEEIIIfwgSziEEEIIIYTwgwTQQgghhBBC+EECaCGEEEIIIfwgAbQQQgghhBB+kABaCCGEEEIIP0gALYQQI0Qp5VZK7VdKHVZK/cFb3tdsX88qpe7yfv20Uqr4CscuV0pd1+/7rymlHjT72AP6/nel1BGl1L8Hoj8hhAhFYz4PtBBCjKAerfVsAKXU74CvAR8XE1BK2bTWLn871Vo/cpVDlgOdwE7v8U9c8Wj/PAakhHtOfCHE2CYz0EIIERreAyZ7Z4ffU0q9AZQrpazeWd09SqmDSqmvglHBSyn130qpY0qpzUDGxY6UUtuUUvO9X9+olNqrlDqglHpXKZWPEaj/tXf2e6lS6l+VUn/jPX62UmqX97Fe91b4utjnT5RSHymljiullg78BbxjjgPKlFJ3B/d0CSHEyJEZaCGEGGFKKRtwE3Cx8thcYIbW+rRS6jGM0rULvKXKP1BKbQTmANOAYowqZuXAMwP6TQeeAq739pXird71BNCptf4P73Gr+jX7LfAtrfV2bwXI7wF/5f2ZTWu9UCl1s/f+1f0fT2t9m1Kq8+KsuhBChCsJoIUQYuRE9ytf+x7wa+A64COt9Wnv/TcAJRfXNwOJwBTgeuBF71KJc0qpLYP0vwjYcbGvq5W5VUolAkla6+3eu54D/tDvkD96/y0D8n36DYUQIgxJAC2EECOnZ+BsrVIKoKv/XRgzwu8MOO7moI/uUn3ef93I+4cQYgyTNdBCCBHa3gG+rpSyAyilpiqlYoEdwN3eNdLjgBWDtN0FXK+Umuhtm+K9vwOIH3iw1roNaOm3vvmLwPaBxwkhxFgnMwhCCBHansZYLrFXGdPTDcAdwOvASoy1z1XAhwMbaq0bvGuo/6iUsgD1wBpgHfCqUup24FsDmj0EPOFNqVcJfCkIv5MQQoxqSms90mMQQgghhBBi1JAlHEIIIYQQQvhBAmghhBBCCCH8IAG0EEIIIYQQfpAAWgghhBBCCD9IAC2EEEIIIYQfJIAWQgghhBDCDxJACyGEEEII4Yf/H4wsydNdqJioAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# define the losses for a target of y=1\n", "hinge_loss = lambda f: np.maximum(1 - f, 0)\n", "l2_loss = lambda f: (1-f)**2\n", "l1_loss = lambda f: np.abs(f-1)\n", "\n", "# plot them\n", "fs = np.linspace(0, 2)\n", "plt.plot(fs, l1_loss(fs), fs, l2_loss(fs), fs, hinge_loss(fs), linewidth=9, alpha=0.5)\n", "plt.legend(['L1 Loss', 'L2 Loss', 'Hinge Loss'])\n", "plt.xlabel('Prediction f')\n", "plt.ylabel('L(y=1,f)')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We can make a few interesting observations:\n", "* The hinge loss is linear like the L1 loss.\n", "* But it only penalizes errors that are on the \"wrong\" side: \n", " * We have an error of $|f-y|$ if true class is $1$ and $f < 1$\n", " * We don't penalize for predicting $f>1$ if true class is $1$." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAD4CAYAAADmbIA7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw9klEQVR4nO3deXCU953v+/e3W60NCe0SoKUFNhgwYAwySHEW77EdByeO7RgbSFJzTmYyJ5nMOalUcmam5szkZKpu7kzlVJ2aZJLce6Y8gI2NnY3ETrzEzmJbAgRmXwVGG6AdoV1q6Xf/kJyrYJbuh5ZaUn9eVSp3P/30t7/989PNRz893T9zziEiIiIiEm98sW5ARERERCQWFIRFREREJC4pCIuIiIhIXFIQFhEREZG4pCAsIiIiInEpIVYPnJub60pLS2P18CIiIiISJ/bs2dPqnMu7dHvMgnBpaSnV1dWxengRERERiRNmVnu57To1QkRERETikoKwiIiIiMSluAnCbd0DvLingbbugVi3IiIiIiJTQMzOEZ5MIyOO1482cfZCP8/srGPt/GzKSrPx+yzWrYmIiEgcGBoaoqGhgf7+/li3MqMlJydTVFREIBAIa/+4CML7Gy5w9sLogTc84njnVBsnm7u5b2kB+bOTY9ydiIiIzHQNDQ2kp6dTWlqKmSbiJoJzjra2NhoaGpg/f35Y97nmqRFm9u9m1mxmh65wu5nZ/zazGjM7YGarIux7QnX2DfHOqbYPbG/pGmDbrnreOtnK0PBIDDoTERGReNHf309OTo5C8AQyM3JyciKadQ/nHOGngfuvcvsDwMKxny8C/xb2o08w5xy/OdrEYOjyQXfEOXafaeeZqloaOnonuTsRERGJJwrBEy/SMb5mEHbO/R5ov8ouDwOb3agqINPM5kbUxQQ5cu4itW3XDrgdvUO8UN3AG8eaGAgNT0JnIiIiIhJr0ThHuBCoH3e9YWzbuUt3NLMvMjprTElJSRQe+sp6BkL87kRLRPfZX9/J6ZYe7l5SwPzcWRPUmYiIiMSbrfvayW04MamP+V/vXfQn19PS0uju7v7j9aeffprq6mr+9V//lR/84AekpqayadOmCevn85//PA899BCPPvrohD1GpCb1w3LOuR8BPwIoKytzE/lYR85dZGAo8nN/u/pD/OzdRpbMTedji/JJSfRPQHciIiIiU8df/MVfxLqFmIjG9wg3AsXjrheNbYupsmAW9y+b4znIHj3XxebKM5xo6sK5Cc3sIiIiIjH1D//wD/zLv/wLAHfccQff+MY3WLNmDYsWLeIPf/gDAL29vTz++OMsXbqUT3/606xdu5bq6moAXn31VSoqKli1ahWPPfbYn8w8X01/fz9f+MIXWL58ObfeeitvvvkmAIcPH2bNmjWsXLmSFStWcPLkSXp6evjEJz7BLbfcwrJly3j++eev+3lHY0Z4B/BlM3sOWAt0Ouc+cFrEZDMzlsydTTAnld8eb+H4+a6Ia/QODvPSgXPckJ/GXYvzSUuKi2+bExERkRmor6+PlStX/vF6e3s769atu+y+oVCIXbt28fLLL/OP//iPvP7663z/+98nKyuLI0eOcOjQoT/Wam1t5dvf/javv/46s2bN4jvf+Q7f/e53+fu///tr9vS9730PM+PgwYMcO3aM++67jxMnTvCDH/yAr371qzz11FMMDg4yPDzMyy+/zLx583jppZcA6OzsvO4xuWayM7NtwB1Arpk1AP8DCAA4534AvAw8CNQAvcAXrrurKEpNTODB5XO5aU46bxxtpnsgFHGNU83dNHT08tGFedw8b7Y+9SkiIiLTTkpKCvv27fvj9ffPEb6cRx55BIDVq1dz5swZAN566y2++tWvArBs2TJWrFgBQFVVFUeOHOH2228HYHBwkIqKirB6euutt/jKV74CwOLFiwkGg5w4cYKKigr+6Z/+iYaGBh555BEWLlzI8uXL+drXvsY3vvENHnroIT7ykY9EPAaXumYQds6tv8btDvgv193JBLshL43CzBTeOtnKwcbIf4MYGBrhtSNNHDvfxb1LCshIDW/FEhEREZHpJikpCQC/308odPVJROcc9957L9u2bYva4z/55JOsXbuWl156iQcffJAf/vCH3HXXXezdu5eXX36Zv/u7v+Puu+8Oa9b5aqJxjvC0kRzwc8/SAh5dXURGircgW9/ey5aqM+yt62BkROcOi4iISHy4/fbb2b59OwBHjhzh4MGDAJSXl/P2229TU1MDQE9PDydOhPcNGR/5yEd45plnADhx4gR1dXXcdNNNnD59mgULFvBXf/VXPPzwwxw4cICzZ8+SmprKhg0b+PrXv87evXuv+znF5UmvxdmpbKwI8s6pNt6t6yDSz8INDTt+d7yFE+e7uGdpAblpSRPTqIiIiMgU8Zd/+Zd87nOfY+nSpSxevJibb76ZjIwM8vLyePrpp1m/fj0DAwMAfPvb32bRokUfqPHnf/7n/PVf/zUAxcXFvPnmm3zpS19i+fLlJCQk8PTTT5OUlMT27dvZsmULgUCAOXPm8Dd/8zfs3r2br3/96/h8PgKBAP/2b9e/hpvF6hsRysrK3JXOS5lM5zr7eP1IE63dg57u7/cZa+Znc1tpNn6fzh0WERGRD/rbbW+Tm5s3qY956fcIX6/h4WGGhoZITk7m1KlT3HPPPRw/fpzExMSoPs71Onr0KEuWLPmTbWa2xzlXdum+cTkjPN7cjBSeXBtk13vt7D7TznCEpzsMjzgqT7Vxsrmbe5cUMCcjeYI6FRERkelqw8psliyJbjCdbL29vdx5550MDQ3hnOP73//+lAvBkYr7IAyjs7oVN+RwY34arx9t4nxnf8Q1WrsGeG53HatKsqi4IYeAP65OvxYREZEZLj09/YrfMjFdKa2Nk5eexGfLivnoojwC/shPc3AO9tR2sLWqlvr23gnoUERERKYrLdA18SIdYwXhS/h8xupgFhvKgxRnp3qqcaF3iBf3NPCbo030Dw1HuUMRERGZbpKTk2lra1MYnkDOOdra2khODv80VZ0acQWZqYl8ZlUhhxov8vuTLQyGRiKucaChk/dae7hrcT4L8tImoEsRERGZDoqKimhoaKClpSXWrcxoycnJFBUVhb2/gvBVmBnLizIozU3ljWPNnG7pibhGV3+In+87y+I56XzspjxSEzXkIiIi8SYQCDB//vxYtyGX0KkRYUhPDrDulnk8uHwuKYl+TzWOne9ic2Utx85f1J9FRERERKYABeEwmRk3zUnncxWlLJ6T7qlG3+Awvzp4nh37z9LVPxTlDkVEREQkEgrCEUpJ9PPA8rk8vHIe6cneTnM43dLD5spaDjZ0anZYREREJEYUhD1akJfGxoogK4oyPN1/MDTC60ebeHFPAxd6va1qJyIiIiLeKQhfh6QEP3cvKeDR1UVkpgY81Wjo6GNrVS17atsZiXBVOxERERHxTkE4CoqzU9lQHqSsNAuLfB0OhoYdvz/RyvPV9bR0DUS/QRERERH5AAXhKAn4fXxkYR7r15SQm57kqcb5zn6e3VnHO6daCQ1H/r3FIiIiIhI+BeEoK5idzJNrSqi4IQe/L/Lp4RHn2Hm6nW276jjX2TcBHYqIiIgIKAhPCL/PKF+Qw5NrS5ibEf4yf+O1dg/y/O56fnfC26p2IiIiInJ1CsITKDcticfLivnYTXkE/JHPDjsHe2s72FpVS3177wR0KCIiIhK/FIQnmM9nrCrJYmN5KSXZqZ5qdPYN8eKeBl470kT/0HCUOxQRERGJTwrCkyQjNcAjqwq5d2kBSQFvw36osZMtlbXUNHdHuTsRERGR+KMgPInMjGWFGWyqKOWG/DRPNboHQvxi/1lePniO3sFQlDsUERERiR8KwjGQlpTAJ1fM5RMr5pKa6PdU4/j5Lv7jnVqOnruoZZpFREREPFAQjhEzY1FBOpsqSlkyd7anGv1Dw/z60Hl+vu8sF/uHotyhiIiIyMymIBxjKYl+7l82h0/dWkh6coKnGu+19rClspb99Rc0OywiIiISJgXhKWJ+7iw2VgS5pTjD0/0HQyO8cayZF/Y00NEzGOXuRERERGYeBeEpJCnBz12LC3isrIis1ICnGo0dfWytqqX6TDsjI5odFhEREbkSBeEpqCgrlafKg9xWmo3PIl+IIzTi+MPJVrbtrqO5q38COhQRERGZ/hSEp6iA38eHF+byxJpi8tKTPNVovjjAtp31vFPTSmhYyzSLiIiIjKcgPMUVzE5m/ZoSbr8xF78v8tnhEefY+V47z+ys4+yFvgnoUERERGR6UhCeBvw+Y838bJ5aW8K8zGRPNdp7BtleXc+bx5sZDGl2WERERCSsIGxm95vZcTOrMbNvXub2EjN708zeNbMDZvZg9FuVnLQkHi8r5o6b8khMiPx3GOdgX90FtlTVUtvWMwEdioiIiEwf10xTZuYHvgc8ACwF1pvZ0kt2+ztgu3PuVuAJ4PvRblRGmRm3lmSxoTxIMCfVU42LfUP8ZG8jrx4+T//QcJQ7FBEREZkewplWXAPUOOdOO+cGgeeAhy/ZxwHvL4+WAZyNXotyORkpAT59ayH33VxAUsDbGS6Hz15kc+UZapq7otydiIiIyNQXToIqBOrHXW8Y2zbePwAbzKwBeBn4yuUKmdkXzazazKpbWlo8tCvjmRk3z8vgcxWl3Jif5qlGz8Awv9h/jl8eOEvPQCjKHYqIiIhMXdH6sNx64GnnXBHwILDFzD5Q2zn3I+dcmXOuLC8vL0oPLbOSEvjkLfN4aMVcZiX5PdU42dTN5spaDp/t1DLNIiIiEhfCCcKNQPG460Vj28b7M2A7gHOuEkgGcqPRoIRvYUE6mypKWTpv9rV3voz+oWFePdzEz/Y10tk3FOXuRERERKaWcILwbmChmc03s0RGPwy345J96oC7AcxsCaNBWOc+xEBywM/Hb57Dp28tZHaKt2Waz7T2srWqln31FzQ7LCIiIjPWNYOwcy4EfBl4BTjK6LdDHDazb5nZurHdvgb8ZzPbD2wDPu+UoGKqNHcWG8uDrCzJxMMqzQyGRnjzWDMvVDfQ3jMY/QZFREREYsxilVfLyspcdXV1TB473jRe6OP1I02eA22Cz1i7IIfVwSxPq9uJiIiIxJKZ7XHOlV26XSvLxYHCzBSeWlvC2vnZ+DxMD4dGHG/XtLJtVx3NF/snoEMRERGRyacgHCcS/D4+dGMu69cWkz87yVONlq4Btu2q562TrYSGtUyziIiITG8KwnEmPz2Z9beV8OGFuSR4OM1hxDl2n2lna1UtjRf6JqBDERERkcmhIByHfD7jttJsnioPUpiV4qlGR+8Q23fX8+axZgZCWqZZREREph8F4TiWPSuRx1YXcdfifBITvB0K++ovsKWyljOtPVHuTkRERGRiKQjHOTPjluJMNlYEKc1N9VSjqz/ET99t5NeHztM3qNlhERERmR4UhAWA2ckBPrWykI/fPIfkgLdlmo+eu8jmyjOcbOrSQhwiIiIy5SkIyx+ZGUvnzWZTRZBFBemeavQODvPLA+f45YFzdA+EotyhiIiISPQoCMsHzEpK4BMr5vLJW+YxK8nb7HBNczebK89wqLFTs8MiIiIyJSkIyxXdmJ/GpopSlhVmeLr/wNAIrx1p4id7G+nsHYpydyIiIiLXR0FYrio54OfepQV8ZlURs1MCnmrUtfeypeoM79Z1MDKi2WERERGZGhSEJSwlOalsLA9ya0kmHlZpZmjY8dvjLbywp5627oHoNygiIiISIQVhCVtigo87bsrn8bJictISPdU4e6GfZ3bWUXW6jWHNDouIiEgMKQhLxOZlpvDkmhLWLsjG52F6eHjEUXmqjWd31dF0sX8COhQRERG5NgVh8STB7+NDN+Ty5NoSCmYne6rR2jXAtl11/OFkC0PDI1HuUEREROTqFITluuSlJ/HEbcV8ZGEuCb7IZ4edg+ozHWytqqWho3cCOhQRERG5PAVhuW4+n1FWms2G8iCFWSmealzoHeKF6gbeONbEQEjLNIuIiMjEUxCWqMmalchjq4u4e0k+iQneDq399Z1sqazlvdaeKHcnIiIi8qcUhCWqzIwVRZlsrAgyP3eWpxpd/SF+9m4jvz50jr5BzQ6LiIjIxFAQlgkxOznAwyvn8cDyOaQkelum+ei5LjZXnuH4+S4t0ywiIiJRpyAsE8bMWDxnNpsqgtw0J91Tjd7BYV4+eI4d+8/SPRCKcociIiISzxSEZcKlJibw4PK5rFs5j7SkBE81Trf0sLnyDIcaOzU7LCIiIlGhICyT5oa8NDZWBFlemOHp/gNDI7x2pIkf722ks3coyt2JiIhIvFEQlkmVHPBzz9ICHl1dREZKwFON+vZetlSdYU9tByNapllEREQ8UhCWmCjOTmVjRZBVwSw8rNLM0LDj9yda2F5dT2v3QPQbFBERkRlPQVhiJuD38bFFeXz2tmJy0xI91TjX2c+zO+uoOt3GsGaHRUREJAIKwhJzczNSeHJtkPIFOfg9LNM8POKoPNXGsztrOd/ZPwEdioiIyEykICxTgt9nVNyQw5NrS5iTkeypRmv3IM/truP3J1oYGh6JcociIiIy0ygIy5SSm5bEZ8uK+eiiPAL+yGeHnYM9tR1sraqlvr13AjoUERGRmUJBWKYcn89YHcxiQ3mQ4uxUTzUu9A7x4p4GXj/SRP+QlmkWERGRDworCJvZ/WZ23MxqzOybV9jncTM7YmaHzezZ6LYp8SgzNZHPrCrkniUFJCZ4+53tYGMnW6tqOd3SHeXuREREZLq75jJfZuYHvgfcCzQAu81sh3PuyLh9FgL/HbjdOddhZvkT1bDEFzNjeVEGpbmpvHGsmdMtPRHX6OoP8fN9Z1k8J52P3ZRHaqK31e1ERERkZglnmm0NUOOcO+2cGwSeAx6+ZJ//DHzPOdcB4Jxrjm6bEu/SkwOsu2UeDy6fS2qi31ONY+e72FxZy7HzF7VMs4iIiIQVhAuB+nHXG8a2jbcIWGRmb5tZlZndf7lCZvZFM6s2s+qWlhZvHUvcMjNumpPOpopSlsxN91Sjb3CYXx08z479Z+nq1zLNIiIi8SxaH5ZLABYCdwDrgf/HzDIv3ck59yPnXJlzriwvLy9KDy3xJiXRz/3L5vLwynmkJ3s7zeF0Sw+bK2s50HBBs8MiIiJxKpwg3AgUj7teNLZtvAZgh3NuyDn3HnCC0WAsMmEW5KWxsSLIiqIMT/cfDI3wm6PNvLingQu9g1HuTkRERKa6cILwbmChmc03s0TgCWDHJfv8jNHZYMwsl9FTJU5Hr02Ry0tK8HP3kgIeXV1EZmrAU42Gjj62VNayp7adES3TLCIiEjeuGYSdcyHgy8ArwFFgu3PusJl9y8zWje32CtBmZkeAN4GvO+faJqppkUsVZ6eyoTxIWWkWFvk6HIRGHL8/0cpzu+tp6RqIfoMiIiIy5Viszo8sKytz1dXVMXlsmdmaLvbz6pEmWj0GWp8Zt83PYk1pNgl+rTkjIiIy3ZnZHudc2aXb9a+8zDgFs5N5ck0JH7ohB78v8unhEefYebqdZ3fVca6zbwI6FBERkalAQVhmJL/PWLsgh6fWljA3I9lTjbbuQZ7fXc/vTrQwGBqJcociIiISawrCMqPlpCXxeFkxH7spj4A/8tlh52BvbQdbq2qpa+udgA5FREQkVhSEZcbz+YxVJVlsLC+lJDvVU43OviF+vLeB14400T80HOUORUREJBYUhCVuZKQGeGRVIfcuLSAp4O3QP9TYyZbKWmqau6PcnYiIiEw2BWGJK2bGssIMNlWUckN+mqca3QMhfrH/LC8dOEfPQCjKHYqIiMhkURCWuJSWlMAnV8zlEyvmkpro91TjRFMXmytrOXruopZpFhERmYYUhCVumRmLCtLZVFHKkrmzPdXoHxrm14fO8/N9Z7nYPxTlDkVERGQiKQhL3EtJ9HP/sjl8+tZC0pMTPNV4r7WHLZW17K+/oNlhERGRaUJBWGRMae4sNlYEWVmc6en+g6ER3jjWzAt7GujoGYxucyIiIhJ1CsIi4yQl+LlzcT6PlRWRlRrwVKOxo4+tVbXsPtPOyIhmh0VERKYqBWGRyyjKSmVDeZDbSrPxWeQLcYRGHG+dbGXb7jqau/onoEMRERG5XgrCIleQ4Pfx4YW5rF9TTF56kqcazRcH2LaznndqWgkNa5lmERGRqURBWOQa8mcns35NCbffmEuCL/LZ4RHn2PleO8/srOPshb4J6FBERES8UBAWCYPfZ6yZn81T5UHmZSZ7qtHeM8j26nrePN7MYEizwyIiIrGmICwSgexZiTxeVsydi/NJTIj85eMc7Ku7wJaqWmrbeiagQxEREQmXgrBIhMyMlcWZbCgPEsxJ9VTjYt8QP9nbyCuHz9M/NBzlDkVERCQcCsIiHmWkBPj0rYXcd3MByQFvyzQfOXuRzZVnqGnuinJ3IiIici0KwiLXwcy4eV4GmyqCLCxI81SjZ2CYX+w/xy8PnKVnIBTlDkVERORKFIRFomBWUgIPrZjHQyvmMivJ2+zwyaZuNlfWcvhsp5ZpFhERmQQKwiJRtLAgnU0VpSydN9vT/fuHhnn1cBM/fbeRzr6hKHcnIiIi4ykIi0RZcsDPx2+ewyOrCpmd4m2Z5tq2XrZW1fJuXYdmh0VERCaIgrDIBAnmzGJjeZCVJZl4WKWZwdAIvz3ewgvVDbT3DEa/QRERkTinICwygRITfNx5Uz6PlRWTPSvRU43GC31srapl13vtDI9odlhERCRaFIRFJkFhZgpPrS1h7fxsfB6mh4dHHG/XtLJtVx3NF/snoEMREZH4oyAsMkkS/D4+dGMu69cWkz87yVONlq4Btu2q562TrYSGtUyziIjI9VAQFplk+enJrL+thA8vzCXBF/ns8Ihz7D7TztaqWhov9E1AhyIiIvFBQVgkBnw+47bSbJ4qD1KYmeKpRkfvENt31/PmsWYGQlqmWUREJFIKwiIxlD0rkcfKirhrcT6JCd5ejvvqL7ClspYzrT1R7k5ERGRmUxAWiTEz45biTDZWBCnNTfVUo6s/xE/fbeTXh87TN6jZYRERkXAoCItMEbOTA3xqZSEfv3kOyQFvyzQfPXeRzZVnONnUpYU4REREriGsIGxm95vZcTOrMbNvXmW/z5iZM7Oy6LUoEj/MjKXzZrOpIsiignRPNXoHh/nlgXP88sA5ugdCUe5QRERk5rhmEDYzP/A94AFgKbDezJZeZr904KvAzmg3KRJvZiUl8IkVc/nkLfOYleRtdrimuZvNlWc41Nip2WEREZHLCGdGeA1Q45w77ZwbBJ4DHr7Mfv8T+A6gb/sXiZIb89PYVFHKssIMT/cfGBrhtSNN/GRvI529Q1HuTkREZHoLJwgXAvXjrjeMbfsjM1sFFDvnXrpaITP7oplVm1l1S0tLxM2KxKPkgJ97lxbwmVVFzE4JeKpR197Llqoz7K3rYETLNIuIiABR+LCcmfmA7wJfu9a+zrkfOefKnHNleXl51/vQInGlJCeVjeVBbi3JxMMqzQwNO353vIUX9tTT1j0Q/QZFRESmmXCCcCNQPO560di296UDy4DfmtkZoBzYoQ/MiURfYoKPO27K5/GyYnLSEj3VOHuhn2d21rHzdBvDmh0WEZE4Fk4Q3g0sNLP5ZpYIPAHseP9G51yncy7XOVfqnCsFqoB1zrnqCelYRJiXmcKTa0pYuyAbn4fp4eERxzun2nh2Vx1NF3Vav4iIxKdrBmHnXAj4MvAKcBTY7pw7bGbfMrN1E92giFxegt/Hh27I5cm1JRTMTvZUo7VrgG276vjDyRaGhkei3KGIiMjUZrH6WqWysjJXXa1JY5FoGBlx7K3roPJUGyGPpztkpQa4e0kBxdneVrcTERGZqsxsj3PuA6ftamU5kRnA5zPKSrPZUB6kKCvFU42O3iFe3NPAb442MRDSMs0iIjLzKQiLzCBZsxJ5dHURdy/JJzHB28v7QEMnWyprea+1J8rdiYiITC0KwiIzjJmxoiiTTRVBFuTN8lSjqz/Ez95t5NeHztE3qNlhERGZmRSERWao9OQA626ZxwPL55CS6G2Z5qPnuthceYbj57u0TLOIiMw4CsIiM5iZsXjObDZVBFk8J91Tjd7BYV4+eI4d+8/SPRCKcociIiKxoyAsEgdSExN4YPlc1q2cR3pygqcap1t62Fx5hkONnZodFhGRGUFBWCSO3JCXxobyIMsLMzzdf2BohNeONPHjvY1c6B2McnciIiKTS0FYJM4kB/zcs7SAR1cXkZka8FSjvr2XrVW17KntYETLNIuIyDSlICwSp4qzU9lQHmR1MAsPqzQzNOz4/YkWnq+up7V7IPoNioiITDAFYZE4FvD7+OiiPJ64rYTctERPNc539vPszjoqT7UxrNlhERGZRhSERYQ5Gck8uTZI+YIc/L7Ip4eHRxxVp9t4dmct5zv7J6BDERGR6FMQFhEA/D6j4oYcnlxbwpyMZE81WrsHeW53Hb8/0cLQ8EiUOxQREYkuBWER+RO5aUl8tqyYjy7KI+CPfHbYOdhT28GWylrq23snoEMREZHoUBAWkQ/w+YzVwSw2lAcpzk71VKOzb4gX9zTw+pEm+oe0TLOIiEw9CsIickWZqYl8ZlUh9y4tIDHB29vFwcZOtlTWcqqlO8rdiYiIXB8FYRG5KjNjWWEGmyqCLMib5alG90CIHfvO8vLBc/QOaplmERGZGhSERSQs6ckB1t0yjweXzyU10e+pxvHzXWyurOXY+YtapllERGJOQVhEwmZm3DQnnU0VpSyZm+6pRt/gML86eJ4d+89ysX8oyh2KiIiET0FYRCKWkujn/mVzeXjlPNKTEzzVON3Sw5bKWg40XNDssIiIxISCsIh4tiAvjY0VQW4pzvB0/8HQCL852syLexro6BmMcnciIiJXpyAsItclKcHPXYsLeHR1EZmpAU81Gjr62FpVS/WZdka0TLOIiEwSBWERiYri7FQ2lAcpK83CIl+Hg9CI4w8nW3ludz0tXQPRb1BEROQSCsIiEjUBv4+PLMxj/ZoSctOTPNVoutjPszvreOdUKyEt0ywiIhNIQVhEoq5gdjJPrinhQzfk4PdFPj084hw7T7fz7K46zl7om4AORUREFIRFZIL4fcbaBTk8tbaEeZnJnmq0dQ+yvbqe3x5vZjCk2WEREYkuBWERmVA5aUk8trqYO27KI+CPfHbYOXi37gJbqmqpa+udgA5FRCReKQiLyITz+YxbS7LYWF5KSXaqpxoX+4b48d4GXj18nv6h4Sh3KCIi8UhBWEQmTUZqgEdWFXLv0gKSAt7efg6fvciWylpqmruj3J2IiMQbBWERmVRmxrLCDDZVlHJjfpqnGt0DIX6x/ywvHThHz0Aoyh2KiEi8UBAWkZhIS0rgoRVzeWjFXFIT/Z5qnGjqYnNlLUfOXtQyzSIiErGwgrCZ3W9mx82sxsy+eZnb/5uZHTGzA2b2GzMLRr9VEZlpzIyFBelsqihlydzZnmr0Dw3zyuHz/HzfWS72D0W5QxERmcmuGYTNzA98D3gAWAqsN7Oll+z2LlDmnFsBvAj839FuVERmrpREP/cvm8Onby0kPTnBU433WnvYUlnL/voLmh0WEZGwhDMjvAaocc6dds4NAs8BD4/fwTn3pnPu/e81qgKKotumiMSD0txZbKwIsrI409P9B0MjvHGsmRf2NNDRMxjd5kREZMYJJwgXAvXjrjeMbbuSPwN+dbkbzOyLZlZtZtUtLS3hdykicSMpwc+di/N5rKyIrNSApxqNHX1srapl95l2RkY0OywiIpcX1Q/LmdkGoAz458vd7pz7kXOuzDlXlpeXF82HFpEZpigrlQ3lQW4rzcZnkS/EERpxvHWylW2762ju6p+ADkVEZLoLJwg3AsXjrheNbfsTZnYP8LfAOufcQHTaE5F4luD38eGFuaxfU0xeepKnGs0XB9i2s563a1oJDWuZZhER+f+FE4R3AwvNbL6ZJQJPADvG72BmtwI/ZDQEN0e/TRGJZ/mzk1m/poTbb8wlwRf57PCIc+x6r51ndtZx9kLfBHQoIiLT0TWDsHMuBHwZeAU4Cmx3zh02s2+Z2bqx3f4ZSANeMLN9ZrbjCuVERDzx+4w187N5qjzIvMxkTzXaewbZXl3Pm8ebGQxpdlhEJN5ZrL5mqKyszFVXV8fksUVkenPOsb+hk7drWj0H2tkpAe5Zkk8wZ1aUuxMRkanGzPY458ou3a6V5URk2jEzVhZnsqE8SGluqqcaF/uG+MneRl45fJ7+oeEodygiItOBgrCITFsZKQE+tbKQ+24uIDngbZnmI2cvsrnyDCebuqLcnYiITHUKwiIyrZkZN8/LYFNFkIUFaZ5q9AwM88sD5/jlgbP0DISi3KGIiExVCsIiMiPMSkrgoRXz+OQtc5mV5G12+GRTN5srazl8tlPLNIuIxAEFYRGZUW7MT2dTRSk3z5vt6f79Q8O8eriJn77bSGffUJS7ExGRqURBWERmnOSAn/tunsMjqwqZneJtmebatl62VtXybl2HZodFRGYoBWERmbGCObPYWB5kZUkmHlZpZjA0wm+Pt7C9up62bi2YKSIy0ygIi8iMlpjg486b8nm8rJjsWYmeapy90M8zO+vY9V47wyOaHRYRmSkUhEUkLszLTOGptSWsnZ+Nz8P08PCI4+2aVrbtqqP5Yv8EdCgiIpNNQVhE4kaC38eHbsxl/dpi8mcnearR0jXAtl31vHWylaFhLdMsIjKdKQiLSNzJT09m/W0lfHhhLgm+yGeHR5xj95l2nqmqpaGjdwI6FBGRyaAgLCJxyeczbivNZkN5kMKsFE81OnqHeKG6gTeONTEQ0jLNIiLTjYKwiMS1rFmJPLa6iLsW55OY4O0tcX99J1sqa3mvtSfK3YmIyERSEBaRuGdm3FKcycaKIPNzZ3mq0dUf4mfvNvLrQ+fpG9TssIjIdKAgLCIyZnZygIdXzuP+ZXNIDnhbpvnouYtsrjzDiaYuLcQhIjLFKQiLiIxjZiyZO5vPfSjIooJ0TzV6B4d56cA5fnHgHN0DoSh3KCIi0aIgLCJyGamJCXxixVw+ecs80pISPNU41dzN5sozHGrs1OywiMgUpCAsInIVN+ansbEiyLLCDE/3Hxga4bUjTfxkbyOdvUNR7k5ERK6HgrCIyDUkB/zcu7SAz6wqIiMl4KlGXXsvW6rOsLeugxEt0ywiMiUoCIuIhKkkJ5UN5UFWBbPwsEozQ8OO3x1vYXt1PW3dA9FvUEREIqIgLCISgcQEHx9blMdnbysmJy3RU41znf08s7OOqtNtDGt2WEQkZhSERUQ8mJuRwpNrSihfkIPPw/Tw8Iij8lQbz+6q43xn/wR0KCIi16IgLCLiUYLfR8UNOTy5toSC2cmearR2DfDc7jr+cLKFoeGRKHcoIiJXoyAsInKd8tKTeOK2Yj66KJcEX+Szw85B9ZkOtlbVUt/eOwEdiojI5SgIi4hEgc9nrA5ms7EiSFFWiqcaF3qHeHFPA7852sRASMs0i4hMNAVhEZEoykxN5NHVRdyzpIDEBG9vsQcaOtlSWUtNc5e+ak1EZAJ5Wy5JRESuyMxYXpRBaW4qbxxr5nRLT8Q1uvpD/GL/OVIS/dyQl8bC/DSKs1Pxezj1QkRELk9BWERkgqQnB1h3yzyON3Xx2+Mt9A1GfrpD3+Awhxo7OdTYSVLAx4LcNG7MTyOYk0rArz/qiYhcDwVhEZEJZGYsnjObkuxUfne8hWPnuzzXGhga4ei5ixw9d5HEBB/zc2dxY34apTmzPJ+GISISzxSERUQmQWpiAg8sn8uiOem8eayZrv7QddUbDI1w/HwXx893keAzgrmzSEvyR6lbEZGJl5mayKqSrJj2oCAsIjKJbshLozAzhbdrWjnQ0BmVmqERx6nm7qjUEhGZLMXZqTEPwmH9Lc3M7jez42ZWY2bfvMztSWb2/NjtO82sNOqdiojMEMkBP3cvKeDR1UVkpgZi3Y6ISNy6ZhA2Mz/wPeABYCmw3syWXrLbnwEdzrkbgf8FfCfajYqIzDTF2alsKA+yOpiFh1WaRUTkOoUzI7wGqHHOnXbODQLPAQ9fss/DwH+MXX4RuNtMb+siItcS8Pv46KI8nrjN+zLNIiLiTTjnCBcC9eOuNwBrr7SPcy5kZp1ADtA6ficz+yLwRYCSkhKPLYuIzDxzMpJZv6aYs539nGzqoqa5+7o/UCciIlc3qR+Wc879CPgRQFlZmZZLEhEZx8wozEyhMDOFjy3Ko+niACebR0Pxhd6hWLcnIjLjhBOEG4HicdeLxrZdbp8GM0sAMoC2qHQoIhKHzIw5GcnMyUjmwzfm0tI9QE1zNzXN3bR1D8a6PRGRGSGcILwbWGhm8xkNvE8AT16yzw7gc0Al8CjwhnNOM74iIlFgZuSnJ5OfnsyHbsilbSwUn2zupqVrINbtiYhMW9cMwmPn/H4ZeAXwA//unDtsZt8Cqp1zO4D/A2wxsxqgndGwLCIiEyAnLYmctCTWLsihs3eI5q5+ejws3ywiEktpSbFfzsJiNXFbVlbmqqurY/LYIiIiIhI/zGyPc67s0u1anF5ERERE4pKCsIiIiIjEJQVhEREREYlLCsIiIiIiEpdi9mE5M2sBamPy4JDLJaveyVVpvCKj8YqMxisyGq/IaLwio/GKjMYrMrEcr6BzLu/SjTELwrFkZtWX++SgXJ7GKzIar8hovCKj8YqMxisyGq/IaLwiMxXHS6dGiIiIiEhcUhAWERERkbgUr0H4R7FuYJrReEVG4xUZjVdkNF6R0XhFRuMVGY1XZKbceMXlOcIiIiIiIvE6IywiIiIicU5BWERERETi0owKwmZ2v5kdN7MaM/vmZW5PMrPnx27faWal427772Pbj5vZxye18RgJY7z+m5kdMbMDZvYbMwuOu23YzPaN/eyY3M5jI4zx+ryZtYwbl/807rbPmdnJsZ/PTW7nsRHGeP2vcWN1wswujLstHo+vfzezZjM7dIXbzcz+99h4HjCzVeNui8fj61rj9dTYOB00s3fM7JZxt50Z277PzKonr+vYCWO87jCzznGvu78fd9tVX8szURjj9fVxY3Vo7D0re+y2eDy+is3szbHMcNjMvnqZfabme5hzbkb8AH7gFLAASAT2A0sv2ecvgR+MXX4CeH7s8tKx/ZOA+WN1/LF+TlNgvO4EUscuf+n98Rq73h3r5zAFx+vzwL9e5r7ZwOmx/2aNXc6K9XOK9Xhdsv9XgH8fdz2ujq+x5/xRYBVw6Aq3Pwj8CjCgHNg5tj3ujq8wx+tD748D8MD74zV2/QyQG+vnMMXG6w7gl5fZHtFreab8XGu8Ltn3k8Ab467H4/E1F1g1djkdOHGZfyOn5HvYTJoRXgPUOOdOO+cGgeeAhy/Z52HgP8YuvwjcbWY2tv0559yAc+49oGas3kx2zfFyzr3pnOsdu1oFFE1yj1NJOMfXlXwceM051+6c6wBeA+6foD6nikjHaz2wbVI6m6Kcc78H2q+yy8PAZjeqCsg0s7nE5/F1zfFyzr0zNh6g969wjq8ruZ73vmkrwvHS+5dz55xze8cudwFHgcJLdpuS72EzKQgXAvXjrjfwwf8Jf9zHORcCOoGcMO8700T6nP+M0d/k3pdsZtVmVmVmn5qA/qaacMfrM2N/8nnRzIojvO9MEvZzHjvlZj7wxrjN8XZ8heNKYxqPx1ekLn3/csCrZrbHzL4Yo56mogoz229mvzKzm8e26fi6CjNLZTS0/Xjc5rg+vmz0tNNbgZ2X3DQl38MSJuuBZPoysw1AGfCxcZuDzrlGM1sAvGFmB51zp2LT4ZTxC2Cbc27AzP6c0b8+3BXjnqaDJ4AXnXPD47bp+JKoMLM7GQ3CHx63+cNjx1c+8JqZHRubAYxnexl93XWb2YPAz4CFsW1pWvgk8LZzbvzscdweX2aWxugvBX/tnLsY637CMZNmhBuB4nHXi8a2XXYfM0sAMoC2MO8704T1nM3sHuBvgXXOuYH3tzvnGsf+exr4LaO//c1k1xwv51zbuDH6f4HV4d53BorkOT/BJX9WjMPjKxxXGtN4PL7CYmYrGH0tPuyca3t/+7jjqxn4KTP/VLhrcs5ddM51j11+GQiYWS46vq7lau9fcXV8mVmA0RD8jHPuJ5fZZUq+h82kILwbWGhm880skdGD89JPm+8A3v804qOMntzuxrY/YaPfKjGf0d+Cd01S37FyzfEys1uBHzIagpvHbc8ys6Sxy7nA7cCRSes8NsIZr7njrq5j9BwpgFeA+8bGLQu4b2zbTBbO6xEzW8zohyMqx22Lx+MrHDuATWOfvC4HOp1z54jP4+uazKwE+Amw0Tl3Ytz2WWaW/v5lRsfrst8MEE/MbM7YZ2YwszWM5oM2wnwtxyMzy2D0L6U/H7ctLo+vsWPn/wBHnXPfvcJuU/I9bMacGuGcC5nZlxkdPD+jn0A/bGbfAqqdczsY/Z+0xcxqGD0J/omx+x42s+2M/mMbAv7LJX+mnXHCHK9/BtKAF8beH+ucc+uAJcAPzWyE0TfL/8s5N6ODSpjj9Vdmto7RY6id0W+RwDnXbmb/k9F/UAC+dcmf0WacMMcLRl+Dz439Qvq+uDu+AMxsG6Of3M81swbgfwABAOfcD4CXGf3UdQ3QC3xh7La4O74grPH6e0Y/A/L9sfevkHOuDCgAfjq2LQF41jn360l/ApMsjPF6FPiSmYWAPuCJsdflZV/LMXgKkyqM8QL4NPCqc65n3F3j8vhidMJiI3DQzPaNbfsboASm9nuYllgWERERkbg0k06NEBEREREJm4KwiIiIiMQlBWERERERiUsKwiIiIiISlxSERURERCQuKQiLiIiISFxSEBYRERGRuPT/AZHpR3uNEZAdAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(fs, hinge_loss(fs), linewidth=9, alpha=0.5)\n", "plt.legend(['Hinge Loss'])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### 12.3.3.1. Properties of the Hinge Loss\n", "\n", "The hinge loss is one of the best losses in machine learning. We summarize here several important properties of the hinge loss.\n", "\n", "* It penalizes errors \"that matter,” hence is less sensitive to outliers.\n", "\n", "* Minimizing a regularized hinge loss optimizes for a high margin.\n", "\n", "* The loss is non-differentiable at point, which may make it more challenging to optimize." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 12.4. Optimization for SVMs\n", "\n", "We have seen a new way to formulate the SVM objective. Let's now see how to optimize it." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.4.0 Review\n", "\n", "### 12.4.0.1. Review: SVM Objective\n", "\n", "Maximizing the margin can be done in the following form:\n", "\n", "$$ \n", "\\min_{\\theta,\\theta_0, \\xi}\\; \\sum_{i=1}^n \\underbrace{\\left(1 - y^{(i)}\\left((x^{(i)})^\\top\\theta+\\theta_0\\right)\\right)^+}_\\text{hinge loss} + \\underbrace{\\frac{\\lambda}{2}||\\theta||^2}_\\text{regularizer} \n", "$$\n", "\n", "* The hinge loss penalizes incorrect predictions.\n", "* The L2 regularizer ensures the weights are small and well-behaved." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We can easily implement this objective in `numpy`.\n", "First we define the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "def f(X, theta):\n", " \"\"\"The linear model we are trying to fit.\n", " \n", " Parameters:\n", " theta (np.array): d-dimensional vector of parameters\n", " X (np.array): (n,d)-dimensional data matrix\n", " \n", " Returns:\n", " y_pred (np.array): n-dimensional vector of predicted targets\n", " \"\"\"\n", " return X.dot(theta)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "And then we define the objective." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "def svm_objective(theta, X, y, C=.1):\n", " \"\"\"The cost function, J, describing the goodness of fit.\n", " \n", " Parameters:\n", " theta (np.array): d-dimensional vector of parameters\n", " X (np.array): (n,d)-dimensional design matrix\n", " y (np.array): n-dimensional vector of targets\n", " \"\"\"\n", " return (np.maximum(1 - y * f(X, theta), 0) + C * 0.5 * np.linalg.norm(theta[:-1])**2).mean()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### 12.4.0.2. Review: Gradient Descent\n", "If we want to optimize $J(\\theta)$, we start with an initial guess $\\theta_0$ for the parameters and repeat the following update:\n", "\n", "$$ \n", "\\theta_i := \\theta_{i-1} - \\alpha \\cdot \\nabla_\\theta J(\\theta_{i-1}). \n", "$$\n", "\n", "As code, this method may look as follows:\n", "```python\n", "theta, theta_prev = random_initialization()\n", "while norm(theta - theta_prev) > convergence_threshold:\n", " theta_prev = theta\n", " theta = theta_prev - step_size * gradient(theta_prev)\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.4.1. A Gradient for the Hinge Loss?\n", "\n", "What is the gradient for the hinge loss with a linear $f$?\n", "\n", "$$ \n", "J(\\theta) = \\max\\left(1 - y \\cdot f_\\theta(x), 0\\right) = \\max\\left(1 - y \\cdot \\theta^\\top x, 0\\right). \n", "$$\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Here, you see the linear part of $J$ that behaves like $1 - y \\cdot f_\\theta(x)$ (when $y \\cdot f_\\theta(x) < 1$) in orange:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAD4CAYAAADmbIA7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3rUlEQVR4nO3deXhV9b33/fd378wEwpAwz2MIMwkgIDigiKggwfZIa496rNpBaxV57H1Or96tbc/1nAfU3latQ1vt3fY4AoqKs1QRirIDCfMQ5oDIEKYkZP49fyRyIgTYe7OSHbI/r+viMuu31v7uz/65svPNytprmXMOEREREZFo44t0ABERERGRSFAjLCIiIiJRSY2wiIiIiEQlNcIiIiIiEpXUCIuIiIhIVIqJ1BOnpqa6nj17RurpRURERCRK5OTkHHLOpZ0+HrFGuGfPngQCgUg9vYiIiIhECTPbVd+4To0QERERkaikRlhEREREolLUNMKFB/ay8vUnKDywN9JRRERERKQJiNg5wo2puqqK/KWvUnW0gM1vP05K+uUMGHU1/pioePkiIiKNqqKigoKCAkpLSyMdRaJMQkICXbt2JTY2Nqjto6IT3Lr6E6qOFtQsVFdxbMNHBPaso8+l3ya1c4/IhhMREWlmCgoKaNmyJT179sTMIh1HooRzjsOHD1NQUECvXr2Cesx5T40wsz+b2QEzW3eW9WZmj5tZvpmtMbORIeZuUMeOHOLIug/PGK8+8RVb332StZ++QUV5WQSSiYiINE+lpaW0a9dOTbA0KjOjXbt2If0lIphzhF8Appxj/bVAv9p/dwF/CPrZG5irrmbLp6/gqsrPtgFFW5eSM/8Rvty1uXHDiYiINGNqgiUSQt3vztsIO+c+BQrPscl04P+6GiuA1mbWKaQUDWT7uhVUHtp+3u1cSSE7P3yOvI9foqy0pBGSiYiIiEikeXGOcBdgT53lgtqxL0/f0MzuouaoMd27d/fgqc+u+MRRDqx+O6THlOwIsPrLLXQZM4NufYc0UDIREZHm7W+5haQWbGnU57z/6v7fWE5OTqaoqOjU8gsvvEAgEOCJJ57g6aefJikpiX/9139tsDy33XYb119/PTfddFODPYdcuEb9sJxz7lngWYCsrCzXkM+1e8PnUBn6ub+u9DgFn/yFQ9uGkH5pNoktWjZAOhEREYmUH/zgB5GOIE2EF9cR3gt0q7PctXYsotJHXU2HMd/CYhPDenxpwVryFsxl58YArrra43QiIiISKb/85S+ZN28eAJdffjkPPfQQo0ePpn///ixduhSAkpISvv3tb5ORkcGMGTMYM2YMgUAAgPfff5+xY8cycuRIvvWtb33jyPO5lJaWcvvttzNkyBBGjBjBkiVLAFi/fj2jR49m+PDhDB06lK1bt1JcXMx1113HsGHDGDx4MC+//HIDzIR4cUR4EXCPmb0EjAGOOefOOC2isZnPR+/BY+jYM51NSxdQtm99yDVceQlfLn+JQ9tXkz7hJpJbtWmApCIiIuK1kydPMnz48FPLhYWFTJs2rd5tKysr+eKLL1i8eDG/+tWv+PDDD3nqqado06YNGzZsYN26dadqHTp0iN/85jd8+OGHtGjRgv/6r//i0Ucf5Re/+MV5Mz355JOYGWvXrmXTpk1MnjyZLVu28PTTT3Pffffx3e9+l/Lycqqqqli8eDGdO3fm7bdrTvM8duzYBc+JnOm8jbCZvQhcDqSaWQHwv4FYAOfc08BiYCqQD5QAtzdU2HAkJacw8trb2b0ll31fvI4rC+63troq9m9m7cJ5pA2/lj5DxmG+qLkhn4iIyEUpMTGR3NzcU8tfnyNcn+zsbAAyMzPZuXMnAJ999hn33XcfAIMHD2bo0KEArFixgg0bNjB+/HgAysvLGTt2bFCZPvvsM+69914A0tPT6dGjB1u2bGHs2LH89re/paCggOzsbPr168eQIUOYPXs2Dz30ENdffz0TJkwIeQ7k/M7bCDvnZp1nvQN+7FmiBtK9/3Dad+vHpmWLOLkrJ/QClWUcDLzOkR259J/4L6S0TfM+pIiIiDS6+Ph4APx+P5WVlefc1jnH1VdfzYsvvujZ83/nO99hzJgxvP3220ydOpVnnnmGK6+8klWrVrF48WJ+/vOfM2nSpKCOOktoourQZkJiC4ZfNYseV34fS2wdVo3KwzvZsOgRNgU+orqqytuAIiIi0iSMHz+eV155BYANGzawdu1aAC655BKWLVtGfn4+AMXFxWzZEtwVMiZMmMDf//53ALZs2cLu3bsZMGAA27dvp3fv3vzkJz9h+vTprFmzhn379pGUlMQtt9zCnDlzWLVqVQO8SomKWyyfrnOvdNK6zGHj8rco3r4CXIgXsKiq5EjeOwR2raHPpd+iXYeuDRNUREREIuJHP/oRt956KxkZGaSnpzNo0CBSUlJIS0vjhRdeYNasWZSV1Vyd6je/+Q39+/c/o8bdd9/NT3/6UwC6devGkiVL+OEPf8iQIUOIiYnhhRdeID4+nldeeYW//vWvxMbG0rFjR/793/+dlStXMmfOHHw+H7GxsfzhD03mfmXNirlQm0CPZGVlubOdq9OYvirYxs5lr1FddDC8Aj4/rQZcRvroyfhjovL3ChERkW/4jxeXkZrauKcQnn4d4QtVVVVFRUUFCQkJbNu2jauuuorNmzcTFxfn6fOI9zZu3MjAgQO/MWZmOc65rNO3jfrOrUPXPqTOnM2mz9/j+JZPoTrE0x2qqzi+8WMCBevoNf5btO/Sq2GCioiIXCRuGd6WgQO9bUwbW0lJCVdccQUVFRU453jqqafUBDdDUd8IA/hjYhg0/joO9RnK9s9eperYvpBrVJ84wLb3nuKrPmNJH3sdsXHxDZBUREREGkPLli3PepUJaT6i6sNy55PasRtZM+6jzdAp4A/jdwTnKMpfTs78eezbudn7gCIiIiLiGTXCp/H5/aSPuoqMG+4npl3PsGq4kiPs+ug5cj96idKTxd4GFBERERFPqBE+i5R2Hcia9iNSM6dDTHjnBJ3cGSB3wTz2bM3zOJ2IiIiIXCg1wudgPh/9hk9g8I0PEtshvJP+XekJCj79K6vefYGSIt0eUURERKSpUCMchJYpbcmc+n06jv0XLDYxrBple9exZsE8dqz/HFdd7XFCEREREQmVGuEgmc9Hr4xRDJv5/xDfZXBYNVzFSfaveJWct5/jxLFCjxOKiIiISCh0+bQQJbZoycgpt7Fnax57v3gdV3oi5BoVB7ay7vV5pA67lr5Dx2M+/T4iIiLNR+q652B/495Qgyv+1xlDycnJFBUVnVp+4YUXCAQCPPHEE4wbN47ly5c3aKTTn7+x7dy5k+uvv55169ZFLMPpZs+ezYcffsjEiRP5/e9/H9Jj/+3f/o233nqL9u3be/aa1IGFqVu/YYyYOYfEnmfcpCQ4leUcynmDlW88ybHDX3kbTkRERM6poZtgOdO2bdtYtmwZeXl5522Cy8vLKS7+5pW3brvtNt59911PM6kRvgDxCUkMn3QzPSbdiSW1CatGVeEuNrz5GJu++IDqqhDvaiciIiJhSU5OBmqOmg4cOJA777yTQYMGMXnyZE6ePAnAr3/9awYMGMCll17KrFmzmDdvHgB/+9vfGD16NMOHD+fuu++mKoSf348++iiDBw9m8ODB/O53vzs1XlxczHXXXcewYcMYPHgwL7/8cr1jdc2dO5fHH38cgPvvv58rr7wSgI8//pjvfve7QM2tout7bWd7Heeaj6+tW7eOcePGnVpetWoVkyZNOufr3rx5M5dffjm7du1ixIgRZzS5X9u4cSOzZ89mwIABbNmy5RvrJk6cSNu2bc/5PKFSI+yBzj0HkDnzQZL7jQez0AtUVXJk7XsEFv4fDu3f431AERGRKHTy5EmGDx9+6t8vfvGLerfbunUrP/7xj1m/fj2tW7dm/vz5rFy5kvnz55OXl8c777xz6i5zGzdu5OWXX2bZsmXk5ubi9/v5+9//HlSenJwcnn/+eT7//HNWrFjBc889x+rVqwF499136dy5M3l5eaxbt44pU6bUO1bXhAkTWLp0KQCBQICioiIqKipYunQpEydOPOtrO9/rONtjvpaRkcH27dtP/QLwwAMPMHfu3G9sU1JSgnPu1PKAAQO49dZb+fWvf83q1atp0aLFqXXFxcU8//zzXHrppdx5551kZGSwZs0aRowYEdS8XgidI+yR2Lh4hkycwcG+I9j+2StUnzgQco2qY/vYuvj37B8wgfTR1xATq3uai4iIhCsxMZHc3NxTy1+fI3y6Xr16MXz4cAAyMzPZuXMnhw4dYvr06SQkJJCQkMANN9wAwEcffUROTg6jRo0Caprt9u3bB5Xns88+Y8aMGaeawOzsbJYuXcqIESMYMmQIs2fP5qGHHuL6669nwoQJ9Y7VlZmZSU5ODsePHyc+Pp6RI0cSCARYunTpqSPF9b22c72OiRMnnvUxX/P5fAwaNIj169ezdetWevTowciRI7+xTe/evVmxYgU9e/Y8NbZ27VqmT59+xrx06tSJoUOH8sc//pH09PSg5tIrOiLssbTOPcnKfoBWA68Enz/0Aq6aE5s+IWfBY3xVsM37gCIiIvIN8fHxp772+/1UVlaedVvnHLfeeiu5ubnk5uayefNmfvnLX15whv79+7Nq1SqGDBnCz3/+cx5++OF6x+qKjY2lV69evPDCC4wbN44JEyawZMkS8vPzGThw4Dlf27leRzDzcckll7Bs2TJ++ctf8p//+Z9nrF+9ejXdunX7xtj69esZPPjMK2+99tprdOnShezsbB5++GF27doV5KxdODXCDcAfE8OgcVPpP/Ve/K27hFWjuugg299/mjX/mE95WanHCUVERORcxo8fz5tvvklpaSlFRUW89dZbAEyaNInXXnuNAwdq/vJbWFgYdOM2YcIEXn/9dUpKSiguLmbhwoWnjvLu27ePpKQkbrnlFubMmcOqVavqHauv5rx585g4cSITJkzg6aefZsSIEdh5TtW8kNcBNY3wz3/+c2bMmEGXLmf2Op06dcLv/58DgidOnCA2NpbExDPvxzB58mRefvllli5dSkpKCtOnT+eqq64640h0Q9CpEQ2oXYeutLnxJ2xZ/Q+OrPsAqs7+G2a9nKN42z9ZtW8j3cfeROdejfvnAhERkWg1atQopk2bxtChQ+nQoQNDhgwhJSWFjIwMfvOb3zB58mSqq6uJjY3lySefpEePHmfUKCkpoWvXrqeWH3jgAW677TZGjx4NwPe///1T58GuXbuWOXPm4PP5iI2N5Q9/+EO9Y6ebMGECv/3tbxk7diwtWrQgISHhjFMo6nO219GxY8eg5ic9PZ34+HgeeuihoLZft25dvUeD62rXrh333Xcf9913H1988cU3GmmAWbNm8Y9//INDhw7RtWtXfvWrX3HHHXcE9fxnY3VPZG5MWVlZrr7zdJqrY4UH2fLpK1Qe3hF2jcTuI0m/dDoJiS3Ov7GIiEiEHHz1AdJSI38d4QtVVFREcnIyJSUlTJw4kWefffaMc2Gj1T333MOoUaO49dZbIx3lDBs3bjx1asjXzCzHOXfGNW91RLiRpLRNI2vaD9m29p8czF0MlWUh1zi5exW587fQadSN9Bgw3PuQIiIiHjg0+E7STmtELkZ33XUXGzZsoLS0lFtvvVVNMDXXAr7uuusYP358k2yCQ6UjwhFQdPwIm5a+RsX+zWHXiO88iPQJ2SQlp3iYTERE5MLVd0ROpLGEckRYH5aLgORWbci89g46jbsZi0sKq0bZvvXkLZjH9nWf46qrPU4oIiIi0vypEY4Q8/noOTCLYdlziO86LLwiFSf56vNXyXnrWY4fPextQBEREZFmTo1whCW2aMnIa75H18tuxRJahVWj4mA+6994hC2rPtHRYRERaRIideqlRLdQ9zs1wk1Et75DGDHzQZJ6nXH6SnAqyzm8+k1WvvEERw/t9zaciIhICBISEjh8+LCaYWlUzjkOHz5MQkJC0I/Rh+WaoC93bWbX8vm4ksLwCvj8tMmYRP+sSfj8YdzdTkRE5AJUVFRQUFBAaaluCCWNKyEhga5duxIbG/uN8bN9WE6NcBNVUV7GphXvUpS/DFx4pzv4WnWkz4Rvk9qxu8fpRERERC4eumrERSY2Lp4hE6fTd8qP8LXsEFaN6uP72br4CdZ9tojKinKPE4qIiIhc3NQIN3FpnXuSlX0/KYOuAl8Ypzm4ak5s/pSc+Y+wf0++9wFFRERELlJqhC8C/pgYMi6ZwoDrfoK/ddfzP6Ae1cWH2fHBM+QteZXyMp2zJSIiIhJUI2xmU8xss5nlm9nP6lnf3cyWmNlqM1tjZlO9jypt23dh1Iyf0HbYVMwfF3oB5yjZ/jmr5s9l7/b13gcUERERuYictxE2Mz/wJHAtkAHMMrOM0zb7OfCKc24EcDPwlNdBpYb5fAzIupKB0x8gJrV3WDXcyWPsXvI8q9//G6UlRR4nFBEREbk4BHNEeDSQ75zb7pwrB14Cpp+2jQO+vhtECrDPu4hSn5Q2qWTd8APaj8qGmPiwapTuySV3wVx2bVrlcToRERGRpi+YRrgLsKfOckHtWF2/BG4xswJgMXBvfYXM7C4zC5hZ4ODBg2HElbrM56PP0HEMzZ5DXMf0sGq4smL2Lftvchb/ieITR70NKCIiItKEefVhuVnAC865rsBU4K9mdkZt59yzzrks51xWWlqaR08tLVq2JvO679N53CwsvkVYNcq/3MiahY+wbc1y3aZZREREokIwjfBeoFud5a61Y3XdAbwC4Jz7J5AApHoRUILXY2Amw7PnkNBteHgFKk5yYOUCct56hmNHDnmaTURERKSpCaYRXgn0M7NeZhZHzYfhFp22zW5gEoCZDaSmEda5DxGQkJTMiMm30O2y27DElLBqVBzcxsY3HmXLqn/o6LCIiIg0W+dthJ1zlcA9wHvARmquDrHezB42s2m1m80G7jSzPOBF4DYXqXs3CwBd+w5m5Mw5JPUeA2YhP95VlXN49VsEXv89Rw5+2QAJRURERCLLItWvZmVluUAgEJHnjjb7d29l1/LXqC4+HF4Bn5+UgVcwIOsq/DEx3oYTERERaWBmluOcyzp9XHeWiwIdu/cjc+ZsWg6YCGd+hvH8qqs4tv5DAgse49C+Xd4HFBEREYkANcJRIiY2jsGXTqPf1HvwteoYVo3qE1+x9d0nWfvpG1RWlHucUERERKRxqRGOMqkduzMq+35aD74afP7QC7hqirYuJfDaPPbv3up9QBEREZFGokY4Cvn8fgaOuYb063+Kv233sGq4kkJ2fPAMeR+/QllpiccJRURERBqeGuEo1iatE6Om30O7ETdg/riwapTs+ILV8+dRkL/O43QiIiIiDUuNcJQzn4/+Iy8j48bZxKb1CauGKz3Onk9eYNV7f+Vk8QmPE4qIiIg0DDXCAkCr1u3IvP5u2o++CWITw6pRVpBH3oK57NqYoxtxiIiISJOnRlhOMZ+PPkMuYeiM2cR1GhRWDVdewr7lL7LqnT9TdPyIxwlFREREvKNGWM7QomVrMqfeTudLb8HiW4RVo3z/JtYunEd+3jIdHRYREZEmSY2wnFWPAcMZnj2HxO4jwytQWcbBwEICi/7AscKD3oYTERERuUBqhOWcEpKSGX71d+h+xR1YYkpYNSoP72DDokfYHPiY6qoqjxOKiIiIhEeNsASlS++BjJw5hxa9LwGz0AtUVVKYt5jA67+n8MBe7wOKiIiIhEiNsAQtLj6BoVfcRK+r78aXnBpWjaqjBWx++3HWL19MVWWlxwlFREREgqdGWELWsVtfMrMfoGX6ZWBh7ELVVRzf+DGBBY9ycN9Oz/OJiIiIBEONsIQlJjaOweNvoN/Ue/G16hRWjeoTB8h/90nWfrqQivIyjxOKiIiInJsaYbkgqR27MSr7p7QZPBl8/tALOEfR1mXkzJ/Hl7s2ex9QRERE5CzUCMsF8/n9pI+ZzMAb7sfftntYNVzJEXZ++Bx5H79EWWmJxwlFREREzqRGWDzTOrUjo6bfQ+rIaRATF1aNkh0BVs+fy578tR6nExEREfkmNcLiKfP56DdiIoOmzyY2rW9YNVzpCQo++Qur3/sLJ4tPeJxQREREpIYaYWkQrVq3I/P6u+h4ybew2MSwapQWrCVvwVx2bFip2zSLiIiI59QIS4Mxn49eg8YwNPtB4jsPCquGKy9h/z9fJmfxHyk6fsTjhCIiIhLN1AhLg0tKTmHktbfTZcItWHxyWDUqvtrC2oXzyM/7TEeHRURExBNqhKXRdO8/nOEz55DYIzO8ApVlHAy8TmDRUxwrPOhtOBEREYk6aoSlUSUktmD4VbPoceX3scTWYdWoPLyTDYseYdPKD6muqvI2oIiIiEQNNcISEZ17pZN50xxa9BkLZqEXqKrkyJp3Cbz+OIe/KvA+oIiIiDR7aoQlYmLj4hl6+Ux6T/4BvuS0sGpUHd3LlsW/Z/3yxVRVVnqcUERERJozNcIScR269iFr5mxapV8R3m2aq6s4vvFjAvMf4cDeHd4HFBERkWZJjbA0Cf6YGAaNv47+U+/Fn9I5rBrVRQfZ9t5TrP1kARXlZR4nFBERkeZGjbA0Ke06dCVrxn20GToF/DGhF3COovzl5Myfx76dm70PKCIiIs2GGmFpcnx+P+mjriLjhvuJadczrBqu5Ai7PnqO3A9fpPRksbcBRUREpFkIqhE2sylmttnM8s3sZ2fZ5ttmtsHM1pvZf3sbU6JRSrsOZE37EamZ0yEmLqwaJ3flkLtgHnu25nmcTkRERC525pw79wZmfmALcDVQAKwEZjnnNtTZph/wCnClc+6ImbV3zh04V92srCwXCAQuNL9EiRPHCtm89DUqvtoSdo34LoNJv3QGSckpHiYTERGRps7McpxzWaePB3NEeDSQ75zb7pwrB14Cpp+2zZ3Ak865IwDna4JFQtUypS2ZU79Px7H/gsUlhVWjbO861iyYx471n+s2zSIiIhJUI9wF2FNnuaB2rK7+QH8zW2ZmK8xsSn2FzOwuMwuYWeDgQd0iV0JjPh+9MkYxLHsOCV2HhFXDVZxk/4pXyXn7OU4cK/Q4oYiIiFxMvPqwXAzQD7gcmAU8Z2atT9/IOfescy7LOZeVlhbeDRREElu0ZMQ1t9J14vewhJZh1ag4sJV1r89j6+pPdXRYREQkSgXTCO8FutVZ7lo7VlcBsMg5V+Gc20HNOcX9vIkoUr9u/YYxYuYcEnueccpPcCrLObRqESvfeJJjh7/yNpyIiIg0ecE0wiuBfmbWy8zigJuBRadt8zo1R4Mxs1RqTpXY7l1MkfrFJyQxfNLN9Jh0J5bUJqwaVYW72LDoUTZ98QHVVVUeJxQREZGm6ryNsHOuErgHeA/YCLzinFtvZg+b2bTazd4DDpvZBmAJMMc5d7ihQoucrnPPAWTOfJDkfuPBLPQC1VUcWfseKxf8jkP795x/exEREbnonffyaQ1Fl0+ThnJw3062f/YK1SfCvHiJ+Wg5YALpo68hJja86xeLiIhI03Ehl08Tuaikde5JVvYDpGRMAp8/9AKumhObPiFnwaN8VbDN+4AiIiLSJKgRlmbJHxNDxthrGXDdT/C3Pv1qf8GpLjrE9vefZs0/5lNeVupxQhEREYk0NcLSrLVt34WsG39Cm2HXgj8m9ALOUbztn6yaP4+92zd6H1BEREQiRo2wNHs+v5/0rElkTJtNTLteYdVwJ4+ye8mfyP3gvyk9WexxQhEREYkENcISNVLappE17YekZc2AmPiwapzcvYrc+XPZtTnX23AiIiLS6NQIS1Qxn4++w8YzZMaDxHYcEFYNV1bEvs/+Rs7i5yk+cdTbgCIiItJo1AhLVEpu1YbMa++g07ibsbiksGqUf7meNQsfYfu6z3WbZhERkYuQGmGJWubz0XNgFsOy5xDfdVh4RSpO8tXnr5Lz1rMcP6p7yIiIiFxM1AhL1Ets0ZKR13yPbpfdhiW0CqtGxcF8Nrz+CFtWfaKjwyIiIhcJNcIitbr2HcyImQ+S1Gt0WI93VeUcXv0mK994gqOH9nucTkRERLymRlikjviEJIZd+W16XnUnltQ2rBpVhbvZ+OZjbPz8PaqrqjxOKCIiIl5RIyxSj049BpB104Mk95sAFsa3SXUVR9d9wMoFj3Fo/27vA4qIiMgFUyMschYxsXEMmTidflN+jK9lh7BqVB/fz9bFT7Dus0VUVpR7nFBEREQuhBphkfNI7dyDrOz7SRl0Ffj8oRdw1ZzY/Ck58x9h/5587wOKiIhIWNQIiwTBHxNDxiVTSL/+p/hbdw2rRnXxYXZ88Ax5S16lvKzU44QiIiISKjXCIiFok9aJUTN+QrsR12P+uNALOEfJ9s9ZNX8ue7ev9z6giIiIBE2NsEiIzOej/8jLGTj9AWJSe4dVw508xu4lz7P6/b9RWlLkcUIREREJhhphkTCltEkl64Yf0H5UNsQmhlWjdE8uuQvmsmvTKo/TiYiIyPmoERa5AObz0WfoOIbOmE1cp4Fh1XBlxexb9t/kLP4TxSeOehtQREREzkqNsIgHWrRsTebUO+g8bhYW3yKsGuVfbmTNwkfYtma5btMsIiLSCNQIi3iox8BMhmfPIaHb8PAKVJzkwMoFBN58mmNHDnmaTURERL5JjbCIxxKSkhkx+Ra6X3E7lpgSVo3KQ9vZ+MajbA58rKPDIiIiDUSNsEgD6dJ7ECNnziGp9xgwC/nxrqqcwrzFBF7/PUcOftkACUVERKKbGmGRBhQXn8CwK75Fr6vuwteiXVg1Ko/sYdNbv2PDinepqqz0OKGIiEj0UiMs0gg6du9H5szZtBwwESyMb7vqKo6t/5DAgsc4tG+X9wFFRESikBphkUYSExvH4Eun0W/qPfhadQyrRvWJr9j67pOs/fQNKivKPU4oIiISXdQIizSy1I7dGZV9P60HXw0+f+gFXDVFW5cSeG0e+3dv9T6giIhIlFAjLBIBPr+fgWOuIf36nxLTpltYNVxJITs+eIa8j1+hrLTE44QiIiLNnxphkQhqk9aJrBvvpd2IGzB/XFg1SnZ8wer58yjIX+dxOhERkeZNjbBIhJnPR/+Rl5Fx42xi0/qEVcOVHmfPJy+w6r2/crL4hMcJRUREmic1wiJNRKvW7ci8/m7aj74JYhPDqlFWkEfegrns2pijG3GIiIicR1CNsJlNMbPNZpZvZj87x3YzzcyZWZZ3EUWih/l89BlyCUNnzCau06CwarjyEvYtf5FV7/yZouNHPE4oIiLSfJy3ETYzP/AkcC2QAcwys4x6tmsJ3Ad87nVIkWjTomVrMqfeTudLb8HiW4RVo3z/JtYunEd+3jIdHRYREalHMEeERwP5zrntzrly4CVgej3b/Rr4L6DUw3wiUa3HgOEMz55DYveR4RWoLONgYCGBRX/gWOFBb8OJiIhc5IJphLsAe+osF9SOnWJmI4Fuzrm3z1XIzO4ys4CZBQ4e1A9lkWAkJCUz/Orv0P2KO7DElLBqVB7ewYZFj7Ap8BHVVVUeJxQREbk4XfCH5czMBzwKzD7fts65Z51zWc65rLS0tAt9apGo0qX3QEbOnEOL3peAWegFqio5kvcOgdd/T+GBvd4HFBERucgE0wjvBepe8b9r7djXWgKDgX+Y2U7gEmCRPjAn4r24+ASGXnETva6+G19yalg1qo4WsPntx9nwz3eoqqz0OKGIiMjFI5hGeCXQz8x6mVkccDOw6OuVzrljzrlU51xP51xPYAUwzTkXaJDEIkLHbn3JzH6AlumXgYXxh53qKo5t+IjAgkc5uG+n5/lEREQuBuf9CeqcqwTuAd4DNgKvOOfWm9nDZjatoQOKSP1iYuMYPP4G+k29F1+rTmHVqD5xgPx3n2TtpwupKC/zOKGIiEjTZs65iDxxVlaWCwR00FjEC9VVVWwJfMSRDR9BdXgfhrOktnQfO5POPQd4nE5ERCSyzCzHOXfGabu6s5xIM+Dz+0kfM5mBN9yPv22PsGq4kkJ2ffQcuR+9RFlpiccJRUREmh41wiLNSOvUjoya/mNSR06DmLiwapzcGWD1/LnsyV/rcToREZGmRY2wSDNjPh/9Rkxk8I0PEtu+X1g1XOkJCj75C6vf+wsni094nFBERKRpUCMs0ky1TGlL5nV30vGSb2GxiWHVKC1YS96CuezYsFK3aRYRkWZHjbBIM2Y+H70GjWFo9oPEdxkcVg1XXsL+f75MzuI/UnT8iMcJRUREIkeNsEgUSEpOYeSU2+gy4RYsoWVYNSq+2sLahfPIz/tMR4dFRKRZUCMsEkW69x/O8OwHSeyRGV6ByjIOBl4nsOgpjh3+yttwIiIijUyNsEiUSUhswfCrZtHjyu9jSW3CqlF5eCcb3nyMTSs/pLoqvOsWi4iIRJoaYZEo1blXOpkzHyS57zgwC71AVSVH1rxLYOH/4fBXBd4HFBERaWBqhEWiWGxcPEMuy6bPNT/Cl5wWVo2qY/vYsvj3rF/2NlWVlR4nFBERaThqhEWE9l16kTVzNq3SrwCfP/QC1VUc37SEwPxHOLB3h/cBRUREGoAaYREBwB8Tw6Dx19F/6r34UzqHVaO66CDb3nuKtZ8soKK8zOOEIiIi3lIjLCLf0K5DV7Jm3EeboVPAHxN6Aecoyl9Ozmtz2bdjk/cBRUREPKJGWETO4PP7SR91FRk33E9Mu55h1XAnj7Lr4z+S++GLlJ4s9jagiIiIB9QIi8hZpbTrQNa0H5GWdSPExIVV4+SuHHLnz2X3llxPs4mIiFwoNcIick7m89F32KUMvvFBYjv0D6uGKyti79K/seqd5ykpOuZxQhERkfCoERaRoLRMaUvm1O/Tcey/YHFJYdUo27eeNQvmsWP957pNs4iIRJwaYREJmvl89MoYxbDsOSR0HRJWDVdxkv0rXiXn7ec4fvSwxwlFRESCp0ZYREKW2KIlI665la4Tv4cltAyrRsWBrax/4xG2rv5UR4dFRCQi1AiLSNi69RvGiJlzSOqVFV6BynIOrVrEyjee5Oih/d6GExEROQ81wiJyQeITkhh25c30mHQnltQmrBpVhbvY+OZjbPr8faqrqjxOKCIiUj81wiLiic49B5A580GS+40Hs9ALVFdxZN37rFzwOw7t3+N9QBERkdOoERYRz8TGxTNk4gz6Tvkxvpbtw6pRffxLti7+PeuWvUllRbnHCUVERP6HGmER8Vxa555kZT9ASsYk8PlDL+CqObHpE3IWPMr+PfneBxQREUGNsIg0EH9MDBljr2XAdT/B37prWDWqiw6x44NnWLPkNcrLSj1OKCIi0U6NsIg0qLbtu5B14720HTYV/DGhF3CO4u0rWDV/Lnu3b/Q+oIiIRC01wiLS4Hx+PwOyriRj2mxi2vUKq4Y7eYzdS/7E6vf/RunJYo8TiohINFIjLCKNJqVtGlnTfkha1gyIiQ+rRumeXHLnz2XX5lxvw4mISNRRIywijcp8PvoOG8+QGQ8S1zE9rBqurIh9n/2NnMXPU3ziqLcBRUQkaqgRFpGISG7VhpHX/hudx83C4pLCqlH+5XrWLHyEbWtX6DbNIiISsqAaYTObYmabzSzfzH5Wz/oHzGyDma0xs4/MrIf3UUWkuTGfjx4DMxmWPYf4rsPCK1JxkgNfvEbOW89y/OhhbwOKiEizdt5G2Mz8wJPAtUAGMMvMMk7bbDWQ5ZwbCrwG/H9eBxWR5iuxRUtGXvM9ul12G5bQKqwaFQfz2fD6I2xZ9YmODouISFCCOSI8Gsh3zm13zpUDLwHT627gnFvinCupXVwBhHfRUBGJal37DmbEzAdJ6jU6rMe7qnIOr36TlW88wdFD+z1OJyIizU0wjXAXYE+d5YLasbO5A3invhVmdpeZBcwscPDgweBTikjUiE9IYtiV36bnVXdiSW3DqlFVuJuNbz7Gxs/fo7qqyuOEIiLSXHj6YTkzuwXIAubWt94596xzLss5l5WWlublU4tIM9OpxwCybnqQ5H4TwMJ4q6qu4ui6D1i54DEO7d/tfUAREbnoBfPTZS/Qrc5y19qxbzCzq4D/AKY558q8iSci0SwmNo4hE6fTb8qP8bXsEFaN6uP72br4CdYtfYPKinKPE4qIyMUsmEZ4JdDPzHqZWRxwM7Co7gZmNgJ4hpom+ID3MUUkmqV27kFW9v2kDLoKfP7QC7hqTmxZSs78R9i/J9/7gCIiclE6byPsnKsE7gHeAzYCrzjn1pvZw2Y2rXazuUAy8KqZ5ZrZorOUExEJiz8mhoxLppB+/U/xtw7v87jVxYfZ8cEz5C15lfKyUo8TiojIxcaccxF54qysLBcIBCLy3CJycXPV1WzN/ZTCNe/jqsI73cESU+h2STZdeg/yOJ2IiDQ1ZpbjnMs6fVx3lhORi475fPQfeTkDpz9AbFqfsGq4k8fYveR5Vr//N0pLijxOKCIiFwM1wiJy0Uppk0rm9XfTflQ2xCaGVaN0Ty65C+aya2OOx+lERKSpUyMsIhc18/noM3QcQ2fMJq7TwLBquLJi9i1/kZzFf6L4xFFvA4qISJOlRlhEmoUWLVuTOfUOOo//DhbfIqwa5V9uZM3CR9i2Zrlu0ywiEgXUCItIs9IjfSTDs+eQ0G14eAUqTnJg5QICbz7NsSOHPM0mIiJNixphEWl2EpKSGTH5FrpfcTuWmBJWjcpD29n4xqNsDnyso8MiIs2UGmERaba69B7EyJlzSOo9BsxCfryrKqcwbzErFz5O4YEzbqgpIiIXOTXCItKsxcUnMOyKb9Hr6rvxtWgXVo2qowVsfvtxNqx4l6rKSo8TiohIpKgRFpGo0LFbXzJnzqblgIlgYbz1VVdxbP2HBBY8xqF9u7wPKCIijU6NsIhEjZjYOAZfOo1+U+/B16pjWDWqT3zF1nefZO2nb1BRXuZxQhERaUxqhEUk6qR27M6o7PtpPfhq8PlDL+CqKdq6lJz5j/Dlrs3eBxQRkUahRlhEopLP72fgmGsYeMP9+Nt2D6uGKylk54fPkffxS5SVlnicUEREGpoaYRGJaq1TOzJq+j20G3ED5o8Lq0bJjgCr589jT/5aj9OJiEhDUiMsIlHPfD76j7yMjBtnE5vWN6warvQ4BZ/8hVXv/ZWTxSc8TigiIg1BjbCISK1WrduRef1ddBjzLYhNDKtGWUEeeQvmsnNjQDfiEBFp4tQIi4jUYT4fvQePYVj2g8R1GhRWDVdewpfLXyLnnT9RdPyIxwlFRMQraoRFROqRlJxC5tTb6XzpLVh8clg1KvZvZu3CeeTnLdPRYRGRJkiNsIjIOfQYMJzhM+eQ2H1keAUqyzgYWEhg0R84VnjQ23AiInJB1AiLiJxHQmILhl/9HbpfcQeW2DqsGpWHd7Bh0SNsCnxEdVWVtwFFRCQsaoRFRILUpfdARs58kBZ9xoJZ6AWqKjmS9w6B1x+n8MBe7wOKiEhI1AiLiIQgLj6BoZfPpPfkH+BLTg2rRtXRvWx++3HWL19MVWWlxwlFRCRYaoRFRMLQoWsfMrMfoFX6FWBhvJVWV3F848cEFjzKgb07Pc8nIiLnp0ZYRCRMMbFxDBp/Hf2m3ouvVaewalSfOMC2955k7acLqSgv8zihiIicixphEZELlNqxG6Oyf0qbIdeAzx96Aeco2rqMnPnz2Ldzs/cBRUSkXmqERUQ84PP7SR99NRnTHsDftkdYNVzJEXZ99By5H71EWWmJxwlFROR0aoRFRDyU0q4Do6b/mNTM6RATF1aNkzsDrJ4/l11b8qiudh4nFBGRr5lzkXmTzcrKcoFAICLPLSLSGE4cK2Tzp69ScWBr2DWOp/Qnpv9k+nROo1vbJPy+MC7bJiIS5cwsxzmXdca4GmERkYbjqqvZuXElX+W8has4GVaNSl8CO9qMoyilH73TWtK3fTI92iUR69cf9UREgnG2RjgmEmFERKKF+Xz0GjSGDj3S2fTZQsr2rgu5Rkx1Kf0Of8xXZfvZWHkpG788TlyMj16pLejbPpme7VoQF6OmWEQkVGqERUQaQVJyCiOn3MburXns++J1XOmJkGsUJv7Ph/DKK6vZvP8Em/efIMZn9EhtQXJ8GFesEBGJkNZJcYzs3iaiGdQIi4g0ou79htG+a182LX+TkzuDPz3sYIv+HEvsVu+6ymrHtgNFXkUUEWkU3domRbwRDupvaWY2xcw2m1m+mf2snvXxZvZy7frPzayn50lFRJqJhMQWDJ90Mz0m3Yklnf+HQIUvkZ2tL2mEZCIi0eW8jbCZ+YEngWuBDGCWmWWcttkdwBHnXF/gMeC/vA4qItLcdO45gMyZD5LcdxzY2a8GsaPteKr8CY2YTEQkOgRzRHg0kO+c2+6cKwdeAqafts104C+1X78GTDI7x7u6iIgAEBsXz5DLsulzzY/wt+xwxvrCxJ4UJvaKQDIRkeYvmEa4C7CnznJB7Vi92zjnKoFjQLvTC5nZXWYWMLPAwYMHw0ssItIMte/Si1Ez76fvmGvp2DqJ+BgfVb44drQZf86jxSIiEr5G/bCcc+5Z4FmouY5wYz63iEhTZ/4Y0gZPIq3nUHpuXszhVhk4upJ/oIijJRWRjici0uwE0wjvBep+VLlr7Vh92xSYWQyQAhz2JKGISLRJTsNGfI9UMyaYcWnfVA4WlZF/oIj8A0UcLiqPdEIRkWYhmEZ4JdDPzHpR0/DeDHzntG0WAbcC/wRuAj52kbplnYhIc+D7nzPXzIz2LRNo3zKBcX1SOVzbFG89UMTBE2URDCkicnE7byPsnKs0s3uA9wA/8Gfn3HozexgIOOcWAX8C/mpm+UAhNc2yiIg0gHbJ8bRLjmdM73YcK6ngwIlSisurIh1LRCQkyfGRv52FRerAbVZWlgsEgr+YvIiIiIhIOMwsxzmXdfq4bk4vIiIiIlFJjbCIiIiIRCU1wiIiIiISldQIi4iIiEhUitiH5czsILArIk8OqcChCD33xUjzFRrNV2g0X6HRfIVG8xUazVdoNF+hieR89XDOpZ0+GLFGOJLMLFDfJwelfpqv0Gi+QqP5Co3mKzSar9BovkKj+QpNU5wvnRohIiIiIlFJjbCIiIiIRKVobYSfjXSAi4zmKzSar9BovkKj+QqN5is0mq/QaL5C0+TmKyrPERYRERERidYjwiIiIiIS5dQIi4iIiEhUalaNsJlNMbPNZpZvZj+rZ328mb1cu/5zM+tZZ93/qh3fbGbXNGrwCAlivh4wsw1mtsbMPjKzHnXWVZlZbu2/RY2bPDKCmK/bzOxgnXn5fp11t5rZ1tp/tzZu8sgIYr4eqzNXW8zsaJ110bh//dnMDpjZurOsNzN7vHY+15jZyDrronH/Ot98fbd2ntaa2XIzG1Zn3c7a8VwzCzRe6sgJYr4uN7Njdb7vflFn3Tm/l5ujIOZrTp25Wlf7ntW2dl007l/dzGxJbc+w3szuq2ebpvke5pxrFv8AP7AN6A3EAXlAxmnb/Ah4uvbrm4GXa7/OqN0+HuhVW8cf6dfUBObrCiCp9usffj1ftctFkX4NTXC+bgOeqOexbYHttf9tU/t1m0i/pkjP12nb3wv8uc5yVO1fta95IjASWHeW9VOBdwADLgE+rx2Puv0ryPka9/U8ANd+PV+1yzuB1Ei/hiY2X5cDb9UzHtL3cnP5d775Om3bG4CP6yxH4/7VCRhZ+3VLYEs9PyOb5HtYczoiPBrId85td86VAy8B00/bZjrwl9qvXwMmmZnVjr/knCtzzu0A8mvrNWfnnS/n3BLnXEnt4gqgayNnbEqC2b/O5hrgA+dcoXPuCPABMKWBcjYVoc7XLODFRknWRDnnPgUKz7HJdOD/uhorgNZm1ono3L/OO1/OueW18wF6/wpm/zqbC3nvu2iFOF96/3LuS+fcqtqvTwAbgS6nbdYk38OaUyPcBdhTZ7mAM/8nnNrGOVcJHAPaBfnY5ibU13wHNb/JfS3BzAJmtsLMbmyAfE1NsPM1s/ZPPq+ZWbcQH9ucBP2aa0+56QV8XGc42vavYJxtTqNx/wrV6e9fDnjfzHLM7K4IZWqKxppZnpm9Y2aDase0f52DmSVR07TNrzMc1fuX1Zx2OgL4/LRVTfI9LKaxnkguXmZ2C5AFXFZnuIdzbq+Z9QY+NrO1zrltkUnYZLwJvOicKzOzu6n568OVEc50MbgZeM05V1VnTPuXeMLMrqCmEb60zvCltftXe+ADM9tUewQwmq2i5vuuyMymAq8D/SIb6aJwA7DMOVf36HHU7l9mlkzNLwU/dc4dj3SeYDSnI8J7gW51lrvWjtW7jZnFACnA4SAf29wE9ZrN7CrgP4Bpzrmyr8edc3tr/7sd+Ac1v/01Z+edL+fc4Tpz9EcgM9jHNkOhvOabOe3PilG4fwXjbHMajftXUMxsKDXfi9Odc4e/Hq+zfx0AFtL8T4U7L+fccedcUe3Xi4FYM0tF+9f5nOv9K6r2LzOLpaYJ/rtzbkE9mzTJ97Dm1AivBPqZWS8zi6Nm5zz90+aLgK8/jXgTNSe3u9rxm63mqhK9qPkt+ItGyh0p550vMxsBPENNE3ygzngbM4uv/ToVGA9saLTkkRHMfHWqsziNmnOkAN4DJtfOWxtgcu1YcxbM9yNmlk7NhyP+WWcsGvevYCwC/rX2k9eXAMecc18SnfvXeZlZd2AB8D3n3JY64y3MrOXXX1MzX/VeGSCamFnH2s/MYGajqekPDhPk93I0MrMUav5S+kadsajcv2r3nT8BG51zj55lsyb5HtZsTo1wzlWa2T3UTJ6fmk+grzezh4GAc24RNf+T/mpm+dScBH9z7WPXm9kr1PywrQR+fNqfaZudIOdrLpAMvFr7/rjbOTcNGAg8Y2bV1LxZ/r/OuWbdqAQ5Xz8xs2nU7EOF1FxFAudcoZn9mpofKAAPn/ZntGYnyPmCmu/Bl2p/If1a1O1fAGb2IjWf3E81swLgfwOxAM65p4HF1HzqOh8oAW6vXRd1+xcENV+/oOYzIE/Vvn9VOueygA7AwtqxGOC/nXPvNvoLaGRBzNdNwA/NrBI4Cdxc+31Z7/dyBF5CowpivgBmAO8754rrPDQq9y9qDlh8D1hrZrm1Y/8OdIem/R6mWyyLiIiISFRqTqdGiIiIiIgETY2wiIiIiEQlNcIiIiIiEpXUCIuIiIhIVFIjLCIiIiJRSY2wiIiIiEQlNcIiIiIiEpX+fwTY780c831LAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(fs, hinge_loss(fs),fs[:25], hinge_loss(fs[:25]), linewidth=9, alpha=0.5)\n", "plt.legend(['Hinge Loss', 'Hinge Loss when $y \\cdot f < 1$'])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "When $y \\cdot f_\\theta(x) < 1$, we are in the \"orange line\" part and $J(\\theta)$ behaves like $1 - y \\cdot f_\\theta(x)$.\n", "\n", "\n", "Hence the gradient in this regime is: \n", "\n", "$$\n", "\\nabla_\\theta J(\\theta) = -y \\cdot \\nabla f_\\theta(x) = -y \\cdot x\n", "$$\n", "\n", "where we used $\\nabla_\\theta \\theta^\\top x = x$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "When $y \\cdot f_\\theta(x) \\geq 1$, we are in the \"flat\" part and $J(\\theta) = 0$.\n", "Hence the gradient is also just zero!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "What is the gradient for the hinge loss with a linear $f$?\n", "\n", "$$ \n", "J(\\theta) = \\max\\left(1 - y \\cdot f_\\theta(x), 0\\right) = \\max\\left(1 - y \\cdot \\theta^\\top x, 0\\right). \n", "$$\n", "\n", "When $y \\cdot f_\\theta(x) = 1$, we are in the \"kink\", and the gradient is not defined!\n", "* In practice, we can either take the gradient when $y \\cdot f_\\theta(x) > 1$ or the gradient when $y \\cdot f_\\theta(x) < 1$ or anything in between. This is called the *subgradient*." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### 12.4.1.1. A Steepest Descent Direction for the Hinge Loss\n", "\n", "We can define a \"gradient\" like function $\\tilde \\nabla_\\theta J(\\theta)$ for the hinge loss\n", "\n", "$$ \n", "J(\\theta) = \\max\\left(1 - y \\cdot f_\\theta(x), 0\\right) = \\max\\left(1 - y \\cdot \\theta^\\top x, 0\\right). \n", "$$\n", "\n", "It equals:\n", "\n", "$$\n", "\\tilde \\nabla_\\theta J(\\theta) = \\begin{cases} -y \\cdot x & \\text{ if $y \\cdot f_\\theta(x) < 1$} \\\\ 0 & \\text{ otherwise} \\end{cases} \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.4.2. (Sub-)Gradient Descent for SVM\n", "\n", "Putting this together, we obtain a gradient descent algorithm (technically, it's called subgradient descent).\n", "\n", "\n", "```python\n", "theta, theta_prev = random_initialization()\n", "while abs(J(theta) - J(theta_prev)) > conv_threshold:\n", " theta_prev = theta\n", " theta = theta_prev - step_size * approximate_gradient\n", "```\n", "\n", "Let's implement this algorithm." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "First we implement the approximate gradient." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "def svm_gradient(theta, X, y, C=.1):\n", " \"\"\"The (approximate) gradient of the cost function.\n", " \n", " Parameters:\n", " theta (np.array): d-dimensional vector of parameters\n", " X (np.array): (n,d)-dimensional design matrix\n", " y (np.array): n-dimensional vector of targets\n", " \n", " Returns:\n", " subgradient (np.array): d-dimensional subgradient\n", " \"\"\"\n", " yy = y.copy()\n", " yy[y*f(X,theta)>=1] = 0\n", " subgradient = np.mean(-yy * X.T, axis=1)\n", " subgradient[:-1] += C * theta[:-1]\n", " return subgradient" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "And then we implement subgradient descent." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Iteration 0. J: 3.728947\n", "Iteration 1000. J: 0.376952\n", "Iteration 2000. J: 0.359075\n", "Iteration 3000. J: 0.351587\n", "Iteration 4000. J: 0.344411\n", "Iteration 5000. J: 0.337912\n", "Iteration 6000. J: 0.331617\n", "Iteration 7000. J: 0.326604\n", "Iteration 8000. J: 0.322224\n", "Iteration 9000. J: 0.319250\n", "Iteration 10000. J: 0.316727\n", "Iteration 11000. J: 0.314800\n", "Iteration 12000. J: 0.313181\n", "Iteration 13000. J: 0.311843\n", "Iteration 14000. J: 0.310667\n", "Iteration 15000. J: 0.309561\n", "Iteration 16000. J: 0.308496\n", "Iteration 17000. J: 0.307523\n", "Iteration 18000. J: 0.306614\n", "Iteration 19000. J: 0.305768\n", "Iteration 20000. J: 0.305068\n", "Iteration 21000. J: 0.304293\n" ] } ], "source": [ "threshold = 5e-4\n", "step_size = 1e-2\n", "\n", "theta, theta_prev = np.ones((3,)), np.zeros((3,))\n", "iter = 0\n", "iris_X['one'] = 1\n", "X_train = iris_X.iloc[:,[0,1,-1]].to_numpy()\n", "y_train = iris_y2.to_numpy()\n", "\n", "while np.linalg.norm(theta - theta_prev) > threshold:\n", " if iter % 1000 == 0:\n", " print('Iteration %d. J: %.6f' % (iter, svm_objective(theta, X_train, y_train)))\n", " theta_prev = theta\n", " gradient = svm_gradient(theta, X_train, y_train)\n", " theta = theta_prev - step_size * gradient\n", " iter += 1" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We can visualize the results to convince ourselves we found a good boundary." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAEGCAYAAABM2KIzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA23ElEQVR4nO3dd5hU5dnH8e+9vReKtF06iqB0C6ACtkQlaCzRRJPoaySWmGiiSTSWaCxR876xt6iJLcYeS4wldlCJNEHBRm/S2cqW2bnfP3bAZV1gB3bmzO7+Pte1FzPPnDl7c5hZfnvPc55j7o6IiIiIiDRPUtAFiIiIiIi0JgrQIiIiIiJRUIAWEREREYmCArSIiIiISBQUoEVEREREopASdAHRyi3o4J27FwVdhoiIiAhJq78IugSJoQUbqta5e+fG460uQHfuXsS1j74UdBkiIiIiZN90RNAlSAwd+9inS5oab3UBWkRERCQoCswCcZgDbWbJZjbLzF5s4rHTzWytmc2OfP0k1vWIiIiIiOyOeHSgfwHMB/K28/jj7v6zONQhIiIiEhV1nKUpMQ3QZlYEHANcC/wylt9LREREZHcpMEtzxHoKx83Ar4HwDrY5wczmmNlTZlbc1AZmNtnMppvZ9LKNG2JRp4iIiIhIs8SsA21mE4E17j7DzMZvZ7MXgMfcvdrMfgo8CBzaeCN3vxe4F6DvoCEem4pFRESkPVLXWaIVyykcY4FJZnY0kAHkmdkj7n7alg3cfX2D7e8DboxhPSIiIiIKzLLbYhag3f0S4BKASAf6oobhOTLezd1XRe5Oov5kQxEREZEWo8AsLS3u60Cb2dXAdHd/Hvi5mU0CQsAG4PR41yMiIiIiEo24BGh3fwt4K3L7igbjW7vUIiIiIi1BHWeJNV2JUERERFo1BWaJt5hfiVBEREREpC1RB1pERERaFXWcJWgK0CIiIpLwFJolkShAi4iISMJRYJZEpjnQIiIiIiJRUAdaREREAqeOs7QmCtAiIiISdwrM0pppCoeIiIiISBTUgRYREZGYU8dZ2hIFaBEREWlxCszSlilAi4iIyG5TYJb2RHOgRURERESioA60iIiI7BJ1naW9UoAWERGRZlFgFqmnKRwiIiIiIlFQB1pERESapI6zSNMUoEVERARQYBZpLk3hEJGEsnr5Ej6dOY2KspKgSxEREWmSOtAikhDKSzdx12UXsGDeR3Qp6sWKRV8y8YeTOfbM8zGzoMsTaZPUcRbZNQrQIpIQ7r/2Ejp2686F/3svKalpbFiziuvPPY0uxb0Z/a1JQZcn0iYoMIu0DAVoEQlc6cYNzH3/HW5/5UNSUtMA6LBHN0465yJee+JBBWiRXaTALBIbmgMtIoGrLCshKzePjMysbcY7detB2aYNAVUlIiLSNHWgRSRwe/ToiXuYzz+awZ5DR24df/+V59h75IEBVibS+qjrLBJ7CtAiErik5GRO++UV3HzxZCb+6Gx69OnPjHdeY/a7b/D7vz0bdHkiCU2BWST+FKBFJCEccPgxdOzanf88+TCzp75J/32H84eHXyC/Y+egSxMREdlGzAO0mSUD04EV7j6x0WPpwEPASGA9cLK7L451TSKSmPrvM5z++wwPugyRhKaOs0jw4tGB/gUwH8hr4rEzgY3u3t/MTgFuAE6OQ00iIiKtggKzSOKJaYA2syLgGOBa4JdNbHIs8PvI7aeA283M3N1jWZeIiEiiUmAWSXyxXsbuZuDXQHg7j/cAlgG4ewgoATo23sjMJpvZdDObXrZRS1qJiIiISHBi1oE2s4nAGnefYWbjd2df7n4vcC9A30FD1J0WEZE2Qx1nkdYnllM4xgKTzOxoIAPIM7NH3P20BtusAIqB5WaWAuRTfzKhiIhIm6XQLNK6xWwKh7tf4u5F7t4bOAV4o1F4Bnge+HHk9omRbdRhFpGE4+6sX72K8tJNQZciIiIBi/s60GZ2NTDd3Z8H7gceNrMvgQ3UB20RkYTy8bQpPHjjFZSVbKS2ppp99j+IM3/3R/IKOwRdmrQS6jiLtC1xCdDu/hbwVuT2FQ3Gq4CT4lGDiMiuWLVkIbddch5nX/1nho2dQPXmSp688yZuvngyV9z3VNDlSYJSYBZp23QlQhGRHXj96Uc49PgfMPygQwHIyMrm1Asv58JjD2bxZ5/Qe6/BAVcoiUCBWaR9ifUydiIirdq6r1ZSPGDgNmNJyckU9d2T9V+tCKgqEREJkjrQIiI70G/wMGa98zpjvnXs1rHKslI+nzOd/7n0ugArkyCp4yzSvilAi4jswITvnsJlpx3Dw3/6PYdM+h6lG9bx5F3/y0FHH0/Hrt2DLk/iRIFZRBpSgBaR3VZTXUW4ro6MrOygS2lxOXkFXHn/Mzz/tzu57bfnkZmTw7hJ3+PQ408NujQREQmIArSI7LKSDet48MYrmPXOf3Cc/vuM4Me/vori/gN3/uRWpLBzF3588VVBlyFxpI6ziOyIArSI7JJwOMyN5/+IQaNGc8cr00nLyODt557g+nNP5Y+Pv6Y1kqXVUWgWkebSKhwiskvmz/iAulCIH1xwGVm5eaSkpnHYiacxZPQhvPsvrY8sIiJtlzrQIrJL1ixfQp+998XMthnvs/cQVi5eEFBVIs2njrOI7CoFaBHZJb32GsQ/77+NulCI5JSvf5R8PO1dhh10WICViTRNgVlEWooCtIjskr6DhlLUby9u+c3ZnPDTX5KRmcWrTzzIysULOO+624MuT0SBWURiRnOgRWSX/eKGO+nZf29uvmgyV//kRGqqq7j8L0+SkZkVdGlbuXvQJYiISBujDrSI7LK0jExOPOdXnHjOr4Iu5Rs+fPNlnrnnzyz5fB6de/Rk4g8nc/hJP/rGnG1pO9RxFpF4UYAWkTZn9tQ3efCGy5l85Z/YZ/+DWPTpXP5y9a8J1dZy1Kk/Cbo8aSEKzCISFE3hEJE25/m/3sGPLr6KIaPHkZScTL/Bwzjv2lt54cG7CNfVBV2eiIi0cupAi0ibs2rJQgYMGbHNWHH/gVRXbaayooycvIJgCpPdpq6ziCQCBWgRaXN69BnAZ7M+5MAjv7N1bMnn88jMyiYrJy/AyiRaCswikogUoEWkzTn2f87jrisuJDU9nX0POJiF8+dy/zW/ZdIZ55GUpJlriUyBWURaAwVoEWlz9j3wEM664iaevfdmbv3NuezRoyfH/HAy4449OejSRESkDVCAFmln1qxYyhN33sRHU98kIyubg485geN+8nPS0jOCLq1FDT/oUIYfdGjQZchOqOMsIq2RArRIO1K2aSN/OOskJhz3fU694DIqSjfx+O03cOflF3DBjXcHXZ60AwrMItIWaDKgSDvy1nOPM2jUGI6ffAGFnbtQ1G8vfnHj3Xzx0XSWL/gs6PJERERaBXWgRdqRZV9+yuD9xmwzlpKaxoCho1j65acU9dsroMqkrVLHWUTaIgVokXaka8/eLPhkNuMmfW/rWLiujkXz5nDsGecFWJm0FQrMItIeKECLtCMTjvs+l/7gKHoNGMQhk06isryMx2/7I12Ke9Nn732DLk9aKYVmEWlvNAdapB0p7NyFS+54lGmvv8T/HLQ3F3xnLJaUzAU33RN0aSIiIq1GzDrQZpYBvAOkR77PU+5+ZaNtTgduAlZEhm539/tiVZOIQG5hB7r27M3qZYtJz8yiW68+pGdk7tK+amuq+ddD9zD15eeoralixMFHcNxPfk5eYYcWrrrtqKmu4oW/3cX7rzxPXaiWURO+xbFnnt+qLi+ujrOItHexnMJRDRzq7uVmlgpMMbN/u/sHjbZ73N1/FsM6RCSisqyUq888gZHjj+TXtz1ERekmnrzzTyxf8BlnX/XnqPd3+6XnU1tTzU+v/BMZ2dm89viD/OGsk7jm4RdJz9y1UN6WuTt/vmgyKampnHvNzaSmpfPyYw9w7U9P4eoHnyM1LT3oEpukwCwisq2YTeHweuWRu6mRL4/V9xORnXv7hSfpM2gIp/3yCnr06c+eQ0dx8S1/ZfbUt1i1ZGFU+1r86ccsmjeHX/7vX+i/73CK+u7JGZdcS+fuRbz/ynMx+hu0bl/MmcHqZYu54MZ76DtoKMX9B/KTy24gOzefD9/4d9DliYhIM8X0JEIzSwZmAP2BO9x9WhObnWBmhwCfAxe6+7JY1iTSni2aP5ehY8ZvM5aWkcnAEQew+NOP6darb1T7Grz/WFJS07YZHzJ6HIs+/ZjxTT+tXVs0fy77HnAwySlf/+g1M4aOGc+i+XMZ8+3jgiuuAXWcRUR2LKYnEbp7nbsPA4qA/c1sn0abvAD0dvchwGvAg03tx8wmm9l0M5tetnFDLEsWadM6dy9iyWfzthkLh8Ms/XwenbsXR7WvTt2LWPL5PNy3/WBp6efz6NStaLdrbYs6das/Zo0t+fwTOnWL7vi3pOybjtjmS0REdiwuq3C4+ybgTeDbjcbXu3t15O59wMjtPP9edx/l7qNydXKSyC6bcNz3mfrvZ3nvlecI19VRWV7Go/93NXmFHem3z7Co9jV4v7F4OMwTd9xI1eZK6kIh3n7+CWZNeYNDvnNibP4CrdywsRMoL9nEM/feTPXmzYRqa3n96UeZN/19xh59XNDliYhIM1nj7lGL7disM1Dr7pvMLBN4FbjB3V9ssE03d18Vuf1d4DfufuCO9tt30BC/9tGXYlKzSHvwxZyZPHjTFaxashAPhxk2dgKn//baXVo5Y+Pa1fz1j7/j42lTMDOK+w/k9N/8gd4DG3/YJFusX72KB667lHnT3yMpKYleew3mjN9eQ3H/gXGrQV1mEZHmOfaxT2e4+6jG47EM0EOon5KRTH2n+wl3v9rMrgamu/vzZnY9MAkIARuAc9z90x3tVwFaWou5H7zDy39/gHVfraDvoCF85/Rz6d67X9BlbVW6cQOpaWlkZufs9r4qy8uoC4XILSjcrf3MmvIGrz7+Nzau+Yp++wxn0unn0KW4927Xt7tKN67nxQfvZu60d8nMzmHcpO9xyHdOwsx2eZ+VZaWEw3Xk5O/eMWsuhebEt76ylmfmb+DjNZXkpSfzrf4FjC3O3aXXWU1dmBc+28j7y8rAYExxLhP3LCQtWZd/EIlG3AN0rChAS2sw5V/P8PgdN/C9c39Ncf+9mD31TV7++/1c/pcn6dF3QNDlJaQ3n32Mf95/G98772J69BnAjLdf4z9PPcyV9z9N1559AqursqyUy3/0HQbtN4Zx3/keJRvW8fQ9f2bwfqM59cLLA6trZxSYW5dNm0Nc9OpixvbM46CeuayrDPHonLWM653HSYM7RbWvsDtXvbWM1OQkjhvYAXfnn59uIOxwxbii3frFT6S92V6A3ukqHGaWDpwA9G64vbtf3ZIFirQV4bo6Hr/jBi646R76DR4GQO+B+5CUnMxzf72Dc/9wc6D1JaJQbQ1P3HkTl9z5KD0H7A3UHzN354UH7+asy28IrLY3n/sHvfYazJmXXr91bK/h+3HhpIM4+rSzKOzcNbDapO148YuNjOqewxnD9wBgQEcY0DGDn7+0iKMGFJKTltzsfX30VSWbqur4v28Vk5xUH5YHdc7igpcXMXdNJUO6ZMfk7yDSnjTns5zngGOpn2ZR0eBLRJqwcd0a6kKhreF5i5HjjuSLOTOCKSrBrV21grSMzK3heYtR44/ky7kzA6qq3pdzZzFq/JHbjOXkFTBgyEgWzpsTUFXfpJU0WrfP121m/x7bTqfqlJVKUV4aS0uqt/Os7exr/WZGdc/ZGp4BkpOMUd1z+Gzd5hapV6S9a8460EXu/u2dbyYiANl5+VRXbaZ043ryCjtuHV+56As67KFuZVNyCwqpKNlERVkJ2bn5W8dXJMAx69C5CysWfbnNWDgcZuXiLwPtPiskty0dMlNYXlrDiO5fj9XWhfmqvJYOmdFdsqFDZgrTV5Z/Y3xZaQ1jinJ3t1QRoXkd6PfMbN+YVyLSRmRkZnHQUd/lvmt+Q3nJRgBWLPqSx269nm+dckbA1SWmnLwC9jv029x/7SVUlJUAsOzLT3nijps48uTTA63t0ONP5fWnHuHjaVNwd2qqq3j89hvI79CZPnvrR6O0jKMGFPLM/PVbO8Sba8PcP3MNAzpm0DUnbSfP3tbYnrl8tr6K1xduoi7s1IWd1xZsYsGGKsb0VIAWaQnbPYnQzOZSf+ntFGAAsBCoBoz6K3UPiVeRDekkQmkNaqqrePhPV/H+K8+TW1DI5opyvnvWBXzrlNODLi1hVW/ezN9uuIwP33iZnIICqjdv5qRzLuLQ438QdGnMnvomD95wOaFQLZsrKthr6CgmX3kT+R07x60GdZzbvqlLS3lg1hqSzKioqWNo12zO279rVPOft1i8sYrb/vsVaytqAdgjO5XzD+hGr4L0li5bpE2LehUOM+u1ox26+5IWqi0qCtDSmlSUlVCyfh2duxeRmqb/uJqjvHQTZRs30Ll70TcuEx6kcDjMmuVLyMzOiWtw3kIBun2oCzurK2rJSUsiLz26qRsNVYXC3D5tFXNXV2IGQ7pk87P9u5KWomXstqeyJsRt/13NZ+s2YwZDu2Rx9igds/Zul5exM7OH3f2HOxuLFwVoEWkPFJhlV4XDYX764kIKMlI4cVD9eRhPfrKe0uo67p7Yh6QkBcLGQuEwk59fSJecVL67dwfqwvDEJ+uoDoW5c2LirN8v8be9AN2cd9HghnfMLJntXHJbREREgvXC5xtxh+sO68UBRbkcUJTL9Yf3IhR2Xl6wKejyEtJTn6wnPcW45tCe7N8jl9HFufzx8F6U1YR5c1FJ0OVJAtru50NmdglwKZBpZqVbhoEa4N441CYi0m6o4ywt5cMVFRzcK4/U5K+XsUtNNg7ulcu05eUcPaBDgNUlplmrKhnXO3+bpf/SU5IYU5zLe8vKmNAnfwfPlvZouwHa3a8Hrjez6939kjjWJCLSLig0SywUZiazvLTmG+PLS2uiXhKvvcjPSGZF6TfX215eWk1xns5fkW/a7hQOMxthZiOAJ7fcbvgVxxpFRESkmU4b0pnZX1Xw/rIy3B13Z+rSUuasruTUIdFdFry9+OHQznywvJzpK8u3HrO3FpXwxfoqfrCvjpl8045+Ff3fyJ8ZwCjgI+qncAwBpgOjY1uaiEjboo6zxEOXnDTOGtGFW6et4t7pSYRxauqcn47qQqesxFnZJpEU56fz46Gd+dPUlWSlJlHnTm2dc/4BXcnLUNdevmlHUzgmAJjZM8AId58bub8P8Pu4VCci0oopMEtQjuxfwKF985i2vBwzOLBHjlbf2ImJe3Xg2wMK+GBZOclJxgE9shPmmK0oreHfX2xkVXkNvQsyOHpAAR2zUoMuq11rzq9Ve20JzwDu/rGZ7R3DmkREWiUFZkkkKUlJjO2ZF3QZrUpKUhIH9UqsYzZ/bSXXvbuCb/cv4Mh+BcxZXcmvXlnMNYf1pEjzswPTnAA9x8zuAx6J3D8VmBO7kkREREQE4K+z13LWyC4cEgn2BxTl0jEzhcfmruPisT0Crq79ak6APgM4B/hF5P47wF0xq0hEpJVQx1lEYqk6FGbhhirGFuduMz6+Tz7PzN8QUFUCzQjQ7l4F/DnyJSLSbikwi0g8pSQZKUlGaXUdhQ2WIFxfWUtuemLMz26vdrSM3RORP+ea2ZzGX/ErUURERKT9SU4yJvTJ4/6Zq6mtCwNQWVvH32av5Yi+BcEW187tqAO9ZcrGxHgUIiKSaNRxFpGgnT5sD27+YBVnPreAXgXpLNxYxUE98zh2oK4oGaQdLWO3KnLzcOAdd/8iPiWJiARHoVmkfXJ3PlxZHlnGDg7qmceQLlmY2c6fHEPpKUmcOWIPnpm3nqUlNXy7fwHHDuy4zWXHJf6aM4GmJ3CPmS00syfN7HwzGxbjukRERETiwt25/b9f8chH6+jXIYPivHTu+vArHv5obdClsWBDFb98eTEAh/XNZ8PmEBe+vIjV5d+8XLvET3NOIrwSwMwygbOAi4GbgeSYViYiEgfqOIvI/HWb+WRNJbcc1Yf0lPre4oQ++Zz3r4Uc2jc/0PWWH5i1mh8N68zhkTnPE/rk89jctTw2dx0XjO4eWF3t3U4DtJldBowFcoBZwEXAuzGuS0QkJhSYRaSxmasqOLhX3tbwDJCbnsyBxbnMWlURWICuDoX5bF0VV03ouc34Ef0KuDDSlZZgNGcd6OOBEPAv4G3gfXevjmlVIiItRIFZRHYmMyWJtRW13xgvra4jMyW45eKSk4zkJKioqSM/4+vIVlpdR1aqlrEL0k6PvruPoP5Ewv8CRwBzzWxKrAsTERERiYdDeuUxZWkpCzdWbR37ZE0lH6+u4MBGFzGJp5Qk46CeeTz00Vrqwg5ATV2YR+asZUKf/MDqkuZN4dgHOBgYB4wClqEpHCKSoNRxFpFodc5O5Zz9unL5G0vpV5hBnTvLSmq4aGwPctKCPeXrf4bvwQ1TVnD2iwvoV5jB/HWb2XePLE7Yu2OgdbV3zZnC8UfqL999K/Chu3/zMw4R+YaS9Wt5+4UnWb9qBX0GDWHMtyaRlpEZdFltTjwCc9id6SvLmf1VJVmpSYzvnRfoSUUNra2o5c1FJWysCrF3pyxGF+eSmqzlrSRxrS6v4YFZa1hdUUuP3DTOGN6ZTllpQZfF2J55jOiWw5zVFSQnGUO6ZJGWvOvTJFaW1fDmohIqauoY2jWbUd1zdmnpuey0ZK4+tCcLN1SxqryG04Z2TpifP+1Zc6ZwTHT3G939vWjCs5llmNl/zewjM/vEzK5qYpt0M3vczL40s2lm1jvK+kUS0oJPZvObk49k9bIldOvdj2mvvcjlP5pEecnGoEuTKNWFnRumrODvc9bROSuFUJ1zyX+W8vbikqBLY85XFfzylcVsrArRPTeNVxZs5HdvLGVzbTjo0kSaNHtVBee/tAgzOLxvPtWhMOf9axGfrq0MujQAMlOTOKAol1Hdc3YrPE9ZWspvXltCTZ2zR3Yqj3+8juvfXU4oMg1jV/TtkMHYnonzy3t7Z+67/o+5wx3Xrzye7e7lZpYKTAF+4e4fNNjmXGCIu59tZqcA33X3k3e0376Dhvi1j74Uk5pFWoK7c9lpx3DMDycz5tvHbR2775rfkJ2bzw8u+F2wBbYB8Zym8c7iUl78fAPXHd6LlEj3aMmmai59fQn3TepPZkAn8oTdOefFhfx0ZBdGdM8B6l9nN0xdSf8OGZw4SB/vSuI56/kvOWpAIcc3mH7w6Jy1TFlayl0T+wVYWcupDoU587kFXH1oMX0LMwAIhZ3L31jKEf0KOFRzl1uVYx/7dIa7j2o8HrOf/F6vPHI3NfLVOK0fCzwYuf0UcJgFfckfkd1Usn4ta1cu48AjJ20dMzOOOOlHzHznPwFW1npl33TENl/x9OHKco7sV7A1PAP0KkinT0EGnwTYNVteWn8RheHdsreOmRlH9y/gwxXl23uaSGBC4TBrK0IcPaBwm/Fj9ixkTRMrYLRW89dupjg/bWt4hvqTAY/sp/dmW9KcOdC7zMySgRlAf+AOd5/WaJMe1J+UiLuHzKwE6Aisa7SfycBkgE5de8SyZJHdlpKaSl1dHaGa6m3mPFdVVpCaro/emiORTgRMTTI2h745JaIqFCYtwEvppiUb1aEwYYeGU543h8KkaQ60JKAkIMnq3zsZDZaG21wbJqkN9c7SUqzJaVRVem+2KdvtQJvZC2b2/Pa+mrNzd69z92FAEbB/ZEWPqLn7ve4+yt1H5RZ22JVdiMRNTn4hew4dxYsP3bN1rLammmfvu4WxRx0XXGGyS8b1zuPFzzdSUhXaOjZteRkbqkIM3iMrsLq65qTROTuVf3/59bz6qlCYp+etZ1yvvMDqEtmepKQkeuSm8fBHa9kyfbQu7Dw8Zy0984M/ibCl7NUxk4raMO8vK9s6Vlpdx/OfbdR7sw3Z7hxoMxu3oye6+9tRfSOzK4BKd/9Tg7FXgN+7+/tmlgJ8BXT2HUzM1hxoaQ3Wr17Fjef/kJTUNIr7D+Tj/05hzyEjOfeaW0lJTQ26vISTSB3npvxj7jpe/HwDw7vlUFIVYklJNZceXMRenYJdVWVlWQ1XvbWM/IwUuuekMuurCvbrkcO5+3VtUx09aTtWl9dw8atLSEs29u6cxcdrKnF3/vztPhRmxvRD8bj6Yv1mrn13BcV5aRRmpDBzVTnfHlDIqft2QjNVW5ftzYGO5UmEnYFad99kZpnAq8AN7v5ig23OA/ZtcBLh8e7+vR3tVwE6GJsrynn/ledZs2Ipffbel5HjjlQQ3IlwXR2ffDiVdatW0HfwUHrtOSjokhJGogfmpqypqGXO6gqyUpMY2S1nm0v+Bmnj5hD/+Hgdaytq2a9HDkf2K9ilpbIAauvCvL+snCUl1XTLTeWgnnnbfNQu0hLC4TAvfbGJLzdUsXfnLI7om0dSUtt7ndXUhZmxsoKKmjr27ZJFl5y202VvT3Y5QJvZAOB6YBCwdUa8u/fdyfOGUH+CYDL1U0WecPerzexqYLq7P29mGcDDwHBgA3CKuy/c0X4VoONv5eIFXHf2KfTbZzi9Bw5mzntvU11VxaV3/52cvIKgy5NWqDUG6ET02brNXPvOcoZ3y6ZbbhrTlpeRm5bMZeOKol6Ca1NViMvfWEpBRgr77pHF5xuqWLKpmmsOLdZ//CLSbu1OgJ4CXAn8GfgOcAaQ5O5XxKLQnVGAjr9rz/4+o8YfybdOOQOoXyrrL1dfTFZuHqf9MpCXgbQyCswtz9352UuLOHVIJ8YU18+rrAs71767nKFdsjl2YHTni9z5369ISzZ+MrLL1rEnP1nHF+uruPSQohatXUSktdidZewy3f116sP2Enf/PXBMSxcoiamyrJQv587k0ON/sHXMzDj6tLP48M1XAqxMElmQy861FyvKaqipCzO6KHfrWHKS8Z09O/DB8rIdPLNpHywvY1Kj0D1xzw7MWFWxWxd/EBFpi5ozY7/azJKAL8zsZ8AKICe2ZUmisMi8NA9vuyRPKBQiOTk5iJIkQSkox1eSGXXh+sX1G854rnNnV6ZAJ1l9B7uhOnes0f5FRKR5HehfAFnAz4GRwA+BH8eyKEkcmdk5DBo5mpce+cvWsXBdHc8/cDsHHjExwMpE2rduOankZyTzxqKvLyleWxfm2fkbGNsz+qWyxvbM46l562k4re/pees5oChnl09KFBFpq5q9CoeZ5VF/gcHoPxtsQZoDHX/rVq3gunO+T2HnLvTaczAf//dd8go7cdEtfyUjM7h1cCVY6jgHb/GmKq5+azm9C9LpmpvGjJXl9CvM4Fdjukcdeitq6rjq7eXUhMLs0yWLL9ZXUVlbx9UTerap5cVERKKxOycRjgL+CmyZaFcC/I+7z2jxKptBAToYodpaZr7zH9auWkbvvfZh0KjRWsuynVFgTkzVoTDTlpezsSrEwE6Z7NkxY5ffm2F3PvqqkiWbquiWm8ao7uo+S721FTU8Nncdm6rrOLhnHhP65AddUsIrr6njvWVllNfUMaxr9jaX9pbWY3sBujlthQeAc939XQAzO4j6QD2kZUuURJaSmsr+hx0VdBki0kh6ShKH9G6Zq5slmTG8WzbDu2W3yP6kbXj5i43cP2sNQ7pk0TU7lb/OWsPT89Zz81G9SWmD6ze3hLmrK7hx6kr27ZJFYUYK17yznP2653D2qC5qPrURzQnQdVvCM4C7TzGz0I6eICKtnzrOIlITCvPX2Wu4eGx39u9R/0H0j4ftwa9eXcz9M9fw01FdA64w8YTCzv+9v4pfjenOsK71v4yeNqQzv3ltCR8sL2d0ce5O9iCtQXMC9Ntmdg/wGPUnfJ8MvGVmIwDcfWYM6xOROFFgFpHG3l5SSn56ytbwDPWfepw4qCN/n7MuwMoS1/y1lXTMTNkangEyU5OYuFch7y0rU4BuI5oToIdG/ryy0fhw6gP1oS1akYiIiCSEsNPksohJZtRHAGnMt3vM6s8zkLZhpwHa3SfEoxARiS91nEVkZyb0zuOBWauZ/VXF1o5q/XKJ6xnaVXPlm7J350xWV9Qyb20lgzrXr1RVHQrz0hebOH7v6K4QKolrpwHazLoA1wHd3f0oMxsEjHb3+2NenYi0KIVmEYlGWkoS39+3E9e+s5wDeuTQJSeVtxaXkmLGWSO67HwH7VBqchK/OKAb172znP175FKYmcLUpaXs1SmTMZq+0WY0Zxm7f1O/6sbv3H2omaUAs9x933gU2JiWsRNpPgVmEWkJy0qqeWTOWsqq6xhdnMsxAwpI0gocO7Rpc4h3lpRSXlvH8K7ZDOyUqRU4WqHdWcauk7s/YWaXALh7yMzqWrxCERERSUjF+elccnBR0GW0KgWZKUwaqCkbbVVzAnSFmXUkcraAmR1I/cVURCTBqOMsIiISe80J0L8Engf6mdlUoDNwYkyrEpFmUWAWERGJv+aswjHTzMYBewEGfObutTGvTEREREQkAW03QJvZfsAyd/8qMu95JHACsMTMfu/uG+JWpYgA6jiLiIgkgh11oO8BDgcws0OAPwLnA8OAe9E0DpGYU2AWERFJPDsK0MkNuswnA/e6+9PA02Y2O+aVibRDCszSlmzYHGLe2kpyUpPZt0sWyU1dnk1EpBXaYYA2sxR3DwGHAZOb+TwREWnnnvhkHf+cv4F9umSxoTJEaXUdl40romd+etCliYjsth0F4ceAt81sHbAZeBfAzPqjZexEWoy6ztLWzFpVwesLS7jjmL4UZtb/N/OfhZu4YcoKbju6D0m6mISItHLbDdDufq2ZvQ50A171ry9ZmET9XGgR2QUKzNLWvbmohOMGdtgangEO65PPs/M38OWGKvbsmBlgdSIiu2+HUzHc/YMmxj6PXTkiItLaVYXC5KQlbzNmZuSmJVNVGw6oKhGRlqO5zCIxpo6ztDcjumfzn4WbGNszd+t0jSWbqlleWsNendR9FpHWTwFapIUpMEt7d1iffN5dUsZlry9lXO981m+u5ZUvN/GTEXuQnpIUdHkiIrtNAVpkNykwi2wrNTmJ348vZuqyUj76qpLctCSumlBM74KMoEtrV2rrnPnrKjFgYKcsUpN18ubO1NaFmb92M0lJxt6dMrX0omxXzAK0mRUDDwFdAKd+HelbGm0zHngOWBQZesbdr45VTSIiEh+pycb43vmM750fdCnt0uxVFdz8wUo6Z6fiwLrKEBce2I2hXbODLi1hTV9Rzq3TVtE1J5WQw6bNIX41pjuD98gKujRJQLHsQIeAX7n7TDPLBWaY2WvuPq/Rdu+6+8QY1iHSotRxFpFEVlIV4k/vreS3B/dgn0j4m/NVBTdOXcldE/uSm568kz20P+sra7n5g1VcNq6IgZF5+jNXlvPHKSu45zt9yUrVMZNtxSxAu/sqYFXkdpmZzQd6AI0DtEhCU2AWkdZkytIyRnbP3hqeAYZ0zWZ4t2ymLC3lqAGFAVaXmN5eUsqY4tyt4RlgRPcc9u6cyQfLyjm0rz5JkW3F5WwOM+sNDAemNfHwaDP7yMz+bWaDt/P8yWY23cyml23c0NQmIiIiAlTU1FGY8c3+WGFGMhU1WkawKRU1YQoyv9ll7pCRQnlNXQAVSaKLeYA2sxzgaeACdy9t9PBMoJe7DwVuA/7Z1D7c/V53H+Xuo3ILO8S0XhGo7zpv+RIRaU2Gds3mvWVlVIe+DstVoTDvLStjaFfN523K0K5ZTF1aRk3d18essraOD1aUa964NCmmq3CYWSr14flRd3+m8eMNA7W7v2Rmd5pZJ3dfF8u6RBpTUBaRtmLPjhkM3iOL3/5nCcfsWQgOL36+kWHdshmgq0A2ad89suhTkM6lry/l6P6FhMLOi59vZExxLr0K0oMuTxJQLFfhMOB+YL67/992tukKrHZ3N7P9qe+Ir49VTSIiicDdWbKpmpBDn4J0LZUlLcrMOP+Arry/rIz3lpUB8L19OnJgUW7AlSUuM+PC0d2ZuqyMD5aVkZxknDqkE/v3yAm6NElQsexAjwV+CMw1s9mRsUuBngDufjdwInCOmYWAzcAp7u4xrEkEUMdZgrNoYxX/+95KasNOapJRFQpz/gFaXkxaVpIZY3vmMbZnXtCltBrJScYhvfI4pJeOmexcLFfhmALssK3i7rcDt8eqBpEtFJglEVSHwlz99nJ+PLQz43rnYWbM/qqCm6au5Naj+9AhU9e2EhFpDfTTWtokBWZJRB+uLKdnfjrj+3y9JNawrtmMLs7hrcUlHL93xwCrExGR5orLMnYiIgIlVXV0yU79xniX7DRKqrRUlohIa6EOtLQJ6jhLazB4j0yemreeqlCYjJT6/kVd2HlvWSnf37dzwNWJiEhzKUBLq6XQLK1N74IMRnbP5nevL+W4gR1ISzZe+mIjeekpjOimkwhFRFoLBWgRkTg6d7+uvLO4lLcWlxAKOwcU5XJE33wtZSci0oooQEuroY6ztAVJZozvk7/NiYQiItK6KEBLwlJgFhERkUSkAC0JQ4FZREREWgMtYyciIiIiEgV1oCUw6jiLiIhIa6QALXGjwCwiIiJtgaZwiIiIJAh3Z8PmEJtrw0GXIiI7oA60xIw6ziIizTd9RTkPzFpDWU0dobBzYFEuZ43cg6zU5KBLE5FGFKClRSk0i4hEb8GGKm6dtopfjunO0C5ZVNSGuX/mam79YBW/Pbgo6PJEpBEFaNktCswiIrvvpS828t29OzCsa/0l3XPSkjlnv66c+dwC1lTUskd2asAVikhDmgMtIiISsDUVtfQqSN9mLC05iW65aaytqA2oKhHZHnWgJSrqOIuItLz+HTKYuaqCEd1yto5t2hxieUk1PfPTd/BMEQmCArTskAKziEjsTdyzkIteXUJ2ahIH98pjTUUtj3y0lqP3LCQ3XScRiiQaBWgREZGAdcxK5Y+H9+SJT9bzh7eXk5eezNEDCjmsb37QpYlIExSgZRvqOIuIBKNLThrnH9At6DJEpBkUoNs5BWYRERGR6GgVDhERERGRKKgD3Q6p6ywiIiKy6xSg2wEFZhEREZGWowDdBikwi4iIiMSO5kCLiIiIiEQhZh1oMysGHgK6AA7c6+63NNrGgFuAo4FK4HR3nxmrmtoqdZwlGstLq3lo9lpmrKogM8UY3yef04Z0JiNFv0+LiIg0RyyncISAX7n7TDPLBWaY2WvuPq/BNkcBAyJfBwB3Rf6UHVBgll21qSrEZW8s47sDO3DB6G6UVdfx8Edr+dPUFVw2rjjo8kRERFqFmLWc3H3Vlm6yu5cB84EejTY7FnjI630AFJiZVpEXiZH/LChhZLdsjh3YgazUZLrkpHHh6O4s3FTN4o1VQZcnIiLSKsTlJEIz6w0MB6Y1eqgHsKzB/eWRsVXxqKu1UMdZWsqy0mqGdMnaZiw5yRjYMZOlpTX0LswIqDIREZHWI+YB2sxygKeBC9y9dBf3MRmYDNCpa+MmdtujwCyxUpyXzvx1mzmsb8HWsbqw8+n6zZw0uGNwhYmIiLQiMQ3QZpZKfXh+1N2faWKTFUDDiZdFkbFtuPu9wL0AfQcN8RiUGjiFZomHw/vlc8HLi3nu0w0c0S9/6xzoPgXp9FH3WUREpFliuQqHAfcD8939/7az2fPAz8zsH9SfPFji7pq+IRIjBRkpXHNoMQ/NXstDH63dugrHz4Z0Dro0ERGRViOWHeixwA+BuWY2OzJ2KdATwN3vBl6ifgm7L6lfxu6MGNaTUNRxlqAU5aVz6SFFQZchIiLSasUsQLv7FMB2so0D58WqhkSiwCwiIiLSNujKCSIiIiIiUYjLMnbtkTrOIiIiIm2TAnQLUWAWERERaR8UoHeRArOIiIhI+6Q50CIiIiIiUVAHupnUcRYRERERUIDeIYVmEREREWlMUzhERERERKKgDnQD6jiLiIiIyM606wCtwCwiIiIi0dIUDhERERGRKLSrDrQ6ziIiIiKyu9p0gFZgFhEREZGW1qYCtAKziIiIiMSa5kCLiIiIiESh1Xeg1XUWERERkXhqdQE6afUXCs0iIiIiEhhN4RARERERiYICtIiIiIhIFBSgRURERESioAAtIiIiIhIFBWgRERERkSgoQIuIiIiIREEBWkREREQkCgrQIiIiIiJRUIAWEREREYmCArSIiIiISBRiFqDN7AEzW2NmH2/n8fFmVmJmsyNfV8SqFhERERGRlpISw33/DbgdeGgH27zr7hNjWIOIiIiISIuKWQfa3d8BNsRq/yIiIiIiQYhlB7o5RpvZR8BK4CJ3/6SpjcxsMjA5crf82Mc+/ayFvn8nYF0L7Ut2jf4NgqXjHywd/2Dp+AdLxz94+jfYuV5NDZq7x+w7mllv4EV336eJx/KAsLuXm9nRwC3uPiBmxTRd33R3HxXP7ynb0r9BsHT8g6XjHywd/2Dp+AdP/wa7LrBVONy91N3LI7dfAlLNrFNQ9YiIiIiINEdgAdrMupqZRW7vH6llfVD1iIiIiIg0R8zmQJvZY8B4oJOZLQeuBFIB3P1u4ETgHDMLAZuBUzyW80madm+cv598k/4NgqXjHywd/2Dp+AdLxz94+jfYRTGdAy0iIiIi0tboSoQiIiIiIlFQgBYRERERiUK7CdBmlmxms8zsxSYeSzezx83sSzObFll+T1rQTo7/6Wa2tsFl3X8SRI1tmZktNrO5keM7vYnHzcxujbwH5pjZiCDqbKuacfzHm1lJg/fAFUHU2VaZWYGZPWVmn5rZfDMb3ehxvf5jqBnHX6//GDKzvRoc29lmVmpmFzTaRu+BKAV9IZV4+gUwH8hr4rEzgY3u3t/MTgFuAE6OZ3HtwI6OP8Dj7v6zONbTHk1w9+0tmH8UMCDydQBwV+RPaTk7Ov4A77r7xLhV077cArzs7ieaWRqQ1ehxvf5ja2fHH/T6jxl3/wwYBvXNLGAF8GyjzfQeiFK76ECbWRFwDHDfdjY5Fngwcvsp4LAtS+zJ7mvG8ZfgHQs85PU+AArMrFvQRYnsLjPLBw4B7gdw9xp339RoM73+Y6SZx1/i5zBggbsvaTSu90CU2kWABm4Gfg2Et/N4D2AZgLuHgBKgY1wqax9uZsfHH+CEyMdGT5lZcXzKalcceNXMZpjZ5CYe3/oeiFgeGZOWsbPjDzDazD4ys3+b2eB4FtfG9QHWAn+NTCO7z8yyG22j13/sNOf4g17/8XIK8FgT43oPRKnNB2gzmwiscfcZQdfSHjXz+L8A9Hb3IcBrfP1pgLScg9x9BPUf051nZocEXVA7s7PjPxPo5e5DgduAf8a5vrYsBRgB3OXuw4EK4LfBltSuNOf46/UfB5HpM5OAJ4OupS1o8wEaGAtMMrPFwD+AQ83skUbbrACKAcwsBchHV0VsKTs9/u6+3t2rI3fvA0bGt8S2z91XRP5cQ/3ct/0bbbL1PRBRFBmTFrCz4+/upe5eHrn9EpBqZp3iXmjbtBxY7u7TIvefoj7QNaTXf+zs9Pjr9R83RwEz3X11E4/pPRClNh+g3f0Sdy9y997Uf3Txhruf1miz54EfR26fGNlGV5hpAc05/o3mWU2i/mRDaSFmlm1muVtuA0cCHzfa7HngR5EzsQ8EStx9VZxLbZOac/zNrOuW8y7MbH/qfzbrl/gW4O5fAcvMbK/I0GHAvEab6fUfI805/nr9x833aXr6Bug9ELX2tArHNszsamC6uz9P/ckND5vZl8AG6oOexFCj4/9zM5sEhKg//qcHWVsb1AV4NvL/Uwrwd3d/2czOBnD3u4GXgKOBL4FK4IyAam2LmnP8TwTOMbMQsBk4Rb/Et6jzgUcjH2EvBM7Q6z+udnb89fqPscgv70cAP20wpvfAbtClvEVEREREotDmp3CIiIiIiLQkBWgRERERkSgoQIuIiIiIREEBWkREREQkCgrQIiIiIiJRUIAWEYkjM/udmX0SuXT9bDM7oIX3P97MXmzueAt8v+PMbFCD+2+Z2aiW/j4iIomk3a4DLSISb2Y2GpgIjHD36sjV1tICLmt3HQe8yDcvTiIi0mapAy0iEj/dgHVbLl3v7uvcfSWAmY00s7fNbIaZvbLlCp2Rju4tkW71x5ErtWFm+5vZ+2Y2y8zea3Clt52KXB3xATP7b+T5x0bGTzezZ8zsZTP7wsxubPCcM83s88hz/mJmt5vZGOqvHnpTpL5+kc1Pimz3uZkd3BIHTkQkkShAi4jEz6tAcSRY3mlm4wDMLBW4DTjR3UcCDwDXNnhelrsPA86NPAbwKXCwuw8HrgCui6KO3wFvuPv+wATqA3B25LFhwMnAvsDJZlZsZt2By4EDgbHAQAB3f4/6SwBf7O7D3H1BZB8pkX1fAFwZRV0iIq2CpnCIiMSJu5eb2UjgYOqD6+Nm9ltgOrAP8Frkkt/JwKoGT30s8vx3zCzPzAqAXOBBMxsAOJAaRSlHApPM7KLI/QygZ+T26+5eAmBm84BeQCfgbXffEBl/EthzB/t/JvLnDKB3FHWJiLQKCtAiInHk7nXAW8BbZjYX+DH1QfMTdx+9vac1cf8PwJvu/l0z6x3ZZ3MZcIK7f7bNYP0JjdUNhurYtf8ntuxjV58vIpLQNIVDRCROzGyvSMd4i2HAEuAzoHPkJEPMLNXMBjfY7uTI+EFASaRDnA+siDx+epSlvAKcb5F2t5kN38n2HwLjzKzQzFKAExo8VkZ9N1xEpN1QgBYRiZ8c6qddzDOzOcAg4PfuXgOcCNxgZh8Bs4ExDZ5XZWazgLuBMyNjNwLXR8aj7fL+gfopH3PM7JPI/e1y9xXUz7H+LzAVWAyURB7+B3Bx5GTEfk3vQUSkbTH3xp8MiohIojCzt4CL3H16wHXkROZwpwDPAg+4+7NB1iQiEhR1oEVEpDl+b2azgY+BRcA/A61GRCRA6kCLiIiIiERBHWgRERERkSgoQIuIiIiIREEBWkREREQkCgrQIiIiIiJRUIAWEREREYnC/wNU7dV5maickwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "xx, yy = np.meshgrid(np.arange(x_min, x_max, .02), np.arange(y_min, y_max, .02))\n", "Z = f(np.c_[xx.ravel(), yy.ravel(), np.ones(xx.ravel().shape)], theta)\n", "Z[Z<0] = 0\n", "Z[Z>0] = 1\n", "\n", "# Put the result into a color plot\n", "Z = Z.reshape(xx.shape)\n", "plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)\n", "\n", "# Plot also the training points\n", "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, edgecolors='k', cmap=plt.cm.Paired)\n", "plt.xlabel('Sepal length')\n", "plt.ylabel('Sepal width')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 12.4.3. Algorithm: Linear Support Vector Machine Classification\n", "The above procedure describes gradient descent optimization for support vector machines. The algorithm card below summarizes this algorithm and its components.\n", "\n", "* __Type__: Supervised learning (binary classification)\n", "\n", "* __Model family__: Linear decision boundaries.\n", "\n", "* __Objective function__: L2-regularized hinge loss.\n", "\n", "* __Optimizer__: Subgradient descent.\n", "\n", "* __Probabilistic interpretation__: No simple interpretation!" ] } ], "metadata": { "accelerator": "GPU", "celltoolbar": "Slideshow", "colab": { "collapsed_sections": [], "name": "neural-ode.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" }, "rise": { "controlsTutorial": false, "height": 900, "help": false, "margin": 0, "maxScale": 2, "minScale": 0.2, "progress": true, "scroll": true, "theme": "simple", "width": 1200 }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } } }, "nbformat": 4, "nbformat_minor": 1 }