{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了运行以下笔记本，如果尚未完成，您需要部署一个使用 `text-embedding-ada-002` 作为基础模型的模型，并在 .env 文件中将部署名称设置为 `AZURE_OPENAI_EMBEDDINGS_ENDPOINT`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from openai import AzureOpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "client = AzureOpenAI(\n",
    "  api_key=os.environ['AZURE_OPENAI_API_KEY'],  # this is also the default, it can be omitted\n",
    "  api_version = \"2024-10-21\",\n",
    "  azure_endpoint = os.environ['AZURE_OPENAI_ENDPOINT']\n",
    "  )\n",
    "\n",
    "model = os.environ['AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT']\n",
    "\n",
    "SIMILARITIES_RESULTS_THRESHOLD = 0.75\n",
    "DATASET_NAME = \"../embedding_index_3m.json\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将把嵌入索引加载到Pandas数据框中。嵌入索引存储在一个名为 `embedding_index_3m.json` 的JSON文件中。嵌入索引包含了截至2023年10月底的每个YouTube文稿的嵌入向量。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_dataset(source: str) -> pd.core.frame.DataFrame:\n",
    "    # Load the video session index\n",
    "    pd_vectors = pd.read_json(source)\n",
    "    return pd_vectors.drop(columns=[\"text\"], errors=\"ignore\").fillna(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将创建一个名为 `get_videos` 的函数，用于在嵌入索引中搜索查询。该函数将返回与查询最相似的前 5 个视频。函数的工作流程如下：\n",
    "\n",
    "1. 首先，创建嵌入索引的副本。\n",
    "2. 接着，使用 OpenAI 嵌入 API 计算查询的嵌入向量。\n",
    "3. 然后，在嵌入索引中创建一个名为 `similarity` 的新列。`similarity` 列包含查询嵌入与每个视频片段嵌入之间的余弦相似度。\n",
    "4. 接下来，根据 `similarity` 列过滤嵌入索引。过滤条件为只包含余弦相似度大于或等于 0.75 的视频。\n",
    "5. 最后，根据 `similarity` 列对嵌入索引进行排序，并返回前 5 个视频。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosine_similarity(a, b):\n",
    "    if len(a) > len(b):\n",
    "        b = np.pad(b, (0, len(a) - len(b)), 'constant')\n",
    "    elif len(b) > len(a):\n",
    "        a = np.pad(a, (0, len(b) - len(a)), 'constant')\n",
    "    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
    "\n",
    "def get_videos(\n",
    "    query: str, dataset: pd.core.frame.DataFrame, rows: int\n",
    ") -> pd.core.frame.DataFrame:\n",
    "    # create a copy of the dataset\n",
    "    video_vectors = dataset.copy()\n",
    "\n",
    "    # get the embeddings for the query    \n",
    "    query_embeddings = client.embeddings.create(input=query, model=model).data[0].embedding\n",
    "\n",
    "    # create a new column with the calculated similarity for each row\n",
    "    video_vectors[\"similarity\"] = video_vectors[\"ada_v2\"].apply(\n",
    "        lambda x: cosine_similarity(np.array(query_embeddings), np.array(x))\n",
    "    )\n",
    "\n",
    "    # filter the videos by similarity\n",
    "    mask = video_vectors[\"similarity\"] >= SIMILARITIES_RESULTS_THRESHOLD\n",
    "    video_vectors = video_vectors[mask].copy()\n",
    "\n",
    "    # sort the videos by similarity\n",
    "    video_vectors = video_vectors.sort_values(by=\"similarity\", ascending=False).head(\n",
    "        rows\n",
    "    )\n",
    "\n",
    "    # return the top rows\n",
    "    return video_vectors.head(rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个函数非常简单，它只是打印搜索查询的结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_results(videos: pd.core.frame.DataFrame, query: str):\n",
    "    def _gen_yt_url(video_id: str, seconds: int) -> str:\n",
    "        \"\"\"convert time in format 00:00:00 to seconds\"\"\"\n",
    "        return f\"https://youtu.be/{video_id}?t={seconds}\"\n",
    "\n",
    "    print(f\"\\nVideos similar to '{query}':\")\n",
    "    for _, row in videos.iterrows():\n",
    "        youtube_url = _gen_yt_url(row[\"videoId\"], row[\"seconds\"])\n",
    "        print(f\" - {row['title']}\")\n",
    "        print(f\"   Summary: {' '.join(row['summary'].split()[:15])}...\")\n",
    "        print(f\"   YouTube: {youtube_url}\")\n",
    "        print(f\"   Similarity: {row['similarity']}\")\n",
    "        print(f\"   Speakers: {row['speaker']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 首先，将嵌入索引加载到 Pandas Dataframe 中。\n",
    "2. 接下来，提示用户输入查询。\n",
    "3. 然后调用 `get_videos` 函数在嵌入索引中搜索查询。\n",
    "4. 最后，调用 `display_results` 函数向用户显示结果。\n",
    "5. 然后提示用户输入另一个查询。此过程会持续，直到用户输入 `exit`。\n",
    "\n",
    "![](../../../../translated_images/zh-CN/notebook-search.1e320b9c7fcbb0bc.webp)\n",
    "\n",
    "系统会提示你输入查询。输入查询后按回车。应用程序会返回与查询相关的视频列表。应用程序还会返回视频中答案所在位置的链接。\n",
    "\n",
    "这里有一些可尝试的查询：\n",
    "\n",
    "- 什么是 Azure 机器学习？\n",
    "- 卷积神经网络是如何工作的？\n",
    "- 什么是神经网络？\n",
    "- 我可以将 Jupyter Notebooks 与 Azure 机器学习一起使用吗？\n",
    "- 什么是 ONNX？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_vectors = load_dataset(DATASET_NAME)\n",
    "\n",
    "# get user query from input\n",
    "while True:\n",
    "    query = input(\"Enter a query: \")\n",
    "    if query == \"exit\":\n",
    "        break\n",
    "    videos = get_videos(query, pd_vectors, 5)\n",
    "    display_results(videos, query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免责声明**：\n本文件由 AI 翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 翻译完成。尽管我们力求准确，但请注意，自动翻译可能包含错误或不准确之处。原始语言版文件应视为权威来源。对于重要信息，建议使用专业人工翻译。我们对因使用本翻译而产生的任何误解或误释不承担责任。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}