{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HozF_O0VM6-B"
      },
      "source": [
        "This notebook is a brief guide of downloading NIRSPEC raw (level 0) data in a given period. It requires PI credentials to access proprietary data. For more detail about PyKOA, please see:\n",
        "http://koa.ipac.caltech.edu/UserGuide/PyKOA/notebooks/PyKOA_NIRSPEC_introduction.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mmPy9VTCNrt-"
      },
      "source": [
        "First, we need to install the KOA python client PyKOA. This line can be removed if PyKOA is already installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T3H-B5qiOHGo",
        "outputId": "6984a9a4-91c1-48d5-915b-38c6f00e5c8e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pykoa\n",
            "  Downloading pykoa-1.7.2-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from pykoa) (2.31.0)\n",
            "Collecting xmltodict (from pykoa)\n",
            "  Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
            "Collecting bs4 (from pykoa)\n",
            "  Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from pykoa) (4.9.3)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4->pykoa) (4.11.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.2.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2023.7.22)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4->pykoa) (2.4.1)\n",
            "Building wheels for collected packages: bs4\n",
            "  Building wheel for bs4 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1256 sha256=7d241eaf41f0a13736c93ca1bc53d333565391bf6a57dbfedceeac7b2350d530\n",
            "  Stored in directory: /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
            "Successfully built bs4\n",
            "Installing collected packages: xmltodict, bs4, pykoa\n",
            "Successfully installed bs4-0.0.1 pykoa-1.7.2 xmltodict-0.13.0\n"
          ]
        }
      ],
      "source": [
        "!pip install pykoa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhVJIWD0OKsS"
      },
      "source": [
        "The next steop is to import pre-requisite packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4G2fXjioOKBa"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import io\n",
        "import os\n",
        "from pykoa.koa import Koa\n",
        "from astropy.table import Table,Column"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jJgI5RBOOOX"
      },
      "source": [
        "Then we need to assign directory that stores the query results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "OB_GgjpsOXVk"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    os.mkdir('./output')\n",
        "except:\n",
        "    print(\" Directory exists already\", flush=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3-Uu0RiOWm9"
      },
      "source": [
        "After that, we use a cookie for KOA log in. We can also spell out the credentials, but for security purpose, it is better to enter the userid and password interactively"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "olSx2E55OcgS",
        "outputId": "0ecee36c-1d19-4a40-f352-628f50a61ef4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Userid: clee\n",
            "Password: ··········\n",
            "Successfully login as clee\n"
          ]
        }
      ],
      "source": [
        "Koa.login ('./tapcookie.txt')\n",
        "#Koa.login ('./tapcookie.txt', userid='clee', password='salami')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBQdEWm9OdOs"
      },
      "source": [
        "We can then proceed to query NIRSPEC data taken in a given period. Note the time period is specified in UT date. The output table will be saved in the designated directory and in IPAC table format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VaeWQLWROveh",
        "outputId": "ecdf9e63-0c12-408d-97bd-c06319c35767"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "submitting request...\n",
            "Result downloaded to file [./output/nirspec_daterange.tbl]\n",
            "        koaid          instrume     targname    ... echlpos filter slitname\n",
            "                                                ...                        \n",
            "---------------------- -------- --------------- ... ------- ------ --------\n",
            "NC.20210720.03740.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03744.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03746.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03749.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03752.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03754.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03757.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03760.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03869.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "NC.20210720.03873.fits  NIRSPEC         unknown ...    60.4  Blank 0.432x24\n",
            "                   ...      ...             ... ...     ...    ...      ...\n",
            "NS.20210720.55116.fits  NIRSPEC          HR8781 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55158.fits  NIRSPEC          HR8781 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55168.fits  NIRSPEC          HR8781 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55177.fits  NIRSPEC          HR8781 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55185.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55194.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55203.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55212.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55222.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "NS.20210720.55230.fits  NIRSPEC Neptune   15:15 ...    60.5 M-wide 0.432x24\n",
            "Length = 2639 rows\n"
          ]
        }
      ],
      "source": [
        "Koa.query_datetime ('nirspec', \\\n",
        "    '2021-07-20 00:00:00/2021-07-20 23:59:59', \\\n",
        "    './output/nirspec_daterange.tbl', overwrite=True, format='ipac', \\\n",
        "    cookiepath='./tapcookie.txt')\n",
        "\n",
        "rec = Table.read ('./output/nirspec_daterange.tbl',format='ipac')\n",
        "print (rec)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KV5BDsU4Osxd"
      },
      "source": [
        "Once we have the query result, we can download files to the designated directory, in this case to dnload_dir_hires/. If the download function is called several times, it will check the designated directory and only download new files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KxRXJeruPLET",
        "outputId": "d58d36a1-158f-45ec-a511-0036b70cca3a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start downloading 3 koaid data you requested;\n",
            "please check your outdir: dnload_dir_nirspec for  progress ....\n",
            "\n",
            "A total of 3 new lev0 FITS files downloaded.\n"
          ]
        }
      ],
      "source": [
        "Koa.download ('./output/nirspec_daterange.tbl',\n",
        "    'ipac',\n",
        "    'dnload_dir_nirspec',\n",
        "#    start_row=0,\n",
        "#    end_row = 2,\n",
        "    cookiepath='./tapcookie.txt')"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}