{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HozF_O0VM6-B"
      },
      "source": [
        "This notebook is a brief guide of downloading HIRES raw
  (level 0) data in a given period. It requires PI credentials to
  access proprietary data. For more detail about PyKOA, please
  see:\n",
        "http://koa.ipac.caltech.edu/UserGuide/PyKOA/notebooks/PyKOA_HIRES_introduction.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mmPy9VTCNrt-"
      },
      "source": [
        "First, we need to install the KOA python client PyKOA. This
  line can be removed if PyKOA is already installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T3H-B5qiOHGo",
        "outputId": "cbd83ac6-fe74-4848-cdaa-93be24dfc290"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pykoa\n",
            "  Downloading pykoa-1.7.2-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: requests in
  /usr/local/lib/python3.10/dist-packages (from pykoa) (2.27.1)\n",
            "Collecting xmltodict (from pykoa)\n",
            "  Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0
  kB)\n",
            "Collecting bs4 (from pykoa)\n",
            "  Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
            "  Preparing metadata (setup.py)
  ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: lxml in
  /usr/local/lib/python3.10/dist-packages (from pykoa) (4.9.3)\n",
            "Requirement already satisfied: beautifulsoup4 in
  /usr/local/lib/python3.10/dist-packages (from bs4->pykoa)
  (4.11.2)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in
  /usr/local/lib/python3.10/dist-packages (from requests->pykoa)
  (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in
  /usr/local/lib/python3.10/dist-packages (from requests->pykoa)
  (2023.7.22)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0
  in /usr/local/lib/python3.10/dist-packages (from requests->pykoa)
  (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in
  /usr/local/lib/python3.10/dist-packages (from requests->pykoa)
  (3.4)\n",
            "Requirement already satisfied: soupsieve>1.2 in
  /usr/local/lib/python3.10/dist-packages (from
  beautifulsoup4->bs4->pykoa) (2.4.1)\n",
            "Building wheels for collected packages: bs4\n",
            "  Building wheel for bs4 (setup.py)
  ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for bs4:
  filename=bs4-0.0.1-py3-none-any.whl size=1256
  sha256=9334e687d56c0477c578bc56c1a2e197bf453c2c18b7d6f7447466c23f5a7776\n",
            "  Stored in directory:
  /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
            "Successfully built bs4\n",
            "Installing collected packages: xmltodict, bs4, pykoa\n",
            "Successfully installed bs4-0.0.1 pykoa-1.7.2
  xmltodict-0.13.0\n"
          ]
        }
      ],
      "source": [
        "!pip install pykoa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhVJIWD0OKsS"
      },
      "source": [
        "The next steop is to import pre-requisite packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4G2fXjioOKBa"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import io\n",
        "import os\n",
        "from pykoa.koa import Koa\n",
        "from astropy.table import Table,Column"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jJgI5RBOOOX"
      },
      "source": [
        "Then we need to assign directory that stores the query
  results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "OB_GgjpsOXVk"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    os.mkdir('./output')\n",
        "except:\n",
        "    print(\" Directory exists already\", flush=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3-Uu0RiOWm9"
      },
      "source": [
        "After that, we use a cookie for KOA log in. We can also spell
  out the credentials, but for security purpose, it is better to enter
  the userid and password interactively"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "olSx2E55OcgS",
        "outputId": "1fd7b11b-15ed-4ce0-b235-0f0495b7732e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Userid: clee\n",
            "Password: ··········\n",
            "Successfully login as clee\n"
          ]
        }
      ],
      "source": [
        "Koa.login ('./tapcookie.txt')\n",
        "#Koa.login ('./tapcookie.txt', userid='clee',
  password='salami')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBQdEWm9OdOs"
      },
      "source": [
        "We can then proceed to query HIRES data taken in a given
  period. Note the time period is specified in UT date. The output
  table will be saved in the designated directory and in IPAC table
  format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VaeWQLWROveh",
        "outputId": "fa9626fb-e01b-46ba-e4f5-d1f08521b111"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "submitting request...\n",
            "Result downloaded to file
  [./output/hires_daterange.tbl]\n",
            "        koaid          ...
  filehand                      \n",
            "----------------------
  ... -----------------------------------------------------\n",
            "HI.20210702.03381.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.03381.fits\n",
            "HI.20210702.03701.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.03701.fits\n",
            "HI.20210702.03826.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.03826.fits\n",
            "HI.20210702.03903.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.03903.fits\n",
            "HI.20210702.04038.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04038.fits\n",
            "HI.20210702.04083.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04083.fits\n",
            "HI.20210702.04129.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04129.fits\n",
            "HI.20210702.04218.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04218.fits\n",
            "HI.20210702.04314.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04314.fits\n",
            "HI.20210702.04383.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.04383.fits\n",
            "                   ... ...
  ...\n",
            "HI.20210702.53089.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.53089.fits\n",
            "HI.20210702.53773.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.53773.fits\n",
            "HI.20210702.54074.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.54074.fits\n",
            "HI.20210702.54353.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.54353.fits\n",
            "HI.20210702.54494.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.54494.fits\n",
            "HI.20210702.54661.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.54661.fits\n",
            "HI.20210702.54937.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.54937.fits\n",
            "HI.20210702.55100.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.55100.fits\n",
            "HI.20210702.55151.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.55151.fits\n",
            "HI.20210702.55235.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.55235.fits\n",
            "HI.20210702.55314.fits
  ... /koadata13/HIRES/20210702/lev0/HI.20210702.55314.fits\n",
            "Length = 113 rows\n"
          ]
        }
      ],
      "source": [
        "Koa.query_datetime ('hires', \\\n",
        "    '2021-07-02 00:00:00/2021-07-02 23:59:59', \\\n",
        "    './output/hires_daterange.tbl', overwrite=True,
  format='ipac', \\\n",
        "    cookiepath='./tapcookie.txt')\n",
        "\n",
        "rec = Table.read
  ('./output/hires_daterange.tbl',format='ipac')\n",
        "print (rec)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KV5BDsU4Osxd"
      },
      "source": [
        "Once we have the query result, we can download files to the
  designated directory, in this case to dnload_dir_hires/. If the
  download function is called several times, it will check the
  designated directory and only download new files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KxRXJeruPLET",
        "outputId": "3b6a5b49-91e8-458a-fe67-79a9035820a7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start downloading 3 koaid data you requested;\n",
            "please check your outdir: dnload_dir_hires for  progress
  ....\n",
            "\n",
            "A total of 3 new lev0 FITS files downloaded.\n"
          ]
        }
      ],
      "source": [
        "Koa.download ('./output/hires_daterange.tbl',\n",
        "    'ipac', \\\n",
        "    'dnload_dir_hires' )\n",
        "#    start_row=0, \\\n",
        "#    end_row = 2)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}