{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "HozF_O0VM6-B"
},
"source": [
"This notebook is a brief guide of downloading NIRES raw (level 0) data in a given period. It requires PI credentials to access proprietary data. For more detail about PyKOA, please see:\n",
"http://koa.ipac.caltech.edu/UserGuide/PyKOA/notebooks/PyKOA_NIRES_introduction.html"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mmPy9VTCNrt-"
},
"source": [
"First, we need to install the KOA python client PyKOA. This line can be removed if PyKOA is already installed."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "T3H-B5qiOHGo",
"outputId": "c56760e3-12a1-46d3-b818-93ab124f36ff"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Collecting pykoa\n",
" Downloading pykoa-1.7.2-py3-none-any.whl (29 kB)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from pykoa) (2.31.0)\n",
"Collecting xmltodict (from pykoa)\n",
" Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
"Collecting bs4 (from pykoa)\n",
" Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from pykoa) (4.9.3)\n",
"Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4->pykoa) (4.11.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2023.7.22)\n",
"Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4->pykoa) (2.4.1)\n",
"Building wheels for collected packages: bs4\n",
" Building wheel for bs4 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1256 sha256=f3706800cdd0951c5f2f3a169631345992aa24530be262cd4dc2ee42ccc94f6c\n",
" Stored in directory: /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
"Successfully built bs4\n",
"Installing collected packages: xmltodict, bs4, pykoa\n",
"Successfully installed bs4-0.0.1 pykoa-1.7.2 xmltodict-0.13.0\n"
]
}
],
"source": [
"!pip install pykoa"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hhVJIWD0OKsS"
},
"source": [
"The next steop is to import pre-requisite packages"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "4G2fXjioOKBa"
},
"outputs": [],
"source": [
"import sys\n",
"import io\n",
"import os\n",
"from pykoa.koa import Koa\n",
"from astropy.table import Table,Column"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9jJgI5RBOOOX"
},
"source": [
"Then we need to assign directory that stores the query results:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "OB_GgjpsOXVk"
},
"outputs": [],
"source": [
"try:\n",
" os.mkdir('./output')\n",
"except:\n",
" print(\" Directory exists already\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q3-Uu0RiOWm9"
},
"source": [
"After that, we use a cookie for KOA log in. We can also spell out the credentials, but for security purpose, it is better to enter the userid and password interactively"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "olSx2E55OcgS",
"outputId": "c87be452-4b97-43d4-85f8-5c63ffc0d26c"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Userid: clee\n",
"Password: ··········\n",
"Successfully login as clee\n"
]
}
],
"source": [
"Koa.login ('./tapcookie.txt')\n",
"#Koa.login ('./tapcookie.txt', userid='clee', password='salami')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kBQdEWm9OdOs"
},
"source": [
"We can then proceed to query NIRES data taken in a given period. Note the time period is specified in UT date. The output table will be saved in the designated directory and in IPAC table format."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VaeWQLWROveh",
"outputId": "ad19e84c-cffc-49ad-e833-f30968dddbcd"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"submitting request...\n",
"Result downloaded to file [./output/nires_daterange.tbl]\n",
" koaid instrume targname ... semid propint\n",
" ... \n",
"---------------------- -------- ---------- ... ---------- -------\n",
"NI.20210721.05574.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.05615.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.07870.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.07987.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08154.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08271.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08387.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08504.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08621.fits NIRES unknown ... 2021a_u130 18\n",
"NR.20210721.08737.fits NIRES unknown ... 2021a_u130 18\n",
" ... ... ... ... ... ...\n",
"NR.20210721.35648.fits NIRES A0V_UXHer ... 2021a_u130 18\n",
"NR.20210721.35703.fits NIRES A0V_UXHer ... 2021a_u130 18\n",
"NR.20210721.35755.fits NIRES A0V_UXHer ... 2021a_u130 18\n",
"NR.20210721.35811.fits NIRES A0V_UXHer ... 2021a_u130 18\n",
"NI.20210721.35967.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"NI.20210721.35979.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"NR.20210721.36046.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"NR.20210721.36382.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"NR.20210721.36717.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"NR.20210721.37053.fits NIRES J1757+1142 ... 2021a_u130 18\n",
"Length = 145 rows\n"
]
}
],
"source": [
"Koa.query_datetime ('nires', \\\n",
" '2021-07-21 00:00:00/2021-07-21 23:59:59', \\\n",
" './output/nires_daterange.tbl', overwrite=True, format='ipac', \\\n",
" cookiepath='./tapcookie.txt')\n",
"\n",
"rec = Table.read ('./output/nires_daterange.tbl',format='ipac')\n",
"print (rec)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KV5BDsU4Osxd"
},
"source": [
"Once we have the query result, we can download files to the designated directory, in this case to dnload_dir_nires/. If the download function is called several times, it will check the designated directory and only download new files."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KxRXJeruPLET",
"outputId": "1c3a3874-8b5e-4ddf-f236-abf2a5adeb21"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Start downloading 3 koaid data you requested;\n",
"please check your outdir: dnload_dir_nires for progress ....\n",
"\n",
"A total of 3 new lev0 FITS files downloaded.\n"
]
}
],
"source": [
"Koa.download ('./output/nires_daterange.tbl',\n",
" 'ipac', \\\n",
" 'dnload_dir_nires', \\\n",
"# start_row=0, \\\n",
"# end_row = 2, \\\n",
" cookiepath='./tapcookie.txt')"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}