{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "HozF_O0VM6-B"
},
"source": [
"This notebook is a brief guide of downloading MOSFIRE raw (level 0) data in a given period. It requires PI credentials to access proprietary data. For more detail about PyKOA, please see:\n",
"http://koa.ipac.caltech.edu/UserGuide/PyKOA/notebooks/PyKOA_MOSFIRE_introduction.html"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mmPy9VTCNrt-"
},
"source": [
"First, we need to install the KOA python client PyKOA. This line can be removed if PyKOA is already installed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "T3H-B5qiOHGo",
"outputId": "56993796-be3c-40c9-9cfb-ec29c61dab55"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Collecting pykoa\n",
" Downloading pykoa-1.7.2-py3-none-any.whl (29 kB)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from pykoa) (2.31.0)\n",
"Collecting xmltodict (from pykoa)\n",
" Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
"Collecting bs4 (from pykoa)\n",
" Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from pykoa) (4.9.3)\n",
"Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4->pykoa) (4.11.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2023.7.22)\n",
"Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4->pykoa) (2.4.1)\n",
"Building wheels for collected packages: bs4\n",
" Building wheel for bs4 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1256 sha256=2c99f09d8e47a0565c3a08d378edcc29446980c88ab222fb2f010f8fdcaeabd0\n",
" Stored in directory: /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
"Successfully built bs4\n",
"Installing collected packages: xmltodict, bs4, pykoa\n",
"Successfully installed bs4-0.0.1 pykoa-1.7.2 xmltodict-0.13.0\n"
]
}
],
"source": [
"!pip install pykoa"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hhVJIWD0OKsS"
},
"source": [
"The next steop is to import pre-requisite packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4G2fXjioOKBa"
},
"outputs": [],
"source": [
"import sys\n",
"import io\n",
"import os\n",
"from pykoa.koa import Koa\n",
"from astropy.table import Table,Column"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9jJgI5RBOOOX"
},
"source": [
"Then we need to assign directory that stores the query results:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OB_GgjpsOXVk"
},
"outputs": [],
"source": [
"try:\n",
" os.mkdir('./output')\n",
"except:\n",
" print(\" Directory exists already\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q3-Uu0RiOWm9"
},
"source": [
"After that, we use a cookie for KOA log in. We can also spell out the credentials, but for security purpose, it is better to enter the userid and password interactively"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "olSx2E55OcgS",
"outputId": "9679166c-df54-49be-e4de-af74c568a6f5"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Userid: clee\n",
"Password: ··········\n",
"Successfully login as clee\n"
]
}
],
"source": [
"Koa.login ('./tapcookie.txt')\n",
"#Koa.login ('./tapcookie.txt', userid='clee', password='salami')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VaeWQLWROveh",
"outputId": "7668e05e-aa58-40c6-b721-cc8ef49b5e5b"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"submitting request...\n",
"Result downloaded to file [./output/mosfire_daterange.tbl]\n",
" koaid instrume targname ... semid propint\n",
" ... \n",
"---------------------- -------- ------------ ... ---------- -------\n",
"MF.20210816.00543.fits MOSFIRE horizon lock ... 2021b_eng 18\n",
"MF.20210816.00585.fits MOSFIRE horizon lock ... 2021b_eng 18\n",
"MF.20210816.09149.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09174.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09199.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09224.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09248.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09273.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09298.fits MOSFIRE unknown ... 2021b_h285 18\n",
"MF.20210816.09323.fits MOSFIRE unknown ... 2021b_h285 18\n",
" ... ... ... ... ... ...\n",
"MF.20210816.55215.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55228.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55241.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55256.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55269.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55282.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55295.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55308.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55322.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"MF.20210816.55336.fits MOSFIRE DOME FLAT ... 2021b_h285 18\n",
"Length = 467 rows\n"
]
}
],
"source": [
"Koa.query_datetime ('mosfire', \\\n",
" '2021-08-16 00:00:00/2021-08-16 23:59:59', \\\n",
" './output/mosfire_daterange.tbl', overwrite=True, format='ipac', \\\n",
" cookiepath='./tapcookie.txt')\n",
"\n",
"rec = Table.read ('./output/mosfire_daterange.tbl',format='ipac')\n",
"print (rec)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kBQdEWm9OdOs"
},
"source": [
"We can then proceed to query MOSFIRE data taken in a given period. Note the time period is specified in UT date. The output table will be saved in the designated directory and in IPAC table format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KV5BDsU4Osxd"
},
"source": [
"Once we have the query result, we can download files to the designated directory, in this case to dnload_dir_mosfire/. If the download function is called several times, it will check the designated directory and only download new files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KxRXJeruPLET",
"outputId": "4899fa7e-bba2-4bc6-d057-ae6c01b87211"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Start downloading 3 koaid data you requested;\n",
"please check your outdir: dnload_dir_mosfire for progress ....\n",
"\n",
"A total of 0 new lev0 FITS files downloaded.\n"
]
}
],
"source": [
"Koa.download ('./output/mosfire_daterange.tbl',\n",
" 'ipac', \\\n",
" 'dnload_dir_mosfire', \\\n",
"# start_row=0, \\\n",
"# end_row=2, \\\n",
" cookiepath='./tapcookie.txt')"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}