{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "HozF_O0VM6-B"
},
"source": [
"This notebook is a brief guide of downloading HIRES raw (level 0) data in a given period. It requires PI credentials to access proprietary data. For more detail about PyKOA, please see:\n",
"http://koa.ipac.caltech.edu/UserGuide/PyKOA/notebooks/PyKOA_HIRES_introduction.html"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mmPy9VTCNrt-"
},
"source": [
"First, we need to install the KOA python client PyKOA. This line can be removed if PyKOA is already installed."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "T3H-B5qiOHGo",
"outputId": "8a8b5f01-816b-4e46-981d-d65dbde17a20"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Collecting pykoa\n",
" Downloading pykoa-1.7.2-py3-none-any.whl (29 kB)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from pykoa) (2.31.0)\n",
"Collecting xmltodict (from pykoa)\n",
" Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
"Collecting bs4 (from pykoa)\n",
" Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from pykoa) (4.9.3)\n",
"Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4->pykoa) (4.11.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->pykoa) (2023.7.22)\n",
"Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4->pykoa) (2.4.1)\n",
"Building wheels for collected packages: bs4\n",
" Building wheel for bs4 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1256 sha256=a421c36805079047d1c68bc8e0031a9f39983ddb28be39ca31e032194e1526b6\n",
" Stored in directory: /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
"Successfully built bs4\n",
"Installing collected packages: xmltodict, bs4, pykoa\n",
"Successfully installed bs4-0.0.1 pykoa-1.7.2 xmltodict-0.13.0\n"
]
}
],
"source": [
"!pip install pykoa"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hhVJIWD0OKsS"
},
"source": [
"The next steop is to import pre-requisite packages"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "4G2fXjioOKBa"
},
"outputs": [],
"source": [
"import sys\n",
"import io\n",
"import os\n",
"from pykoa.koa import Koa\n",
"from astropy.table import Table,Column"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9jJgI5RBOOOX"
},
"source": [
"Then we need to assign directory that stores the query results:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "OB_GgjpsOXVk"
},
"outputs": [],
"source": [
"try:\n",
" os.mkdir('./output')\n",
"except:\n",
" print(\" Directory exists already\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q3-Uu0RiOWm9"
},
"source": [
"After that, we use a cookie for KOA log in. We can also spell out the credentials, but for security purpose, it is better to enter the userid and password interactively"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "olSx2E55OcgS",
"outputId": "e409f760-51f8-499d-a378-c2988200cec3"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Userid: clee\n",
"Password: ··········\n",
"Successfully login as clee\n"
]
}
],
"source": [
"Koa.login ('./tapcookie.txt')\n",
"#Koa.login ('./tapcookie.txt', userid='clee', password='salami')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kBQdEWm9OdOs"
},
"source": [
"We can then proceed to query HIRES data taken in a given period. Note the time period is specified in UT date. The output table will be saved in the designated directory and in IPAC table format."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VaeWQLWROveh",
"outputId": "9718ef12-0ab5-4bc4-84b7-74e67b8e9022"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"submitting request...\n",
"Result downloaded to file [./output/hires_daterange.tbl]\n",
" koaid ... filehand \n",
"---------------------- ... -----------------------------------------------------\n",
"HI.20210702.03381.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.03381.fits\n",
"HI.20210702.03701.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.03701.fits\n",
"HI.20210702.03826.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.03826.fits\n",
"HI.20210702.03903.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.03903.fits\n",
"HI.20210702.04038.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04038.fits\n",
"HI.20210702.04083.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04083.fits\n",
"HI.20210702.04129.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04129.fits\n",
"HI.20210702.04218.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04218.fits\n",
"HI.20210702.04314.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04314.fits\n",
"HI.20210702.04383.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.04383.fits\n",
" ... ... ...\n",
"HI.20210702.53089.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.53089.fits\n",
"HI.20210702.53773.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.53773.fits\n",
"HI.20210702.54074.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.54074.fits\n",
"HI.20210702.54353.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.54353.fits\n",
"HI.20210702.54494.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.54494.fits\n",
"HI.20210702.54661.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.54661.fits\n",
"HI.20210702.54937.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.54937.fits\n",
"HI.20210702.55100.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.55100.fits\n",
"HI.20210702.55151.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.55151.fits\n",
"HI.20210702.55235.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.55235.fits\n",
"HI.20210702.55314.fits ... /koadata13/HIRES/20210702/lev0/HI.20210702.55314.fits\n",
"Length = 113 rows\n"
]
}
],
"source": [
"Koa.query_datetime ('hires', \\\n",
" '2021-07-02 00:00:00/2021-07-02 23:59:59', \\\n",
" './output/hires_daterange.tbl', overwrite=True, format='ipac', \\\n",
" cookiepath='./tapcookie.txt')\n",
"\n",
"rec = Table.read ('./output/hires_daterange.tbl',format='ipac')\n",
"print (rec)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KV5BDsU4Osxd"
},
"source": [
"Once we have the query result, we can download files to the designated directory, in this case to dnload_dir_hires/. If the download function is called several times, it will check the designated directory and only download new files."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KxRXJeruPLET",
"outputId": "1e09c0f1-67ea-484e-aa38-d1b45d5bef43"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Start downloading 3 koaid data you requested;\n",
"please check your outdir: dnload_dir_hires for progress ....\n",
"\n",
"A total of 3 new lev0 FITS files downloaded.\n"
]
}
],
"source": [
"Koa.download ('./output/hires_daterange.tbl',\n",
" 'ipac', \\\n",
" 'dnload_dir_hires', \\\n",
"# start_row=0, \\\n",
"# end_row = 2, \\\n",
" cookiepath='./tapcookie.txt')"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}