{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59d19f73",
   "metadata": {},
   "source": [
    "# Python API Example - Netbacks Data Import and Storage in Dataframe\n",
    "\n",
    "This guide is designed to provide an example of how to access the Spark API:\n",
    "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n",
    "- This script has been designed to display the raw outputs of requests from the API, and how to format those outputs to enable easy reading and analysis\n",
    "- This script can be copy and pasted by customers for quick use of the API\n",
    "- Once comfortable with the process, you can change the variables that are called to produce your own custom analysis products. (Section 2 onwards in this guide).\n",
    "\n",
    "__N.B. This guide is just for Netbacks data. If you're looking for other API data products (such as Price releases or Freight routes), please refer to their according code example files.__ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0a05be4",
   "metadata": {},
   "source": [
    "### Have any questions?\n",
    "\n",
    "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n",
    "\n",
    "__data@sparkcommodities.com__\n",
    "\n",
    "or refer to our API website for more information about this endpoint:\n",
    "https://www.sparkcommodities.com/api/lng-cargo/netbacks.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e00ae34",
   "metadata": {},
   "source": [
    "## 1. Importing Data\n",
    "\n",
    "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n",
    "\n",
    "__This section can remain unchanged for most Spark API users.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1161e807",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import sys\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from base64 import b64encode\n",
    "from pprint import pprint\n",
    "from urllib.parse import urljoin\n",
    "import datetime\n",
    "from io import StringIO\n",
    "\n",
    "\n",
    "try:\n",
    "    from urllib import request, parse\n",
    "    from urllib.error import HTTPError\n",
    "except ImportError:\n",
    "    raise RuntimeError(\"Python 3 required\")\n",
    "\n",
    "\n",
    "API_BASE_URL = \"https://api.sparkcommodities.com\"\n",
    "\n",
    "\n",
    "def retrieve_credentials(file_path=None):\n",
    "    \"\"\"\n",
    "    Find credentials either by reading the client_credentials file or reading\n",
    "    environment variables\n",
    "    \"\"\"\n",
    "    if file_path is None:\n",
    "        client_id = os.getenv(\"SPARK_CLIENT_ID\")\n",
    "        client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n",
    "        if not client_id or not client_secret:\n",
    "            raise RuntimeError(\n",
    "                \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n",
    "            )\n",
    "    else:\n",
    "        # Parse the file\n",
    "        if not os.path.isfile(file_path):\n",
    "            raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n",
    "\n",
    "        with open(file_path) as fp:\n",
    "            lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n",
    "\n",
    "        if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n",
    "            client_id, client_secret = lines[1].split(\",\")\n",
    "        else:\n",
    "            print(\"First line read: '{}'\".format(lines[0]))\n",
    "            raise RuntimeError(\n",
    "                \"The specified file {} doesn't look like to be a Spark API client \"\n",
    "                \"credentials file\".format(file_path)\n",
    "            )\n",
    "\n",
    "    print(\">>>> Found credentials!\")\n",
    "    print(\n",
    "        \">>>> Client_id={}, client_secret={}****\".format(client_id, client_secret[:5])\n",
    "    )\n",
    "\n",
    "    return client_id, client_secret\n",
    "\n",
    "\n",
    "def do_api_post_query(uri, body, headers):\n",
    "    url = urljoin(API_BASE_URL, uri)\n",
    "\n",
    "    data = json.dumps(body).encode(\"utf-8\")\n",
    "\n",
    "    # HTTP POST request\n",
    "    req = request.Request(url, data=data, headers=headers)\n",
    "    try:\n",
    "        response = request.urlopen(req)\n",
    "    except HTTPError as e:\n",
    "        print(\"HTTP Error: \", e.code)\n",
    "        print(e.read())\n",
    "        sys.exit(1)\n",
    "\n",
    "    resp_content = response.read()\n",
    "\n",
    "    # The server must return HTTP 201. Raise an error if this is not the case\n",
    "    assert response.status == 201, resp_content\n",
    "\n",
    "    # The server returned a JSON response\n",
    "    content = json.loads(resp_content)\n",
    "\n",
    "    return content\n",
    "\n",
    "\n",
    "def do_api_get_query(uri, access_token, format='json'):\n",
    "    \"\"\"\n",
    "    After receiving an Access Token, we can request information from the API.\n",
    "    \"\"\"\n",
    "    url = urljoin(API_BASE_URL, uri)\n",
    "\n",
    "    if format == 'json':\n",
    "        headers = {\n",
    "            \"Authorization\": \"Bearer {}\".format(access_token),\n",
    "            \"Accept\": \"application/json\",\n",
    "        }\n",
    "    elif format == 'csv':\n",
    "        headers = {\n",
    "            \"Authorization\": \"Bearer {}\".format(access_token),\n",
    "            \"Accept\": \"text/csv\"\n",
    "        }\n",
    "    else:\n",
    "        raise ValueError('The format parameter only takes `csv` or `json` as inputs')\n",
    "\n",
    "    # HTTP GET request\n",
    "    req = request.Request(url, headers=headers)\n",
    "    try:\n",
    "        response = request.urlopen(req)\n",
    "    except HTTPError as e:\n",
    "        print(\"HTTP Error: \", e.code)\n",
    "        print(e.read())\n",
    "        sys.exit(1)\n",
    "\n",
    "    resp_content = response.read()\n",
    "    #status = response.status\n",
    "\n",
    "    # The server must return HTTP 200. Raise an error if this is not the case\n",
    "    assert response.status == 200, resp_content\n",
    "\n",
    "    # Storing response based on requested format\n",
    "    if format == 'json':\n",
    "        content = json.loads(resp_content)\n",
    "    elif format == 'csv':\n",
    "        content = resp_content\n",
    "\n",
    "    return content\n",
    "\n",
    "\n",
    "def get_access_token(client_id, client_secret):\n",
    "    \"\"\"\n",
    "    Get a new access_token. Access tokens are the thing that applications use to make\n",
    "    API requests. Access tokens must be kept confidential in storage.\n",
    "\n",
    "    # Procedure:\n",
    "\n",
    "    Do a POST query with `grantType` in the body. A basic authorization\n",
    "    HTTP header is required. The \"Basic\" HTTP authentication scheme is defined in\n",
    "    RFC 7617, which transmits credentials as `clientId:clientSecret` pairs, encoded\n",
    "    using base64.\n",
    "    \"\"\"\n",
    "\n",
    "    # Note: for the sake of this example, we choose to use the Python urllib from the\n",
    "    # standard lib. One should consider using https://requests.readthedocs.io/\n",
    "\n",
    "    payload = \"{}:{}\".format(client_id, client_secret).encode()\n",
    "    headers = {\n",
    "        \"Authorization\": \"Basic {}\".format(b64encode(payload).decode()),\n",
    "        \"Accept\": \"application/json\",\n",
    "        \"Content-Type\": \"application/json\",\n",
    "    }\n",
    "    body = {\n",
    "        \"grantType\": \"clientCredentials\",\n",
    "    }\n",
    "\n",
    "    content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n",
    "\n",
    "    print(\n",
    "        \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n",
    "            content[\"accessToken\"][:5], content[\"expiresIn\"]\n",
    "        )\n",
    "    )\n",
    "\n",
    "    return content[\"accessToken\"]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f40a040b",
   "metadata": {},
   "source": [
    "## N.B. Credentials\n",
    "\n",
    "Here we call the above functions, and input the file path to our credentials.\n",
    "\n",
    "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already.  Instructions for downloading your credentials can be found here:\n",
    "\n",
    "https://api.sparkcommodities.com/redoc#section/Authentication/Create-an-Oauth2-Client\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeb3bd88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input the path to your client credentials here\n",
    "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n",
    "\n",
    "# Authenticate:\n",
    "access_token = get_access_token(client_id, client_secret)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c527e40",
   "metadata": {},
   "source": [
    "## Reference Date endpoint\n",
    "\n",
    "This query shows an overview on all available netbacks, showing all available ports and possible routes to/from these destinations (i.e. via Suez, Panama etc.).\n",
    "\n",
    "An example of pulling all available Netbacks data for a specific route (e.g. Netbacks data for Sabine Pass via Suez) is shown later in this script. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ada4f167",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the function for listing all netbacks\n",
    "def list_netbacks(access_token):\n",
    "\n",
    "    content = do_api_get_query(\n",
    "        uri=\"/v1.0/netbacks/reference-data/\", access_token=access_token\n",
    "    )\n",
    "\n",
    "    # retrieve release dates separately\n",
    "    reldates = content[\"data\"][\"staticData\"][\"sparkReleases\"]\n",
    "\n",
    "    # return reference data\n",
    "    refdata_dict = content[\"data\"]\n",
    "\n",
    "    return reldates, refdata_dict\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4003cf3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch reference data:\n",
    "reldates, refdata_dict = list_netbacks(access_token)\n",
    "\n",
    "# Shows the structure of the raw dictionary called\n",
    "refdata_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc9904e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# formatting as a Dataframe\n",
    "available_df = pd.json_normalize(refdata_dict['staticData']['fobPorts'])\n",
    "available_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e447d6b2",
   "metadata": {},
   "source": [
    "## Fetching Netbacks Data specific to one port\n",
    "\n",
    "Input your required Netbacks parameters. Refer to the reference data above (\"available_df\") for available port UUIDs & via points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4480909",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetching port & via point inputs\n",
    "port = \"Sabine Pass\"\n",
    "my_ticker = available_df[available_df[\"name\"] == port][\"uuid\"].iloc[0]\n",
    "my_via = 'cogh'\n",
    "\n",
    "# choosing a date for the \"release-date\" parameter\n",
    "my_release = reldates[0]\n",
    "\n",
    "print(my_ticker)\n",
    "print(my_release)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1bf4c0",
   "metadata": {},
   "source": [
    "## Defining data calling function\n",
    "\n",
    "Details on the available parameters can be found on our API docs website:\n",
    "\n",
    "https://www.sparkcommodities.com/api/lng-cargo/netbacks.html\n",
    "\n",
    "__N.B.:__ This endpoint provides the option to return a JSON or CSV formatted response. Metadata is only available in the JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb563eb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import StringIO\n",
    "\n",
    "def fetch_netback(access_token, ticker, release=None, via=None, laden=None, ballast=None, start=None, end=None, percent_hire=None, format='json'):\n",
    "    \n",
    "    query_params = \"?fob-port={}\".format(ticker)\n",
    "    if release is not None:\n",
    "        query_params += \"&release-date={}\".format(release)\n",
    "    if start is not None:\n",
    "        query_params += \"&start={}\".format(start)\n",
    "    if end is not None:\n",
    "        query_params += \"&end={}\".format(end)\n",
    "    if via is not None:\n",
    "        query_params += \"&via-point={}\".format(via)\n",
    "    if laden is not None:\n",
    "        query_params += \"&laden-congestion-days={}\".format(laden)\n",
    "    if ballast is not None:\n",
    "        query_params += \"&ballast-congestion-days={}\".format(ballast)\n",
    "    if percent_hire is not None:\n",
    "        query_params += \"&percent-hire={}\".format(percent_hire)\n",
    "    \n",
    "    \n",
    "    content = do_api_get_query(\n",
    "        uri=\"/v1.0/netbacks/{}\".format(query_params),\n",
    "        access_token=access_token, format=format,\n",
    "    )\n",
    "    \n",
    "    if format == 'json':\n",
    "        data = content\n",
    "    elif format == 'csv':\n",
    "        # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n",
    "        if len(content) == 0:\n",
    "            data = content\n",
    "            print('No Data to Show')\n",
    "        else:\n",
    "            data = content.decode('utf-8')\n",
    "            data = pd.read_csv(StringIO(data))  # automatically converting into a Pandas DataFrame when choosing CSV format\n",
    "\n",
    "    return data\n",
    "\n",
    "\n",
    "## Calling data in JSON format\n",
    "\n",
    "json_data = fetch_netback(access_token, my_ticker, release=my_release, via=my_via, format='json')\n",
    "json_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16fb96d1",
   "metadata": {},
   "source": [
    "### CSV Response\n",
    "\n",
    "Calling the data in CSV format & formatting as a Pandas DataFrame. Here, we also utilise the \"start\" and \"end\" parameters to call a range of historical data in one call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cde48ff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_csv = fetch_netback(access_token, start='2025-01-01', end='2026-01-28', ticker=my_ticker, via=my_via, format='csv')\n",
    "df_csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50637607",
   "metadata": {},
   "source": [
    "# Analytics Gallery\n",
    "Want to gain market insights using our data?\n",
    "\n",
    "Take a look at our [Analytics Gallery](https://www.sparkcommodities.com/api/code-examples/analytics-examples.html) on the Spark API website, which includes:\n",
    "\n",
    "- __US Arb Monthly Tracker__ - Analyse the variations of the US Arb via COGH across multiple years.\n",
    "\n",
    "- __US Front Month Historical Arb__ - Explore how the historical US front month Arb has evolved over time and compare them across different via points (Panama, Suez, COGH).\n",
    "\n",
    "Take a look at all our Analytics charts [here](https://www.sparkcommodities.com/api/code-examples/analytics-examples.html). \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}