{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Python API Example - Access Slots Import and Storage in Dataframe\n", "\n", "This guide is designed to provide an example of how to access the Spark API:\n", "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n", "- This script has been designed to display the raw outputs of requests from the API\n", "- This script can be copied and pasted by customers for quick use of the API\n", "\n", "__N.B. This guide is just for Access terminal slots data. If you're looking for other API data products (such as contract prices, Freight routes or Netbacks), please refer to their according code example files.__ " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Have any questions?\n", "\n", "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n", "\n", "__data@sparkcommodities.com__\n", "\n", "or refer to our API website for more information about this endpoint: https://www.sparkcommodities.com/api/request/access.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Importing Data\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "This section can remain unchanged for most Spark API users." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# import libraries for importing data\n", "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "from base64 import b64encode\n", "from urllib.parse import urljoin\n", "from pprint import pprint\n", "import requests\n", "from io import StringIO\n", "import time\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# defining functions \n", "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " print(\n", " \">>>> Client_id={}****, client_secret={}****\".format(\n", " client_id[:5], client_secret[:5]\n", " )\n", " )\n", "\n", " return client_id, client_secret\n", "\n", "def do_api_post_query(uri, body, headers):\n", " \"\"\"\n", " OAuth2 authentication requires a POST request with client credentials before accessing the API. \n", " This POST request will return an Access Token which will be used for the API GET request.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "def get_access_token(client_id, client_secret):\n", " \"\"\"\n", " Get a new access_token. Access tokens are the thing that applications use to make\n", " API requests. Access tokens must be kept confidential in storage.\n", "\n", " # Procedure:\n", "\n", " Do a POST query with `grantType` and `scopes` in the body. A basic authorization\n", " HTTP header is required. The \"Basic\" HTTP authentication scheme is defined in\n", " RFC 7617, which transmits credentials as `clientId:clientSecret` pairs, encoded\n", " using base64.\n", " \"\"\"\n", "\n", " # Note: for the sake of this example, we choose to use the Python urllib from the\n", " # standard lib. One should consider using https://requests.readthedocs.io/\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " \"scopes\": \"read:access\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already. Instructions for downloading your credentials can be found here:\n", "\n", "https://api.sparkcommodities.com/redoc#section/Authentication/Create-an-Oauth2-Client\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">>>> Found credentials!\n", ">>>> Client_id=875f4****, client_secret=6cdf8****\n", ">>>> Successfully fetched an access token eyJhb****, valid 604799 seconds.\n" ] } ], "source": [ "# Insert file path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Latest Slot Release\n", "\n", "Here we call the latest slot release and print it in a readable format. This is done using the URL:\n", "\n", "__/beta/terminal-slots/releases/latest/__\n", "\n", "\n", "We then save the entire dataset as a local variable called `latest`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Function to get the latest slot release\n", "def get_latest_slots():\n", " uri = urljoin(API_BASE_URL,'/beta/terminal-slots/releases/latest/')\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"accept\": \"text/csv\"\n", " }\n", " response = requests.get(uri, headers=headers)\n", " if response.status_code == 200:\n", " df = response.content.decode('utf-8')\n", " df = pd.read_csv(StringIO(df))\n", " else:\n", " print('Bad Request')\n", " return df" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ReleaseDateTerminalCodeTerminalNameTotalM+0M+1M+2M+3M+4M+5...M+4StartDateM+5StartDateM+6StartDateM+7StartDateM+8StartDateM+9StartDateM+10StartDateM+11StartDateM+12StartDateM>12StartDate
02025-01-14adriaticAdriatic0000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
12025-01-14brunsbuttelBrunsbuttel0000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
22025-01-14deutsche-ostseeDeutsche Ostsee59256643...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
32025-01-14dragonDragon0000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
42025-01-14dunkerqueDunkerque1100000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " ReleaseDate TerminalCode TerminalName Total M+0 M+1 M+2 M+3 \\\n", "0 2025-01-14 adriatic Adriatic 0 0 0 0 0 \n", "1 2025-01-14 brunsbuttel Brunsbuttel 0 0 0 0 0 \n", "2 2025-01-14 deutsche-ostsee Deutsche Ostsee 59 2 5 6 6 \n", "3 2025-01-14 dragon Dragon 0 0 0 0 0 \n", "4 2025-01-14 dunkerque Dunkerque 1 1 0 0 0 \n", "\n", " M+4 M+5 ... M+4StartDate M+5StartDate M+6StartDate M+7StartDate \\\n", "0 0 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "1 0 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "2 4 3 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "3 0 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "4 0 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "\n", " M+8StartDate M+9StartDate M+10StartDate M+11StartDate M+12StartDate \\\n", "0 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "1 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "2 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "3 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "4 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "\n", " M>12StartDate \n", "0 2026-02-01 \n", "1 2026-02-01 \n", "2 2026-02-01 \n", "3 2026-02-01 \n", "4 2026-02-01 \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Call latest slots function\n", "latest = get_latest_slots()\n", "latest.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Slot Release by Release Date\n", "\n", "Here we call the slot data by choosing a specific date and print it in a readable format. This is done using the URL:\n", "\n", "__/beta/terminal-slots/releases/{date}/__ where __date__ is the release date, in the \"YYYY-MM-DD\" string format.\n", "\n", "\n", "We then save the entire dataset as a local variable called `release_df`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Function to get the slot releases for a specific date\n", "def get_slot_releases(date):\n", " uri = urljoin(API_BASE_URL, f'/beta/terminal-slots/releases/{date}/')\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"accept\": \"text/csv\"\n", " }\n", "\n", " response = requests.get(uri, headers=headers)\n", "\n", " if response.status_code == 200:\n", " df = response.content.decode('utf-8')\n", " df = pd.read_csv(StringIO(df))\n", " return df\n", "\n", " elif response.content == b'{\"errors\":[{\"code\":\"object_not_found\",\"detail\":\"Object not found\"}]}':\n", " print('Bad Date')\n", " return None\n", " \n", " else:\n", " print('Bad Request')\n", " return None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### N.B. Plan Limits\n", "\n", "__Premium__ Users can choose any release date, as they have full access to the dataset.\n", "\n", "__Trial__ Users must choose a release date within the last 2 weeks." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ReleaseDateTerminalCodeTerminalNameTotalM+0M+1M+2M+3M+4M+5...M+4StartDateM+5StartDateM+6StartDateM+7StartDateM+8StartDateM+9StartDateM+10StartDateM+11StartDateM+12StartDateM>12StartDate
02024-10-22adriaticAdriatic0000000...2025-02-012025-03-012025-04-012025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-01
12024-10-22brunsbuttelBrunsbuttel0000000...2025-02-012025-03-012025-04-012025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-01
22024-10-22deutsche-ostseeDeutsche Ostsee20134111...2025-02-012025-03-012025-04-012025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-01
32024-10-22dragonDragon3120000...2025-02-012025-03-012025-04-012025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-01
42024-10-22eems-energy-terminalEemsEnergyTerminal7160000...2025-02-012025-03-012025-04-012025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-01
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " ReleaseDate TerminalCode TerminalName Total M+0 M+1 M+2 \\\n", "0 2024-10-22 adriatic Adriatic 0 0 0 0 \n", "1 2024-10-22 brunsbuttel Brunsbuttel 0 0 0 0 \n", "2 2024-10-22 deutsche-ostsee Deutsche Ostsee 20 1 3 4 \n", "3 2024-10-22 dragon Dragon 3 1 2 0 \n", "4 2024-10-22 eems-energy-terminal EemsEnergyTerminal 7 1 6 0 \n", "\n", " M+3 M+4 M+5 ... M+4StartDate M+5StartDate M+6StartDate M+7StartDate \\\n", "0 0 0 0 ... 2025-02-01 2025-03-01 2025-04-01 2025-05-01 \n", "1 0 0 0 ... 2025-02-01 2025-03-01 2025-04-01 2025-05-01 \n", "2 1 1 1 ... 2025-02-01 2025-03-01 2025-04-01 2025-05-01 \n", "3 0 0 0 ... 2025-02-01 2025-03-01 2025-04-01 2025-05-01 \n", "4 0 0 0 ... 2025-02-01 2025-03-01 2025-04-01 2025-05-01 \n", "\n", " M+8StartDate M+9StartDate M+10StartDate M+11StartDate M+12StartDate \\\n", "0 2025-06-01 2025-07-01 2025-08-01 2025-09-01 2025-10-01 \n", "1 2025-06-01 2025-07-01 2025-08-01 2025-09-01 2025-10-01 \n", "2 2025-06-01 2025-07-01 2025-08-01 2025-09-01 2025-10-01 \n", "3 2025-06-01 2025-07-01 2025-08-01 2025-09-01 2025-10-01 \n", "4 2025-06-01 2025-07-01 2025-08-01 2025-09-01 2025-10-01 \n", "\n", " M>12StartDate \n", "0 2025-11-01 \n", "1 2025-11-01 \n", "2 2025-11-01 \n", "3 2025-11-01 \n", "4 2025-11-01 \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calling slot release function\n", "release_df = get_slot_releases(\"2024-10-22\")\n", "release_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Terminal List\n", "\n", "Here we call the list of terminals and their uuid, and print it in a readable format. This is done using the URL:\n", "\n", "__'beta/terminal-slots/terminals/'__\n", "\n", "\n", "We then save the entire dataset as a local variable called `terminal_list`." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# Function to get the list of terminals and their uuids (as well as their start and latest release date)\n", "def get_terminal_list():\n", " uri = urljoin(API_BASE_URL,'beta/terminal-slots/terminals/')\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"accept\": \"text/csv\"\n", " }\n", " response = requests.get(uri, headers=headers)\n", " if response.status_code == 200:\n", " df = response.content.decode('utf-8')\n", " df = pd.read_csv(StringIO(df))\n", " else:\n", " print('Bad Request')\n", " return df" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TerminalUUIDTerminalCodeTerminalNamefirstAvailableReleaselatestAvailableRelease
000317185-978a-4df5-970c-2c28d3ab893cgrain-lngIsle of Grain2023-11-282025-01-14
10031994e-f370-4927-ba88-a4e7a78c42dbzeebruggeZeebrugge2023-11-282025-01-14
200338f3f-8875-435d-87a9-f83d9a5c5241dunkerqueDunkerque2024-11-152025-01-14
300344dd2-5608-4413-a2f4-c52c747a286adragonDragon2023-11-282025-01-14
4003497c6-ed32-412f-95ef-c3b1f962464ebrunsbuttelBrunsbuttel2024-05-282025-01-14
\n", "
" ], "text/plain": [ " TerminalUUID TerminalCode TerminalName \\\n", "0 00317185-978a-4df5-970c-2c28d3ab893c grain-lng Isle of Grain \n", "1 0031994e-f370-4927-ba88-a4e7a78c42db zeebrugge Zeebrugge \n", "2 00338f3f-8875-435d-87a9-f83d9a5c5241 dunkerque Dunkerque \n", "3 00344dd2-5608-4413-a2f4-c52c747a286a dragon Dragon \n", "4 003497c6-ed32-412f-95ef-c3b1f962464e brunsbuttel Brunsbuttel \n", "\n", " firstAvailableRelease latestAvailableRelease \n", "0 2023-11-28 2025-01-14 \n", "1 2023-11-28 2025-01-14 \n", "2 2024-11-15 2025-01-14 \n", "3 2023-11-28 2025-01-14 \n", "4 2024-05-28 2025-01-14 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Call terminal list function\n", "terminal_list = get_terminal_list()\n", "terminal_list.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fetching Slots Data specific to one terminal\n", "\n", "Now that we can see all the terminal data available to us, we can start to define what terminal we want to call slots data for (by referring to `terminal_list` above).\n", "\n", "The first step is to choose which terminal uuid (`my_uuid`), then the request will return all the historical data available for that terminal." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# Function to collect and store historical slots for one specific terminal\n", "def get_individual_terminal(terminal_uuid):\n", " uri = urljoin(API_BASE_URL, f'/beta/terminal-slots/terminals/{terminal_uuid}/')\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"accept\": \"text/csv\"\n", " }\n", " response = requests.get(uri, headers=headers)\n", " if response.status_code == 200:\n", " df = response.content.decode('utf-8')\n", " df = pd.read_csv(StringIO(df))\n", " return df\n", "\n", " elif response.content == b'{\"errors\":[{\"code\":\"object_not_found\",\"detail\":\"Object not found\"}]}':\n", " print('Bad Terminal Request')\n", " return None\n", " else:\n", " print('Bad Request')\n", " return None" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TerminalCodeTerminalNameReleaseDateTotalM+0M+1M+2M+3M+4M+5...M+4StartDateM+5StartDateM+6StartDateM+7StartDateM+8StartDateM+9StartDateM+10StartDateM+11StartDateM+12StartDateM>12StartDate
0dragonDragon2023-11-282020000...2024-03-012024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-01
1dragonDragon2023-11-292020000...2024-03-012024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-01
2dragonDragon2023-11-302200000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
3dragonDragon2023-12-012200000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
4dragonDragon2023-12-041100000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
..................................................................
280dragonDragon2025-01-080000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
281dragonDragon2025-01-090000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
282dragonDragon2025-01-100000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
283dragonDragon2025-01-130000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
284dragonDragon2025-01-140000000...2025-05-012025-06-012025-07-012025-08-012025-09-012025-10-012025-11-012025-12-012026-01-012026-02-01
\n", "

285 rows × 32 columns

\n", "
" ], "text/plain": [ " TerminalCode TerminalName ReleaseDate Total M+0 M+1 M+2 M+3 M+4 \\\n", "0 dragon Dragon 2023-11-28 2 0 2 0 0 0 \n", "1 dragon Dragon 2023-11-29 2 0 2 0 0 0 \n", "2 dragon Dragon 2023-11-30 2 2 0 0 0 0 \n", "3 dragon Dragon 2023-12-01 2 2 0 0 0 0 \n", "4 dragon Dragon 2023-12-04 1 1 0 0 0 0 \n", ".. ... ... ... ... ... ... ... ... ... \n", "280 dragon Dragon 2025-01-08 0 0 0 0 0 0 \n", "281 dragon Dragon 2025-01-09 0 0 0 0 0 0 \n", "282 dragon Dragon 2025-01-10 0 0 0 0 0 0 \n", "283 dragon Dragon 2025-01-13 0 0 0 0 0 0 \n", "284 dragon Dragon 2025-01-14 0 0 0 0 0 0 \n", "\n", " M+5 ... M+4StartDate M+5StartDate M+6StartDate M+7StartDate \\\n", "0 0 ... 2024-03-01 2024-04-01 2024-05-01 2024-06-01 \n", "1 0 ... 2024-03-01 2024-04-01 2024-05-01 2024-06-01 \n", "2 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", "3 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", "4 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", ".. ... ... ... ... ... ... \n", "280 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "281 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "282 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "283 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "284 0 ... 2025-05-01 2025-06-01 2025-07-01 2025-08-01 \n", "\n", " M+8StartDate M+9StartDate M+10StartDate M+11StartDate M+12StartDate \\\n", "0 2024-07-01 2024-08-01 2024-09-01 2024-10-01 2024-11-01 \n", "1 2024-07-01 2024-08-01 2024-09-01 2024-10-01 2024-11-01 \n", "2 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", "3 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", "4 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", ".. ... ... ... ... ... \n", "280 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "281 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "282 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "283 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "284 2025-09-01 2025-10-01 2025-11-01 2025-12-01 2026-01-01 \n", "\n", " M>12StartDate \n", "0 2024-12-01 \n", "1 2024-12-01 \n", "2 2025-01-01 \n", "3 2025-01-01 \n", "4 2025-01-01 \n", ".. ... \n", "280 2026-02-01 \n", "281 2026-02-01 \n", "282 2026-02-01 \n", "283 2026-02-01 \n", "284 2026-02-01 \n", "\n", "[285 rows x 32 columns]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Call individual terminal function (Dragon)\n", "my_uuid = terminal_list[terminal_list['TerminalName']=='Dragon']['TerminalUUID'].tolist()[0]\n", "dragon_hist_df = get_individual_terminal(my_uuid)\n", "dragon_hist_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Historical Slots\n", "\n", "Here we collect all the historical slots for each available terminal. This is done using the URL:\n", "\n", "__/beta/terminal-slots/terminals/{terminal_uuid}/__\n", "\n", "First we define the function that imports the data from the Spark API.\n", "\n", "We then call that function, and define the list of terminal uuids.\n", "\n", "\n", "We save the output as a local variable called `all_terminals_historical`.\n", "\n", "### N.B. Plan Limits\n", "\n", "__Premium__ Users have full access to the dataset.\n", "\n", "__Trial__ Users only have access the latest 2 weeks' worth of datapoints." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "# Function to collect and store each terminal's historical slots data\n", "def get_all_terminal_data(terminal_list):\n", " terminals_all = pd.DataFrame()\n", " for i in range(len(terminal_list)):\n", " print(terminal_list['TerminalName'].loc[i])\n", " terminal_df = get_individual_terminal(terminal_list['TerminalUUID'].loc[i])\n", " time.sleep(0.1)\n", " terminals_all = pd.concat([terminals_all,terminal_df])\n", " return terminals_all" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Isle of Grain\n", "Zeebrugge\n", "Dunkerque\n", "Dragon\n", "Brunsbuttel\n", "OLT Toscana\n", "Le Havre\n", "Piombino\n", "EemsEnergyTerminal\n", "Adriatic\n", "Fos Cavaou\n", "Montoir\n", "Wilhelmshaven 1\n", "South Hook\n", "Inkoo\n", "Deutsche Ostsee\n", "Klaipeda\n", "Gate\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TerminalCodeTerminalNameReleaseDateTotalM+0M+1M+2M+3M+4M+5...M+4StartDateM+5StartDateM+6StartDateM+7StartDateM+8StartDateM+9StartDateM+10StartDateM+11StartDateM+12StartDateM>12StartDate
0grain-lngIsle of Grain2023-11-281010000...2024-03-012024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-01
1grain-lngIsle of Grain2023-11-291010000...2024-03-012024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-01
2grain-lngIsle of Grain2023-11-302200000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
3grain-lngIsle of Grain2023-12-012200000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
4grain-lngIsle of Grain2023-12-042200000...2024-04-012024-05-012024-06-012024-07-012024-08-012024-09-012024-10-012024-11-012024-12-012025-01-01
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " TerminalCode TerminalName ReleaseDate Total M+0 M+1 M+2 M+3 M+4 \\\n", "0 grain-lng Isle of Grain 2023-11-28 1 0 1 0 0 0 \n", "1 grain-lng Isle of Grain 2023-11-29 1 0 1 0 0 0 \n", "2 grain-lng Isle of Grain 2023-11-30 2 2 0 0 0 0 \n", "3 grain-lng Isle of Grain 2023-12-01 2 2 0 0 0 0 \n", "4 grain-lng Isle of Grain 2023-12-04 2 2 0 0 0 0 \n", "\n", " M+5 ... M+4StartDate M+5StartDate M+6StartDate M+7StartDate \\\n", "0 0 ... 2024-03-01 2024-04-01 2024-05-01 2024-06-01 \n", "1 0 ... 2024-03-01 2024-04-01 2024-05-01 2024-06-01 \n", "2 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", "3 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", "4 0 ... 2024-04-01 2024-05-01 2024-06-01 2024-07-01 \n", "\n", " M+8StartDate M+9StartDate M+10StartDate M+11StartDate M+12StartDate \\\n", "0 2024-07-01 2024-08-01 2024-09-01 2024-10-01 2024-11-01 \n", "1 2024-07-01 2024-08-01 2024-09-01 2024-10-01 2024-11-01 \n", "2 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", "3 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", "4 2024-08-01 2024-09-01 2024-10-01 2024-11-01 2024-12-01 \n", "\n", " M>12StartDate \n", "0 2024-12-01 \n", "1 2024-12-01 \n", "2 2025-01-01 \n", "3 2025-01-01 \n", "4 2025-01-01 \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calling all terminal data function\n", "all_terminal_historical = get_all_terminal_data(terminal_list)\n", "all_terminal_historical.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Analytics Gallery\n", "Want to gain market insights using our data?\n", "\n", "Take a look at our [Analytics Gallery](https://www.sparkcommodities.com/api/code-examples/analytics-examples.html) on the Spark API website, which includes\n", "\n", "- __Regas Slots vs US Arb__ - See how the US Arb (M+1, M+2, M+3) correlates with monthly average slots for all European Terminals.\n", "__Note: Requires Cargo and Access subscriptions__\n", "\n", "\n", "Take a look at our Regas Slots vs US Arb chart [here](https://www.sparkcommodities.com/api/code-examples/analytics-examples.html). " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 2 }