{ "cells": [ { "cell_type": "markdown", "id": "92475c69", "metadata": {}, "source": [ "# Python API Example - Intraday Arbs\n", "\n", "Here we import Live, Historical and Revised prices from the Spark Intraday Arbs Python API. \n", "\n", "__N.B. This guide is just for Intraday Arbs data. If you're looking for other API data products (such as Freight routes or Netbacks), please refer to their according code samples.__ \n", "\n", "### Have any questions?\n", "\n", "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n", "\n", "__data@sparkcommodities.com__\n", "\n", "or refer to our API website for more information about this endpoint:\n", "https://www.sparkcommodities.com/api/" ] }, { "cell_type": "markdown", "id": "c5716130", "metadata": {}, "source": [ "## 1. Importing Data\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "This section can remain unchanged for most Spark API users." ] }, { "cell_type": "code", "execution_count": null, "id": "c9a04e15", "metadata": {}, "outputs": [], "source": [ "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "import numpy as np\n", "from base64 import b64encode\n", "from urllib.parse import urljoin\n", "from io import StringIO\n", "import datetime\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")" ] }, { "cell_type": "code", "execution_count": null, "id": "33fb0640", "metadata": {}, "outputs": [], "source": [ "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " \n", " return client_id, client_secret\n", "\n", "\n", "def do_api_post_query(uri, body, headers):\n", " \"\"\"\n", " OAuth2 authentication requires a POST request with client credentials before accessing the API.\n", " This POST request will return an Access Token which will be used for the API GET request.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def do_api_get_query(uri, access_token, format='json'):\n", " \"\"\"\n", " After receiving an Access Token, we can request information from the API.\n", " Supports both JSON (default) and CSV responses.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " if format == 'json':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"application/json\",\n", " }\n", " elif format == 'csv':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"text/csv\",\n", " }\n", " else:\n", " raise ValueError(\"The format parameter only takes 'csv' or 'json' as inputs\")\n", "\n", " print(\"Fetching {}\".format(url))\n", "\n", " # HTTP GET request\n", " req = request.Request(url, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 200. Raise an error if this is not the case\n", " assert response.status == 200, resp_content\n", "\n", " # Storing response based on requested format\n", " if format == 'json':\n", " content = json.loads(resp_content)\n", " elif format == 'csv':\n", " content = resp_content\n", "\n", " return content\n", "\n", "\n", "def get_access_token(client_id, client_secret):\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]" ] }, { "cell_type": "markdown", "id": "fd3171a8", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already.\n", "\n", "The code then prints the available prices that are callable from the API, and their corresponding Python ticker names are displayed as a list at the bottom of the Output." ] }, { "cell_type": "code", "execution_count": null, "id": "fd7e89bf", "metadata": {}, "outputs": [], "source": [ "# Insert file path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)" ] }, { "cell_type": "markdown", "id": "10bafcf1", "metadata": {}, "source": [ "# 2. Reference Data\n", "\n", "The Reference Data endpoint returns the UUID codes associated with each FoB Port available on the Arbs endpoint. \n", "\n", "If there are two arbs available for a single FoB Port, e.g. \"Sabine Pass via COGH\" or \"Sabine Pass via Panama\", these are included as separate entries in the Reference Data response. The FoB Port UUID remains the same between these two entries. " ] }, { "cell_type": "code", "execution_count": null, "id": "4d211cdb", "metadata": {}, "outputs": [], "source": [ "# define reference data function\n", "\n", "def fetch_reference_data(access_token, format='json'):\n", "\n", " uri=\"/v1.0/intraday/arbs/reference-data/\"\n", " print(uri)\n", " \n", " content = do_api_get_query(\n", " uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data\n", "\n", "\n", "# call data\n", "ref_df = fetch_reference_data(access_token, format='csv')\n", "ref_df" ] }, { "cell_type": "code", "execution_count": null, "id": "1389e400", "metadata": {}, "outputs": [], "source": [ "# Save your chosen FoB Port UUID as a variable\n", "my_fobport = ref_df[ref_df['FobPortName'] == 'Sabine Pass']['FobPortUuid'].iloc[0]\n", "\n", "print(my_fobport)" ] }, { "cell_type": "markdown", "id": "7138d5ee", "metadata": {}, "source": [ "# 3. Live Feed\n", "\n", "Here we define the function used to call the live price release for Spark Intraday Arbs. This endpoint takes 4 parameters - details on these parameters can be found on the API docs website:\n", "\n", "https://www.sparkcommodities.com/api/intraday/arbs.html\n", "\n", "Users can also choose which format to retrieve the data in, 'json' or 'csv'.\n", "\n", "__N.B__ Metadata is only available via the JSON format.\n", "\n", "__N.B__ This function will only run once and retrieve the latest forward curve. For a continuous feed of prices, you need to repeatedly call this function every 10 minutes for each new price release" ] }, { "cell_type": "code", "execution_count": null, "id": "b4186220", "metadata": {}, "outputs": [], "source": [ "## Defining the function\n", "\n", "\n", "def fetch_live_releases(access_token, ticker, via, percent_hire, unit, extended_curve, format='json'):\n", " query_params = \"?fob-port={}\".format(ticker)\n", " query_params += \"&via-point={}\".format(via)\n", " query_params += \"&unit={}\".format(unit)\n", " query_params += \"&percent-hire={}\".format(percent_hire)\n", " query_params += \"&extended-curve={}\".format(extended_curve)\n", "\n", " uri=\"/v1.0/intraday/arbs/live/{}\".format(query_params)\n", " print(uri)\n", " \n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data\n", "\n", "# calling the function to fetch the latest curves\n", "json_live = fetch_live_releases(access_token, ticker='003dec0a-ce8f-41db-8c24-4d7ef6addf70', via='cogh', \n", " percent_hire=100, unit='usd-per-mmbtu', extended_curve=False, format='json')\n", "json_live" ] }, { "cell_type": "markdown", "id": "f0798174", "metadata": {}, "source": [ "### Data included\n", "\n", "The JSON includes 4 fields:\n", "- 'errors': will be empty if the API call is successful, otherwise returns relevant error code\n", "- 'data': contains the full forward curve of the requested contract\n", "- 'revisions': contains any revisions made since the last price release - this can be for a revision for any historical price\n", "- 'metaData': provides details on the fetched data and time of API call" ] }, { "cell_type": "code", "execution_count": null, "id": "690b953b", "metadata": {}, "outputs": [], "source": [ "print(json_live.keys())\n", "print(json_live['metaData'])" ] }, { "cell_type": "markdown", "id": "06e78d2d", "metadata": {}, "source": [ "### CSV Format" ] }, { "cell_type": "code", "execution_count": null, "id": "f65eae7b", "metadata": {}, "outputs": [], "source": [ "csv_live = fetch_live_releases(access_token, ticker='003dec0a-ce8f-41db-8c24-4d7ef6addf70', via='cogh', \n", " percent_hire=100, unit='usd-per-mmbtu', extended_curve=False, format='csv')\n", "csv_live.head(5)" ] }, { "cell_type": "markdown", "id": "e27e05fb", "metadata": {}, "source": [ "# 4. Historical Data\n", "\n", "Here we define the function used to call the historical price release for Spark Intraday Arbs. This endpoint takes 6 parameters - details on these parameters can be found on the API docs website:\n", "\n", "https://www.sparkcommodities.com/api/intraday/arbs.html\n", "\n", "Users can also choose which format to retrieve the data in, 'json' or 'csv'.\n", "\n", "__N.B__ Metadata is only available via the JSON format." ] }, { "cell_type": "code", "execution_count": null, "id": "24adff55", "metadata": {}, "outputs": [], "source": [ "## Defining the function\n", "\n", "\n", "def fetch_historical_releases(access_token, ticker, via, percent_hire, unit, extended_curve, start, end, format='json'):\n", " query_params = \"?fob-port={}\".format(ticker)\n", " query_params += \"&via-point={}\".format(via)\n", " query_params += \"&unit={}\".format(unit)\n", " query_params += \"&percent-hire={}\".format(percent_hire)\n", " query_params += \"&extended-curve={}\".format(extended_curve)\n", "\n", " query_params += \"&start={}\".format(start)\n", " query_params += \"&end={}\".format(end)\n", "\n", " uri=\"/v1.0/intraday/arbs/historical/{}\".format(query_params)\n", " print(uri)\n", " \n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data\n", "\n", "# call function to fetch historical data\n", "\n", "hist_df = fetch_historical_releases(access_token, ticker='003dec0a-ce8f-41db-8c24-4d7ef6addf70', via='cogh', \n", " percent_hire=100, unit='usd-per-mmbtu', extended_curve=True,\n", " start='2025-10-01', end='2025-10-31', format='csv')\n", "\n", "hist_df.head(5)" ] }, { "cell_type": "code", "execution_count": null, "id": "22f3ae40", "metadata": {}, "outputs": [], "source": [ "# Group dataframe by \"Contract\" field\n", "contract_groups = hist_df.groupby('PeriodName')\n", "\n", "# list available contracts (not listed chronologically)\n", "contract_keys = list(contract_groups.groups.keys())\n", "print(contract_keys)" ] }, { "cell_type": "code", "execution_count": null, "id": "d7d19d51", "metadata": {}, "outputs": [], "source": [ "# Fetch historical prices for a specific contract\n", "cdf = contract_groups.get_group(contract_keys[0])\n", "cdf.head(10)" ] }, { "cell_type": "markdown", "id": "b3fa015e", "metadata": {}, "source": [ "## N.B. Historical Data Limits\n", "\n", "Currently, a maximum of 20 historical days can be called at one time due to the size of the data file. \n", "\n", "If more data points are required, the below code can be used, which splits the requested date range into looped calls of 20 historical days at a time." ] }, { "cell_type": "code", "execution_count": null, "id": "129599ed", "metadata": {}, "outputs": [], "source": [ "def loop_historical_data(access_token, ticker, via, percent_hire, unit, extended_curve, hist_start, hist_end):\n", " \n", " # calculating the amount of days to retrieve data for\n", " hist_diff = (datetime.datetime.strptime(hist_end, '%Y-%m-%d') - datetime.datetime.strptime(hist_start, '%Y-%m-%d')).days\n", " t = 0\n", "\n", " starts = []\n", " ends = []\n", " \n", " # iterating through the historical data, 20 days at a time\n", " while t < hist_diff:\n", " # initialising dataframe\n", " if t == 0 and hist_diff >= 21:\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(days=20), '%Y-%m-%d')\n", " hist_df = fetch_historical_releases(access_token, ticker=ticker, via=via, percent_hire=percent_hire, unit=unit, extended_curve=extended_curve,\n", " start=hist_start, end=diff_end, format='csv')\n", "\n", " starts.append(hist_start)\n", " ends.append(diff_end)\n", " \n", " # in case the date range is smaller than 20 days\n", " elif t==0 and hist_diff < 21:\n", " hist_df = fetch_historical_releases(access_token, ticker=ticker, via=via, percent_hire=percent_hire, unit=unit, extended_curve=extended_curve,\n", " start=hist_start, end=hist_end, format='csv')\n", "\n", " # looping through the rest of the dates\n", " else:\n", " if t < hist_diff-20:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(days=t), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=20), '%Y-%m-%d')\n", " hist2 = fetch_historical_releases(access_token, ticker=ticker, via=via, percent_hire=percent_hire, unit=unit, extended_curve=extended_curve, \n", " start=diff_start, end=diff_end, format='csv')\n", " hist_df = pd.concat([hist_df,hist2])\n", " starts.append(hist_start)\n", " ends.append(diff_end)\n", " # final loop\n", " else:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(days=t), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=hist_diff-t), '%Y-%m-%d')\n", " hist2 = fetch_historical_releases(access_token, ticker=ticker, via=via, percent_hire=percent_hire, unit=unit, extended_curve=extended_curve, \n", " start=diff_start, end=diff_end, format='csv')\n", " hist_df = pd.concat([hist_df,hist2])\n", " starts.append(hist_start)\n", " ends.append(diff_end)\n", " \n", " t += 20\n", "\n", " hist_df['Value'] = pd.to_numeric(hist_df['Value'])\n", " hist_df['AsOf'] = pd.to_datetime(hist_df['AsOf'], utc=True).dt.tz_localize(None)\n", "\n", " return hist_df" ] }, { "cell_type": "code", "execution_count": null, "id": "a3ac2ad6", "metadata": {}, "outputs": [], "source": [ "# Calling the extended historical data function\n", "\n", "\"\"\"\n", "hist_df = loop_historical_data(access_token, ticker='003dec0a-ce8f-41db-8c24-4d7ef6addf70', via='cogh', \n", " percent_hire=100, unit='usd-per-mmbtu', extended_curve=True,\n", " hist_start='2025-10-01', hist_end='2026-01-31')\n", "\n", "hist_df.head(5)\n", "\"\"\"" ] }, { "cell_type": "markdown", "id": "922a757e", "metadata": {}, "source": [ "# 5. Revisions\n", "\n", "Here we define the function used to call revisions in the last 24h for Spark Intraday Arbs. This endpoint takes 4 parameters - details on these parameters can be found on the API docs website:\n", "\n", "https://www.sparkcommodities.com/api/intraday/arbs.html\n", "\n", "Users can also choose which format to retrieve the data in, 'json' or 'csv'.\n", "\n", "__N.B__ Metadata is only available via the JSON format." ] }, { "cell_type": "code", "execution_count": null, "id": "054ccebe", "metadata": {}, "outputs": [], "source": [ "def fetch_historical_revisions(access_token, ticker, via, percent_hire, unit, extended_curve, format='json'):\n", " \n", " query_params = \"?fob-port={}\".format(ticker)\n", " query_params += \"&via-point={}\".format(via)\n", " query_params += \"&unit={}\".format(unit)\n", " query_params += \"&percent-hire={}\".format(percent_hire)\n", " query_params += \"&extended-curve={}\".format(extended_curve)\n", "\n", " uri=\"/v1.0/intraday/arbs/revisions/{}\".format(query_params)\n", " print(uri)\n", " \n", " content = do_api_get_query(\n", " uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data\n", "\n", "\n", "revs = fetch_historical_revisions(access_token, ticker='003dec0a-ce8f-41db-8c24-4d7ef6addf70', via='cogh', \n", " percent_hire=100, unit='usd-per-mmbtu', extended_curve=True, format='csv')" ] }, { "cell_type": "markdown", "id": "45988eac", "metadata": {}, "source": [ "__N.B.__\n", "\n", "Revisions are only stored in this endpoint for all revisions published since the start of the previous business day. If you'd like to check for any price revisions published earlier than this, we recommend downloading our historical datasets as these will have the most up-to-date prices (including revised prices)." ] }, { "cell_type": "code", "execution_count": null, "id": "c19aff5b", "metadata": {}, "outputs": [], "source": [ "revs" ] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }