{ "cells": [ { "cell_type": "markdown", "id": "92475c69", "metadata": {}, "source": [ "# Python API Example - Intraday Historical\n", "## Importing historical Intraday curves\n", "\n", "Here we import historical price release data from the Spark Intraday Python API. \n", "\n", "This guide is designed to provide an example of how to access the Spark API:\n", "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n", "- This script has been designed to display the raw outputs of requests from the API, and then shows you how to format those outputs to enable easy reading and analysis\n", "- This script can be copied and pasted by customers for quick use of the API\n", "\n", "__N.B. This guide is just for Intraday Historical data. If you're looking for other API data products (such as Freight routes or Netbacks), please refer to their according code example files.__ " ] }, { "cell_type": "markdown", "id": "c5716130", "metadata": {}, "source": [ "## 1. Importing Data\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "This section can remain unchanged for most Spark API users." ] }, { "cell_type": "code", "execution_count": 1, "id": "33fb0640", "metadata": {}, "outputs": [], "source": [ "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "import numpy as np\n", "from base64 import b64encode\n", "from pprint import pprint\n", "from urllib.parse import urljoin\n", "import datetime\n", "\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")\n", "\n", "\n", "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " print(\n", " \">>>> Client_id={}, client_secret={}****\".format(client_id, client_secret[:5])\n", " )\n", "\n", " return client_id, client_secret\n", "\n", "\n", "def do_api_post_query(uri, body, headers):\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def do_api_get_query(uri, access_token):\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"accept\": \"application/json\",\n", " }\n", "\n", " # HTTP POST request\n", " req = request.Request(url, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 200, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def get_access_token(client_id, client_secret):\n", " \"\"\"\n", " Get a new access_token. Access tokens are the thing that applications use to make\n", " API requests. Access tokens must be kept confidential in storage.\n", "\n", " # Procedure:\n", "\n", " Do a POST query with `grantType` and `scopes` in the body. A basic authorization\n", " HTTP header is required. The \"Basic\" HTTP authentication scheme is defined in\n", " RFC 7617, which transmits credentials as `clientId:clientSecret` pairs, encoded\n", " using base64.\n", " \"\"\"\n", "\n", " # Note: for the sake of this example, we choose to use the Python urllib from the\n", " # standard lib. One should consider using https://requests.readthedocs.io/\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " #\"scopes\": \"read:intraday\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "fd3171a8", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already.\n", "\n", "The code then prints the available prices that are callable from the API, and their corresponding Python ticker names are displayed as a list at the bottom of the Output." ] }, { "cell_type": "code", "execution_count": null, "id": "fd7e89bf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">>>> Found credentials!\n", ">>>> Client_id=01c23590-ef6c-4a36-8237-c89c3f1a3b2a, client_secret=80763****\n", ">>>> Successfully fetched an access token eyJhb****, valid 604799 seconds.\n", "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXBlIjoiYWNjZXNzVG9rZW4iLCJzdWIiOiIwMWMyMzU5MC1lZjZjLTRhMzYtODIzNy1jODljM2YxYTNiMmEiLCJzdWJUeXBlIjoib2F1dGgtY2xpZW50IiwiZXhwIjoxNzUzMTc4NTI5LCJoYXNoZWRTZWNyZXQiOiJwYmtkZjJfc2hhMjU2JDYwMDAwMCRoTXRMNDlrMUZUaVVzTE42Njlqc2pPJHVCSXNxcml5b1NHVzJTS1AvaHVLNHh3eTZ4d3VDN001aUdGRm43N2l4S1U9Iiwib3JnVXVpZCI6IjQ5MzhiMGJiLTVmMjctNDE2NC04OTM4LTUyNTdmYmQzNTNmZiIsInNjb3BlcyI6W10sImNsaWVudFR5cGUiOiJvYXV0aC1jbGllbnQifQ.8RGUrjVvl2pDgM-dszIc9LpmQ1bcWkiYa6AjKeLXuBE\n" ] } ], "source": [ "# Insert file path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)\n", "print(access_token)" ] }, { "cell_type": "markdown", "id": "922a757e", "metadata": {}, "source": [ "# Data Calling function\n", "\n", "Here we define the function used to call the 'intraday/historical' endpoint. This endpoint retrieves the historical published curves from the Intraday platform, and takes 4 required parameters:\n", "\n", "- \"contract\": which contract you'd like to pull curves for ('jkm-ttf')\n", "- \"unit\": which unit you'd like the prices to be in ('usd-per-mmbtu')\n", "- \"start\": which date you'd like the data to start from\n", "- \"end\": which date you'd like the data to run up until" ] }, { "cell_type": "code", "execution_count": null, "id": "144ef79d", "metadata": {}, "outputs": [], "source": [ "## Defining the function\n", "def fetch_historical_releases(access_token, contract=None, unit=None, start=None, end=None):\n", " query_params = \"?contract={}\".format(contract)\n", " query_params += \"&unit={}\".format(unit)\n", " if start is not None:\n", " query_params += \"&start={}\".format(start)\n", " if end is not None:\n", " query_params += \"&end={}\".format(end)\n", "\n", " uri=\"/beta/intraday/historical{}\".format(query_params)\n", " print(uri)\n", " \n", " content = do_api_get_query(\n", " uri=\"/beta/intraday/historical{}\".format(query_params), access_token=access_token\n", " )\n", "\n", " return content\n", "\n", "# call function to fetch historical data\n", "\n", "historical = fetch_historical_releases(access_token, contract='jkm-ttf', unit='usd-per-mmbtu', start='2025-06-30', end ='2025-07-03')" ] }, { "cell_type": "markdown", "id": "a4c4fcd1", "metadata": {}, "source": [ "## Data included\n", "\n", "The JSON includes 4 fields:\n", "- 'errors': will be empty if the API call is successful, otherwise returns relevant error code\n", "- 'data': contains the historical dataset of the requested contract\n", "- 'metaData': provides details on the fetched data and time of API call\n", "\n", "__N.B.__ for historical revisions, please refer to the \"Intraday Revisions\" code sample" ] }, { "cell_type": "code", "execution_count": 25, "id": "4d76461d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['errors', 'data', 'metaData'])" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# available fields\n", "historical.keys()" ] }, { "cell_type": "code", "execution_count": null, "id": "cb77f760", "metadata": {}, "outputs": [], "source": [ "# raw output\n", "historical['data']" ] }, { "cell_type": "code", "execution_count": null, "id": "b1a6fc36", "metadata": {}, "outputs": [], "source": [ "# converting to Pandas DataFrame\n", "hist_df = pd.json_normalize(historical['data'])\n", "hist_df" ] }, { "cell_type": "markdown", "id": "4f31642c", "metadata": {}, "source": [ "# Fetching Historical evolution of a single contract\n", "\n", "A simple example of how to filter the data to fetch the historical evolution of a single contract, e.g. JKM-TTF Cal26 prices" ] }, { "cell_type": "code", "execution_count": null, "id": "bcf7bca6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Apr26', 'Apr27', 'Aug26', 'Cal26', 'Cal27', 'Dec25', 'Dec26', 'Feb26', 'Feb27', 'Jan26', 'Jan27', 'Jul26', 'Jul27', 'Jun26', 'Jun27', 'Mar26', 'Mar27', 'May26', 'May27', 'Nov25', 'Nov26', 'Oct25', 'Oct26', 'Q126', 'Q127', 'Q226', 'Q227', 'Q326', 'Q425', 'Q426', 'Sep25', 'Sep26', 'Sum26', 'Win25', 'Win26']\n" ] } ], "source": [ "# Group dataframe by \"Contract\" field\n", "contract_groups = hist_df.groupby('Contract')\n", "\n", "# list available contracts (not listed chronologically)\n", "contract_keys = list(contract_groups.groups.keys())\n", "print(contract_keys)\n" ] }, { "cell_type": "code", "execution_count": 30, "id": "bf980553", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Release DateContractPrice
60062025-06-30 07:00:00+01:00Cal2611.827
60072025-06-30 07:10:00+01:00Cal2611.814
60082025-06-30 07:20:00+01:00Cal2611.78
60092025-06-30 07:30:00+01:00Cal2611.783
60102025-06-30 07:40:00+01:00Cal2611.744
60112025-06-30 07:50:00+01:00Cal2611.77
60122025-06-30 08:00:00+01:00Cal2611.788
60132025-06-30 08:10:00+01:00Cal2611.785
60142025-06-30 08:20:00+01:00Cal2611.721
60152025-06-30 08:30:00+01:00Cal2611.687
\n", "
" ], "text/plain": [ " Release Date Contract Price\n", "6006 2025-06-30 07:00:00+01:00 Cal26 11.827\n", "6007 2025-06-30 07:10:00+01:00 Cal26 11.814\n", "6008 2025-06-30 07:20:00+01:00 Cal26 11.78\n", "6009 2025-06-30 07:30:00+01:00 Cal26 11.783\n", "6010 2025-06-30 07:40:00+01:00 Cal26 11.744\n", "6011 2025-06-30 07:50:00+01:00 Cal26 11.77\n", "6012 2025-06-30 08:00:00+01:00 Cal26 11.788\n", "6013 2025-06-30 08:10:00+01:00 Cal26 11.785\n", "6014 2025-06-30 08:20:00+01:00 Cal26 11.721\n", "6015 2025-06-30 08:30:00+01:00 Cal26 11.687" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Fetch historical prices for a specific contract\n", "cdf = contract_groups.get_group('Cal26')\n", "cdf.head(10)" ] }, { "cell_type": "markdown", "id": "82958821", "metadata": {}, "source": [ "## N.B. Historical Data Limits\n", "\n", "Currently, a maximum of 30 historical days can be called at one time due to the size of the data file. \n", "\n", "If more data points are required, the below code can be used. It calls 30 historical days at a time. To call more history, increase the 'loops' parameter in the first line of the code. The 'loops' parameter describes the number of historical data requests to be executed." ] }, { "cell_type": "code", "execution_count": null, "id": "2c6339c3", "metadata": {}, "outputs": [], "source": [ "def loop_historical_data(token, hist_contract, hist_unit, hist_start, hist_end):\n", " \n", " # calculating number of days within defined period\n", " hist_diff = (datetime.datetime.strptime(hist_end, '%Y-%m-%d') - datetime.datetime.strptime(hist_start, '%Y-%m-%d')).days\n", " \n", " # initialising loop counter\n", " t = 0\n", "\n", " # looping through defined period 20 days at a time\n", " while t < hist_diff:\n", " # first loop\n", " if t == 0:\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(20, unit='days'), '%Y-%m-%d')\n", " historical = fetch_historical_releases(token, contract=hist_contract, unit=hist_unit, start=hist_start, end=diff_end)\n", " hist_df = pd.json_normalize(historical['data'])\n", "\n", " # All other loops\n", " else:\n", " if t < hist_diff-20:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(t, unit='days'), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(20, unit='days'), '%Y-%m-%d')\n", " historical = fetch_historical_releases(token, contract=hist_contract, unit=hist_unit, start=diff_start, end=diff_end)\n", " hist_df = pd.concat([hist_df,pd.json_normalize(historical['data'])])\n", "\n", " # final loop\n", " else:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(hist_start, '%Y-%m-%d') + pd.Timedelta(t, unit='days'), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(hist_diff-t, unit='days'), '%Y-%m-%d')\n", " historical = fetch_historical_releases(token, contract=hist_contract, unit=hist_unit, start=diff_start, end=diff_end)\n", " hist_df = pd.concat([hist_df,pd.json_normalize(historical['data'])])\n", " \n", " t += 20\n", "\n", " # converting data from strings into correct data types \n", " hist_df['value'] = pd.to_numeric(hist_df['value'])\n", " hist_df['asOf'] = pd.to_datetime(hist_df['asOf'])\n", "\n", " return hist_df" ] }, { "cell_type": "code", "execution_count": null, "id": "474d8f28", "metadata": {}, "outputs": [], "source": [ "# Example of using historical loop function\n", "hdf = loop_historical_data(access_token, 'jkm-ttf', 'usd-per-mmbtu', '2025-05-10', '2025-07-16')" ] }, { "cell_type": "code", "execution_count": null, "id": "d4a4337b", "metadata": {}, "outputs": [], "source": [ "# Preview of dataframe\n", "hdf" ] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }