{ "cells": [ { "cell_type": "markdown", "id": "92475c69", "metadata": {}, "source": [ "# Python API Example - Hubs (USGC LNG) API\n", "## Calling the latest & historical Hubs posts\n", "\n", "Here we call currently Active & Historical Hubs USGC posts.\n", "\n", "This guide is designed to provide an example of how to access the Spark API:\n", "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n", "- This script has been designed to display the raw outputs of requests from the API, and then shows you how to format those outputs to enable easy reading and analysis\n", "- This script can be copied and pasted by customers for quick use of the API\n", "\n", "__N.B. This guide is just for Hubs data. If you're looking for other API data products (such as Regas Costs or Netbacks), please refer to their according code example files.__ " ] }, { "cell_type": "markdown", "id": "c5716130", "metadata": {}, "source": [ "## 1. Importing Data\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "__This section can remain unchanged for most Spark API users.__" ] }, { "cell_type": "code", "execution_count": null, "id": "33fb0640", "metadata": {}, "outputs": [], "source": [ "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "import numpy as np\n", "from base64 import b64encode\n", "from pprint import pprint\n", "from urllib.parse import urljoin\n", "from datetime import datetime\n", "\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")\n", "\n", "\n", "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " print(\n", " \">>>> Client_id={}, client_secret={}****\".format(client_id, client_secret[:5])\n", " )\n", "\n", " return client_id, client_secret\n", "\n", "\n", "def do_api_post_query(uri, body, headers):\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def do_api_get_query(uri, access_token, format='json'):\n", " \"\"\"\n", " After receiving an Access Token, we can request information from the API.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " if format == 'json':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"application/json\",\n", " }\n", " elif format == 'csv':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"text/csv\"\n", " }\n", "\n", " # HTTP POST request\n", " req = request.Request(url, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 200, resp_content\n", "\n", " # Storing response based on requested format\n", " if format == 'json':\n", " content = json.loads(resp_content)\n", " elif format == 'csv':\n", " content = resp_content\n", "\n", " return content\n", "\n", "\n", "def get_access_token(client_id, client_secret):\n", " \"\"\"\n", " Get a new access_token. Access tokens are the thing that applications use to make\n", " API requests. Access tokens must be kept confidential in storage.\n", "\n", " # Procedure:\n", "\n", " Do a POST query with `grantType` and `scopes` in the body. A basic authorization\n", " HTTP header is required. The \"Basic\" HTTP authentication scheme is defined in\n", " RFC 7617, which transmits credentials as `clientId:clientSecret` pairs, encoded\n", " using base64.\n", " \"\"\"\n", "\n", " # Note: for the sake of this example, we choose to use the Python urllib from the\n", " # standard lib. One should consider using https://requests.readthedocs.io/\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "fd3171a8", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already.\n", "\n", "The code then prints the available prices that are callable from the API, and their corresponding Python ticker names are displayed as a list at the bottom of the Output." ] }, { "cell_type": "code", "execution_count": null, "id": "fd7e89bf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">>>> Found credentials!\n", ">>>> Client_id=01c23590-ef6c-4a36-8237-c89c3f1a3b2a, client_secret=80763****\n", ">>>> Successfully fetched an access token eyJhb****, valid 604799 seconds.\n", "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIwMWMyMzU5MC1lZjZjLTRhMzYtODIzNy1jODljM2YxYTNiMmEiLCJzdWJfdHlwZSI6Im9hdXRoLWNsaWVudCIsImV4cCI6MTc2Mjc4MTI2NCwidHlwZSI6ImFjY2Vzc1Rva2VuIiwiaGFzaGVkX3NlY3JldCI6InBia2RmMl9zaGEyNTYkNjAwMDAwJGhNdEw0OWsxRlRpVXNMTjY2OWpzak8kdUJJc3FyaXlvU0dXMlNLUC9odUs0eHd5Nnh3dUM3TTVpR0ZGbjc3aXhLVT0iLCJvcmdfdXVpZCI6IjQ5MzhiMGJiLTVmMjctNDE2NC04OTM4LTUyNTdmYmQzNTNmZiIsImNsaWVudF90eXBlIjoib2F1dGgtY2xpZW50In0.B2R4oX6xvcU-V5C_aY8q_RUo9It6dwELFTov4nLRUhM\n" ] } ], "source": [ "# Insert file path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)\n", "print(access_token)" ] }, { "cell_type": "markdown", "id": "0994ce16", "metadata": {}, "source": [ "# 2. Live Feed\n", "\n", "Here we define the function used to call the currently live Hubs posts, including both swap and outright posts. This function takes no parameters, and can be retrieved in JSON or CSV format.\n", "\n", "__N.B__ Metadata is only available via the JSON format." ] }, { "cell_type": "code", "execution_count": null, "id": "dff2524b", "metadata": {}, "outputs": [], "source": [ "from io import StringIO\n", "\n", "## Defining the function\n", "\n", "def fetch_live(access_token, format='json'):\n", " \n", " content = do_api_get_query(\n", " uri=\"beta/lng/hubs/usgc/live/\", access_token=access_token, format=format\n", " )\n", " \n", " if format == 'json':\n", " my_dict = content\n", " elif format == 'csv':\n", " my_dict = content.decode('utf-8')\n", " my_dict = pd.read_csv(StringIO(my_dict)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " \n", " return my_dict" ] }, { "cell_type": "code", "execution_count": 38, "id": "6844b461", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "v1.0/lng/hubs/usgc/live/\n" ] }, { "data": { "text/plain": [ "{'terminalCodes': {'003dec0a-ce8f-41db-8c24-4d7ef6addf70': 'sabine-pass',\n", " '0030c461-9a63-403d-8f53-9327ea773517': 'corpus-christi',\n", " '003adf4e-01b5-4680-9cd7-49613d5d0e3e': 'plaquemines',\n", " '003e005e-9aeb-4e31-b44f-6f1a5d79ea67': 'calcasieu-pass',\n", " '0039b921-cc5c-47e3-9a31-9dfb0c55d345': 'freeport',\n", " '00356046-86ab-4a76-a689-99af1b3027ad': 'cameron-liqu',\n", " '003e8539-3a98-48fa-b35d-0ba061beea4e': 'cove-point',\n", " '00352a22-e959-4233-b93d-d23a0da3dfed': 'elba-island',\n", " '00339787-d27c-4605-8409-e1c544820cec': 'atlantic-lng'}}" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calling the data as a JSON\n", "data_json = fetch_live(access_token, format='json')\n", "\n", "# printing metaData\n", "data_json['metaData']" ] }, { "cell_type": "code", "execution_count": 39, "id": "c8bc66b2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "v1.0/lng/hubs/usgc/live/\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idhubRegioninterestTypeorderTypeposterNamestatusconnectionsterminalCodesterminalsHiddenloadDateHidden...indicativePriceFixedValueindicativePriceIndexindicativePriceIndexPercentageindicativePriceIndexConstantindicativePriceHiddenotherInfootherInfoHiddenpostedAtUtcvalidUntilUtclastUpdatedAtUtc
0LNG-USG-S-0104USGCswapofferedAnonymousactive0sabine-pass,corpus-christiFalseFalse...NaNNaNNaNNaNTrueNaNTrue2025-10-29T18:02:15.738565+00:002025-11-05T18:00:00+00:002025-10-29T18:02:15.738648+00:00
1LNG-USG-S-0104USGCswaprequestedAnonymousactive0sabine-pass,corpus-christi,plaquemines,calcasi...FalseFalse...NaNNaNNaNNaNTrueNaNFalse2025-10-29T18:02:15.738565+00:002025-11-05T18:00:00+00:002025-10-29T18:02:15.738648+00:00
2LNG-USG-S-0100USGCswapofferedAnonymousactive1corpus-christiFalseFalse...NaNNaNNaNNaNFalseNaNFalse2025-10-24T13:38:17.936934+00:002025-11-29T00:01:00+00:002025-10-24T14:09:01.747082+00:00
\n", "

3 rows × 32 columns

\n", "
" ], "text/plain": [ " id hubRegion interestType orderType posterName status \\\n", "0 LNG-USG-S-0104 USGC swap offered Anonymous active \n", "1 LNG-USG-S-0104 USGC swap requested Anonymous active \n", "2 LNG-USG-S-0100 USGC swap offered Anonymous active \n", "\n", " connections terminalCodes \\\n", "0 0 sabine-pass,corpus-christi \n", "1 0 sabine-pass,corpus-christi,plaquemines,calcasi... \n", "2 1 corpus-christi \n", "\n", " terminalsHidden loadDateHidden ... indicativePriceFixedValue \\\n", "0 False False ... NaN \n", "1 False False ... NaN \n", "2 False False ... NaN \n", "\n", " indicativePriceIndex indicativePriceIndexPercentage \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 NaN NaN \n", "\n", " indicativePriceIndexConstant indicativePriceHidden otherInfo \\\n", "0 NaN True NaN \n", "1 NaN True NaN \n", "2 NaN False NaN \n", "\n", " otherInfoHidden postedAtUtc \\\n", "0 True 2025-10-29T18:02:15.738565+00:00 \n", "1 False 2025-10-29T18:02:15.738565+00:00 \n", "2 False 2025-10-24T13:38:17.936934+00:00 \n", "\n", " validUntilUtc lastUpdatedAtUtc \n", "0 2025-11-05T18:00:00+00:00 2025-10-29T18:02:15.738648+00:00 \n", "1 2025-11-05T18:00:00+00:00 2025-10-29T18:02:15.738648+00:00 \n", "2 2025-11-29T00:01:00+00:00 2025-10-24T14:09:01.747082+00:00 \n", "\n", "[3 rows x 32 columns]" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calling data as a CSV\n", "live_df = fetch_live(access_token, format='csv')\n", "live_df.head(3)" ] }, { "cell_type": "markdown", "id": "e3bb7625", "metadata": {}, "source": [ "# 3. Historical Slots\n", "\n", "Here we define the function used to call the all Expired swaps and outrights posts. This function takes no parameters, and can be retrieved in JSON or CSV format." ] }, { "cell_type": "code", "execution_count": null, "id": "ea4e7fc3", "metadata": {}, "outputs": [], "source": [ "## Defining the function\n", "\n", "\n", "def fetch_historical_releases(access_token, format='json'):\n", "\n", " content = do_api_get_query(\n", " uri=\"/beta/lng/hubs/usgc/historical/\", access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " my_dict = content\n", " elif format == 'csv':\n", " my_dict = content.decode('utf-8')\n", " my_dict = pd.read_csv(StringIO(my_dict))\n", " \n", " return my_dict" ] }, { "cell_type": "code", "execution_count": 41, "id": "38327f98", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idhubRegioninterestTypeorderTypeposterNamestatusconnectionsterminalCodesterminalsHiddenloadDateHidden...indicativePriceFixedValueindicativePriceIndexindicativePriceIndexPercentageindicativePriceIndexConstantindicativePriceHiddenotherInfootherInfoHiddenpostedAtUtcvalidUntilUtclastUpdatedAtUtc
0LNG-USG-S-0103USGCswapofferedAnonymousexpired0calcasieu-passFalseFalse...NaNNaNNaNNaNFalseNaNFalse2025-10-28T22:24:39.047882+00:002025-10-29T17:00:00+00:002025-10-29T17:30:02.659928+00:00
1LNG-USG-S-0103USGCswaprequestedAnonymousexpired0sabine-pass,corpus-christi,plaquemines,calcasi...FalseFalse...NaNNaNNaNNaNFalseNaNFalse2025-10-28T22:24:39.047882+00:002025-10-29T17:00:00+00:002025-10-29T17:30:02.659928+00:00
2LNG-USG-S-0102USGCswapofferedAnonymousexpired0sabine-passFalseFalse...NaNNaNNaNNaNFalseNaNFalse2025-10-28T16:37:07.294311+00:002025-10-31T23:00:00+00:002025-10-31T23:30:06.739368+00:00
\n", "

3 rows × 32 columns

\n", "
" ], "text/plain": [ " id hubRegion interestType orderType posterName status \\\n", "0 LNG-USG-S-0103 USGC swap offered Anonymous expired \n", "1 LNG-USG-S-0103 USGC swap requested Anonymous expired \n", "2 LNG-USG-S-0102 USGC swap offered Anonymous expired \n", "\n", " connections terminalCodes \\\n", "0 0 calcasieu-pass \n", "1 0 sabine-pass,corpus-christi,plaquemines,calcasi... \n", "2 0 sabine-pass \n", "\n", " terminalsHidden loadDateHidden ... indicativePriceFixedValue \\\n", "0 False False ... NaN \n", "1 False False ... NaN \n", "2 False False ... NaN \n", "\n", " indicativePriceIndex indicativePriceIndexPercentage \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 NaN NaN \n", "\n", " indicativePriceIndexConstant indicativePriceHidden otherInfo \\\n", "0 NaN False NaN \n", "1 NaN False NaN \n", "2 NaN False NaN \n", "\n", " otherInfoHidden postedAtUtc \\\n", "0 False 2025-10-28T22:24:39.047882+00:00 \n", "1 False 2025-10-28T22:24:39.047882+00:00 \n", "2 False 2025-10-28T16:37:07.294311+00:00 \n", "\n", " validUntilUtc lastUpdatedAtUtc \n", "0 2025-10-29T17:00:00+00:00 2025-10-29T17:30:02.659928+00:00 \n", "1 2025-10-29T17:00:00+00:00 2025-10-29T17:30:02.659928+00:00 \n", "2 2025-10-31T23:00:00+00:00 2025-10-31T23:30:06.739368+00:00 \n", "\n", "[3 rows x 32 columns]" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calling the function defined above, but only for the latest data\n", "hist_df = fetch_historical_releases(access_token, format='csv')\n", "hist_df.head(3)" ] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }