{ "cells": [ { "cell_type": "markdown", "id": "59d19f73", "metadata": {}, "source": [ "# Cargo Netbacks - Determining Arb Drivers through Component Analysis\n", "\n", "Looking at the underlying components that make up our Cargo Netbacks & Arbs to understand which component is driving daily changes in the arb.\n", "\n", "For a full explanation of how to import our Cargo Netbacks data, please refer to our Python Jupyter Notebook Code Samples:\n", "\n", "https://www.sparkcommodities.com/api/code-examples/jupyter.html\n" ] }, { "cell_type": "markdown", "id": "b0a05be4", "metadata": {}, "source": [ "### Have any questions?\n", "\n", "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n", "\n", "__data@sparkcommodities.com__" ] }, { "cell_type": "markdown", "id": "9e00ae34", "metadata": {}, "source": [ "## 1. API Base Call Functions\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "__This section can remain unchanged for most Spark API users.__" ] }, { "cell_type": "code", "execution_count": null, "id": "e1b0b454", "metadata": {}, "outputs": [], "source": [ "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "import numpy as np\n", "from base64 import b64encode\n", "from urllib.parse import urljoin\n", "from pprint import pprint\n", "from io import StringIO\n", "import datetime\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")" ] }, { "cell_type": "code", "execution_count": null, "id": "1161e807", "metadata": {}, "outputs": [], "source": [ "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " \n", " return client_id, client_secret\n", "\n", "\n", "def do_api_post_query(uri, body, headers):\n", " \"\"\"\n", " OAuth2 authentication requires a POST request with client credentials before accessing the API.\n", " This POST request will return an Access Token which will be used for the API GET request.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def do_api_get_query(uri, access_token, format='json'):\n", " \"\"\"\n", " After receiving an Access Token, we can request information from the API.\n", " Supports both JSON (default) and CSV responses.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " if format == 'json':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"application/json\",\n", " }\n", " elif format == 'csv':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"text/csv\",\n", " }\n", " else:\n", " raise ValueError(\"The format parameter only takes 'csv' or 'json' as inputs\")\n", "\n", " print(\"Fetching {}\".format(url))\n", "\n", " # HTTP GET request\n", " req = request.Request(url, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 200. Raise an error if this is not the case\n", " assert response.status == 200, resp_content\n", "\n", " # Storing response based on requested format\n", " if format == 'json':\n", " content = json.loads(resp_content)\n", " elif format == 'csv':\n", " content = resp_content\n", "\n", " return content\n", "\n", "\n", "def get_access_token(client_id, client_secret):\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]" ] }, { "cell_type": "markdown", "id": "f40a040b", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already. Instructions for downloading your credentials can be found here:\n", "\n", "https://www.sparkcommodities.com/api/introduction.html\n" ] }, { "cell_type": "code", "execution_count": null, "id": "eeb3bd88", "metadata": {}, "outputs": [], "source": [ "# Input the path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)" ] }, { "cell_type": "markdown", "id": "9c527e40", "metadata": {}, "source": [ "# 2. Defining Functions to fetch Cargo Netbacks data " ] }, { "cell_type": "code", "execution_count": null, "id": "ada4f167", "metadata": {}, "outputs": [], "source": [ "# Define the function for listing all netback reference data\n", "def list_netbacks(access_token):\n", "\n", " content = do_api_get_query(\n", " uri=\"/v1.0/netbacks/reference-data/\", access_token=access_token\n", " )\n", "\n", " # retrieve release dates separately\n", " reldates = content[\"data\"][\"staticData\"][\"sparkReleases\"]\n", "\n", " # return reference data\n", " refdata_dict = content[\"data\"]\n", "\n", " return reldates, refdata_dict\n", "\n", "# Fetch reference data:\n", "reldates, refdata_dict = list_netbacks(access_token)\n", "\n", "# formatting as a Dataframe\n", "available_df = pd.json_normalize(refdata_dict['staticData']['fobPorts'])\n", "available_df.head(3)" ] }, { "cell_type": "code", "execution_count": null, "id": "eb563eb4", "metadata": {}, "outputs": [], "source": [ "# Defining function to fetch netbacks data\n", "def fetch_netback(access_token, fob_port_uuid, release=None, via_point=None, laden=None, ballast=None, percent_hire=None, start=None, end=None, format='json'):\n", " \n", " query_params = \"?fob-port={}\".format(fob_port_uuid)\n", " if release is not None:\n", " query_params += \"&release-date={}\".format(release)\n", " if start is not None:\n", " query_params += \"&start={}\".format(start)\n", " if end is not None:\n", " query_params += \"&end={}\".format(end)\n", " if via_point is not None:\n", " query_params += \"&via-point={}\".format(via_point)\n", " if laden is not None:\n", " query_params += \"&laden-congestion-days={}\".format(laden)\n", " if ballast is not None:\n", " query_params += \"&ballast-congestion-days={}\".format(ballast)\n", " if percent_hire is not None:\n", " query_params += \"&percent-hire={}\".format(percent_hire)\n", " \n", " \n", " content = do_api_get_query(\n", " uri=\"/v1.0/netbacks/{}\".format(query_params),\n", " access_token=access_token, format=format,\n", " )\n", " \n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns raw response (empty string) and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = content\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", " data['ReleaseDate'] = pd.to_datetime(data['ReleaseDate'])\n", " data['LoadDate'] = pd.to_datetime(data['LoadDate'])\n", "\n", " return data\n" ] }, { "cell_type": "markdown", "id": "3729b0f0", "metadata": {}, "source": [ "## N.B. Historical Data Limits\n", "\n", "Currently, a maximum of 1 year's worth of historical data can be called at one time due to the size of the data file. \n", "\n", "If more data points are required, the below code can be used: the function calls the historical data 1 year at a time and combines the data into one Pandas DataFrame" ] }, { "cell_type": "code", "execution_count": null, "id": "9163a05a", "metadata": {}, "outputs": [], "source": [ "\n", "def loop_historical_data(access_token, fob_port_uuid=None, via_point=None, percent_hire=None, start=None, end=None):\n", " \n", " hist_diff = (datetime.datetime.strptime(end, '%Y-%m-%d') - datetime.datetime.strptime(start, '%Y-%m-%d')).days\n", " t = 0\n", "\n", " starts = []\n", " ends = []\n", "\n", " w = 365\n", "\n", " while t < hist_diff:\n", " # initialising dataframe\n", " if t == 0 and hist_diff>w:\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=w), '%Y-%m-%d')\n", " hist_df = fetch_netback(access_token, fob_port_uuid=fob_port_uuid, via_point=via_point, start=start, end=diff_end, \n", " percent_hire=percent_hire, format='csv')\n", "\n", " starts.append(start)\n", " ends.append(diff_end)\n", " \n", " elif t==0 and hist_diff<=w:\n", " hist_df = fetch_netback(access_token, fob_port_uuid=fob_port_uuid, via_point=via_point, start=start, end=end, \n", " percent_hire=percent_hire, format='csv')\n", "\n", " # appending additional historical data\n", " else:\n", " if t < hist_diff-w:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=t+1), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=w), '%Y-%m-%d')\n", " historical_addition = fetch_netback(access_token, fob_port_uuid=fob_port_uuid, via_point=via_point, start=diff_start, end=diff_end, \n", " percent_hire=percent_hire, format='csv')\n", " \n", " try:\n", " hist_df = pd.concat([hist_df,historical_addition])\n", " #exception if hist_df is empty\n", " except:\n", " hist_df = historical_addition.copy()\n", " starts.append(start)\n", " ends.append(diff_end)\n", " else:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=t+1), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=hist_diff-t), '%Y-%m-%d')\n", "\n", " historical_addition = fetch_netback(access_token, fob_port_uuid=fob_port_uuid, via_point=via_point, start=diff_start, end=diff_end, \n", " percent_hire=percent_hire, format='csv')\n", "\n", " hist_df = pd.concat([hist_df, historical_addition])\n", " starts.append(start)\n", " ends.append(diff_end)\n", " \n", " #looping by year\n", " t += w\n", "\n", " for c in list(hist_df.columns)[4:]:\n", " hist_df[c] = pd.to_numeric(hist_df[c])\n", " hist_df['ReleaseDate'] = pd.to_datetime(hist_df['ReleaseDate'])\n", " hist_df['LoadMonthDate'] = pd.to_datetime(hist_df['LoadMonth'])\n", " hist_df['CalMonth'] = hist_df['LoadMonthDate'].dt.strftime('%b%y')\n", "\n", " hist_df = hist_df.drop_duplicates()\n", " hist_df = hist_df.sort_values(['ReleaseDate', 'LoadDate'])\n", "\n", " return hist_df" ] }, { "cell_type": "markdown", "id": "4b8096e9", "metadata": {}, "source": [ "# 3. Defining Inputs & Calling Data" ] }, { "cell_type": "code", "execution_count": null, "id": "cde48ff9", "metadata": {}, "outputs": [], "source": [ "# Defining date-range for data call\n", "start = '2026-03-01'\n", "end = datetime.datetime.today().strftime('%Y-%m-%d')\n", "\n", "# Fetching port & via point inputs\n", "sab = available_df[available_df[\"name\"] == 'Sabine Pass'][\"uuid\"].iloc[0]\n", "\n", "# Fetching data with defined input params\n", "df_csv = loop_historical_data(access_token, start=start, end=end, fob_port_uuid=sab, via_point='cogh')\n", "\n", "# Converting Freight columns to negative to indicate them as costs\n", "df_csv['NeaMetaRouteCost'] = -df_csv['NeaMetaRouteCost']\n", "df_csv['NweMetaRouteCost'] = -df_csv['NweMetaRouteCost']\n", "\n", "df_csv" ] }, { "cell_type": "code", "execution_count": null, "id": "2fd057a9", "metadata": {}, "outputs": [], "source": [ "# Calculating Arb Components from underlying Netbacks components: DES LNG pricing\n", "df_csv['DeltaMetaDesLng'] = df_csv['NeaMetaDesLng'] - df_csv['NweMetaDesLng']\n", "df_csv['DeltaMetaTtfPrice'] = df_csv['NeaMetaTtfPrice'] - df_csv['NweMetaTtfPrice']\n", "df_csv['DeltaMetaTtfBasis'] = (df_csv['NeaMetaTtfBasis'] - df_csv['NweMetaTtfBasis'])\n", "\n", "# Calculating Arb Components from underlying Netbacks components: Freight costs\n", "df_csv['DeltaMetaRouteCost'] = (df_csv['NeaMetaRouteCost'] - df_csv['NweMetaRouteCost'])\n", "df_csv['DeltaMetaVolAdj'] = df_csv['NeaMetaVolAdj'] - df_csv['NweMetaVolAdj']\n", "df_csv['DeltaFreight'] = df_csv['DeltaMetaRouteCost'] + df_csv['DeltaMetaVolAdj']" ] }, { "cell_type": "code", "execution_count": null, "id": "5ddee530", "metadata": {}, "outputs": [], "source": [ "# Creating Front Month dataframe\n", "m1 = df_csv[df_csv['LoadMonthIndex'] == 1].copy()\n", "\n", "# make sure data is sorted\n", "m1 = m1.sort_values('ReleaseDate', ascending=False)\n", "\n", "# Creating day-on-day diffs for each component\n", "m1['DiffDesLng'] = m1['DeltaMetaDesLng'].diff(periods=-1)\n", "m1['DiffTtfPrice'] = m1['DeltaMetaTtfPrice'].diff(periods=-1)\n", "m1['DiffTtfBasis'] = m1['DeltaMetaTtfBasis'].diff(periods=-1)\n", "m1['DiffRouteCost'] = m1['DeltaMetaRouteCost'].diff(periods=-1)\n", "m1['DiffVolAdj'] = m1['DeltaMetaVolAdj'].diff(periods=-1)\n", "\n", "m1.head(3)" ] }, { "cell_type": "markdown", "id": "054ef4d4", "metadata": {}, "source": [ "# 4. Plotting" ] }, { "cell_type": "code", "execution_count": null, "id": "86dc4419", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.dates as mdates\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "sns.set_style('whitegrid')\n", "\n", "# Component naming & colour mapping\n", "comp_cols = {\n", " 'DiffTtfPrice': 'TTF Price',\n", " 'DiffTtfBasis': 'TTF Basis',\n", " 'DiffRouteCost': 'Route Cost',\n", " 'DiffVolAdj': 'Vol Adj',\n", "}\n", "comp_colours = {\n", " 'TTF Price': '#008bcc',\n", " 'TTF Basis': '#9B59B6',\n", " 'Route Cost': '#1E8449',\n", " 'Vol Adj': '#52BE80',\n", "}\n", "\n", "filt_threshold = 0.03 # grey out days whose net move is below this ($/MMBtu)\n", "hide_weekend = False # set to True if you want to exclude weekends (i.e. set to white) from \"dominant component\" graph\n", "\n", "# Creating function to stack positive components and negative components separately\n", "def stacked_signed_bars(ax, x, df, cols, colours, width=0.8):\n", " pos = np.zeros(len(x))\n", " neg = np.zeros(len(x))\n", " handles = []\n", " for col, label in cols.items():\n", " v = df[col].fillna(0).to_numpy(float)\n", " bottoms = np.where(v >= 0, pos, neg)\n", " h = ax.bar(x, v, bottom=bottoms, width=width, color=colours[label],\n", " label=label, edgecolor='white', linewidth=0.4, zorder=1)\n", " pos = pos + np.where(v >= 0, v, 0.0)\n", " neg = neg + np.where(v < 0, v, 0.0)\n", " handles.append(h)\n", " return handles\n", "\n", "#\n", "net = m1[list(comp_cols)].sum(axis=1) # Daily diff in arb\n", "signed = m1[list(comp_cols)].mul(np.sign(net), axis=0) # calc direction of effect of each component\n", "valid = signed.notna().any(axis=1) # rows with at least one non-NaN component\n", "dom_label = signed[valid].idxmax(axis=1).map(comp_cols) # dominant component, valid rows only\n", "dom_label = dom_label.reindex(signed.index) # restore all-NaN rows as NaN\n", "dom_colour = dom_label.map(comp_colours) # fetching colour of dominant component\n", "dom_colour[net.abs() < filt_threshold] = '#dddddd' # aplying filter threshold\n", "dom_colour = dom_colour.fillna('#dddddd') # applying grey to dates with no dominant component\n", "\n", "# Figure: main chart on top, dominance strip below (shared x)\n", "fig, (ax, ax_strip) = plt.subplots(\n", " 2, 1, figsize=(14, 7.6), sharex=True,\n", " gridspec_kw={'height_ratios': [14, 1], 'hspace': 0.10}\n", ")\n", "\n", "# Bars on a twin y-axis, arb line on the primary axis\n", "# remove the first line below, and change all \"axb\" values to \"ax\", to plot on same y-axis\n", "axb = ax.twinx()\n", "bar_handles = stacked_signed_bars(axb, m1['ReleaseDate'], m1, comp_cols, comp_colours)\n", "axb.axhline(0, color='grey', linewidth=0.8, zorder=0)\n", "axb.set_ylabel('Daily change in arb components ($/MMBtu)')\n", "axb.grid(False)\n", "\n", "line, = ax.plot(m1['ReleaseDate'], m1['DeltaNeaNwe'], color='#1A3A6B',\n", " label='US Arb (NEA via COGH)', linewidth=3.0, zorder=3, alpha=0.5)\n", "ax.scatter(m1['ReleaseDate'].iloc[0], m1['DeltaNeaNwe'].iloc[0],\n", " color='#1A3A6B', marker='o', s=80, zorder=4, alpha=0.5)\n", "ax.set_ylabel('Arb level ($/MMBtu)')\n", "\n", "# Line axis in front of the bar axis\n", "ax.set_zorder(axb.get_zorder() + 1)\n", "ax.patch.set_visible(False)\n", "\n", "ax.set_xlim([m1['ReleaseDate'].iloc[-1] - pd.Timedelta(1, 'day'),\n", " m1['ReleaseDate'].iloc[0] + pd.Timedelta(2, 'day')])\n", "\n", "# creating legend\n", "handles = [line] + bar_handles\n", "ax.legend(handles, [h.get_label() for h in handles], loc='lower right')\n", "\n", "# ── Dominance strip ────────────────────────────────────────────────────────\n", "# creating dataframe for the dominant component strip\n", "strip = (pd.DataFrame({'date': m1['ReleaseDate'].to_numpy(),\n", " 'color': dom_colour.to_numpy()})\n", " .sort_values('date').reset_index(drop=True))\n", "xnum = mdates.date2num(strip['date'])\n", "\n", "# calculating segment positions for each bar to align with arb chart\n", "if hide_weekend:\n", " lefts = xnum - 0.5\n", " widths = np.full(len(xnum), 1.0)\n", "else:\n", " mid = (xnum[:-1] + xnum[1:]) / 2\n", " edges = np.concatenate([[xnum[0] - (xnum[1] - xnum[0]) / 2],\n", " mid,\n", " [xnum[-1] + (xnum[-1] - xnum[-2]) / 2]])\n", " lefts = edges[:-1]\n", " widths = np.diff(edges)\n", "\n", "# Plotting dominant component strip chart\n", "ax_strip.bar(lefts, height=1.0, width=widths, bottom=0, align='edge',\n", " color=strip['color'].to_numpy(), linewidth=0)\n", "ax_strip.set_ylim(0, 1)\n", "ax_strip.set_yticks([])\n", "ax_strip.set_ylabel('Dominant\\ndriver', rotation=0, ha='right', va='center', labelpad=25)\n", "ax_strip.set_xlabel('Release Date')\n", "ax_strip.grid(False)\n", "\n", "plt.tight_layout()\n", "plt.show()" ] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.9" } }, "nbformat": 4, "nbformat_minor": 5 }