{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25c8380d",
   "metadata": {},
   "source": [
    "# Python API Example - Cargo Contracts\n",
    "\n",
    "This guide is designed to provide an example of how to call the Spark Cargo Contracts endpoint\n",
    "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n",
    "- This script has been designed to display the raw outputs of requests from the API, and then shows you how to format those outputs to enable easy reading and analysis\n",
    " \n",
    "__N.B. This guide is just for Cargo Contracts data. If you're looking for other API data products (such as Freight routes or Netbacks), please refer to their according code example files.__ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f555e29c",
   "metadata": {},
   "source": [
    "### Have any questions?\n",
    "\n",
    "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n",
    "\n",
    "__data@sparkcommodities.com__\n",
    "\n",
    "or refer to our API website for more information about this endpoint: https://www.sparkcommodities.com/api/lng-cargo/contracts.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84f27f99",
   "metadata": {},
   "source": [
    "## 1. Importing Data\n",
    "\n",
    "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n",
    "\n",
    "This section can remain unchanged for most Spark API users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1172a5c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import sys\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from base64 import b64encode\n",
    "from urllib.parse import urljoin\n",
    "from io import StringIO\n",
    "import datetime\n",
    "\n",
    "try:\n",
    "    from urllib import request, parse\n",
    "    from urllib.error import HTTPError\n",
    "except ImportError:\n",
    "    raise RuntimeError(\"Python 3 required\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fe087d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "API_BASE_URL = \"https://api.sparkcommodities.com\"\n",
    "\n",
    "\n",
    "def retrieve_credentials(file_path=None):\n",
    "    \"\"\"\n",
    "    Find credentials either by reading the client_credentials file or reading\n",
    "    environment variables\n",
    "    \"\"\"\n",
    "    if file_path is None:\n",
    "        client_id = os.getenv(\"SPARK_CLIENT_ID\")\n",
    "        client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n",
    "        if not client_id or not client_secret:\n",
    "            raise RuntimeError(\n",
    "                \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n",
    "            )\n",
    "    else:\n",
    "        # Parse the file\n",
    "        if not os.path.isfile(file_path):\n",
    "            raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n",
    "\n",
    "        with open(file_path) as fp:\n",
    "            lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n",
    "\n",
    "        if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n",
    "            client_id, client_secret = lines[1].split(\",\")\n",
    "        else:\n",
    "            print(\"First line read: '{}'\".format(lines[0]))\n",
    "            raise RuntimeError(\n",
    "                \"The specified file {} doesn't look like to be a Spark API client \"\n",
    "                \"credentials file\".format(file_path)\n",
    "            )\n",
    "\n",
    "    print(\">>>> Found credentials!\")\n",
    "    \n",
    "    return client_id, client_secret\n",
    "\n",
    "\n",
    "def do_api_post_query(uri, body, headers):\n",
    "    \"\"\"\n",
    "    OAuth2 authentication requires a POST request with client credentials before accessing the API.\n",
    "    This POST request will return an Access Token which will be used for the API GET request.\n",
    "    \"\"\"\n",
    "    url = urljoin(API_BASE_URL, uri)\n",
    "\n",
    "    data = json.dumps(body).encode(\"utf-8\")\n",
    "\n",
    "    # HTTP POST request\n",
    "    req = request.Request(url, data=data, headers=headers)\n",
    "    try:\n",
    "        response = request.urlopen(req)\n",
    "    except HTTPError as e:\n",
    "        print(\"HTTP Error: \", e.code)\n",
    "        print(e.read())\n",
    "        sys.exit(1)\n",
    "\n",
    "    resp_content = response.read()\n",
    "\n",
    "    # The server must return HTTP 201. Raise an error if this is not the case\n",
    "    assert response.status == 201, resp_content\n",
    "\n",
    "    # The server returned a JSON response\n",
    "    content = json.loads(resp_content)\n",
    "\n",
    "    return content\n",
    "\n",
    "\n",
    "def do_api_get_query(uri, access_token, format='json'):\n",
    "    \"\"\"\n",
    "    After receiving an Access Token, we can request information from the API.\n",
    "    Supports both JSON (default) and CSV responses.\n",
    "    \"\"\"\n",
    "    url = urljoin(API_BASE_URL, uri)\n",
    "\n",
    "    if format == 'json':\n",
    "        headers = {\n",
    "            \"Authorization\": \"Bearer {}\".format(access_token),\n",
    "            \"Accept\": \"application/json\",\n",
    "        }\n",
    "    elif format == 'csv':\n",
    "        headers = {\n",
    "            \"Authorization\": \"Bearer {}\".format(access_token),\n",
    "            \"Accept\": \"text/csv\",\n",
    "        }\n",
    "    else:\n",
    "        raise ValueError(\"The format parameter only takes 'csv' or 'json' as inputs\")\n",
    "\n",
    "    print(\"Fetching {}\".format(url))\n",
    "\n",
    "    # HTTP GET request\n",
    "    req = request.Request(url, headers=headers)\n",
    "    try:\n",
    "        response = request.urlopen(req)\n",
    "    except HTTPError as e:\n",
    "        print(\"HTTP Error: \", e.code)\n",
    "        print(e.read())\n",
    "        sys.exit(1)\n",
    "\n",
    "    resp_content = response.read()\n",
    "\n",
    "    # The server must return HTTP 200. Raise an error if this is not the case\n",
    "    assert response.status == 200, resp_content\n",
    "\n",
    "    # Storing response based on requested format\n",
    "    if format == 'json':\n",
    "        content = json.loads(resp_content)\n",
    "    elif format == 'csv':\n",
    "        content = resp_content\n",
    "\n",
    "    return content\n",
    "\n",
    "\n",
    "def get_access_token(client_id, client_secret):\n",
    "\n",
    "    payload = \"{}:{}\".format(client_id, client_secret).encode()\n",
    "    headers = {\n",
    "        \"Authorization\": b64encode(payload).decode(),\n",
    "        \"Accept\": \"application/json\",\n",
    "        \"Content-Type\": \"application/json\",\n",
    "    }\n",
    "    body = {\n",
    "        \"grantType\": \"clientCredentials\",\n",
    "    }\n",
    "\n",
    "    content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n",
    "\n",
    "    print(\n",
    "        \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n",
    "            content[\"accessToken\"][:5], content[\"expiresIn\"]\n",
    "        )\n",
    "    )\n",
    "\n",
    "    return content[\"accessToken\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd3171a8",
   "metadata": {},
   "source": [
    "## N.B. Credentials\n",
    "\n",
    "Here we call the above functions, and input the file path to our credentials.\n",
    "\n",
    "N.B. You must have downloaded your client credentials CSV file before proceeding. Please refer to the API documentation if you have not dowloaded them already. Instructions for downloading your credentials can be found here:\n",
    "\n",
    "https://www.sparkcommodities.com/api/request/authentication.html\n",
    "\n",
    "\n",
    "The code then prints the available prices that are callable from the API, and their corresponding Python ticker names are displayed as a list at the bottom of the Output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd7e89bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Insert file path to your client credentials here\n",
    "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n",
    "\n",
    "# Authenticate:\n",
    "access_token = get_access_token(client_id, client_secret)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3638e7a7",
   "metadata": {},
   "source": [
    "## 2. Listing all available contract tickers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce36499b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define function for listing contracts from API\n",
    "def list_contracts(access_token):\n",
    "    \"\"\"\n",
    "    Fetch available contracts. Return contract ticker symbols\n",
    "\n",
    "    # Procedure:\n",
    "\n",
    "    Do a GET query to /v1.0/contracts/ with a Bearer token authorization HTTP header.\n",
    "    \"\"\"\n",
    "    content = do_api_get_query(uri=\"/v1.0/contracts/\", access_token=access_token)\n",
    "\n",
    "    print(\">>>> All the contracts you can fetch\")\n",
    "    tickers = []\n",
    "    for contract in content[\"data\"]:\n",
    "        print(contract[\"fullName\"])\n",
    "        tickers.append(contract[\"id\"])\n",
    "\n",
    "    return tickers\n",
    "\n",
    "# Fetch all contracts:\n",
    "tickers = list_contracts(access_token)\n",
    "\n",
    "print(tickers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc9cf152",
   "metadata": {},
   "source": [
    "## 3. Latest Price Release\n",
    "\n",
    "Here we define a function to fetch the latest price release. The available parameters can be found on our API docs:\n",
    "\n",
    "Cargo: https://www.sparkcommodities.com/api/lng-cargo/contracts.html\n",
    "\n",
    "__N.B.:__ This endpoint provides the option to return a JSON or CSV formatted response. Metadata is only available in the JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d026eb33",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Defining the function\n",
    "\n",
    "\n",
    "def fetch_latest_price_releases(access_token, ticker):\n",
    "    \"\"\"\n",
    "    For a contract, fetch then display the latest price release\n",
    "\n",
    "    # Procedure:\n",
    "\n",
    "    Do GET queries to /v1.0/contracts/{contract_ticker_symbol}/price-releases/latest/\n",
    "    with a Bearer token authorization HTTP header.\n",
    "    \"\"\"\n",
    "    content = do_api_get_query(\n",
    "        uri=\"/v1.0/contracts/{}/price-releases/latest/\".format(ticker),\n",
    "        access_token=access_token,\n",
    "    )\n",
    "\n",
    "    return content[\"data\"]\n",
    "\n",
    "\n",
    "## Calling that function and storing the output\n",
    "data = fetch_latest_price_releases(access_token, 'sparknwe-b-f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04e61ee1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shows how the raw output is formatted\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0e0e030",
   "metadata": {},
   "source": [
    "## 3. Historical Price Releases\n",
    "\n",
    "Here we define a function to fetch the historical price releases. The available parameters can be found on our API docs:\n",
    "\n",
    "Cargo: https://www.sparkcommodities.com/api/lng-cargo/contracts.html\n",
    "\n",
    "__N.B.:__ This endpoint provides the option to return a JSON or CSV formatted response. Metadata is only available in the JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff4d0dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fetch_historical_price_releases(access_token, ticker, limit=4, offset=None):\n",
    "\n",
    "    query_params = \"?limit={}\".format(limit)\n",
    "    if offset is not None:\n",
    "        query_params += \"&offset={}\".format(offset)\n",
    "\n",
    "    content = do_api_get_query(\n",
    "        uri=\"/v1.0/contracts/{}/price-releases/{}\".format(ticker, query_params),\n",
    "        access_token=access_token,\n",
    "    )\n",
    "\n",
    "    data = content[\"data\"]\n",
    "\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4836aa94",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Define which price product you want to retrieve\n",
    "example_hist_data = fetch_historical_price_releases(access_token, ticker='sparknwe-b-f', limit=10)\n",
    "example_hist_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99be9416",
   "metadata": {},
   "source": [
    "### Formatting into a Pandas DataFrame\n",
    "\n",
    "The outputted data has several nested lists and dictionaries. If we are aware of what variables we want, we can externally store these values as lists and create a Pandas DataFrame.\n",
    "\n",
    "The dictionary is then transformed into a Pandas Dataframe for readability and ease of use. \n",
    "\n",
    "\n",
    "__N.B.__ This JSON structure is not consistent across all datasets, and so might need to be amended when calling other Spark contracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27782530",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the function for storing and formatting the data into a Pandas DataFrame\n",
    "def store_and_format(dict_hist):\n",
    "    stored_data = {\n",
    "        \"ReleaseDate\": [],\n",
    "        \"Ticker\": [],\n",
    "        \"PeriodStart\": [],\n",
    "        \"USDpermmbtu\": [],\n",
    "        \"USDpermmbtuMax\": [],\n",
    "        \"USDpermmbtuMin\": [],\n",
    "    }\n",
    "\n",
    "    for release in dict_hist:\n",
    "\n",
    "        data_points = release[\"data\"][0][\"dataPoints\"]\n",
    "\n",
    "        for data_point in data_points:\n",
    "            stored_data['Ticker'].append(release[\"contractId\"])\n",
    "            stored_data['ReleaseDate'].append(release[\"releaseDate\"])\n",
    "\n",
    "            period_start_at = data_point[\"deliveryPeriod\"][\"startAt\"]\n",
    "            stored_data['PeriodStart'].append(period_start_at)\n",
    "\n",
    "            stored_data['USDpermmbtu'].append(data_point[\"derivedPrices\"][\"usdPerMMBtu\"][\"spark\"])\n",
    "            stored_data['USDpermmbtuMax'].append(data_point[\"derivedPrices\"][\"usdPerMMBtu\"][\"sparkMin\"])\n",
    "            stored_data['USDpermmbtuMin'].append(data_point[\"derivedPrices\"][\"usdPerMMBtu\"][\"sparkMax\"])\n",
    "            \n",
    "    historical_df = pd.DataFrame(stored_data)\n",
    "    \n",
    "    historical_df[\"USDpermmbtu\"] = pd.to_numeric(historical_df[\"USDpermmbtu\"])\n",
    "    historical_df[\"USDpermmbtuMax\"] = pd.to_numeric(historical_df[\"USDpermmbtuMax\"])\n",
    "    historical_df[\"USDpermmbtuMin\"] = pd.to_numeric(historical_df[\"USDpermmbtuMin\"])\n",
    "\n",
    "    historical_df[\"ReleaseDate\"] = pd.to_datetime(historical_df[\"ReleaseDate\"])\n",
    "        \n",
    "    return historical_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66656ee3",
   "metadata": {},
   "source": [
    "# 4. Calling Cargo Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b72f5322",
   "metadata": {},
   "source": [
    "# 4.1. Front Month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56aa19be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling and storing SparkNWE Front Month data (TTF basis)\n",
    "sparknwe_basis_data = fetch_historical_price_releases(access_token, ticker='sparknwe-b-f', limit=10)\n",
    "sparknwe_basis = store_and_format(sparknwe_basis_data)\n",
    "sparknwe_basis.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3664778",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling and storing SparkNWE Front Month data (Outright price)\n",
    "sparknwe_outright_data = fetch_historical_price_releases(access_token, ticker='sparknwe-f', limit=10)\n",
    "sparknwe_outright = store_and_format(sparknwe_outright_data)\n",
    "sparknwe_outright.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e89d22c5",
   "metadata": {},
   "source": [
    "## 4.2. Forward Curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57de7454",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling and storing SparkNWE Forward Curve data (TTF basis)\n",
    "sparknwefo_basis_data = fetch_historical_price_releases(access_token, ticker='sparknwe-b-fo', limit=10)\n",
    "sparknwefo_basis = store_and_format(sparknwefo_basis_data)\n",
    "sparknwefo_basis.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68204e17",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling and storing SparkNWE Forward Curve data (Outright)\n",
    "sparknwefo_outright_data = fetch_historical_price_releases(access_token, ticker='sparknwe-fo', limit=10)\n",
    "sparknwefo_outright = store_and_format(sparknwefo_outright_data)\n",
    "sparknwefo_outright.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838a25c1",
   "metadata": {},
   "source": [
    "# Analytics Gallery\n",
    "Want to gain market insights using our data?\n",
    "\n",
    "Take a look at our [Analytics Gallery](https://www.sparkcommodities.com/api/code-examples/analytics-examples.html) on the Spark API website\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}