{ "cells": [ { "cell_type": "markdown", "id": "dd2f10a9", "metadata": {}, "source": [ "# Python API Example - Gas Transit Capacity\n", "\n", "This guide is designed to provide an example of how to call the Gas Transit Capacity API endpoints, and store the data accordingly.\n", "- The path to your client credentials is the only input needed to run this script (just before Section 2)\n", "\n", "__N.B. This guide is just for Gas Transit Capacity data. If you're looking for other API data products (such as Gas Transit costs, Contracts or Netbacks), please refer to their according code example files.__" ] }, { "cell_type": "markdown", "id": "0a7c8ef9", "metadata": {}, "source": [ "### Have any questions?\n", "\n", "If you have any questions regarding our API, or need help accessing specific datasets, please contact us at:\n", "\n", "__data@sparkcommodities.com__\n", "\n", "or refer to our API website for more information about this endpoint:\n", "https://www.sparkcommodities.com/api/gas/transit-capacity.html" ] }, { "cell_type": "markdown", "id": "8e21285b", "metadata": {}, "source": [ "## 1. Importing Data\n", "\n", "Here we define the functions that allow us to retrieve the valid credentials to access the Spark API.\n", "\n", "__This section can remain unchanged for most Spark API users.__" ] }, { "cell_type": "code", "id": "0c195b75", "metadata": {}, "source": [ "import json\n", "import os\n", "import sys\n", "import pandas as pd\n", "import numpy as np\n", "from base64 import b64encode\n", "from urllib.parse import urljoin\n", "from io import StringIO\n", "import datetime\n", "\n", "try:\n", " from urllib import request, parse\n", " from urllib.error import HTTPError\n", "except ImportError:\n", " raise RuntimeError(\"Python 3 required\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "9e888ad1", "metadata": {}, "source": [ "API_BASE_URL = \"https://api.sparkcommodities.com\"\n", "\n", "\n", "def retrieve_credentials(file_path=None):\n", " \"\"\"\n", " Find credentials either by reading the client_credentials file or reading\n", " environment variables\n", " \"\"\"\n", " if file_path is None:\n", " client_id = os.getenv(\"SPARK_CLIENT_ID\")\n", " client_secret = os.getenv(\"SPARK_CLIENT_SECRET\")\n", " if not client_id or not client_secret:\n", " raise RuntimeError(\n", " \"SPARK_CLIENT_ID and SPARK_CLIENT_SECRET environment vars required\"\n", " )\n", " else:\n", " # Parse the file\n", " if not os.path.isfile(file_path):\n", " raise RuntimeError(\"The file {} doesn't exist\".format(file_path))\n", "\n", " with open(file_path) as fp:\n", " lines = [l.replace(\"\\n\", \"\") for l in fp.readlines()]\n", "\n", " if lines[0] in (\"clientId,clientSecret\", \"client_id,client_secret\"):\n", " client_id, client_secret = lines[1].split(\",\")\n", " else:\n", " print(\"First line read: '{}'\".format(lines[0]))\n", " raise RuntimeError(\n", " \"The specified file {} doesn't look like to be a Spark API client \"\n", " \"credentials file\".format(file_path)\n", " )\n", "\n", " print(\">>>> Found credentials!\")\n", " \n", " return client_id, client_secret\n", "\n", "\n", "def do_api_post_query(uri, body, headers):\n", " \"\"\"\n", " OAuth2 authentication requires a POST request with client credentials before accessing the API.\n", " This POST request will return an Access Token which will be used for the API GET request.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " data = json.dumps(body).encode(\"utf-8\")\n", "\n", " # HTTP POST request\n", " req = request.Request(url, data=data, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 201. Raise an error if this is not the case\n", " assert response.status == 201, resp_content\n", "\n", " # The server returned a JSON response\n", " content = json.loads(resp_content)\n", "\n", " return content\n", "\n", "\n", "def do_api_get_query(uri, access_token, format='json'):\n", " \"\"\"\n", " After receiving an Access Token, we can request information from the API.\n", " Supports both JSON (default) and CSV responses.\n", " \"\"\"\n", " url = urljoin(API_BASE_URL, uri)\n", "\n", " if format == 'json':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"application/json\",\n", " }\n", " elif format == 'csv':\n", " headers = {\n", " \"Authorization\": \"Bearer {}\".format(access_token),\n", " \"Accept\": \"text/csv\",\n", " }\n", " else:\n", " raise ValueError(\"The format parameter only takes 'csv' or 'json' as inputs\")\n", "\n", " print(\"Fetching {}\".format(url))\n", "\n", " # HTTP GET request\n", " req = request.Request(url, headers=headers)\n", " try:\n", " response = request.urlopen(req)\n", " except HTTPError as e:\n", " print(\"HTTP Error: \", e.code)\n", " print(e.read())\n", " sys.exit(1)\n", "\n", " resp_content = response.read()\n", "\n", " # The server must return HTTP 200. Raise an error if this is not the case\n", " assert response.status == 200, resp_content\n", "\n", " # Storing response based on requested format\n", " if format == 'json':\n", " content = json.loads(resp_content)\n", " elif format == 'csv':\n", " content = resp_content\n", "\n", " return content\n", "\n", "\n", "def get_access_token(client_id, client_secret):\n", "\n", " payload = \"{}:{}\".format(client_id, client_secret).encode()\n", " headers = {\n", " \"Authorization\": b64encode(payload).decode(),\n", " \"Accept\": \"application/json\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " body = {\n", " \"grantType\": \"clientCredentials\",\n", " }\n", "\n", " content = do_api_post_query(uri=\"/oauth/token/\", body=body, headers=headers)\n", "\n", " print(\n", " \">>>> Successfully fetched an access token {}****, valid {} seconds.\".format(\n", " content[\"accessToken\"][:5], content[\"expiresIn\"]\n", " )\n", " )\n", "\n", " return content[\"accessToken\"]" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "efd0c53b", "metadata": {}, "source": [ "## N.B. Credentials\n", "\n", "Here we call the above functions, and input the file path to our credentials.\n", "\n", "N.B. You must have downloaded your client credentials CSV file before proceeding. Instructions for downloading your credentials can be found here:\n", "\n", "https://app.sparkcommodities.com/data-integrations/api" ] }, { "cell_type": "code", "id": "4494e0b2", "metadata": {}, "source": [ "# Input the path to your client credentials here\n", "client_id, client_secret = retrieve_credentials(file_path=\"/tmp/client_credentials.csv\")\n", "\n", "# Authenticate:\n", "access_token = get_access_token(client_id, client_secret)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "1850e940", "metadata": {}, "source": [ "## 2. Reference Data\n", "\n", "Lists every Gas Transit route. `RouteId` is the value used to filter the capacity endpoints, and `CapacityFirstReleaseDate` is the first release with capacity data for that route (empty if the route has no capacity data yet)." ] }, { "cell_type": "code", "id": "073c3261", "metadata": {}, "source": [ "def fetch_ref_data(access_token, format='json'):\n", "\n", " content = do_api_get_query(\n", " uri=\"/v1.0/gas/transit/reference-data/\", access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data\n", "\n", "ref_df = fetch_ref_data(access_token, format='csv')\n", "ref_df.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "cbd54a2d", "metadata": {}, "source": [ "# routes with capacity data\n", "ref_df[ref_df['CapacityFirstReleaseDate'].notna()].head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "64e800d7", "metadata": {}, "source": [ "## 3. Monthly Capacity\n", "\n", "Technical, Booked and Available capacity for each route, by delivery month. The available parameters can be found on our API docs:\n", "\n", "https://www.sparkcommodities.com/api/gas/transit-capacity.html\n", "\n", "__N.B.:__ This endpoint provides the option to return a JSON or CSV formatted response. Metadata is only available in the JSON format." ] }, { "cell_type": "code", "id": "ee74ca7b", "metadata": {}, "source": [ "def fetch_monthly_capacity(access_token, origin=None, destination=None, route_id=None, capacity_type=None,\n", " start=None, end=None, limit=None, offset=None, unit=None, format='json'):\n", "\n", " query_params = ''\n", "\n", " if origin is not None:\n", " query_params += '&origin={}'.format(origin)\n", " if destination is not None:\n", " query_params += '&destination={}'.format(destination)\n", " if route_id is not None:\n", " query_params += '&route-id={}'.format(route_id)\n", " if capacity_type is not None:\n", " query_params += '&type={}'.format(capacity_type)\n", " if start is not None:\n", " query_params += '&start={}'.format(start)\n", " if end is not None:\n", " query_params += '&end={}'.format(end)\n", " if limit is not None:\n", " query_params += '&limit={}'.format(limit)\n", " if offset is not None:\n", " query_params += '&offset={}'.format(offset)\n", " if unit is not None:\n", " query_params += '&unit={}'.format(unit)\n", "\n", " # replacing the first '&' with '?'\n", " if len(query_params) > 0:\n", " query_params = '?{}'.format(query_params[1:])\n", "\n", " uri = \"/v1.0/gas/transit/capacities/monthly/{}\".format(query_params)\n", " print(uri)\n", "\n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "35da3171", "metadata": {}, "source": [ "#### JSON Response" ] }, { "cell_type": "code", "id": "883d9907", "metadata": {}, "source": [ "# latest release for one route\n", "json_data = fetch_monthly_capacity(access_token, route_id='ttf-the', capacity_type='firm', limit=1)\n", "json_data" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "d7e1240f", "metadata": {}, "source": [ "# checking metaData\n", "json_data['metaData']" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "f3373593", "metadata": {}, "source": [ "#### CSV Response\n", "\n", "One route, every release in the last 30 days." ] }, { "cell_type": "code", "id": "f7ecbf65", "metadata": {}, "source": [ "today = datetime.datetime.now().strftime('%Y-%m-%d')\n", "start_30d = (datetime.datetime.now() - datetime.timedelta(days=30)).strftime('%Y-%m-%d')\n", "\n", "csv_data = fetch_monthly_capacity(access_token, route_id='ttf-the', start=start_30d, end=today, format='csv')\n", "csv_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "b730215a", "metadata": {}, "source": [ "#### Latest release for all routes\n", "\n", "Leave out the route filters and use `limit=1` to retrieve every route for the latest release." ] }, { "cell_type": "code", "id": "71310a46", "metadata": {}, "source": [ "latest_data = fetch_monthly_capacity(access_token, limit=1, format='csv')\n", "latest_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "ab57ad7a", "metadata": {}, "source": [ "## 4. Daily Capacity\n", "\n", "The same data by gas day. It takes the same parameters as the monthly endpoint, plus `gas_day_start` and `gas_day_end`, which filter the gas days inside each release.\n", "\n", "__N.B.:__ Without a route filter, this endpoint returns one release per call (`limit=1`). With a route filter, the `start`/`end` range can be up to 31 days." ] }, { "cell_type": "code", "id": "41192d51", "metadata": {}, "source": [ "def fetch_daily_capacity(access_token, origin=None, destination=None, route_id=None, capacity_type=None, start=None,\n", " end=None, gas_day_start=None, gas_day_end=None, limit=None, offset=None, unit=None,\n", " format='json'):\n", "\n", " query_params = ''\n", "\n", " if origin is not None:\n", " query_params += '&origin={}'.format(origin)\n", " if destination is not None:\n", " query_params += '&destination={}'.format(destination)\n", " if route_id is not None:\n", " query_params += '&route-id={}'.format(route_id)\n", " if capacity_type is not None:\n", " query_params += '&type={}'.format(capacity_type)\n", " if start is not None:\n", " query_params += '&start={}'.format(start)\n", " if end is not None:\n", " query_params += '&end={}'.format(end)\n", " if gas_day_start is not None:\n", " query_params += '&gas-day-start={}'.format(gas_day_start)\n", " if gas_day_end is not None:\n", " query_params += '&gas-day-end={}'.format(gas_day_end)\n", " if limit is not None:\n", " query_params += '&limit={}'.format(limit)\n", " if offset is not None:\n", " query_params += '&offset={}'.format(offset)\n", " if unit is not None:\n", " query_params += '&unit={}'.format(unit)\n", "\n", " # replacing the first '&' with '?'\n", " if len(query_params) > 0:\n", " query_params = '?{}'.format(query_params[1:])\n", "\n", " uri = \"/v1.0/gas/transit/capacities/daily/{}\".format(query_params)\n", " print(uri)\n", "\n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "d5998b2c", "metadata": {}, "source": [ "#### JSON Response" ] }, { "cell_type": "code", "id": "dd9db1d6", "metadata": {}, "source": [ "# first week of next month, from the latest release\n", "next_month = (datetime.datetime.now().replace(day=1) + datetime.timedelta(days=32)).replace(day=1)\n", "gas_day_start = next_month.strftime('%Y-%m-%d')\n", "gas_day_end = (next_month + datetime.timedelta(days=6)).strftime('%Y-%m-%d')\n", "\n", "daily_json = fetch_daily_capacity(access_token, route_id='ttf-the', capacity_type='firm', limit=1,\n", " gas_day_start=gas_day_start, gas_day_end=gas_day_end)\n", "daily_json" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "28a18603", "metadata": {}, "source": [ "# checking metaData\n", "daily_json['metaData']" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "f3646591", "metadata": {}, "source": [ "#### CSV Response\n", "\n", "One route, every release in the last 30 days." ] }, { "cell_type": "code", "id": "f9e74714", "metadata": {}, "source": [ "daily_data = fetch_daily_capacity(access_token, route_id='ttf-the', start=start_30d, end=today, format='csv')\n", "daily_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "3172063e", "metadata": {}, "source": [ "#### Latest release for all routes\n", "\n", "Leave out the route filters to retrieve every route for the latest release (around 42,000 rows)." ] }, { "cell_type": "code", "id": "46e26f62", "metadata": {}, "source": [ "daily_latest = fetch_daily_capacity(access_token, limit=1, format='csv')\n", "daily_latest.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "a207f33e", "metadata": {}, "source": [ "## 5. Override Impact\n", "\n", "Route capacity with and without Spark overrides, side by side, for the routes where an override is in place. These endpoints take the same parameters as the monthly and daily endpoints, plus `indicator` (`technical`, `booked` or `available`)." ] }, { "cell_type": "code", "id": "ee9e0d48", "metadata": {}, "source": [ "def fetch_monthly_override_impact(access_token, origin=None, destination=None, route_id=None, capacity_type=None,\n", " indicator=None, start=None, end=None, limit=None, offset=None, unit=None,\n", " format='json'):\n", "\n", " query_params = ''\n", "\n", " if origin is not None:\n", " query_params += '&origin={}'.format(origin)\n", " if destination is not None:\n", " query_params += '&destination={}'.format(destination)\n", " if route_id is not None:\n", " query_params += '&route-id={}'.format(route_id)\n", " if capacity_type is not None:\n", " query_params += '&type={}'.format(capacity_type)\n", " if indicator is not None:\n", " query_params += '&indicator={}'.format(indicator)\n", " if start is not None:\n", " query_params += '&start={}'.format(start)\n", " if end is not None:\n", " query_params += '&end={}'.format(end)\n", " if limit is not None:\n", " query_params += '&limit={}'.format(limit)\n", " if offset is not None:\n", " query_params += '&offset={}'.format(offset)\n", " if unit is not None:\n", " query_params += '&unit={}'.format(unit)\n", "\n", " # replacing the first '&' with '?'\n", " if len(query_params) > 0:\n", " query_params = '?{}'.format(query_params[1:])\n", "\n", " uri = \"/v1.0/gas/transit/capacities/monthly/override-impact/{}\".format(query_params)\n", " print(uri)\n", "\n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "2b59cf1b", "metadata": {}, "source": [ "def fetch_daily_override_impact(access_token, origin=None, destination=None, route_id=None, capacity_type=None,\n", " indicator=None, start=None, end=None, gas_day_start=None, gas_day_end=None,\n", " limit=None, offset=None, unit=None, format='json'):\n", "\n", " query_params = ''\n", "\n", " if origin is not None:\n", " query_params += '&origin={}'.format(origin)\n", " if destination is not None:\n", " query_params += '&destination={}'.format(destination)\n", " if route_id is not None:\n", " query_params += '&route-id={}'.format(route_id)\n", " if capacity_type is not None:\n", " query_params += '&type={}'.format(capacity_type)\n", " if indicator is not None:\n", " query_params += '&indicator={}'.format(indicator)\n", " if start is not None:\n", " query_params += '&start={}'.format(start)\n", " if end is not None:\n", " query_params += '&end={}'.format(end)\n", " if gas_day_start is not None:\n", " query_params += '&gas-day-start={}'.format(gas_day_start)\n", " if gas_day_end is not None:\n", " query_params += '&gas-day-end={}'.format(gas_day_end)\n", " if limit is not None:\n", " query_params += '&limit={}'.format(limit)\n", " if offset is not None:\n", " query_params += '&offset={}'.format(offset)\n", " if unit is not None:\n", " query_params += '&unit={}'.format(unit)\n", "\n", " # replacing the first '&' with '?'\n", " if len(query_params) > 0:\n", " query_params = '?{}'.format(query_params[1:])\n", "\n", " uri = \"/v1.0/gas/transit/capacities/daily/override-impact/{}\".format(query_params)\n", " print(uri)\n", "\n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "da92f83b", "metadata": {}, "source": [ "#### JSON Response" ] }, { "cell_type": "code", "id": "810ea61c", "metadata": {}, "source": [ "# latest release for one route - returns an empty list if the route has no override in place\n", "impact_json = fetch_monthly_override_impact(access_token, route_id='psv-sivtp', indicator='available', limit=1)\n", "impact_json" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "cd293717", "metadata": {}, "source": [ "#### CSV Response" ] }, { "cell_type": "code", "id": "8e1bbb33", "metadata": {}, "source": [ "# monthly - every route with an override, latest release\n", "impact_data = fetch_monthly_override_impact(access_token, limit=1, format='csv')\n", "impact_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "133efc1b", "metadata": {}, "source": [ "# daily - every route with an override, latest release\n", "daily_impact_data = fetch_daily_override_impact(access_token, limit=1, format='csv')\n", "daily_impact_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "020aad25", "metadata": {}, "source": [ "## 6. Override Log\n", "\n", "The log of every Spark override: each one added, updated or removed, with the reason and a link to the source." ] }, { "cell_type": "code", "id": "dd6bc868", "metadata": {}, "source": [ "def fetch_override_log(access_token, origin=None, destination=None, route_id=None, tso=None, capacity_type=None,\n", " indicator=None, change_type=None, start=None, end=None, limit=None, offset=None,\n", " format='json'):\n", "\n", " query_params = ''\n", "\n", " if origin is not None:\n", " query_params += '&origin={}'.format(origin)\n", " if destination is not None:\n", " query_params += '&destination={}'.format(destination)\n", " if route_id is not None:\n", " query_params += '&route-id={}'.format(route_id)\n", " if tso is not None:\n", " query_params += '&tso={}'.format(tso)\n", " if capacity_type is not None:\n", " query_params += '&type={}'.format(capacity_type)\n", " if indicator is not None:\n", " query_params += '&indicator={}'.format(indicator)\n", " if change_type is not None:\n", " query_params += '&change-type={}'.format(change_type)\n", " if start is not None:\n", " query_params += '&start={}'.format(start)\n", " if end is not None:\n", " query_params += '&end={}'.format(end)\n", " if limit is not None:\n", " query_params += '&limit={}'.format(limit)\n", " if offset is not None:\n", " query_params += '&offset={}'.format(offset)\n", "\n", " # replacing the first '&' with '?'\n", " if len(query_params) > 0:\n", " query_params = '?{}'.format(query_params[1:])\n", "\n", " uri = \"/v1.0/gas/transit/capacities/overrides/log/{}\".format(query_params)\n", " print(uri)\n", "\n", " content = do_api_get_query(\n", " uri=uri, access_token=access_token, format=format\n", " )\n", "\n", " if format == 'json':\n", " data = content\n", " elif format == 'csv':\n", " # if there's no data to show, returns an empty DataFrame and \"No Data to Show\" message\n", " if len(content) == 0:\n", " data = pd.DataFrame()\n", " print('No Data to Show')\n", " else:\n", " data = content.decode('utf-8')\n", " data = pd.read_csv(StringIO(data)) # automatically converting into a Pandas DataFrame when choosing CSV format\n", "\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "ebca81a8", "metadata": {}, "source": [ "#### JSON Response" ] }, { "cell_type": "code", "id": "e11e4cbf", "metadata": {}, "source": [ "# the most recent entry date\n", "log_json = fetch_override_log(access_token, limit=1)\n", "log_json" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "94102e79", "metadata": {}, "source": [ "#### CSV Response" ] }, { "cell_type": "code", "id": "5b0218e7", "metadata": {}, "source": [ "log_data = fetch_override_log(access_token, format='csv')\n", "log_data.head(5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "36016544", "metadata": {}, "source": [ "## N.B. Historical Data Limits\n", "\n", "A maximum of 1 year's worth of release dates can be called at one time on the monthly endpoints, and 31 days on the daily endpoints.\n", "\n", "The function below calls the historical data one window at a time and combines it into one Pandas DataFrame. It works with any of the capacity and override impact functions above: set `w=31` for the daily endpoints. Use a route filter here: the daily endpoints need one for a date range, and requests for all routes over a long date range are too large to return in one call." ] }, { "cell_type": "code", "id": "8fdee673", "metadata": {}, "source": [ "def loop_historical_data(access_token, fetch_function, start, end, w=365, **params):\n", "\n", " hist_diff = (datetime.datetime.strptime(end, '%Y-%m-%d') - datetime.datetime.strptime(start, '%Y-%m-%d')).days\n", " t = 0\n", "\n", " while t < hist_diff:\n", " # initialising dataframe\n", " if t == 0 and hist_diff>w:\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=w), '%Y-%m-%d')\n", " hist_df = fetch_function(access_token, start=start, end=diff_end, format='csv', **params)\n", "\n", " elif t==0 and hist_diff<=w:\n", " hist_df = fetch_function(access_token, start=start, end=end, format='csv', **params)\n", "\n", " # appending additional historical data\n", " else:\n", " if t < hist_diff-w:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=t+1), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=w), '%Y-%m-%d')\n", " historical_addition = fetch_function(access_token, start=diff_start, end=diff_end, format='csv', **params)\n", "\n", " try:\n", " hist_df = pd.concat([hist_df,historical_addition])\n", " #exception if hist_df is empty\n", " except:\n", " hist_df = historical_addition.copy()\n", " else:\n", " diff_start = datetime.datetime.strftime(datetime.datetime.strptime(start, '%Y-%m-%d') + pd.Timedelta(days=t+1), '%Y-%m-%d')\n", " diff_end = datetime.datetime.strftime(datetime.datetime.strptime(diff_start, '%Y-%m-%d') + pd.Timedelta(days=hist_diff-t), '%Y-%m-%d')\n", " historical_addition = fetch_function(access_token, start=diff_start, end=diff_end, format='csv', **params)\n", "\n", " hist_df = pd.concat([hist_df, historical_addition])\n", "\n", " #looping by window\n", " t += w\n", "\n", " if len(hist_df) > 0:\n", " hist_df['ReleaseDate'] = pd.to_datetime(hist_df['ReleaseDate'])\n", " hist_df = hist_df.drop_duplicates()\n", " hist_df = hist_df.sort_values('ReleaseDate')\n", "\n", " return hist_df" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "58e0c9ee", "metadata": {}, "source": [ "today = datetime.datetime.now().strftime('%Y-%m-%d')\n", "\n", "# monthly capacity - 1 year at a time\n", "hist_df = loop_historical_data(access_token, fetch_monthly_capacity, start='2026-08-18', end=today, route_id='ttf-the')\n", "hist_df" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "c7695269", "metadata": {}, "source": [ "# daily capacity - 31 days at a time, needs a route filter\n", "daily_hist_df = loop_historical_data(access_token, fetch_daily_capacity, start='2026-08-18', end=today, w=31, route_id='ttf-the')\n", "daily_hist_df" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "7fc4860e", "metadata": {}, "source": [ "# monthly override impact - 1 year at a time\n", "impact_hist_df = loop_historical_data(access_token, fetch_monthly_override_impact, start='2026-08-18', end=today, route_id='psv-sivtp')\n", "impact_hist_df" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "id": "cc5bc1c5", "metadata": {}, "source": [ "# daily override impact - 31 days at a time, needs a route filter\n", "daily_impact_hist_df = loop_historical_data(access_token, fetch_daily_override_impact, start='2026-08-18', end=today, w=31, route_id='psv-sivtp')\n", "daily_impact_hist_df" ], "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.9" } }, "nbformat": 4, "nbformat_minor": 5 }