{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\\rightarrow$Run All).\n",
    "\n",
    "Make sure you fill in any place that says `YOUR CODE HERE` or \"YOUR ANSWER HERE\", as well as your name and collaborators below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"\"\n",
    "COLLABORATORS = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 23 - first half\n",
    "\n",
    "The main goals of this inclass activity are\n",
    "\n",
    "- get practice with curl in the terminal\n",
    "- learn how to use curl in an ipynb setting, with %%bash\n",
    "- practice with learning about APIs\n",
    "- practice with GET requests and file extensions\n",
    "- practice with POST\n",
    "\n",
    "In the reading, you learned about APIs for GitHub and TMDB. With this activity and the homework you'll learn about Kiva loans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding Paginated Results\n",
    "\n",
    "**Step 1**\n",
    "\n",
    "1. Read the Terms of Use Agreement: https://www.kiva.org/legal/terms\n",
    "2. Read the Code of Conduct: https://www.kiva.org/build/code-of-conduct\n",
    "3. Study the API: we've given you the info you need to avoid this in the current worksheet\n",
    "\n",
    "The API has changed in the past year to become more complicated, but you can still access data from Kiva using the old API approach. That old approach is decribed at the following link. This link uses the \"Way Back Machine\", which archives web pages from points back in time:\n",
    "\n",
    "https://web.archive.org/web/20190629032937/https://build.kiva.org/api\n",
    "\n",
    "**Step 2**\n",
    "\n",
    "Start playing with how to get data from http://api.kivaws.org\n",
    "\n",
    "API methods can be tested easily with most any browser. As an example, try out the loans/search method using HTML output:\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.html?status=fundraising\n",
    "\n",
    "API calls with the .html extension are designed for testing or debugging. If the browser or tool you are using easily supports viewing XML output you might try using the .xml extension instead:\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.xml?status=fundraising\n",
    "\n",
    "Try changing up some of the parameters and see how the search results change. What URL would you use to access the same data in JSON format?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e811eed5f4e5978effbd21e136fa9926",
     "grade": true,
     "grade_id": "cell-e9d61998a6961cb3",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the reading, it should be clear to you that we are using a *query parameter* corresponding to a Python dictionary `{'status':'fundraising'}`. Just like in the reading, we can use `&` to provide the URI with multiple query parameters at once. Here are two more that we commonly use (and that come up on your homework):\n",
    "\n",
    "- page\n",
    "  -  A number for the page of data to return (results are segmented into pages).\n",
    "- per_page\n",
    "  -  A number telling how many results per page you want to see\n",
    "\n",
    "Both of these parameters will hopefully be familiar to you from times you have used a search engine like Google.\n",
    "\n",
    "In our example link above, the default is to take you to Page 1 (out of 145 at the time of writing):\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.html?status=fundraising\n",
    "\n",
    "You can see \"Page 1 out of 145\" in the top line of the results. Note that, by the time you go to this link, it might have changed, if more loans were made and the data source updated correspondingly.\n",
    "\n",
    "To get the second page of results instead of the first, you can add a query parameter using `page`:\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.html?status=fundraising&page=2\n",
    "    \n",
    "Note that the first line now says \"Page 2 out of 145\". You can also change how many results you want to see per page, just like a search engine. For example, to see 100 results per page, you would do:\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.html?status=fundraising&per_page=100\n",
    "\n",
    "Note that when showing 100 results per page, you only need 29 pages to get through all the results, instead of 145. How would you modify this URI to show you the third page, with 50 pages per day?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fd37cfa10a70e6ed11475aedf7961f85",
     "grade": true,
     "grade_id": "cell-af0ad831ed9468e5",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some more parameters that the loans/search method can take:\n",
    "\n",
    "- status\n",
    "  -  Any of: fundraising,funded,in_repayment,paid,defaulted\n",
    "- gender\n",
    "  -  Any of: male,female\n",
    "- sector\n",
    "  -  Matches against a sector name such as agriculture\n",
    "- region\n",
    "  -  Any of: na,ca,sa,af,as,me,ee\n",
    "- country_code\n",
    "  -  Matches a two-digit ISO country code.\n",
    "- partner\n",
    "  -  Matches one or more partner IDs.\n",
    "- q\n",
    "  -  A general search string to match against various properties of loans\n",
    "- sort_by\n",
    "  -  Any of: popularity,loan_amount,oldest,expiration,newest, amount_remaining,repayment_term\n",
    "\n",
    "Here's how you'd make a request for all loans in Cambodia or Mongolia that are actively paying back, sorted by the amount of the loan:\n",
    "\n",
    "    http://api.kivaws.org/v1/loans/search.html?country_code=kh,mn&sort;_by=loan_amount&status;=in_repayment\n",
    "\n",
    "Please come up with three more examples that make use of the first eight parameters above (that is, everything except `page`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a7001d8107c16b575353ea493d485d04",
     "grade": true,
     "grade_id": "cell-ca7ff148c051caa7",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "bb54647da68c7e58fe72195842687460",
     "grade": true,
     "grade_id": "cell-b4cb353a51a266b3",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8ab8e358e44c81381b7ea38c3a41a79a",
     "grade": true,
     "grade_id": "cell-0ddf3a906570f85e",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Curl\n",
    "\n",
    "In all the examples above, you needed to physically copy and paste the URLs into a web browser to test your results. This is obviously terrible from a computer science perspective.\n",
    "\n",
    "One step in the right direction (to at least have a reproducible workflow all in one document) is to use `curl`. For example, with the first link given above, we have\n",
    "\n",
    "    curl --get --url http://api.kivaws.org/v1/loans/search.html?status=fundraising\n",
    "\n",
    "We can run this command in Jupyter via the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "curl --get --url http://api.kivaws.org/v1/loans/search.html?status=fundraising"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please use three `%%bash` or `!curl` cells below to run `curl` commands for the three links provided above in the examples about the query parameters `page=` and `per_page=`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6e19c2ee6395c7208d9f6824904e8a31",
     "grade": true,
     "grade_id": "cell-4909451708b58fac",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fdd09e17a810766ad98b027bd1a1e9fe",
     "grade": true,
     "grade_id": "cell-35ac756e8b72a9e5",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0703fc4784ca9e6090c38990c2afda34",
     "grade": true,
     "grade_id": "cell-c61463b9bbb10e95",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Programmatic\n",
    "\n",
    "While `curl` is fun and powerful, we have not illustrated yet how to actually store the results of a `curl` GET command for use in a program. Thankfully, we previously learned how to get results from the web into Python native data structures using the `requests` module. Please run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import io\n",
    "from lxml import etree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We return now to our Kiva loan problem.\n",
    "\n",
    "**Step 3**\n",
    "\n",
    "Write, in a global cell, the programmatic way to get the data as xml and yields the root Element of the xml tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://api.kivaws.org/v1/loans/search.xml\"\n",
    "searchTerms = {'status': 'fundraising'}\n",
    "resp = requests.get(url, params=searchTerms)\n",
    "print(resp.status_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldata = etree.parse(io.BytesIO(resp.content)).getroot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for child in xmldata:\n",
    "    print(child.tag, child.attrib, child.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4** Make the above a **function** `getRootKivas()` with no parameters that returns a Python data structure, or `None`, if there was a problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e175df426315f970397ceb544ea40c38",
     "grade": true,
     "grade_id": "cell-bdb5c296e38d17bf",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 5** Refine your function to take and use a **page** parameter, so `getRootKivas(p)` gives the results from page `p`. Please solve this by creating a URI with a `?` and bringing `p` into a format string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "69dda1915b2ad8fe253c24cf731e66d0",
     "grade": true,
     "grade_id": "cell-0cb2937531630834",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = getRootKivas(42)\n",
    "r.request.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5b** Now solve the same problem by passing a dictionary to `get`. Here is an example. Please write a general function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kiva_api = 'http://api.kivaws.org/v1/'\n",
    "endpoint = 'loans/newest.json'\n",
    "getargs = {'page': 42}\n",
    "r = req.get(kiva_api + endpoint, params=getargs)\n",
    "n = json.loads(r.text)\n",
    "print(n['paging'])\n",
    "print()\n",
    "print(n['loans'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a6abf9d4dc0e6f7a7b5cae3847fa5b68",
     "grade": true,
     "grade_id": "cell-e3e7306459aaf5c5",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 6** Refine your function further to take and use a **per_page** parameter, so `getRootKivas(p,n)` gives the results from page `p`, when each page has `n` results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "384c9de0f9e5dbde2342943659058b61",
     "grade": true,
     "grade_id": "cell-415481264087982f",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 7** Refine your function further to take and use a **sector** parameter, so `getRootKivas(p,n,s)` gives the results from page `p`, when each page has `n` results, and the sector is `s` (e.g., 'Agriculture')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "91b8cb9203e93b19035eab646ba209eb",
     "grade": true,
     "grade_id": "cell-1b1b897afb4890d6",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing cell\n",
    "r = getRootKivas(2,10,'agriculture')\n",
    "n = json.loads(r.text)\n",
    "print(n['paging'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 8** Refine your function further to take and use **theme** and **status** parameters, so `getRootKivas(p,n,sec,theme,stat)` gives the results from page `p`, when each page has `n` results, and the sector is `sec`, and the theme is `theme` (e.g., 'Higher Eduction') and the status is `stat` (e.g. 'funded')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f126a35f11bffe32441a99cc48a0f719",
     "grade": true,
     "grade_id": "cell-db48b0879afc478d",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing cell\n",
    "r = getRootKivas(1,10,'agriculture','Higher Education','funded')\n",
    "n = json.loads(r.text)\n",
    "print(n['paging'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 9** Refine your function further to take a parameter for the endpoint_type. In all the examples above, it was `newest`, but poking around on the Kiva website, `search` would have also worked. Think about what other types work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f81c4a431c64abd3f53056a3f323d3dc",
     "grade": true,
     "grade_id": "cell-0e01f8dd9e88d17d",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 10** Can you make your function even more general? For example, every invocation above seeks data along the path `http://api.kivaws.org/v1/loans/`. What else in that path can be modified to be a parameter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dfe24363a0c254a156d03ba055ba56d5",
     "grade": true,
     "grade_id": "cell-81db66f1d305853e",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A more in-depth example\n",
    "\n",
    "In the example above, we assumed the data would come to us in XML form. We now generalize that, then show how to build in query parameters.\n",
    "\n",
    "**Q1** Please write a function \n",
    "\n",
    "    kiva_newest(baseurl, apiobject, method, form = 'json')\n",
    "    \n",
    "that takes four string parameters (where the fourth is optional), builds a correct URL, and executes a `requests.get`, returning the result (or `None` if there is a problem). The fourth parameter is the format of the data. Please refer to the examples above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5696a3c69399e4a22eac34d775201115",
     "grade": true,
     "grade_id": "cell-c7bc169a49ddba50",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Testing cell\n",
    "\n",
    "baseurl = \"https://api.kivaws.org/v1\"\n",
    "apiobject = \"loans\"\n",
    "method = \"newest\"\n",
    "\n",
    "resp = kiva_newest(baseurl, apiobject, method)\n",
    "print(resp.text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "resp = kiva_newest(baseurl, apiobject, method,'json')\n",
    "print(resp.text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "resp = kiva_newest(baseurl, apiobject, method,'xml')\n",
    "print(resp.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By now we should be familiar with the query parameters for `page` and `per_page`. Another useful query parameter is `ids_only` which can be either True or False. Please take a moment to familiarize yourself with this parameter, e.g., by playing with the following URI:\n",
    "\n",
    "https://api.kivaws.org/v1/loans/newest.json?page=3&ids_only=true\n",
    "\n",
    "For more practice (in a different setting than the previous problem), please solve the following. Pay careful attention to the test invocation to understand the parameters.\n",
    "\n",
    "**Q2** Please write a function with five string parameters\n",
    "\n",
    "    kiva_query(result,page,pp,ids_only,endpoint)\n",
    "\n",
    "that uses `endpoint` and `result` to create a correct endpoint path for *newest* loans, then creates a dictionary for the other three parameters suitable for query parameters to go with the Kiva API. Please pass the URI and query parameters to `requests.get`, returning the result. Once you have a working version, please think about how to generalize it to make all arguments optional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "be1d76b26ec0bb1aaad62c1ce1e91b9c",
     "grade": true,
     "grade_id": "cell-7842c8700e611d27",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_end=\"https://api.kivaws.org/v1/loans\"\n",
    "\n",
    "r = kiva_query(result='xml', page=5, pp=10, ids_only='false',endpoint=my_end)\n",
    "r.request.path_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stripparser = etree.XMLParser(remove_blank_text=True)\n",
    "tree = etree.parse(io.BytesIO(r.content), stripparser)\n",
    "root = tree.getroot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "root = etree.parse(io.BytesIO(r.content)).getroot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(etree.tostring(root, pretty_print=True).decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q3** Please use the result from the previous problem, and XPath, to extract a list `sector_list` of `sectors` that appear in your query, e.g. \"Agriculture\", etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f3ad87b1fc973498b33b39c0a642bc2a",
     "grade": true,
     "grade_id": "cell-d2b39eb91a37196b",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sector_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice with POST\n",
    "\n",
    "Please visit the following link in Google Chrome and use the drop-down menus to select all years from 2002 to 2015\n",
    "\n",
    "http://www.rateinflation.com/consumer-price-index/usa-historical-cpi.php\n",
    "\n",
    "Now use Dev Tools so you can see what POST is really doing, and write down what you learned. \n",
    "\n",
    "**Q4** Write a function `makePostDict(from_year,to_year)` that takes the from year and to year (as strings) and returns a correct dictionary that could be sent with the POST.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "44f4924ea0c9bff6b117ea4c4b18ccb0",
     "grade": true,
     "grade_id": "cell-5609d9092cfa40da",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing cell\n",
    "D = makePostDict('2004','2011')\n",
    "print(D)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing cell\n",
    "endpoint = 'http://httpbin.org/post'\n",
    "r = requests.post(endpoint, data=D)\n",
    "r.request.body"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q5** Following what you learned in the book, formulate a `curl` POST for the example of 2002 to 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4233045e145bb92c5ceeee05d1e43dd7",
     "grade": true,
     "grade_id": "cell-02e55259541be790",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q6** Write a function `get_inflation(from_year,to_year)` that uses the `requests` module to issue a POST request whose body is obtained via a call to `makePostDict`, returning the result of the `requests.post` invocation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0831e4e06fefc94748616b5d447e2656",
     "grade": true,
     "grade_id": "cell-0bb09cf5806accee",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = get_inflation('2004','2011')\n",
    "r.status_code\n",
    "r.text\n",
    "r.request.url\n",
    "r.request.body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}