Community Data Science Workshops (Spring 2014)/Saturday May 3rd lecture

Lecture Outline

1. Introduction to APIs
   • definition of API: just an interface for programs
   • definition of web API
     • way to ask for data (almost always a URL)
     • way to get data back (almost always in a format called JSON)
     • every API is different, and documented
   • to use APIs to build a dataset we will need:
     • all our tools from last session: variables, etc
     • the ability to open urls on the web
     • the ability to create custom URLS
     • the ability to save to files
     • the ability to understand (i.e., parse) JSON data that APIs usually give us
2. Review material from last session
   • variables
   • lists
   • dictionaries
   • if statements
   • for loops
   • printing
   • modules
3. New programming concepts:
   • urllib2 and urlopen
   • interpolate variables into a string using % and %()s
   • open files and write to them
4. placekitten.com
   • API that takes specially crafted URLs and gives appropriately sized picture of kittens
   • example of placekitten in browser
     • visit the API documentation
     • kittens of different sizes
     • kittens in greyscale or color
   • show how to use place
   • write a small program to grab arbitrary square from placekitten by asking for the size on standard in
5. JSON file (JavaScript Object Notation)
   • what is json: useful for more structure data
   • import json; json.loads()
   • like Python (except no single quotes)
   • simple lists, dictionaries
   • can reflect more complicated data structures
   • Example file at http://mako.cc/cdsw.json
   • download it and parse it
6. Wikipedia API
   • explain MediaWiki, exists on other wikis
   • navigate to api page and show the documentation, point out examples
   • looking at the images within a page http://en.wikipedia.org/w/api.php?action=query&titles=Seattle&prop=images&imlimit=20&format=jsonfm
   • change the city with a custom URL
   • (if time) parse through the output
   • (if time) edit count http://en.wikipedia.org/w/api.php?action=query&list=users&ususers=Benjamin_Mako_Hill%7CJtmorgan%7CSj%7CMindspillage&usprop=editcount&format=jsonfm
   • (if time) give me the content of the main page http://en.wikipedia.org/w/api.php?format=json&action=query&titles=Main%20Page&prop=revisions&rvprop=content
7. Other APIs
   • every API is different, so read the documentation!
   • for popular APIs, there are python modules that help you make requests and parse json!
   • rate limiting
   • authenticaiton
   • text encoding issues