Difference between revisions of "Community Data Science Workshops (Spring 2014)/Saturday May 3rd lecture"

From OpenHatch wiki
Jump to navigation Jump to search
imported>Mako
imported>Jtmorgan
m (moved to wiki.communitydata.cc)
 
(9 intermediate revisions by one other user not shown)
Line 1: Line 1:
  +
{{CDSW_Moved}}
 
== Lecture Outline ==
 
== Lecture Outline ==
   
  +
# Introduction to APIs
# API
 
 
#* definition of API: just an interface for programs
 
#* definition of API: just an interface for programs
 
#* definition of web API
 
#* definition of web API
 
#** way to ask for data (almost always a URL)
 
#** way to ask for data (almost always a URL)
 
#** way to get data back (almost always in a format called JSON)
 
#** way to get data back (almost always in a format called JSON)
  +
#** every API is different, and documented
 
#* to use APIs to build a dataset we will need:
 
#* to use APIs to build a dataset we will need:
 
#** all our tools from last session: variables, etc
 
#** all our tools from last session: variables, etc
Line 13: Line 15:
 
#** the ability to understand (i.e., parse) JSON data that APIs usually give us
 
#** the ability to understand (i.e., parse) JSON data that APIs usually give us
 
# Review material from last session
 
# Review material from last session
#* variables, different types
+
#* variables
#* printing
 
#* if statements
 
 
#* lists
 
#* lists
 
#* dictionaries
 
#* dictionaries
 
#* if statements
 
#* for loops
 
#* for loops
 
#* printing
 
#* modules
 
#* modules
#* example python program
 
 
# New programming concepts:
 
# New programming concepts:
 
#* urllib2 and urlopen
 
#* urllib2 and urlopen
 
#* interpolate variables into a string using % and %()s
 
#* interpolate variables into a string using % and %()s
  +
#* open files and write to them
 
# [http://placekitten.com/ placekitten.com]
 
# [http://placekitten.com/ placekitten.com]
 
#* API that takes specially crafted URLs and gives appropriately sized picture of kittens
 
#* API that takes specially crafted URLs and gives appropriately sized picture of kittens
 
#* example of placekitten in browser
 
#* example of placekitten in browser
  +
#** visit the API documentation
 
#** kittens of different sizes
 
#** kittens of different sizes
 
#** kittens in greyscale or color
 
#** kittens in greyscale or color
Line 32: Line 35:
 
#* write a small program to grab arbitrary square from placekitten by asking for the size on standard in
 
#* write a small program to grab arbitrary square from placekitten by asking for the size on standard in
 
# JSON file (JavaScript Object Notation)
 
# JSON file (JavaScript Object Notation)
#* what is json
+
#* what is json: useful for more structure data
 
#* import json; json.loads()
 
#* import json; json.loads()
 
#* like Python (except no single quotes)
 
#* like Python (except no single quotes)
Line 38: Line 41:
 
#* can reflect more complicated data structures
 
#* can reflect more complicated data structures
 
#* Example file at http://mako.cc/cdsw.json
 
#* Example file at http://mako.cc/cdsw.json
#* open a file and write to it
+
#* download it and parse it
# Wikipedia API
 
#* explain MediaWiki, exists on other wikis
 
#* navigate to [http://en.wikipedia.org/w/api.php api page] and show the documentation, point out examples
 
#* looking at the images within a page http://en.wikipedia.org/w/api.php?action=query&titles=Seattle&prop=images&imlimit=20&format=jsonfm
 
#* looking at within two pages http://en.wikipedia.org/w/api.php?action=query&titles=Seattle|Bellevue,_Washington&prop=images&imlimit=50&format=jsonfm
 
#* edit count http://en.wikipedia.org/w/api.php?action=query&list=users&ususers=Benjamin_Mako_Hill|Jtmorgan|Sj|Mindspillage&usprop=editcount&format=jsonfm
 
#* give me the content of the main page http://en.wikipedia.org/w/api.php?format=json&action=query&titles=Main%20Page&prop=revisions&rvprop=content
 
 
# Other APIs
 
# Other APIs
 
#* every API is different, so read the documentation!
 
#* every API is different, so read the documentation!
  +
#* for popular APIs, there are python modules that help you make requests and parse json!
 
#* rate limiting
 
#* rate limiting
 
#* authenticaiton
 
#* authenticaiton

Latest revision as of 22:04, 15 March 2015

Page Moved
All material related to the Community Data Science Workshops have been moved from the OpenHatch wiki to a new dedicated wiki and this page is no longer being updated here. Please visit the new version of the page on the Community Data Science Collective wiki.

Lecture Outline

  1. Introduction to APIs
    • definition of API: just an interface for programs
    • definition of web API
      • way to ask for data (almost always a URL)
      • way to get data back (almost always in a format called JSON)
      • every API is different, and documented
    • to use APIs to build a dataset we will need:
      • all our tools from last session: variables, etc
      • the ability to open urls on the web
      • the ability to create custom URLS
      • the ability to save to files
      • the ability to understand (i.e., parse) JSON data that APIs usually give us
  2. Review material from last session
    • variables
    • lists
    • dictionaries
    • if statements
    • for loops
    • printing
    • modules
  3. New programming concepts:
    • urllib2 and urlopen
    • interpolate variables into a string using % and %()s
    • open files and write to them
  4. placekitten.com
    • API that takes specially crafted URLs and gives appropriately sized picture of kittens
    • example of placekitten in browser
      • visit the API documentation
      • kittens of different sizes
      • kittens in greyscale or color
    • show how to use place
    • write a small program to grab arbitrary square from placekitten by asking for the size on standard in
  5. JSON file (JavaScript Object Notation)
    • what is json: useful for more structure data
    • import json; json.loads()
    • like Python (except no single quotes)
    • simple lists, dictionaries
    • can reflect more complicated data structures
    • Example file at http://mako.cc/cdsw.json
    • download it and parse it
  6. Other APIs
    • every API is different, so read the documentation!
    • for popular APIs, there are python modules that help you make requests and parse json!
    • rate limiting
    • authenticaiton
    • text encoding issues