Community Data Science Workshops (Spring 2014)/Saturday May 3rd lecture: Difference between revisions
Content added Content deleted
imported>Mako |
imported>Jtmorgan m (moved to wiki.communitydata.cc) |
||
(17 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
{{CDSW_Moved}} |
|||
== Lecture Outline == |
== Lecture Outline == |
||
# Introduction to APIs |
|||
⚫ | |||
⚫ | |||
#** way to ask for data (almost always a URL) |
|||
#** way to get data back (almost always in a format called JSON) |
|||
#** every API is different, and documented |
|||
#* to use APIs to build a dataset we will need: |
|||
#** all our tools from last session: variables, etc |
|||
#** the ability to open urls on the web |
|||
#** the ability to create custom URLS |
|||
#** the ability to save to files |
|||
#** the ability to understand (i.e., parse) JSON data that APIs usually give us |
|||
# Review material from last session |
# Review material from last session |
||
#* variables |
#* variables |
||
⚫ | |||
⚫ | |||
#* lists |
#* lists |
||
#* dictionaries |
#* dictionaries |
||
⚫ | |||
#* for loops |
#* for loops |
||
⚫ | |||
#* modules |
#* modules |
||
#* example python program |
|||
# New programming concepts: |
# New programming concepts: |
||
⚫ | |||
#* open a file and append to it |
|||
#* join a string together |
|||
⚫ | |||
#* from syntax for importing |
|||
#* urllib2 and urlopen |
#* urllib2 and urlopen |
||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
# API |
|||
⚫ | |||
⚫ | |||
# [http://placekitten.com/ placekitten.com] |
# [http://placekitten.com/ placekitten.com] |
||
#* API that takes specially crafted URLs and gives appropriately sized picture of kittens |
#* API that takes specially crafted URLs and gives appropriately sized picture of kittens |
||
#* example of placekitten in browser |
#* example of placekitten in browser |
||
#** visit the API documentation |
|||
#** kittens of different sizes |
#** kittens of different sizes |
||
#** kittens in greyscale or color |
#** kittens in greyscale or color |
||
#* show how to use place |
#* show how to use place |
||
#* write a small program to grab arbitrary square from placekitten by asking for the size on standard in |
#* write a small program to grab arbitrary square from placekitten by asking for the size on standard in |
||
⚫ | |||
# Wikipedia API |
|||
#* what is json: useful for more structure data |
|||
#* explain MediaWiki, exists on other wikis |
|||
⚫ | |||
#* navigate to [http://en.wikipedia.org/w/api.php api page] and show the documentation, point out examples |
|||
⚫ | |||
#* looking at the images within a page http://en.wikipedia.org/w/api.php?action=query&titles=Seattle&prop=images&imlimit=20&format=jsonfm |
|||
⚫ | |||
#* looking at within two pages http://en.wikipedia.org/w/api.php?action=query&titles=Seattle|Bellevue,_Washington&prop=images&imlimit=50&format=jsonfm |
|||
⚫ | |||
#* edit count http://en.wikipedia.org/w/api.php?action=query&list=users&ususers=Benjamin_Mako_Hill|Jtmorgan|Sj|Mindspillage&usprop=editcount&format=jsonfm |
|||
⚫ | |||
#* give me the content of the main page http://en.wikipedia.org/w/api.php?format=json&action=query&titles=Main%20Page&prop=revisions&rvprop=content |
|||
#* download it and parse it |
|||
# Other APIs |
# Other APIs |
||
#* every API is different, so read the documentation! |
#* every API is different, so read the documentation! |
||
#* for popular APIs, there are python modules that help you make requests and parse json! |
|||
#* rate limiting |
#* rate limiting |
||
#* authenticaiton |
#* authenticaiton |
Latest revision as of 22:04, 15 March 2015
Lecture Outline
- Introduction to APIs
- definition of API: just an interface for programs
- definition of web API
- way to ask for data (almost always a URL)
- way to get data back (almost always in a format called JSON)
- every API is different, and documented
- to use APIs to build a dataset we will need:
- all our tools from last session: variables, etc
- the ability to open urls on the web
- the ability to create custom URLS
- the ability to save to files
- the ability to understand (i.e., parse) JSON data that APIs usually give us
- Review material from last session
- variables
- lists
- dictionaries
- if statements
- for loops
- printing
- modules
- New programming concepts:
- urllib2 and urlopen
- interpolate variables into a string using % and %()s
- open files and write to them
- placekitten.com
- API that takes specially crafted URLs and gives appropriately sized picture of kittens
- example of placekitten in browser
- visit the API documentation
- kittens of different sizes
- kittens in greyscale or color
- show how to use place
- write a small program to grab arbitrary square from placekitten by asking for the size on standard in
- JSON file (JavaScript Object Notation)
- what is json: useful for more structure data
- import json; json.loads()
- like Python (except no single quotes)
- simple lists, dictionaries
- can reflect more complicated data structures
- Example file at http://mako.cc/cdsw.json
- download it and parse it
- Other APIs
- every API is different, so read the documentation!
- for popular APIs, there are python modules that help you make requests and parse json!
- rate limiting
- authenticaiton
- text encoding issues