Community Data Science Workshops (Fall 2014)/Day 2 lecture: Difference between revisions

Content added Content deleted

Inline

Revision as of 04:54, 10 November 2014

Lecture Outline

Introduction and context

You can write some tools in Python now. Congratulations!
Today we'll learn how to find/create data sets
Next week we'll get into data science (asking and answering questions)

Outline

What did we learn in Session 1?
What is an API?
How do we use one to fetch interesting datasets?
How do we write programs that use the internet?
How can we use the placekitten API to fetch kitten pictures?
Introduction to structured data (JSON)
How do we use APIs in general?

What is a (web) API?

API: a structured way for programs to talk to each other (aka an interface for programs)
Web APIs: like a website your programs can visit (you:a website::your program:a web API)

How do we use an API to fetch datasets?

Basic idea: your program sends a request, the API sends data back

Where do you direct your request? The site's API endpoint.
- For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
How do I write my request? Put together a URL; it will be different for different web APIs.
- Check the documentation, look for code samples
How do you send a request?
- Python has modules you can use, like requests (they make HTTP requests)
What do you get back?
- Structured data (usually in the JSON format)
How do you understand (i.e. parse) the data?
- There's a module for that!

What did we learn in Session 1?

Navigating in the terminal and using it to run programs
Writing Python:
- using variables to manipulate data
- types of data: strings, integers, lists, dictionaries
- if statements
- for loops
- printing
- importing modules, so you can use code other people have written for you!

How do we write Python programs that make web requests?

New programming concepts:
- requests
- interpolate variables into a string using % and %()s
- open files and write to them

- to use APIs to build a dataset we will need:
  - all our tools from last session: variables, etc
  - the ability to open urls on the web
  - the ability to create custom URLS
  - the ability to save to files
  - the ability to understand (i.e., parse) JSON data that APIs usually give us

How do we use an API to fetch kitten pictures?

placekitten.com
- API that takes specially crafted URLs and gives appropriately sized picture of kittens
- example of placekitten in browser
  - visit the API documentation
  - kittens of different sizes
  - kittens in greyscale or color
- show how to use place
- write a small program to grab arbitrary square from placekitten by asking for the size on standard in

Introduction to structured data (JSON)

JSON file (JavaScript Object Notation)
- what is json: useful for more structure data
- import json; json.loads()
- like Python (except no single quotes)
- simple lists, dictionaries
- can reflect more complicated data structures
- Example file at http://mako.cc/cdsw.json
- download it and parse it

Using other APIs

every API is different, so read the documentation!
If the documentation isn't helpful, search online!
for popular APIs, there are python modules that help you make requests and parse json!
rate limiting
authenticaiton
text encoding issues

@@ Line 18: / Line 18: @@
 * Web APIs: like a website your programs can visit (you:a website::your program:a web API)
-; How do we use an API to fetch our datasets?
+; How do we use an API to fetch datasets?
-Using APIs: your program sends a request, the API sends data back
+Basic idea: your program sends a request, the API sends data back
-** Where do you direct your request? The site's API endpoint.
+* Where do you direct your request? The site's API endpoint.
-*** For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
+** For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
-** How do I write my request? Put together a URL; it will be different for different web APIs.
+* How do I write my request? Put together a URL; it will be different for different web APIs.
-*** Check the documentation, look for code samples
+** Check the documentation, look for code samples
-** How do you send a request?
+* How do you send a request?
-*** Python has modules you can use, like <code>requests</code> (they make HTTP requests)
+** Python has modules you can use, like <code>requests</code> (they make HTTP requests)
-** What do you get back?
+* What do you get back?
-*** Structured data (usually in the JSON format)
+** Structured data (usually in the JSON format)
-** How do you understand (i.e. parse) the data?
+* How do you understand (i.e. parse) the data?
-*** There's a module for that!
+** There's a module for that!
-What did we learn in Session 1?
+; What did we learn in Session 1?
 * Navigating in the terminal and using it to run programs
 * Writing Python:
@@ Line 40: / Line 40: @@
 ** printing
 ** importing modules, so you can use code other people have written for you!
+; How do we write Python programs that make web requests?
 # New programming concepts:
@@ Line 53: / Line 55: @@
 #** the ability to understand (i.e., parse) JSON data that APIs usually give us
+; How do we use an API to fetch kitten pictures?
 # [http://placekitten.com/ placekitten.com]
 #* API that takes specially crafted URLs and gives appropriately sized picture of kittens
@@ Line 63: / Line 65: @@
 #* write a small program to grab arbitrary square from placekitten by asking for the size on standard in
+; Introduction to structured data (JSON)
 # JSON file (JavaScript Object Notation)
 #* what is json: useful for more structure data
@@ Line 72: / Line 75: @@
 #* download it and parse it
-# Other APIs
+; Using other APIs
-#* every API is different, so read the documentation!
+* every API is different, so read the documentation!
 * If the documentation isn't helpful, search online!
 * for popular APIs, there are python modules that help you make requests and parse json!