Community Data Science Workshops (Fall 2014)/Day 2 lecture: Difference between revisions

From OpenHatch wiki
Content added Content deleted
imported>Fhocutt
(draft lecture outline)
imported>Fhocutt
mNo edit summary
Line 1: Line 1:
== Lecture Outline ==
== Lecture Outline ==
Introduction and context
Introduction and context
* You know how to use Python now. Congratulations!
* You can write some tools in Python now. Congratulations!
* Today we'll learn how to find/create data sets
* Today we'll learn how to find/create data sets
* Next week we'll get into data science (asking and answering questions)
* Next week we'll get into data science (asking and answering questions)
Line 10: Line 10:
* How do we use one to fetch interesting datasets?
* How do we use one to fetch interesting datasets?
* How do we write programs that use the internet?
* How do we write programs that use the internet?
* How do we use the placekitten API to fetch kitten pictures?
* How can we use the placekitten API to fetch kitten pictures?
* Introduction to structured data (JSON)
* Introduction to structured data (JSON)
* How do we use APIs in general?
* How do we use APIs in general?
Line 17: Line 17:
* API: a structured way for programs to talk to each other (aka an interface for programs)
* API: a structured way for programs to talk to each other (aka an interface for programs)
* Web APIs: like a website your programs can visit (you:a website::your program:a web API)
* Web APIs: like a website your programs can visit (you:a website::your program:a web API)
* You send a request to the API, it gives you data back
* Using APIs: your program sends a request, the API sends data back
** Where do you direct my request? The site's API endpoint.
** Where do you direct your request? The site's API endpoint.
*** For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
*** For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
** How do I write my request? Put together a URL; it will be different for different web APIs.
** How do I write my request? Put together a URL; it will be different for different web APIs.
Line 26: Line 26:
** What do you get back?
** What do you get back?
*** Structured data (usually in the JSON format)
*** Structured data (usually in the JSON format)
** Congratulations, you have data!
** How do you understand (i.e. parse) the data?
*** There's a module for that!


#** way to ask for data (almost always a URL)
#** way to get data back (almost always in a format called JSON)
#** every API is different, and documented
#* to use APIs to build a dataset we will need:
#* to use APIs to build a dataset we will need:
#** all our tools from last session: variables, etc
#** all our tools from last session: variables, etc
Line 41: Line 39:
* Navigating in the terminal and using it to run programs
* Navigating in the terminal and using it to run programs
* Writing Python:
* Writing Python:
* variables
** using variables to manipulate data
** types of data: strings, integers, lists, dictionaries
* lists
⚫
** if statements
* dictionaries
⚫
** for loops
⚫
* if statements
⚫
** printing
⚫
#* for loops
** importing modules, so you can use code other people have written for you!
⚫
#* printing
#* modules


# New programming concepts:
# New programming concepts:

Revision as of 04:47, 10 November 2014

Lecture Outline

Introduction and context

  • You can write some tools in Python now. Congratulations!
  • Today we'll learn how to find/create data sets
  • Next week we'll get into data science (asking and answering questions)

Outline:

  • What did we learn in Session 1?
  • What is an API?
  • How do we use one to fetch interesting datasets?
  • How do we write programs that use the internet?
  • How can we use the placekitten API to fetch kitten pictures?
  • Introduction to structured data (JSON)
  • How do we use APIs in general?

What is a (web) API?

  • API: a structured way for programs to talk to each other (aka an interface for programs)
  • Web APIs: like a website your programs can visit (you:a website::your program:a web API)
  • Using APIs: your program sends a request, the API sends data back
    • Where do you direct your request? The site's API endpoint.
    • How do I write my request? Put together a URL; it will be different for different web APIs.
      • Check the documentation, look for code samples
    • How do you send a request?
      • Python has modules you can use, like requests (they make HTTP requests)
    • What do you get back?
      • Structured data (usually in the JSON format)
    • How do you understand (i.e. parse) the data?
      • There's a module for that!
    • to use APIs to build a dataset we will need:
      • all our tools from last session: variables, etc
      • the ability to open urls on the web
      • the ability to create custom URLS
      • the ability to save to files
      • the ability to understand (i.e., parse) JSON data that APIs usually give us

What did we learn in Session 1?

  • Navigating in the terminal and using it to run programs
  • Writing Python:
    • using variables to manipulate data
    • types of data: strings, integers, lists, dictionaries
    • if statements
    • for loops
    • printing
    • importing modules, so you can use code other people have written for you!
  1. New programming concepts:
    • requests
    • interpolate variables into a string using % and %()s
    • open files and write to them
  2. placekitten.com
    • API that takes specially crafted URLs and gives appropriately sized picture of kittens
    • example of placekitten in browser
      • visit the API documentation
      • kittens of different sizes
      • kittens in greyscale or color
    • show how to use place
    • write a small program to grab arbitrary square from placekitten by asking for the size on standard in
  1. JSON file (JavaScript Object Notation)
    • what is json: useful for more structure data
    • import json; json.loads()
    • like Python (except no single quotes)
    • simple lists, dictionaries
    • can reflect more complicated data structures
    • Example file at http://mako.cc/cdsw.json
    • download it and parse it
  1. Other APIs
    • every API is different, so read the documentation!
  • If the documentation isn't helpful, search online!
  • for popular APIs, there are python modules that help you make requests and parse json!
  • rate limiting
  • authenticaiton
  • text encoding issues
Retrieved from "https://wiki.openhatch.org/wiki/Community_Data_Science_Workshops_(Fall_2014)/Day_2_lecture?oldid=15301"