Django for Designers/Adding models

Part 3: Models, our database, and making it visible
Remember to make a new branch for section 3 based off the official branch!

====Introduction to databases and the ORM, or: finally, something we couldn't've done with plain HTML/CSS/JS!====

Okay, being able to design your URLs however you want, without them having to correspond to your actual file structure, is pretty neat. But besides that, we haven't done anything yet that you couldn't do with just HTML, CSS, and JavaScript. Time to fix that.

One key thing that separates web apps from web pages is that apps typically store data somewhere. The code that runs the app chooses what data seems most important to show at the moment. When writing an app using Django, we configure the storage through Django models.

Every Django app comes with a models.py in which you list each kind of data you want to store. These are configured through Python classes (to name the kind of data) with a sequence of attributes (which control the pieces of data that make up the model). If you're familiar with, or interested in, SQL, these correspond to tables and columns respectively.

The models that you define here are built on top of Django's object-relational mapper, or ORM. Because you define your models as Python objects, you can effectively write queries against a relational database by instead coding in Python with reference to your objects. Django then maps your Python code into relational database queries in SQL.

By configuring your data access in Python, Django makes it easy to take advantage of your data layout in other places. For example, you'll see later how the ORM makes it easy to automatically generate forms that ask for exactly the information you need. It is also generally convenient to write your entire app in one language, rather than being required to switch to SQL to do queries. The ORM also handles "escaping," which makes your SQL queries handle Unicode and other strange characters properly. These conveniences let you avoid mojibake-style data corruption and SQL injection attacks effortlessly.

Creating a basic model
Let's make it so we can store some real live data in our application! In our simple bookmarking app, we’ll create two models: Bookmarks and Tags. What information do we need to store about these objects?

A bookmark has:
 * A URL
 * A title (optionally)
 * A timestamp for when the bookmark was made

A tag has:
 * The tag slug/name

A tag also needs to know which bookmarks it applies to.

How do we represent this information in a Django model?

Open up bookmarks/models.py in your editor.

What's going on here?
 * We've made a Bookmark class that inherits from Django's model class.
 * We gave it a url, which is a URLField (a field that expects a string that parses as a valid URL).
 * We gave it a timestamp, which is a DateTimeField (a field that expects a Python datetime object). auto_now_add means that the field will automatically set itself to the time when the model is created.
 * Finally, we gave it a title, which is a CharField (a field that expects a string of some length or less, in this case less than 200 characters). "blank=True" means that Django will permit this field to be blank. We also set a default of the empty string, so that if someone fails to specify a title we permit that.

Now let's add a tag model! In the same file, below the Bookmark model:

The tag slug is stored in another CharField, which we've seen before. But what's this ManyToManyField? Well, it's one type of field for denoting a relationship between two models.

In relational databases, there's two basic types of relationships. In a ForeignKey relationship, one model relates to one and only one other model. You could imagine this like a Car model and a Wheel model. Each Wheel instance belongs to one and only one Car, so it would have a ForeignKey field for its car. The car, of course, can have multiple wheels.

With a ManyToManyField, on the other hand, the relationship isn't exclusive for either of the models involved. For instance, you could imagine having a Pizza model and a Topping model. Each Pizza can have multiple Toppings, and each Topping can be on multiple Pizzas.

(There are other variants of these fields, but this is the basic concept. See https://docs.djangoproject.com/en/dev/ref/models/fields/#module-django.db.models.fields.related for more details for how these relationship fields work in Django.)

In this case, each Bookmark can have multiple Tags, and each Tag can apply to multiple Bookmarks. So we use a ManyToManyField to store that information.

Creating database tables for your app (carefully, with South)
When you first create your models, you might not always know exactly what fields your models will need in advance. Maybe someday your bookmarks app will have preview thumbnail images! Then you would want to add another field to the model to store that information. Maybe someday you'll decide that tracking all the timestamps is silly, and want to delete that field.

Unfortunately, Django (and most database-using software) can’t figure out how to handle model changes very well on its own. Fortunately, a Django app called South that we installed earlier can handle these database changes--called ‘migrations’--for us.

Now that we’ve made our first version of our models file, let’s set up our bookmarks app to work with South so that we can make migrations with it in the future!

On the command line, write:

As you can see, that’s created a migrations directory for us, and automatically made a new migration file inside it.

All we need to do now is apply our new migration:

Great! Now our database file knows about bookmarks and its new models, and if we need to change our models, South is set up to handle those changes. We’ll come back to South later.

IMPORTANT: You can't migrate an app if it's already been synced in the database using

. But you do need to run syncdb at least once before you use south (since south itself uses syncdb to give itself space in your database). That's why it's super important that when you run syncdb, south should be listed under INSTALLED_APPS, but none of your own apps should be, and after you add your app to INSTALLED_APPS, you must not run syncdb again until after you've already set up migrations with that app.

Add and commit all your work, including the migrations folder that South generated for you!

Add some bookmarks via the command line
Now, let’s hop into the interactive Python shell and play around with the free API ("Application programming interface" -- APIs aren't just data doodads that hip web startups provide for you, they're an important concept in software architecture.) that Django gives you. To invoke the Python shell, use this command:

We’re using this instead of simply typing “python”, because manage.py's shell sets up the project’s environment for you. “Setting up the environment” involves two things:
 * Making sure bookmarks (and any other apps you might have) are on the right path to be imported.
 * Setting the DJANGO_SETTINGS_MODULE environment variable, which gives Django the path to your settings.py file.

Once you’re in the shell, let's explore the database API. Let's import the model classes we just wrote:

List all the current Bookmarks:

How many bookmarks is this? It's an empty list, so zero!

Let's add a bookmark:

We could specify the value for timestamp as well here, but we don't have to since when we wrote its model definition, we specified that timestamp has a default value (of when the bookmark is created). We did specify a title for the bookmark, even though we didn't have to since it's an optional field.

Try getting the ID number of our new bookmark 'b' by typing:

Python has given you back your prompt because the value of b.id is None. That is because we have not yet saved the object; to the database, it does not exist.

Save your bookmark to the database. In Django, you have to call save explicitly.

Now, try again to get the id of the Bookmark instance. Because it’s been saved, it has an ID in the database now! Even though we didn't specify one in our models.py, every saved model instance automatically has an id field.

Access the database columns (Fields, in Django parlance) as Python attributes:

We can change the bookmark title by changing its title attribute, then calling save.

If we now ask Django to show a list of all the Bookmark objects available, we can see it's no longer an empty list!

Fix the hideous default model representation
Wait a minute!  is an utterly unhelpful representation of this object. Let’s fix that by editing the Bookmark model. Use your text editor to open the bookmarks/models.py file and adding a __unicode__ method to both Bookmark and Tag:

Let's see our new shiny __unicode__ methods in action. Start a new Python interactive shell by running:

Save and commit your changes.

Adding more data via the shell
We've created a bunch of bookmarks, but no tags! Let's change that.

Slicing and dicing Django data
There are many methods for searching across your Django models.

We can filter our bookmarks, for instance for ones with a particular URL:

Or for bookmarks which have titles:

If you try to use filter to search for a question that does not exist, filter will give you the empty list.

The get method, on the other hand, returns exactly one hit. If it finds zero matches, or more than one match, it will raise an exception.

More information on making queries with Django's ORM can be found in the Django docs at https://docs.djangoproject.com/en/dev/topics/db/queries/#retrieving-objects.

Part 3.5: Changing our mind and adding users
D'oh! You know what every social bookmarking app has, that ours doesn't have? Users!

I don't mean like the number of people using it--I mean a way to store different users' accounts and keep track of who owns which bookmarks. So let's change our app to include this feature.

Lucky for us, Django comes with an app for user accounts and authentication from the get-go! In fact, it's already installed. If you look back at your settings.py file, you'll see that in INSTALLED_APPS, there is an entry for 'django.contrib.auth'.

That's our built-in authentication app!

Let's open the Django shell and play with this app a bit.

Whaaaaat?? There's already a User here. How can that be?

You might recall making a 'superuser' account when you first set up your Django project. That superuser was, in fact, created using Django's built-in auth app.

What is our user account's id number?

Neato!

Add user field to bookmark
Now we need to create a relationship between the built-in User model and our Bookmark model.

First, in our models.py file, we need to import django.contrib.auth's built-in User model so that we can refer to it in our models.

Now we need to think--what kind of relationship do users and bookmarks have?

Well, a user can have multiple bookmarks. But (right now, anyway) a bookmark should only have one user. So that means that we should use a ForeignKey field to add the user to our Bookmark model.

While we're at it, let's update the __unicode__ method too to let us know to whom a bookmark belongs to.

Make a migration in South
Now that we've added a field to our model, we are going to need to create a database migration. South can help us do this!

To create our migration, run

Note that we're now using --auto instead of --initial (which we used back when we first wrote our models and set up our app to use South).

Eep! Before it will make our migration file, South wants some information from us:

We could in theory modify our models to specify a default value for author, or make it optional. But neither of those sound like good options. We *want* future bookmarks to be forced to have an author! So we'll choose 2, to set up a default value for our new author field just for the purposes of this migration.

We then need to come up with a default value. Well, right now there's only one User in our system who the sample bookmarks we'd entered so far could belong to--our superuser account, which (if you don't remember) had an ID number of 1.

So let's enter 1 for our default.

Remember, the first step creates the migration, but doesn't run it. So let's do what South says and run a command to apply our migration!

Save and commit your work to add users to your bookmarks app!

Templates and links for login/logout/etc
Django's auth app comes with built-in views, which we can use to handle login and logout functionality for our users. Once we point some URLs at them, that is.

First, let's edit urls.py:

Now if anyone goes to localhost:8000/login/, the built-in login view will get triggered.

Run your dev server and try that. What error do you see?

While there is a built-in login view, there is no built-in login template. We need to build one for it. While we could put it anywhere and tell login explicitly where to look, by default the login view expects the template to reside at templates/registration/login.html. So we may as well put it there.

Create a new file within the new registration directory called login.html, and put this inside:

  {% endblock %}

Note the hidden input with the name "next". This input tells the login view what URL to send the user to after they successfully log in. We have it set to '/', so it'll just take them back to the home page.

We need to add a logout view too! Let's do that. Back in urls.py:

The dictionary after the logout URL sends some extra arguments to the logout view. Specifically it tells the logout view where to send the user after they log out. We could make a special goodbye splash page or something, but nah, let's just send them back to the home page again.

Now we need to add a login/logout link to our site so people can actually use these views! We want this link to be at the top of every page, so we'll edit our base template:

Our template checks to see if there's a logged-in user, and if so, shows a hello message and a logout link. If the user isn't logged in, it shows a login link instead.

Check http://localhost:8000 and try logging in with the superadmin username and password you created before! It should work. :)

You might be wondering--where did the 'user' variable in the template come from? If you look at views.py, you'll notice we never added a user variable to our context dictionary. So how did this happen?

The answer is in the function we are using to render our templates, render. render automatically uses the request we sent it to create a Django RequestContext, which contains a bunch of extra context variables that get sent along to every view that uses a RequestContext. The Django auth app adds the current user to the RequestContext automatically. This is handy, since you don't want to have to look up the current user to every single view you ever write separately, just so the nav section on your website will work everywhere!

There are other functions that return an HTMLResponse, like render, but don't include a RequestContext. render_to_response is one common shortcut function that doesn't include it by default; another was the HttpResponse function we used earlier! Just something to remember--if you're trying to send a piece of data to almost every page in your web app, 1.) you probably want to use render in your views, and 2.) you want to find a way to add your data to your app's RequestContext. (https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.RequestContext says more about this!)

Save and commit your changes in adding login/logout functionality.

Modify views and templates to use model data
Let's edit our views and templates so they use real bookmark data from our database!

Add more data
First, let's add a bit more bookmark and tag data to our database. We'll use a Python loop to add a lot of bookmarks at once.

That will bring us to the Django management shell. Within that special Python prompt, type the following:

(Remember, any time you see ">>>", you don't have to type those characters.)

This tells Python we'll be accessing the user model, and then grabs the first user, storing it in a variable called me.

Keeping that Python prompt open, run the following commands:

This creates a number of bookmarks, tagged appropriately! You can exit the shell by typing:

Get views.py to talk to our models
Then, we'll need to have our views.py file import the bookmark model and send data to the index view.

We'll add an import statement:

And we'll edit index(request):

Wait! We don't actually want to load every bookmark in our database when we go to the front page. If we have lots of bookmarks, that will get slow and unwieldy quickly.

Instead, let's show the 10 most recent bookmarks:

We also want our tag view to show real bookmarks. We only want to show bookmarks that have been tagged with the given tag. So we'll edit the import statement we just added:

And alter the tag definition as well:

Change templates to handle bookmark data
Now, let's modify our templates to use this data.

In index.html, update just block content per the following. Make sure to keep the extends directive and the block subheader directive.

Instead of writing out each bookmark list element individually ahead of time, we are using a Django template language for loop to create a list element for each bookmark. Thus, we only have to specify the HTML formatting of our list elements once!

Start your development server and look at http://localhost:8000 now. You'll see that instead of the fake HTML we had before, each link's text is the unicode representation of each bookmark in your database. Cool, huh?

That's okay, but the programmer-friendly unicode representation isn't really what we want to go there. Let's fix that:

If the bookmark has a title, we'll make the title the link. Otherwise we'll show the URL. We also fill in the URL on the  tag, to make the link work.

We also want to fill in the other metadata, like tags and author info. To do that:

We check if there are any tags for the bookmark, and if so loop over the tags to put each of them in.

Additionally, we use Django's {% url %} tag to generate the URL for our tag's link. The URL tag takes first an argument which is the path of a particular view function (bookmarks.views.tag), then additional arguments for any input variables that the function expects to glean from the URL. (As you may recall, the tag view takes an argument tag_name, which is the name of the tag in question).

Why use {% url %} instead of just writing "/tags/{&#8288;{ tag.slug }}"? Django principle of DRY--Don't Repeat Yourself--means that we want to avoid duplicating work as much as possible. If later down the line we decided to change our URL structure so that tag pages would appear at "/bookmarks/by_tag/" instead, we'd have to go in and fix all these hard-coded URL patterns by hand. Using the {% url %} tag makes Django generate our URL for us, based on our urls.py file, so any changes we make automatically get propagated outward!

Run

and confirm that your tag links working and our other changes are visible. It's easy to typo or make a mistake with {% url %} tags, so we want to make sure everything is working.

The default timestamp formatting that Django gave us is pretty neat, but in our original mockup we just used the date--not this long timestamp. Fortunately, Django's built-in filters make it easy to format a date or time any way we want! Since we just want to show the date, we'll use the date filter:

We tell Django that we're using a filter via the |filter_name syntax. The arguments that come after the colon are a standard Python code for describing different ways of formatting dates. You can read more about the date filter and all the different formatting codes at https://docs.djangoproject.com/en/dev/ref/templates/builtins/#date. For our purposes, Y outputs the full four-digit year, while m and d outputs the month and the day as two digit numbers. The hyphens we put between them are included in the formatting, too -- if we wanted the date to use slashes instead, we'd simply write |date:"Y/m/d".

Spin up your dev server, if you haven't got it running already, and check out your changes! Then save and commit your precious work.

Updating the tag template, and dealing with errors
Let's update our tag.html template to use the same formatting as index.html. Since we're using the same formatting for the list of bookmarks, and there aren't any other content block differences between index.html and tag.html, we can simply modify our tag template to inherit from index.html. Edit tag.html to have only the following contents. (That means you can remove the mockup "block content" section in tags.html. Yay!)

Now we can go to http://localhost:8000/tags/funny and see a nicely styled list of bookmarks tagged with that tag!

What happens if we go to http://localhost:8000/tags/asdfjkl ?

Eek, a DoesNotExist error! That's not so great. Our application is erroring on one of our view lines:

Fortunately, Django has a shortcut function that can help us -- get_object_or_404. This function will attempt to get a Django model based on the parameters you give it, and if it fails, automatically throw a standard HTTP "not found" error, number 404.

Now http://localhost:8000/tags/asdfjkl will throw a nicer 404 error. We could even make a pretty 404.html template to handle such errors! There are also shortcuts in Django for 500 and 403 (Forbidden) errors, if you want to handle and style those as well.

Save and commit your error-catching work.

Let me show off
So far, we've all only been able to access our own Django-powered sites. In this section, you will see how to access my (the instructor's) super cool bookmarks site! (We'll talk later about how you can share your code the same way. It might require firewall configuration changes on your computer, so we save that complexity for later.)

Assuming that you and the instructor are on the same wifi network, you can visit a temporary website on the instructor's computer. Visit this link:

http://192.168.1.1:8000/

(If you are wondering exactly who can connect to the instructor's app, it is typically only people on the same wifi/wired network as her, rather than the whole Internet. This is due to the a technique in wide use called "network address translation", and it is not considered perfect security.)

When you visit the instructor's app, you are interacting with the database stored on her laptop. So all the changes made by other people in the room are reflected in what you see! So try not to be too silly.

Part 4: CRUD
Right now, we have a nice website that displays data from our database. Unfortunately, though, while we have a mock bookmark form in the app header, currently the only way to create new bookmarks is in the Python shell. That's not fun at all.

This section deals with "CRUD" functionality, which are key to all web applications. CRUD stands for Create, Read, Update, and Delete. We already have Reading bookmarks covered; now we need to handle Creating bookmarks!

Django forms
Django comes with some built-in classes that make it easy to create forms with built-in validation and other functionality. There are plain django.forms classes (which are useful for encapsulating forms-related processing functions all in one place) as well as ModelForm classes that create a form based on a Django model that can automagically save instances of that model. For this application, let's use ModelForms to make a form for our Bookmark model.

Making a basic, working ModelForm
Inside our bookmarks app folder, let's make a file named forms.py. Edit it to import ModelForm and our models, then create a BookmarkForm class:

The generated BookmarkForm class will have a form field for every Bookmark model field--the field type based on some defaults.

Let's add our BookmarkForm to our views and templates!

Edit views.py:

Then edit index.html to take advantage of the new form object in our context:

The .as_p call makes the form put each field inside a paragraph tag. There are similar methods for making the form appear inside a table or inside divs.

The other thing we need to do here is add a CSRF token to our form, since that isn't included by default. A CSRF token is a special bit of code, built into Django, that protects your site from Cross Site Request Forgeries. It tells Django that a POST from this form really came from this form, not some other malicious site. Django makes this super easy:

Just add the csrf_token tag inside your form, and it's good to go.

Also note that the form call doesn't include the external form tags or the submit button -- Django leaves those for you to write yourself. This makes things more flexible if you want to add additional elements inside the form tag, or if you want to combine multiple Django Form objects into one HTML form.

Spin up http://localhost:8000/, login if you're not logged in already, take a look at your beautiful new form!

Right now, if you try to submit bookmarks with this form, it'll just reload the page. That's because we haven't told the view to do anything with any form data that gets sent to it! Let's edit our index view in views.py to make it do something with this form's data:

What's going on here? We first check to see if the request method was a POST (as opposed to a GET, the usual method your browser uses when you're just reading a web page). If so, we use our ModelForm class to make an instance of the ModelForm using the data that we received via the POST from the HTML form's fields. If the form's built-in validator functions come back clean, we save the ModelForm, which makes a shiny new Bookmark in our database!

Save your work, then try making some new bookmarks via your new form. The page should reload, with your new bookmark at the top of the list! High five!

Save and commit your wonderful bookmark-creating Django form.

Customize your form fields
So our form works, but it doesn't look the way we originally expected. For one, there's a dropdown asking us to specify the bookmark's author. For another, we're missing a field for tags. We'll need to customize our BookmarkForm if we want it to look the way we want.

First, let's make the default author always the currently logged-in user. In views.py:

Now if you reload, thanks to the initial data we provided our BookmarkForm, you'll see that your account is chosen by default in the author dropdown.

Then let's hide the author field from the user. We'll do this in our ModelForm specification in forms.py:

Now if you restart your server, you'll see the author field appears to be gone! Instead of the default text input widget, we told Django to use a hidden field for the author, so we no longer see it.

Since titles aren't even required, the URL is the most important piece of data for a bookmark. Let's change the order of the fields, so the URL comes first. Again in forms.py

Save and commit your changes.

Save tags with our form
Finally, let's add a tags field:

Note that since tags isn't a field in the Bookmark model, we're going to need to make sure it gets saved separately. For that, let's go back to views.py:

What's happening here? First, we're saving our ModelForm to save our original bookmark, as is. Then we're accessing our ModelForm's cleaned_data attribute to look for our string of comma-delineated tags. (We use cleaned_data instead of data because that one was pre-sanitized when we ran form.is_valid.) We split the string on the commas, clean up the tags by removing excess whitespace, making them all lowercase, turning spaces into hyphens, and then using urllib to quote any remaining special characters. Then we use a model shortcut function called get_or_create. What get_or_create does is see if there's already a tag with this slug. If so, it returns us the old tag, plus a False argument since it didn't make anything new. If not, it creates a new tag with the slug, and returns that (plus True, since it did make a new tag).

Finally, we save our tag, then add our new bookmark to its bookmarks Many-to-Many attribute to link them together.

Phew! Save your work, let your development server automatically restart, and try adding a bookmark with some tags. It should work!

Make sure to commit your work.

CRUD with asynchronous Javascript
That's pretty cool, but what if we don't want to make our users reload the page every time we want them to see new data? Most web apps are going to involve asynchronous Javascript in one way or another.

Let's modify our application to send our new bookmarks asynchronously to the server, and make the new bookmark appear on the page without reloading!

(Warning: we'll be writing some Javascript in this section.)

First, we need to tell our templates to import some Javascript files. In base.html:

Right now, our static/js/script.js is blank. Let's write a quick and dirty JS function that gets triggered when someone submits the bookmarks form:

What this does is when the user submits the bookmarks form, it prevent the page from reloading (like it would normally), serializes all the form fields, POSTs them to localhost:8000, and upon successfully receiving a response from the server displays a message at the top of our list of bookmarks.

If you try this out, you'll see that our JS-y form makes real bookmarks, just like the old version of our form did! However, it's putting junk in our bookmarks list on the page; we have to reload to actually see the bookmark our ajax call created.

There's several ways we could fix this. We could make our views.py send back a JSON serialization of our bookmark data, and have our JS file turn that into a list element somehow (using either a JS template or a giant string). Or we could have Django do the template rendering for us, and send our JS script the raw HTML we want it to use. This is called AHAH (Asychronous HTML and HTTP), or sometimes PJAX.

For the sake of speed and simplicity, we'll go with the latter. First, though, we need to refactor our templates a little.

Create a new template called bookmark.html, and paste in the bookmark list element from index.html in there.

Note that unlike the other templates, this template doesn't inherit anything -- it's just a block of HTML with some template markup.

Then, rewrite index.html so the content block looks like this:

Reload the page. Nothing should have changed, in terms of how the page looks. We've just changed the structure of the templates. Using the include tag, the index template drops in the contents of bookmarks.html rendered with the values for a given bookmark, for each bookmark in the list.

Now we're ready to teach our view to send back a partial bit of HTML.

What are we doing here? First, we're turning our request.POST test into a true choice -- we no longer go on to send back the full index.html rendered template if the request is a POST. Second, if we manage to create a bookmark successfully, we send back just our bookmark.html -- not the full index.html -- rendered with the data for our new bookmark.

Reload your development server and your web page, and try creating a bookmark. It should appear right away on the page now!

Try submitting the form with a tag but no URL. Nothing happens, right? If you check the server log in your terminal window, you'll see an error message:

Eek. Our index view doesn't handle the case where the request is a POST, but the form doesn't validate, and Django noticed this and returned an error. Let's fix this:

Now in the error case, we cheekily send back some HTML containing the errors that Django found, so there's no circumstance in which this view fails to provide some sort of response. (If we wanted to go to more effort, we could send back the errors as JSON, teach our callback function how to tell the difference between JSON and plain HTML responses, and render the former differently, instead of sticking our form error messages in the bookmarks list. But we'll run with this for now.)

Save and commit your JS-ification work!

In the real world, if you were doing lots of this sort of manipulation, instead of AHAH you might want to be using a Javascript framework such as Backbone to avoid getting confused, messy code. You'd also want to use templates for the elements that your Javascript hooks are adding and modifying. Ideally, you'd want those templates to be the same ones that your Django application used, to avoid repeating yourself! There's a lot of ways Django users deal with these problems. One way to do this is to use Django as an API data engine and do all the routing, application logic, and template rendering--at least, on the JS-heavy pages--via a Javascript-based template language. Tastypie is a popular Django application for making APIs for this sort of thing. Another approach is to teach Django to speak the language of a JS-based templating language. Projects like django-handlebars or djangobars are examples of this approach!