Bug tracker import code (moved): Difference between revisions

Content added Content deleted

Inline

Revision as of 23:45, 23 August 2010

Here's what you need to know:

Each style (like "Bugzilla", "Roundup", and so forth) of bug tracker has its code in mysite/customs/bugtrackers/style.
We write tests in mysite/customs/tests.py for each bug tracker type.

The Roundup code is instructive. Look at that in your favorite editor. You'll see:

class RoundupTracker
- This class's __init__() is built so that an instance has enough data that a call to grab() will start downloading bug data.

The rest of that file is trivial subclasses that pre-fill the data to __init__(). That way, they can be called once a day by the mysite/customs/management/commands/customs_daily_tasks.py .

So if you want to add a new Roundup-based bug tracker, add a new subclass to that file, and submit a patch.

(Wondering how to submit a patch? Read how we handle patches.)

@@ Line 1: / Line 1: @@
+Here's what you need to know:
-The code we use for crawling bug trackers lives in mysite/search/tasks/. Some of the imports are in __init__.py, however many have been migrated out to separate modules.  It's fairly straight-forward.
+* Each ''style'' (like "Bugzilla", "Roundup", and so forth) of bug tracker has its code in mysite/customs/bugtrackers/'''style'''.
-There are a few different designs of our bug tracker import code, and different bug trackers that we can import from use different styles. The most recent style is the only one I'll document here, since I like it the most so far. The Fedora Bugzilla "fitandfinish" importer is an example of that.
+* We write tests in mysite/customs/tests.py for each bug tracker type.
+The Roundup code is instructive. Look at that in your favorite editor. You'll see:
-It's separated into three tasks:
+* '''class RoundupTracker'''
-* One task accepts a Fedora Bugzilla bug ID as input. It checks if our copy of it is more than one day old. If so, it refreshes it by pulling the fresh data from the remote bug tracker. If we do not have a local copy of that Fedora bug at all, it downloads it and stores a copy in our database. It's called ''LookAtOneFedoraBug'' and lives in mysite/search/tasks/bugzilla_instances.py.
+** This class's __init__() is built so that an instance has enough data that a call to grab() will start downloading bug data.
+The rest of that file is trivial subclasses that pre-fill the data to __init__(). That way, they can be called once a day by the mysite/customs/management/commands/customs_daily_tasks.py .
-* One task is a "Periodic" task, basically a Python-based cron job. It wakes up every day and grabs a list of Fedora bugs from the Fedora bug tracker. For each such bug, it enqueues a LookAtOneFedoraBug instance. It's called LearnAboutNewFedoraFitAndFinishBugs.
+So if you want to add a new Roundup-based bug tracker, add a new subclass to that file, and submit a patch.
-* The final task is another Periodic task. It wakes up once a day and, for each bug in our bug tracker that seems to come from Fedora, it enqueues a LookAtOneFedoraBug task to make sure we look at that the bug. This task is needed so that we learn when bugs should be marked as CLOSED (note that the LearnAboutNewFedoraFitAndFinishBugs task would not look at CLOSED bugs since those wouldn't appear in the Bugzilla search -- ask me to reword this explaination if necessary!).
+(Wondering how to submit a patch? Read [[how we handle patches]].)
-Why all this talk about "tasks"? That way, these can each be small bits of code, and we can use a job queuing system to make sure we don't hammer the Fedora Bugzilla too much by downloading LOTS of bugs in parallel. We use the [http://ask.github.com/celery/ celery task queue] to manage these jobs.
-These are the workers, but they're not where the real bug tracker parsing work is done. That lives in '''mysite/customs/''', in various files there. We do write tests for the bug tracker importers, and those live in '''mysite/customs/tests.py'''. So far we don't much test the tasks, but instead the core import code that the tasks use.
-[[Category:Hacking OpenHatch]] [[Category:Out of date]]
+[[Category:Hacking OpenHatch]]