Bug tracker import code (moved): Difference between revisions

Content added Content deleted

Inline

Latest revision as of 18:44, 22 August 2012

This is a page about improving or modifying OpenHatch.

We call that "Hacking OpenHatch," and there is a whole category of pages about that.

We are updating our documentation system. This page is now included in our project package, and is automatically generated by sphinx at openhatch.readthedocs.org Bug tracker import code

@@ Line 1: / Line 1: @@
+{{Hacking OpenHatch}}
-The code we use for crawling bug trackers lives in mysite/search/tasks/. Some of the imports are in __init__.py, however many have been migrated out to separate modules.  It's fairly straight-forward.
+We are updating our documentation system. This page is now included in our project package, and is automatically generated by sphinx at openhatch.readthedocs.org [http://openhatch.readthedocs.org/en/latest/internals/index.html#bug-tracker-import-code Bug tracker import code]
-There are a few different designs of our bug tracker import code, and different bug trackers that we can import from use different styles. The most recent style is the only one I'll document here, since I like it the most so far. The Fedora Bugzilla "fitandfinish" importer is an example of that.
-It's separated into three tasks:
-* One task accepts a Fedora Bugzilla bug ID as input. It checks if our copy of it is more than one day old. If so, it refreshes it by pulling the fresh data from the remote bug tracker. If we do not have a local copy of that Fedora bug at all, it downloads it and stores a copy in our database. It's called ''LookAtOneFedoraBug'' and lives in mysite/search/tasks/bugzilla_instances.py.
-* One task is a "Periodic" task, basically a Python-based cron job. It wakes up every day and grabs a list of Fedora bugs from the Fedora bug tracker. For each such bug, it enqueues a LookAtOneFedoraBug instance. It's called LearnAboutNewFedoraFitAndFinishBugs.
-* The final task is another Periodic task. It wakes up once a day and, for each bug in our bug tracker that seems to come from Fedora, it enqueues a LookAtOneFedoraBug task to make sure we look at that the bug. This task is needed so that we learn when bugs should be marked as CLOSED (note that the LearnAboutNewFedoraFitAndFinishBugs task would not look at CLOSED bugs since those wouldn't appear in the Bugzilla search -- ask me to reword this explaination if necessary!).
-Why all this talk about "tasks"? That way, these can each be small bits of code, and we can use a job queuing system to make sure we don't hammer the Fedora Bugzilla too much by downloading LOTS of bugs in parallel. We use the [http://ask.github.com/celery/ celery task queue] to manage these jobs.
-These are the workers, but they're not where the real bug tracker parsing work is done. That lives in '''mysite/customs/''', in various files there. We do write tests for the bug tracker importers, and those live in '''mysite/customs/tests.py'''. So far we don't much test the tasks, but instead the core import code that the tasks use.
-[[Category:Hacking OpenHatch]] [[Category:Out of date]]