View source for Monitoring (moved)

{{Hacking_OpenHatch}}

== Monitoring ==

=== The basics===
* <code>linode.openhatch.org</code> is the main OpenHatch box, which runs the website.
* <code>linode2.openhatch.org</code> is the secondary server for OpenHatch. It hosts the Hudson continuous integration server, as well as Nagios!
* The Nagios configuration is owned by a user called ''nagios'' on ''linode2.openhatch.org''.

=== Access ===

* We use ssh keys for login. 
* If you want SSH access to that account, file a bug requesting it, and attach an SSH key. You should hear back within 2 days; if you don't hear back by then, try to find paulproteus or jesstess on IRC.
* Then you can do:

 ssh nagios@linode2.openhatch.org

* You'll know it's working if you are logged in. If you see a "Password:" prompt, then it is not working.

=== Notifications ===

* Nagios notifications go to [http://lists.openhatch.org/mailman/listinfo/monitoring monitoring@lists.openhatch.org]. Anyone can subscribe to this list or read its archives.

=== Making changes ===

In brief, here's what you need to know:

* Edit files in ~nagios/
* Once you know what changes you want to make, create a local branch with those changes:
 git checkout -b my_changes
* As you make changes, make meaningful commits. Also, tell "git commit" to use your identity:
 git commit --author="Some Body <some.body@example.com>"
* After you have made the changes, ask someone to review them and merge the changes to ''master''.
* '''Rationale''': If you stick to the above process, it is fairly easy to roll back to the "master" branch of the Nagios configuration.
* '''History''': We came up with this process during [https://openhatch.org/bugs/issue332 issue332].

=== Viewing the web interface, and handling the daemon ===

* On <code>linode2</code>, <code>~nagios/secrets/</code> contains the mailman and Nagios web interface passwords.
* View the Nagios web interface at <code>http://linode2.openhatch.org/nagios3/</code>
* To restart the Nagios daemon, run 

 sudo /etc/init.d/nagios3 restart

===In case of emergency===

* See [[Emergency operations for the openhatch server]]. People with ssh keys set up for the Linode Shell (Lish) can reboot the box and have other limited emergency capabilities.

== TODOs ==

# Send Nagios notifications to IRC (<code>#openhatch-auto</code>?)?
# Make the Nagios web interface world-viewable. 
# Version the monitoring configurations.
# Send SMS alerts to people who want them.
# Add historical trending (Munin)?

== Related ==

* See also [[Emergency operations for the openhatch server]]
* See also the page about the [[Login team]]