Editing Monitoring (moved)

Jump to navigation Jump to search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 1: Line 1:
{{Hacking OpenHatch}}
+
{{Hacking_OpenHatch}}
  
We are updating our documentation system. This page is now included in our project package, and is automatically generated by sphinx at openhatch.readthedocs.org [[http://openhatch.readthedocs.org/en/latest/internals/monitoring.html Monitoring]
+
== Monitoring ==
 +
 
 +
=== The basics===
 +
* <code>linode.openhatch.org</code> is the main OpenHatch box, which runs the website.
 +
* <code>linode2.openhatch.org</code> is the secondary server for OpenHatch. It hosts Nagios!
 +
* <code>vm3.openhatch.org</code> is a third server, hosted at GPLHost, that runs the Jenkins continuous integration server.
 +
* The Nagios configuration is owned by a user called ''nagios'' on ''linode2.openhatch.org''.
 +
 
 +
=== Access ===
 +
 
 +
* We use ssh keys for login.
 +
* If you want SSH access to that account, file a bug requesting it, and attach an SSH key. You should hear back within 2 days; if you don't hear back by then, try to find paulproteus or jesstess on IRC.
 +
* Then you can do:
 +
 
 +
ssh nagios@linode2.openhatch.org
 +
 
 +
* You'll know it's working if you are logged in. If you see a "Password:" prompt, then it is not working.
 +
 
 +
=== Notifications ===
 +
 
 +
* Nagios notifications go to [http://lists.openhatch.org/mailman/listinfo/monitoring monitoring@lists.openhatch.org]. Anyone can subscribe to this list or read its archives.
 +
 
 +
=== Making changes ===
 +
 
 +
In brief, here's what you need to know:
 +
 
 +
* Edit files in ~nagios/
 +
* Once you know what changes you want to make, create a local branch with those changes:
 +
git checkout -b my_changes
 +
* As you make changes, make meaningful commits. Also, tell "git commit" to use your identity:
 +
git commit --author="Some Body <some.body@example.com>"
 +
* After you have made the changes, ask someone to review them and merge the changes to ''master''.
 +
* '''Rationale''': If you stick to the above process, it is fairly easy to roll back to the "master" branch of the Nagios configuration.
 +
* '''History''': We came up with this process during [https://openhatch.org/bugs/issue332 issue332].
 +
 
 +
=== Viewing the web interface, and handling the daemon ===
 +
 
 +
* On <code>linode2</code>, <code>~nagios/secrets/</code> contains the mailman and Nagios web interface passwords.
 +
* View the Nagios web interface at <code>http://linode2.openhatch.org/nagios3/</code>
 +
* To restart the Nagios daemon, run
 +
 
 +
sudo /etc/init.d/nagios3 restart
 +
 
 +
===In case of emergency===
 +
 
 +
* See [[Emergency operations for the openhatch server]]. People with ssh keys set up for the Linode Shell (Lish) can reboot the box and have other limited emergency capabilities.
 +
 
 +
== TODOs ==
 +
 
 +
# Send Nagios notifications to IRC (<code>#openhatch-auto</code>?)?
 +
# Make the Nagios web interface world-viewable.
 +
# Version the monitoring configurations.
 +
# Send SMS alerts to people who want them.
 +
# Add historical trending (Munin)?
 +
 
 +
== Related ==
 +
 
 +
* See also [[Emergency operations for the openhatch server]]
 +
* See also the page about the [[Login team]]

Please note that all contributions to OpenHatch wiki are considered to be released under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) (see OpenHatch wiki:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

Cancel Editing help (opens in new window)

Templates used on this page: