Overview

Kosmos has not been maintained very well lately, for several reasons, and for the common reasons: some have the abilities but lack the time, some have the time and interest but lack the skills. It’s about time to change this and to share skills and the time spent on server administration.

Meetings

We, the sysadmins of kosmos, and those taking part in the skill sharing sessions, meet on IRC, irc.indymedia.org, port 6667 (6697 for SSL encrypted chat), in channel “#kosmos”.

If you do not have an IRC client you can use the web chat instead. If you get an SSL error on this web page, there is another one which does not use encryption. However, you should try to use encryption so if the encrypted webchat does not work for you, be sure to read these instructions.

Topics

The following skill sharing session topics have been suggested:

  1. Introduction to docs.indymedia.org; update documentation on kosmos
    Sysadmin.KosmosSkillSharing
    Sysadmin.KosmosUpgrade200909
    Sysadmin.KosmosInfo
    Sysadmin Web
  2. Find a backup location
  3. Schedule regular offsite backups
  4. Server virtualization by example: Xen (Alster)
    Host vs Guest, Xen Hypervisor vs Dom0 vs DomU, hardware-assisted virtualization (HVM) vs para-virtualization (PVM), hardware support (Intel VT + AMD-V), Xen Dom0/DomU kernel patches, compare to KVM, compare to openvz/lxr/linux-vserver/chroot
  5. Introduction to the shell (BASH)
    What is BASH, how does it work? What’s the very basics?
    ls, pwd, cd, cat, less
  6. Introduction to GNU screen
    GNU screen allows us to login to the server and watch one of us executing commands (and their output) on the server, or join in, running commands on our own (and have the others watch as we do it). It also prevents running programs from failing just because our SSH session gets disconnected.
  7. Introduction to common shell commands pt 1
    nano or pico, vim, tail, file, df, free, w, tee
    Output redirection (‘>’ vs ‘>>’), pipes
  8. Introduction to common shell commands pt 2
    Text manipulation and regular expressions with cut, sed, awk, grep
  9. Introduction to regular expressions (Alster)
    Both simple and extended regular expressions will be discussed
  10. Introduction to common shell commands pt 3
    Networking with wget, netcat
    First steps with common application protocols: HTTP, SMTP
  11. Introduction to metche (and/or etckeeper)
    Metche and etckeeper are utilities which make system administration easier, easpecially when you administrate a server as a group of sysadmins. We shall install and configure one or both, get used to them and take provisions to make and keep everyone aware they are supposed to be used on this server (e.g. by adding notes to /etc/motd{,.tail})
  12. Introduction to APT, apt-get and aptitude
    APT is the way Debian organizes software to be installed/removed etc.
  13. Install cron-apt, apt-listbugs and (optionally) apt-listchanges
  14. Introduction to cron-apt
    Install and configure cron-apt. Cron-apt sends you email when there are important updates available for your server.
  15. Introduction to outbound-only mail servers
    We will pick and configure a sendmail compatible outbound-only mail server such as nullmailer, ssmtp etc., and configure it so that email for root is delivered to who shoudl actually receive it
  16. Introduction to logcheck
    Install the ‘logcheck’ log monitoring utility, configure it nicely, make it send email to the sysadmins
  17. Tweaking logcheck
    Logcheck usually requires some tweaking to make it less noisy and thus actually useful in the long term. Once we have received its first reports by email, we can configure it to ignore some events
  18. Securing !OpenSSHd
    Use public/private key authentication and disable password authentication (communicate this first!). Alternatively, install a utility to thwart brute force attacks against SSH (‘denyhosts’ etc.)
  19. Determine which user accounts are still in use, delete those which are not
    We currently have many user accounts on kosmos, some of which are probably no longer in use and should thus be removed or disabled to decrease the attack surface.
    Some accounts have already been deleted, and by Nov 15 2009 additional accounts can be deleted (cf. Alster’s Kosmos-sysad Account cleanup emails).
  20. Configure munin-node (and possibly munin, too)
    Per request from a Nadir member, munin-node is already installed and monitoring from double.nadir.org is taking place. The results are probably available at double.nadir.org but we do not currently have access credentials for it. So we may want to set up our own munin instance.
  21. Tweak/tune the system in an attempt to reduce load
    Analyse common jobs, high/low loads; reschedule cron jobs; uninstall unneccessary daemons and other software; optimise configuration of required daemons; determine and delete unneccessary packages
  22. Backups
    Check whether backups are currently happening, and if so, of what, and whether that is sufficient;
    develop a backup scenario, find backup locations, find utilities to create backups, install and configure them
  23. shorewall + iptables
    Install ‘iptables’ firewall and ‘shorewall’ firewall frontend; use shorewall to configure a firewall
  24. Creating SSL certificates with CAcert
    In order to avoid security advises on the web browser when visiting CMIs, an SSL certificate can be created and set on the website.

Please add whatever you can think of – whether or not you could teach others on doing it. If you can do it yourself and would like to, please add your name in brackets.