IT Watchdogs SuperGoose (WxGoos-2) Review

Some time ago it became apparent that we would require environmental monitoring in our server room. The primary reason being that our server room was never initially intended to be a server room and the after-the-fact A/C unit installation (size, vent placement, etc.) is definitely less than optimal. Not to mention the A/C unit is likely overloaded as well, judging by some of the data we gathered after installing the environmental monitoring equipment and software. Basically, I needed to be made aware of any potential problems with the environment in that room so that should anything go wrong, I can act quickly. A secondary use of the data is to trend the environment changes in order to reveal specific patterns that may help with long-term planning.

Time Navigator HA Cluster Agent Configuration

I’ve been wanting to post about a configuration that allows for seamless file-level backup of storage attached to an active/passive high availability cluster in an uninterrupted fashion using Atempo’s Time Navigator and I’m finally going to do it.

The Problem

The initial difficulty lies in the requirement that the data must be consistently backed up at every interval, no matter which cluster node is currently the active node with the backend storage mounted. To do this, an agent is required to be configured as a cluster resource in order to “follow” the mounting/exporting of the storage to any cluster node. So in order to accomplish this,  N + 1 tina agents are required. That is, if you have two cluster nodes, you need three agents to successfully backup each node with the local agent and the storage, as it floats about the cluster nodes depending on failure or migration events.

Luckily for me, the good people at Atempo have engineered the agent in such a way that multiple agents can be ran on a single node, each binding to it’s own IP address and each individually controlled via it’s own init script. Of course, we need to make some file edits to make all this happen and that’s what I’m going share!

Read the rest of this entry »

Cfengine 3 Snippets Part 1: DenyHosts

I’ve recently begun looking into configuration management with cfengine 3. I’ve ignored this growing sub-field of system administration for too long and I just can’t ignore it anymore. After spending quite some time researching the philosophies, methods and different tools out there, I settled on starting out with cfengine 3. There’s no special reason that I chose cfengine instead of puppet, bcfg2, chef or AutomateIT. I haven’t used any of these tools and thus I cannot pass judgement on them or their methods. All these projects seem to have intelligent and highly motivated people behind them. I simply gravitated towards cfengine because of its strong academic background and the fact that version 3 now represents the most recent and modern research in the field by Mark Burgess et. al.

As part of my learning experience with cfengine, I’ve decided to start posting some of the code that I’ve begun developing in the hopes that by writing about it, I can learn better, faster and maybe even receive some helpful comments from readers along the way. Beware, I’m a cfengine newbie and so what I post here should NOT be copy and pasted into your environment unless you’re ok with the potential of wildly breaking things!

The first snippet of code I want to discuss is related to managing our DenyHosts configuration. As part of our “security policy”, I would like to ensure that every RedHat/CentOS system is running a properly configured DenyHosts instance. Here is what I’ve come up with so far.

Read the rest of this entry »

RHEL/CentOS, NFS and Firewalls

I recently decided that it’s about time to setup consistent, explicit and tight firewall policy across our Linux (mostly RHEL/CentOS) servers. One of the initial issues I faced was NFS. NFS implementations are very well known to make use of the portmapper and dynamically assigned port for rpc.mountd and because of this dynamic assignment, firewalling NFS can be challenging.

Luckily, RedHat’s /etc/sysconfig/nfs configuration file read by  various “nfs”, “nfslock” and RPC services init scripts provides an easy means of locking down specific ports for all the NFS-related services so that one doesn’t have to work around the dynamic port assignment problem when it comes to firewalling.

Read the rest of this entry »

Atempo Time Navigator 4.2 Archive Media Selection Tunable

Just a quick post here to share a non-obvious tunable for Atempo’s Time Navigator 4.2 regarding archiving and media selection.

Before upgrading from 4.1 to 4.2 Time Navigator’s media selection for archive jobs with standalone drives behaved as expected: If existing partly filled and open cartridges in the associated media pool existed, Time Navigator would request those media be placed in the drives upon the start a new archive operation, effectively only asking for new, unlabeled media to be inserted once the existing media was full.

However, with the upgrade to 4.2 we found that Time Navigator was no longer requesting the existing, partly filled, open cartridges and was instead requesting new, unlabeled media to be inserted into the drives instead! The result of this new behavior was that Time Navigator would use new tapes for every new archive operation, no matter if existing, partly filled and open media was available in the media pool. Basically 4.2′s default behavior was preventing us from filling any archive media unless the particular archive job would happen to be larger than a single tape.

While I don’t know why the functionality changed, I do know what tunable to modify in order to make 4.2 behave like 4.1. The tunable is “check_external_cart_when_recycling“. Setting this tunable to “Yes” has restored the 4.1 behavior, allowing us to make full use of all archive media capacity by only requesting new media when all the existing media in the media pool has been filled.

I believe we only faced this problem because we use standalone archive tape drives that do not have an autoloader or robot nor an “inventory” of online tape. Each tape must be manually loaded. I suspect that if we had an autoloader for our tape drives, that 4.2 would have made the correct/expected selection of media.

I doubt that anyone else is going to face this problem but it took about 3 weeks with Atempo’s R&D department to figure out the problem so I figure if posting here can save anyone that amount of time, then I’ll have done my part!