– The Practice of System and Network Administration, Thomas A. Limoncelli and Christine Hogan.

Bingo. But can there be a third hybrid model?

I currently represent the decentralized model and I must agree with these two fine authors that the benefit of my close working relationship with the individual department/group is that the service provided is highly customized and focused. The central IT department(s) are understandably focused on large-scale issues (“Infrastructure”, “Communications”, “Collaboration”, “Applications”) and as such do not always represent the most ideal channel for delivery of IT services to the various research groups and departments on campus, often with more nuanced, specialized and micro-level issues.

One of my developing long-term goals is to (warning: business jargon) “bridge the gap” between the focused local support that I currently represent and the value proposition(s) of centralized IT services. I’m not yet entirely certain of how to accomplish this but I am certain that there is a way to improve the delivery of IT services to researchers across all our campuses and I want to be involved.

Does such an approach warrant the definition of a third hybrid model or is this so-called bridging of the gap already encapsulated in the model of centralized vs. decentralized?

Some of the challenges I face specifically as a “standalone” decentralized sysadmin on campus are:

• Dealing with all IT needs from desktop support to infrastructure development to data security
• Developing and maintaining vendor contacts and relationships
• No immediate peers in our environment to bounce specific ideas around with
• Weak purchasing power and negotiation leverage
• Duplication of effort
• Career progression is potentially limited
• All too easy to develop a “King of the Castle” attitude
• Complacency

Some of the concerns I hear about when introducing researchers to the idea of centralized IT support:

• General lack of trust/faith in the centralized IT department
• Perceived lack of personal attention and focus (turn around times, site knowledge, etc.)
• Perceived lack of “control” over their environment (and data!) under the centralized model
• Charge-back models for IT services are viewed as grant-unfriendly
• Physical hardware ownership appears to remain important for many researchers

Of course, this is but a snapshot of the challenges I face and the concerns I’ve been hearing but they do serve as decent examples. It must also be noted that I am seeing great progress is many of these areas already because there are very bright people here already working on these challenges. My interest in this field is absolutely not unique.

For the immediate future, I’m focusing on improving my collaborations and communication with centralized IT services by helping them out where I can and leaning on them more often for our localized problems. My hope is that by constantly forging a closer working relationship will increasingly expose me (and in turn, our group) to the benefits of the centralized IT model while providing the central IT group with greater insight into our environment and how we work.

The next steps are still a mystery to me but I’m keeping my eyes open for new opportunities to bring better IT to research.

No related posts.

As if there wasn’t already enough discussion surrounding AoE vs. iSCSI in mailing lists, forums and blogs, I am going to add more baseless opinion to the existing overwhelming heap of information on the subject. I’m sure this will be lost in the noise but after having implemented AoE with CORAID devices and iSCSI with an IBM (well, LSI) device and iSCSI with software targets in the past I feel I finally have something share.

This isn’t a technical analysis. I’m not dissecting the protocols nor am I suggesting implementation of either protocol for your project. What I am doing is sharing some of my experiences and observations simply because I can. Read on, brave souls.

Background

My experiences with AoE and iSCSI are limited to fairly small implementations by most standards. Multi-terabyte and mostly file serving with a little bit of database thrown in there for good measure. The reasoning behind all the AoE and iSCSI implementations I’ve setup is basically to detach storage from physical servers to achieve:

1. Independently managed storage that can grow without pain
2. High availability services front-end (multiple servers connecting to the same storage device(s))

There are plenty of other uses for these technologies (and other technologies that may satisfy these requirement), but that’s where I draw my experiences from. I’ve not deployed iSCSI or AoE for virtual infrastructure which does seem to be a pretty hot topic these days, so if that’s what you’re doing, your mileage will vary.

Performance

Yeah, yeah, yeah, everyone wants the performance numbers. Well, I don’t have them. You can find people comparing AoE and iSCSI performance elsewhere (even if many of the tests are flawed). Any performance numbers I may accidentally provide while typing this up in a mad frenzy are entirely subjective and circumstantial… I may not even end up providing any! Do you own testing, it’s the only way you’ll ever be sure.

The Argument For or Against

I don’t really want to be trying to convince anyone to use a certain technology here. However, I will say it: I lean towards AoE for the types of implementations I mentioned above. Why? One reason: SIMPLICITY. Remember the old KISS adage? Well, kiss me AoE because you’ve got the goods!

iSCSI has the balls to do a lot, for a lot of different situations. iSCSI is routable in layer 3 by nature. AoE is not. iSCSI has a behemoth sized load of options and settings that can be tweaked for any particular implementation needs. iSCSI has big vendor backing in both the target and the initiator markets. Need to export an iSCSI device across a WAN link? Sure, you can do it, never mind that the performance might be less than optimal but the point is it’s not terribly involved or “special” to route iSCSI over a WAN because iSCSI is designed from the get-go to run over the Internet. While AoE over a WAN has been demonstrated with GRE, it’s not inherent to the design of AoE and never will be.

So what does AoE have that iSCSI doesn’t? Simplicity and less overhead. AoE doesn’t have myriad of configuration options to get wrong, it’s really so straight forward that it’s hard to get it wrong. iSCSi is easy to get wrong. Tune your HBA firmware settings or software initiator incorrectly (and the factory defaults can easily be “wrong” for any particular implementation) and watch all hell be unleashed before your eyes. If you’ve ever looked at the firmware options provided to by QLogic in their HBAs and you’re not an iSCSI expert, you’ll know what I’m talking about.

Simplicity Example: Multipath I/O

A great example of AoE’s simplicity vs. iSCSI is when it comes to multipath I/O. Multipath I/O is defined as utilizing multiple paths to the same device/LUN/whatever to gain performance and/or redundancy. This is generally implemented with multiple HBAs or NICs on the initiator side and multiple target interfaces on the target side.

With iSCSI, every path to the same device provides the operating system with a separate device. In Linux, that’ll be /dev/sdd, /dev/sde, /dev/sdf, etc. A software layer (MPIO) is required to manage I/O across all the devices in an organized and sensible fashion.

While I’m a fairly big fan of the latest device-mapper-multipath MPIO layer in modern Linux variants, I find AoE’s multipath I/O method much, much better for the task of providing multiple paths to a storage device because it has incredibly low overhead to setup and manage. AoE’s implementation has the advantage that it doesn’t need to be everything to every storage subsystem, which fortunately or unfortunately device-mapper-multipath has to be.

The AoE Linux driver totally abstracts multiple paths in a way that iSCSI does not by handling all the multipath stuff internally. The host is only provided with a single device in /dev that is managed identically to any other non-multipath device. You don’t even need to configure the driver in any special way, just plug in the interfaces and go! That’s a long shot from what is necessary with MPIO layers and iSCSI.

There’s nothing wrong about device-mapper-multipath and it is quite flexible, but it certainly doesn’t have the simplicity of AoE’s multipath design.

Enterprise Support

Enterprise support is where iSCSI shines in this comparison. Show me a major storage vendor that doesn’t have at least one iSCSI device, even if they are just rebranded. Ok, maybe there are a few vendors out there without an iSCSI solution but for the most part all the big boys are flaunting some kind of iSCSI solution. NetApp, EMC, Dell, IBM, HDS and HP all have iSCSI solutions. On the other hand, AoE only has only a single visible company backing it at the commercial level: CORAID, a spin-off company started by Brantley Coile (yeah, the guy who invented the now-Cisco PIX and AoE). I’m starting to see some Asian manufacturers backing AoE on the hardware level but when it comes to your organization buying rack mount AoE compatible disk trays, CORAID is the only vendor I would suggest at this time.

This isn’t so fantastic for getting AoE into businesses but it’s a start. With AoE in the Linux kernel and Asian vendors packing AoE into chips things will likely pickup for AoE from an enterprise support point of view: It’s cheap, it’s simple and performance is good.

Conclusion

AoE rocks! iSCSI is pretty cool too, but I’ve certainly undergone much worse pain working with much more expensive iSCSI SAN devices vs the CORAID devices. And no performance benefit that I could realize with moderate to heavy file serving and light database workloads. I like AoE over iSCSI but there are plenty of reasons not to like it as well.

To each their own, I say.

Related posts:

1. Migration Weekend: Success

NOTE: IT Watchdogs has since released the SuperGoose II. Since I haven’t compared the two products, I can’t comment on any differences.

IT Watchdogs

I really don’t know a lot about these guys but they appear to be a fairly small company. Their name came up several times during my searches for environmental monitoring solutions. What I do know is that their support people get back to you fairly quickly and they aren’t demanding a product serial number before assisting you with your IT Watchdogs related product questions. Their support gives you the impression you’re speaking with people that understand the product very well.

They provide firmware upgrades on their website where you can download them and get instructions on the upgrade process. There is no service contract necessary to continue getting firmware updates, which is very nice considering one of the main points they seem to be competing on is price and thus you not only get good initial value for your dollar, you get long term value in the software improvements over the lifetime of the device.

It’s a plane, it’s a bird, it’s a SuperGoose!

IT Watchdogs have a wide range of monitoring products but the SuperGoose (also known as the WxGoos-2) is (was) their flagship product. With a $499 price tag, it’s not dirt cheap but comparing it’s feature list to much of the competition out there and it really does start to look like a bargain.

The unit itself is a tiny litte 1U rackmount box that is no deeper than it is tall. That is, it’s about 1U deep. The unit has an LCD on the front that will cycle through readings from your various attached sensors that provide visual stimulus for those lonely nights in the server room. The unit also has a audible alarm which is of no use to me, but we’ll come to that later.

The build quality of the SuperGoose is good. It’s made out of metal, which is always nice. Everything is solid and well manufactured. There is just one design feature of the SuperGoose that I just don’t get: It requires an external power adapter (included), the kind you use to charge your cellphone or to power your crappy D-Link hub you’ve got installed in your office that the IT guys keep ragging on you about. It really isn’t a big deal but I just don’t see why they couldn’t have integrated the power transformer (A/C to D/C) into the chassis of the device and allowed the common and standard IEC C13 plug for power. Such a solution would have been cleaner and more compliant with the existing power infrastructure in the average server room/data center. Not to mention that the current power adapter plugs into the front of the SuperGoose, another little design issue I don’t fully understand. Either way, this is a minor gripe but something worth noting regardless as you’ll need to have a suitable PDU or extension cable to power the thing.

The SuperGoose allows for two kinds of external sensors to be added in addition to the full set of built-in sensors. The first kind of sensor is connected via RJ-12 jacks. The sensors supported in the RJ-12 jacks are digital sensors which means there is some kind of tiny microchip in the sensor itself. It’s not “passive”, if you will. While only five physical RJ-12 ports are included, the SuperGoose supports up to 16 sensors by using an RJ-12 splitter. The second kind of sensor is analog. In our case, we’re only using digital sensors so I cannot comment on the analog sensor functionality at this point, unfortunately.

Setting up the SuperGoose

Setup is really quite easy:

1. Mount the SuperGoose in your rack (and cable it up).
2. Grab the MAC address from the front of the device and setup a DHCP address for it (or just leave the default IP and setup a host on that subnet to initially configure it).
3. Power it up.
4. Point your browser to the SuperGoose’s IP address and begin configuring!

Of course, you can add many sensors as well at this point, which is key to a general purpose environmental monitoring solution.

The SuperGoose has a robust web server where you can view the collected data and graphs and configure various things such as networking, NTP, email, SNMP, camera setting and the device information. You can also configure alarm thresholds for your sensors so that you can get an email or SNMP trap fired off in case of abnormal readings. While useful for smaller shops lacking a centralized monitoring solution, these built-in alerts are not a factor when integrating the SuperGoose into a larger environments where a centralized monitoring solution has been implemented. It’s nice to have them, but you might not need them.

The configuration page is clear and straight forward. The language used to describe configuration options should be totally familiar to anyone with cursory knowledge of the various technologies, standards and protocols in use. The SuperGoose provides three level of access to the device: View-only, Control and Administrative. Each access level has a configurable account name and password. The SuperGoose unfortunately does not support multiple accounts of the same level. That said, enterprise use of the SuperGoose will often involve SNMP data collection instead of users and admins logging into the SuperGoose web server directly and so, more fine grained per-user/admin accounts can be configured in the SNMP monitoring solution to control access to data while at the same time locking out access to the SuperGoose directly to all but the SNMP/env administrators in order to achieve “lock down” and logging on who can see/do what.

Digital Sensors & SNMP Monitoring

Digital sensors have unique IDs to which names can be assigned within the configuration interface. This means that when you connect/disconnect your digital sensors, or the SuperGoose reboots, the graphs produced by the SuperGoose and the data it has collected remain valid and consistent.

One snag I did run into momentarily is that although the SuperGoose will internally track the digital sensors correctly, the SNMP OIDs for each sensor were not consistent upon reboot of the SuperGoose! My SNMP monitoring station would collect false data when the SuperGoose was rebooted because the sensors were detected in a different order and thus the OIDs for each sensor changed! The reason this happened is because I was monitoring the OIDs in static fashion by associating a sensor definition in the monitoring software with a particular, static OID. When the OID for the any sensor changed (as it often does upon reboot), the monitoring station would be collecting information for a particular item in the monitoring database from the wrong physical sensor. The solution is to use SNMP monitoring software that supports using dynamic SNMP indexes. Zabbix is such an open source monitoring solution. It took me a little while to configure Zabbix correctly for dynamics SNMP indexes but it was worth it: no more collecting data from the wrong sensor and thus more consistent and correct long term trending data is collected.

For what it’s worth the following is an example of the “OID” string I use within a Zabbix “item” to implement the dynamic index feature:

IT-WATCHDOGS-MIB::tempSensorTempC[index,IT-WATCHDOGS-MIB::tempSensorName,Ambient Front]

Where “Ambient Front” is the name of the sensor, as set in the SuperGoose configuration.

Well Folks, that’s Part 1

Well, that’s part 1 of the review. Part 2 of the review will arrive sometime in the future… I’ve just been sitting on this much of the review for too long now and it needs to be posted. I hope everyone enjoys half reviews (for now)!

Related posts:

1. IBM Changed UPS Vendors
2. OpenGear CM4116 Review

This won’t have a large functional difference but it pisses me off none-the-less. I’ve been using Network UPS Tools (NUT) with our APC UPS devices over USB and now I come to realize that this Eaton UPS doesn’t support USB, or rather it doesn’t come with a USB cable although the documentation says it does. Now to integrate this Eaton UPS with NUT I need to use SNMP since the Eaton only comes with a serial cable, which is useless to me for monitoring purposes. I guess I get to find out how NUT’s SNMP support is now.

This silent switch with a minor model change/update just rubs me the wrong way. I’m not so happy with the management of this UPS either…

UPDATE: Got the Powerware 5125 working with NUT via SNMP… seems good so far.

For the sake of posterity, here is my /etc/ups/ups.conf for the 5125 to work with NUT:

[ibm3000_1]
  driver = snmp-ups
  port = ibm3000-1.mydomain.com
  mibs = pw
  pollfreq = 30
  community = mycommunity
  snmp_version = v1
  desc = "Rack \#1 - IBM UPS 3000 HV @ U1/2"

This of course assume you’ve already properly configured the 5125 with a DNS name or IP address and properly enabled SNMP, allowing your NUT client to connect

UPDATE 2: The Powerware is starting to grow on me a little. It looks like it packs more juice than the same 2U unit from APC: 45% load and 12 minutes run time compared to an APC with 48% load and 8 minute run time. As long as I ignore the web management interface and stick to NUT via SNMP, I actually kind of like this UPS.

Related posts:

1. Check Your UPS Batteries!