iPad & Tablet Thoughts From Your Friendly Neighbourhood Irrelevant Internet Guy

Let me preface this by saying that I do not own an iPad nor do I own any other “tablet” like device. I’ve played with an iPad and some of the older generation of Windows tablets (Laptops with swivel screens and stylus input running a “Tablet Edition” of Window) but I have no exceptional experience with anything I’m going to rant about. That said, I’m still going to talk about them.

I’m going to try to address various talking points I’ve been reading in articles, comments and blogs but don’t sue me if I miss anything.

[ Read More ]»

Migration Weekend: Success

It was a long weekend of watching tape restores and restarting them as necessary but it’s finally over and everything appears to be mostly hunky dory!

I did discovery yet more small misconfigurations and strange behaviour along the way:

1. OpenLDAP’s syncrepl using “refereshAndPersist” wasn’t working how I expected it to, no new changes were replicating to the slave LDAP server! I changed the directive to “refreshOnly” and set a 10 minute interval. I made several changes and monitored the slave LDAP server. Changes propagated in about 10 minutes, every time.
2. Despite iSCSI’s maturity and the maturity of QLogic’s HBAs I still noticed strange, unexplained target drop outs. Two HBAs per server, two controllers in the IBM DS3300 and just one target out of four was dropping. At first, I couldn’t figure out how to properly reconnect the target on a live system so I rebooted. Later, I discovered you can “disable” and then “enable” the specific target in SANsurfer or iscli, which worked to bring back the dropped target on a live system. Multipath picked up the “new” path right away, as expected.
3. Always remember to leave free physical extents in any LVM Volume Group in which you are taking snapshots of the Logical Volumes. It’s freakin’ obvious but I forgot and when I went to do snapshot backups, the snapshots were failing. Now I’m growing some LUNs on the DS3300 so that my VGs have room for snapshots.

All in all, a good weekend that was mostly filled with success.

Migration Weekend

The big data migration is upon me!

This weekend we will be migrating about 3TB of data from aging 5 year old servers with internal DAS RAID over to the new infrastructure I’ve been building over the last two months. Part of my genius plan is to migrate the data using our backup & restore software. The genius I believe comes from the fact that doing the migration via tape restore will provide a long overdue full test of our capability to restore in the event of catastrophic storage failure. Data migration and restore testing, two birds with one stone.

Wish me luck!

LVM filters and initrd

Another “don’t make the same mistake I did” post, you say? Yippee! I seem to be running into quite a few of these (semi) complex gotchas lately but I suppose they at least fuel techslaves with a bit of content which I can’t be too angry about these days.

Today’s gotcha is all about LVM filters and initrd and really this wouldn’t be a big problem but because I’ve never sat down to appreciate the initrd process in any great depth, it took me two days to figure out just exactly what went wrong (recovery however, was much faster).

[ Read More ]»

When using Syncrepl…

Quick OpenLDAP tip boys & girls…

When using syncrepl to replicate from a master LDAP server to a slave LDAP server, always remember to configure the ACLs on the master LDAP server to allow the “sync dn” to read everything.

I know it sounds entirely obvious but today I realized that the order in which I had defined the ACLs on the master LDAP server was preventing the sync dn from reading the “userPassword” attribute and thus also preventing it from syncing it to the slave. The consequence of which was that users would not be able to authenticate against the slave server! Shit!

Of course, since everything else was syncing properly, all the NSS (lookup) stuff worked fine but anything authentication related like PAM wouldn’t work because the user bind would fail with “Invalid credentials” in /var/log/secure. It had a been some time since I tested authentication so I must never have actually tested authentication against the slave (whoops!) and thus didn’t notice until now. I know I tested lookups but testing authentication must have slipped by somehow. Grrr, testing.

Good thing I caught the problem early and it never escalated into a problem, that really could have sucked down the line.

Don’t make the same mistake I did.