Not for Domains

It’s important to keep in mind that these instructions are not for a integrating FreeIPA with a Samba domain controller but merely a Samba file server. My understanding is that FreeIPA will never conveniently/properly support the necessary bits to make it a suitable backend for a Samba 3 PDC. I believe FreeIPA will eventually look towards Samba 4 integration (using Domain trusts) for this kind of integration but don’t quote me on that. Either way, these instructions are not for Samba domain controllers, just Samba file servers.

The Assumptions

There are some basic assumptions that these instructions make.

• FreeIPA is installed and functional
• You have a general idea of how to use LDAP command line tools
• If you have a nice GUI LDAP browser, you can use it to apply the example LDIFs and edit the tree instead of the ldap CLI tools
• The LDAP commands are executed on the FreeIPA server
• Samba and FreeIPA are installed on the same server (although it shouldn’t be difficult to use TLS encryption with separate servers)
• Your LDAP suffix is “dc=domain,dc=tld”
• You know the difference between the “admin” account and the directory manager and their passwords

The Goods

Let’s not beat around the bush any further.

1. Determine your Samba server SID by executing the following command while smbd is running and jot it down:

root@ipaserver:~
# net getlocalsid
SID for IPASERVER domain  is: S-1-5-21-3180075094-3458813485-3821849995

2. With the “admin” kerberos ticket, add two attributes to “cn=ipaConfig,dc=etc,dc=domain,dc=tld” that tell FreeIPA to setup each account as a Samba account and each group as a Samba group:

ldapmodify -Y GSSAPI <<EOF
dn: cn=ipaconfig,cn=etc,dc=domain,dc=tld
changetype: modify
add: ipaUserObjectClasses
ipaUserObjectClasses: sambaSAMAccount
-
add: ipaGroupObjectClasses
ipaGroupObjectClasses: sambaGroupMapping
EOF

3. With the directory manager password and the Samba SID you jotted down from above, create an instance of the 389 DS DNA plugin that will automatically generate SIDs for your users and groups which are necessary for use with Samba:

ldapadd -x -D "cn=Directory Manager" -W <<EOF
dn: cn=SambaGroupSid,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
objectClass: top
objectClass: extensibleObject
dnatype: sambaSID
dnaprefix: S-1-5-21-3180075094-3458813485-3821849995-
dnainterval: 1
dnamagicregen: assign
dnafilter: (|(objectclass=sambasamaccount)(objectclass=sambagroupmapping))
dnascope: dc=domain,dc=tld
cn: SambaSid
dnanextvalue: 15277
EOF

The thing to note here is that the “dnaprefix” is set to the SID your jotted down… PLUS a hyphen (“-”) appended to the end!

4. Now we have to start modifying the FreeIPA API, CLI and WebUI to allow us to specify the “sambaGroupType” attribute at group creation time. We have to set “sambaGroupType” because it is a required attribute for the objectClass “sambaGroupMapping” which we are automatically adding to every group with the “ipaGroupObjectClasses” setting from earlier.

Although the value is going to be “4″ for every conceivable case in this non-domain configuration, I was not able to figure out how to make the DNA plugin insert static values like it can set incrementing values so I decided to allow setting it through the CLI and WebUS with defaults enabled instead. If anyone knows how to setup 389 to automatically add an attribute with a static value upon DN creation of DNs with specific objectClasses, please tell me.

There are a few steps required to make this CLI/UI stuff happen but the FreeIPA developers have actually made this quite simple.

The rule is: Extend the FreeIPA schema first, then the CLI, then the WebUI.

4.1. Extend the FreeIPA schema with a custom field by adding the attribute “ipaCustomFields” with a value of “Samba Group Type,sambagrouptype,true” to “cn=ipaConfig,dc=etc,dc=domain,dc=tld” with an “admin” kerberos ticket:

ldapmodify -Y GSSAPI <<EOF
dn: cn=ipaconfig,cn=etc,dc=domain,dc=tld
changetype: add
add: ipaCustomFields
ipaCustomFields: "Samba Group Type,sambagrouptype,true"
EOF

As there can only be one “ipaCustomFields” attribute, if you have multiple custom fields you need to separate each definition with a “$” like so: “Samba Group Type,sambagrouptype,true$Description,attrname,isrequiredboolean”.

4.2. Extend the CLI for groups by editing the python file “/…/site-packages/ipalib/plugins/group.py” to define the custom field and specify a default if not implicitly defined (diff):

--- group.py.orig	2011-08-15 14:59:48.570715207 -0700
+++ group.py	2011-08-16 12:43:43.493236507 -0700
@@ -118,6 +118,13 @@
             label=_('GID'),
             doc=_('GID (use this option to set it manually)'),
         ),
+        Int('sambagrouptype',
+            cli_name='sgt',
+            label=_('Samba Group Type'),
+            doc=_('Samba Group Type (default is 4)'),
+            default=4,
+            autofill=True,
+        ),
     )

 api.register(group)

Important: Restart “httpd” at this point!

4.3. Test the CLI. With an “admin” (or equivalent priv) kerberos ticket, try creating a new group:

account@ipaserver:~
$ ipa group-add testgrp --desc="Testing the group.py CLI mods"
---------------------
Added group "testgrp"
---------------------
  Group name: testgrp
  Description: Testing the group.py CLI mods
  GID: 1234500010
  Samba Group Type: 4

4.4 With the CLI functioning properly, we can move on to extending the WebUI. To extend the WebUI for group attributes, edit “/usr/share/ipa/ui/group.js” like so (diff):

--- group.js.orig	2011-08-15 10:01:28.515209121 -0700
+++ group.js	2011-08-16 13:52:59.587352034 -0700
@@ -34,6 +34,7 @@
                 column({name: 'cn'}).
                 column({name: 'gidnumber'}).
                 column({name: 'description'}).
+                column({name: 'sambagrouptype'}).
                 dialog(
                     IPA.add_dialog({
                         'name': 'add',
@@ -41,6 +42,7 @@
                     }).
                         field(IPA.text_widget({name: 'cn', undo: false})).
                         field(IPA.text_widget({name: 'description', undo: false})).
+                        field(IPA.select_widget({name: 'sambagrouptype', undo: false, options: [{label: 'Local', value: 4}, {label: 'Domain', value: 2}]})).
                         field(IPA.checkbox_widget({
                             name: 'posix',
                             label: IPA.messages.objects.group.posix,
@@ -56,6 +58,7 @@
                     }).
                         input({name: 'cn' }).
                         input({name: 'description'}).
+                        input({name: 'sambagrouptype'}).
                         input({name: 'gidnumber' }))).
         facet(
             IPA.group_member_user_facet({

And then these the WebUI to ensure that you can both see the attribute in the group list, but also add it via the select widget added to the new/edit group dialog.

That should be it. Questions, comments, suggestions, correction and more… all are welcome!

Related posts:

1. LDAP User Management Tools and User Private Groups
2. Life Support!

I arrived shortly after 8AM, registered to receive my badge and t-shirt then milled around the vendor booths until the keynotes were ready to start. I watched the keynotes (Jim Zemlin, Linux Foundation and Jim  Whitehurts, Red Hat), went to every session I could and came back to the main ballroom for the panel discussion with Jon “Maddog” Hall, Eben Moglen and Dan Frye and the following interview of Linus Torvalds by Greg Kroah-Hartman to wrap things up for day 1. So far, so good.

The Keynotes

Jim Zemlin’s opening keynote “Imagining a World Without Linux” was decent. While he did take some inevitable potshots at Microsoft, the message was generally very positive and uplifting. I won’t go into details but basically Jim described a world without Linux as one that would be black & white as opposed to the colour filled world we know today (due to Linux). Jim is a smiley and positive person on stage, his style helped kick off LinuxCon 2011 with a good vibe.

Jim Whitehurst, CEO of Red Hat had a similar approach of sending positive vibes but focused on how the progress of Linux and Open Source has enabled businesses and business models. He said that Google wouldn’t exist (at least not in it’s current form) without Linux and basically implying the same about other major well know Linux-powered companies such as Amazon and Facebook. Jim struck me as a fairly modest fellow but he wasn’t shy about mentioning Red Hat’s penetration into Fortune 500 companies. Nor was he reserved about how Linux has powered, enabled, strongly driven by or directly benefitted various global forces that may or may not be angels (U.S. Navy, NSA, Russian Military, NYSE/Wall Street). While his examples spoke to the breadth of applications for and the wide reach of Linux, I couldn’t help but think about how the pervasiveness of Linux is not only helping drive great positive change int he world but may also be powering negative forces as well.

Overall both Jim’s did a good job and left me excited for the rest of LinuxCon to come.

First Day Sessions

I attended four sessions on day 1:

1. Centralized User Administration with FreeIPA and sssd by Stephen Gallagher
2. Watching Mad Men and Thinking About Open Source by Karen Copenhaver
3. 20 Years – And More – of Kernel Development by Jon Cobert
4. What to Expect from Linux Storage by James Bottomley

Centralized User Administration with FreeIPA and sssd

My first LinuxCon session was by Stephen Gallagher of Red Hat. As is clear by the title, it was about FreeIPA and sssd, two emerging Red Hat driven projects relating to centralized directory and authentication services. Stephen wasn’t the most natural speaker I’ve had the pleasure to watch and I suspect that presentations aren’t something he does on a regular basis but he clearly knew his material and he was able to field the post-presentation questions with ease. The presentation material was fairly spot on to what I expected. I should stop by the Red Hat booth and speak with Stephen tomorrow as there are a few FreeIPA/sssd related questions I have which I didn’t ask during the question period. Overall, I was satisfied.

Watching Mad Men and Thinking About Open Source

First of all, Karen is a more natural speaker than Stephen but I suppose that’s to be expected: She is legal counsel for the Linux Foundation. The material in this session while clear and understandable was maybe not quite as impactful as I had hoped. Karen had some very nice points and brought good historical reference to the table but it wasn’t really anything that I didn’t already think think about in my own internal dialog, for the most part.

Some key points that Karen made early which did resonate with me:

• “It’s a privilege to work on something so important”, I believe she was quoting Linus Torvalds. This hits home for me as my work is only to enable the much more important and relevant work of others.
• The observation that the open source community generally doesn’t have time for anything but the truth which is a nice ideal but perhaps isn’t necessarily reflective of the entire open source world so much as a few of the important luminaries.
• Identify the things that you value and… well I missed that part. But I do think identifying the things you value is, well, valuable.

These are all straight forward things but to hear someone say them can be powerful. This session was good but it wasn’t quite as hard hitting as I thought it might be based on the title and description. It was no let down, though.

20 Year – And More – Of Linux Kernel Development

Ok, now we’re getting way out of my league. Jon Corbet is a high profile Linux kernel contributor and he knows what he is talking about. This man has confidence and ostensibly the knowledge to back it up. His overview of the last 20 years of Linux kernel development was excellent and spotted with just enough humour to keep the real developers cracking up and the rest of us only getting every second joke.

Jon’s timeline approach to describing the history of kernel development was excellent and enabled him to visually map releases, events and growth in a very simple and understandable way. He made an excellent observation regarding the pace (measure by lines of code) of Linux kernel development during the dot com bust not slowing down one bit despite industry turmoil and job loss and pointed out the correlation between important points in Linux kernel development time with other events that may not be obvious to every outsider (BitKeeper, Git, time between certain releases, Merge Window, etc.).

While this session was developer focused, it wasn’t so technical to be devoid of value for anyone else, in fact I think it really helped frame the history of Linux kernel development for me in a way that I had never experienced before. Way to go, Jon.

What To Expect From Linux Storage

I’m not sure why James’ talk was titled what it was because for the best of what I could tell, the majority of the talk was about what already is, not what to expect. That’s not to say it was devoid of important information regarding “what to expect” and maybe it was because James ran out of time and had to skip some slides but I did find the title interesting in that capacity none-the-less.

James is charismatic. He makes jokes, he wears a bow tie, he speaks with an attractive accent. He’s also clearly very knowledgable about his part of the Linux kernel: the Block layer.

Being a sysadmin, knowing more about the block layer and James’ perspective on storage was hugely beneficial. He has historical reference that I never will and deep knowledge of the kernel which I’ll never achieve. With that said, some of his opinions regarding specific technologies and methods, I personally already held myself! How is it that a Linux kernel rube such as myself could had gleaned the same opinions on specific technologies as one of the people  who understands these technologies the best of anyone? iSCSI was an example. I think it’s safe to say James thinks iSCSI is an abhorrent mess that simply tries to solve a problem in entirely the wrong way. I’m also not a big fan of iSCSI and his reasoning  resonated with me, despite my lack of in depth knowledge.

I could go on because I liked this session but I already feel like I’m burning myself out on this summary of day 1 and we haven’t even gotten to the panel discussion or Linus interview yet.

Panel Discussion

The panel discussion with Jon Hall, Eben Moglen and Dan Frye was fairly profound despite Eben using the platform for an interesting but strangely placed speech that appeared entirely scripted/written. That’s not to say I didn’t like his speech or that I don’t agree with him or his world views but the way he momentarily took over the panel with what was clearly a pre-planned speech during a panel discussion main-hall format was strange indeed.

Dan Frye struck me as level-headed and one of those business people whom can take the challenge of  balancing the need to run a profitable business with social awareness and decency and excel at it. I’ve never really doubted IBM’s commitment to Linux and I know their commitment is based on profitability but the way that Dan framed the reasons that he and his team knew Linux meant good business for IBM put a smile on my face.

Jon Hall’s experience in the computing industry is staggering and humbling, even for today’s big shots. What a dude. Level head, very articulated, sense of humour and a huge white beard. It’s hard not to love the guy after watching that panel discussion. Jon talked about his hopes for how Linux and the open source model will foster the next generation of great thinkers, movers and shakers and enable them to do great things. I liked that.

I’m not really sure what to say about Eben. I agreed with everything he said but he just wasn’t as loveable as Jon Hall. Must be because he’s a lawyer :D I suppose that slightly awkward speech about the troubled times that are looming (mounting patent threats and inevitable “10-20 billion” dollar war) could have been a factor as well. That said, he seemed positive despite the heavy and serious tone he used to describe the battles ahead.

On one hand, the panel discussion left me feeling good and uplifted but on the other hand I was left with a feeling of powerlessness. I’m not one of the next great thinkers, doers or talkers. What’s my place in the Linux and open source world, then? Everything that was discussed revolved around the greatest minds in open source and the huge impacts made by major players. I almost felt a little left out as a lowly sysadmin whom has to deploy at least some non-RMS blessed systems alongside the requisite Linux systems. What’s my role in all this?

Interview with Linus

I really don’t have much to say about this one. Linus is down to earth, but strong in his opinions. He admits when something is outside of his immediate expertise, as evidenced by his answers to many non-Linux kernel specific questions. He talks well and he would have preferred if the crowd did not give him a standing ovation at the end but I suppose you cannot make a room full of Linux geeks sit down when their proverbial leader is being applauded.

I liked a lot of what Linus talked about regarding the modern direction of Linux such as the version numbering changes, the idea that we should be looking backwards at how to improve existing subsystems and layers instead of always looking forward to new feature inclusions. I liked how he described the cross-pollination of various parts of Linux that exist when everyone from embedded systems to massively parallel SMP systems are made to use the exact same kernel instead of everyone having their own specialized forks.

Linus was clam and cool, just like Linux and I had a seriously good time at LinuxCon today. Rock on, LinuxCon!

Related posts:

1. Is Ubuntu Ready for the Enterprise?
2. LVM filters and initrd

Ever since we transitioned from the RHEL4 environment to Fedora 14, people have been reporting terrible slowness and delays in nautilus when browsing our NFS shares. Reports of waiting over a minute for an NFS automount root-level directory with < 100 sub directories to display the contents are not good.

This wasn’t a problem on our old RHEL4 terminal server and I couldn’t for the life of me understand how nautilus could have become so slow in the years since RHEL4 was released. It just didn’t make sense. I started to think something had to be wrong and that this wasn’t just the new normal expected behaviour but I had nothing to go on.

I tried the basic recommendations: Disable thumbnails, disable preview, disable directory item counts. That didn’t help the user experience in any dramatic way. At this point, I started recommended pcmanfm and thunar as a way to workaround nautilus’ terrible performance. I even wrote a fairly concise script for modifying the default file manager and desktop-drawing application so that using a different file manager wouldn’t be so foreign in GNOME.

Then one day I started looking at the verbose level output from automount while browsing the NFS mounts with nautilus and found a substantial amount of this in the logs:

Apr 28 11:19:10 hostname automount[18959]: attempting to mount entry /home/.svn
Apr 28 11:19:10 hostname automount[18959]: key ".svn" not found in map source(s).
Apr 28 11:19:10 hostname automount[18959]: failed to mount /home/.svn

Oh my! Why are there repeated access attempts for “.svn”? What is causing automount to perform map lookups for “.svn” in the automount-controlled directories? Could it be nautilus?

Why yes!

As it turns out the GNOME SVN integration package “gnubversion” includes a nautilus extension and this extension was causing Nautilus to look for “.svn” directories everywhere and it just so happens that looking for “.svn” in a root-level automount directory causes slow map lookup failures that (presumably) kill the perceptible performance of browsing automounted NFS shares.

I removed gnubversion (as no one was using it) and the user experience for nautilus has normalized. While nautilus still isn’t as speedy as pcmanfm or thunar, its no longer a cause of forceful hair removal incidents… and all is well in the world.

Related posts:

1. RHEL/CentOS, NFS and Firewalls
2. Is Ubuntu Ready for the Enterprise?
3. POSIX Default ACLs, umask and Project Directories

Right now one machine is running an old way out of support RHEL4 installation and the other is on Fedora 12, which is no longer supported by the Fedora Project. Paid support/subscription is not a consideration yet for this project, but I do want to run a modern Linux distribution for the associated modern application software and maintenance.

I basically need to move these servers to something free and supportable. I’m finding out that there aren’t a lot of options in PPC Linux as when I was last interested in this architecture. It’s pretty much just:

• Debian
• CRUX PPC
• Gentoo

I realize there is RHEL and SuSE Enterprise for PPC64 but those are subscription products without free binaries available. I’m not prepared to build an RPM-based distro from source at this point so I need something with binaries or something where building from source is highly automated and integrated, such as Gentoo. Digression…

The question is which of these distros do I go with? To answer the question I suppose I need to define the roles.

These two pSeries servers a redundant pair running LDAP/Auth Service, NTP, DNS and DHCP. The load is low but I want a solid modern software platform on both these servers from now until they are replaced with in the future (which is likely to be integration into a centralized architecture).

With that said, and with my familiarity level of these distros, I would first lean towards Debian and then to Gentoo and finally to CRUX PPC.

Debian is a binary distribution, which is nice for maintaining a server. Debian is more familiar to me. What are the arguments for Gentoo or CRUX PPC?

Agree or Disagree?

No related posts.

I’m in a bit of a pickle.

Traditionally, we’ve always allowed wide-open SSH access from anywhere to our main terminal server for remote access. Since we use NX (neatx, FreeNX, NXclient, etc.), all we ever needed open was SSH to make it all work nicely. Sure, SSH is a big bruteforce target but with DenyHosts and low thresholds things are pretty well under control. I realize huge distributed bruteforce attacks are still possible against a DenyHosts protected SSH daemon but we have to factor in ease of use when thinking about security and the low risk of massively distributed bruteforce attacks.

With the deployment of a new terminal server we have the opportunity to use the Cisco-based campus VPN service delegated to us via a customized VPN group. This is all well and good except for one thing: There is no ability to set custom routes for the clients based on those VPN groups. When a remote user connects to the VPN, *all* their network traffic has to be routed via the VPN. This typically makes sense, it is a VPN after all. However, we have a strong use case for which this simply won’t work. We have remote collaborators from anywhere in the world and mandating that they route all their Internet traffic via us when they need to remote in to our systems is unacceptable. Not to mention we have local users working from home and the same applies. We cannot mandate that these users have to route all their non-work related private traffic via us whenever they also need to access our resources at the same time.

Alternatives

So what are we left with? What can we do to provide increased security while also maintaining the ease of use of direct SSH access? I’ve thought of a couple things, but they also have disadvantages:

1. Run our own VPN with OpenVPN on dd-wrt or a generic box.

This sounds great at first but this requires that we support another box and another service directly. The campus VPN service is managed by the central IT group, our own VPN would add support costs of our own and specifically with the client installation side of things. I’d have to come up with our own instructions for installing and configuring the clients on multiple operating systems and hope that our users don’t have serious problems getting it working.

2. Use SSH but mandate the use of SSH keys.

No doubt some of our users have bad passwords and using SSH keys would prevent password bruteforce attempts including massively distributed ones. But mandating the use of SSH keys seems like hell from a support perspective. Asking users to generate keys and have them available on any client they wish to use is really pushing the boundary of what they’ll be able to successfully do on their own. I just know I’d get the evil eye from everyone if I handed them the instructions for how to accomplish this.

Push Back Times Two

What makes this worse is no matter what I choose, I’ll get push back from one side or the other. Make things more secure but more complicated and my user base will be seriously unhappy. Continue allowing direct SSH access as the long term policy and the central IT group(s) will tar and feather me: “Bad practice! Bad practice!”. Damn this pickle!

So, again, what are we left with for secure remote access solutions that are both secure and simple enough for anyone to use? Any suggestions dear Internet?

Related posts:

1. Remote Access Solution: Follow Up

Nothing like running compiz smoothly on a dual monitor thin client :D

The problem I was having was that despite the X server on the thin client being fully configured and tested to use hardware acceleration locally, when connected to the terminal server over the secure LDM tunnel I was getting direct rendering with the software renderer which results in a big fail for compiz.

The key to avoiding the software renderer from being used for DRI was setting LIBGL_ALWAYS_INDIRECT=1 as an environment variable. I don’t know why with everything configured correctly that the system defaults to using the software renderer instead of indirect rendering + hardware renderer but at least forcing this environment variable in a global profile script allows for sexy hardware accelerated compiz goodness from securely connected thin clients.

Without the environment variable to force indirect rendering, glxinfo output with the LIBGL_DEBUG=verbose env variable set was complaining that the “drm device” didn’t exist. I suspect this is because glxinfo was expecting to somehow find the /dev/dri/card0 device on the terminal server itself instead of on the thin client and of course it doesn’t exist on the server… the OpenGL card is installed on the thin client!

There must be a way to get this working without the LIBGL_ALWAYS_INDIRECT environment variable but I couldn’t figure it out… this really smells of a hack but since it’s very easy to apply globally and it works just how I expect things to work, I’ll have to leave it in place until the time I can figure out another non-hacky way of getting the results I want with this configuration.

Related posts:

1. Interesting New Developments…
2. A Correction

We’ve got these two IBM p505 servers that actually work pretty well. They were purchased on some kind of clear out two-for-one deal that my predecessor jumped on and while I probably wouldn’t be the guy to buy these machines in the first place, I’ve come to strangely like them. These server run our DNS, DHCP and soon-to-be LDAP stuff. It’s all distributed, replicated and zone-transfered goodness.

However, as of this writing they are both sportin’ a solid amber light on the LightPath diagnostics and the procedure to clear the amber light is… well… rather unclear. I think it’s unclear because we don’t have an HMC (Hardware Management Console) so we don’t get a lot of the spiffy external management features that these systems offer. Add to the fact that we run Linux on these hosts as opposed to AIX, which apparently has OS-level tools for querying the event log and flipping the light switches. I can’t find anything equivalent on Linux for p-Series systems… yet.

Googling doesn’t offer much in the way of help for clearing amber lights without an HMC and neither does IBM’s website. Looks like I’ll have to reboot one of them and go into the management controller to see if there are any options in there.

Updates coming if I can find the dang off switch…

Update 2: Yay! I finally discovered the utility necessary to turn the amber lights on an off via software on a live Linux p5o5 system! It’s usysattn from the Powerpc-utils or “Linux on Power Service Tools” project. It seem backed by IBM in some way because they link to these tools quite heavily throughout their online documentation, but I haven’t investigated any further at this time.

Regardless, once you have the latest usysattn from Powerpc-utils installed just use this command to turn off the amber light:

# usysattn

To list all indicators (lights). Then once you have the “location code” of the light you want to turn off:

# usysattn -l [location_code] -s normal

The amber lamp is now off!

Update: I Found the option in the service processor configuration menus available over the serial port. Unfortunately this means I have to reboot my servers to clear the lights but I suppose if something caused the lights to go on in the first place, it’s probably worth checking it out and scheduling some downtime to resolve it.

For posterity, the sequence necessary to turn off the amber lights after logging into the service processor is:

System name: Server-9115-505-XXXXXX
Version: SF240_358
User: admin
Copyright ? 2002-2008 IBM Corporation. All rights reserved.
1. Power/Restart Control
2. System Service Aids
3. System Information
4. System Configuration
5. Network Services
6. Performance Setup
7. On Demand Utilities
8. Concurrent Maintenance
9. Login Profile
99. Log out

S1> 4

System Configuration
1. System Name
2. Processing Unit Identifier
3. Configure I/O Enclosures
4. Time Of Day
5. Firmware Update Policy
6. PCI Error Injection Policy
7. Interposer Plug Count
8. I/O Adapter Enlarged Capacity
9. Hardware Management Consoles
10. Virtual Ethernet Switches
11. Hardware Deconfiguration
12. Program Vital Product Data
13. Service Indicators
98. Return to previous menu
99. Log out

S1> 13

Service Indicators
1. System Attention Indicator
2. Enclosure Indicators
3. Indicators by Location code
4. Lamp Test
98. Return to previous menu
99. Log out

S1> 1

System Attention Indicator
Currently: On
Turn off the system attention indicator
Enter 1 to confirm or 2 to cancel:
The system attention indicator is turned off.
PRESS ENTER TO CONTINUE:

Related posts:

1. Which Distro for PPC64 Server?

In my last post, “Interesting New Developements…“, I mentioned how AIGLX and DRI weren’t working with LDM. Turns out, if I use LDM_DIRECTX=true in lts.conf, that was indeed the case. But if I was using LDM_DIRECTX=false I would receive proper software rendering support reported by glxinfo instead of that BadRequest problem I had with the direct X option. I still haven’t figured out why but I’m sure it’s related to the SSH tunnelling involved in the non-direct X11 connection.

Also, I ragged on LDM quite a bit. Despite the perceived shortcomings, those interface issues can be resolved over time and I hope they will be because the real meat of LDM seems to work pretty well.

Related posts:

1. LTSP 5 and AIGLX

At present I’m doing a technology review for implementing a new terminal server. Our existing terminal server is a 4-way AMD Opteron 848 system that’s about 5 years old right now. It runs CentOS 4 and has been so mega-customized over those 5 years, I’ve never wanted to go through the pain of in-place upgrading to CentOS 5. We also have a simple IBM 1U server running Windows 2003 Server for windows purposes. It’s ok but also about 5 years old.

The idea is to roll both these servers into a large single physical server with some kind of virtualization. The large system would also have the resources to run other VMs, as necessary. Development/test boxes or what not.

Hardware and Virtualization

The server is a 48 core AMD 6172 (2.1GHz) with 64GB of DDR3 ECC RAM with a bunch of 15K RPM SAS drives. I’m not sure whether the AMD solution is entirely the right solution. Intel does score amazingly well on virtualization workload benchmarks… but our workload is different than the benchmark workloads. What we need is one huge guest with a crap-ton of VCPUs and one guest with 2 VCPU and 4GB of RAM. The last NUMA node/cell is pinned to the Windows (smaller) VM and the Linux (large) VM is pinned across several NUMA cells. The Linux VM is a multi-user system where many users will want to run intensive computation but desktop as well. The existing server would sometimes be starved of resources when too many users would start intensive programs simultaneously.

What I can’t quite seem to figure out regarding this whole KVM this yet is if the guest is smart enough and if KVM allows for mapping the VCPUs and the guest memory according to physical NUMA topology thus reducing likelyhood of slow inter-cell memory access?

Software

Anyways… now that the server has arrived, I’ve started out with RHEL6 beta2 as the “hypervisor”, if you will. I’m obviously using KVM and libvirt as this is what RedHat is backing. So far, so good. I’ve only used virt-manager and virtsh thus far, I’ll explore other tools a little later. Fedora 14 is being used as the Linux VM and Windows 7 Enterprise for the guest… I’m going to try out using the Terminal Server multi-user hack and see how that goes. If it won’t go, I’ll recommend actually buying the correct Windows Server license, I suppose. Or buy into the VDI stuff that’s going on around me. I’ll check out the pricing I suppose… but I digress.

Linux Terminal Server: LTSP5

Fedora 14 with LTSP5 works pretty well. But there are some caveats.

1. The chroot

The current ltsp* packages in Fedora14 aren’t able to build Fedora 14 chroots. You can currently only build Fedora 13 or older because it takes some work to make a complete kickstart LTSP chroot from a new release. Instead of  invoking “ltsp-build-client” blindly, you’ll need to do something like:

# ltsp-build-client --release 13

This isn’t a huuuuuuge deal, but in an ideal world I would prefer to use the same release for clients and servers. It’s just cleaner and makes maintaing everything a bit come congruent for me. It also has some troubleshooting benefits. Bah!

2. The Display Manager Situation

LDM, the LTSP5 display manager is both wonderful and woeful. There are some really nice things that LTSP5 can tout due to LDM, but it’s also a step backwards compared to GDM (yes, even the new all-gtk GDM with reduced XDMCP functionality) or KDM in many ways.

What LDM does so well is proper setup and teardown of LOCALDEV and sound via pulse. It’s actually pretty slick, especially in GNOME where it gives the users desktop drive icons for portable USB drives/keys. The automatic unmounting of the drive is actually pretty slick but initially it was highly counterintuitive to someone like me who expects to require ejection/unmounting of the drive before pulling it out.

What LDM does poorly is… well frankly a few things. First off, the feedback from password prompt is poor. It can’t tell WHY your password failed due to the way it interacts with SSH. It’s a hard problem to solve, apparently but it’s a terrible user experience. Second, when you type your password incorrectly, it pauses for some time, tells you it can’t connect to the server and X restarts to load LDM again. Again, bad user experience. It’s also not highly customizable. As an example, the login box will span multiple monitors by default so it’s split across the bezels of your sweet dual monitor thin client. While you can “hack” it by providing a wide logo to force the login box off to the right, it’s not exactly super slick that way.

The last thing LDM doesn’t seem to do at all is allow for AIGLX/DRI2. If I login using LDM, glxinfo/glxgears barfs on a BadRequest related to DRI2/DRI2Connect. With an XDMCP connection to GDM on the same server, with the same client chroot, lts.conf and xorg configuration and glxinfo displays software rendering but that’s a different story. Even using a X11 forwarding to another server provides yet more different results. Either way, it appears LDM basically breaks functionality where it would otherwise run, but run fairly slowly. Could be the open source “radeon” driver I’m using on the test box, I suppose. I hear the intel driver works well…

Since the LOCALDEV and sound routing stuff is tightly integrated to LDM, it appears to be some work to get it working through GDM or KDM… then there is the problem that I haven’t had any success making the new GDM (~ >=2.30, I think) to act as an XDMCP chooser. I do have other XDMCP hosts that I want to connect to…

There is some hope to be had from this interesting post on the Ubuntu help/documentation wiki. It details how to install GDM on the LTSP chroot and get LOCALDEV working with it. They claim sound just works on that older version of Ubuntu but I don’t recall my sound device showing up when using XDMCP and remote GDM. Either way, LOCALDEV is more important.

Windows Terminal Server: Windows 7 Enterprise

I can’t start without saying that Windows 7 via RDP feels “slower” than Windows 2003 over RDP. I detect more mouse and menu lag from the same client systems and versions. I disabled aero and it helped but only a very little bit. I’m not ready to give up yet, there may be more things I can do here.

As mentioned, I’m trying the termsrv.dll hack that allows for multiple RDP users to non-Windows Server Terminal Services hosts. I wonder if it’s part of the performance issue I’m seeing… I should quickly revert the hack to test that possibility.

On the terminal server and with dual 4:3 monitors, not much of the GUI pizzaz in Windows 7 is all that interesting or useful. Some UI changes compared to XP make things a tad unfamiliar for me but overall it’s the same experience with a few nice things and seemingly better stability and driver support.

Related posts:

1. LTSP 5 and AIGLX
2. A Correction
3. Fresh Win2k Install and Windows Update Error

As if there wasn’t already enough discussion surrounding AoE vs. iSCSI in mailing lists, forums and blogs, I am going to add more baseless opinion to the existing overwhelming heap of information on the subject. I’m sure this will be lost in the noise but after having implemented AoE with CORAID devices and iSCSI with an IBM (well, LSI) device and iSCSI with software targets in the past I feel I finally have something share.

This isn’t a technical analysis. I’m not dissecting the protocols nor am I suggesting implementation of either protocol for your project. What I am doing is sharing some of my experiences and observations simply because I can. Read on, brave souls.

Background

My experiences with AoE and iSCSI are limited to fairly small implementations by most standards. Multi-terabyte and mostly file serving with a little bit of database thrown in there for good measure. The reasoning behind all the AoE and iSCSI implementations I’ve setup is basically to detach storage from physical servers to achieve:

1. Independently managed storage that can grow without pain
2. High availability services front-end (multiple servers connecting to the same storage device(s))

There are plenty of other uses for these technologies (and other technologies that may satisfy these requirement), but that’s where I draw my experiences from. I’ve not deployed iSCSI or AoE for virtual infrastructure which does seem to be a pretty hot topic these days, so if that’s what you’re doing, your mileage will vary.

Performance

Yeah, yeah, yeah, everyone wants the performance numbers. Well, I don’t have them. You can find people comparing AoE and iSCSI performance elsewhere (even if many of the tests are flawed). Any performance numbers I may accidentally provide while typing this up in a mad frenzy are entirely subjective and circumstantial… I may not even end up providing any! Do you own testing, it’s the only way you’ll ever be sure.

The Argument For or Against

I don’t really want to be trying to convince anyone to use a certain technology here. However, I will say it: I lean towards AoE for the types of implementations I mentioned above. Why? One reason: SIMPLICITY. Remember the old KISS adage? Well, kiss me AoE because you’ve got the goods!

iSCSI has the balls to do a lot, for a lot of different situations. iSCSI is routable in layer 3 by nature. AoE is not. iSCSI has a behemoth sized load of options and settings that can be tweaked for any particular implementation needs. iSCSI has big vendor backing in both the target and the initiator markets. Need to export an iSCSI device across a WAN link? Sure, you can do it, never mind that the performance might be less than optimal but the point is it’s not terribly involved or “special” to route iSCSI over a WAN because iSCSI is designed from the get-go to run over the Internet. While AoE over a WAN has been demonstrated with GRE, it’s not inherent to the design of AoE and never will be.

So what does AoE have that iSCSI doesn’t? Simplicity and less overhead. AoE doesn’t have myriad of configuration options to get wrong, it’s really so straight forward that it’s hard to get it wrong. iSCSi is easy to get wrong. Tune your HBA firmware settings or software initiator incorrectly (and the factory defaults can easily be “wrong” for any particular implementation) and watch all hell be unleashed before your eyes. If you’ve ever looked at the firmware options provided to by QLogic in their HBAs and you’re not an iSCSI expert, you’ll know what I’m talking about.

Simplicity Example: Multipath I/O

A great example of AoE’s simplicity vs. iSCSI is when it comes to multipath I/O. Multipath I/O is defined as utilizing multiple paths to the same device/LUN/whatever to gain performance and/or redundancy. This is generally implemented with multiple HBAs or NICs on the initiator side and multiple target interfaces on the target side.

With iSCSI, every path to the same device provides the operating system with a separate device. In Linux, that’ll be /dev/sdd, /dev/sde, /dev/sdf, etc. A software layer (MPIO) is required to manage I/O across all the devices in an organized and sensible fashion.

While I’m a fairly big fan of the latest device-mapper-multipath MPIO layer in modern Linux variants, I find AoE’s multipath I/O method much, much better for the task of providing multiple paths to a storage device because it has incredibly low overhead to setup and manage. AoE’s implementation has the advantage that it doesn’t need to be everything to every storage subsystem, which fortunately or unfortunately device-mapper-multipath has to be.

The AoE Linux driver totally abstracts multiple paths in a way that iSCSI does not by handling all the multipath stuff internally. The host is only provided with a single device in /dev that is managed identically to any other non-multipath device. You don’t even need to configure the driver in any special way, just plug in the interfaces and go! That’s a long shot from what is necessary with MPIO layers and iSCSI.

There’s nothing wrong about device-mapper-multipath and it is quite flexible, but it certainly doesn’t have the simplicity of AoE’s multipath design.

Enterprise Support

Enterprise support is where iSCSI shines in this comparison. Show me a major storage vendor that doesn’t have at least one iSCSI device, even if they are just rebranded. Ok, maybe there are a few vendors out there without an iSCSI solution but for the most part all the big boys are flaunting some kind of iSCSI solution. NetApp, EMC, Dell, IBM, HDS and HP all have iSCSI solutions. On the other hand, AoE only has only a single visible company backing it at the commercial level: CORAID, a spin-off company started by Brantley Coile (yeah, the guy who invented the now-Cisco PIX and AoE). I’m starting to see some Asian manufacturers backing AoE on the hardware level but when it comes to your organization buying rack mount AoE compatible disk trays, CORAID is the only vendor I would suggest at this time.

This isn’t so fantastic for getting AoE into businesses but it’s a start. With AoE in the Linux kernel and Asian vendors packing AoE into chips things will likely pickup for AoE from an enterprise support point of view: It’s cheap, it’s simple and performance is good.

Conclusion

AoE rocks! iSCSI is pretty cool too, but I’ve certainly undergone much worse pain working with much more expensive iSCSI SAN devices vs the CORAID devices. And no performance benefit that I could realize with moderate to heavy file serving and light database workloads. I like AoE over iSCSI but there are plenty of reasons not to like it as well.

To each their own, I say.

Related posts:

1. Migration Weekend: Success