Not for Domains

It’s important to keep in mind that these instructions are not for a integrating FreeIPA with a Samba domain controller but merely a Samba file server. My understanding is that FreeIPA will never conveniently/properly support the necessary bits to make it a suitable backend for a Samba 3 PDC. I believe FreeIPA will eventually look towards Samba 4 integration (using Domain trusts) for this kind of integration but don’t quote me on that. Either way, these instructions are not for Samba domain controllers, just Samba file servers.

The Assumptions

There are some basic assumptions that these instructions make.

• FreeIPA is installed and functional
• You have a general idea of how to use LDAP command line tools
• If you have a nice GUI LDAP browser, you can use it to apply the example LDIFs and edit the tree instead of the ldap CLI tools
• The LDAP commands are executed on the FreeIPA server
• Samba and FreeIPA are installed on the same server (although it shouldn’t be difficult to use TLS encryption with separate servers)
• Your LDAP suffix is “dc=domain,dc=tld”
• You know the difference between the “admin” account and the directory manager and their passwords

The Goods

Let’s not beat around the bush any further.

1. Determine your Samba server SID by executing the following command while smbd is running and jot it down:

root@ipaserver:~
# net getlocalsid
SID for IPASERVER domain  is: S-1-5-21-3180075094-3458813485-3821849995

2. With the “admin” kerberos ticket, add two attributes to “cn=ipaConfig,dc=etc,dc=domain,dc=tld” that tell FreeIPA to setup each account as a Samba account and each group as a Samba group:

ldapmodify -Y GSSAPI <<EOF
dn: cn=ipaconfig,cn=etc,dc=domain,dc=tld
changetype: modify
add: ipaUserObjectClasses
ipaUserObjectClasses: sambaSAMAccount
-
add: ipaGroupObjectClasses
ipaGroupObjectClasses: sambaGroupMapping
EOF

3. With the directory manager password and the Samba SID you jotted down from above, create an instance of the 389 DS DNA plugin that will automatically generate SIDs for your users and groups which are necessary for use with Samba:

ldapadd -x -D "cn=Directory Manager" -W <<EOF
dn: cn=SambaGroupSid,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
objectClass: top
objectClass: extensibleObject
dnatype: sambaSID
dnaprefix: S-1-5-21-3180075094-3458813485-3821849995-
dnainterval: 1
dnamagicregen: assign
dnafilter: (|(objectclass=sambasamaccount)(objectclass=sambagroupmapping))
dnascope: dc=domain,dc=tld
cn: SambaSid
dnanextvalue: 15277
EOF

The thing to note here is that the “dnaprefix” is set to the SID your jotted down… PLUS a hyphen (“-”) appended to the end!

4. Now we have to start modifying the FreeIPA API, CLI and WebUI to allow us to specify the “sambaGroupType” attribute at group creation time. We have to set “sambaGroupType” because it is a required attribute for the objectClass “sambaGroupMapping” which we are automatically adding to every group with the “ipaGroupObjectClasses” setting from earlier.

Although the value is going to be “4″ for every conceivable case in this non-domain configuration, I was not able to figure out how to make the DNA plugin insert static values like it can set incrementing values so I decided to allow setting it through the CLI and WebUS with defaults enabled instead. If anyone knows how to setup 389 to automatically add an attribute with a static value upon DN creation of DNs with specific objectClasses, please tell me.

There are a few steps required to make this CLI/UI stuff happen but the FreeIPA developers have actually made this quite simple.

The rule is: Extend the FreeIPA schema first, then the CLI, then the WebUI.

4.1. Extend the FreeIPA schema with a custom field by adding the attribute “ipaCustomFields” with a value of “Samba Group Type,sambagrouptype,true” to “cn=ipaConfig,dc=etc,dc=domain,dc=tld” with an “admin” kerberos ticket:

ldapmodify -Y GSSAPI <<EOF
dn: cn=ipaconfig,cn=etc,dc=domain,dc=tld
changetype: add
add: ipaCustomFields
ipaCustomFields: "Samba Group Type,sambagrouptype,true"
EOF

As there can only be one “ipaCustomFields” attribute, if you have multiple custom fields you need to separate each definition with a “$” like so: “Samba Group Type,sambagrouptype,true$Description,attrname,isrequiredboolean”.

4.2. Extend the CLI for groups by editing the python file “/…/site-packages/ipalib/plugins/group.py” to define the custom field and specify a default if not implicitly defined (diff):

--- group.py.orig	2011-08-15 14:59:48.570715207 -0700
+++ group.py	2011-08-16 12:43:43.493236507 -0700
@@ -118,6 +118,13 @@
             label=_('GID'),
             doc=_('GID (use this option to set it manually)'),
         ),
+        Int('sambagrouptype',
+            cli_name='sgt',
+            label=_('Samba Group Type'),
+            doc=_('Samba Group Type (default is 4)'),
+            default=4,
+            autofill=True,
+        ),
     )

 api.register(group)

Important: Restart “httpd” at this point!

4.3. Test the CLI. With an “admin” (or equivalent priv) kerberos ticket, try creating a new group:

account@ipaserver:~
$ ipa group-add testgrp --desc="Testing the group.py CLI mods"
---------------------
Added group "testgrp"
---------------------
  Group name: testgrp
  Description: Testing the group.py CLI mods
  GID: 1234500010
  Samba Group Type: 4

4.4 With the CLI functioning properly, we can move on to extending the WebUI. To extend the WebUI for group attributes, edit “/usr/share/ipa/ui/group.js” like so (diff):

--- group.js.orig	2011-08-15 10:01:28.515209121 -0700
+++ group.js	2011-08-16 13:52:59.587352034 -0700
@@ -34,6 +34,7 @@
                 column({name: 'cn'}).
                 column({name: 'gidnumber'}).
                 column({name: 'description'}).
+                column({name: 'sambagrouptype'}).
                 dialog(
                     IPA.add_dialog({
                         'name': 'add',
@@ -41,6 +42,7 @@
                     }).
                         field(IPA.text_widget({name: 'cn', undo: false})).
                         field(IPA.text_widget({name: 'description', undo: false})).
+                        field(IPA.select_widget({name: 'sambagrouptype', undo: false, options: [{label: 'Local', value: 4}, {label: 'Domain', value: 2}]})).
                         field(IPA.checkbox_widget({
                             name: 'posix',
                             label: IPA.messages.objects.group.posix,
@@ -56,6 +58,7 @@
                     }).
                         input({name: 'cn' }).
                         input({name: 'description'}).
+                        input({name: 'sambagrouptype'}).
                         input({name: 'gidnumber' }))).
         facet(
             IPA.group_member_user_facet({

And then these the WebUI to ensure that you can both see the attribute in the group list, but also add it via the select widget added to the new/edit group dialog.

That should be it. Questions, comments, suggestions, correction and more… all are welcome!

Related posts:

1. LDAP User Management Tools and User Private Groups
2. Life Support!

I arrived shortly after 8AM, registered to receive my badge and t-shirt then milled around the vendor booths until the keynotes were ready to start. I watched the keynotes (Jim Zemlin, Linux Foundation and Jim  Whitehurts, Red Hat), went to every session I could and came back to the main ballroom for the panel discussion with Jon “Maddog” Hall, Eben Moglen and Dan Frye and the following interview of Linus Torvalds by Greg Kroah-Hartman to wrap things up for day 1. So far, so good.

The Keynotes

Jim Zemlin’s opening keynote “Imagining a World Without Linux” was decent. While he did take some inevitable potshots at Microsoft, the message was generally very positive and uplifting. I won’t go into details but basically Jim described a world without Linux as one that would be black & white as opposed to the colour filled world we know today (due to Linux). Jim is a smiley and positive person on stage, his style helped kick off LinuxCon 2011 with a good vibe.

Jim Whitehurst, CEO of Red Hat had a similar approach of sending positive vibes but focused on how the progress of Linux and Open Source has enabled businesses and business models. He said that Google wouldn’t exist (at least not in it’s current form) without Linux and basically implying the same about other major well know Linux-powered companies such as Amazon and Facebook. Jim struck me as a fairly modest fellow but he wasn’t shy about mentioning Red Hat’s penetration into Fortune 500 companies. Nor was he reserved about how Linux has powered, enabled, strongly driven by or directly benefitted various global forces that may or may not be angels (U.S. Navy, NSA, Russian Military, NYSE/Wall Street). While his examples spoke to the breadth of applications for and the wide reach of Linux, I couldn’t help but think about how the pervasiveness of Linux is not only helping drive great positive change int he world but may also be powering negative forces as well.

Overall both Jim’s did a good job and left me excited for the rest of LinuxCon to come.

First Day Sessions

I attended four sessions on day 1:

1. Centralized User Administration with FreeIPA and sssd by Stephen Gallagher
2. Watching Mad Men and Thinking About Open Source by Karen Copenhaver
3. 20 Years – And More – of Kernel Development by Jon Cobert
4. What to Expect from Linux Storage by James Bottomley

Centralized User Administration with FreeIPA and sssd

My first LinuxCon session was by Stephen Gallagher of Red Hat. As is clear by the title, it was about FreeIPA and sssd, two emerging Red Hat driven projects relating to centralized directory and authentication services. Stephen wasn’t the most natural speaker I’ve had the pleasure to watch and I suspect that presentations aren’t something he does on a regular basis but he clearly knew his material and he was able to field the post-presentation questions with ease. The presentation material was fairly spot on to what I expected. I should stop by the Red Hat booth and speak with Stephen tomorrow as there are a few FreeIPA/sssd related questions I have which I didn’t ask during the question period. Overall, I was satisfied.

Watching Mad Men and Thinking About Open Source

First of all, Karen is a more natural speaker than Stephen but I suppose that’s to be expected: She is legal counsel for the Linux Foundation. The material in this session while clear and understandable was maybe not quite as impactful as I had hoped. Karen had some very nice points and brought good historical reference to the table but it wasn’t really anything that I didn’t already think think about in my own internal dialog, for the most part.

Some key points that Karen made early which did resonate with me:

• “It’s a privilege to work on something so important”, I believe she was quoting Linus Torvalds. This hits home for me as my work is only to enable the much more important and relevant work of others.
• The observation that the open source community generally doesn’t have time for anything but the truth which is a nice ideal but perhaps isn’t necessarily reflective of the entire open source world so much as a few of the important luminaries.
• Identify the things that you value and… well I missed that part. But I do think identifying the things you value is, well, valuable.

These are all straight forward things but to hear someone say them can be powerful. This session was good but it wasn’t quite as hard hitting as I thought it might be based on the title and description. It was no let down, though.

20 Year – And More – Of Linux Kernel Development

Ok, now we’re getting way out of my league. Jon Corbet is a high profile Linux kernel contributor and he knows what he is talking about. This man has confidence and ostensibly the knowledge to back it up. His overview of the last 20 years of Linux kernel development was excellent and spotted with just enough humour to keep the real developers cracking up and the rest of us only getting every second joke.

Jon’s timeline approach to describing the history of kernel development was excellent and enabled him to visually map releases, events and growth in a very simple and understandable way. He made an excellent observation regarding the pace (measure by lines of code) of Linux kernel development during the dot com bust not slowing down one bit despite industry turmoil and job loss and pointed out the correlation between important points in Linux kernel development time with other events that may not be obvious to every outsider (BitKeeper, Git, time between certain releases, Merge Window, etc.).

While this session was developer focused, it wasn’t so technical to be devoid of value for anyone else, in fact I think it really helped frame the history of Linux kernel development for me in a way that I had never experienced before. Way to go, Jon.

What To Expect From Linux Storage

I’m not sure why James’ talk was titled what it was because for the best of what I could tell, the majority of the talk was about what already is, not what to expect. That’s not to say it was devoid of important information regarding “what to expect” and maybe it was because James ran out of time and had to skip some slides but I did find the title interesting in that capacity none-the-less.

James is charismatic. He makes jokes, he wears a bow tie, he speaks with an attractive accent. He’s also clearly very knowledgable about his part of the Linux kernel: the Block layer.

Being a sysadmin, knowing more about the block layer and James’ perspective on storage was hugely beneficial. He has historical reference that I never will and deep knowledge of the kernel which I’ll never achieve. With that said, some of his opinions regarding specific technologies and methods, I personally already held myself! How is it that a Linux kernel rube such as myself could had gleaned the same opinions on specific technologies as one of the people  who understands these technologies the best of anyone? iSCSI was an example. I think it’s safe to say James thinks iSCSI is an abhorrent mess that simply tries to solve a problem in entirely the wrong way. I’m also not a big fan of iSCSI and his reasoning  resonated with me, despite my lack of in depth knowledge.

I could go on because I liked this session but I already feel like I’m burning myself out on this summary of day 1 and we haven’t even gotten to the panel discussion or Linus interview yet.

Panel Discussion

The panel discussion with Jon Hall, Eben Moglen and Dan Frye was fairly profound despite Eben using the platform for an interesting but strangely placed speech that appeared entirely scripted/written. That’s not to say I didn’t like his speech or that I don’t agree with him or his world views but the way he momentarily took over the panel with what was clearly a pre-planned speech during a panel discussion main-hall format was strange indeed.

Dan Frye struck me as level-headed and one of those business people whom can take the challenge of  balancing the need to run a profitable business with social awareness and decency and excel at it. I’ve never really doubted IBM’s commitment to Linux and I know their commitment is based on profitability but the way that Dan framed the reasons that he and his team knew Linux meant good business for IBM put a smile on my face.

Jon Hall’s experience in the computing industry is staggering and humbling, even for today’s big shots. What a dude. Level head, very articulated, sense of humour and a huge white beard. It’s hard not to love the guy after watching that panel discussion. Jon talked about his hopes for how Linux and the open source model will foster the next generation of great thinkers, movers and shakers and enable them to do great things. I liked that.

I’m not really sure what to say about Eben. I agreed with everything he said but he just wasn’t as loveable as Jon Hall. Must be because he’s a lawyer :D I suppose that slightly awkward speech about the troubled times that are looming (mounting patent threats and inevitable “10-20 billion” dollar war) could have been a factor as well. That said, he seemed positive despite the heavy and serious tone he used to describe the battles ahead.

On one hand, the panel discussion left me feeling good and uplifted but on the other hand I was left with a feeling of powerlessness. I’m not one of the next great thinkers, doers or talkers. What’s my place in the Linux and open source world, then? Everything that was discussed revolved around the greatest minds in open source and the huge impacts made by major players. I almost felt a little left out as a lowly sysadmin whom has to deploy at least some non-RMS blessed systems alongside the requisite Linux systems. What’s my role in all this?

Interview with Linus

I really don’t have much to say about this one. Linus is down to earth, but strong in his opinions. He admits when something is outside of his immediate expertise, as evidenced by his answers to many non-Linux kernel specific questions. He talks well and he would have preferred if the crowd did not give him a standing ovation at the end but I suppose you cannot make a room full of Linux geeks sit down when their proverbial leader is being applauded.

I liked a lot of what Linus talked about regarding the modern direction of Linux such as the version numbering changes, the idea that we should be looking backwards at how to improve existing subsystems and layers instead of always looking forward to new feature inclusions. I liked how he described the cross-pollination of various parts of Linux that exist when everyone from embedded systems to massively parallel SMP systems are made to use the exact same kernel instead of everyone having their own specialized forks.

Linus was clam and cool, just like Linux and I had a seriously good time at LinuxCon today. Rock on, LinuxCon!

Related posts:

1. Is Ubuntu Ready for the Enterprise?
2. LVM filters and initrd

Ever since we transitioned from the RHEL4 environment to Fedora 14, people have been reporting terrible slowness and delays in nautilus when browsing our NFS shares. Reports of waiting over a minute for an NFS automount root-level directory with < 100 sub directories to display the contents are not good.

This wasn’t a problem on our old RHEL4 terminal server and I couldn’t for the life of me understand how nautilus could have become so slow in the years since RHEL4 was released. It just didn’t make sense. I started to think something had to be wrong and that this wasn’t just the new normal expected behaviour but I had nothing to go on.

I tried the basic recommendations: Disable thumbnails, disable preview, disable directory item counts. That didn’t help the user experience in any dramatic way. At this point, I started recommended pcmanfm and thunar as a way to workaround nautilus’ terrible performance. I even wrote a fairly concise script for modifying the default file manager and desktop-drawing application so that using a different file manager wouldn’t be so foreign in GNOME.

Then one day I started looking at the verbose level output from automount while browsing the NFS mounts with nautilus and found a substantial amount of this in the logs:

Apr 28 11:19:10 hostname automount[18959]: attempting to mount entry /home/.svn
Apr 28 11:19:10 hostname automount[18959]: key ".svn" not found in map source(s).
Apr 28 11:19:10 hostname automount[18959]: failed to mount /home/.svn

Oh my! Why are there repeated access attempts for “.svn”? What is causing automount to perform map lookups for “.svn” in the automount-controlled directories? Could it be nautilus?

Why yes!

As it turns out the GNOME SVN integration package “gnubversion” includes a nautilus extension and this extension was causing Nautilus to look for “.svn” directories everywhere and it just so happens that looking for “.svn” in a root-level automount directory causes slow map lookup failures that (presumably) kill the perceptible performance of browsing automounted NFS shares.

I removed gnubversion (as no one was using it) and the user experience for nautilus has normalized. While nautilus still isn’t as speedy as pcmanfm or thunar, its no longer a cause of forceful hair removal incidents… and all is well in the world.

Related posts:

1. RHEL/CentOS, NFS and Firewalls
2. Is Ubuntu Ready for the Enterprise?
3. POSIX Default ACLs, umask and Project Directories

We run Dovecot 1.1 with a MySQL backend and Postfix for MTA duties. Everything requires both TLS and SSL for authentication and everything requires authentication except for sending mail from the local subnet. It works pretty well. I never touch the thing anymore, it just runs. However, it doesn’t support push email and it certainly doesn’t support ActiveSync. So I went looking for something that could do push email to my spankin’ new phone.

I was surprisingly happy to discover z-push, an open source, standalone ActiveSync implementation in PHP. Well hot damn!

I initially installed the latest stable release, but then quickly tried the SVN trunk for any potential fixes that have yet to make it out to the stable release because I wasn’t having much success. After a few simple problems got resolved and I was updated to the SVN trunk things started to work… kind of. The initial sync takes forever! I didn’t have the patience to wait for all my mail to download because it appeared to be taking several minutes per email. The folder list loaded right up and my nearly empty inbox too but any folder with more than a few messages was taking forever to sync. Not to mention the apache server started to churn CPU pretty hard on the server. Also, it seemed the sync would only even start to work if I had “No Limit” selected on the iPhone for history of emails to sync. Maybe the large volume initial sync by using “No Limit” is just too taxing and that’s why it’s brutally slow but I kept getting “Cannot Get Mail – The connection to the server failed.” on my iPhone if I selected any option besides “No Limit”.

On top of that, push didn’t work!

I’ll keep plugging away at it next week, maybe post on the z-push forums to see if I can get this figured out. Cheers for now.

Related posts:

1. When using Syncrepl…

Now that I want to write a site-specific GUI-based LDAP management tool as referenced in a previous post, I’m jumping back into software development a little bit. I’ve decided to use wxPython for a few reasons. We already use it within our group, we have in-house expertise in the form of an actual developer and the GUI-builder tools seem to work best with wxPython (wxGlade was producing bad wxPerl code, go figure).

Let me also say that I’ve never been a big python guy. I’m really a perl kind of dude, so this learning a new object oriented API while learning python at the same time is a challenge… but I’m making progress! Basically, wxPython seriously kicked my ass for about 3 days but now I’m gaining speed and things are moving faster than I expected. Python and wxPython are starting to make more sense and behave.

So far I have a frame with menubar, statusbar and a three-tabbed notebook with a grid on each tab of the notebook. The program is able to connect to an LDAP server (with TLS) and query the directory for all the users, groups and autofs information and then display that information in the grid… and that’s about it so far. I need to build in new user, group and autofs functionality as well as basic editing of existing entries (in place editing with wx.Grid looks really nice but I haven’t tried it yet!).

Although my ass has been kicked for the last few days, I’m actually feeling pretty optimistic about it now.

Hopefully I can share the kludge code at some point, though it will never be easily transportable to different environments since I’m not building this to be the end-all-be-all LDAP user/group/autofs management tool, just one tailored for our environment.

No related posts.

I’m well aware of the FreeIPA project and that project does in fact support UPG, probably because it’s a RedHat project but I’ve determined that FreeIPA is too comprehensive for my needs. Despite Kerberos being the “right” solution in every sense of the term, I’d rather have the simplicity of binding to the LDAP server for authentication, even though I know that using LDAP as an authentication service is “wrong”.

My question, loyalty challenged readers, is: Are there any LDAP user management tools out there that support UPG?

Let me start the list:

• LAM – NO
• phpLDAPadmin – NO
• Luma – NO
• LAT – NO
• Gosa – NO
• smbldap-tools – Maybe?

Not to bash any of those tools, but I’ve decided to start writing my own simple “useradd” script for now because the workflow for creating a user with the UPG scheme with any of these tools is an annoying multi-step process. While my solution is site-specific and non-comprehensive, it just exactly the job I need done, done. And fast. I used perl and Net::LDAP, among other modules. Once I figured out if I want to it keep it on the console or move it to the web, I’ll post the results… even if it won’t be useful to anyone as-is.

Related posts:

1. Which Distro for PPC64 Server?
2. FreeIPA and Samba 3 Integration
3. Migration Weekend: Success

The example is that of a very, very basic sudoers policy but the principles are easily extended to create much more complex policy. The general idea here is that we want cfengine to ensure that specific rules are always in place. Instructed properly, cfengine accomplishes this very well.

Warning: I don’t know anything. I’m just someone learning cfengine 3 and posting about it. If I’m wrong about something, let me know! If you find this at all useful, be my guest. That is all.

################################################################################
##
## FILE: sudo.cf
## DESC: Control /etc/sudoers file on various servers
##
################################################################################

bundle agent sudo
{

vars:

  "sudoers" string => "/etc/sudoers";

  "sudo"     slist => {
                      "%admin ALL = ALL",
                      "%sysadmin ALL = /sbin/mount",
                      "%devel ALL = /sbin/mount"
                      };

packages:

  Night::

  "sudo" -> "Security policy"
    comment               => "Ensure sudo is up to date every 24 hours (and only at night)",
    package_policy        => "update",
    package_method        => yum,
    package_architectures => { "$(sys.arch)" },
    action                => if_elapsed("1440");

files:

  "$(sudoers)" -> "Security Policy"
    comment      => "Append common configuration to sudoers",
    edit_line    => append_if_no_lines("$(sudo)");

}

As with last snippet I posted, the above does not even resemble a complete cfengine policy/configuration, just a small portion that can be contained in it’s own bundle. It can be put in a separate .cf file, imported by promises.cf and added to the bundle sequence, inheriting variables and classes! Also, just like last time I’m using cfengine’s built in interface for package management systems to ensure “sudo” is always installed via yum at night, every 24 hours.

What’s new here is file editing with edit_line and the use of iteration which proves to be very powerful in cfengine 3.

Editing Files

Editing files with cfengine is supposed to be easy but initially it seemed a bit awkward to me.

First you have the promise file promise:

files:

  "$(sudoers)" -> "Security Policy"
    comment      => "Append common configuration to sudoers",
    edit_line    => append_if_no_lines("$(sudo)");

Which makes some reference to the edit_line facility and what looks like a function name with append_if_no_lines(…).

Then you have the edit_line bundle defined elsewhere:

bundle edit_line append_if_no_lines(list)
{
insert_lines:

 "$(list)";
}

Which describes what “append_if_no_lines” actually does.

Of course I’m just learning cfengine and new things can often seem strange and scary but I am finally warming up to editing files with cfengine… I think. What I seemed to have initial trouble with was with the bundles necessary for edit_lines, as described above. The bundle within a bundle concept. The append_if_no_lines and append_if_no_line bundles I’m using are implemented in the cfengine std library, which is highly recommended so that you may avoid re-inventing the wheel a little bit.

For basic promises, to add or remove, comment or uncomments lines and the such there are good edit_lines bundles available in the stdlib. For other more complex or customized file editing, writing your own bundles will be necessary. Either way, understanding what a bundle is and how to create your own is key to fully grasping file editing and getting the most out of it. This seems obvious in retrospect but something I didn’t pickup instantly.

See the cfengine documentation for more about editing files, check the cfengine documentation. There’s waaaaay more good information over there and it’s from the cfengine team, not some random newb.

Iteration

Iteration is powerful mechanism within cfengine that harnesses the power of lists to express a large possible number of actions/operations with very little amount of code. When lists are used, single actions can be made to repeat for every item in a list by using the $(varname) syntax to refer to the list… which as it turns out is the same for scalar values! Funny that!

So cfengine allows us to define X different lines of code to ensure are in a file using only a single file: promise all with the same simple syntax as scalar variables? Brilliant!

A demonstration of iteration can be seen with the $(sudo) slist and the “Append common configuration to sudoers” file: promise. With this single promise definition, up to 6 actual promises are made because the $(sudo) variable is an slist. Each element or item in the list is iterated over in sequence and the promise is evaluated and acted upon, if necessary. The reason that up to 6 promises will be evaluated is the ifvarclass property of promise, ensuring the promise will only be kept if we’re in the context of the class… and looking at the promise to find out which class, we see another example of iteration using the $(sudo) list and the canonify function that turns a string into a class. Thusly, if the host currently running this policy defines all the classes that are tested by the ifvarclass iteration, 6 promises will be made. If the host defines 3 of the classes, then 3 promises will be made and so on, and so forth.

As a beginner, using lists and iteration effectively and creatively seems fairly important to getting the most out of cfengine 3.

Editing Files vs. Copying Files

In my previous snippet, I demonstrated how to promise to copy a file from a secure remote server if the local file does not match the server’s file in order to manage a configuration with cfengine. This time, I’m promising to add lines to a configuration file if they do not already exist exactly as provided.

This represents two rather different takes on policy. The first says: “The configuration must always be exactly like this file, byte per byte!” the second says “These lines must exist but I don’t care about anything else in the file”. The file copy method is what I would call hard policy and the second is soft policy. In the cfengine community solutions, they recommend managing sudo by copying an /etc/sudoers from a remote server. That way is great (just like my DenyHosts example) but this is just another way if you have a use case for cfengine not owning every byte of your configuration file.

Conclusion

Yeah, that’s about it. Enjoy.

Related posts:

1. Cfengine 3 Snippets Part 1: DenyHosts
2. Nanorcs: Ultrasimplistic Configuration File Revision Control
3. RHEL/CentOS, NFS and Firewalls

I did discovery yet more small misconfigurations and strange behaviour along the way:

1. OpenLDAP’s syncrepl using “refereshAndPersist” wasn’t working how I expected it to, no new changes were replicating to the slave LDAP server! I changed the directive to “refreshOnly” and set a 10 minute interval. I made several changes and monitored the slave LDAP server. Changes propagated in about 10 minutes, every time.
2. Despite iSCSI’s maturity and the maturity of QLogic’s HBAs I still noticed strange, unexplained target drop outs. Two HBAs per server, two controllers in the IBM DS3300 and just one target out of four was dropping. At first, I couldn’t figure out how to properly reconnect the target on a live system so I rebooted. Later, I discovered you can “disable” and then “enable” the specific target in SANsurfer or iscli, which worked to bring back the dropped target on a live system. Multipath picked up the “new” path right away, as expected.
3. Always remember to leave free physical extents in any LVM Volume Group in which you are taking snapshots of the Logical Volumes. It’s freakin’ obvious but I forgot and when I went to do snapshot backups, the snapshots were failing. Now I’m growing some LUNs on the DS3300 so that my VGs have room for snapshots.

All in all, a good weekend that was mostly filled with success.

Related posts:

1. Migration Weekend
2. When using Syncrepl…
3. Atempo Time Navigator 4.2 Archive Media Selection Tunable

The Problem

The initial difficulty lies in the requirement that the data must be consistently backed up at every interval, no matter which cluster node is currently the active node with the backend storage mounted. To do this, an agent is required to be configured as a cluster resource in order to “follow” the mounting/exporting of the storage to any cluster node. So in order to accomplish this,  N + 1 tina agents are required. That is, if you have two cluster nodes, you need three agents to successfully backup each node with the local agent and the storage, as it floats about the cluster nodes depending on failure or migration events.

Luckily for me, the good people at Atempo have engineered the agent in such a way that multiple agents can be ran on a single node, each binding to it’s own IP address and each individually controlled via it’s own init script. Of course, we need to make some file edits to make all this happen and that’s what I’m going share!

System Configuration

This configuration is based on CentOS 5.x and Time Navigator 4.2 but should the concepts should be mostly portable to other popular Linux or UNIX distributions. The underlying cluster software used for the majority of my experience with this configuration is Heartbeat 2.1.3, right before the Pacemaker split but has also been more recently tested on Pacemaker 1.0 / Heartbeat 3.0.x. DRBD is used to provide the active/passive cluster-aware state and configuration information to where I’ve installed the Atempo Time Navigator agent but it is possible to install a second agent on each cluster node and configure it identically but this just seems like more work. DRBD does a great job of making sure the latest cluster-aware tina agent is consistently configured and available on the active cluster node, no matter which node that actually is.

For the purpose of this post, I’m going to assume you already have a working Heartbeat/Pacemaker/DRBD configuration up and running with proper STONITH and all that good jazz. Maybe some other time.

Installing and Configuring the Agent on DRBD

The first thing that needs to be done is the tina agent must be installed to a filesystem hosted on DRBD. I generally just SSH around the Linux-X64.tar or Linux-X86.tar Time Navigator installation archive and then decompress and run the install script.

Assuming the dedicated (to this agent resource) DRBD filesystem is mounted as /cluster/tina on the active cluster node:

$ cd /cluster/tina
$ scp user@remote.fqdn:/path/to/Linux-X86.tar ./
$ tar -xf Linux-X86.tar
$ cd Linux-X86
$ ./install.sh

This will bring up the GUI installer. Alternatively use the batch install method, whatever works for you.

Set /cluster/tina as the installation directory and otherwise proceed normally as per site configuration. Unique ports do not need to be used for the second cluster agent as this configuration bind to a floating cluster resource IP address while the local agent binds to (one of) the servers “real” IP address(es).

Once installed, there is one important edit to make in the tina agent environment configuration scripts named .tina.sh (sh/bash) and .tina.csh (csh/tcsh) located in the installation directory (/cluster/tina). The key change to make in the relevant script is to modify the value where the $TINA environment variable is being set. In .tina.sh that would be changing the line:

TINA=tina

to instead read something like this:

TINA=tina_ha

where tina_ha is a unique identifier for this instance of the agent. Basically, it needs to be anything BUT tina. This is one of two key components that had me tricked for a while. I had first tried modifying the $TINA_SERVICE_NAME environment variable but that was a giant red herring because uniquely setting that variable to something other than tina does not produce the desired effect, despite what the looking through the tina environment scripts and init scripts might have you believe.

The second thing we must do is to create an LSB-compliant init script for the cluster-aware tina agent. The LSB compliance is very important to ensure the cluster can manage the resource properly. If any return codes are out of the LSB spec, the cluster will behave erratically and unpredictably when dealing with starting, stopping and monitoring the tina agent.

Since the installation creates a good init script for us, we can copy that script with a new name and edit it.

$ cp /etc/init.d/tina.tina /etc/init.d/tina.tina_ha
$ nano /etc/init.d/tina.tina_ha

First, replace every instance of the path to the local agent’s tina install path with that of the cluster agent’s installation path. A simple search (Ctrl-W) then replace (Cntrl-R) in nano should suffice.

Additionally, we need a small section at the top that will exit the script in case the DRBD filesystem is not mounted. The HA cluster will do resource status checks on all nodes in the cluster and we need the init script to be able to exit with a sane exit code, even if the DRBD filesystem is not accessible (as it is on all passive nodes). Something like this:

if [ -f /cluster/tina/.tina.sh ] ; then
  . /cluster/tina/.tina.sh > /dev/null 2>&1
else
  echo "Unable to start Time Navigator daemon"
  echo "because the \"/cluster/tina/.tina.sh\" file does not exist"
  retval=3
fi

In order to make the script LSB compliant, we need to ensure the correct exit status is returned during the correct operations. Instead of pointing out each specific place I had to edit in order for this to happen, I will simply post my entire “/etc/init.d/tina.tina_ha” init script:

#!/bin/sh
# UPDATED BY SETUP - BEGIN
########################################################
#WARNING :
#THIS FILE IS GENERATED AUTOMATICALLY
#AND WILL BE OVERWRITTEN WHEN UPGRADING
#YOUR VERSION OF Time Navigator PRODUCT
########################################################
PATH="$PATH:/bin:/usr/bin:/sbin:/usr/sbin:/etc:/usr/etc"
export PATH
if [ "${TINA_HOME:+$TINA_HOME}" != "" ] ; then
	if [ "/cluster/tina" != "$TINA_HOME" ] ; then
		echo "Unable to start Time Navigator daemon for \"/cluster/tina\""
		echo "because the Time Navigator environment is already set by \"$TINA_HOME\""
		retval=3
	fi
fi
if [ -f /cluster/tina/.tina.sh ] ; then
	. /cluster/tina/.tina.sh > /dev/null 2>&1
else
	echo "Unable to start Time Navigator daemon"
	echo "because the \"/cluster/tina/.tina.sh\" file does not exist"
	retval=3
fi
# UPDATED BY SETUP - END
# @(#) $Id: rc.tina.orig,v 1.1.6.10.4.4.2.4 2007/09/20 16:26:50 dle Exp $
#
# Time Navigator startup script
# (C) 1999-2005 - Atempo
# tina_daemon starting...
#

OS_TYPE=`uname -s`

if echo "\c" | grep "c">/dev/null ; then
	ECHOMODE=Bsd
else
	ECHOMODE=Sys5
fi

ECHONOCR() {
	if [ "$ECHOMODE" = Bsd ] ; then
		echo -n "$*"
	else
		echo "$*\c"
	fi
}

PING() {
    os_type=`uname -s`
    case $os_type in
        HP-UX) result=`ping $1 -n 2 2>/dev/null`; return $?;;
        *) result=`ping -c 2 $1 2>/dev/null`; return $?;;
    esac
}

ISREDHATLIKE=1
# Source function library.
if [ -f /etc/init.d/functions ] ; then
	. /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
	. /etc/rc.d/init.d/functions
else
	ISREDHATLIKE=0
fi

ISSUSE=1
if [ -f /etc/rc.status ] ; then
	. /etc/rc.status
else
	ISSUSE=0
fi

RCStart()
{
	if [ -x ${TINA_HOME}/Bin/ndmpd ] ; then
		echo "Starting NDMP Data Server..."
		${TINA_HOME}/Bin/ndmpd
	elif [ -x ${TINA_HOME}/Bin/tina_nts ] ; then
		echo "Starting NDMP Tape Server..."
		${TINA_HOME}/Bin/tina_nts
	fi

	TINA_DAEMON=$TINA_HOME/Bin/tina_daemon
	if [ -x "$TINA_DAEMON" ]; then
		ECHONOCR "Starting Time Navigator ($TINA_SERVICE_NAME)..."
		if [ -d /var/lock/subsys ] ; then
			touch /var/lock/subsys/tina.$TINA_SERVICE_NAME
		fi
		i=1
		while [ $i -le 60 ] ; do
			if [ $OS_TYPE = "Darwin" ] ; then
				echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon" >> /var/log/system.log
			fi
			echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon $i" >> ${TINA_HOME}/Adm/auto_start.log
			hostname=`hostname 2>/dev/null`
			if [ ! -z "$hostname" ] ; then
				echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon: hostname $hostname is defined" >> ${TINA_HOME}/Adm/auto_start.log
				PING $hostname
				status=$?
				if [ $status -eq 0 ] ; then
					echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon: ping $hostname is ok" >> ${TINA_HOME}/Adm/auto_start.log
					$TINA_DAEMON
					sleep 2
					RCStatus no_mess
					if [ ! -z "$is_running" ] ; then
						if [ $OS_TYPE = "Darwin" ] ; then
							echo `date` "tina_daemon ($TINA_SERVICE_NAME) daemon is started" >> /var/log/system.log
						fi
						echo `date` "tina_daemon ($TINA_SERVICE_NAME) daemon is started" >> ${TINA_HOME}/Adm/auto_start.log
						break
					else
						echo `date` "tina_daemon ($TINA_SERVICE_NAME) daemon is not started" >> ${TINA_HOME}/Adm/auto_start.log
					fi
				else
					echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon: ping $hostname is ko" >> ${TINA_HOME}/Adm/auto_start.log
				fi
			else
				echo `date` "Trying to start tina_daemon ($TINA_SERVICE_NAME) daemon: hostname is not defined" >> ${TINA_HOME}/Adm/auto_start.log
			fi
			sleep 5
			i=`expr $i + 1`
		done

		if [ $ISREDHATLIKE -eq 1 ]; then
			echo_success
			echo
		elif [ $ISSUSE -eq 1 ]; then
			rc_status -v
		else
			echo
		fi

		# Start ACSLS daemons (mini_el and ssi)
		if [ -d "$TINA_HOME/Vtl" ] ; then
			for VL_path in $TINA_HOME/Vtl/*
			do
				[ ! -d $VL_path ] && continue
				VL_name=`basename $VL_path`
				if [ $VL_name = "Install" -o $VL_name = "Bin" -o $VL_name = "Log" -o $VL_name = "Tmp" ] ; then
					continue
				fi

				# If there is no tina_stk.conf, give up
				[ ! -f "$VL_path/tina_stk.conf" ] && continue

				[ ! -x "$TINA_HOME/Vtl/Bin/ACSLS/start.sh" ] && continue

				ECHONOCR "Starting ACSLS client daemon for $VL_name virtual library ..."
				$TINA_HOME/Vtl/Bin/ACSLS/start.sh $VL_name
				echo
			done
		fi
	elif [ ! -f ${TINA_HOME}/.ndmp.sh ] ; then
		if [ $ISREDHATLIKE -eq 1 ]; then
			ECHONOCR "Starting Time Navigator (${TINA_SERVICE_NAME})..."
			echo_failure
			echo
		elif [ $ISSUSE -eq 1 ]; then
			rc_failed 1
		else
			echo
		fi
	fi
}

RCStop()
{
	#Stop ndmp daemon
	NDMPDAEMON=""
	if [ -x ${TINA_HOME}/Bin/ndmpd ] ; then
		NDMPDAEMON="ndmpd"
	elif [ -x ${TINA_HOME}/Bin/tina_nts ] ; then
		NDMPDAEMON="tina_nts"
	fi
	if [ ! -z "$NDMPDAEMON" ] ; then
		file="/var/tmp/$NDMPDAEMON.pid"
		if [ -f $file ] ; then
			if [ "$NDMPDAEMON" = ndmpd ] ; then
				echo "Shutting down NDMP Data Server..."
			elif [ "$NDMPDAEMON" = tina_nts ] ; then
				echo "Shutting down NDMP Tape Server..."
			fi
			kill `cat $file`
		fi
	fi

	#Stop Time Navigator daemon
	if [ -x ${TINA_HOME}/Bin/tina_stop ]; then
		if [ -d /var/lock/subsys ] ; then
			rm -f /var/lock/subsys/tina.$TINA_SERVICE_NAME
		fi
		ECHONOCR "Shutting down Time Navigator ($TINA_SERVICE_NAME)..."
		if [ $OS_TYPE = "Darwin" ] ; then
			echo `date` "Stopping tina_daemon ($TINA_SERVICE_NAME) daemon" >> /var/log/system.log
		fi
		echo `date` "Stopping tina_daemon ($TINA_SERVICE_NAME) daemon" >> ${TINA_HOME}/Adm/auto_start.log
		$TINA_HOME/Bin/tina_stop > /dev/null
		retval=0
		if [ $ISREDHATLIKE -eq 1 ]; then
			echo_success
			echo
		elif [ $ISSUSE -eq 1 ]; then
			rc_status -v
		else
			echo
		fi
	elif [ ! -f ${TINA_HOME}/.ndmp.sh ] ; then
		if [ $ISREDHATLIKE -eq 1 ]; then
			echo "Shutting down Time Navigator ($TINA_SERVICE_NAME)..."
			echo_failure
			echo
		elif [ $ISSUSE -eq 1 ]; then
			rc_failed 1
		else
			echo
		fi
	fi
}

RCStatus()
{
	## Check status with checkproc(8), if process is running
	## checkproc will return with exit status 0.

	# Status has a slightly different for the status command:
	# 0 - service running
	# 1 - service dead, but /var/run/ pid file exists
	# 2 - service dead, but /var/lock/ lock file exists
	# 3 - service not running

	if [ -f $TINA_HOME/Conf/hosts ] ; then
		host_to_ping=`cat $TINA_HOME/Conf/hosts | grep ^localhostname | awk '{print $2}' 2>/dev/null`
		if [ $? != 0 -o -z "$host_to_ping" ] ; then
			host_to_ping="127.0.0.1"
		fi
	else
		host_to_ping="127.0.0.1"
	fi

	is_running=`$TINA_HOME/Bin/tina_ping -host $host_to_ping -language English | grep "is running"`
	if [ $# -eq 0 ] ; then
		ECHONOCR "Checking for Time Navigator ($TINA_SERVICE_NAME): "
		if [ $OS_TYPE = "Darwin" ] ; then
			echo `date` "Checking tina_daemon ($TINA_SERVICE_NAME) daemon" >> /var/log/system.log
		fi
		echo `date` "Checking tina_daemon ($TINA_SERVICE_NAME) daemon" >> ${TINA_HOME}/Adm/auto_start.log
		if [ ! -z "$is_running" ] ; then
			echo "tina_daemon is running"
			echo `date` "Checking tina_daemon ($TINA_SERVICE_NAME) daemon: tina_daemon is running" >> ${TINA_HOME}/Adm/auto_start.log
			retval=0
		else
			echo "tina_daemon is stopped"
			echo `date` "Checking tina_daemon ($TINA_SERVICE_NAME) daemon: tina_daemon is stopped" >> ${TINA_HOME}/Adm/auto_start.log
                        retval=3
		fi
	fi
}

test "$ISSUSE" -eq 1 && rc_reset

case "$1" in
start)
	RCStart
	retval=0
	;;

stop)
	RCStop
	retval=0
	;;

start_msg)
	echo "Starting Time Navigator ($TINA_SERVICE_NAME)" ;;

stop_msg)
	echo "Shutting down Time Navigator ($TINA_SERVICE_NAME)" ;;

restart)
	RCStop
	sleep 3
	RCStart ;;

status)
	RCStatus ;;

*)
	echo "usage: /etc/init.d/tina {start|stop|restart|status}" ;;
esac

exit $retval

One final Time Navigator configuration change must be made. The tina agent “hosts” file must be configured to set the “localhostname” of our agent to the FQDN of the floating or virtual IP address service so that the agent will only try to bind to that IP address instead of all IP addresses on the system.

$ cd /cluster/tina/Conf
$ cp hosts.sample hosts
$ nano hosts

Add a line to the file specifying the “localhostname” like so:

localhostname myserver.company.com

For this to work properly, you must also set any other tina agents running on the cluster nodes to also have a “localhostname” set in their respective “hosts” file to prevent other host-based agents from binding to all IP addresses on the host, including the virtual IP address.

That’s it! The tina service can be added to the HA cluster as an LSB resource agent, grouped with your storage resource agents so it will always be running on the same node as your storage.

Conclusion

Ok, so I rushed the end. Big deal. Sue me. I doubt anyone cares anyways!

Related posts:

1. Atempo Time Navigator 4.2 Archive Media Selection Tunable
2. Nanorcs: Ultrasimplistic Configuration File Revision Control
3. Migration Weekend: Success

As part of my learning experience with cfengine, I’ve decided to start posting some of the code that I’ve begun developing in the hopes that by writing about it, I can learn better, faster and maybe even receive some helpful comments from readers along the way. Beware, I’m a cfengine newbie and so what I post here should NOT be copy and pasted into your environment unless you’re ok with the potential of wildly breaking things!

The first snippet of code I want to discuss is related to managing our DenyHosts configuration. As part of our “security policy”, I would like to ensure that every RedHat/CentOS system is running a properly configured DenyHosts instance. Here is what I’ve come up with so far.

################################################################################
#
# FILE: denyhosts.cf
# DESC: Install, update, configure and ensure DenyHosts is running
# DATE: May 2010
#
#################################################################################

bundle agent denyhosts
{

packages:

  "denyhosts" -> "Security policy"
    comment               => "Ensure denyhosts is installed once a week",
    package_policy        => "add",
    package_method        => yum,
    package_architectures => { "noarch" },
    action                => if_elapsed("10080");

  Night::

  "denyhosts" -> "Security policy"
    comment               => "Check for update to denyhosts every 24 hours (and only at night)",
    package_policy        => "update",
    package_method        => yum,
    package_architectures => { "noarch" },
    action                => if_elapsed("1440");

files:

  "/etc/denyhosts.conf" -> "Security policy"
    comment   => "Standard base DenyHosts configuration",
    copy_from => mycopy("$(g.confdir)/denyhosts/denyhosts.conf", "$(g.cfserver)"),
    classes   => cdefine("denyhosts_restart", "denyhosts_conf_copy_failed"),
    perms     => mo("400", "root"),
    action    => if_elapsed("1440");

processes:

  "python /usr/bin/denyhosts.py" -> "Security policy"
    comment       => "Define denyhosts_restart class if denyhost is NOT running",
    restart_class => canonify("denyhosts_restart");

commands:

  "/sbin/service denyhosts restart" -> "Security policy"
     comment    => "Restarting DenyHosts after configuration change or death",
     ifvarclass => canonify("denyhosts_restart");

}

If you’re familiar with cfengine at all, you’ll quickly realize this is not a complete configuration. I am relying on the cfengine standard library for several body definitions as well as custom site variables defined in the common bundle named “g” (not shown). And of course, there are no control bodies, bundlesequence or many other things that make up a complete cfengine configuration, hence “snippet”.

Let’s ignore what’s lacking for now and focus on the meat of the promises.

Packages

The first part of the denyhosts bundle is dealing with packages. I’m making two promises regarding the “denyhosts” package. The first promise is that the package has been added to the system via yum and the second is that the package is up to date via yum. I’m not entirely clear on how to best manage promises like this yet so perhaps I’m missing some cute shorthand for both adding and keeping packages up to date. For now, I’ll stick with two separate promises.

You’ll also notice that I’m only checking to see if the package is installed once a week (via action => if_elapsed) and only checking to see if the package is up to date once every 24 hours (if_elapsed, again). The update promise is also subject to the Night class to ensure that package updates only occur at night and not during the work day. This is just a matter of preference. I’d prefer if updates occur at night, you may not.

Since I’m using a “smart” package manager to ensure that denyhosts is installed, I can count on having yum resolve any dependencies (such as python) for me automatically. I would loath to describe every dependency for every package I want control over by hand.

Files

There is only one file that I’m concerned with when it comes to denyhosts and that is the denyhosts configuration file in /etc/denyhosts.conf. Instead of doing file edits on the default denyhosts.conf file provided in the denyhosts packages that I’ve promised to install, I simply copy a pre-defined configuration from my cfengine server. This file has our site’s default denyhosts configuration all ready to go. If I needed to customize the configuration on a per-host or per-host-type basis, I could copy the base file to a temporary location then perform edits on the temp file and write out the changes to the final location of the default configuration file or simply maintain several pre-configured versions of denyhosts.conf and copy the appropriate file.

Also of note about files is that if the file promise must be repaired (if the file must be copied because it’s changed), I’m setting a class to be defined so that DenyHosts can be restarted. More on that later.

Processes

In this case, I’m only checking to see if the DenyHosts python process is running or not. If DenyHosts is running, we do nothing. If DenyHosts is not running, we define a class using the same name as the class we define if we have to copy the configuration file.

Commands

Finally, in the commands section we tell cfengine how and when to restart DenyHosts. If the “denyhosts_restart” class is defined, we instruct cfengine to restart DenyHosts with the “/sbin/service denyhosts restart” command. The canonify and cdefine special functions in cfengine provide a very powerful way of defining some rather complex relationships.

What is Missing?

Well, probably a lot of stuff. One obvious thing is that I’m not promising that DenyHosts is set to startup at boot time using the hosts’ native init system. Of course, this shouldn’t be a big deal because cfengine will start it up if it’s not running at the next cf-agent run, but perhaps it would be nice to make that promise anyways.

I’m also not using many (or any!) classes to limit the scope of where (to what hosts) these promises will apply. Right now, I’m just working with a test environment so it’s easy to get away with that but I’m learning that it’s good to be as explicit at possible from the start when building promises.

UPDATE: Ah, how could I forget.. Reporting is totally missing! I knew I was setting some of those classes for a reason. In the next installment, I’ll include the most basic of reporting functionality.

I think that’s all for now. Please critique my amateur use of cfengine 3 in the comments, I want to hear from you!

Related posts:

1. Cfengine 3 Snippets Part 2: sudo
2. RHEL/CentOS, NFS and Firewalls
3. Nanorcs: Ultrasimplistic Configuration File Revision Control