The Pragmatic Administrator

Automatically set the hostname during Kickstart Installation

Sun, 08 Jan 2012 22:14:00 -0500

I hate having to manually set the hostname in kickstart file, so when I found a fix, I was very happy. I wish I can take credit, but it was originally made by somebody who was trying to figure out a way to automatically set the hostname for VMWare ESX machines. Unfortunately, I lost that link, so I can’t refer to the other page for credit. So the best I can do is to explain how it is done and hopefully I find that link later and update this post, so that the right person is properly attributed.

To explain how the solution works, its good to understand how Linux boots a system, which this article does a very good job of explaining. However, if you are impatient, this is short version:

Computer turns on (DUH!)
BIOS kick in, which performs POST, local device enumeration and initialization and then searches for active and bootable devices.
Stage 1 (MBR) kicks in, looks for boot loader (in our case, GRUB)
Grub (Stage 2) then loads kernel with an optional ramdisk.
kernel boots, initializes and then starts init (or some other process) that then starts up other processes

Now with that mind, let’s take a look at our grub on jenkins:

[root@jenkins chef]# cat /etc/grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_centos6-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-220.2.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_centos6/lv_root  KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet rd_NO_DM
	initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img

As you can see, it boots the kernel, as well as set parameters such as root file system, language, keyboard and others things needs for the system to boot up properly. That information is actually still available in the running kernel by viewing the following file:

[root@jenkins chef]# cat /proc/cmdline 
ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_centos6/lv_root  KEYBOARDTYPE=pc KEYTABLE=us  rhgb quiet rd_NO_DM
[root@jenkins chef]

Notice that in this file, you will find the same parameters as you find in the grub.conf. In some ways, if init (at least on System-V systems) is the mother of all process, the kernel is the grandmother, quietly hidden in the background.

What if you were to pass a parameter that it doesn’t recognize? In most cases, it will probably ignore it, but it will still in the kernel list. So lets insert:

FOO=BAR

to the kernel line right between “crashkernel=auto” and “rhgb” (either in grub or at kernel line at boot loader page during stage 2):

kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_centos6/lv_root  KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto FOO=BAR rhgb quiet rd_NO_DM

Now lets view /proc/cmdline again:

[root@jenkins ~]# cat /proc/cmdline 
ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_centos6/lv_root  KEYBOARDTYPE=pc KEYTABLE=us  FOO=BAR rhgb quiet rd_NO_DM
[root@jenkins ~]#

As we can see, FOO=BAR is in there, with no ill effects to the system boot.

So why would we want to pass a value that the kernel doesn’t use? So that we can do this:

[rilindo@jenkins ~]$ for x in `cat /proc/cmdline`
> do
> case $x in FOO*)
> eval $x
> echo "${FOO}" 
> ;;
> esac
> done
BAR
[rilindo@jenkins ~]$

What this script does is to get the output of /proc/cmdline as a series of positional elements (think of it like a list or an array) and loop through it. Then we will test each element through a case statement and if it matches (in this case, FOO), then it evaluates it to a variable. We then echo that variable, which will then return a value. In other words, we look for a section that has “FOO”, and get “BAR” out of it.

That is essentially how we automatically set the hostname in our installation. Using this technique, we put this script in our %pre section of our kickstart:

%pre
#!/bin/sh
for x in `cat /proc/cmdline`; do
        case $x in SERVERNAME*)
	        eval $x
		echo "network --device eth0 --bootproto dhcp --hostname ${SERVERNAME}.monzell.com" > /tmp/network.ks
                ;;
	        esac;
	done
%end

Here, we look for a value called SERVERNAME and evaluates that value into a variable. We will then echo the network setup with the variable (which we will use as part of the hostname setup) and redirect into the file under /tmp. Then we will include that file in our installation section:

At this point, we are essentially done. To use it, we just need to pass SERVERNAME=X (where X is the name of the hostname you want to set) in our kickstart setup. In our case, we build virtual machines with KVM via virt-install, so we pass that in the following line:

virt-install --name jenkins --disk path=/home/vms/jenkins,size=50,bus=virtio --vnc --noautoconsole --vcpus=1 --ram=512 --network bridge=br0,mac=52:54:00:91:95:30 --location=http://192.168.15.100/mirrors/centos/6.2/os/x86_64/ -x "ks=http://192.168.15.100/mirrors/ks/6.2/kvm/x86_64-Ruby-test.cfg SERVERNAME=jenkins"

Here is my entire kickstart file:

install
url --url http://192.168.15.100/mirrors/centos/6.2/os/x86_64/
lang en_US.UTF-8
keyboard us
text
%include /tmp/network.ks

rootpw  --iscrypted PUTPASSWORDHERE
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
clearpart --all --drives=vda --initlabel

part /boot --fstype=ext4 --size=500
part pv.EPlgaf-h1b4-YqDI-2wfs-3C7I-SPPt-Agk5O7 --grow --size=1

volgroup vg_centos6 --pesize=4096 pv.EPlgaf-h1b4-YqDI-2wfs-3C7I-SPPt-Agk5O7
logvol / --fstype=ext4 --name=lv_root --vgname=vg_centos6 --grow --size=1024 --maxsize=51200
logvol swap --name=lv_swap --vgname=vg_centos6 --grow --size=1008 --maxsize=2016

repo --name="Local CentOS 6 - x86_64"  --baseurl=http://192.168.15.100/mirrors/centos/6.2/os/x86_64
repo --name="Local CentOS 6 - x86_64 - Updates"  --baseurl=http://192.168.15.100/mirrors/centos/6.2/updates/x86_64
repo --name="Local Custom Installs" --baseurl=http://192.168.15.100/mirrors/customrepos/centos/x86_64

%packages
@base
@console-internet
@core
@debugging
@directory-client
@hardware-monitoring
@large-systems
@network-file-system-client
@performance
@perl-runtime
@scalable-file-systems
@server-platform
gcc
gcc-c++
pax
oddjob
sgpio
certmonger
pam_krb5
krb5-workstation
nscd
pam_ldap
nss-pam-ldapd
perl-DBD-SQLite
ruby-1.9.3p0
rubygems-1.8.12
%end

%pre
#!/bin/sh
for x in `cat /proc/cmdline`; do
        case $x in SERVERNAME*)
	        eval $x
		echo "network --device eth0 --bootproto dhcp --hostname ${SERVERNAME}.example.com" > /tmp/network.ks
                ;;
	        esac;
	done
%end

%post --log=/root/my-post-log

setsebool -P use_nfs_home_dirs on
mkdir /home/users
mkdir /etc/chef

URLPOSTCONF="http://192.168.15.100/mirrors/ks"
curl ${URLPOSTCONF}/6.2/repos/CentOS-Custom.repo -o /etc/yum.repos.d/CentOS-Custom.repo
curl ${URLPOSTCONF}/6.2/autofs/auto.master -o /etc/auto.master
curl ${URLPOSTCONF}/6.2/autofs/auto.home -o /etc/auto.home
curl ${URLPOSTCONF}/keys/cacert.pem -o /etc/openldap/cacerts/cacert.pem


curl ${URLPOSTCONF}/chef/validation.pem -o /etc/chef/validation.pem
curl ${URLPOSTCONF}/chef/client.rb -o /etc/chef/client.rb
curl ${URLPOSTCONF}/chef/first-run.json -o /etc/chef/first-run.json
rpm --import ${URLPOSTCONF}/keys/legacy.key
rpm --import ${URLPOSTCONF}/keys/custom.key

authconfig --enablesssd --enableldap --enableldaptls --ldapserver=kerberos.monzell.com --ldapbasedn="dc=monzell,dc=com" --enableldapauth --update

echo "nameserver 192.168.15.57" >> /etc/resolv.conf
echo "nameserver 192.168.15.71" >> /etc/resolv.conf

gem install chef
chef-client -j /etc/chef/first-run.json
chkconfig chef-client on
chkconfig rpcbind on
chkconfig sssd on
chkconfig ntpd on
sync


%end

reboot

Let me know if this is useful. And again, I didn’t originally came up with this, so I plead innocent to charges of plagiarism. :)

Updated to chef-client on FreeBSD (aka part III)

Sun, 08 Jan 2012 20:49:41 -0500

Found the solution.

Essentially, I just need to add this:

supports :status => true, :restart => true, :reload => true

This means that it will start up the service if isn’t running. Now it works as expected when I add chef-client to the run list.

Here is the updated code:

when "bsd"
  case node['platform']
    when "freebsd"

      directory "/etc/rc.conf.d" do
        owner "root"
        group "wheel"
        mode "0644"
        action :create
      end
      template "/etc/rc.d/chef-client" do
        source "#{dist_dir}/rc.d/chef-client.erb"
        owner "root"
        group "wheel"
        mode 0755
      end

      template "/etc/rc.conf.d/chef" do
        source "#{dist_dir}/rc.conf.d/chef.erb"
        mode 0644
        notifies :start, "service[chef-client]", :delayed
      end

      service "chef-client" do
        supports :status => true, :restart => true, :reload => true
        action [:start]
      end

    else
      log "You specified service style 'bsd'. You will need to set up your rc.local file."
      log "Hint: chef-client -i #{node["chef_client"]["client_interval"]} -s #{node["chef_client"]["client_splay"]}"
  end
else
  log "Could not determine service init style, manual intervention required to start up the chef-client service."
end

FreeBSD and chef-client - a part II (of sorts)

Mon, 02 Jan 2012 11:24:00 -0500

Finally changed chef-client with an updated recipe to support FreeBSD.

Under the chef-repo/chef-client directory, I added the following files:

./templates/freebsd/rc.d/chef-client.erb
./templates/freebsd/rc.conf.d/chef.erb

And updated:

./recipes/service.rb

The locations corresponds to the directory location under the default #{conf} directory, (which is apparently /etc) The templates are .erb files that corresponds to the configuration files on the server.

chef-client.erb:

[rilindo@chef chef-client]$ cat ./templates/freebsd/rc.d/chef-client.erb 
#!/bin/sh

# PROVIDE: chef
# REQUIRE: LOGIN
# KEYWORD: nojail shutdown

. /etc/rc.subr

name="chef"
rcvar=`set_rcvar`
stop_cmd="chef_stop"
command="/usr/local/bin/${name}-client"
command_args="-i -s -d -L /var/log/chef/client.log -c /etc/chef/client.rb -P /var/run/chef.pid"
load_rc_config $name
export rc_pid
chef_stop()
{
	pidfile="/var/run/chef.pid"
	rc_pid=`cat ${pidfile}`
        kill $rc_pid
}

run_rc_command "$1"

chef.erb

[rilindo@chef chef-client]$ cat ./templates/freebsd/rc.conf.d/chef
chef_enable="YES"

With ERB, I could have easily have placeholders in the code so that it can be populated with node-specific information automatically. I did not do that in this case, though. That is for another time.

Finally, I updated the service code from:

when "bsd"
  log "You specified service style 'bsd'. You will need to set up your rc.local file."
  log "Hint: chef-client -i #{node["chef_client"]["client_interval"]} -s #{node["chef_client"]["client_splay"]}"
  
else
  log "Could not determine service init style, manual intervention required to start up the chef-client service."
end

when "bsd"
  case node['platform']
    when "freebsd"

      directory "/etc/rc.conf.d" do
        owner "root"
        group "wheel"
        mode "0644"
        action :create
      end
      template "/etc/rc.d/chef-client" do
        source "#{dist_dir}/rc.d/chef-client.erb"
        owner "root"
        group "wheel"
        mode 0755
      end

      template "/etc/rc.conf.d/chef" do
        source "#{dist_dir}/rc.conf.d/chef.erb"
        mode 0644
        notifies :start, "service[chef-client]", :delayed
      end

      service "chef-client" do
        action [:start]
      end

    else
      log "You specified service style 'bsd'. You will need to set up your rc.local file."
      log "Hint: chef-client -i #{node["chef_client"]["client_interval"]} -s #{node["chef_client"]["client_splay"]}"
  end
else
  log "Could not determine service init style, manual intervention required to start up the chef-client service."
end

I am not sure if this is quite “rubyish”, but it works.

At that point, I uploaded the cookbook:

knife cookbook upload chef-client

Added the recipe to the freebsd node:

knife node run_list add freebsddev.monzell.com  "recipe[chef-client]"

And ran chef-client. The chef-client program sees the receipe and install the files to the appropriate locations:

freebsddev# /usr/local/bin/chef-client
[Sun, 01 Jan 2012 23:48:28 -0500] INFO: *** Chef 0.10.8 ***
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Run List is [recipe[chef-client]]
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Run List expands to [chef-client]
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Starting Chef Run for freebsddev.monzell.com
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Running start handlers
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Start handlers complete.
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Loading cookbooks [chef-client]
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Storing updated cookbooks/chef-client/recipes/default.rb in the cache.
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Storing updated cookbooks/chef-client/recipes/delete_validation.rb in the cache.
[Sun, 01 Jan 2012 23:48:34 -0500] INFO: Storing updated cookbooks/chef-client/recipes/service.rb in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Storing updated cookbooks/chef-client/recipes/config.rb in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Storing updated cookbooks/chef-client/attributes/default.rb in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Storing updated cookbooks/chef-client/metadata.json in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Storing updated cookbooks/chef-client/README.md in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Storing updated cookbooks/chef-client/metadata.rb in the cache.
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing directory[/var/run] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing directory[/var/chef/cache] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing directory[/var/chef/backup] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing directory[/var/log/chef] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing directory[/etc/rc.conf.d] action create (chef-client::service line 203)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing template[/etc/rc.d/chef-client] action create (chef-client::service line 209)
[Sun, 01 Jan 2012 23:48:35 -0500] INFO: Processing template[/etc/rc.conf.d/chef] action create (chef-client::service line 216)
[Sun, 01 Jan 2012 23:48:36 -0500] INFO: Processing service[chef-client] action start (chef-client::service line 222)
[Sun, 01 Jan 2012 23:48:36 -0500] INFO: Chef Run complete in 2.340431224 seconds
[Sun, 01 Jan 2012 23:48:36 -0500] INFO: Running report handlers
[Sun, 01 Jan 2012 23:48:36 -0500] INFO: Report handlers complete

I am mostly done now. I just need to start it up with:

/etc/rc.d/chef-client start

I should be able to start automatically. However, getting it to start up automatically upon installation has so far just returns me with:

freebsddev# /usr/local/bin/chef-client
[Sun, 01 Jan 2012 23:46:26 -0500] INFO: *** Chef 0.10.8 ***
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Run List is [recipe[chef-client]]
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Run List expands to [chef-client]
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Starting Chef Run for freebsddev.monzell.com
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Running start handlers
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Start handlers complete.
[Sun, 01 Jan 2012 23:46:32 -0500] INFO: Loading cookbooks [chef-client]
[Sun, 01 Jan 2012 23:46:33 -0500] INFO: Storing updated cookbooks/chef-client/recipes/default.rb in the cache.
[Sun, 01 Jan 2012 23:46:33 -0500] INFO: Storing updated cookbooks/chef-client/recipes/delete_validation.rb in the cache.
[Sun, 01 Jan 2012 23:46:33 -0500] INFO: Storing updated cookbooks/chef-client/recipes/service.rb in the cache.
[Sun, 01 Jan 2012 23:46:33 -0500] INFO: Storing updated cookbooks/chef-client/recipes/config.rb in the cache.
[Sun, 01 Jan 2012 23:46:33 -0500] INFO: Storing updated cookbooks/chef-client/attributes/default.rb in the cache.
[Sun, 01 Jan 2012 23:46:34 -0500] INFO: Storing updated cookbooks/chef-client/metadata.json in the cache.
[Sun, 01 Jan 2012 23:46:34 -0500] INFO: Storing updated cookbooks/chef-client/README.md in the cache.
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Storing updated cookbooks/chef-client/metadata.rb in the cache.
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing directory[/var/run] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing directory[/var/chef/cache] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing directory[/var/chef/backup] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing directory[/var/log/chef] action create (chef-client::service line 42)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing directory[/etc/rc.conf.d] action create (chef-client::service line 203)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing template[/etc/rc.d/chef-client] action create (chef-client::service line 209)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing template[/etc/rc.conf.d/chef] action create (chef-client::service line 216)
[Sun, 01 Jan 2012 23:46:35 -0500] INFO: Processing service[chef-client] action restart (chef-client::service line 222)
[Sun, 01 Jan 2012 23:46:36 -0500] ERROR: service[chef-client] (chef-client::service line 222) has had an error
[Sun, 01 Jan 2012 23:46:36 -0500] ERROR: service[chef-client] (/var/chef/cache/cookbooks/chef-client/recipes/service.rb:222:in `from_file') had an error:
service[chef-client] (chef-client::service line 222) had an error: Chef::Exceptions::Exec: /etc/rc.d/chef-client stop returned 1, expected 0
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/mixin/command.rb:127:in `handle_command_failures'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/mixin/command.rb:74:in `run_command'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/provider/service/init.rb:45:in `stop_service'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/provider/service/init.rb:55:in `restart_service'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/provider/service.rb:78:in `action_restart'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource.rb:440:in `run_action'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/runner.rb:45:in `run_action'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/runner.rb:81:in `each'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/runner.rb:81:in `block in converge'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection.rb:94:in `block in execute_each_resource'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/resource_collection.rb:92:in `execute_each_resource'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/runner.rb:76:in `converge'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/client.rb:312:in `converge'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/client.rb:160:in `run'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/application/client.rb:239:in `block in run_application'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/application/client.rb:229:in `loop'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/application/client.rb:229:in `run_application'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/lib/chef/application.rb:67:in `run'
/usr/local/lib/ruby/gems/1.9/gems/chef-0.10.8/bin/chef-client:26:in `'
/usr/local/bin/chef-client:19:in `load'
/usr/local/bin/chef-client:19:in `'
[Sun, 01 Jan 2012 23:46:36 -0500] ERROR: Running exception handlers
[Sun, 01 Jan 2012 23:46:36 -0500] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[Sun, 01 Jan 2012 23:46:36 -0500] ERROR: Exception handlers complete
[Sun, 01 Jan 2012 23:46:36 -0500] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[Sun, 01 Jan 2012 23:46:36 -0500] FATAL: Chef::Exceptions::Exec: service[chef-client] (chef-client::service line 222) had an error: Chef::Exceptions::Exec: /etc/rc.d/chef-client stop returned 1, expected 0

Essentially, it couldn’t find the PID file in the expected location, which is no surprise, as I had been running chef-client manually without any arguments. Hopefully I can figure out a fix for that soon.

FreeBSD and chef-client - a part I?

Sun, 01 Jan 2012 22:26:00 -0500

As I mentioned before, Chef appears to work well on mostly Debian and Ubuntu. You will have to do a bit more work on the other OSes: In the case of FreeBSD, a lot more.

Here is one example: The recipe chef-client is used to install startup scripts on the nodes (rc scripts for Red Hat, upstart for Ubuntu, etc). it works on most OSes - except for BSD systems. In fact, in the code, when it noticed it is on a BSD systems, it puts out the following:

when "bsd"
  log "You specified service style 'bsd'. You will need to set up your rc.local file."
  log "Hint: chef-client -i #{node["chef_client"]["client_interval"]} -s #{node["chef_client"]["client_splay"]}"
  
else
  log "Could not determine service init style, manual intervention required to start up the chef-client service."
end

in other words, it doesn’t even bother.

I am not sure if it is out of laziness or just having limited resources that they didn’t create rc scripts for BSD (I could understand OpenBSD, but FreeBSD?), so I created the following rc script:

#!/bin/sh

# PROVIDE: chef
# REQUIRE: LOGIN
# KEYWORD: nojail shutdown

. /etc/rc.subr

name="chef"
rcvar=`set_rcvar`
stop_cmd="chef_stop"
command="/usr/local/bin/${name}-client"
command_args="-i -s -d -L /var/log/chef/client.log -c /etc/chef/client.rb -P /var/run/chef.pid"
load_rc_config $name
export rc_pid
chef_stop()
{
	pidfile="/var/run/chef.pid"
	rc_pid=`cat ${pidfile}`
        kill $rc_pid
}

run_rc_command "$1"

Ordinarily, I shouldn’t have to create a separate function to kill a chef process, but for some reason, the rc functions within FreeBSD can’t find the PID.

Interestingly enough, during my debugging with the script, through the use of truss I found an undocumented feature where instead of adding the entry to enable a service in /etc/rc.conf, you can put it in /etc/rc.conf.d - which is what I did:

freebsd82# pwd
/etc/rc.d
freebsd82# cd ../rc.conf.d
freebsd82# ls
chef
freebsd82# cat chef
chef_enable="YES"

Apparently it came from NetBSD.

With that, I got a working chef init script. Now to see if I can update the chef-client recipe and working on FreeBSD.

Always Crispy: SSH Tunnels - 2 ways

Sun, 01 Jan 2012 09:20:05 -0500

Always Crispy: SSH Tunnels - 2 ways :

verycrispy:

Recently, I have been googling how to make tunnels so I thought I would post what I do. A SSH tunnel allows you to connect to server, A, through server B, from client C.

You generally only want to setup a tunnel when you need to connect to server A but only have access to server B from your…

Installing Chef on CentOS

Sat, 31 Dec 2011 22:28:00 -0500

I have been playing around with Chef for the past week and while I liked it, it was a pain it setup. It seems to be work well if you run Debian and Ubuntu. Everything else … not so much.

First sign of trouble is when I attempt to bootstrap the install. The install calls for installing Ruby from the RBEL repo. Which I don’t have too much with trouble - in fact, they have binary RPMs of chef already available, so I used that initially and installed with:


yum install rubygem-chef-server --disablerepo=updates --disablerepo=CentOS-Custom --disablerepo=extras

(Centos-Custom is my own repo, by the way).

That went well - until it turns out that it installed Ruby 1.8 along with it.

So I got that removed. I spent the next few hours of trying (and failing) to install Ruby 1.9 while avoiding have to install 1.8. In the end, I gave up. Instead, what I did is the following:

Installed the prerequisites for ruby (including my build of Ruby 1.9 and Rubygems).
Then, I ran “gem install ruby-shadow”, as there was no RPM for it in the CentOS repo.
Then I installed the EPEL repo (instead of the RBEL repo). That allow to proceed with the install of chef with “gem install chef”. That, in turn, took care of all the requirements and package installation.

The next step is to configure a web proxy, as detail here. I decided to deviate slight and just use Red Hat’s utility with:

genkey chef.monzell.com

And then open the firewall ports.

However, because I had SELinux running, apache is not able to communicate to another application (as they are in different security context. So I had to enable access with:

setsebool -P httpd_can_network_connect on

That got me further, but I still had issues. After tailing the audit log and cat the output to audit2allow, I found that I still need to open a port in SELinux:

#============= httpd_t ==============
allow httpd_t reserved_port_t:tcp_socket name_bind;

I enabled access with:

[root@chef audit]# tail audit.log  | audit2allow -M chef444
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i chef444.pp

Installed the module and got the web access working.

There is more, but that’s for another post. :)

(as a side note, is there a tumbler theme that is code friendly - that is, I can paste in code and command line snippets without looking like snot?)

EDIT: Nevermind, looks like I’ll be poking around with CSS again to get it working the way I like.

FreeBSD and OpenLDAP Grief

Sat, 19 Nov 2011 16:56:00 -0500

Finally got my FreeBSD client to authenticate against my OpenLDAP server.

The configuration is fairly straightforward. What took the time was compilation the dependencies (running it in a VM can do that to it). That and the following issues.

- It seems that Perl is not a requirement for a FreeBSD install. Not a big deal, (thinking about it, it make sense historically), but I needed to get the certs installed - which mean a install of Perl. Fun.

- ca-root no longer exists. Had to use ca-root-nss to build.

- After working with Red Hat for a while, manually setting up pam was pain.

- I couldn’t get pass pam_ldap almost all night and part of the afternoon, until I tailed /var/log/auth.log, which showed me this:

User rfoster not allowed because shell /bin/bash does not exist

Bash is not installed by default. Another compile. But afterwards, I was finally able to login.

From there, it was a matter of using amd to work so that I can automount the directories. Using this as a guideline, I setup the symlinks in /usr/home to the mounts:

ln -sf /host/kerberos.monzell.com/exports/users .

Then I add my ldap user to wheel group (so that I can become root):

freebsd82# pw groupmod wheel -m rfoster

freebsd82# pw groupshow wheel

wheel:*:0:rilindo,rfoster

freebsd82#

And… I am done.

Next, configure SuSE Enterprise Linux 11 with LDAP authentication. :)

TLS problems with OpenLDAP Client

Fri, 18 Nov 2011 16:22:00 -0500

I ran into an interesting problem sometime back that I only now resolved.

Originally, I was running Scientific Linux on most of my VM. I have since upgraded most of them to Centos 6.0 - and converted on in particular to Centos CR.

That “broke” my ldap authentication - when connecting to the server with the ldapuser credentials, sssd returns with the following:

Could not start TLS encryption. TLS error -8172:Unknown code ___f 20

At the time, I thought that there was a bug with the updated sssd package on Centos CR, so I ignored it for while - until I logged into a fairly new Scientific Linux 6.1 VM today - and it gave me the same message.

That was curious. So I dug deep into searching for a solution.

As it turns out, the problem was my configuration. Apparently I had been connecting with a self-signed certificate, which the following:

[root@localhost ~]# openssl verify cacert.pem 
backup.cacert.pem: C = US, ST = Georgia, O = Monzell Management Systems, OU = IT, CN = example.com, emailAddress = rilindo.foster@example.com
error 18 at 0 depth lookup:self signed certificate
OK

It seems with the initial version of sssd with 6.0 , it was allowing me to connect without complaining about it being a self-sign cert. With the updated version, it is now refusing to connect without a valid certificate. I can confirm that by running:

>
ldapsearch -d -1 -vvvvv -w PASSWORD -ZZZ  -H ldap://ldap.example.com  -D "cn=root,dc=example,dc=com"   "(uid=joeuser)"
<snip>
                                          ..                
TLS: certificate [CN=StartCom Certification Authority,OU=Secure Digital Certificate Signing,O=StartCom Ltd.,C=IL] is not valid - error -8172:Unknown code ___f 20.
tls_write: want=7, written=7
  0000:  15 03 01 00 02 02 30                               ......0           
TLS: error: connect - force handshake failure: errno 0 - moznss error -8172
TLS: can't connect: TLS error -8172:Unknown code ___f 20.

After I put in the correct certificate, I was able to connect:

<snip>

tls_read: want=48, got=48 0000: f0 85 60 72 54 c1 3b c8 6f 53 c4 f0 89 82 27 17 ..`rT.;.oS....'. 0010: 3c 3f 99 8f 18 64 22 ae 41 28 d4 a6 0b 0f a4 de <?...d".A(...... 0020: 36 10 3e d4 6c f5 73 fb cb 12 04 af 64 7f 14 69 6.>.l.s.....d..i TLS certificate verification: subject: E=webmaster@example.com,CN=kerberos.example.com, <REDACTED> ldap_sasl_bind ldap_send_initial_request ldap_send_server_request

<snip>

And openssl returned with no errors:

[root@localhost cacerts]# openssl verify cacert.pem  cacert.pem: OK

Linux: ### openSUSE 12.1 released ###Version 12.1 of the popular openSUSE...

Thu, 17 Nov 2011 16:40:19 -0500

Linux: ### openSUSE 12.1 released ###Version 12.1 of the popular openSUSE...:

reallinux:

### openSUSE 12.1 released ###
Version 12.1 of the popular openSUSE distribution got released. It’s the first version numbered after the new numbering scheme and it brings some major changes.
In version 12.1 of the distribution, Gnome now comes in version 3.2 (in openSUSE 11.4, Gnome 3 was only…

A Note on Installing FreeBSD on KVM + DHCP

Sun, 30 Oct 2011 00:42:00 -0400

If you install FreeBSD on KVM, you encounter a problem where you are able to install it over the network, but not be able to get it to obtain an IP address after the initial reboot.

The solution is to add the following in /etc/rc.conf:

synchronous_dhclient=”YES”

And restart the network with:

/etc/rc.d/netif restart

Restart your machine to ensure that it is able to obtain an IP address after boot.

Explanation: By default, the dhcpclient will start in the background concurrently with init startup. With this option, the remaining startup options will wait until the client is able to obtain an IP address successfully:

Source

helencho: anthonybourdain: No explanation other than...

Sat, 29 Oct 2011 14:24:27 -0400

helencho:

anthonybourdain:

No explanation other than awesomeness!

yep.

Linux: CentOS CR for CentOS 6.0

Thu, 29 Sep 2011 22:09:43 -0400

Linux: CentOS CR for CentOS 6.0:

reallinux:

CentOS has been criticized heavily earlier this year, mainly because of the big delay their distribution normally has (compared to the original RHEL releases).
One of the problems that comes with this delay is that security fixes that RedHat writes for their latest release are unavailable for…

/dev/random/thoughts: Creating new LV and moving files from old location

Wed, 28 Sep 2011 22:44:18 -0400

/dev/random/thoughts: Creating new LV and moving files from old location:

hadret:

Recently I had some disk issues on my server and unfortunately it went to load average above 20 (which was killing for this machine). It forced me to power down the machine completely and to investigate it further. (It turned out that issue was with space on /home partition — sharing file system…

Sys::Log: kill: Creating a core dump

Sun, 25 Sep 2011 22:44:43 -0400

Sys::Log: kill: Creating a core dump:

sys-log:

Sometimes when dealing with application problems you run into a point where logs and environmental data just don’t seem to provide you with the exact issue.

For this reason there are many times where a developer may ask you to create a core file for their application. Core files contain a lot of…

The simple way of setting up NAT with iptables and Red Hat / CentOS and Scientific Linux

Sat, 24 Sep 2011 18:36:00 -0400

I always feel intimidated with setting NAT. I knew, of course, how to filter packets with iptables, but I always felt that NAT to be just a tad bit confusing. Then I studied for the RHCE exam and it turns out that it is a lot easier than I expected, with a little help from Red Hat’s firewall tool.

With earlier releases of Red Hat, the firewall tool is limited to just filtering by ports with no NAT or ACLs support. With version 6, the tool was significantly revamped. While ACLs are still not supported, you can now setup NAT with the firewall GUI. To get started, log yourself into the X-Windows GUI and run the following from the terminal.

system-config-firewall

Alternatively, you may run it from the menu -> System- Administration -> firewall. Or if you are logged remotely and you have X-Window running on your workstation, you can forward the tool over SSH to run it locally on your workstation by logging to the server with:

ssh -X username@hostname

and then run the “system-config-firewall.”

(One notable caution: if you have an existing iptables configuration, running this tool will wipe out your rules. Be sure to back /etc/system/iptables before you proceed further on this.)

From there, you will see the following options available:

Select “Masquerading” and then to your right, select the interface or interfaces you want your traffic to go through.

Congratulations, NAT is now setup! The next step is to setup forwarding to your devices or servers. Select “Port Forwarding” to your left:

Then click on add button at your right:

{stub}

Then near the top of the following screen under the Source heading, click on the interface button, select your interface and click on OK

Then click the protocol button, select your protocol and then click on OK

Then click on the port button and select (or enter) your port(s) and then click on OK:

Now under the Destination heading, select “Forward to another port” and enter the IP address of the machine you will forward the packets to in the field:

Then select the Port button or enter the port(s) you will be forwarding to:

Click on OK, then OK to return to the main screen

From there, click on apply (which will write to the /etc/sysconfig/iptables file) and then reload (which will restart iptables).

(If you have backed up your iptables file, you may be able restore them with the custom rules option. However, the iptable backup must be in the iptables-save format, which presumably means that a custom-made one will not work).

You are mostly done here, but there is a couple of more changes youneed to make. First of all, though, lets login in via the command and take a look at the iptables file:

[root@sl6vmware sysconfig]# cat iptables

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.

*nat

:PREROUTING ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

:POSTROUTING ACCEPT [0:0]

-A POSTROUTING -o eth+ -j MASQUERADE

-A PREROUTING -i eth+ -p tcp --dport 20:21 -j DNAT --to-destination 192.168.15.36:20-21

COMMIT

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -i eth+ -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

-A FORWARD -p icmp -j ACCEPT

-A FORWARD -i lo -j ACCEPT

-A FORWARD -i eth+ -j ACCEPT

-A FORWARD -o eth+ -j ACCEPT

-A FORWARD -i eth+ -m state --state NEW -m tcp -p tcp -d 192.168.15.36 --dport 20:21 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

As you can see there, setting up NAT comprised of a few steps:

1) you set the table type, which at the beginning, would be *nat. If you were to do it from the command line, it would be the first part of the following statement

iptables -t nat <statement>

2) Then, you would set POSTROUTING s. From the command line, it will the second part of the statement we mention earlier.

iptables -t nat -A POSTROUTING -o eth+ -j MASQUERADE

At that point, the nat is setup.

3) From this point, it is just the matter of forwarding the packets based on source port (you can add the ACLs at this point, but the GUI tool will most likely overwrite them. Again, make sure that you have a backup!) . That is done by setting up a PREROUTING rule for a port, with:

-A PREROUTING -i eth+ -p tcp —dport 20:21 -j DNAT —to-destination 192.168.15.36:20-21

From the command line, it would be:

iptables -t nat -A PREROUTING -i eth+ -p tcp —dport 20:21 -j DNAT —to- destination 192.168.15.36:20-21

And then, you would add a FORWARD rule with:

-A FORWARD -i eth+ -m state —state NEW -m tcp -p tcp -d 192.168.15.36 — dport 20:21 -j ACCEPT

Which, from the command line would be:

iptables -t filter -A FORWARD -i eth+ -m state —state NEW -m tcp -p tcp -d 192.168.15.36 —dport 20:21 -j ACCEPT

Note that statement must be under the filter table, not under the nat table.

Getting this right via the command-line is fairly tricky even for experienced administrations, which is why the firewall tool is a great way to manage your rules for NAT (assuming that they are simple enough)

But you are not done yet. First, you need to make sure that you actually enable IP forwarding with the kernel. You can set it dynamically with either:

sysctl -w net.ipv4.conf.all.forwarding =1

Or:

echo “1” > /proc/sys/net/ipv4/ip_forward

To change it permanently (which is what you want), add or change the net.ipv4.conf.all.forwarding value in /etc/sysctl.conf to:

net.ipv4.ip_forward = 1

And then reboot (assuming that you didn’t make the change dynamically).

At this point, there is one more thing you may need to do, , particularly if you are running a FTP service.

With FTP, when a client connects to a server, the FTP server returns the response via different port (usually a high-number port above 1024). That becomes a problem when the client attempts to respond to that same port when the server is behind either a firewall or a NAT , as that port will most likely be blocked.

So there are 4 options you choose to address this problem.

1) Use SFTP. It is is encrypted and travels through the same port (22), so that is the best option. However, not all clients would have thats support.

2) Have the FTP server return a response to a specific port and open that port.

3) Have the client use passive mode

4) Modify the nat/firewall dynamically as well as modify the PORT and PASV commands dynamically.

We will go with the last option. Fortunately, it is not complicated in this case. We just need the open the following file:

/etc/sysconfig/iptables-config

And add (or change) the following:

IPTABLES_MODULES=”nf_nat_ftp”

This is a helper module that allows us to dynamically set the ports and track the FTP connectivity. It will load once you restart type iptables, either from the gui or from the command line with:

service iptables restart

If you were to prefer the load the module manual, you would run:

modprobe nf_nat_ftp

Which will load that module as the other supporting modules. You confirm whether it is loaded by running:

lsmod | grep ftp

Which will present you with the following output

nf_nat_ftp 3473 0

nf_conntrack_ftp 12879 1 nf_nat_ftp

nf_nat 22788 3 nf_nat_ftp,ipt_MASQUERADE,iptable_nat

nf_conntrack 79611 8 nf_nat_ftp,nf_conntrack_ftp,ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,nf_conntrack_ipv6,xt_state

Now you are done with NAT. You can start forwarding traffic to your internal servers and have them accessible from outside your network.

For more information on setting up NAT, go to the following links:

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch14_:_Linux_Firewalls_Using_iptables - A longer guide to setting up IP tables, with a section for NAT.

http://debianclusters.org/index.php/NAT_with_IPTables - Setting up IP tables with Debian-type distributions.

http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html - The problems you will encounter with with NAT and FTP.

bash: Field Separator Variable

Tue, 20 Sep 2011 23:46:26 -0400

sys-log:

By default when using a for loop in bash the field separator is set to a space.

Example:

[bcane@bcane ~]$ for x in list:like:this; do echo $x; done
list:like:this

One of the cool things about bash is that you can change this by setting a simple variable $IFS

[bcane@bcane ~]$ IFS=”:”
[bcane@bcane ~]$ for x in list:like:this; do echo $x; done
list like this

"The exit code from a batch job is a standard Unix termination status, the same sort of number you..."

Sat, 17 Sep 2011 22:24:51 -0400

“

The exit code from a batch job is a standard Unix termination status, the same sort of number you get in a shell script from checking the “$?” variable after executing a command.

Typically, exit code 0 (zero) means successful completion. Codes 1-127 are typically generated by your job itself calling exit() with a non-zero value to terminate itself and indicate an error. In BaBar we don’t make very much use of this. The most common such value you might see is 64, which is the value used by Framework to say that its event loop is being stopped before all the requested data have been read, typically because time ran out. In recent BaBar releases you might also see 125, which we use as a code for a generic “severe error”; the job log should contain a message stating what the error was.

Exit codes in the range 129-255 represent jobs terminated by Unix “signals”. Each type of signal has a number, and what’s reported as the job exit code is the signal number plus 128. Signals can arise from within the process itself (as for SEGV, see below) or be sent to the process by some external agent (such as the batch control system, or your using the “bkill” command).

By way of example, then, exit code 64 means that the job deliberately terminated its execution by calling “exit(64)”, exit code 137 means that the job received a signal 9, and exit code 140 represents signal 12.

”

- (via arisawa)

Sys::Log: mysql: Backup your user privileges

Fri, 16 Sep 2011 19:12:28 -0400

Sys::Log: mysql: Backup your user privileges:

sys-log:

While I am sure there are multiple ways to do this and some probably easier here is a way to backup your mysql user privileges to a CSV file.

First you will need to get to the mysql cli.

# mysql -uroot -p
Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL…

UNIX BASH scripting: Rename file to uppercase except extension - Bash

Thu, 15 Sep 2011 23:09:55 -0400

UNIX BASH scripting: Rename file to uppercase except extension - Bash

Welcome to Tumblr, Rilindo!

Thu, 15 Sep 2011 23:08:41 -0400

Thanks!