I installed Chef Client on an AIX server and added it to my Chef Server. The „normal“ mode of operations for Chef is that Chef client runs on regular basis, polls Chef server, downloads run-lists and required configurations and performs the tasks on the node, where it runs.
After the installation, I’ve made, I can run the client manually from the command line by calling chef-client:
But sorry, I am not a fan to sit the whole day before a console and to start chef-client manually every hour or so.
The easiest way to start chef-client regularly and without an operator is to make an entry in crontab, something like:
root@delta# crontab -e 55 3,7,11,15,19,23 * * * /usr/bin/chef-client >/dev/console 2>&1
It is also possible to start chef-client in a daemon mode:
root@delta# chef-client -d -i 1800 -s 300
The option -i specifies an interval in seconds, how often chef-client polls its Chef Server. The option -s is to define splay – chef-client will choose a random number between 0 and splay seconds and add it to the interval. With this settings (-i 1800 -s 300) chef-client will poll its Chef Server every 30-35 minutes.
One can write a startup script and add it into /etc/rc.d/rc2.d or just make an entry in /etc/inittab to start the service with the system.
There is a third way – the Chef way. Chef authors have prepared a special cookbook with all required tasks to automate startup of chef-client. The cookbook is called chef-client and is known to work on AIX. IBM has checked it and published a document on Advanced Technical Support site. This is what I’d like to test.
But first I have to find the cookbook and download it. I use the command knife on my Chef Server for it:
chef$ knife cookbook site search chef-client ... chef-client: cookbook: https://supermarket.chef.io/api/v1/cookbooks/chef-client cookbook_description: Manages client.rb configuration and chef-client service cookbook_maintainer: chef cookbook_name: chef-client ...
Here it is:
chef$ knife cookbook site show chef-client average_rating: category: Other created_at: 2010-12-16T23:00:45.000Z deprecated: false description: Manages client.rb configuration and chef-client service external_url: http://github.com/opscode-cookbooks/chef-client foodcritic_failure: false issues_url: latest_version: https://supermarket.chef.io/api/v1/cookbooks/chef-client/versions/4.3.2 maintainer: chef ... name: chef-client source_url: http://github.com/opscode-cookbooks/chef-client up_for_adoption: updated_at: 2015-11-05T17:21:41.356Z versions: https://supermarket.chef.io/api/v1/cookbooks/chef-client/versions/4.3.2 ...
Let’s try to download it:
chef$ knife cookbook site download chef-client Downloading chef-client from Supermarket at version 4.3.2 to /chef-repo/chef-client-4.3.2.tar.gz Cookbook saved: /chef-repo/chef-client-4.3.2.tar.gz
The next step is to extract it and to upload to the Chef Server:
chef$ tar xzf chef-client-4.3.2.tar.gz -C cookbooks chef$ knife cookbook upload chef-client --cookbook-path cookbooks Uploading chef-client [4.3.2] ERROR: Cookbook chef-client depends on cookbooks which are not currently ERROR: being uploaded and cannot be found on the server. ERROR: The missing cookbook(s) are: 'cron' version '>= 1.2.0', 'logrotate' version '>= 1.2.0', 'windows' version '~> 1.37'
Oops. Something went wrong. We have to install dependencies for the cookbook. The same procedure, as for chef-client cookbook:
chef$ knife cookbook site download cron Downloading cron from Supermarket at version 1.7.2 to /chef-repo/cron-1.7.2.tar.gz Cookbook saved: /chef-repo/cron-1.7.2.tar.gz chef$ knife cookbook site download logrotate Downloading logrotate from Supermarket at version 1.9.2 to /chef-repo/logrotate-1.9.2.tar.gz Cookbook saved: /chef-repo/logrotate-1.9.2.tar.gz chef$ knife cookbook site download windows Downloading windows from Supermarket at version 1.38.4 to /chef-repo/windows-1.38.4.tar.gz Cookbook saved: /chef-repo/windows-1.38.4.tar.gz chef$ tar xzf cron-1.7.2.tar.gz -C cookbooks/ chef$ tar xzf logrotate-1.9.2.tar.gz -C cookbooks/ chef$ tar xzf windows-1.38.4.tar.gz -C cookbooks/ chef$ knife cookbook upload cron --cookbook-path cookbooks Uploading cron [1.7.2] Uploaded 1 cookbook. chef$ knife cookbook upload logrotate --cookbook-path cookbooks Uploading logrotate [1.9.2] Uploaded 1 cookbook. chef$ knife cookbook upload windows --cookbook-path cookbooks Uploading windows [1.38.4] ERROR: Cookbook windows depends on cookbooks which are not currently ERROR: being uploaded and cannot be found on the server. ERROR: The missing cookbook(s) are: 'chef_handler' version '>= 0.0.0'
Windows make problems as usual. Download and install one more dependency.
chef$ knife cookbook site download chef_handler Downloading chef_handler from Supermarket at version 1.2.0 to /chef-repo/chef_handler-1.2.0.tar.gz Cookbook saved: /chef-repo/chef_handler-1.2.0.tar.gz chef$ tar xzf chef_handler-1.2.0.tar.gz -C cookbooks/ chef$ knife cookbook upload chef_handler --cookbook-path cookbooks Uploading chef_handler [1.2.0] Uploaded 1 cookbook.
Now everything runs well:
chef$ knife cookbook upload windows --cookbook-path cookbooks Uploading windows [1.38.4] Uploaded 1 cookbook. chef$ knife cookbook upload chef-client --cookbook-path cookbooks Uploading chef-client [4.3.2] Uploaded 1 cookbook.
We’ve installed the cookbook, now we can try to use it on our AIX server. There are several recipes in it and several attributes, which affect these recipes. I don’t plan to change any attributes now, because I think the default behavior is quite good for the test, but I plan to use 2 recipes from the cookbook.
The first recipe delete_validation removes the file /etc/chef/validation.pem. It is a validation certificate, which was used previously to register and authorize the Chef Client on the Chef Server. We don’t need it anymore, because after the registration we received our own certificate from the server.
The second recipe is the default one, which installs a service on AIX, running chef-client.
Let’s add them:
chef$ knife node run_list add delta.enfence.net "recipe[chef-client::delete_validation]" delta.enfence.net: run_list: recipe[chef-client::delete_validation] chef$ knife node run_list add delta.enfence.net "recipe[chef-client]" delta.enfence.net: run_list: recipe[chef-client::delete_validation] recipe[chef-client]
After adding the recipes to the run-list we can start chef-client on the AIX side to converge the configuration:
root@delta # chef-client [2015-12-15T22:48:55+01:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support. Starting Chef Client, version 12.5.1 resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"] Synchronizing Cookbooks: - chef-client (4.3.2) - cron (1.7.2) - logrotate (1.9.2) ================================================================================ Error Syncing Cookbooks: ================================================================================ Unexpected Error: ----------------- ThreadError: can't create Thread: Resource temporarily unavailable Running handlers: [2015-12-15T22:49:25+01:00] ERROR: Running exception handlers Running handlers complete [2015-12-15T22:49:25+01:00] ERROR: Exception handlers complete Chef Client failed. 0 resources updated in 16 seconds [2015-12-15T22:49:25+01:00] ERROR: Failed to post reporting data to server (can't create Thread: Resource temporarily unavailable) [2015-12-15T22:49:25+01:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [2015-12-15T22:49:25+01:00] ERROR: can't create Thread: Resource temporarily unavailable [2015-12-15T22:49:26+01:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Something went obviously wrong. After a little bit googling I’ve found a bug report and a notice in Release Notes, that I have to change 2 parameters and to reboot AIX:
root@delta # chsec -f /etc/security/limits -s default -a "rss=-1" root@delta # chsec -f /etc/security/limits -s default -a "nofiles=50000" root@delta # shutdown -Fr
I really don’t think that it is a good idea to change limits for all users. I run chef-client under only one user – root. May be if I’d run some other recipes, they’d run under other users and require these parameters too. I am not sure. Even bigger problem for me is to reboot a server. If I had some production server, where I need to install chef-client, I really-really-really don’t want to reboot it. Some servers have downtime only once per year. That’s why I tried first to set up these parameters only for user root and as usual – logoff/login to make them active.
root@delta # chuser rss=-1 nofiles=50000 root
Unfortunately it didn’t help. Then I did what Chef suggests – changed the default values and rebooted the server. It didn’t help either – I received ThreadError again.
Then I changed two more attributes for root and logged off/in to activate them:
root@delta # chuser stack=-1 data=-1 root
It helped. I didn’t see the error anymore.
root@delta # chef-client [2015-12-15T22:57:11+01:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support. Starting Chef Client, version 12.5.1 resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"] Synchronizing Cookbooks: - chef-client (4.3.2) - cron (1.7.2) - logrotate (1.9.2) - windows (1.38.4) - chef_handler (1.2.0) Compiling Cookbooks... Converging 9 resources Recipe: chef-client::delete_validation * file[/etc/chef/validation.pem] action delete - delete file /etc/chef/validation.pem Recipe: chef-client::src_service * directory[/var/run/chef] action create - create new directory /var/run/chef - change owner from '' to 'root' - change group from '' to 'system' * directory[/var/spool/chef] action create - create new directory /var/spool/chef - change owner from '' to 'root' - change group from '' to 'system' * directory[/var/lib/chef] action create - create new directory /var/lib/chef - change owner from '' to 'root' - change group from '' to 'system' * directory[/var/adm/chef] action create - create new directory /var/adm/chef - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'system' * directory[/etc/chef] action create (up to date) * execute[install chef in SRC] action run - execute mkssys -s chef -p /usr/bin/chef-client -u root -S -n 15 -f 9 -o /var/adm/chef/client.log -e /var/adm/chef/client.log -a '-i 1800 -s 300' * execute[enable chef] action run - execute mkitab 'chef:2:once:/usr/bin/startsrc -s chef > /dev/console 2>&1' * service[chef] action start - start service service[chef] Running handlers: Running handlers complete Chef Client finished, 8/9 resources updated in 15 seconds
As you can see from the output the recipe created some directories and made a standard AIX service for chef-client. It also added an entry to /etc/inittab to start the service, and started it.
root@delta # lsitab chef chef:2:once:/usr/bin/startsrc -s chef > /dev/console 2>&1 root@delta # lssrc -s chef Subsystem Group PID Status chef 3145906 active root@delta # ps -ef | grep chef root 3145906 1900758 0 22:57:32 - 0:08 /opt/chef/embedded/bin/ruby
I also checked the log file in /var/adm/chef:
root@delta # cd /var/adm/chef root@delta # ls -l total 8 -rwxr-xr-x 1 root system 144 Dec 15 22:57 client.log root@delta # tail client.log [2015-12-15T22:57:33+01:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support.
At this point I was really annoyed by the message about the absent locale and decided to install it. No, I didn’t do it using AIX cookbook, I just installed it manually.
root@delta # lslpp -Lqc bos.loc.utf.EN_US bos.loc.utf.EN_US:bos.loc.utf.EN_US:18.104.22.168: : :C: :Base System Locale UTF Code Set - U. S. English : : : : : : :0:0:/:1543