Category Archives: Linux

Pivotal BOSH Director Setup Error – Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I was trying to install Pivotal Kubernetes Services on vSphere.  I setup the inital Operations Manger .ova appliance without issue.  Since I was deploying on vSphere, I needed to configure the BOSH Director installation through the vSphere tile next.  I ran through the configuration and tried to deploy once.. and it failed.  I tried again, and was dead-stopped at the above error over and over again.  I believe this came up because I deleted the BOSH/0 VM and tried to have the installer run again.

When in this state, it continually fails with the following error:

Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I had no idea what that meant, so I found this on the tech support site:
https://discuss.pivotal.io/hc/en-us/articles/115000488247-OpsManager-Install-Updates-error-Could-not-find-VM-for-stemcell-xxxxx-

Same error, but I didn’t have BOSH director even setup yet so it didn’t apply.

The full log readout is below:

{“type”: “step_started”, “id”: “bosh_product.deploying”}
===== 2018-05-10 20:29:13 UTC Running “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”
Deployment manifest: ‘/var/tempest/workspaces/default/deployments/bosh.yml’
Deployment state: ‘/var/tempest/workspaces/default/deployments/bosh-state.json’

Started validating
Validating release ‘bosh’… Finished (00:00:00)
Validating release ‘bosh-vsphere-cpi’… Finished (00:00:00)
Validating release ‘uaa’… Finished (00:00:01)
Validating release ‘credhub’… Finished (00:00:00)
Validating release ‘bosh-system-metrics-server’… Finished (00:00:01)
Validating release ‘os-conf’… Finished (00:00:00)
Validating release ‘backup-and-restore-sdk’… Finished (00:00:04)
Validating cpi release… Finished (00:00:00)
Validating deployment manifest… Finished (00:00:00)
Validating stemcell… Finished (00:00:03)
Finished validating (00:00:12)

Started installing CPI
Compiling package ‘ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254’… Finished (00:00:00)
Compiling package ‘iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8’… Finished (00:00:00)
Compiling package ‘vsphere_cpi/3049e51ead9d72268c1f6dfb5b471cbc7e2d6816’… Finished (00:00:00)
Installing packages… Finished (00:00:00)
Rendering job templates… Finished (00:00:01)
Installing job ‘vsphere_cpi’… Finished (00:00:00)
Finished installing CPI (00:00:02)

Starting registry… Finished (00:00:00)
Uploading stemcell ‘bosh-vsphere-esxi-ubuntu-trusty-go_agent/3541.12’… Skipped [Stemcell already uploaded] (00:00:00)

Started deploying
Creating VM for instance ‘bosh/0’ from stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’… Failed (00:00:02)
Failed deploying (00:00:02)

Stopping registry… Finished (00:00:00)
Cleaning up rendered CPI jobs… Finished (00:00:00)

Deploying:
Creating instance ‘bosh/0’:
Creating VM:
Creating vm with stemcell cid ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’:
CPI ‘create_vm’ method responded with error: CmdError{“type”:”Unknown”,”message”:”Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29‘”,”ok_to_retry”:false}

Exit code 1
===== 2018-05-10 20:29:31 UTC Finished “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”; Duration: 18s; Exit Status: 1
{“type”: “step_finished”, “id”: “bosh_product.deploying”}
Exited with 1.

I did end up resolving this by deleting the bosh-state.json file. It apparently held some erroneous setup info about the stem cells that was causing the setup process to try and use a stemcell it had not yet downloaded.

I was able to SSH into the PKS Operations Manager VM and run this to fix it:

sudo rm /var/tempest/workspaces/default/deployments/bosh-state.json

Then, I was able to re-run the deployment with success.

Installing CloudFoundry User Account and Authentication Client (cf-uaac) on Windows

I’m doing some playing round with Pivotal CloudFoundry and Kubernetes and ran into an issue where during the setup I needed to use their cf-uaac tool (written in Ruby) to complete the setup and manage authentication to the system.

There are a lot of instructions out there on how to do this on Linux and pretty much all of them assume you have an Internet connection. I found not only can you install this on Windows, but you can do so on a machine that does not have Internet access.

Below, I detail how to install cf-uaac on Ubuntu Linux and Windows both with and without an Internet connection.

Prerequisites for Either Installation Method

Whether or not you have Internet access on your target machine, you need to follow these steps to setup your machine to leverage the Ruby gems.

For Linux
# Build-essential is a prerequisite for a lot of Ruby gems.
apt install -y build-essential ruby ruby-dev
For Windows
  • Download ruby (with the devkit): https://rubyinstaller.org/downloads
  • Install MSYS2
    • The devkit installer will do this for you if your machine has Internet access.
    • Otherwise, the installer will run with errors and you have to manually install it afterwards from here.
  • Make sure c:\Rubyxx-x64\bin is in your PATH environment variable (where xx is the current Ruby version)

Installing cf-uaac From the Internet

This is pretty easy and detailed in a lot of other places on the Internet. For brevity, I included quick instructions here:

For Either Windows or Linux
gem install cf-uaac

Installing cf-uaac Without a Direct Internet Connection

This method assumes you have a workstation that has Internet access from which you can download the gems. Then, you can copy them to the target machine that you need to run uaac from.

CF-UAAC has a list of required gems (as of this writing):

rack-1.6.9.gem
highline-1.6.21.gem
cookiejar-0.3.3.gem
addressable-2.5.2.gem
launchy-2.4.3.gem
eventmachine-1.2.5.gem
em-http-request-1.1.5.gem
httpclient-2.8.3.gem
cf-uaac-4.1.0.gem
json_pure-1.8.6.gem
public_suffix-3.0.2.gem
em-socksify-0.3.2.gem
multi_json-1.12.2.gem
cf-uaa-lib-3.13.0.gem
http_parser.rb-0.6.0.gem

Note that cf-uaac doesn’t require (moreover, doesn’t allow) the latest versions of all of these plugins. You need to make sure you observe the version requirements as listed. For instance, the runtime dependencies for cf-uaac are currently:

uaac-requirements

You need em-http-request version >= 1.1.2 and < 1.2. For more info on pessimistic versioning and constraints in Ruby, see this article.

Download each gem by visiting its page on Rubygems.org and clicking the “Download” link on the page.

Once you have each gem (and each gem’s dependencies) downloaded, you can move the .gem files you downloaded to somewhere reachable by your target machine.

Installing On Linux
# Install a single gem:
gem install --local /tmp/mygem.gem

# Or install a series of gems from a directory:
for file in /tmp/PKS/*.gem; do gem install --local "$file"; done
Installing On Windows
# Install a single gem:
gem install --local c:\temp\mygem.gem

# Install a series of gems from a directory:
Get-Item -Path "c:\temp\PKS\*.gem" | Sort-Object -Property Name | Foreach-Object { gem install --local "$($_.FullName)" }

Once these steps are complete, the uaac binary should be added to the Ruby/bin (Windows) or /usr/loca/bin (Linux) path and can be executed by typing uaac from your console (PowerShell or Bash).

Most issues I had getting this working were because the prerequisites weren’t present. Make sure build-essential, ruby and ruby-dev are installed on Linux machines and that Ruby with the devkit and MSYS2 is installed on Windows machines.

With all of this done, I was able to manage my PKS UAA component from the CLI on my Windows and Linux machines.

Puppet File Sync Not Working – LOCK_FAILURE

I had a recent issue where Puppet was not properly syncing code from the code-staging directory to the code directory.  I verified it was pulling the new code from my Git repository to code-staging without issue.  However, file-sync was not pushing the new code to the code directory.

Here is what I was seeing in the /var/log/puppetlabs/puppetserver/puppetserver.log

2017-06-26 11:08:49,026 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:08:54,051 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:08:59,077 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:04,103 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:09,129 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:14,155 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.

I had no idea what this meant, and I wasn’t sure how to resolve it so I took a snapshot of my Puppet Master VM and tried a few things.

The first thing I tried was going to the directory indicated and taking a look:

ll /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code/production.git/
total 44
drwxr-xr-x 7 pe-puppet pe-puppet 4096 Jun 26 11:03 ./
drwxr-xr-x 3 pe-puppet pe-puppet 4096 Apr 25 2016 ../
drwxr-xr-x 2 pe-puppet pe-puppet 4096 Apr 25 2016 branches/
-rw-r—– 1 pe-puppet pe-puppet 307 Jun 26 11:03 config
-rw-r—– 1 pe-puppet pe-puppet 148 Jun 26 10:28 FETCH_HEAD
-rw-r–r– 1 pe-puppet pe-puppet 23 Apr 25 2016 HEAD
drwxr-xr-x 2 pe-puppet pe-puppet 4096 Apr 25 2016 hooks/
drwxr-xr-x 3 pe-puppet pe-puppet 4096 Apr 25 2016 logs/
drwxr-xr-x 4 pe-puppet pe-puppet 4096 Jun 26 10:28 objects/
drwxr-xr-x 4 pe-puppet pe-puppet 4096 Apr 25 2016 refs/
-rw-r—– 1 pe-puppet pe-puppet 41 Jun 26 10:28 synced-commit

/opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git/production.git had the same contents but for one file:

synced-commit.lock

I wasn’t sure this file belonged there, so I  removed it.  Once I did that, the file-sync service stopped throwing errors and successfully synced my files!

Hope this helps!

Puppet Agent on Windows – Module not found

The first step I take when developing a new Puppet configuration is to install the Puppet Agent on a standalone test Windows server and build the configuration files locally there.  I then use the puppet apply utility to test it and make sure it works.  This saves a lot of time since it avoids having to do hundreds of pushes and merge requests to our source control system as I tweak and debug the config files to get them working the way I want.

I had some challenges getting this setup initially though.  I attempted to follow advice given to me by my Puppet SE, and researched and tried to implement Roles and Profiles as a means of developing layered configurations.  It make sense to do it this way, especially as your configuration base grows, but it requires a bit of know-how to get working properly.  One of the major stumbling blocks I hit was getting Puppet to recognize classes located in a non-standard directory.  The normal, standard directory structure looks like this:

C:/ProgramData/PuppetLabs/code
    /modules # This is the default $basemodulepath
    /environments/production
        /manifests  # This is where it expects site.pp and any other code you write
        /modules     # Your downloaded and custom modules can also go here

In my case, I wanted to create a “site” directory in which I stored my role and profile configurations per the design above.  My structure looked like this:

c:/programdata/puppetlabs/code/environments/production
    /site
        /profile/manifests
        /role/manifests

Since this was not in the default $basemodulepath directory  or the environment module directory I’d receive an error stating the class could not be found:

ModuleNotFound

This is easy enough to figure out.  Puppet is highly configurable, and as such you can add additional directories to the list of those it looks in for classes it can use.  In my case, I simply edited the environment.conf file found at C:\ProgramData\PuppetLabs\code\environments\production\environment.conf  and commented-in the modulepath variable.  I then added my site folder.  I changed this line:

# modulepath = ./modules:$basemodulepath

To look like this:

modulepath = modules:site:$basemodulepath

However, I found I would still receive the same error as before.  A clue for me was when I ran the puppet config print modulepath command:

PS C:\ProgramData\PuppetLabs\code\environments\production\manifests&amp;gt; (puppet config print modulepath) -split &quot;;&quot;

You can see it lists the following paths:

C:/ProgramData/PuppetLabs/code/environments/production/modules
C:/ProgramData/PuppetLabs/code/modules
C:/opt/puppetlabs/puppet/modules

None of these were my site directory.  It’s as if the change I made to environment.conf was simply ignored.

Essentially, I found it was.  Even though the inital example show in the environment.conf files shows this (note the colon delimiter):

# modulepath = ./modules:$basemodulepath

I found the Windows Agent uses semicolons, not colons as a delimiter for multiple paths.  This is kind of documented here.

Path Separator

Make sure to use a semi-colon (;) as the path separator on Windows, e.g., modulepath=path1;path2

Plain enough, but this document does not reference the environment.conf file specifically, or even the Puppet Agent (this seems to be just a general Windows thing).  Also, the Puppet Agent installer lays down the environment.conf file with the colons in place, so it’s very misleading.

In any case, I found that if I changed the file to look like this, everything worked:

modlepath = modules;site;$basemodulepath

Running puppet config print modulepath confirmed my site path now shows up:

C:/ProgramData/PuppetLabs/code/environments/production/modules
C:/ProgramData/PuppetLabs/code/environments/production/site
C:/ProgramData/PuppetLabs/code/modules
C:/opt/puppetlabs/puppet/modules

So, in summary, if you are using any non-standard paths for your modules or classes on a Windows machine, make sure and use semicolons to delimit multiple paths for the modulepath setting, rather than the default colon.

Confusing, but easy to fix fortunately.

hiera-eyaml and Puppet Enterpise – Command not found?

I’m in the process of evaluating Puppet Enterprise as a configuration management solution for my company.  A glaring issue I hit early on is figuring out how to secure credentials that are fed to the various Puppet configurations.  By default, there is no way I’m aware of to obfuscate credentials in the configuration areas (including hiera files and class parameters in the GUI).  This is an issue as I can’t expose certain credentials to the general public.

Fortunately, hiera-eyaml was easy-to-find and does the trick.  There’s a lot of good documentation out there on how to set this up, and I won’t belabor that point, but to a Puppet noob the documentation makes a lot of assumptions.  The main assumption I want to clear up is how to get it up-and-running on your Puppet Master server using the eyaml utility from the CLI.

 

The GitHub document appears easy-to-follow:

https://github.com/TomPoulton/hiera-eyaml

The first step makes perfect sense, and worked without issue:

puppetserver gem install hiera-eyaml

The problem was after this.  I could not call the eyaml executable.  If I typed “eyaml –help”, “eyaml encrypt” or any valid variation of the command I received a “eyaml:  command not found” error.

Long story short, the issue is the Puppet master server does not have the ruby interpreter setup by default for command line use. The command above does make hiera-eyaml available for the Puppet software’s use, and you can go about configuring  it and using as stated in the GitHub readme for Puppet, but the eyaml calls will not work for you on the CLI.  The assumption they make is that you know to install the Ruby interpreter and gem separately for CLI usage.  To do this, do the following from the Puppet master (or any Linux station):

apt-get install ruby
gem install hiera-eyaml

Now the ruby interpreter is available for use to you on the CLI and you can call the eyaml executable as noted in the GitHub article.

I’m sure this is obvious to a Ruby/Linux expert, but it took me about 3/4 of a day to figure this out, so hopefully this helps save someone some time down the road.

Installing the ASUS PCE-N53 WIFI Card in a Machine Running SteamOS

Recently, I took the dive into building a custom SteamOS box.  So far, it’s been a fun exercise.  SteamOS basically is the Big Picture Mode for the Steam Client running on top of a debian-based Linux platform.  There are plenty of articles that go into the installation of it, so I won’t bother with that.

The main issue I see is that there is not a whole lot of support for various hardware.  In my case, I had a few-year-old ASUS PCE-N53 WIFI card that I wanted to install and use.   However, the SteamOS did not detect the card at all, thus began an adventure in research into getting it working.

The below procedure walks you through installing the drivers for the card.  To do this, you have to get into the desktop mode on the SteamOS and be familiar with the terminal.

I’m using SteamOS version 1.0 Update 161.

Installing WIFI PCE-N53 Card

Get the driver patch

ASUS has a Linux driver, but the driver is not compatible with Linux kernels under version 3.0.  There is an unsupported patch out there, it worked for me, but use at your own risk!

  1. Get the v3 patch:

http://gridlox.net/diff/rt5592sta_fix_64bit_3.8.patch

2.  Put the patch on the SteamBox in the /home/desktop/Downloads folder

In my case, I used WinSCP from a Windows Desktop.  You can use wget from the SteamOS console.

Get the ASUS Driver

  1. Download the driver from the ASUS site:

http://support.asus.com/download.aspx?SLanguage=en&p=11&m=PCE-N53&hashedid=F3giHhIS4hp6aWBW

Drop this in the /home/desktop/Downloads folder

Patch and Compile the Driver

To perform these steps, you need to be logged into the Linux terminal on the steamOS box.  You can do this in two ways:

  1. Log into the desktop in the SteamOS and launch the terminal
  2. Enable SSH on the SteamOS box and SSH in
  3. Log onto the SteamOS terminal

Go to the SteamOS desktop, then to the terminal

Or, alternately, SSH to the desktop shell

Install 7-zip and the make utility

sudo apt-get install p7zip-full
sudo apt-get install build-essential
  1. Unpack the driver
7z x Linux_PCE_N53_1008.zip
cd Linux
7z x DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326.tar.bz2
7z x DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326.tar

2.  Patch the driver

cd DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326/
patch -p1 < ~/Downloads/rt5592sta_fix_64bit_3.8.patch

3.  Compile

sudo make

4.  Install the compiled driver

sudo make install

5.  Rescan for the new card

sudo modprobe rt5592sta

Viola!  Now the card works. I can configure it from the SteamOS desktop network configuration utility.  Hope this is of benefit to someone out there.

Setting Up Libvirt on an Ubuntu Machine to Power on ESXi VMs

I recently setup a MAAS cluster on Ubuntu Linux in my home lab so that I could play around with Openstack and Juju.  I wanted MAAS to be able to automatically power on and off my newly-provisioned vms, but I had a hard time finding an easy-to-follow set of instructions on how to get this working on VMware ESXi.  I did eventually get it working and put together a procedure below.  Hopefully it helps someone out.

My Environment

  • VMware ESXi (Standalone, no VirtualCenter server) version 5.5.0
  • Ubuntu MAAS on VMware VMs (no physical servers)
  • libvirt version 1.2.8
  • Ubuntu Linux 14.04

Procedure

Install and Build the libvirt 1.2.8

cd /tmp
wget
http://libvirt.org/sources/libvirt-1.2.8.tar.gz

  • Untar the file

tar -xzf /tmp/libvirt-1.2.8.tar.gz -C /tmp/

  • Uninstall any existing version of libvir

sudo apt-get remove libvirt-bin

  • Install the prerequisites

sudo apt-get install gcc make libxml2-dev libgnutls-dev libdevmapper-dev libcurl4-gnutls-dev  libpciaccess-dev  libnl-dev

  • Make and Install the new version
    • Specify the /usr directory when calling the configure command to specify where the non-architecture specific stuff gets installed

cd /tmp/libvirt-1.2.8
./configure –prefix=/usr –with-esx
make
sudo make install

Setup Authentication

You have to setup the /etc/libvirt/auth.conf file with credentials in order to use virsh without entering a password.  This is necessary for MAAS to power nodes up automatically.

sudo nano /etc/libvirt/auth.conf

virsh1

  • The credentials-esx block defines the credentials
    • credentials-esx  #  The esx in this part defines the name of the credential set
  • The second block defines the service and hostname
    • auth-esx #  This defines which credential block to use.  In this case, the esx credentials
    • auth-esx-hostname #  This defines the host to use the esx credentials for
  • Restart libvirt-bin

sudo service libvirt-bin restart

Test your Configuration

You can make a test connection to your ESXi host to test your virsh configuration with the command below:

virsh -c esx://root@esxhostname1?no_verify=1 list –all

This will prompt you for your root password and then list all VMs on the VM host.

3/25/2015 – UPDATE:  I found that this will only work on the paid (non-evaulation) version of vSphere ESXi.  If you have the eval version, it will work until your eval expires and reverts to the unpaid license.  MAAS will silently fail to power on or off VMs when this happens.  You will get this error when trying to run the command manually from the CLI:

virsh -c esx://root@esxhostname?no_verify=1 start –domain myvmname

error:  Failed to start the domain myvmname
error:  internal error:  HTTP response code 500 for call to ‘PowerOnVM_Task’.  Fault:  ServerFaultCode – Current license or ESXi version prohibits executio nof the requested operation.

Pretty clear, I guess.

Configuring MAAS for Virsh

Now you have to configure your new MAAS nodes with virsh.  To do this, we’ll simulate adding a new node to a MAAS cluster.

  • First, use the VSphere client to connect to your ESXi host and create a new VM.
  • Make sure and set the OS Type to Ubuntu Linux (I use 64-bit)
  • Once the vm is created, right-click it and Edit Settings
  • Make sure you go to the Boot tab and force it to the BIOS for the next boot.

virsh3

  • Start the new VM and go into the BIOS.
  • Go to the boot tab in the BIOS and set the network card to be the first boot option (Use SHIFT plus to move it up)

virsh4

  • Go to the Exit menu and choose Save.
  • Power off the VM.
  • Edit the settings again
  • Select the network card and copy the MAC address

virsh5

virsh2

  • Fill out the fields as specified:

virsh6

  • Power Type:  Select virsh from the drop down
  • Power Address:  esx://root@esxhost1/system?no_verify=1
    • Where esxhost1 is the name of your ESXi host
  • Power ID:  The name of the VM as displayed in the VM inventory in the vSphere client.
  • Now click “Add an additional MAC address”
  • Paste in the MAC address of the VMs virtual network card you got above.

Done!  I’ve not found a way to get past manually booting the node for the first time.  After you manually power it on once, it will enlist and shut itself off.  From there on out, you can click the node in the MAAS web interface and click “Start Node” or “Stop Node” to power it on and off.

References

http://www.gremwell.com/node/155
http://www.libvirt.org
http://manpages.ubuntu.com/manpages/lucid/man1/virsh.1.html