Category Archives: Uncategorized

Setting Up a vSphere Service Account for Pivotal BOSH Director using PowerCLI

BOSH Director requires a fairly powerful vCenter service account to do all of the things it does.

The list of permissions required is here, and it’s extensive.

You can always take the shortcut and make your account an Administrator of the vSphere environment, but that violates the whole “least privilege” principle and I don’t like that in production environments.

I wrote a working PowerCLI code function that will automatically create this vCenter role and add the specified user/group to it.

It greatly reduces the time to set this up.  Hope this helps someone out.


Pivotal BOSH Director Setup Error – Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I was trying to install Pivotal Kubernetes Services on vSphere.  I setup the inital Operations Manger .ova appliance without issue.  Since I was deploying on vSphere, I needed to configure the BOSH Director installation through the vSphere tile next.  I ran through the configuration and tried to deploy once.. and it failed.  I tried again, and was dead-stopped at the above error over and over again.  I believe this came up because I deleted the BOSH/0 VM and tried to have the installer run again.

When in this state, it continually fails with the following error:

Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I had no idea what that meant, so I found this on the tech support site:

Same error, but I didn’t have BOSH director even setup yet so it didn’t apply.

The full log readout is below:

{“type”: “step_started”, “id”: “bosh_product.deploying”}
===== 2018-05-10 20:29:13 UTC Running “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”
Deployment manifest: ‘/var/tempest/workspaces/default/deployments/bosh.yml’
Deployment state: ‘/var/tempest/workspaces/default/deployments/bosh-state.json’

Started validating
Validating release ‘bosh’… Finished (00:00:00)
Validating release ‘bosh-vsphere-cpi’… Finished (00:00:00)
Validating release ‘uaa’… Finished (00:00:01)
Validating release ‘credhub’… Finished (00:00:00)
Validating release ‘bosh-system-metrics-server’… Finished (00:00:01)
Validating release ‘os-conf’… Finished (00:00:00)
Validating release ‘backup-and-restore-sdk’… Finished (00:00:04)
Validating cpi release… Finished (00:00:00)
Validating deployment manifest… Finished (00:00:00)
Validating stemcell… Finished (00:00:03)
Finished validating (00:00:12)

Started installing CPI
Compiling package ‘ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254’… Finished (00:00:00)
Compiling package ‘iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8’… Finished (00:00:00)
Compiling package ‘vsphere_cpi/3049e51ead9d72268c1f6dfb5b471cbc7e2d6816’… Finished (00:00:00)
Installing packages… Finished (00:00:00)
Rendering job templates… Finished (00:00:01)
Installing job ‘vsphere_cpi’… Finished (00:00:00)
Finished installing CPI (00:00:02)

Starting registry… Finished (00:00:00)
Uploading stemcell ‘bosh-vsphere-esxi-ubuntu-trusty-go_agent/3541.12’… Skipped [Stemcell already uploaded] (00:00:00)

Started deploying
Creating VM for instance ‘bosh/0’ from stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’… Failed (00:00:02)
Failed deploying (00:00:02)

Stopping registry… Finished (00:00:00)
Cleaning up rendered CPI jobs… Finished (00:00:00)

Creating instance ‘bosh/0’:
Creating VM:
Creating vm with stemcell cid ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’:
CPI ‘create_vm’ method responded with error: CmdError{“type”:”Unknown”,”message”:”Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29‘”,”ok_to_retry”:false}

Exit code 1
===== 2018-05-10 20:29:31 UTC Finished “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”; Duration: 18s; Exit Status: 1
{“type”: “step_finished”, “id”: “bosh_product.deploying”}
Exited with 1.

I did end up resolving this by deleting the bosh-state.json file. It apparently held some erroneous setup info about the stem cells that was causing the setup process to try and use a stemcell it had not yet downloaded.

I was able to SSH into the PKS Operations Manager VM and run this to fix it:

sudo rm /var/tempest/workspaces/default/deployments/bosh-state.json

Then, I was able to re-run the deployment with success.

Puppet File Sync Not Working – LOCK_FAILURE

I had a recent issue where Puppet was not properly syncing code from the code-staging directory to the code directory.  I verified it was pulling the new code from my Git repository to code-staging without issue.  However, file-sync was not pushing the new code to the code directory.

Here is what I was seeing in the /var/log/puppetlabs/puppetserver/puppetserver.log

2017-06-26 11:08:49,026 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:08:54,051 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:08:59,077 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:04,103 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:09,129 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.
2017-06-26 11:09:14,155 ERROR [clojure-agent-send-off-pool-3] [p.e.file-sync-errors] Error syncing repo :puppet-code: File sync successfully fetched from the server repo, but update-ref result was LOCK_FAILURE on 8c346001ee2f834a4be05d3d9788d2d712b212c5. Name: puppet-code. Directory: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git.

I had no idea what this meant, and I wasn’t sure how to resolve it so I took a snapshot of my Puppet Master VM and tried a few things.

The first thing I tried was going to the directory indicated and taking a look:

ll /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code/production.git/
total 44
drwxr-xr-x 7 pe-puppet pe-puppet 4096 Jun 26 11:03 ./
drwxr-xr-x 3 pe-puppet pe-puppet 4096 Apr 25 2016 ../
drwxr-xr-x 2 pe-puppet pe-puppet 4096 Apr 25 2016 branches/
-rw-r—– 1 pe-puppet pe-puppet 307 Jun 26 11:03 config
-rw-r—– 1 pe-puppet pe-puppet 148 Jun 26 10:28 FETCH_HEAD
-rw-r–r– 1 pe-puppet pe-puppet 23 Apr 25 2016 HEAD
drwxr-xr-x 2 pe-puppet pe-puppet 4096 Apr 25 2016 hooks/
drwxr-xr-x 3 pe-puppet pe-puppet 4096 Apr 25 2016 logs/
drwxr-xr-x 4 pe-puppet pe-puppet 4096 Jun 26 10:28 objects/
drwxr-xr-x 4 pe-puppet pe-puppet 4096 Apr 25 2016 refs/
-rw-r—– 1 pe-puppet pe-puppet 41 Jun 26 10:28 synced-commit

/opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git/production.git had the same contents but for one file:


I wasn’t sure this file belonged there, so I  removed it.  Once I did that, the file-sync service stopped throwing errors and successfully synced my files!

Hope this helps!

Encrypting Credentials In PowerShell Scripts

I have a long-standing dislike of hard-coding credentials in scripts.  In a production environment, it’s never a good idea to leave sensitive account passwords hard-coded in plain text in scripts.  To that end, I’ve developed an easy method in PowerShell to protect sensitive information.

The functions I present below allow you to store usernames and passwords, where the passwords are encrypted, in a form that can be later decrypted inside a script.  By default, only the user account that encrypted the credentials can decrypt them, and only from that same machine.  It all uses native .NET stuff, so you don’t need any third-party stuff to get it working.

Where I find this most useful is for services or scheduled tasks that run as system accounts that execute PowerShell scripts.  You can log into the machine as that service account, encrypt a set of credentials, then when that scheduled task runs as that service account it is able to read them.

Using the export function I show below, you can either export your credentials to an xml file on the file system, or a registry value in the Windows registry.

Here is an example:

First, save the credential to a variable and export it to an xml file:

$cred = Get-Credential username
$cred | Export-PSCredential -Path c:\temp\creds.xml

This outputs the path to the xml file you created with the encrypted credentials:


Alternately, you can export to a registry key instead:

$cred = Get-Credential username
$cred | Export-PSCredential -RegistryPath HKCU:\software\test -Name mycreds

In the registry, you can see your exported credentials:


The major thing that needs to be understood about this is the encryption key that is used to encrypt these credentials is tied to both the userid used to encrypt them AND the machine you encrypted from.  Unless you specify a keyphrase, you cannot decrypt these credentials as another user or from another machine.  The idea is if you have a script that reads these encrypted credentials, you have to log in as the user the script runs as on the machine the script runs from and encrypt them.  However, as described above, if you provide a keyphrase, you can decrypt them from anywhere as any user.  You just have to somehow protect the keyphrase.

Importing the credentials again is pretty simple:

$cred = Import-PSCredential -Path C:\temp\creds.xml
# OR
$cred = Import-PSCredential -RegistryPath HKCU:\Software\test -Name mycreds


Specifying a keyphrase involves specifying the -KeyPhrase parameter on either the import or export function.

Below is the code.  Simply paste these three functions into your PowerShell session or into your script and away you go.

Note the Get-EncryptionKey function is required for both the import and export functions!

Creating a Directory Tree in Puppet

As you can probably tell from the flurry of blog posts I’ve made concerning Puppet, I’m going through the process of learning and setting up Puppet Enterprise.

One thing that irked me early on is the inability of the file resource to create a directory if the parent directory does not exist.  For example:

file { 'mydirectory' :
  ensure         => 'directory',
  path           => 'c:/parentdir/childdir'

If c:\parentdir does not exist, this fails.

Error: Cannot create C:/parentdir/childdir; parent directory C:/parentdir does not exist
Error: /Stage[main]/Profile::Myclass/File[mydirectory]/ensure: change from absent to directory failed: Cannot create C:/parentdir/childdir; parent directory C:/parentdir does not exist

You can alternately specify it like this to get it to work:

file { ['c:/parentdir', 'c:/parentdir/childdir'] :
  ensure         => 'directory'

This works, and for the most part is OK.  In my case though, I have the user provide the directory name through a class parameter:

class myclass ([String] $mydirectory)
  file { 'mydirectory' :
   ensure         => 'directory',
   path           => $mydirectory

If the user specifies c:/parentdir/childdir, and c:/parentdir does not exist, it explodes.  I could adjust the code and advise my users to pass in arrays of strings representing the directories, but that’s not very clear or clean.

Fortunately, Puppet supports PowerShell and PowerShell is awesome:

class myclass (String $directory)
   exec { 'mydirectory' :
     command => "c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe -noprofile -noninteractive -command \"New-Item -ItemType Directory -Path \"$directory\" \"",
     onlyif  => "c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe -noprofile -noninteractive -command \"if (Test-Path -Path \"$directory\" -PathType Container) { exit 99 }\""

This code block creates the entire directory tree without issue.  The onlyif parameter ensures that the exec block is not fired off if the directory already exists.


Puppet Agent on Windows – Module not found

The first step I take when developing a new Puppet configuration is to install the Puppet Agent on a standalone test Windows server and build the configuration files locally there.  I then use the puppet apply utility to test it and make sure it works.  This saves a lot of time since it avoids having to do hundreds of pushes and merge requests to our source control system as I tweak and debug the config files to get them working the way I want.

I had some challenges getting this setup initially though.  I attempted to follow advice given to me by my Puppet SE, and researched and tried to implement Roles and Profiles as a means of developing layered configurations.  It make sense to do it this way, especially as your configuration base grows, but it requires a bit of know-how to get working properly.  One of the major stumbling blocks I hit was getting Puppet to recognize classes located in a non-standard directory.  The normal, standard directory structure looks like this:

    /modules # This is the default $basemodulepath
        /manifests  # This is where it expects site.pp and any other code you write
        /modules     # Your downloaded and custom modules can also go here

In my case, I wanted to create a “site” directory in which I stored my role and profile configurations per the design above.  My structure looked like this:


Since this was not in the default $basemodulepath directory  or the environment module directory I’d receive an error stating the class could not be found:


This is easy enough to figure out.  Puppet is highly configurable, and as such you can add additional directories to the list of those it looks in for classes it can use.  In my case, I simply edited the environment.conf file found at C:\ProgramData\PuppetLabs\code\environments\production\environment.conf  and commented-in the modulepath variable.  I then added my site folder.  I changed this line:

# modulepath = ./modules:$basemodulepath

To look like this:

modulepath = modules:site:$basemodulepath

However, I found I would still receive the same error as before.  A clue for me was when I ran the puppet config print modulepath command:

PS C:\ProgramData\PuppetLabs\code\environments\production\manifests> (puppet config print modulepath) -split ";"

You can see it lists the following paths:


None of these were my site directory.  It’s as if the change I made to environment.conf was simply ignored.

Essentially, I found it was.  Even though the inital example show in the environment.conf files shows this (note the colon delimiter):

# modulepath = ./modules:$basemodulepath

I found the Windows Agent uses semicolons, not colons as a delimiter for multiple paths.  This is kind of documented here.

Path Separator

Make sure to use a semi-colon (;) as the path separator on Windows, e.g., modulepath=path1;path2

Plain enough, but this document does not reference the environment.conf file specifically, or even the Puppet Agent (this seems to be just a general Windows thing).  Also, the Puppet Agent installer lays down the environment.conf file with the colons in place, so it’s very misleading.

In any case, I found that if I changed the file to look like this, everything worked:

modlepath = modules;site;$basemodulepath

Running puppet config print modulepath confirmed my site path now shows up:


So, in summary, if you are using any non-standard paths for your modules or classes on a Windows machine, make sure and use semicolons to delimit multiple paths for the modulepath setting, rather than the default colon.

Confusing, but easy to fix fortunately.

hiera-eyaml and Puppet Enterpise – Command not found?

I’m in the process of evaluating Puppet Enterprise as a configuration management solution for my company.  A glaring issue I hit early on is figuring out how to secure credentials that are fed to the various Puppet configurations.  By default, there is no way I’m aware of to obfuscate credentials in the configuration areas (including hiera files and class parameters in the GUI).  This is an issue as I can’t expose certain credentials to the general public.

Fortunately, hiera-eyaml was easy-to-find and does the trick.  There’s a lot of good documentation out there on how to set this up, and I won’t belabor that point, but to a Puppet noob the documentation makes a lot of assumptions.  The main assumption I want to clear up is how to get it up-and-running on your Puppet Master server using the eyaml utility from the CLI.


The GitHub document appears easy-to-follow:

The first step makes perfect sense, and worked without issue:

puppetserver gem install hiera-eyaml

The problem was after this.  I could not call the eyaml executable.  If I typed “eyaml –help”, “eyaml encrypt” or any valid variation of the command I received a “eyaml:  command not found” error.

Long story short, the issue is the Puppet master server does not have the ruby interpreter setup by default for command line use. The command above does make hiera-eyaml available for the Puppet software’s use, and you can go about configuring  it and using as stated in the GitHub readme for Puppet, but the eyaml calls will not work for you on the CLI.  The assumption they make is that you know to install the Ruby interpreter and gem separately for CLI usage.  To do this, do the following from the Puppet master (or any Linux station):

apt-get install ruby
gem install hiera-eyaml

Now the ruby interpreter is available for use to you on the CLI and you can call the eyaml executable as noted in the GitHub article.

I’m sure this is obvious to a Ruby/Linux expert, but it took me about 3/4 of a day to figure this out, so hopefully this helps save someone some time down the road.