Category Archives: Kubernetes

Setting Up a vSphere Service Account for Pivotal BOSH Director using PowerCLI

BOSH Director requires a fairly powerful vCenter service account to do all of the things it does.

The list of permissions required is here, and it’s extensive.

You can always take the shortcut and make your account an Administrator of the vSphere environment, but that violates the whole “least privilege” principle and I don’t like that in production environments.

I wrote a working PowerCLI code function that will automatically create this vCenter role and add the specified user/group to it.

It greatly reduces the time to set this up.  Hope this helps someone out.

 

Pivotal BOSH Director Setup Error – Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I was trying to install Pivotal Kubernetes Services on vSphere.  I setup the inital Operations Manger .ova appliance without issue.  Since I was deploying on vSphere, I needed to configure the BOSH Director installation through the vSphere tile next.  I ran through the configuration and tried to deploy once.. and it failed.  I tried again, and was dead-stopped at the above error over and over again.  I believe this came up because I deleted the BOSH/0 VM and tried to have the installer run again.

When in this state, it continually fails with the following error:

Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’

I had no idea what that meant, so I found this on the tech support site:
https://discuss.pivotal.io/hc/en-us/articles/115000488247-OpsManager-Install-Updates-error-Could-not-find-VM-for-stemcell-xxxxx-

Same error, but I didn’t have BOSH director even setup yet so it didn’t apply.

The full log readout is below:

{“type”: “step_started”, “id”: “bosh_product.deploying”}
===== 2018-05-10 20:29:13 UTC Running “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”
Deployment manifest: ‘/var/tempest/workspaces/default/deployments/bosh.yml’
Deployment state: ‘/var/tempest/workspaces/default/deployments/bosh-state.json’

Started validating
Validating release ‘bosh’… Finished (00:00:00)
Validating release ‘bosh-vsphere-cpi’… Finished (00:00:00)
Validating release ‘uaa’… Finished (00:00:01)
Validating release ‘credhub’… Finished (00:00:00)
Validating release ‘bosh-system-metrics-server’… Finished (00:00:01)
Validating release ‘os-conf’… Finished (00:00:00)
Validating release ‘backup-and-restore-sdk’… Finished (00:00:04)
Validating cpi release… Finished (00:00:00)
Validating deployment manifest… Finished (00:00:00)
Validating stemcell… Finished (00:00:03)
Finished validating (00:00:12)

Started installing CPI
Compiling package ‘ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254’… Finished (00:00:00)
Compiling package ‘iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8’… Finished (00:00:00)
Compiling package ‘vsphere_cpi/3049e51ead9d72268c1f6dfb5b471cbc7e2d6816’… Finished (00:00:00)
Installing packages… Finished (00:00:00)
Rendering job templates… Finished (00:00:01)
Installing job ‘vsphere_cpi’… Finished (00:00:00)
Finished installing CPI (00:00:02)

Starting registry… Finished (00:00:00)
Uploading stemcell ‘bosh-vsphere-esxi-ubuntu-trusty-go_agent/3541.12’… Skipped [Stemcell already uploaded] (00:00:00)

Started deploying
Creating VM for instance ‘bosh/0’ from stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’… Failed (00:00:02)
Failed deploying (00:00:02)

Stopping registry… Finished (00:00:00)
Cleaning up rendered CPI jobs… Finished (00:00:00)

Deploying:
Creating instance ‘bosh/0’:
Creating VM:
Creating vm with stemcell cid ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29’:
CPI ‘create_vm’ method responded with error: CmdError{“type”:”Unknown”,”message”:”Could not find VM for stemcell ‘sc-b0131c8f-ef44-456b-8e7c-df3951236d29‘”,”ok_to_retry”:false}

Exit code 1
===== 2018-05-10 20:29:31 UTC Finished “/usr/local/bin/bosh –no-color –non-interactive –tty create-env /var/tempest/workspaces/default/deployments/bosh.yml”; Duration: 18s; Exit Status: 1
{“type”: “step_finished”, “id”: “bosh_product.deploying”}
Exited with 1.

I did end up resolving this by deleting the bosh-state.json file. It apparently held some erroneous setup info about the stem cells that was causing the setup process to try and use a stemcell it had not yet downloaded.

I was able to SSH into the PKS Operations Manager VM and run this to fix it:

sudo rm /var/tempest/workspaces/default/deployments/bosh-state.json

Then, I was able to re-run the deployment with success.

Installing CloudFoundry User Account and Authentication Client (cf-uaac) on Windows

I’m doing some playing round with Pivotal CloudFoundry and Kubernetes and ran into an issue where during the setup I needed to use their cf-uaac tool (written in Ruby) to complete the setup and manage authentication to the system.

There are a lot of instructions out there on how to do this on Linux and pretty much all of them assume you have an Internet connection. I found not only can you install this on Windows, but you can do so on a machine that does not have Internet access.

Below, I detail how to install cf-uaac on Ubuntu Linux and Windows both with and without an Internet connection.

Prerequisites for Either Installation Method

Whether or not you have Internet access on your target machine, you need to follow these steps to setup your machine to leverage the Ruby gems.

For Linux
# Build-essential is a prerequisite for a lot of Ruby gems.
apt install -y build-essential ruby ruby-dev
For Windows
  • Download ruby (with the devkit): https://rubyinstaller.org/downloads
  • Install MSYS2
    • The devkit installer will do this for you if your machine has Internet access.
    • Otherwise, the installer will run with errors and you have to manually install it afterwards from here.
  • Make sure c:\Rubyxx-x64\bin is in your PATH environment variable (where xx is the current Ruby version)

Installing cf-uaac From the Internet

This is pretty easy and detailed in a lot of other places on the Internet. For brevity, I included quick instructions here:

For Either Windows or Linux
gem install cf-uaac

Installing cf-uaac Without a Direct Internet Connection

This method assumes you have a workstation that has Internet access from which you can download the gems. Then, you can copy them to the target machine that you need to run uaac from.

CF-UAAC has a list of required gems (as of this writing):

rack-1.6.9.gem
highline-1.6.21.gem
cookiejar-0.3.3.gem
addressable-2.5.2.gem
launchy-2.4.3.gem
eventmachine-1.2.5.gem
em-http-request-1.1.5.gem
httpclient-2.8.3.gem
cf-uaac-4.1.0.gem
json_pure-1.8.6.gem
public_suffix-3.0.2.gem
em-socksify-0.3.2.gem
multi_json-1.12.2.gem
cf-uaa-lib-3.13.0.gem
http_parser.rb-0.6.0.gem

Note that cf-uaac doesn’t require (moreover, doesn’t allow) the latest versions of all of these plugins. You need to make sure you observe the version requirements as listed. For instance, the runtime dependencies for cf-uaac are currently:

uaac-requirements

You need em-http-request version >= 1.1.2 and < 1.2. For more info on pessimistic versioning and constraints in Ruby, see this article.

Download each gem by visiting its page on Rubygems.org and clicking the “Download” link on the page.

Once you have each gem (and each gem’s dependencies) downloaded, you can move the .gem files you downloaded to somewhere reachable by your target machine.

Installing On Linux
# Install a single gem:
gem install --local /tmp/mygem.gem

# Or install a series of gems from a directory:
for file in /tmp/PKS/*.gem; do gem install --local "$file"; done
Installing On Windows
# Install a single gem:
gem install --local c:\temp\mygem.gem

# Install a series of gems from a directory:
Get-Item -Path "c:\temp\PKS\*.gem" | Sort-Object -Property Name | Foreach-Object { gem install --local "$($_.FullName)" }

Once these steps are complete, the uaac binary should be added to the Ruby/bin (Windows) or /usr/loca/bin (Linux) path and can be executed by typing uaac from your console (PowerShell or Bash).

Most issues I had getting this working were because the prerequisites weren’t present. Make sure build-essential, ruby and ruby-dev are installed on Linux machines and that Ruby with the devkit and MSYS2 is installed on Windows machines.

With all of this done, I was able to manage my PKS UAA component from the CLI on my Windows and Linux machines.