Author Archives: Brandon

About Brandon

I've been in the IT industry for 17 years and currently work as a Systems Architect for Plex Systems Inc. in Auburn Hills, MI. I specialize process automation and orchestration, focusing on IT Infrastructure operations. I have a range of technical interests I enjoy writing about in both personal and professional pursuits.

Puppet Enterprise – Adding Windows Scheduled Tasks

So, continuing on the path I’ve been on, I’ve had to create quite a few custom “resources” in my Puppet profiles to deploy or configure items I could not find right out-of-the-box.  In this case, I have a server that requires a standard set of Windows scheduled tasks.

For this purpose, I created a new pseudo-resource called “windows_scheduled_task”.  As with the other items I’ve published, I call this a pseudo-resource because it’s not really a Puppet resource.  It’s a custom class that is used just like a resource.  The approach I took here leverages PowerShell and assumes the presence of the ScheduledTasks module, which is only available in PowerShell v4 and higher.

The class requires the use of a module class (.pp file) and an accompanying template file (.epp).  The .pp file goes in the manifests folder in your module, and the template in your templates folder.  The assumed folder structure is like so:

/manifests/windows_server
  /scheduled_task.pp
/templates/windows_server
  /scheduled_task_add.epp

If you change the paths, that’s OK, but you have to make sure the class namespace in the .pp file matches your new folder structure. The default is

class windows_server::scheduled_task()

which assumes the folder /manifests/windows_server

You also have to make sure the epp() function call in the .pp file references the correct path to the template (if you change it). Right now, it’s set to look at /templates/windows_server/scheduled_task_add.epp.

Here is the .pp file class:

class windows_server::scheduled_task()
{

  define windows_scheduled_task
  (
    String $description = "No description.",
    String $path = "",
    String $executionTimeLimit = "01.00:00:00",
    String $userName = "NT AUTHORITY\\SYSTEM",
    String $password = "",
    Boolean $deployEnabled = true,
    Array[Hash] $actions,
    Array[Hash] $triggers = []
  )
  {
    #  name (string)                - Specifies the name of the task
    #  description (string)         - Specifies a description of the task
    #  path (string)                - Specifies the folder to place the task in.  Default is "\" (the root folder).  NOTE:  This must begin with a slash but not end with one!  Example:  /Restore
    #  executionTimeLimit (string)  - Specifies the length of time the task can run before being automatically stopped.  Specify as a TimeSpan.
    #  deployEnabled (bool)         - Determines whether the task should deployed in an enabled state or not.  This state is not enforced going forward.
    #  actions (Hash[]) -
    #    workingDirectory (string)      - Specifies the working directory for the action.  Default is C:\windows\system32
    #    command (string)               - Specifies the command to execute.
    #    arguments (string[])           - Specifies the arguments to pass to the command.
    #    isPowerShell (bool)            - If specified, then the command and arguments are automatically constructed.  You only need pass the powershell script you want to run for the command.

    #  triggers (Hash[]) -
    #    atDateTime (String)          - Specifies the date and time to start running the task.
    #    repetitionInterval (string)  - Specifies how often to re-run the task after the atDateTime occurs.  Specify as a Timespan.
    #    repetitionDuration (string)  - Specifies how long to repeat the task executions for.  Specify as a Timespan.  Default is [Timespan]::MaxValue (forever)

    #  If your command is a PowerShell script, you have to escape double-quotes with backslashes.
    #  Example:
    #  windows_server::scheduled_task::windows_scheduled_task { 'Test Scheduled Task':
    #   userName          =>  $taskCredentials['userName'],
    #   password          =>  $taskCredentials['password'],
    #   path              => '\MyTasks',
    #   actions           => [{
    #    isPowerShell        => true,
    #    command             => "c:\\scripts\\Run-MyPowerShellScript.ps1 -Param1 value1 -Param2 \"value 2\" -Param3 ${puppetVariableHere}  "
    #   }],
    #   triggers              => [{
    #    atDateTime          => "9/1/2016 12:30 AM",
    #    repetitionInterval  => "00:30:00"
    #   }],
    #}

    exec { "scheduled_task_${title}" :
      command       => epp("windows_server/scheduled_task_add.epp", {
                        name                => $name,
                        description         => $description,
                        path                => $path,
                        executionTimeLimit  => $executionTimeLimit,
                        userName            => $userName,
                        password            => $password,
                        deployEnabled       => $deployEnabled,
                        actions             => $actions,
                        triggers            => $triggers
                      }),
      onlyif        => "if ( ScheduledTasks\\Get-ScheduledTask | Where-Object { \$_.TaskName -ieq \"${name}\" -and \$_.TaskPath -ieq \"${path}\\\" } ) { \$host.SetShouldExit(99); exit 99 }",
      returns       => [0],
      provider      => powershell,
      logoutput     => true,
    }
  }
}

The template file is here:

<%- | String $name,
      String $description = "No description",
      String $path = "\\",
      String $executionTimeLimit = "01.00:00:00",
      String $userName = "NT AUTHORITY\\SYSTEM",
      String $password = "",
      Boolean $deployEnabled = true,
      Array[Hash] $actions,
      Array[Hash] $triggers = []
|
  #  name (string) - Specifies the name of the task
  #  description (string) - Specifies a description of the task
  #  path (string) - Specifies the folder to place the task in.  Default is "\" (the root foler)
  #  executionTimeLimit (string) - Specifies the length of time the task can run before being automatically stopped.  Specify as a TimeSpan.
  #  userName (string) - Specifies the user to execute the task as.  Default is local system,.
  #  password (string) - Specifies the password for the given user.
  #  actions (Hash[]) -
  #    workingDirectory (string) - Specifies the working directory for the action.  Default is C:\windows\system32
  #    command (string) - Specifies the command to execute.
  #    arguments (string[]) - Specifies the arguments to pass to the command.
  #    isPowerShell (bool) - If specified, then the command and arguments are automatically constructed.  You only need pass the powershell script you want to run for the command.

  #  triggers (Hash[]) -
  #    atDateTime (String) - Specifies the date and time to start running the task.
  #    repetitionInterval (string) - For daily repetition - Specifies how often to re-run the task after the atDateTime occurs.  Specify as a Timespan.
  #    repetitionDuration (string) - For daily repetition - Specifies how long to repeat the task executions for.  Specify as a Timespan.  Default is [Timespan]::MaxValue (forever)
  #    daysOfTheWeek (Array[string]) - For weekly repetition - Specifies the days of the week to run the task.  Specify an array of Monday | Tuesday | Wednesday | Thursday | Friday | Saturday | Sunday
  #    weeksInterval (Integer) - For weekly repetition - Specifies whether to run the schedule every week (value 1) or every n weeks (value n).  Default is every 1 week.
%>
$acts = @();
<% $actions.each | Hash $act | { -%>
$arg = @();
<%  if ( $act['isPowerShell'] )
{
  $cmd = "powershell.exe"
-%>
$arg += "-noprofile"
$arg += "-command `"<%= regsubst($act['command'],'\"', '\\\`"', 'GI') -%>`""
<% }
else
{
  $cmd = $act['command']
  if ( $act['arguments'] and is_array($act['arguments']) )
  {
    $act['arguments'].each | String $ar |
    { -%>
$arg += "<%= $ar -%>";
<%
    }
  }
  else
  { -%>
$arg += "<%= $act['arguments'] -%>"
<%}
}
if ( $act['workingDirectory'] )
{
  $wd = "-WorkingDirectory \"${act['workingDirectory']}\" "
}
else
{
  $wd = ""
} -%>
$params = @{}
if ( $arg )
{
  $params.Add("Argument", ($arg -join " "))
}

$acts += New-ScheduledTaskAction <%= $wd -%>-Execute "<%= $cmd -%>" @params
<% } -%>

$params = @{};
$trigs = @();
<% $triggers.each | Hash $trig |
{
  if ( $trig['weeksInterval'] or $trig['daysOfTheWeek'] )
  {
    #  Weekly Trigger:
    if ( $trig['weeksInterval'] )
    {
      $weeksInterval = $trig['weeksInterval']
    }
    else
    {
      $weeksInterval = 1
    }
-%>
$trigs += New-ScheduledTaskTrigger -Weekly -At "<%= $trig['atDateTime'] -%>" -WeeksInterval <%= $weeksInterval %> -DaysOfWeek <%= $trig['daysOfTheWeek'].join(",") %>;
<%
  }
  else
  {
    if ( $trig['repetitionDuration'] )
    {
      $repDuration = "<%= $trig['repetitionDuration'] -%>"
    }
    else
    {
      $repDuration = "([TimeSpan]::MaxValue)"
    }
#  Daily Trigger:
-%>
$trigs += New-ScheduledTaskTrigger -Once -At "<%= $trig['atDateTime'] -%>" -RepetitionInterval "<%= $trig['repetitionInterval'] -%>" -RepetitionDuration <%= $repDuration -%>;
<%
  }
}
-%>
if ( $trigs )
{
  $params.Add("Trigger", $trigs);
}

<% if ( $path == "" )
{
  $taskPath = "\\"
}
else
{
  $taskPath = $path
}
-%>
$sett = New-ScheduledTaskSettingsSet -ExecutionTimeLimit "<%= $executionTimeLimit -%>" -RunOnlyIfIdle:$false -DontStopOnIdleEnd;
$task = Register-ScheduledTask -TaskName "<%= $name -%>" -TaskPath "<%= $taskPath -%>" -Action $acts -Force -User "<%= $userName -%>" -Settings $sett<% if ( $password != "" ) { %> -Password "<%= $password -%>"<% } %> -RunLevel Highest @params;
<% if ( $deployEnabled == false ) { -%>
$task = $task | Disable-ScheduledTask;
<% } -%>

You can get both in my PuppetResources GitHub repo here.

Here is an example of a sample Scheduled Task:

mymodule::scheduled_task::windows_scheduled_task { 'Sample Scheduled Task':
    userName          =>  'MyTaskUserName',
    password          =>  'MyTaskPassword',
    deployEnabled     =>  true,
    description       => 'This task does some stuff.',
    actions           => [{
      command             => "c:\\scripts\\test-powershellscript.ps1",
      isPowerShell        => true
    }],
    triggers              => [{
      atDateTime          => "9/1/2016 11:00 PM",
      weeksInterval       => 1,
      daysOfTheWeek       => ["Monday","Tuesday","Wednesday","Thursday","Friday"]
    }],
  }

Enjoy!

Adding SQL Server Agent Jobs using Puppet

I find Puppet Enterprise to be very useful for configuring our many SQL Servers.  It does a nice job of setting up the SQL Instance and doing some base configuration.  There were a few things I wanted to add that it didn’t do out of the box that I thought I’d share.  One need I had was there was a set of specific SQL Agent jobs that I deployed out to our servers that I wanted Puppet to lay down for me.  I was able to build a pseudo-resource using the PE sqlserver forge module and some T-SQL.  I call it a pseudo-resource because it’s not a real resource in Puppet (with all the backing Ruby classes), but it behaves very much like a resource.

To do this, I needed the puppetlabs/sqlserver module and I had to create two files in my Puppet code repository.

NOTE: You must have Puppet Enterprise to use the puppetlabs/sqlserver module!

The first file I had to create was a T-SQL template that would generate the code needed to add the SQL Agent job.  This template is not 100% fully-featured, and a lot  more variables can be added to fully flesh out all of its options, but this is a very solid start.  I named this file sql_agent_job.epp and dropped it in my “templates” folder.  It looks like this:

<%- | String $name, String $description, String $notifyOperator = "", Array[Hash] $steps, Any $schedules = undef | -%>
BEGIN TRANSACTION
BEGIN TRY
  DECLARE @ReturnCode INT
SELECT @ReturnCode = 0
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)
BEGIN
EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N'JOB', @type=N'LOCAL', @name=N'[Uncategorized (Local)]';
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
END;

This isn’t the complete file (see my link below for the entire thing), but it gives you the idea. The template gets called by the .pp class file, which is below.

The second file is the actual Puppet class (.pp extension).  This is the file that implements the template and makes the whole thing “resource-like”.  This file belongs in your “manifests” folder in your repository or module:

class sqlserver::sql_agent_job()
{
  define sql_agent_job
  (
    String $sqlInstanceName,
    String $description,
    String $notifyOperator,
    Array[Hash] $steps,
    Any $schedules = undef
  )
  {
   sqlserver_tsql { "${title}_${sqlInstanceName}_sql_agent_job" :
      instance    => $sqlInstanceName,
      command     => epp("sqlserver/sql_add_job.epp", {
                        name            => $name,
                        description     => $description,
                        notifyOperator  => $notifyOperator,
                        steps           => $steps,
                        schedules       => $schedules
                      }),
      onlyif      => "IF NOT EXISTS ( SELECT * FROM msdb.dbo.sysjobs WHERE name = '${name}' ) BEGIN
                        THROW 51000, '${name} job not present.', 10;
                    END;"
    }
  }
}

Note:  You have to make sure the call to epp(…) above points to the path your template is at.  In the example above, I presume it’s in the same module in the templates/sqlserver folder.  Your folder structure should look roughly like this:

manifests/
     /sqlserver/sql_add_job.pp
templates/
     /sqlserver/sql_add_job.epp

This is the resource you will actually drop in you profile classes to add jobs to servers. The input parameters are as follows:

#  PARAMETERS:
    # name                          => (namevar) Specifies the name of the agent job.   - https://msdn.microsoft.com/en-us/library/ms182079.aspx
    # sqlInstanceName               => Specifies the SQL Server instance.
    # description                   => Specifies the description on the job.
    # notifyOperator                => Specifies the name of the job operator to notify.
    # steps                         => An array of hashes specifying the job steps:
    #   name                          => String - The name of the job step
    #   command                       => String - The T-SQL to execute
    #   database                      => String - The name of the database to execute against if the subsystem is TSQL.
    #   onSuccess                     => Integer - 3(next)|2(quitfail)|1(quitsuccess)|4(gotostep), default is 1
    #   onFail                        => Integer - 3(next)|2(quitfail)|1(quitsuccess)|4(gotostep), default is 2
    #   onSuccessStepId               => Integer - The stepid to go to on success
    #   onFailStepId                  => Integer - The stepid to to go in failure
    #   subsystem                     => String - Specify either "TSQL" or "CmdExec".  Default is TSQL.
    #   outputFileName                => String - Specify the path to the file to write the output to.
    # schedules                     => (optional) A hash specifying a job schedule.     - https://msdn.microsoft.com/en-us/library/ms366342.aspx
    #   frequencyType                 => Integer - 1(once)|4(daily)|8(weekly)|16(monthly), default 4
    #   frequencyInterval             => Integer - (once) - not used | (daily) - every frequencyInterval days | (weekly) - frequencyinterval determines day of wek | (monthly) - determines day of the month
    #   frequencySubdayType           => Integer - 1(attime)|4(minutes)|8(hours), default 1
    #   frequencySubdayInterval       => Integer - number of minutes/hours
    #   frequencyRecurrenceFactor     => Integer - Number of weeks/months between exectutions.  Nonzero value required if frequencytype is 8|16|32 (not used otherwise).  Default is 0.
    #   activeStartTime               => "HHMMSS, default 0",
    #   activeEndTime                 => "HHMMSS, default 235959"

You’ll probably notice the parameter names and values are pretty much identical to the input parameters for sp_add_job, sp_add_jobstep and sp_add_jobschedule stored procedures. A trick I use when I want to take a job and add it to Puppet is to add the job to SQL Server first, set it up the way I want, then script the job out. The parameters in the T-SQL script will pretty much translate to the sql_agent_job resource.

Here is an example of a profile with the sql_agent_job resource in use:

profile::sqlserver::component::sql_agent_job::sql_agent_job { "${name}_my_agent_job":
      name                  => "My SQL Agent Job",
      sqlInstanceName       => $name,
      description           => 'This is my SQL Agent Job being deploying wiht Puppet.',
      notifyOperator        => 'SQLTeam',
      steps                 => [{
                                name      => 'Execute Test Script',
                                database  => 'master',
                                subsystem => 'TSQL',
                                command   => "SELECT 'test data here'",
                                onSuccess => 1,
                                onFail    => 2
                              }],
      schedules             => {
                                frequencyType           => 4,
                                frequencyInterval       => 1,
                                frequencySubdayType     => 4,
                                frequencySubdayInterval => 30,
                                activeStartTime         => 000000,
                                activeEndTime           => 235959
                              },
    }
The full versions of these files can be found in my GitHub repository here:
Enjoy!

Creating a Directory Tree in Puppet

As you can probably tell from the flurry of blog posts I’ve made concerning Puppet, I’m going through the process of learning and setting up Puppet Enterprise.

One thing that irked me early on is the inability of the file resource to create a directory if the parent directory does not exist.  For example:

file { 'mydirectory' :
  ensure         => 'directory',
  path           => 'c:/parentdir/childdir'
}

If c:\parentdir does not exist, this fails.

Error: Cannot create C:/parentdir/childdir; parent directory C:/parentdir does not exist
Error: /Stage[main]/Profile::Myclass/File[mydirectory]/ensure: change from absent to directory failed: Cannot create C:/parentdir/childdir; parent directory C:/parentdir does not exist

You can alternately specify it like this to get it to work:

file { ['c:/parentdir', 'c:/parentdir/childdir'] :
  ensure         => 'directory'
}

This works, and for the most part is OK.  In my case though, I have the user provide the directory name through a class parameter:

class myclass ([String] $mydirectory)
{
  file { 'mydirectory' :
   ensure         => 'directory',
   path           => $mydirectory
  }
}

If the user specifies c:/parentdir/childdir, and c:/parentdir does not exist, it explodes.  I could adjust the code and advise my users to pass in arrays of strings representing the directories, but that’s not very clear or clean.

Fortunately, Puppet supports PowerShell and PowerShell is awesome:

class myclass (String $directory)
{
   exec { 'mydirectory' :
     command => "c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe -noprofile -noninteractive -command \"New-Item -ItemType Directory -Path \"$directory\" \"",
     onlyif  => "c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe -noprofile -noninteractive -command \"if (Test-Path -Path \"$directory\" -PathType Container) { exit 99 }\""
   } 
}

This code block creates the entire directory tree without issue.  The onlyif parameter ensures that the exec block is not fired off if the directory already exists.

Enjoy!

Puppet Agent on Windows – Module not found

The first step I take when developing a new Puppet configuration is to install the Puppet Agent on a standalone test Windows server and build the configuration files locally there.  I then use the puppet apply utility to test it and make sure it works.  This saves a lot of time since it avoids having to do hundreds of pushes and merge requests to our source control system as I tweak and debug the config files to get them working the way I want.

I had some challenges getting this setup initially though.  I attempted to follow advice given to me by my Puppet SE, and researched and tried to implement Roles and Profiles as a means of developing layered configurations.  It make sense to do it this way, especially as your configuration base grows, but it requires a bit of know-how to get working properly.  One of the major stumbling blocks I hit was getting Puppet to recognize classes located in a non-standard directory.  The normal, standard directory structure looks like this:

C:/ProgramData/PuppetLabs/code
    /modules # This is the default $basemodulepath
    /environments/production
        /manifests  # This is where it expects site.pp and any other code you write
        /modules     # Your downloaded and custom modules can also go here

In my case, I wanted to create a “site” directory in which I stored my role and profile configurations per the design above.  My structure looked like this:

c:/programdata/puppetlabs/code/environments/production
    /site
        /profile/manifests
        /role/manifests

Since this was not in the default $basemodulepath directory  or the environment module directory I’d receive an error stating the class could not be found:

ModuleNotFound

This is easy enough to figure out.  Puppet is highly configurable, and as such you can add additional directories to the list of those it looks in for classes it can use.  In my case, I simply edited the environment.conf file found at C:\ProgramData\PuppetLabs\code\environments\production\environment.conf  and commented-in the modulepath variable.  I then added my site folder.  I changed this line:

# modulepath = ./modules:$basemodulepath

To look like this:

modulepath = modules:site:$basemodulepath

However, I found I would still receive the same error as before.  A clue for me was when I ran the puppet config print modulepath command:

PS C:\ProgramData\PuppetLabs\code\environments\production\manifests&amp;gt; (puppet config print modulepath) -split &quot;;&quot;

You can see it lists the following paths:

C:/ProgramData/PuppetLabs/code/environments/production/modules
C:/ProgramData/PuppetLabs/code/modules
C:/opt/puppetlabs/puppet/modules

None of these were my site directory.  It’s as if the change I made to environment.conf was simply ignored.

Essentially, I found it was.  Even though the inital example show in the environment.conf files shows this (note the colon delimiter):

# modulepath = ./modules:$basemodulepath

I found the Windows Agent uses semicolons, not colons as a delimiter for multiple paths.  This is kind of documented here.

Path Separator

Make sure to use a semi-colon (;) as the path separator on Windows, e.g., modulepath=path1;path2

Plain enough, but this document does not reference the environment.conf file specifically, or even the Puppet Agent (this seems to be just a general Windows thing).  Also, the Puppet Agent installer lays down the environment.conf file with the colons in place, so it’s very misleading.

In any case, I found that if I changed the file to look like this, everything worked:

modlepath = modules;site;$basemodulepath

Running puppet config print modulepath confirmed my site path now shows up:

C:/ProgramData/PuppetLabs/code/environments/production/modules
C:/ProgramData/PuppetLabs/code/environments/production/site
C:/ProgramData/PuppetLabs/code/modules
C:/opt/puppetlabs/puppet/modules

So, in summary, if you are using any non-standard paths for your modules or classes on a Windows machine, make sure and use semicolons to delimit multiple paths for the modulepath setting, rather than the default colon.

Confusing, but easy to fix fortunately.

hiera-eyaml and Puppet Enterpise – Command not found?

I’m in the process of evaluating Puppet Enterprise as a configuration management solution for my company.  A glaring issue I hit early on is figuring out how to secure credentials that are fed to the various Puppet configurations.  By default, there is no way I’m aware of to obfuscate credentials in the configuration areas (including hiera files and class parameters in the GUI).  This is an issue as I can’t expose certain credentials to the general public.

Fortunately, hiera-eyaml was easy-to-find and does the trick.  There’s a lot of good documentation out there on how to set this up, and I won’t belabor that point, but to a Puppet noob the documentation makes a lot of assumptions.  The main assumption I want to clear up is how to get it up-and-running on your Puppet Master server using the eyaml utility from the CLI.

 

The GitHub document appears easy-to-follow:

https://github.com/TomPoulton/hiera-eyaml

The first step makes perfect sense, and worked without issue:

puppetserver gem install hiera-eyaml

The problem was after this.  I could not call the eyaml executable.  If I typed “eyaml –help”, “eyaml encrypt” or any valid variation of the command I received a “eyaml:  command not found” error.

Long story short, the issue is the Puppet master server does not have the ruby interpreter setup by default for command line use. The command above does make hiera-eyaml available for the Puppet software’s use, and you can go about configuring  it and using as stated in the GitHub readme for Puppet, but the eyaml calls will not work for you on the CLI.  The assumption they make is that you know to install the Ruby interpreter and gem separately for CLI usage.  To do this, do the following from the Puppet master (or any Linux station):

apt-get install ruby
gem install hiera-eyaml

Now the ruby interpreter is available for use to you on the CLI and you can call the eyaml executable as noted in the GitHub article.

I’m sure this is obvious to a Ruby/Linux expert, but it took me about 3/4 of a day to figure this out, so hopefully this helps save someone some time down the road.

Installing the ASUS PCE-N53 WIFI Card in a Machine Running SteamOS

Recently, I took the dive into building a custom SteamOS box.  So far, it’s been a fun exercise.  SteamOS basically is the Big Picture Mode for the Steam Client running on top of a debian-based Linux platform.  There are plenty of articles that go into the installation of it, so I won’t bother with that.

The main issue I see is that there is not a whole lot of support for various hardware.  In my case, I had a few-year-old ASUS PCE-N53 WIFI card that I wanted to install and use.   However, the SteamOS did not detect the card at all, thus began an adventure in research into getting it working.

The below procedure walks you through installing the drivers for the card.  To do this, you have to get into the desktop mode on the SteamOS and be familiar with the terminal.

I’m using SteamOS version 1.0 Update 161.

Installing WIFI PCE-N53 Card

Get the driver patch

ASUS has a Linux driver, but the driver is not compatible with Linux kernels under version 3.0.  There is an unsupported patch out there, it worked for me, but use at your own risk!

  1. Get the v3 patch:

http://gridlox.net/diff/rt5592sta_fix_64bit_3.8.patch

2.  Put the patch on the SteamBox in the /home/desktop/Downloads folder

In my case, I used WinSCP from a Windows Desktop.  You can use wget from the SteamOS console.

Get the ASUS Driver

  1. Download the driver from the ASUS site:

http://support.asus.com/download.aspx?SLanguage=en&p=11&m=PCE-N53&hashedid=F3giHhIS4hp6aWBW

Drop this in the /home/desktop/Downloads folder

Patch and Compile the Driver

To perform these steps, you need to be logged into the Linux terminal on the steamOS box.  You can do this in two ways:

  1. Log into the desktop in the SteamOS and launch the terminal
  2. Enable SSH on the SteamOS box and SSH in
  3. Log onto the SteamOS terminal

Go to the SteamOS desktop, then to the terminal

Or, alternately, SSH to the desktop shell

Install 7-zip and the make utility

sudo apt-get install p7zip-full
sudo apt-get install build-essential
  1. Unpack the driver
7z x Linux_PCE_N53_1008.zip
cd Linux
7z x DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326.tar.bz2
7z x DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326.tar

2.  Patch the driver

cd DPO_GPL_RT5592STA_LinuxSTA_v2.6.0.0_20120326/
patch -p1 < ~/Downloads/rt5592sta_fix_64bit_3.8.patch

3.  Compile

sudo make

4.  Install the compiled driver

sudo make install

5.  Rescan for the new card

sudo modprobe rt5592sta

Viola!  Now the card works. I can configure it from the SteamOS desktop network configuration utility.  Hope this is of benefit to someone out there.

Updated FC Drivers on ESXi

I had an issue where I had to change out the driver ESXi uses for it’s QLogic fiber channel hbas.  Below is the procedure I used to make the change from the qlnativefc and qla-2xxx driver sets.  I was running ESXi 5.5, but the procedure for other versions is likely similar.

Updating the Qlogic Driver on an ESXi Host

ESXi hosts with Qlogic adapters use the qlnativefc driver out of the box.

On some servers, including Dell PowerEdge R7xx servers, the adapter does not support NPIV without a different driver.  Follow the procedure below to update the driver

Update the Qlogic Driver to an NPIV-enabled Driver

      1. Download the qlogic driver for Vmware.

VMware ESXi5.x FC-FCoE Driver for QLogic and OEM-branded Fibre Channel and Converged Network Adapter

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI5X-QLOGIC-QLA2XXX-9345320-1VMW&productId=285

Example:  qla2xxx-934.5.38.0-1872208.zip

      1. Extract the zip file and find the offine_bundle.zip file if it’s not already extracted.
      2. SCP the offline bundle file to a datastore on the ESXi host that all of the other hosts can see.

Example:   /vmfs/volumes/vmware_host_swap_03/qla2xxx-934.5.38.0-offline_bundle-1872208.zip

For each host:

      1. Log onto the VM host
      2. Put the host in maintenance mode.
      3. Do the following:

esxcli software vib install -d /vmfs/volumes/vmware_host_swap_01/qla2xxx-934.5.38.0-offline_bundle-1872208.zip

Remove the old driver

esxcli software vib remove -n qlnativefc

      1. Reboot the host
      2. Verify after the host is back up:

esxcli software vib list | grep -I qla2xxx

Rolling Back the Driver Upgrade

Vmware QLNativeFC Driver:

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-QLOGIC-QLNATIVEFC-11390-1&productId=353&download=true&fileId=838e130270df93ad2aca6bd64be27f06&secureParam=&uuId=08b248a1-29dc-4ce5-8da3-6353d2997bce&downloadType=

Example:  VMW-ESX-5.5.0-qlnativefc-1.1.39.0-2243137.zip

      1. Extract the zip file and find the offine_bundle.zip file if it’s not already extracted.
      2. SCP the offline bundle file to a datastore on the ESXi host that all of the other hosts can see.

Example:   /vmfs/volumes/ah-3270-2:vmware_host_swap_03/VMW-ESX-5.5.0-qlnativefc-1.1.39.0-offline_bundle-2243137.zip

For each host:

      1. Log onto the VM host
      2. Put the host in maintenance mode.
      3. Do the following:

esxcli software vib install -d  /vmfs/volumes/ah-3270-2:vmware_host_swap_03/VMW-ESX-5.5.0-qlnativefc-1.1.39.0-offline_bundle-2243137.zip

      1. Remove the old driver

esxcli software vib remove -n scsi-qla2xxx

      1. Reboot the host
      2. Verify after the host is back up:

esxcli software vib list | grep -i ql

Setting up and troubleshooting NPIV on VSphere

I recently setup NPIV for several of my VMs in my production environment.  Let’s just say it was an adventure.  I wanted to detail my experience below, including some step-by-step instructions for setting it up and some advanced troubleshooting I had to do to resolve my issues.

My environment consists of VSphere ESXi 5.5 hosts with Brocade DCX backbones.  Though this setup revolves around my particular setup, the steps here should work with most setups.

Definition

NPIV is the process where a Fiber Channel switch port can advertise multiple Port WWPNs for the same fiber channel connection.  This allows you to have a single switch port represent the WWPN of an ESXI host port and also a virtualized WWPN assigned directly to a VM.  Cisco UCS also does this, with the physical ports of the edge device representing the different virtual WWPNs of the different virtual HBAs on the service profiles provisioned on the Cisco hardware.

This is useful when you want to identify a particular VMs fiber channel traffic by identifying it by its own unique WWPN.  Also, with Hitachi Server Priority Manager (QOS), you have to assign QOS limits by selecting a WWPN to assign a quota to.  Without NPIV, the best you could do is limit the ESXi server HBA to a limit, thereby limiting ALL vms on the ESXi host.

We use VirtualWisdom, which is a Fiber Channel performance and health monitoring software package made by VirtualInstruments.  With NPIV, we can now trace IOPS, MB/s and latency right to the VM as opposed to ESXi HBA port.

Requirements

NPIV has some strict requirements, and failing to adhere to them can give you headaches.

  • Your switch has to support NPIV (Most modern switches, including DCX backbones do)
  • NPIV needs to be enabled on your switch ports (Enabled by default on DCX backbones)
  • ESXi needs to support NPIV (versions past 4.0 do for sure.  I see references to it back to v3.5)
  • NPIV can only be used in conjunction with RDM devices.  .vmdk disks from a datastore are still identified by the ESXi host’s wwpns and cannot use NPIV.
  • The driver on the ESXi host has to support NPIV (Esxi qlnative drivers on 1.1.39 is confirmed to work)

Zoning Notice

You have to make sure the NPIV wwpns are zoned to see ALL STORAGE the cluster hosts can see, even if the VM does not attach to the storage.

Example Configuration

Enabling NPIV

VM Name:  npiv_test_01

1) First, shut down the VM.
2) Log into the vSphere web client and find the VM in your inventory.
3) Edit the settings of the VM.
4) Go to the VM Options Tab, then to “Fibre Channel NPIV”
5)  Uncheck the “Temporarily disable NPIV for this Virtual Machine”
6)  Select “Generate new WWNs”
7)  Select the appropriate number of wwpns vs wwnns.
For one FC fabric, one WWNN and one WWPN should work.  For two, I’d do one WWNN and two WWPNs to simulate a dual-port HBA,each connected to one fabric for redundancy.
8)  Click OK and exit.
9)  Edit settings again and note the WWPN(s) generated for your VM. You will need these later.  For purposes of this exercise, we will assume the following:

NPIV1

Node WWN:
28:2c:00:0c:29:00:00:30

Port WWNs:
28:2c:00:0c:29:00:00:31, 28:2c:00:0c:29:00:00:32

Connect your RDM to it at this time.  When we are done, the VM should be accessing your RDM via your NPIV-generated wwpns.  If this fails for some reason, it will fall back on the wwpns of the ESXi host HBAs.  Remember, it will ONLY see RDMs this way, not .vmdk disks sitting in a datastore.  This ALWAYS go through the ESXi host wwpns.

Now, before you power up the VM, you have to set the zoning and LUN masking or you will have issues while booting or vMotioning it.

FC Fabric Zoning and Configuration

Verify NPIV is enabled

Run the following command from a Brocade backbone

portcfgshow 1/26
(Where 1/26 is the slot/port of your ESXi host HBA port)

NPIV2

Verify the NPIV Capability setting is set to ON (which should be the default for Brocade)

Zone the Fabric(s)

1)  Create a FC alias on each fabric.  Each one will contain one of the two NPIV wwpns you generated above.  On a Brocade backbone:

Fabric 1
alicreate npiv_test_01_hba0,28:2c:00:0c:29:00:00:31

Fabric 2
alicreate npiv_test_01_hba1,28:2c:00:0c:29:00:00:32

2) Create your zones.  Recall that you *must* zone the NPIV wwpn to ALL storage systems your ESXi cluster hosts can see, even if the VM will never see or use the storage!

Fabric 1
zonecreate npiv_test_01_hba0_to_storage_array_01_1a,”ah_npiv_test_01_hba0; storage_array_01_1a”
cfgadd “mycfg”,”npiv_test_01_hba0_to_storage_array_01_1a”
.. add any additional zones for other storage systems…

Fabric 2
zonecreate npiv_test_01_hba1_to_storage_array_01_1b,”ah_npiv_test_01_hba1; storage_array_01_1b”
cfgadd “mycfg”,”npiv_test_01_hba1_to_storage_array_01_1b”
.. add any additional zones for other storage systems…

3) Save and commit your config.

cfgsave
cfgenable mycfg

Mask Your Luns

Now you have to mask your LUNs.  It’s best to add the wwpns for your NPIV VM to the same host or initiator group as your ESXi hosts.  In my environment, we create one group for all wwpns for all ESXI HBAs in the given cluster.  Here is an example:

esxi_host_01
HBA1 (goes to fabric 1):  59:00:00:00:00:00:00:01
HBA2 (goes to fabric 2):  59:00:00:00:00:00:00:02

esxi_host_02
HBA1 (goes to fabric 1):  59:00:00:00:00:00:00:03
HBA2 (goes to fabric 2):  59:00:00:00:00:00:00:04

Your existing host group will look like this:
esxi_cluster_01_target_1a:  59:00:00:00:00:00:00:01,59:00:00:00:00:00:00:03
esxi_cluster_01_target_1b:  59:00:00:00:00:00:00:02,59:00:00:00:00:00:00:04

So in this case, you’d add your fabric1 NPIV address to esxi_cluster_01_target_1a and your fabric2 wwpn to esxi-cluster_01_target_1b:

esxi_cluster_01_target_1a:
59:00:00:00:00:00:00:01,59:00:00:00:00:00:00:03,28:2c:00:0c:29:00:00:31
esxi_cluster_01_target_1b:
59:00:00:00:00:00:00:02,59:00:00:00:00:00:00:04,28:2c:00:0c:29:00:00:32

There is some flexibility at this point.  Different storage systems mask differently, and I don’t think NPIV will freak out if it can’t see the LUN.  In this case, it just won’t work and will fall back to the ESXi host wwpn.

Finishing the Configuration

Now you can power up the VM.  I’d advise you to watch the vmkernel.log as the VM boots so you can see if it worked.  Log into the ESXi host using SSH and use the following command:

tail -f /var/log/vmkernel.log

Messages like below indicate success.

2015-03-20T23:13:32.838Z cpu30:61525)ScsiNpiv: 1701: Physical Path : adapter=vmhba3, channel=0, target=3, lun=2

2015-03-20T23:13:32.838Z cpu30:61525)ScsiNpiv: 1160: NPIV vport rescan complete, [2:2] (0x41092fd6c1c0) [0x41092fd42440] vport exists

2015-03-20T23:13:32.838Z cpu30:61525)ScsiNpiv: 1848: Vport Create status for world:61526 num_wwpn=2, num_vports=4, paths=4, errors=0

If it posts within a minute or two, you should be OK.  If it takes a lot longer, you probably have an issue.

You can verify it worked by using this command:

cat /var/log/vmkernel.log | grep -i num_vport

…num_wwpn=2, num_vports=4, paths=4, errors=0

If you see errors=0, you should be OK.  You should see a vPort per path to the LUN, and no errors.

Verification

There are a number of ways to verify NPIV is working.  I used my VirtualInstruments software to verify it could see the NPIV wwpn.  I created a Host entity and added the two NPIV wwpns as HBA_port objects to it, then graphed it:

NPIV3

This tells me without a doubt an external system can see the traffic from the VM through the NPIV port.

Troubleshooting

I had a number of issues getting this working, but I was able to figure them all out with some help from VMware tech support.  I’ll detail them below:

Error code bad00030 

Translated, this means VMK_NOT_FOUND.  Basically, it means no LUN paths could be found via the NPIV wwpns.  In my case, this was due to a bad driver.  On my Dell PowerEdge 710/720 servers, I had to install the qla-2xxx driver as opposed to the qlnativefc driver to get NPIV to work. I have a separate post forthcoming that details this procedure.

Error code bad0040

Like bad00030, this indicates NPIV can’t reach the LUN via your npiv wwpns.  I watched the log and noticed it would scan each path up to 5 times for each HBA, throwing this error each time until it timed out.  This would take 15 or more minutes, then the VM would eventually post and come up.  If you tried to vMotion it, it would try to enable NPIV on the host it was moving to, time out, then try and fail back to the original host and go through the process again and time out all over again.  This would essentially hang the VM for up to 30 minutes before VSphere would drop the VM on its’ face and power it off.  The VMKernel messages look like this:

2015-03-13T18:15:35.770Z cpu2:330868)WARNING: ScsiPsaDriver: 1272: Failed adapter create path; vport:vmhba64 with error: bad0040
2015-03-13T18:15:37.772Z cpu2:330868)ScsiNpiv: 1152: NPIV vport rescan complete, [3:1] (0x41092fd6ad40) [0x41092fd3fe40] status=0xbad0040
2015-03-13T18:15:37.772Z cpu2:330868)WARNING: ScsiNpiv: 1788: Failed to Create vport for world 330869, vmhba2, rescan failed, status=bad0001
2015-03-13T18:15:37.773Z cpu24:33516)ScsiAdapter: 2806: Unregistering adapter vmhba64

Basically, you can see it creates vmhba64 (This is your virtual NPIV adapter.  The number after vmhba varies).  It tries to scan for your RDM LUN (Target 3, LUN id 1) and fails.  After several retries, it gives up and deletes the vmhba.

This was an issue caused by the fact that I did not zone my npiv wwpns to ALL storage systems my ESXi hosts were connected to.  We had zoned our ESXi cluster to a NetApp storage system, but at some point disconnected all of the LUNs from the ESXi cluster.  Even though the ESXI hosts didn’t need the zones, and the VM certainly did not have any LUNs from that storage system attached, not zoning the NPIV wwpns to that storage system broke NPIV.

How I figured this out was I turned on verbose logging for the qlnativefc driver using this command:

esxcfg-module -s ‘ql2xextended_error_logging=1’ qlnativefc

Then reboot the host for it to take effect.

Disable it later with:

esxcfg-module -s ‘ql2xextended_error_logging=0’ qlnativefc

Then reboot again.

After that, I ran the following:

/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -a | less

There is a section in this output for each vmhba, and it lists the target ports the physical ESXi hba sees and the target numbers it assigned to them:

vmhba2

FC Target-Port List:

scsi-qla0-target-0=500000000bbb1155:01ac00:0:Online;
scsi-qla0-target-1=500000000bbb1166:015400:1:Online;
scsi-qla0-target-2=500000000aaa1102:01d800:2:Online;
scsi-qla0-target-3=500000000aaa1103:019800:3:Online;

You can see the wwpns of your storage system target ports, followed by the internal ID ESXi assigns to it (01ac00), and its status.  In my case, the first two target ports were from the storage system I didn’t realize was connected.

With the verbose logging turned on, I ran tail -f /var/log/vmkernel.log while the VM booted up and noted the following:

2015-05-04T19:44:08.112Z cpu19:33504)qlnativefc: vmhba64(4:0.0): GID_PT entry – nn 282c000c29000030 pn 282c000c29000031 portid=012b01.
2015-05-04T19:44:08.113Z cpu19:33504)qlnativefc: vmhba64(4:0.0): GID_PT entry – nn 500000000aaa1101 pn 500000000aaa1102 portid=012900.
2015-05-04T19:44:08.113Z cpu19:33504)qlnativefc: vmhba64(4:0.0): GID_PT entry – nn 500000000aaa1101 pn 500000000aaa1103 portid=016900.
2015-05-04T19:44:08.132Z cpu19:33504)qlnativefc: vmhba64(4:0.0): device wrap (016900)
2015-05-04T19:44:08.132Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Trying Fabric Login w/loop id 0x0008 for port 012900.
2015-05-04T19:44:08.132Z cpu19:33504)qlnativefc: vmhba64(4:0.0): qla24xx_login_fabric(6): failed to complete IOCB — completion status (31) ioparam=1b/fffc01.
2015-05-04T19:44:08.132Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Trying Fabric Login w/loop id 0x0009 for port 012900.
2015-05-04T19:44:08.135Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Fabric Login successful w/loop id 0x0009 for port 012900.
2015-05-04T19:44:08.135Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Assigning new target ID 0 to fcport 0x410a6c407e00
2015-05-04T19:44:08.135Z cpu19:33504)qlnativefc: vmhba64(4:0.0): fcport 500000000aaa1102 (targetId = 0) ONLINE
2015-05-04T19:44:08.135Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Trying Fabric Login w/loop id 0x000a for port 016900.
2015-05-04T19:44:08.138Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Fabric Login successful w/loop id 0x000a for port 016900.
2015-05-04T19:44:08.138Z cpu19:33504)qlnativefc: vmhba64(4:0.0): Assigning new target ID 1 to fcport 0x410a6c410500
2015-05-04T19:44:08.138Z cpu19:33504)qlnativefc: vmhba64(4:0.0): fcport 500000000aaa1103 (targetId = 1) ONLINE
2015-05-04T19:44:08.138Z cpu19:33504)qlnativefc: vmhba64(4:0.0): LOOP READY

The lines that start with GID_PT show the target ports that the NPIV wwpn sees (this is a separate discovery process than the ESXi HBA).  You notice it only sees two of the target ports.

If we concentrate on the first target port it sees, it’s labeled ID 12900.  Later, you see it assigns target id 0 to the target port.  However, from the above steps, you can see using the vmkmgmt_keyval command the ESXi HBA sees wwpn 500000000aaa1102 as Target ID 2.  They don’t match, and thus NPIV fails to work with the bad0040 error.

Once I saw this, I traced the extra wwpns back to the storage system I realized I was connected to and remove it from the zoning of the ESXi HBA ports.  After a rescan of the storage on the ESXi cluster and a reboot of the hosts to be safe, I booted the VM up again and viola!

Summary

Make sure you are meeting all of the requirements for NPIV.  VERIFY you are zoning your NPIV wwpns to ALL STORAGE SYSTEMS the cluster can see, even if it will not use any storage from the storage systems.  Be aware that if you connect any new storage systems to a cluster with one or more NPIV-enabled  VMs, you will need to add the new zones so the NPIV wwpns can see the new storage system target ports, or it will probably start failing.

Powershell Module for Logging to a File

Despite out best efforts as coders, automated processes sometimes fail.  One of the principle ways to troubleshoot such processes is to log data to a file so you can follow what happened after-the-fact.  I have a TON of scripts that have to do this, so it made sense to cobble together some functions that make doing this easier.

To this end, I’ve written a script module called bsti.logging.  It features the following functions:

logging_functions

Once you import the module, you call New-LogFile to setup a new file to write to.  You can specify options to append the weekday to the file or a more specific timestamp (e.g. MyLog_mmddyyyyhhmmss.log) to the log file.  For timestamped log files, you can also setup retention so old log files get automatically deleted after a period of time or after so many accumulate.

logging_new_logfile

I have three basic types of ways to handle log file naming that you need to be clear on to get good use out of the module:

1) Standard – The log file path you pass is will be unchanged by the function.  The purging parameters are ignored, you must use -Append or it will be overwritten if it exists already.

2) Circular – The log file will have _weekday appended to the file name before the extension.  If you pass in C:\temp\log\MyLogFile.log for example, you get:
MyLogFile_Monday.log
MyLogFIle_Tuesday.log
etc…
When you call New-LogFile with circular naming and the *same* log file path again that same day, it will automatically be appended to.  When the next Monday rolls by, it *automatically* overwrites it.

This scheme is good if you call the script that writes to the log file frequently, don’t want to manage a large number of log files, and don’t need more than 7 days of log file history.

3) DateStamped – This appends a detailed datestamp to the log file name before the extension.  In the example above, you get:
MyLogFile_03292015200500.log (Assuming 3/29/2015 10:05 PM)
This means every time you call New-LogFile (Assuming you wait at least 1 second!), you get a unique log file.  Append is essentially ignored.
The PurgeAfter and KeepNumberOfFIles, if specified, will cause the New-LogFile function to call Remove-LogFiles automatically and keep your log files trimmed as you specify.  If you specify both PurgeAfter AND KeepNumberOfFiles, both thresholds are observed (meaning the file needs to be older than what you specified with PurgeAfter AND you have to have KeepNumberOfFiles remaining).

This scheme is good if you need a specified history of log files and want individual log files for each run of your process.  The automatic cleanup is a bonus.

Once you’ve setup your new log file, you call the following functions to write to it.  These functions also echo to the console, so you can replace any calls to Write-Host with these functions and get messages to your console and to a log file:

Write-Message
Write-Object
Write-Banner
Write-BlankLines

logging_write

Makes things pretty simple.

As with all of my modules and module functions, I heavily document them using Powershell comments-based help.  Just try:

Get-Help New-LogFile -Full

This module does depend on my bsti.conversion module, so if you use this module as a whole you need both modules.  I posted about that module here.

Here is a link to the new bsti.logging module.
Here is a link to the bsti.conversion module.

TimeSpan Conversion Function and Module

I have a ton of Powershell code I’ve written over the last 6 years or so that I’m in the process of cleaning up and looking to share.  I plan on re-presenting them as a set of scripts and script modules I’ll be featuring in a series of blog posts coming in the near future.

I’ll start simply.  I have a new script module called bsti.conversion that has a single function:  ConvertTo-TimeSpan

I always thought it was neat that you could type in 1kb and Powershell would convert that value to 1024.  You can also use mb (megabytes), tb (terabytes), and pb (petabytes).  I don’t see that eb (exabytes) works.  In any case, I always wished I could do the same with time units like this:

12m (minutes) or 24h (hours) or 7d (days)

The ConvertTo-TimeSpan function I’ve included in this module does just that.

What I use this for is I have functions and scripts I like to write that require the user to pass in an easy time unit value.

This functionality can also be achieved by Timespan conversion like so:

[Timespan](“1.00:00:00”)  # 1 day
[Timespan](“1.6:00:00”)  # 1 day, 6 hours
[Timespan](“1:22:00”)  # 1 hour, 22 minutes

The conversion function is a little less precise, but a bit more human-readable, which is important to me since most of my users are not .NET programmers and don’t understand the format of a timespan object right offhand.

The function in this module supports both formats:

bsti.conversion1

The module files can be downloaded here.
Once downloaded, extract the folder to the C:\windows\system32\windowspowershell\v1.0\modules directory.  The final structure should look like this:
C:\windows\system32\windowspowershell\v1.0\modules\bsti.conversion

Then just use the Import-Module bsti.conversion command as shown above.

Not bad for a start, hope you enjoy.

UPDATE:  I’m adding my stuff to GitHub.  Bear with me while I come up-to-speed on how to use it.  Find this module project here:

https://github.com/Roadkill4545/bsti.Powershell/tree/master/bsti.conversion