PowerShell – Page 4 – Donnie's Azure, Automation, Monitoring, and Management Blog

Powershell and Parsing html code in Core

When working on a PowerShell webservice, I came across an interesting problem that I think is only going to crop up more and more. This webservice takes a json payload, performs some simple manipulation on the data, kicks off some automation, and logs some events. Part of the payload is html code, however. Of course the json payload doesn’t care, but when it was coming into PowerShell it was being interpreted as a string. That meant that when you looked at the string, you literally saw html code:

<BODY><H3>OPEN Problem 256 in environment <I>EPG</I></H3>
                               <HR>
                               <B>1 impacted infrastructure component</B>
                               <HR>
                               <BR>
                               <DIV><SPAN>Process</SPAN><BR><B><SPAN style="FONT-SIZE: 120%; COLOR: #dc172a">bosh-dns-health</SPAN></B><BR>
                               <P style="MARGIN-LEFT: 1em"><B><SPAN style="FONT-SIZE: 110%">Network problem</SPAN></B><BR>Packet retransmission rate for process bosh-dns-health on host 
                               cloud_controller/<GUID> has increased to 18 %</P></DIV>
                               <HR>
                               Root cause
                               <HR>
                               
                               <DIV><SPAN>Process</SPAN><BR><B><SPAN style="FONT-SIZE: 120%; COLOR: #dc172a">bosh-dns-health</SPAN></B><BR>
                               <P style="MARGIN-LEFT: 1em"><B><SPAN style="FONT-SIZE: 110%">Network problem</SPAN></B><BR>Packet retransmission rate for process bosh-dns-health on host 
                               cloud_controller/<GUID> has increased to 18 %</P></DIV>
                               <HR>
                               
                               <P><A href="https://redacted.somewhere.com/e/<GUID>/#problems/problemdetails;pid=-<GUID>">Open in Browser</A></P></BODY>

I wanted to put this data in the events I was logging, but I obviously didn’t want it to look like this. There isn’t a PowerShell cmdlet for ‘ConvertFrom-HTML’, although that would be great. I could have tried to parse the text and remove the formatting, but that would be a pain trying to escape the right characters and account for all of the html tags. I found several ways online to handle this – namely creating a ‘HTMLFile’ object, writing the html code into it, and then extracting out the innertext property of the object. This works fine in some environments, but throws a weird error if you don’t have Office installed. Here is the code:

$HTML = New-Object -Com "HTMLFile"
$html.IHTMLDocument2_write($htmlcode)

And the error it throws when Office isn’t installed:

Method invocation failed because [System.__ComObject] does not contain a method named 'IHTMLDocument2_write'.

There is also a catch here – that error will also be thrown if you are trying this in PowerShell Core/Microsoft PowerShell (not Windows PowerShell – i.e. anything under 5.x)! Fortunately, there is a quick fix:

$HTML = New-Object -Com "HTMLFile"
try {
    $html.IHTMLDocument2_write($htmlcode)
}
catch {
    $encoded = [System.Text.Encoding]::Unicode.GetBytes($htmlcode)
    $html.write($encoded)
}
$text = ($html.all | Where-Object { $_.tagname -eq 'body' } | Select-Object -Property innerText).innertext

A simple try/catch, and it works great. I am going to create a new repo in GitHub and make this a new function. Hopefully you haven’t wasted too much time trying to track this down – it took me a while, and it wasn’t until I stumbled upon a similar solution on StackOverflow (https://stackoverflow.com/questions/46307976/unable-to-use-ihtmldocument2) that I was able to wrap it up. Expect the cmdlet/function soon!

Powershell – what modules are in use?

There comes a time when you are neck deep in code, and come to the realization that one of your modules could really use a tweak. There is just one problem – what other scripts are using this module? What if you wanted to inventory your scripts and the modules that each script is using? This was the main use-case for this post. I have a scripts directory, and want to get a quick inventory of what modules each one is using.

First, the function:

#requires -version 3.0
Function Test-ScriptFile {
    <#
    .Synopsis
    Test a PowerShell script for cmdlets
    .Description
    This command will analyze a PowerShell script file and display a list of detected commands such as PowerShell cmdlets and functions. Commands will be compared to what is installed locally. It is recommended you run this on a Windows 8.1 client with the latest version of RSAT installed. Unknown commands could also be internally defined functions. If in doubt view the contents of the script file in the PowerShell ISE or a script editor.
    You can test any .ps1, .psm1 or .txt file.
    .Parameter Path
    The path to the PowerShell script file. You can test any .ps1, .psm1 or .txt file.
    .Example
    PS C:\> test-scriptfile C:\scripts\Remove-MyVM2.ps1
     
    CommandType Name                                   ModuleName
    ----------- ----                                   ----------
        Cmdlet Disable-VMEventing                      Hyper-V
        Cmdlet ForEach-Object                          Microsoft.PowerShell.Core
        Cmdlet Get-VHD                                 Hyper-V
        Cmdlet Get-VMSnapshot                          Hyper-V
        Cmdlet Invoke-Command                          Microsoft.PowerShell.Core
        Cmdlet New-PSSession                           Microsoft.PowerShell.Core
        Cmdlet Out-Null                                Microsoft.PowerShell.Core
        Cmdlet Out-String                              Microsoft.PowerShell.Utility
        Cmdlet Remove-Item                             Microsoft.PowerShell.Management
        Cmdlet Remove-PSSession                        Microsoft.PowerShell.Core
        Cmdlet Remove-VM                               Hyper-V
        Cmdlet Remove-VMSnapshot                       Hyper-V
        Cmdlet Write-Debug                             Microsoft.PowerShell.Utility
        Cmdlet Write-Verbose                           Microsoft.PowerShell.Utility
        Cmdlet Write-Warning                           Microsoft.PowerShell.Utility

    .Notes
    Original script provided by Jeff Hicks at (https://www.petri.com/powershell-problem-solver-find-script-commands)

    #>
     
    [cmdletbinding()]
    Param(
        [Parameter(Position = 0, Mandatory = $True, HelpMessage = "Enter the path to a PowerShell script file,",
            ValueFromPipeline = $True, ValueFromPipelineByPropertyName = $True)]
        [ValidatePattern( "\.(ps1|psm1|txt)$")]
        [ValidateScript( { Test-Path $_ })]
        [string]$Path
    )
     
    Begin {
        Write-Verbose "Starting $($MyInvocation.Mycommand)"  
        Write-Verbose "Defining AST variables"
        New-Variable astTokens -force
        New-Variable astErr -force
    }
     
    Process {
        Write-Verbose "Parsing $path"
        $AST = [System.Management.Automation.Language.Parser]::ParseFile($Path, [ref]$astTokens, [ref]$astErr)

        #group tokens and turn into a hashtable
        $h = $astTokens | Group-Object tokenflags -AsHashTable -AsString
     
        $commandData = $h.CommandName | where-object { $_.text -notmatch "-TargetResource$" } | 
        ForEach-Object {
            Write-Verbose "Processing $($_.text)" 
            Try {
                $cmd = $_.Text
                $resolved = $cmd | get-command -ErrorAction Stop
                if ($resolved.CommandType -eq 'Alias') {
                    Write-Verbose "Resolving an alias"
                    #manually handle "?" because Get-Command and Get-Alias won't.
                    Write-Verbose "Detected the Where-Object alias '?'"
                    if ($cmd -eq '?') { 
                        Get-Command Where-Object
                    }
                    else {
                        $resolved.ResolvedCommandName | Get-Command
                    }
                }
                else {
                    $resolved
                }
            } 
            Catch {
                Write-Verbose "Command is not recognized"
                #create a custom object for unknown commands
                [PSCustomobject]@{
                    CommandType = "Unknown"
                    Name        = $cmd
                    ModuleName  = "Unknown"
                }
            }
        } 

        write-output $CommandData
    }

    End {
        Write-Verbose -Message "Ending $($MyInvocation.Mycommand)"
    }
}

This function was originally from Jeff Hicks . I made a few tweaks, but the majority of this is his code. As always, Jeff puts out amazing work.

Next, a simple script to call the function on all of the scripts in a certain directory:

if ($IsWindows -or $IsWindows -eq $null){$ModulesDir = (Get-ItemProperty HKLM:\SOFTWARE\PWSH).PowerShell_Modules}
else{get-content '/var/opt/tifa/settings.json'}
$log = $PSScriptRoot + '\'+ ($MyInvocation.MyCommand.Name).split('.')[0] + '.log'
$ModulesToImport = 'logging', 'SQL', 'PoshKeePass', 'PoshRSJob', 'SCOM', "OMI", 'DynamicMonitoring', 'Nix','Utils'
foreach ($module in $ModulesToImport) {Get-ChildItem $ModulesDir\$module\*.psd1 -Recurse | resolve-path | ForEach-Object { import-module $_.providerpath -force }}

$files = (Get-ChildItem 'D:\pwsh\' -Include *.ps1 -Recurse | where-object { $_.FullName -notmatch "\\modules*" }).fullname

[System.Collections.ArrayList]$AllData = @()
foreach ($file in $files) {
    [array]$data = Test-ScriptFile -Path $file
    foreach ($dataline in $data) {
        $path = $Dataline.module.path
        $version = $dataline.Module.version
        $dataobj = [PSCustomObject]@{
            File          = $File;
            Type          = $dataline.CommandType;
            Name          = $dataline.Name;
            ModuleName    = $dataline.ModuleName;
            ModulePath    = $path;
            ModuleVersion = $version;
        }
        $AllData.add($dataobj)|out-null
    }
}
$AllData|select-object -Property file, type, name, modulename,modulepath,moduleversion -Unique|out-gridview

There are a few gotchas you want to keep an eye out for – in order to correctly identify the cmdlets, you will need to load the module that contains said cmdlet. You can see where I am loading several modules – SCOM, SQL, logging, etc. The modules in my script are loaded from modules stored in a reg key – HKLM:\Software\PWSH. Load your modules in any way that works for you. Also note that the last line sends the output to a gridview – change that output to something more useful unless you just previewing it.

And there you have it! Thanks to Jeff for the core function, and I hope this helps keep your modules organized!

SQL Process Automation with PowerShell

Here is a handy way to handle SQL processes that you find yourself needing to schedule. Of course you can always setup that kind scheduling via the SQL Server Agent, but there are two good reasons to do this kind of scheduling via PowerShell.

1: You don’t have rights to add jobs to via the SQL Server Agent. Some security teams will restrict non-dba access to either the agent or insist on setting the agent to a manual start.
2: You wish to have easier tracking, easier configuration, and just want to do something cool with PowerShell.

The other option you have when trying to schedule something like SQL processes would be to simply use Task Scheduler. Indeed – in my solution I actually use Task Scheduler as a base engine to run every minute or so. What I don’t like about Task Scheduler is trying to put SQL command lines in it. It’s flat out a pain. So, I built something that was easy to configure – even by someone who is not skilled in PowerShell, easy to implement, and has all of the typical good stuff you want with PowerShell.

The solution utilizes 3 main pieces. First, is a script that we will schedule to run every minute via Task Scheduler. Second, a configuration file in JSON format. I choose JSON since it’s simple to read and easy to write. Lastly, we will have XML file that tracks when the last time something ran. Let’s examine each piece:

The script file is fairly straight forward, and there are only a couple of pieces that need explaining. In simple terms this script file will be used as the engine that calls the various SQL commands we specify in the config.json file. We import a few modules, setup a couple of base variables, and then loop through the SQL commands, updating the runhistory.xml file with the timestamp. Schedule this file to run every minute via task scheduler.

$log = $PSScriptRoot + '\'+ ($MyInvocation.MyCommand.Name).split('.')[0] + '.log'
$ModulesDir = 'D:\pwsh\modules'
$ModulesToImport = 'DDTLogging','SQL'
foreach ($module in $ModulesToImport){Get-ChildItem $ModulesDir\$module\*.psd1 -Recurse | resolve-path | ForEach-Object { import-module $_.providerpath -force }}
$Date = Get-Date
$Commands = (get-content $PSScriptRoot\config.json|convertfrom-json).commands
[xml]$RunHistory = get-content "$PSScriptRoot\runhistory.xml"
foreach ($command in $Commands){
    $commandname = $command.name
    $LastRunTime = ($runhistory.Catalog.dataset|where-object {$_.name -eq $commandname}|select-object -Property time).time
    if ($LastRunTime -lt $Date.AddMinutes(-$command.TimePeriod) -or $LastRunTime -eq $null)
    {
        if ($LastRunTime -eq $null){
            $newdata = $RunHistory.Catalog.AppendChild($RunHistory.CreateElement("dataset"))
            $newdata.SetAttribute("name","$commandname")
            $newdata.SetAttribute("time","$date")
            $RunHistory.Save("$PSScriptRoot\runhistory.xml")
        }
        else {
            $RunHistory.Catalog.dataset|where-object {$_.name -eq $commandname}|foreach {$_.time = [string]$date}
            $RunHistory.Save("$PSScriptRoot\runhistory.xml")
        }
        try{
            invoke-sql -server $command.server -database $command.database -method $command.commandtype -integratedsecurity $true -statement $command.statement
        }
        catch{
            write-log -text "Error in $commandname.  $_" -level ERROR -log $log
        }
    }
}

If you are wondering about the Invoke-SQL command, it’s something that I wrapped up in a quick module. You can get the module here.

Now that we have the base script, let’s look at the config.json file. This is where the meat of the information about your commands come from:

{
    "Commands":
    [
        {
            "name":"SQL_Views_Stored_Proc",
            "timeperiod":15,
            "statement":"exec refreshviews",
            "server":"sqlserver1.constoso.com",
            "database":"customers",
            "commandtype":"execute"
        },
        {
            "name":"Update_Time",
            "timeperiod":15,
            "statement":"update cogsuppliers set time = getdate()",
            "server":"sqlserver2.constoso.com",
            "database":"suppliers",
            "commandtype":"update"
        }       
    ]
}

As you can see – it’s a pretty straight forward. Put in the frequency you would like the statement to run (timeperiod), the statement itself, the server and database to run on, and finally the ‘type’ of command it is. This is just to tell the SQL module whether or not to load the data into a data table. That’s it! Anyone with a basic knowledge of how to write a json file can add or remove from this config file quickly!

The final piece is the runshistory.xml file. This simply lets the main script keep track of the last time each statement was run. You shouldn’t have to ever update this file manually.

<Catalog>
  <dataset name="SQL_Views_Stored_Proc" time="11/15/2018 10:49:12" />
  <dataset name="Update_Time" time="11/15/2018 10:49:12" />
</Catalog>

For full transparency, there are a few things I just need to swing back around and address. First – the commands runs sequentially. Long running SQL statements might cause others to fail or fall behind. The plan for that is to use something like PoshRSJob to run the jobs in parallel, and each in it’s own runspace. Secondly, there is a small chance that if someone was to manually edit the runhistory.xml file and remove one of the lines the script could error. I will update the script with a catch to make sure this doesn’t happen.

PowerShell Logging – What about file locking?

Today I was working on one of my large-scale enterprise automation scripts, when a common annoyance reared it’s ugly head. When dealing with multi-threaded applications, in this case Runspaces, it’s common to have one runspace trying to access a resource another runspace is currently using. For example – writing some output data to a log file. If two or more runspaces attempt to access the same log at the same time you will receive an error message. This isn’t a complete stoppage when dealing with log files, but if a critical resource is locked then your script could certainly fail.

So what is the easiest way to get around this? Enter the humble Mutex. There are several other posts that deal with Mutex locking, so I won’t go over the basics. Here I want to share some simple code that makes use of the mutex for writing to a log file.

The code below is a sample write-log function that takes 4 parameters. 3 of them are mandatory – Text for the log entry, name of the log to write to, and the ‘level’ of the entry. Level is really only used to help color coding and reading of the log in something like CMTrace. The last of the four parameters is ‘UseMutex’. This is what is going to tell our function whether or not to lock the resource being accessed.

<#
    .SYNOPSIS
        Simple function to write to a log file
    
    .DESCRIPTION
        This function will write to a log file, pre-pending the date/time, level of detail, and supplied information
    
    .PARAMETER text
        This is the main text to log
    
    .PARAMETER Level
        INFO,WARN,ERROR,DEBUG
    
    .PARAMETER Log
        Name of the log file to send the data to.
    
    .PARAMETER UseMutex
        A description of the UseMutex parameter.
    
    .EXAMPLE
        write-log -text "This is the main problem." -level ERROR -log c:\test.log
    
    .NOTES
        Created by Donnie Taylor.
        Version 1.0     Date 4/5/2016
#>
function Write-Log
{
    param
    (
        [Parameter(Mandatory = $true,
                   Position = 0)]
        [ValidateNotNull()]
        [string]$text,
        [Parameter(Mandatory = $true,
                   Position = 1)]
        [ValidateSet('INFO', 'WARN', 'ERROR', 'DEBUG')]
        [string]$level,
        [Parameter(Mandatory = $true,
                   Position = 2)]
        [string]$log,
        [Parameter(Position = 3)]
        [boolean]$UseMutex
    )
    
    Write-Verbose "Log:  $log"
    $date = (get-date).ToString()
    if (Test-Path $log)
    {
        if ((Get-Item $log).length -gt 5mb)
        {
            $filenamedate = get-date -Format 'MM-dd-yy hh.mm.ss'
            $archivelog = ($log + '.' + $filenamedate + '.archive').Replace('/', '-')
            copy-item $log -Destination $archivelog
            Remove-Item $log -force
            Write-Verbose "Rolled the log."
        }
    }
    $line = $date + '---' + $level + '---' + $text
    if ($UseMutex)
    {
        $LogMutex = New-Object System.Threading.Mutex($false, "LogMutex")
        $LogMutex.WaitOne()|out-null
        $line | out-file -FilePath $log -Append
        $LogMutex.ReleaseMutex()|out-null
    }
    else
    {
        $line | out-file -FilePath $log -Append
    }
}

To write to the log, you use the function like such:

write-log -log c:\temp\output.txt -level INFO -text "This is a log entry" -UseMutex $true

Let’s test this. First, let’s make a script that will fail due to the file being in use. For this example, I am going to use PoshRSJob, which is a personal favorite of mine. I have saved the above function as a module to make sure I can access it from inside the Runspace. Run this script:

import-module C:\blog\PoshRSJob\PoshRSJob.psd1
$I = 1..3000
$I|start-rsjob -Throttle 250 -ScriptBlock{
    param($param)
    import-module C:\blog\write-log.psm1
    Write-Log -log c:\temp\mutex.txt  -level INFO -text $param
}|out-null

Assuming you saved your files in the same locations as mine, and this runs, then you should see something like this when you do a get-rsjob|receive-rsjob:

So what exactly happened here? Well, we told 3000 Runspaces to write their number ($param) to the log file….250 at a time (throttle). That’s obviously going to cause some contention. In fact, if we examine the output file (c:\temp\mutex.txt), and count the actual lines written to it, we will have missed a TON of log entries. On my PC, out of the 3000 that should write, we ended up with only 2813 entries. That is totally unacceptable to miss that many log entries. I’ve exaggerated the issue you will normally see using these large numbers, but this happens all the time when using smaller sets as well. To fix this, we are going to run the same bit of code, but we are going to use the ‘UseMutex’ option in write-log function. This tells each runspace to grab the mutex and attempt to write to the log. If it can’t grab the mutex, it will wait until it can (in this case forever – $LogMutex.WaitOne()|out-null). Run this code:

import-module C:\blog\PoshRSJob\PoshRSJob.psd1
$I = 1..3000
$I|start-rsjob -Throttle 250 -ScriptBlock{
    param($param)
    import-module C:\blog\write-log.psm1
    Write-Log -log c:\temp\mutex.txt  -level INFO -text $param -UseMutex $true
}|out-null

See the ‘-UseMutex’ switch? That should fix our problem. A get-rsjob|receive-rsjob now returns this:

Success! If we examine our output file, we find that all 3000 lines have been written. Using our new write-log function that uses a Mutex, we have solved our locking problem. Coming soon, I will publish the actual code on Github – stay tuned!

PowerShell – Get is optional

Here is something I learned a while back from Mr. Snover himself, and it was something I just couldn’t believe. Sure enough it’s true, and it’s still true in PowerShell Core 6. The “Get-” part of almost all “Get-” commands is completely optional. Yeah – you heard that right. “Get” is optional for almost all. Get-Process can’t run correctly, mainly because “Process” expects some arguments. Otherwise, give it a try!

PS  C:\Blog (9:24:23 PM) > timezone


Id                         : Central Standard Time
DisplayName                : (UTC-06:00) Central Time (US & Canada)
StandardName               : Central Standard Time
DaylightName               : Central Daylight Time
BaseUtcOffset              : -06:00:00
SupportsDaylightSavingTime : True

PS  C:\Blog (9:24:27 PM) > random
1240576702

PS  C:\Blog (9:25:18 PM) > computerinfo


WindowsBuildLabEx                                       : 16299.15.amd64fre.rs3_release.170928-1534
WindowsCurrentVersion                                   : 6.3
WindowsEditionId                                        : EnterpriseN
WindowsInstallationType                                 : Client
WindowsInstallDateFromRegistry                          : 12/7/2017 10:29:37 AM
WindowsProductId                                        : 00330-00180-00000-AA567
WindowsProductName                                      : Windows 10 Enterprise N
WindowsRegisteredOrganization                           :
WindowsRegisteredOwner                                  : Draith
WindowsSystemRoot                                       : C:\WINDOWS
WindowsVersion                                          : 1709
WindowsUBR                                              : 192
BiosCharacteristics                                     : {7, 11, 12, 15...}
BiosBIOSVersion                                         : {ALASKA - 1072009, V1.12, American Megatrends - 4028F}
BiosBuildNumber                                         :
BiosCaption                                             : V1.12
BiosCodeSet                                             :
BiosCurrentLanguage                                     : en|US|iso8859-1
BiosDescription                                         : V1.12
BiosEmbeddedControllerMajorVersion                      : 255
BiosEmbeddedControllerMinorVersion                      : 255
BiosFirmwareType                                        : Uefi
BiosIdentificationCode                                  :
BiosInstallableLanguages                                : 1
BiosInstallDate                                         :
BiosLanguageEdition                                     :
BiosListOfLanguages                                     : {en|US|iso8859-1}
BiosManufacturer                                        : American Megatrends Inc.
BiosName                                                : V1.12
BiosOtherTargetOS                                       :
BiosPrimaryBIOS                                         : True
BiosReleaseDate                                         : 8/10/2015 7:00:00 PM
BiosSerialNumber                                        : To be filled by O.E.M.
BiosSMBIOSBIOSVersion                                   : V1.12
BiosSMBIOSMajorVersion                                  : 2
BiosSMBIOSMinorVersion                                  : 8
BiosSMBIOSPresent                                       : True
BiosSoftwareElementState                                : Running
BiosStatus                                              : OK
BiosSystemBiosMajorVersion                              : 4
BiosSystemBiosMinorVersion                              : 6
BiosTargetOperatingSystem                               : 0
BiosVersion                                             : ALASKA - 1072009
CsAdminPasswordStatus                                   : Disabled
CsAutomaticManagedPagefile                              : True
CsAutomaticResetBootOption                              : True
CsAutomaticResetCapability                              : True
CsBootOptionOnLimit                                     :
CsBootOptionOnWatchDog                                  :
CsBootROMSupported                                      : True
CsBootStatus                                            : {0, 0, 0, 0...}
CsBootupState                                           : Normal boot
CsCaption                                               : DESKTOP-KL1CDTP
CsChassisBootupState                                    : Safe
CsChassisSKUNumber                                      : To be filled by O.E.M.
CsCurrentTimeZone                                       : -360
CsDaylightInEffect                                      : False
CsDescription                                           : AT/AT COMPATIBLE
CsDNSHostName                                           : <>
CsDomain                                                : WORKGROUP
CsDomainRole                                            : StandaloneWorkstation
CsEnableDaylightSavingsTime                             : True
CsFrontPanelResetStatus                                 : Disabled
CsHypervisorPresent                                     : True
CsInfraredSupported                                     : False
CsInitialLoadInfo                                       :
CsInstallDate                                           :
CsKeyboardPasswordStatus                                : Disabled
CsLastLoadInfo                                          :
CsManufacturer                                          : MSI
CsModel                                                 : MS-7917
CsName                                                  : <>
CsNetworkAdapters                                       : {Ethernet 2, Ethernet, vEthernet (Default Switch), vEthernet
                                                          (ExternalSwitch)...}
CsNetworkServerModeEnabled                              : True
CsNumberOfLogicalProcessors                             : 8
CsNumberOfProcessors                                    : 1
CsProcessors                                            : {Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz}
CsOEMStringArray                                        : {To Be Filled By O.E.M.}
CsPartOfDomain                                          : False
CsPauseAfterReset                                       : -1
CsPCSystemType                                          : Desktop
CsPCSystemTypeEx                                        : Desktop
CsPowerManagementCapabilities                           :
CsPowerManagementSupported                              :
CsPowerOnPasswordStatus                                 : Disabled
CsPowerState                                            : Unknown
CsPowerSupplyState                                      : Safe
CsPrimaryOwnerContact                                   :
CsPrimaryOwnerName                                      : Draith
CsResetCapability                                       : Other
CsResetCount                                            : -1
CsResetLimit                                            : -1
CsRoles                                                 : {LM_Workstation, LM_Server, NT}
CsStatus                                                : OK
CsSupportContactDescription                             :
CsSystemFamily                                          : To be filled by O.E.M.
CsSystemSKUNumber                                       : To be filled by O.E.M.
CsSystemType                                            : x64-based PC
CsThermalState                                          : Safe
CsTotalPhysicalMemory                                   : 34305863680
CsPhysicallyInstalledMemory                             : 33554432
CsUserName                                              : <>\Draith
CsWakeUpType                                            : PowerSwitch
CsWorkgroup                                             : WORKGROUP
OsName                                                  : Microsoft Windows 10 Enterprise N
OsType                                                  : WINNT
OsOperatingSystemSKU                                    : WindowsEnterprise
OsVersion                                               : 10.0.16299
OsCSDVersion                                            :
OsBuildNumber                                           : 16299
OsHotFixes                                              : {KB4048951, KB4053577, KB4054022, KB4055237...}
OsBootDevice                                            : \Device\HarddiskVolume2
OsSystemDevice                                          : \Device\HarddiskVolume4
OsSystemDirectory                                       : C:\WINDOWS\system32
OsSystemDrive                                           : C:
OsWindowsDirectory                                      : C:\WINDOWS
OsCountryCode                                           : 1
OsCurrentTimeZone                                       : -360
OsLocaleID                                              : 0409
OsLocale                                                : en-US
OsLocalDateTime                                         : 1/24/2018 9:25:52 PM
OsLastBootUpTime                                        : 1/20/2018 4:14:27 PM
OsUptime                                                : 4.05:11:25.3101843
OsBuildType                                             : Multiprocessor Free
OsCodeSet                                               : 1252
OsDataExecutionPreventionAvailable                      : True
OsDataExecutionPrevention32BitApplications              : True
OsDataExecutionPreventionDrivers                        : True
OsDataExecutionPreventionSupportPolicy                  : OptIn
OsDebug                                                 : False
OsDistributed                                           : False
OsEncryptionLevel                                       : 256
OsForegroundApplicationBoost                            : Maximum
OsTotalVisibleMemorySize                                : 33501820
OsFreePhysicalMemory                                    : 22288576
OsTotalVirtualMemorySize                                : 38482556
OsFreeVirtualMemory                                     : 25010532
OsInUseVirtualMemory                                    : 13472024
OsTotalSwapSpaceSize                                    :
OsSizeStoredInPagingFiles                               : 4980736
OsFreeSpaceInPagingFiles                                : 4980736
OsPagingFiles                                           : {C:\pagefile.sys}
OsHardwareAbstractionLayer                              : 10.0.16299.192
OsInstallDate                                           : 12/7/2017 4:29:37 AM
OsManufacturer                                          : Microsoft Corporation
OsMaxNumberOfProcesses                                  : 4294967295
OsMaxProcessMemorySize                                  : 137438953344
OsMuiLanguages                                          : {en-US}
OsNumberOfLicensedUsers                                 :
OsNumberOfProcesses                                     : 194
OsNumberOfUsers                                         : 2
OsOrganization                                          :
OsArchitecture                                          : 64-bit
OsLanguage                                              : en-US
OsProductSuites                                         : {TerminalServicesSingleSession}
OsOtherTypeDescription                                  :
OsPAEEnabled                                            :
OsPortableOperatingSystem                               : False
OsPrimary                                               : True
OsProductType                                           : WorkStation
OsRegisteredUser                                        : Draith
OsSerialNumber                                          : 00330-00180-00000-AA567
OsServicePackMajorVersion                               : 0
OsServicePackMinorVersion                               : 0
OsStatus                                                : OK
OsSuites                                                : {TerminalServices, TerminalServicesSingleSession}
OsServerLevel                                           :
KeyboardLayout                                          : en-US
TimeZone                                                : (UTC-06:00) Central Time (US & Canada)
LogonServer                                             : \\<>
PowerPlatformRole                                       : Desktop
HyperVisorPresent                                       : True
HyperVRequirementDataExecutionPreventionAvailable       :
HyperVRequirementSecondLevelAddressTranslation          :
HyperVRequirementVirtualizationFirmwareEnabled          :
HyperVRequirementVMMonitorModeExtensions                :
DeviceGuardSmartStatus                                  : Running
DeviceGuardRequiredSecurityProperties                   : {0}
DeviceGuardAvailableSecurityProperties                  : {BaseVirtualizationSupport, DMAProtection}
DeviceGuardSecurityServicesConfigured                   : {0}
DeviceGuardSecurityServicesRunning                      : {0}
DeviceGuardCodeIntegrityPolicyEnforcementStatus         : Off
DeviceGuardUserModeCodeIntegrityPolicyEnforcementStatus : Off

Super-fast mass update of management servers for OpsMgr

Here’s a quick one – you want to update the failover management servers on your agents en-mass, and don’t want to wait 12 years for it to complete. Why do you want to set it? Maybe you only want certain agents talking to certain data-centers, or specific management servers have very limited resources. Regardless of the reasons, if you do need to update the agent config, it can be a bit slow. Here is a quick little script that can make those update a LOT quicker.

First thing first – download PoshRSJob from Boe Prox. It’s about the best thing since sliced bread, and I use it constantly. Download the module and place it in one of your module directories (C:\Windows\System32\WindowsPowerShell\v1.0\Modules, for example). Next, create a CSV called FailOverPairs.csv. This should have 2 columns – Primary and Failover. For example:

Primary,Failover
MS01,MS02
MS03,MS04
MS05,MS06

You will want that header line – mainly because it saves us a couple of lines of code in PowerShell. Next, save that CSV in the same directory as the script below. This CSV will be used to set the appropriate failover partner. Save the script below in the same directory as the csv, and you are good to go! Here is the script:

Import-Module PoshRSJob -Force
Import-Module OperationsManager -Force
$modules = (Get-Module | Where-Object{ $_.Name -notlike 'Microsoft.*' -and $_.Name -ne 'PoshRSJob' -and $_.Name -ne 'ISE' }).path
try
{
    $agents = Get-SCOMAgent
}
catch
{
    write-verbose "Cannot load agent list"
}
$Pairs = Import-Csv -Path $PSScriptRoot + '\FailoverPairs.csv' -Header Primary,Failover
$agents|Start-RSJob -Name { $_.DisplayName } -Throttle 20 -ModulesToImport $modules -ScriptBlock {
    param($agent)
    $Pairs = $using:Pairs
    $primary = $agent.PrimaryManagementServerName
    $CurrentFailover = ($agent.GetFailoverManagementServers().DisplayName)
    foreach ($Pair in $Pairs)
    {
        if ($Pair.Primary -eq $primary){$secondary = $Pair.Failover}
        if ($Pair.Failover -eq $primary){$secondary = $Pair.primary}
    }
    if ($secondary -ne $CurrentFailover)
    {
        $AgentName = $agent.DisplayName
        write-verbose "$AgentName Secondary wrong. Primary $Primary, Current Secondary $CurrentFailover, Discovered Secondary $secondary"
        try
        {
            $Failover = Get-SCOMManagementServer | Where-Object {$_.Name -eq $secondary}
            if ($Failover.IsGateway -eq $true)
            {
                $FailOverServerObject = Get-SCOMGatewayManagementServer | Where-Object {$_.Name -eq $secondary}
            }
            else 
            {
                $FailOverServerObject = $Failover
            }
            Set-SCOMParentManagementServer -Agent $agent -FailoverServer $FailOverServerObject
            write-verbose "$AgentName $secondary set."
        }
        catch
        {
            $ErrorText = $error[0]
            write-verbose "$AgentName Failed to set failover. Current Failover $CurrentFailover, Discovered Failover $secondary.$ErrorText"
        }
    }
}|Out-Null
get-rsjob|Wait-RSJob|Remove-RSJob -force|Out-Null

Let’s examine some of this – the imports are obvious. If you have any issue with unblocking files or execution policy, leave a comment and I will help you through the import. The next line is different:

$modules = (Get-Module | Where-Object{ $_.Name -notlike 'Microsoft.*' -and $_.Name -ne 'PoshRSJob' -and $_.Name -ne 'ISE' }).path

What we are doing here is to get a list of the loaded modules, then exclude some of them. We are doing this because when we run this script, we are creating a ton of Runspaces. By default, these runspaces will need to know which modules to load. We don’t need them to load PoshRSJob, and we don’t need them to load things like the ISE because they are ephemeral – they will go away after they have completed their processing. This line can be modified if you don’t need to load other modules. It will load the OperationsManager module, which is the heavy lifter of this script.

Next, we get all of the agents from the management group. This script needs to be run from a SCOM server, but you could easily modify this script to run from a non-SCOM system by adding the “-computername” switch to the get-scomagent command. Then we import the CSV that contains our failover pairs.

Now the fun starts – this line starts the magic:

$agents|Start-RSJob -Name { $_.DisplayName } -Throttle 20 -ModulesToImport $modules -ScriptBlock {

This is the magic. We are feeding the list of SCOM agents (via the pipeline) to the start-rsjob cmdlet. The “-name” parameter tells the runspaces to use the Agent name as the job name, and the “-Throttle” parameter is set to control the number of runspaces we want running at once. I typically find that there isn’t a lot of benefit to going much over 2 or 3 times the number of logical cores. Maybe if you have remote processes that were very long running it might be beneficial to go up to 5-10 times the number of processors, but for this I found 2-3 to be the sweet spot. You will also see that we are telling start-rsjob what modules to import (see above).

The rest of the script is the scriptblock we want PoshRSJob to run. This is actually pretty straight forward – we set some variables (some of the we have to get with “$using:“). Then we find the current primary and failover, see if they match our pairs, and if they don’t we correct them. This isn’t a fast process, but if you are doing 20 of them at a time, it goes by a lot faster!

At the end of the script, we are simply waiting for the jobs to finish. In fact, if you want to track the progress, comment out this line:

get-rsjob|Wait-RSJob|Remove-RSJob -force|Out-Null

If you comment that line out, you can track how fast your jobs are completing by using this:

get-rsjob|group -property state

We’ve been able to check several thousand systems daily in very little time to make sure our primary and failover pairs are set correctly. I hope you guys get some use from this, and go give Boe some love for his awesome module! Leave a comment if you have any questions, or hit me up on Twitter.

Customize your PowerShell profile for useful startup actions

Did you know you can make PowerShell run any commands you want when you start a shell? This is amazingly useful for gathering information, making settings changes, or kicking off processes – all at shell startup. There are plenty of places that talk about the profiles, so I won’t go into each type, but long story short there are almost 10 different profiles when you account for 32 and 64 bit PowerShell. Some of them are explained in detail here.

For my example here, I am going to deal with the %UserProfile%\My Documents\WindowsPowerShell\profile.ps1 profile, which affects the current user, but all shells. This is useful for when you are switching back and forth between the ISE and console – say when you are testing new scripts. By default this file won’t exist – you will have to create it if it doesn’t.

#See if your profile file exists.  Checks the 'My Documents\WindowsPowerShell' directory
Test-Path $profile

The profile file is really just a .ps1 file. You can put any PowerShell you want in this file. Say you want to get a random Cat Fact every time you start a shell? (who am I to judge?)

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
((((invoke-webrequest -uri https://catfact.ninja/fact -Method get).content).split(':')[1])).trim('","length"')

There are some really useful things you can do now that you know you can run anything. For example – this will show the number of running PowerShell processes, along with WinRM service status:

write-host -ForegroundColor Green "PowerShell Processes: " (Get-process 'PowerShell').count
write-host -ForegroundColor Green "WinRM Status: " (Get-service 'winrm').status

And this will show the PowerShell module paths:

write-host -ForegroundColor Green "Module Paths:"
foreach ($module in ($env:PSModulePath).Split(";")){write-host -ForegroundColor Green "    "$module}

Notice here – I also added a simple “cd c:\blog” to my profile – really useful for starting straight in your scripts directory (no more “cd c:\workspaces\app\development\scripts….” every time you start a console)

But you can do even more! Full functions can be loaded into your profile and are available immediately. One of my favorites is one that will add a Start-RDP function, so I can initiate a remote desktop session without ever touching the start menu! How cool is this?

write-host -ForegroundColor Green "PowerShell Processes: " (Get-process 'PowerShell').count
write-host -ForegroundColor Green "WinRM Status: " (Get-service 'winrm').status
write-host -ForegroundColor Green "Module Paths:"
foreach ($module in ($env:PSModulePath).Split(";")){write-host -ForegroundColor Green "    "$module}
cd C:\Blog
function Start-RDP
{
    param  
    (  
        [Parameter(
            Position = 0,
            ValueFromPipeline=$true,
            Mandatory=$true
        )]
        [ValidateNotNullOrEmpty()]
        [string]
        $ServerName
    )
    mstsc /v:$ServerName 
}

There is also a special function you can put in your profile – it’s the ‘prompt’ function. This will change the PowerShell command line prompt to whatever you want! Just create a new function called ‘prompt’, write-host anything you want into the function, and make sure you put a ‘return ” “‘ at the end – it’s that’s simple! You can put some great data right on the prompt – for example, you can make the current time show up each time the prompt is shown! This is really useful for measuring how long something takes to run if you don’t want to pull out measure-object! Here is my prompt, along with the full profile:

function prompt
{
    $time = "("+(get-date).ToLongTimeString()+")"
    write-host -NoNewline -ForegroundColor Green "PS " (Get-Location).Path $time ">"
    return " "
}
write-host -ForegroundColor Green "PowerShell Processes: " (Get-process 'PowerShell').count
write-host -ForegroundColor Green "WinRM Status: " (Get-service 'winrm').status
write-host -ForegroundColor Green "Module Paths:"
foreach ($module in ($env:PSModulePath).Split(";")){write-host -ForegroundColor Green "    "$module}
cd C:\Blog
function Start-RDP
{
    param  
    (  
        [Parameter(
            Position = 0,
            ValueFromPipeline=$true,
            Mandatory=$true
        )]
        [ValidateNotNullOrEmpty()]
        [string]
        $ServerName
    )
    mstsc /v:$ServerName 
}

And the outcome:

It’s that simple! Customize your profile, and start being productive quicker! Leave a comment below to tell me what your favorite profile modifications are!!

Austin PowerShell User Group Survey Responses!

Last week, I sent out a survey where we were asking people questions about the Austin PowerShell User Group. Basically we are trying to find out what you all want when it comes to location, meeting times, etc… Well, here are the results!!

To begin with we had around 40 responses, which is excellent! Thank you all for responding!

The first question – “Would you be interested in attending an Austin PowerShell User Group meeting in the future?”. I know – I know – softball question. If someone is responding, then they are probably interested in attending. Not surprisingly, the percent that answered yes was 100%! That’s great!

When we asked what days worked best, a couple of clear winners emerged – Friday and Doesn’t Matter.

Next – How often should we meet? Again, a pretty clear winner – Once every other month took almost 50% of the votes!

Now we get to the fun questions – Where and for how long should we meet. First, here are the primary and secondary choices for location. North Austin and Round Rock/Pflugerville appear to be the leaders for Primary choice, and North Austin taking the bulk of the Secondary choice!

When we asked how long each meeting should be, we got some great varied responses! All day and Afternoon took 2/3rds from the Primary, while Afternoon and Mornings dominated the Secondary Choice!

We know we can’t pick everyone’s preferred time and/or location, so next we asked how likely you would still be able to attend if the selection didn’t go your way. All in all, everyone seemed somewhat flexible!

Now on the to the free-form text! Some great suggestions on venues:

Microsoft or Dell Campus
Employer
Microsoft, Member Facilities
Just happy to be aware of this
Dave & Busters, Alamo Drafthouse, MSFT Store (free)
Domain area
eBay (Daytime only), Microsoft (daytime only), User Group Member Businesses
Private companies to host
Yes 🙂
Microsoft Austin on Stonelake

I don’t know who the smart-ass was that said “Yes” with a smiley face, but I will find you 🙂

I should have know better than ask for open comments, but here they are.

I think this is a great idea! It would be great to meet other PS developers in the area
I could participate more easily on days I could not attend if we had live feed or if the presentations were available on you tube or something
Maybe we could expand CTSMUG and devote a session to Powershell every time we meet – We could schedule it after lunch to allow for those attendees who cant take an entire day – Or whenever during the CTSMUG Day that makes the most sense. The technologies are complimentary and it benefits the CTSMUGers as much as the PUGers. Or barring that how about a Powershell Happy Hour post CTSMUG.
Newcomers’ meeting would be cool for a start.
Meetings during business hours opens up more options for locations because you don’t have to pay for extra security or host in a Retail/Food location which may be too noisy. I work for eBay and can easily host meetings (small or large) given enough advanced notice but it must be during the week, during business hours.
Ask for more volunteers to lead the group so we can spread the load.

Here is something nice – someone thanked me!

Thanks Donnie !

And then there’s Duncan McAlynn’s comment (Yes, I know it was you)

Donnie’s an asshole.

Thanks to everyone that took the survey (except Duncan) – we REALLY appreciate it! We will munch on this data, and send out invites shortly!

Run _Anything_ with Flow. PowerShell Triggers

Want to start PowerShell commands from a Tweet? Yeah you do, and you didn’t even know you wanted to.

Earlier this month, a great Flow of the Week was posted that highlighted the ability to use a .net filesystemwatcher to kick off local processes. This sparked an idea – I think we can expand on this and basically run anything we want. Here’s how:

First, let’s start with the Connected Gateway. The link above goes into a bit of detail on how to configure the connection. Nothing special there.
Second, on the Connected Gateway, run this PowerShell script:

$FileSystemWatcher = New-Object System.IO.FileSystemWatcher
$FileSystemWatcher.path = "C:\temp\WatchMe"
$FileSystemWatcher.Filter = "Flow.txt"
$FileSystemWatcher.EnableRaisingEvents = $true
 
Register-ObjectEvent $FileSystemWatcher "Changed" -Action {
$content =  get-content C:\temp\WatchMe\Flow.txt |select-object -last 1
powershell.exe $content
}

This script sets up a FileSystemWatcher on the C:\temp\WatchMe\Flow.txt file. The watcher will only perform an action if the file is changed. There are several options for the “Changed” parameter – Created, Deleted, Renamed, Error, etc… Once created, the watcher will look at the last line of the c:\temp\WatchMe\Flow.txt file, and launch a PowerShell process that takes that last line as the input.

Third – This is the best part. Since we have a FileSystemWatcher, and that watcher is reading the last line of the C:\temp\WatchMe\Flow.txt file and kicking that process off, all we have to do is append a line to that file to start a PowerShell session. Flow has a built-in connection for FileSystem. You can see where this is going. Create a new Flow, and add an input action – I am fond of the Outlook.com Email Arrives action. Supply a suitable trigger in the subject, and add the ‘Append File’ action from the FileSystem service. Here is how mine is configured:

The only catch with this particular setup is that the body of the email needs to be in plain text – Windows 10 Mail app, for example, will not send in plain text. The body of the mail is the PowerShell command we want to run. For example, maybe we want PowerShell to get a list of processes that have a certain name, and dump those to a text file for parsing later. Simply send an email that has the body of “get-process -name chrome|out-file c:\temp\ChromeProcesses.txt”. Here is what that results in:
Before we send the email:

The Email:

After a few minutes – an new folder appears!:

The contents of the text file:

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName                                                  
-------  ------    -----      -----     ------     --  -- -----------                                                  
   1956      97   131588     191648     694.98    728   1 chrome                                                       
    249      22    34268      43356       1.63   4264   1 chrome                                                       
    381      81   307592     331312     145.16   6080   1 chrome                                                       
    149      12     2140      10076       0.05   7936   1 chrome                                                       
    632      86   277900     557484     974.00   9972   1 chrome                                                       
    431      31   147956     159404     182.11  10056   1 chrome                                                       
    219      12     2132       9608       0.08  11636   1 chrome                                                       
    283      50   135932     141512      98.05  12224   1 chrome                                                       
    396      54   133912     297432      18.58  12472   1 chrome                                                       
    253      46   107348     106752      50.13  13276   1 chrome                                                       
    381      48   114452     128836     242.89  14328   1 chrome

Think about what you could do with this – Perhaps you want to do an Invoke-WebRequest every time a RSS Feed updates. Maybe start a set of diagnostic commands when an item is added to Sharepoint. Kick off actions with a Tweet. If you want full scripts to run instead of commands, change the Action section of the FileSystemWatcher to “PowerShell.exe -file $content”. Easy as pie.

PowerShell WSMAN Configuration for Massive Scale

In my day job, I constantly strive to push PowerShell to the limit, attempting to use absolutely every bit of processor/memory/network bandwidth available. One way I do this is with PoshRSJob written by Boe Prox. PoshRSJob is a wonder multi-threading tool, and I use it at pretty heavy scale – typically at a 100 thread throttle.

Sometimes, when you are running a lot of concurrent threads attaching to remote machines, you will run into WinRM connection limitations. They typically will show up in error messages like this when you try to do commands line “invoke-command -computername remoteserver01” :
“This user is allowed a maximum number of 5 concurrent shells, which has been exceeded. “

Configuring typical WSMAN connection limits are fairly well documented, but I was running into another type of error. This error was occurring even after I had upped the connection limits:
“The maximum number of concurrent shells allowed for this plugin has been exceeded.”

This was driving me crazy, until I realized the slightly different wording. Browsing the WSMAN PSDrive, I was eventually able to solve it. The key word in the second error was “plugin”. I had to configure the limits on the plugin, not just the shell. After I realized the difference, I was able to find the right settings. I have compiled them here in a small script that will enable WinRM, and set the limits very high for both the shell and plugin.

enable-psremoting -force 
cd WSMan:\localhost\Shell 
set-item MaxConcurrentUsers 100 
set-item MaxProcessesPerShell 10000 
set-item MaxMemoryPerShellMB 1024 
set-item MaxShellsPerUser 1000 
cd WSMan:\localhost\Plugin\microsoft.powershell\Quotas 
set-item MaxConcurrentUsers 100 
set-item MaxProcessesPerShell 10000 
set-item MaxShells 1000 
set-item MaxShellsPerUser 1000 
restart-service winrm

Obviously test this before you deploy to production. I also found a neat one-liner to monitor the number of WSMan connections of a target system (set the $computername variable to the target, or use localhost):

while($true){(Get-WSManInstance -ComputerName $ComputerName -ResourceURI Shell -Enumerate).count;start-sleep 1}