Use retry functionality when working with PowerShell cmdlets that are not thread safe

Problem

Thread safety is a concept often overlooked when developing PowerShell cmdlets. When used for manual administration tasks such as running commands in a PowerShell console, the lack of thread safety is rarely an issue. But when using the same commands in a multi-threaded environment such as the ZervicePoint Provisioning system, activities that work perfectly fine during service development and testing may start failing intermittently when deployed in a production environment and subjected to load with a higher level of concurrent processing.

Whether or not a PowerShell cmdlet is thread safe may not be immediately obvious, but there are ways to tell. For example one may create a simple service containing a provider activity, stop the process system, then create a lot of orders and start the process system again. The result is that all the activities are processed simultaneously, which should reveal any thread safety issues. Tests like that should always be conducted during all provider development.

If a thread safety issue is detected, measures must be taken to mitigate the effects. Often a simple Try-Catch statement within a While loop is enough to resolve the issue, but the problem may sometimes be more complex and require a different approach.

Solution

The following is a demonstration of how to use a While loop with random delays to overcome the thread safety issues in the Service Manager Get-SCSMClass cmdlet.

The code below is not thread safe and will fail when two or more activities are executed simultaneously.

Code that will fail when executed concurrently

$srClass = Get-SCSMClass -Name System.WorkItem.ServiceRequest$

Implementing the following code will resolve the issue.

Code that will retry the operation with random delays until is succeeds

$startTime = Get-Date
While (-not $srClass) {
    Try {
        $srClass = Get-SCSMClass -Name System.WorkItem.ServiceRequest$ -ErrorAction Stop
    } Catch {
        If ((New-TimeSpan -Start $startTime -End (Get-Date)).TotalSeconds -gt 300) { Throw }
        Start-Sleep -Seconds (Get-Random -Minimum 1 -Maximum 5)
    }
}