Generating Code Coverage from PowerShell Scripts

If you're testing your PowerShell scripts (manually or automatically,) one of the first questions you'll end up asking yourself is, "did I test enough?"

This is common problem in all of software development. To the rescue is a simple metric known as "Code Coverage" – a measure of how much code you exercised during the testing of that code.

However, the measurement is just the end result – you need a tool to get you there. When it comes to measuring code coverage in PowerShell scripts, though, there simply aren't any tools yet.

Like performance measurement tools, Code Coverage tools are sometimes driven by instrumentation of the source code, and sometimes driven by sampling the code during runtime. Instrumentation gives the highest accuracy, but we can go a long way by runtime analysis alone. To accomplish that, we'll use our favourite feature to abuse – PowerShell script tracing. The last time we pushed it, we got a sampling profiler out of the deal. Let's do it again for code coverage.

Take, for example, the following source code:

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Err0r")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}     

We want to run through three parameters that it takes, and make sure we're exercising everything. Notice how we're even being diligent by testing the "Error" case!

PS C:\temp> $tests = @()
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Test }
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 SomethingElse }
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Error }
PS C:\temp>
PS C:\temp> .\Get-ScriptCoverage.ps1 .\Test-CodeCoverage.ps1 $tests 

What does that give us?

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Err0r")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}
Coverage Statistics: 66.6666666666667%
PS C:\temp>   

Ouch! Why is the error handling code (in red) not being hit? Ah, after further investigation, it turns out that we have a typo in our string comparison. We fix it:

...

elseif($args[0] -eq "Err0r")

...

Becomes

...

elseif($args[0] -eq "Error")

...

And run code coverage again:

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Error")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}
Coverage Statistics: 100%
PS C:\temp>    

Much better.

Here is the script – under 80 lines of (heavily commented) code:

## Get-ScriptCoverage.ps1
## Test the script named by $testScript for code coverage.
## The command given by $command must exercise this named
## script.
param([string] $testScript, [ScriptBlock[]] $command)

# Store the content of the script to be tested
$fileContent = gc $testScript -ea Stop

## Start a transcript, and log it to a file
$tempFile = [IO.Path]::GetTempFilename()
Start-Transcript $tempFile

## Turn on line-level tracing, run the command(s),
## then turn off line-level tracing again.
Set-PsDebug -Trace 1
$command | Foreach-Object { & $_ }
Set-PsDebug -Trace 0

## Stop the transcript
Stop-Transcript

## Get the result of the script coverage run
$coverageContent = (gc $tempFile) -match "^DEBUG:"
Remove-Item -LiteralPath $tempFile

Clear-Host

## Clean up interference from other scripts
$scriptLines = @()
$processedLines = @{}

foreach($originalLine in $coverageContent)
{
    # Make sure we only process unique lines in the
    # transcript
    if($processedLines[$originalLine]) { continue }
    $processedLines[$originalLine] = $true

    ## Recover as much as possible from the original script line
    ## without its debugging information
    $originalLine = $originalLine -replace " <<<< ",""
    $line = $originalLine -replace '\D*\d+\+ (.*)','$1'

    ## Go through each line in the original script, and see if
    ## this is actually in the script
    foreach($fileLine in $fileContent)
    {
        ## If it is, add the debug line to the list of lines
        ## covered by this scenario
        if($fileLine.Contains($line))
        {
            $scriptLines += $originalLine
        }
    }
}

## Find out which line numbers were covered
$coveredLines = $scriptLines |
    % { $_ -replace '\D*(\d+)\+ .*','$1' } | Sort -Unique

$coverageCount = 0
$possibleCoveredLines = 0
for($counter = 1; $counter -le $fileContent.Count; $counter++)
{
    $color = "Red"
    $line = $fileContent[$counter - 1]

    ## Ignore comments, blank lines, curly
    ## braces, and fall-through conditional statements
    ## in coverage computation (as they are never
    ## traced in Set-PsDebug tracing
    if(($line -notmatch '^\s*#') -and
       ($line -notmatch '^\s*{\s*$') -and
       ($line -notmatch '^\s*}\s*$') -and
       ($line -notmatch '^\s*else') -and
       ($line -notmatch '^\s*param\(') -and
       ($line.Trim()))
    {
        $possibleCoveredLines++
    }
    else { $color = "Gray" }

    ## If this line was hit in code coverage, colour it
    ## green
    if($coveredLines -contains $counter)
    {
        $color = "Green"
        $coverageCount++
    }

    ## Display the line in the appropriate colour
    Write-Host -Fore $color $line
}

## Output the coverage statistics
Write-Host ("Coverage Statistics: " +
    "$($coverageCount / $possibleCoveredLines * 100)%")               

2 Responses to “Generating Code Coverage from PowerShell Scripts”

  1. PowerShell Magazine » New feature in Pester 3.0: Code Coverage metrics writes:

    […] for PowerShell yet. The two examples that I found came from Lee Holmes and James Brundage:  https://www.leeholmes.com/blog/2008/05/20/generating-code-coverage-from-powershell-scripts/ and http://scriptcoverage.start-automating.com/ , respectively. Both of these gave me valuable […]

  2. Jim writes:

    So, I realize this is an extremely old post (hoping you’ll still reply), but I found it while trying to figure out a way to log debug output to a file. I thought using a transcript file was a pretty neat way to do that, but I tried something similar in my own code and it never logged any of the debug output to the transcript file. I thought perhaps I was doing something wrong so I tried your code with the same results, no debug output logged. I’m on Windows 10 using PowerShell 5 so I don’t know if something has changed. I was wondering if you know if this code should still work or if something has changed to break it and if it has been broken if you know another way to log debug output to a file.

    Thanks!

Leave a Reply