Precision Computing

If you're testing your PowerShell scripts (manually or automatically,) one of the first questions you'll end up asking yourself is, "did I test enough?"

This is common problem in all of software development. To the rescue is a simple metric known as "Code Coverage" – a measure of how much code you exercised during the testing of that code.

However, the measurement is just the end result – you need a tool to get you there. When it comes to measuring code coverage in PowerShell scripts, though, there simply aren't any tools yet.

Like performance measurement tools, Code Coverage tools are sometimes driven by instrumentation of the source code, and sometimes driven by sampling the code during runtime. Instrumentation gives the highest accuracy, but we can go a long way by runtime analysis alone. To accomplish that, we'll use our favourite feature to abuse – PowerShell script tracing. The last time we pushed it, we got a sampling profiler out of the deal. Let's do it again for code coverage.

Take, for example, the following source code:

trap { "Error handling!"; continue }                                  
                                                                      
"Got here"                                                            
                                                                      
if($args[0] -eq "Test")                                               
{                                                                     
   "Got TEST as an argument"                                          
}                                                                     
elseif($args[0] -eq "Err0r")                                          
{                                                                     
   throw "Catch Me!"                                                  
}                                                                     
else                                                                  
{                                                                     
   "Didn't get TEST as an argument"                                   
}     

We want to run through three parameters that it takes, and make sure we're exercising everything. Notice how we're even being diligent by testing the "Error" case!

PS C:\temp> $tests = @()                                              
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Test }                
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 SomethingElse }       
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Error }               
PS C:\temp>                                                           
PS C:\temp> .\Get-ScriptCoverage.ps1 .\Test-CodeCoverage.ps1 $tests 

What does that give us?

trap { "Error handling!"; continue }                                  
                                                                      
"Got here"                                                            
                                                                      
if($args[0] -eq "Test")                                               
{                                                                     
   "Got TEST as an argument"                                          
}                                                                     
elseif($args[0] -eq "Err0r")                                          
{                                                                     
   throw "Catch Me!"                                                  
}                                                                     
else                                                                  
{                                                                     
   "Didn't get TEST as an argument"                                   
}                                                                     
Coverage Statistics: 66.6666666666667%                                
PS C:\temp>   

Ouch! Why is the error handling code (in red) not being hit? Ah, after further investigation, it turns out that we have a typo in our string comparison. We fix it:

...
elseif($args[0] -eq "Err0r")
...

Becomes

...
elseif($args[0] -eq "Error")
...

And run code coverage again:

trap { "Error handling!"; continue }                                  
                                                                      
"Got here"                                                            
                                                                      
if($args[0] -eq "Test")                                               
{                                                                     
   "Got TEST as an argument"                                          
}                                                                     
elseif($args[0] -eq "Error")                                          
{                                                                     
   throw "Catch Me!"                                                  
}                                                                     
else                                                                  
{                                                                     
   "Didn't get TEST as an argument"                                   
}                                                                     
Coverage Statistics: 100%                                             
PS C:\temp>    

Much better.

Here is the script – under 80 lines of (heavily commented) code:

## Get-ScriptCoverage.ps1                                             
## Test the script named by $testScript for code coverage.            
## The command given by $command must exercise this named             
## script.                                                            
param([string] $testScript, [ScriptBlock[]] $command)                 
                                                                      
# Store the content of the script to be tested                        
$fileContent = gc $testScript -ea Stop                                
                                                                      
## Start a transcript, and log it to a file                           
$tempFile = [IO.Path]::GetTempFilename()                              
Start-Transcript $tempFile                                            
                                                                      
## Turn on line-level tracing, run the command(s),                    
## then turn off line-level tracing again.                            
Set-PsDebug -Trace 1                                                  
$command | Foreach-Object { & $_ }                                    
Set-PsDebug -Trace 0                                                  
                                                                      
## Stop the transcript                                                
Stop-Transcript                                                       
                                                                      
## Get the result of the script coverage run                          
$coverageContent = (gc $tempFile) -match "^DEBUG:"                    
Remove-Item -LiteralPath $tempFile                                    
                                                                      
Clear-Host                                                            
                                                                      
## Clean up interference from other scripts                           
$scriptLines = @()                                                    
$processedLines = @{}                                                 
                                                                      
foreach($originalLine in $coverageContent)                            
{                                                                     
    # Make sure we only process unique lines in the                   
    # transcript                                                      
    if($processedLines[$originalLine]) { continue }                   
    $processedLines[$originalLine] = $true                            
                                                                      
    ## Recover as much as possible from the original script line      
    ## without its debugging information                              
    $originalLine = $originalLine -replace " <<<< ",""                
    $line = $originalLine -replace '\D*\d+\+ (.*)','$1'               
                                                                      
    ## Go through each line in the original script, and see if        
    ## this is actually in the script                                 
    foreach($fileLine in $fileContent)                                
    {                                                                 
        ## If it is, add the debug line to the list of lines          
        ## covered by this scenario                                   
        if($fileLine.Contains($line))                                 
        {                                                             
            $scriptLines += $originalLine                             
        }                                                             
    }                                                                 
}                                                                     
                                                                      
## Find out which line numbers were covered                           
$coveredLines = $scriptLines |                                        
    % { $_ -replace '\D*(\d+)\+ .*','$1' } | Sort -Unique             
                                                                      
$coverageCount = 0                                                    
$possibleCoveredLines = 0                                             
for($counter = 1; $counter -le $fileContent.Count; $counter++)        
{                                                                     
    $color = "Red"                                                    
    $line = $fileContent[$counter - 1]                                
                                                                      
    ## Ignore comments, blank lines, curly                            
    ## braces, and fall-through conditional statements                
    ## in coverage computation (as they are never                     
    ## traced in Set-PsDebug tracing                                  
    if(($line -notmatch '^\s*#') -and                                 
       ($line -notmatch '^\s*{\s*$') -and                             
       ($line -notmatch '^\s*}\s*$') -and                             
       ($line -notmatch '^\s*else') -and                              
       ($line -notmatch '^\s*param\(') -and                           
       ($line.Trim()))                                                
    {                                                                 
        $possibleCoveredLines++                                       
    }                                                                 
    else { $color = "Gray" }                                          
                                                                      
    ## If this line was hit in code coverage, colour it               
    ## green                                                          
    if($coveredLines -contains $counter)                              
    {                                                                 
        $color = "Green"                                              
        $coverageCount++                                              
    }                                                                 
                                                                      
    ## Display the line in the appropriate colour                     
    Write-Host -Fore $color $line                                     
}                                                                     
                                                                      
## Output the coverage statistics                                     
Write-Host ("Coverage Statistics: " +                                 
    "$($coverageCount / $possibleCoveredLines * 100)%")