Archives for the Month of May, 2008

Generating Code Coverage from PowerShell Scripts

If you're testing your PowerShell scripts (manually or automatically,) one of the first questions you'll end up asking yourself is, "did I test enough?"

This is common problem in all of software development. To the rescue is a simple metric known as "Code Coverage" – a measure of how much code you exercised during the testing of that code.

However, the measurement is just the end result – you need a tool to get you there. When it comes to measuring code coverage in PowerShell scripts, though, there simply aren't any tools yet.

Like performance measurement tools, Code Coverage tools are sometimes driven by instrumentation of the source code, and sometimes driven by sampling the code during runtime. Instrumentation gives the highest accuracy, but we can go a long way by runtime analysis alone. To accomplish that, we'll use our favourite feature to abuse – PowerShell script tracing. The last time we pushed it, we got a sampling profiler out of the deal. Let's do it again for code coverage.

Take, for example, the following source code:

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Err0r")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}     

We want to run through three parameters that it takes, and make sure we're exercising everything. Notice how we're even being diligent by testing the "Error" case!

PS C:\temp> $tests = @()
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Test }
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 SomethingElse }
PS C:\temp> $tests += { .\Test-CodeCoverage.ps1 Error }
PS C:\temp>
PS C:\temp> .\Get-ScriptCoverage.ps1 .\Test-CodeCoverage.ps1 $tests 

What does that give us?

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Err0r")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}
Coverage Statistics: 66.6666666666667%
PS C:\temp>   

Ouch! Why is the error handling code (in red) not being hit? Ah, after further investigation, it turns out that we have a typo in our string comparison. We fix it:

...

elseif($args[0] -eq "Err0r")

...

Becomes

...

elseif($args[0] -eq "Error")

...

And run code coverage again:

trap { "Error handling!"; continue }

"Got here"

if($args[0] -eq "Test")
{
   "Got TEST as an argument"
}
elseif($args[0] -eq "Error")
{
   throw "Catch Me!"
}
else
{
   "Didn't get TEST as an argument"
}
Coverage Statistics: 100%
PS C:\temp>    

Much better.

Here is the script – under 80 lines of (heavily commented) code:

## Get-ScriptCoverage.ps1
## Test the script named by $testScript for code coverage.
## The command given by $command must exercise this named
## script.
param([string] $testScript, [ScriptBlock[]] $command)

# Store the content of the script to be tested
$fileContent = gc $testScript -ea Stop

## Start a transcript, and log it to a file
$tempFile = [IO.Path]::GetTempFilename()
Start-Transcript $tempFile

## Turn on line-level tracing, run the command(s),
## then turn off line-level tracing again.
Set-PsDebug -Trace 1
$command | Foreach-Object { & $_ }
Set-PsDebug -Trace 0

## Stop the transcript
Stop-Transcript

## Get the result of the script coverage run
$coverageContent = (gc $tempFile) -match "^DEBUG:"
Remove-Item -LiteralPath $tempFile

Clear-Host

## Clean up interference from other scripts
$scriptLines = @()
$processedLines = @{}

foreach($originalLine in $coverageContent)
{
    # Make sure we only process unique lines in the
    # transcript
    if($processedLines[$originalLine]) { continue }
    $processedLines[$originalLine] = $true

    ## Recover as much as possible from the original script line
    ## without its debugging information
    $originalLine = $originalLine -replace " <<<< ",""
    $line = $originalLine -replace '\D*\d+\+ (.*)','$1'

    ## Go through each line in the original script, and see if
    ## this is actually in the script
    foreach($fileLine in $fileContent)
    {
        ## If it is, add the debug line to the list of lines
        ## covered by this scenario
        if($fileLine.Contains($line))
        {
            $scriptLines += $originalLine
        }
    }
}

## Find out which line numbers were covered
$coveredLines = $scriptLines |
    % { $_ -replace '\D*(\d+)\+ .*','$1' } | Sort -Unique

$coverageCount = 0
$possibleCoveredLines = 0
for($counter = 1; $counter -le $fileContent.Count; $counter++)
{
    $color = "Red"
    $line = $fileContent[$counter - 1]

    ## Ignore comments, blank lines, curly
    ## braces, and fall-through conditional statements
    ## in coverage computation (as they are never
    ## traced in Set-PsDebug tracing
    if(($line -notmatch '^\s*#') -and
       ($line -notmatch '^\s*{\s*$') -and
       ($line -notmatch '^\s*}\s*$') -and
       ($line -notmatch '^\s*else') -and
       ($line -notmatch '^\s*param\(') -and
       ($line.Trim()))
    {
        $possibleCoveredLines++
    }
    else { $color = "Gray" }

    ## If this line was hit in code coverage, colour it
    ## green
    if($coveredLines -contains $counter)
    {
        $color = "Green"
        $coverageCount++
    }

    ## Display the line in the appropriate colour
    Write-Host -Fore $color $line
}

## Output the coverage statistics
Write-Host ("Coverage Statistics: " +
    "$($coverageCount / $possibleCoveredLines * 100)%")               

First Steps in Flight

For a long time, I've been thinking about learning to fly. One of the random long-distance-drive conversations I like to have is, "What job would you do if you were for some reason prevented from doing what you do now?" For me, flying has always been something I would entertain, but only romantically. The downsides to doing it professionally are huge -- the time away from your family and strict seniority-based promotions being the two largest factors. So instead, I wanted to reward myself with flying lessons the next time I thought it was appropriate. A King of the Hill episode, of all things, changed my mind on that.

image

In that episode, Hank (the father) and Peggy (the mother) were going through marital problems, and spoke with a counselor. The counselor asked them what they wanted to do when they retired -- something they had been planning toward for a long time. Their dream was to buy some motorcycles and tour the United States together. The counselor's suggestion was simple -- holding off on buying the bikes doesn't really help anything, so just buy the darn things and start enjoying yourselves! The cross-country tours can wait until you have the time to do them, but little local trips can happen right now.

That started the slow chemical reaction in my brain, which ultimately led me to start investigating flying lessons more seriously.

My typical obsessive online research turned up lots of information about learning to fly. Tom Unger’s “Flying Lessons” journal was indispensable, gave a lot of great background information, and what to expect with the process. However, I found very little about picking a flight school around the Seattle area, and sent a mail to an internal mailing list asking for opinions. Four suggestions came up most frequently:

I ruled out Galvin immediately – their responses to customer complaints on http://www.airnav.com/airport/KBFI/GALVIN were ridiculous and showed complete disregard for their customers.

I ended up looking for operations out of the Renton Municipal Airport (KRNT) as my primary factor. It’s close to my house, and commute time seems to be the most important element in choosing a base. Airport traffic and other factors can be learned at other airports (such as Crest Airpark for its crazy small field, and Boeing Field for its busy airspace,) since most of your practice take-offs and landings come from explicit training exercises (as opposed to being dictated by the base you fly from.)

I first checked out Pro-Flight during a 9:00 AM intro flight, taking up a well-loved Cessna 172. Introductory flights are an amazing deal – even as a leisure activity. They cost under $100 for a scenic tour of the Puget Sound, and you get to do most of the flying yourself.

image        image

The biggest challenge (as compared to driving a car) was adapting to a new mental model, and adapting to a very manual control system. When taxiing, having 4 control surfaces at my feet was pretty complex – seeing as I kept on thinking of it like controlling handles on a bike steering column. I would instinctively push right expecting to turn left, but got the hang of it more when I thought about it as differential braking (even when I was using the nose wheel part of the control.) My only point of minor helmet fire came at this point – when we needed to use differential braking to make a turn, but my feet were only on the nose wheel portion of the control (as suggested by the instructor.)

image

Oh, and ignoring my internal alarm bells is hard enough when starting an automatic transmission car without depressing a clutch – an airplane was much worse.

What surprised me most was how much input you had to give the controls. It felt like there was a good 8 inches of travel in the foot pedals – much more than I was expecting. I was sure I was going to rip the knob off of the dash when I was priming the engine.

After we taxied, takeoff went pretty well. There was no traffic, so we didn’t burn engine hours waiting in line. Actually leaving the pavement was weird, as I again was expecting to need a subtle touch. After pushing the throttle forward slowly like you see them do on big airplanes, the instructor told me to just gun it 🙂 The climb felt surprisingly manual as well, although the instructor did give me a rate of ascent to aim for, which helped a lot. Once I visually leveled off, the instructor pointed out that I was still climbing.  The nose definitely points a lot lower during steady flight than it does during takeoff, but adapting to that came shortly.

We did some minor orientation and directional work, and even got to do some nice steep turns. The ceiling was really indefinite, and we had to avoid some cloud cover. The runway approach went really well as we approached from the North. I guess since there was no traffic, we basically did the 45 degree entry directly into an extended final for the traffic pattern. Lining up felt pretty natural, although I made the approach too low – I was aiming for the numbers instead of the first dash.

image

So all-in-all, it was a great time up – and they seem like a very capable flight school.

The next day, I took a flight with AcuWings, and I can understand why there is so much support for them. Their office was a little hard to find at first (barely any signs on an already nondescript office building,) but they seemed well-run. Their office was off-base, so we had a very short drive across the road to the actual airplanes. They will be getting a building on the field in the near future, so that will be convenient. I followed the owner to the field who, despite his precision flying and deep knowledge of flight rules, failed to use his signal light even once.

For that intro flight, we flew their Cirrus SR20 (since I wanted to try a completely different airplane.) Different it was – an ’07 model, tons of electronics, tons of power, and no traditional yoke! You control the throttle with your right hand, and control the direction with a joystick-like device on your left. The weather was beautiful: in the mid-60s with unlimited ceiling, and 10 miles of visibility. This made for a much busier airport, which was also good to experience.

image     image

My daughter joined us on this ride -- an experience I was thrilled to have been able to share with her.

The aircraft was a lot more responsive than the Cessna, so I initially pitched it up to like 1100 ft/min correcting to the proper 700 ft/min or so during takeoff. We climbed to 3500 feet, and were treated to an awesome view of Mount Rainier. With all of the interactive electronics and gadgets, I had to consciously stop myself from getting too absorbed in the pretty displays, and instead use the cockpit window for what it was intended for 🙂 We did some more level turns, played with the autopilot a bit, and practiced throttle control.

I was amazed to learn that airplanes will “skip” in the atmosphere. If you let them go with wings level and drop the power, they will just settle on a new lower altitude that matches the new power.

After about half an hour in the air, we came back to the Renton base and ran a more typical approach. We flew parallel to the runway (which I was surprised to realize was 30ish degrees off of the North-South axis,) and made a short final from the South due to heavy traffic. The landing went well, probably because it was almost entirely under the control of my instructor 🙂

After landing, it became very clear to me that this hobby has much more in store for me.