burn-console: Optimized, and Ready For Marshmallows

In the last article, I introduced a library function to help you invoke inline C# from your scripts.  Since we carefully profiled the burn-console script’s performance, we know exactly where to apply these inline performance optimizations: to the updateBuffer function, and the updateScreen function.

The completed script is included below. The inline optimizations bring us to about 30 - 40 frames per second, easily fast enough to provide smooth animation.

Library for Inline C# in MSH

In the last post, we got nearly as far as we could in improving the performance of our MSH script.  We used the profiler to help target our performance optimizations.  After we were finished, setting variables, incrementing counters, and comparing colour values took up the vast majority of our time.  We can’t make these statements more efficient, nor can we execute them less frequently.

To really push the performance of this script, we’ll write the highly critical sections using inline C#, rather than MSH.

Scripting with the Microsoft Shell -- Now on Script Center

The scripting guys continue to impart their magic scripting touch, now with a great big dose of MSH.

Their “Scripting with the Microsoft Shell” Script Center went live recently, and it’s already chock-full of sample scripts – most applying the power of WMI interop.

Their first article covers Accessing WMI From the Microsoft Shell – check it out!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Accessing Performance Counters in PowerShell

The question came up on the newsgroup on how to monitor CPU usage in PowerShell.  I wrote a script to demonstrate this some time ago with the intent to write about it – so now is probably an ideal time.

The following poll-process script retrieves the process name, main window title, processor usage, disk activity, and working set.  It continually refreshes the display to give you a task manager-like experience.

burn-console Part III: The Most Efficient MSH Script

Now that we have our script profiler up and running, we instrument our script a little to mark regions we are concerned about.  You can download that starting script here: [burn-console-1.profiler.msh]   After running the script profiler, we get the performance breakdown:

Breakdown by line:
----------------------------
 15%: Line  123 -             if($colour -lt 20) { $colour -= 1 }
 14%: Line  122 -             if($colour -le 70) { $colour -= 3 }
 11%: Line  124 -             if($colour -lt 0) { $colour = 0 }
 10%: Line  128 -             $tempWorkingBuffer[$baseOffset -
  8%: Line  117 -             $colour /= 4.0
  7%: Line  121 -             if($colour -gt 70) { $colour -= 1 }
  6%: Line  154 -             $nextScreen[$row, $column] = `
  6%: Line  113 -             $colour = $screenBuffer[$baseOffset]
  6%: Line  115 -             $colour += $screenBuffer[$baseOffset + 1]
  6%: Line  116 -             $colour += $screenBuffer[$baseOffset +
  5%: Line  114 -             $colour += $screenBuffer[$baseOffset - 1]
  5%: Line  109 -             $baseOffset = ($windowWidth * $row) +
  0%: Line   90 -         if($random.NextDouble() -ge 0.20)
  0%: Line  152 -         for($column = 0; $column -lt $windowWidth;
  0%: Line   94 -             $screenBuffer[($windowHeight - 2) *

Breakdown by marked regions:
----------------------------
  6%: updateScreen
  0%: startFireLastRow
 93%: propigate
  0%: Unmarked
  0%: main

It looks like we’re spending 93% of our time propagating the fire.  When we investigate how we’re spending the time, none of the hot spots are individually egregious.  Since we’re in such a tight loop, we’re spending most of the time comparing colours.  We can be a little smarter with our ‘if’ statements, attempting to minimize the number of comparisons and variable assignments.  Before optimization, we have 4 checks per loop, and (a weighted average of) just over 1 assignment per loop.  If we write it as below, we reduce that to (a weighted average of) 1 check per loop, and (a weighted average even closer to) 1 assignment per loop.

burn-console Digression: An MSH Script Profiler Part 2

In the last article, we talked about the theory behind a sampling profiler.  Let’s go over the code to implement one.  We’ll use this to guide our future optimizations.  You can also download the script here: profile-transcript.msh.

################################################################################
##
## profile-transcript.msh
##
## Computes the performance characteristics of a script, based on the transcript
## of it running at trace level 1.
##
## To profile a script:
##    1) Turn on MSH script tracing in the window that will run the script: 
##       set-mshdebug –trace 1
##    2) Turn on the transcript for the window that will run the script:
##       start-transcript
##       (Note the filename that Monad provides as the logging destination.)
##    3) Type in the script name, but don't actually start it.
##    4) Open another MSH window, and navigate to the directory holding 
##       this script.  Type in '.\profile-transcript.msh <path-to-transcript>',
##       replacing <path-to-transcript> with the path given in step 2.  Don't
##       press <Enter> yet.
##    5) Switch to the profiled script window, and start the script.
##       Switch to the window containing profile-transcript, and press <Enter>
##    6) Wait until your profiled script exits, or has run long enough to be
##       representative of its work.  To be statistically accurate, your script
##       should run for at least ten seconds.
##    7) Switch to the window running profile-transcript.msh, and press a key.
##    8) Switch to the window holding your profiled script, and type:
##       stop-transcript
##    9) Delete the transcript.
##
## Note: You can profile regions of code (ie: functions) rather than just lines
## by placing the following call at the start of the region:
##       write-debug "ENTER <region_name>"
## and the following call and the end of the region:
##       write-debug "EXIT"
## This is implemented to account exclusively for the time spent in that 
## region, and does not include time spent in regions contained within the
## region.  For example, if FunctionA calls FunctionB, and you've surrounded
## each by region markers, the statistics for FunctionA will not include the
## statistics for FunctionB.
##
################################################################################
param($logFilePath = $(throw "Please specify a path to the transcript log file."))

function Main
{
    ## Run the actual profiling of the script.  $uniqueLines gets
    ## the mapping of line number to actual script content.
    ## $samples gets a hashtable mapping line number to the number of times
    ## we observed the script running that line.
    $uniqueLines = @{}
    $samples = GetSamples $uniqueLines
    
    "Breakdown by line:"
    "----------------------------"

    ## Create a new hash table that flips the $samples hashtable -- 
    ## one that maps the number of times sampled to the line sampled.
    ## Also, figure out how many samples we got altogether.
    $counts = @{}
    $totalSamples = 0;
    foreach($item in $samples.Keys) 
    { 
       $counts[$samples[$item]] = $item 
       $totalSamples += $samples[$item]
    }

    ## Go through the flipped hashtable, in descending order of number of 
    ## samples.  As we do so, output the number of samples as a percentage of
    ## the total samples.  This gives us the percentage of the time our script
    ## spent executing that line.
    foreach($count in ($counts.Keys | sort -desc))
    {
       $line = $counts[$count]
       $percentage = "{0:#0}" -f ($count * 100 / $totalSamples)
       "{0,3}%: Line {1,4} -{2}" -f $percentage,$line,
          $uniqueLines[$line]
    }

    ## Go through the transcript log to figure out which lines are part of any
    ## marked regions.  This returns a hastable that maps region names to
    ## the lines they contain.
    ""
    "Breakdown by marked regions:"
    "----------------------------"
    $functionMembers = GenerateFunctionMembers
    
    ## For each region name, cycle through the lines in the region.  As we
    ## cycle through the lines, sum up the time spent on those lines and output
    ## the total.
    foreach($key in $functionMembers.Keys)
    {
        $totalTime = 0
        foreach($line in $functionMembers[$key])
        {
            $totalTime += ($samples[$line] * 100 / $totalSamples)
        }
        
        $percentage = "{0:#0}" -f $totalTime
        "{0,3}%: {1}" -f $percentage,$key
    }
}

## Run the actual profiling of the script.  $uniqueLines gets
## the mapping of line number to actual script content.
## Return a hashtable mapping line number to the number of times
## we observed the script running that line.
function GetSamples($uniqueLines)
{
    ## Open the log file.  We use the .Net file I/O, so that we keep monitoring
    ## just the end of the file.  Otherwise, we would make our timing innacurate
    ## as we scan the entire length of the file every time.
    $logStream = [System.IO.File]::Open($logFilePath, "Open", "Read", "ReadWrite")
    $logReader = new-object System.IO.StreamReader $logStream

    $random = new-object Random
    $samples = @{}

    $lastCounted = $null
    
    ## Gather statistics until the user presses a key.
    while(-not $host.UI.RawUI.KeyAvailable)
    {
       ## We sleep a slightly random amount of time.  If we sleep a constant
       ## amount of time, we run the very real risk of improperly sampling 
       ## scripts that exhibit periodic behaviour.
       $sleepTime = [int] ($random.NextDouble() * 100.0)
       start-sleep -m $sleepTime

       ## Get any content produced by the transcript since our last poll.
       ## From that poll, extract the last DEBUG statement (which is the last
       ## line executed.)
       $rest = $logReader.ReadToEnd()
       $lastEntryIndex = $rest.LastIndexOf("DEBUG: ")

       ## If we didn't get a new line, then the script is still working on the
       ## last line that we captured.
       if($lastEntryIndex -lt 0) 
       { 
          if($lastCounted) { $samples[$lastCounted]++ }
          continue; 
       }
       
       ## Extract the debug line.
       $lastEntryFinish = $rest.IndexOf("\n", $lastEntryIndex)
       if($lastEntryFinish -eq -1) { $lastEntryFinish = $rest.length }

       $scriptLine = $rest.Substring($lastEntryIndex, ($lastEntryFinish - $lastEntryIndex)).Trim()
       if($scriptLine -match 'DEBUG:[ \t]*([0-9]*)\+(.*)')
       {
           ## Pull out the line number from the line
           $last = $matches[1]
           
           $lastCounted = $last
           $samples[$last]++
           
           ## Pull out the actual script line that matches the line number
           $uniqueLines[$last] = $matches[2]
       }

       ## Discard anything that's buffered during this poll, and start waiting
       ## again
       $logReader.DiscardBufferedData()
    }

    ## Clean up
    $logStream.Close()
    $logReader.Close()
    
    return $samples
}

## Go through the transcript log to figure out which lines are part of any
## marked regions.  This returns a hastable that maps region names to
## the lines they contain.
function GenerateFunctionMembers
{
    ## Create a stack that represents the callstack.  That way, if a marked
    ## region contains another marked region, we attribute the statistics
    ## appropriately.
    $callstack = new-object System.Collections.Stack
    $currentFunction = "Unmarked"
    $callstack.Push($currentFunction)

    $functionMembers = @{}

    ## Go through each line in the transcript file, from the beginning
    foreach($line in (get-content $logFilePath))
    {
        ## Check if we're entering a monitor block
        ## If so, store that we're in that function, and push it onto
        ## the callstack.
        if($line -match 'write-debug "ENTER (.*)"')
        {
            $currentFunction = $matches[1]
            $callstack.Push($currentFunction)
        }
        ## Check if we're exiting a monitor block
        ## If so, clear the "current function" from the callstack,
        ## and store the new "current function" onto the callstack.
        elseif($line -match 'write-debug "EXIT"')
        {
            [void] $callstack.Pop()
            $currentFunction = $callstack.Peek()
        }
        ## Otherwise, this is just a line with some code.
        ## Add the line number as a member of the "current function"
        else
        {
            if($line -match 'DEBUG:[ \t]*([0-9]*)\+')
            {
                ## Create the arraylist if it's not initialized
                if(-not $functionMembers[$currentFunction])
                {
                    $functionMembers[$currentFunction] = new-object `
                        System.Collections.ArrayList
                }
                
                ## Add the current line to the ArrayList
                if(-not $functionMembers[$currentFunction].Contains($matches[1]))
                {
                    [void] $functionMembers[$currentFunction].Add($matches[1])
                }
            }
        }
    }
    
    return $functionMembers
}

. Main

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

A Legal Take on the JC Penny Price Gouging

AKA: Welcome Don McGowan to the Party :)  Don works in the Law and Corporate Affairs organization at Microsoft, and has always been a bright and illuminating voice in our internal discussions.

Case in point – a bunch of people internally are pretty upset that JC Penny is gouging customers by charging $800 for a vanilla Xbox  Premium system.  Where most people speculate, Don lays down this awesome summary of the situation:

burn-console digression: an MSH script profiler

We now have implemented a working fire effect in a form that’s extremely easy to read.  The intention of the code is clear, our functions are nice and modular, and the algorithms are clear.  However, we need to deal with some performance issues.  For that, we’ll take a small digression.

At this point, if performance is of utmost importance, we might consider simply porting our script to a compiled C# cmdlet.  However, we’ll continue this in MSH, applying highly-targeted optimizations to bring the performance up to an acceptable level.

burn-console.msh Part II - A working implementation

Now that we’ve generated the palette, the most complex part of the algorithm is actually behind us.  The remaining code implements the fire algorithm, and is fairly simple:

  1. Generate fire on the bottom row of the screen.
  2. Move the already existing fire upwards on the screen.  Rather than just move every cell of fire upwards, though, we take into account the fact that heat is always affected by nearby heat.  So the fire that we move upwards will actually be the average heat from its four neighboring cells.

Another technique that we implement is called “double buffering.”  If we were to update each cell on the screen as we compute it, the graceful effect of frame-by-frame animation would be destroyed.  It looks like the progressive text of a teleprompter, as compared to the subtle scrolling of credits at the end of a movie.  To combat this, we draw each new frame onto a pretend screen (which is one buffer.)  As soon as we’re finished our calculations, we bulk-update the real screen (which is the second buffer.)

Cracking Safes With Thermal Imaging

This came up on SecurityFocus a few days ago, and it’s extremely cool: cracking safes with thermal imaging.

The idea behind the attack is that your fingers transfer enough heat to the keypad to be visible for a significant amount of time afterwards.  The equipment costs around $10,000 – but that may be a pittance compared to how much the safe has inside.

http://lcamtuf.coredump.cx/tsafe/