Precision Computing

We now have implemented a working fire effect in a form that’s extremely easy to read.  The intention of the code is clear, our functions are nice and modular, and the algorithms are clear.  However, we need to deal with some performance issues.  For that, we'll take a small digression.

At this point, if performance is of utmost importance, we might consider simply porting our script to a compiled C# cmdlet.  However, we’ll continue this in MSH, applying highly-targeted optimizations to bring the performance up to an acceptable level.

When tackling performance problems, the first rule is to measure the problem.  I can’t stress this highly enough.  It stuns (and saddens) me when I see people parade around on an optimization binge without ever enlisting the help of a profiler, or other measurement tool.  Unless you can guide your optimization efforts with hard performance data, you are almost certainly directing them to the wrong spots.  Random cute performance hacks will quickly turn your code to garbage, often with no appreciable performance gain.  I’ve done my fair share of micro-optimization, but it’s always been under the guidance of hard data to support it. 

Now, let’s investigate sources of our performance problems by measuring them.

The easiest way is to probably comment out function calls to see their effect on our refresh rate.  If we comment out the calls to both updateBuffer and updateScreen in our Main function, then we get about 8,000 frames per second.  If we comment out only the call to updateBuffer, we get about 3.7 frames per second.  If we comment out only the call to updateScreen, we get about 0.54 frames per second.  We can continue to comment out chunks of code, slowly determining what causes the most drastic effect on our script’s performance.

However, that’s not very elegant.  It’s slow, and inefficient.  We should really rely on a profiler to provide us the information we need.  But MSH doesn’t have a script profiler.

Is that going to stop us?  Not in the least.  In fact, it will barely slow us down.  With a surprisingly small amount of bubble gum and handy twine, we’ll write our own – in MSH, of course.

There are two approaches one can take when writing a profiler.  The first is an instrumenting profiler, and the second is a sampling profiler.

Instrumenting profilers inject code at the beginning and end of every call to measure the time spent in that call.  The idea is fairly straight-forward, but the technique is fairly invasive.  The instrumentation process itself can skew the performance profile of a program, so the profiler needs to calibrate itself to account for that.  In addition, robust code instrumentation is extremely difficult.  Binary instrumentation is much easier, so nearly all instrumenting profilers take this approach.

Sampling profilers semi-randomly peek into a program to see what statement it is currently executing.  If we see the program executing a certain statement half of the times we check, we can be fairly confident that our program is spending about 50% of its time executing that statement.  Likewise, if we see the program executing a certain statement 8 out of the 50 times we check, we can be fairly confident that our program is spending about 16% of its time executing that statement.  When we apply this idea to a much larger sample set, we can fairly accurately map the execution time of each individual statement.

We’re going to primarily implement a sampling profiler for several reasons:

• There is no concept of an MSH Script Binary.  If we implement an instrumenting profiler, we have only the very perilous path of mangling the script itself.
• Monad has several bits of infrastructure available that make a sampling profiler surprisingly easy to implement.

So how do we peek in on the MSH engine to see which line of script it’s currently executing?  Very indirectly. 

1. Turn on MSH script tracing (at level 1,) as outlined as part of Jon Newman’s excellent series of posts.
   MSH:116 C:\Temp >set-mshdebug –trace 1
2. Turn on file logging for our session:
   MSH:117 C:\Temp >start-transcript
3. Start our script

Once the script starts, we get entries similar to this in the transcript log for our session:

(…)
DEBUG:  113+             $colour = $screenBuffer[$baseOffset]
DEBUG:  114+             $colour += $screenBuffer[$baseOffset - 1]
DEBUG:  115+             $colour += $screenBuffer[$baseOffset + 1]
DEBUG:  116+             $colour += $screenBuffer[$baseOffset +
$windowWidth]
(…)

Now, we can see what line of script the engine is currently executing by looking at the last bit of output that made it to the log.

Here is an example of the output – we’ll go over the code next time:

Breakdown by line:
----------------------------
 15%: Line  123 -             if($colour -lt 20) { $colour -= 1 }
 14%: Line  122 -             if($colour -le 70) { $colour -= 3 }
 11%: Line  124 -             if($colour -lt 0) { $colour = 0 }
 10%: Line  128 -             $tempWorkingBuffer[$baseOffset -
  8%: Line  117 -             $colour /= 4.0
  7%: Line  121 -             if($colour -gt 70) { $colour -= 1 }
  6%: Line  154 -             $nextScreen[$row, $column] = `
  6%: Line  113 -             $colour = $screenBuffer[$baseOffset]
  6%: Line  115 -             $colour += $screenBuffer[$baseOffset + 1]
  6%: Line  116 -             $colour += $screenBuffer[$baseOffset +
  5%: Line  114 -             $colour += $screenBuffer[$baseOffset - 1]
  5%: Line  109 -             $baseOffset = ($windowWidth * $row) +
  0%: Line   90 -         if($random.NextDouble() -ge 0.20)
  0%: Line  152 -         for($column = 0; $column -lt $windowWidth;
  0%: Line   94 -             $screenBuffer[($windowHeight - 2) *

Breakdown by marked regions:
----------------------------
  6%: updateScreen
  0%: startFireLastRow
 93%: propigate
  0%: Unmarked
  0%: main

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Now that we’ve generated the palette, the most complex part of the algorithm is actually behind us.  The remaining code implements the fire algorithm, and is fairly simple:

1. Generate fire on the bottom row of the screen.
2. Move the already existing fire upwards on the screen.  Rather than just move every cell of fire upwards, though, we take into account the fact that heat is always affected by nearby heat.  So the fire that we move upwards will actually be the average heat from its four neighboring cells.

Another technique that we implement is called “double buffering.”  If we were to update each cell on the screen as we compute it, the graceful effect of frame-by-frame animation would be destroyed.  It looks like the progressive text of a teleprompter, as compared to the subtle scrolling of credits at the end of a movie.  To combat this, we draw each new frame onto a pretend screen (which is one buffer.)  As soon as we’re finished our calculations, we bulk-update the real screen (which is the second buffer.)

The updated implementation is below, and also here: burn-console-1.working.msh.  The two new functions are updateBuffer and updateScreen, along with a little bit of new code in the main function.  The performance is atrocious, though.  I get about 0.4 frames per second on my computer, so we’ll work on improving that over the next two posts.  In fact, we’ll be in the mid 30s for frames per second by the time we’re done.

BTW – for a bit more visual realism, change your console font to either the 4x6 raster font, or the 5pt Lucida Console font.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

This came up on SecurityFocus a few days ago, and it's extremely cool: cracking safes with thermal imaging.

The idea behind the attack is that your fingers transfer enough heat to the keypad to be visible for a significant amount of time afterwards.  The equipment costs around $10,000 -- but that may be a pittance compared to how much the safe has inside.

http://lcamtuf.coredump.cx/tsafe/

As a deeper example of interfacing with the MshHostRawUserInterface class, the next few articles will deal with implementing the classic “Fire Effect” seen in many of the demos that I so fondly remember.  As part of the journey, we’ll also explore some of the ways in which you can improve the performance of time-sensitive portions of your script.

[This GIF is animated.  Some Ad blocking software restricts animation]

What is the fire effect?  It’s a relatively simple algorithm to simulate fire – usually in a graphical buffer.  We’re going to do it in the console text buffer.  In the algorithm, you seed the bottom row randomly with hot pixels.  Then, you progress the fire upwards.  As you progress the fire upwards, you cool it off a little bit – and simulate heat dissipation by averaging the heat from nearby buffer cells.

You can find an excellent write-up of the concepts at http://freespace.virgin.net/hugo.elias/models/m_fire.htm

The first problem we run into is properly representing the shades of the fire.  The fire effect traditionally uses 256 shades of red as its palette.  Colour 255 is the hottest, while colour 0 is the coldest.  That’s difficult when your fire-like console palette is restricted to four fire-like colours from [System.Console]::ConsoleColor – Yellow, Red, DarkRed, and Black.  To get around this problem, we’ll dither our colours.  Newspapers use dithering with great success: by carefully controlling the density in which they place dots of ink, they can simulate a passable grayscale.

Similarly, we can generate many new colours using the high ASCII characters ░ (ALT+176), ▒ (ALT+177), ▓ (ALT+178), and █ (ALT+219) to mix foreground and background colours.  For example, a DarkRed foreground and Black background, applied to character ▒ produces a very dark red. 

Now, visual design isn’t my core competency.  I’d be hard pressed to mix these four colours, using only the ratios implicit in the dithering characters, to create a fire-like palette.  So I’ll cheat, and let the computer do it for me.  I’ll assign a number to each colour that represents its luminosity / intensity, let the computer generate all possible mixes of colours and characters, and then have it organize the resulting mixes it order of total visual intensity.

The code below accomplishes that.  You can also download the script: burn-console-1.palette.msh.

###############################################################################
## burn-console.msh
##
## Create a fire effect in MSH, using the Console text buffer as the rendering
## surface.
##
## Great overview of the fire effect algorithm here: 
## http://freespace.virgin.net/hugo.elias/models/m_fire.htm
##
###############################################################################

function main
{
    ## Rather than a simple red fire, we'll introduce oranges and yellows
    ## by including Yellow as one of the base colours
    $colours = "Yellow","Red","DarkRed","Black"
    
    ## The four characters that we use to dither with, along with the 
    ## percentage of the foreground colour that they show
    $dithering = "█","▓","▒","░"
    $ditherFactor = 1,0.75,0.5,0.25
    
    ## Hold the palette.  We actually store each entry as a BufferCell,
    ## since we need to retain a foreground colour, background colour,
    ## and dithering character.
    $palette = `
        @(new-object System.Management.Automation.Host.BufferCell) * 256
    
    ## Generate, then display, the palette
    clear-host
    generatePalette
    displayPalette

    ## Clean up and exit
    $host.UI.RawUI.ForegroundColor = "Gray"
    $host.UI.RawUI.BackgroundColor = "Black"
}

## Generates a palette of 256 colours.  We create every combination of 
## foreground colour, background colour, and dithering character, and then
## order them by their visual intensity.
##
## The visual intensity of a colour can be expressed by the NTSC luminance 
## formula.  That formula depicts the apparent brightness of a colour based on 
## our eyes' sensitivity to different wavelengths that compose that colour.
## http://en.wikipedia.org/wiki/Luminance_%28video%29
function generatePalette
{
    ## The apparent intensities of our four primary colours.
    ## However, the formula under-represents the intensity of our straight
    ## red colour, so we artificially inflate it.
    $luminances = 225.93,106.245,38.272,0
    $apparentBrightnesses = @{}

    ## Cycle through each foreground, background, and dither character
    ## combination.  For each combination, find the apparent intensity of the 
    ## foreground, and the apparent intensity of the background.  Finally,
    ## weight the contribution of each based on how much of each colour the
    ## dithering character shows.
    ## This provides an intensity range between zero and some maximum.
    ## For each apparent intensity, we store the colours and characters
    ## that create that intensity.
    $maxBrightness = 0
    for($fgColour = 0; $fgColour -lt $colours.Count; $fgColour++)
    {
        for($bgColour = 0; $bgColour -lt $colours.Count; $bgColour++)
        {
            for($ditherCharacter = 0; 
                $ditherCharacter -lt $dithering.Count; 
                $ditherCharacter++)
            {
                $apparentBrightness = `
                    $luminances[$fgColour] * $ditherFactor[$ditherCharacter] +`
                    $luminances[$bgColour] * 
                        (1 - $ditherFactor[$ditherCharacter])
                    
                if($apparentBrightness -gt $maxBrightness) 
                { 
                    $maxBrightness = $apparentBrightness 
                }
                    
                $apparentBrightnesses[$apparentBrightness] = `
                    "$fgColour$bgColour$ditherCharacter"
            }
       }
    }

    ## Finally, we normalize our computed intesities into a pallete of
    ## 0 to 255.  If a given intensity is 30% towards our maximum intensity,
    ## then it should be in the palette at 30% of index 255.
    $paletteIndex = 0
    foreach($key in ($apparentBrightnesses.Keys | sort))
    {
        $keyValue = $apparentBrightnesses[$key]
        do
        {
            $character = $dithering[[Int32]::Parse($keyValue[2])]
            $fgColour = $colours[[Int32]::Parse($keyValue[0])]
            $bgColour = $colours[[Int32]::Parse($keyValue[1])]
            
            $bufferCell = `
                new-object System.Management.Automation.Host.BufferCell `
                    $character, `
                    $fgColour, `
                    $bgColour, `
                    "Complete"
                    
            $palette[$paletteIndex] = $bufferCell
            $paletteIndex++
        } while(($paletteIndex / 256) -lt ($key / $maxBrightness));
    }
}

## Dump the palette to the screen.
function displayPalette
{
    for($paletteIndex = 254; $paletteIndex -ge 0; $paletteIndex--)
    {
        $bufferCell = $palette[$paletteIndex]
        $fgColor = $bufferCell.ForegroundColor
        $bgColor = $bufferCell.BackgroundColor
        $character = $bufferCell.Character

        $host.UI.RawUI.ForegroundColor = $fgColor
        $host.UI.RawUI.BackgroundColor = $bgColor
        write-host -noNewLine $character
    }
    
    write-host
}

. main

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

For those of you looking to slake your thirst for Monad on something other than Blogs and Newsgroups, Andy Oakley’s book will soon be available from Amazon: http://www.amazon.com/gp/product/0596100094

It’s called “Monad – Introducing the MSH Command Shell and Language.”  I read it (several times) as one of the technical editors, and it’s a good piece of work.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

As your scripts become more complex, it’s possible that you will have to interact with the user in a manner more complex than write- and read-host.  For example, you might decide to implement a text-based menu system, or even a game of Aliens.

To support this low-level access to the console buffer, hosts (for example, ours,) extend the MshHostRawUserInterface class.  This class provides most of the common operations required by a console-mode raw user interface, including window management, blitting, scrolling, and colour control.

The following script demonstrates some of the ways in which you can work with Monad’s Raw UI.  It illustrates:

• Getting the coordinates of the viewport in relation to the window buffer
• Getting the current cursor location
• Obtaining direct access to the console’s buffer contents
• Scrolling the buffer’s contents
• Directly writing to the console’s buffer
• Working within the constraints of the console’s current width and height

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Hopefully you’ve been following along with Eric’s regular expression exercises, because we’re about to add another cool tool to your Monad toolbox.  If your regex-fu is strong, you will soon dice text streams with ease.

As you well know, one of the strongest features of Monad is that the pipeline is object-based.  You don’t waste your energy creating, destroying, and recreating the object representation of your data.  In past shells, you destroy the full- fidelity representation of data when the pipeline converts it to pure text.  You can regain some of it through excessive text parsing, but not all of it. 

However, we still often have to interact with low-fidelity input originating from outside of Monad.  Text-based data files and legacy programs are two examples.

If you’re used to searching through files with Grep, you’ve hopefully discovered Monad’s match-string cmdlet.  If you’re used to dynamically replacing patterns in a stream of text with Sed, you’ve hopefully discovered the [Regex]::Replace() method.  If you’re used to extracting text from a stream with Awk, you’ve hopefully discovered… [String]::Split()?  OK, it’s the best you have so far, but it gets better.

The following parse-textObject script allows you to convert many text streams into a meaningful object-based representation.  From there, you can use all of Monad’s powerful object-based filtering cmdlets as you would normally.

Here’s an example, using the output of a source control system we use at work.

MSH:48 D:\enlistment > sd opened
//depot/main/dirs#2 - edit default change (text)
//depot/main/output.txt#0 - add default change (text)
//depot/main/sdb.ini#1 - delete default change (text)
MSH:49 D:\enlistment >
MSH:49 D:\enlistment > $sdObjectDefinition = @("Path","Revision","Change","ChangeList","Type")
MSH:50 D:\enlistment > $sdFormat = "(.*)#([^ \t]+) - ([^ \t]+) (.*) \((.*)\)"
MSH:51 D:\enlistment > $results = sd opened | parse-textobject -ParseExpression:$sdFormat `
>> -ObjectDefinition:$sdObjectDefinition
>>

MSH:52 D:\enlistment > $results | format-table

Path                      Revision                  Change                    ChangeList                Type
----                      --------                  ------                    ----------                ----
//depot/main/dirs         2                         edit                      default change            text
//depot/main/output.txt   0                         add                       default change            text
//depot/main/sdb.ini      1                         delete                    default change            text

MSH:53 D:\enlistment > $results | where { $_.Revision -gt 1 }

Path       : //depot/main/dirs
Revision   : 2
Change     : edit
ChangeList : default change
Type       : text

And here's the script:

param([string] $parseExpression, [string[]] $propertyName, [type[]] $propertyType, [string] $delimiter, [switch] $unitTest)

################################################################################################
##
##    Parse-TextObject.ps1 -- Parse a simple string into a custom MshObject.
##
##    Parameters:
##        [switch] unitTest
##        Runs the unit tests.  Defaults to "false"
##
##        [string] delimiter
##        If specified, gives the .NET Regular Expression with which to split the string.
##        The script generates properties for the resulting object out of the elements resulting
##        from this split.
##        If not specified, defaults to splitting on the maximum amount of whitespace: "\s+",
##        as long as ParseExpression is not specified either.
##
##        [string] parseExpression
##        If specified, gives the .NET Regular Expression with which to parse the string.
##        The script generates properties for the resulting object out of the groups captured by
##        this regular expression.
##        
##        ** NOTE ** Delimiter and ParseExpression are mutually exclusive.
##
##        [string[]] propertyName
##        If specified, the script will pair the names from this object definition with the 
##        elements from the parsed string.  If not specified (or the generated object contains
##        more properties than you specify,) the script adds notes in the pattern of
##        Property1,Property2,...,PropertyN
##
##        [type[]] propertyType
##        If specified, the script will pair the types from this list with the properties
##        from the parsed string.  If not specified (or the generated object contains
##        more properties than you specify,) the script sets the properties to be of type [string]
##
##
##    Example usage:
##        "Hello World" | parse-textobject
##        Generates an Object with "Property1=Hello" and "Property2=World"
##
##        "Hello World" | parse-textobject -delimiter "ll"
##        Generates an Object with "Property1=He" and "Property2=o World"
##
##        "Hello World" | parse-textobject -parseExpression "He(ll.*o)r(ld)"
##        Generates an Object with "Property1=llo Wo" and "Property2=ld"
##
##        "Hello World" | parse-textobject -propertyName FirstWord,SecondWord
##        Generates an Object with "FirstWord=Hello" and "SecondWord=World
##
##        "123 456" | parse-textobject -propertyType $([string],[int])
##        Generates an Object with "Property1=123" and "Property2=456"
##              These properties are integers, as opposed to strings
##
##
################################################################################################

function Main($inputObjects, $parseExpression, $propertyType, $propertyName, $delimiter, $unitTest)
{
    if($unitTest -eq $true)
    {
        UnitTest
        ""
    } 
    else 
    {
        $delimiterSpecified = [bool] $delimiter
        $parseExpressionSpecified = [bool] $parseExpression

        ## If they've specified both ParseExpression and Delimiter, show usage
        if($delimiterSpecified -and $parseExpressionSpecified)
        {
            Usage
            return
        }
        
        ## If they enter no parameters, assume a default delimiter of spaces
        if(-not $($delimiterSpecified -or $parseExpressionSpecified))
        {
            $delimiter = "\s+"
            $delimiterSpecified = $true
        }
        
        ## Cycle through the $inputObjects, and parse it into objects
        foreach($inputObject in $inputObjects)
        {
                        if(-not $inputObject) { $inputObject = "" }
            foreach($inputLine in $inputObject.ToString())
            {
                ParseTextObject $inputLine $delimiter $parseExpression $propertyType $propertyName
            }
        }
    }
}

function Usage
{
    "Usage: "
    " parse-textobject"
    " parse-textobject -unitTest"
    " parse-textobject -objectDefinition objectDefinition"
    " parse-textobject -parseExpression parseExpression -propertyName objectDefinition"
    " parse-textobject -delimiter delimiter -propertyName objectDefinition"
    return
}

## Function definition -- ParseTextObject.
## Perform the heavy-lifting -- parse a string into its components.
## for each component, add it as a note to the mshObject that we return
function ParseTextObject
{
    $textInput = $args[0]
    $delimiter = $args[1]
    $parseExpression = $args[2]
        $propertyTypes = $args[3]
    $propertyNames = $args[4]
    
    $parseExpressionSpecified = -not $delimiter
    
    $returnObject = new-mshobject
    
    $matches = $null
    $matchCount = 0;
    if($parseExpressionSpecified)
    {
        ## Populates the matches variable by default
        [void] ($textInput -match $parseExpression)
        $matchCount = $matches.Count
    }
    else
    {
        $matches = [Regex]::Split($textInput, $delimiter)
        $matchCount = $matches.Length
    }
    
    $counter = 0
    if($parseExpressionSpecified) { $counter++ }
    for(; $counter -lt $matchCount; $counter++)
    {
        $propertyName = "None"
                $propertyType = [string]

        
        ## Parse by Expression
        if($parseExpressionSpecified)
        {
            $propertyName = "Property$counter"
            
                        ## Get the property name
            if($counter -le $propertyNames.Length)
            {
                if($propertyName[$counter - 1])
                {
                    $propertyName = $propertyNames[$counter - 1] 
                }
            }

                        ## Get the property value
            if($counter -le $propertyTypes.Length)
            {
                if($types[$counter - 1])
                {
                    $propertyType = $propertyTypes[$counter - 1] 
                }
            }

        }
        ## Parse by delimiter
        else
        {
            $propertyName = "Property$($counter + 1)"
            
                        ## Get the property name
            if($counter -lt $propertyNames.Length) 
            {
                if($propertyNames[$counter])
                {
                    $propertyName = $propertyNames[$counter] 
                }
            }

                        ## Get the property value
            if($counter -lt $propertyTypes.Length)
            {
                if($propertyTypes[$counter])
                {
                    $propertyType = $propertyTypes[$counter] 
                }
            }
        }