Precision Computing

Hopefully you’ve been following along with Eric’s regular expression exercises, because we’re about to add another cool tool to your Monad toolbox.  If your regex-fu is strong, you will soon dice text streams with ease.

As you well know, one of the strongest features of Monad is that the pipeline is object-based.  You don’t waste your energy creating, destroying, and recreating the object representation of your data.  In past shells, you destroy the full- fidelity representation of data when the pipeline converts it to pure text.  You can regain some of it through excessive text parsing, but not all of it. 

However, we still often have to interact with low-fidelity input originating from outside of Monad.  Text-based data files and legacy programs are two examples.

If you’re used to searching through files with Grep, you’ve hopefully discovered Monad’s match-string cmdlet.  If you’re used to dynamically replacing patterns in a stream of text with Sed, you’ve hopefully discovered the [Regex]::Replace() method.  If you’re used to extracting text from a stream with Awk, you’ve hopefully discovered… [String]::Split()?  OK, it’s the best you have so far, but it gets better.

The following parse-textObject script allows you to convert many text streams into a meaningful object-based representation.  From there, you can use all of Monad’s powerful object-based filtering cmdlets as you would normally.

Here’s an example, using the output of a source control system we use at work.

MSH:48 D:\enlistment > sd opened
//depot/main/dirs#2 - edit default change (text)
//depot/main/output.txt#0 - add default change (text)
//depot/main/sdb.ini#1 - delete default change (text)
MSH:49 D:\enlistment >
MSH:49 D:\enlistment > $sdObjectDefinition = @("Path","Revision","Change","ChangeList","Type")
MSH:50 D:\enlistment > $sdFormat = "(.*)#([^ \t]+) - ([^ \t]+) (.*) \((.*)\)"
MSH:51 D:\enlistment > $results = sd opened | parse-textobject -ParseExpression:$sdFormat `
>> -ObjectDefinition:$sdObjectDefinition
>>

MSH:52 D:\enlistment > $results | format-table

Path                      Revision                  Change                    ChangeList                Type
----                      --------                  ------                    ----------                ----
//depot/main/dirs         2                         edit                      default change            text
//depot/main/output.txt   0                         add                       default change            text
//depot/main/sdb.ini      1                         delete                    default change            text

MSH:53 D:\enlistment > $results | where { $_.Revision -gt 1 }

Path       : //depot/main/dirs
Revision   : 2
Change     : edit
ChangeList : default change
Type       : text

And here's the script:

param([string] $parseExpression, [string[]] $propertyName, [type[]] $propertyType, [string] $delimiter, [switch] $unitTest)

################################################################################################
##
##    Parse-TextObject.ps1 -- Parse a simple string into a custom MshObject.
##
##    Parameters:
##        [switch] unitTest
##        Runs the unit tests.  Defaults to "false"
##
##        [string] delimiter
##        If specified, gives the .NET Regular Expression with which to split the string.
##        The script generates properties for the resulting object out of the elements resulting
##        from this split.
##        If not specified, defaults to splitting on the maximum amount of whitespace: "\s+",
##        as long as ParseExpression is not specified either.
##
##        [string] parseExpression
##        If specified, gives the .NET Regular Expression with which to parse the string.
##        The script generates properties for the resulting object out of the groups captured by
##        this regular expression.
##        
##        ** NOTE ** Delimiter and ParseExpression are mutually exclusive.
##
##        [string[]] propertyName
##        If specified, the script will pair the names from this object definition with the 
##        elements from the parsed string.  If not specified (or the generated object contains
##        more properties than you specify,) the script adds notes in the pattern of
##        Property1,Property2,...,PropertyN
##
##        [type[]] propertyType
##        If specified, the script will pair the types from this list with the properties
##        from the parsed string.  If not specified (or the generated object contains
##        more properties than you specify,) the script sets the properties to be of type [string]
##
##
##    Example usage:
##        "Hello World" | parse-textobject
##        Generates an Object with "Property1=Hello" and "Property2=World"
##
##        "Hello World" | parse-textobject -delimiter "ll"
##        Generates an Object with "Property1=He" and "Property2=o World"
##
##        "Hello World" | parse-textobject -parseExpression "He(ll.*o)r(ld)"
##        Generates an Object with "Property1=llo Wo" and "Property2=ld"
##
##        "Hello World" | parse-textobject -propertyName FirstWord,SecondWord
##        Generates an Object with "FirstWord=Hello" and "SecondWord=World
##
##        "123 456" | parse-textobject -propertyType $([string],[int])
##        Generates an Object with "Property1=123" and "Property2=456"
##              These properties are integers, as opposed to strings
##
##
################################################################################################

function Main($inputObjects, $parseExpression, $propertyType, $propertyName, $delimiter, $unitTest)
{
    if($unitTest -eq $true)
    {
        UnitTest
        ""
    } 
    else 
    {
        $delimiterSpecified = [bool] $delimiter
        $parseExpressionSpecified = [bool] $parseExpression

        ## If they've specified both ParseExpression and Delimiter, show usage
        if($delimiterSpecified -and $parseExpressionSpecified)
        {
            Usage
            return
        }
        
        ## If they enter no parameters, assume a default delimiter of spaces
        if(-not $($delimiterSpecified -or $parseExpressionSpecified))
        {
            $delimiter = "\s+"
            $delimiterSpecified = $true
        }
        
        ## Cycle through the $inputObjects, and parse it into objects
        foreach($inputObject in $inputObjects)
        {
                        if(-not $inputObject) { $inputObject = "" }
            foreach($inputLine in $inputObject.ToString())
            {
                ParseTextObject $inputLine $delimiter $parseExpression $propertyType $propertyName
            }
        }
    }
}

function Usage
{
    "Usage: "
    " parse-textobject"
    " parse-textobject -unitTest"
    " parse-textobject -objectDefinition objectDefinition"
    " parse-textobject -parseExpression parseExpression -propertyName objectDefinition"
    " parse-textobject -delimiter delimiter -propertyName objectDefinition"
    return
}

## Function definition -- ParseTextObject.
## Perform the heavy-lifting -- parse a string into its components.
## for each component, add it as a note to the mshObject that we return
function ParseTextObject
{
    $textInput = $args[0]
    $delimiter = $args[1]
    $parseExpression = $args[2]
        $propertyTypes = $args[3]
    $propertyNames = $args[4]
    
    $parseExpressionSpecified = -not $delimiter
    
    $returnObject = new-mshobject
    
    $matches = $null
    $matchCount = 0;
    if($parseExpressionSpecified)
    {
        ## Populates the matches variable by default
        [void] ($textInput -match $parseExpression)
        $matchCount = $matches.Count
    }
    else
    {
        $matches = [Regex]::Split($textInput, $delimiter)
        $matchCount = $matches.Length
    }
    
    $counter = 0
    if($parseExpressionSpecified) { $counter++ }
    for(; $counter -lt $matchCount; $counter++)
    {
        $propertyName = "None"
                $propertyType = [string]

        
        ## Parse by Expression
        if($parseExpressionSpecified)
        {
            $propertyName = "Property$counter"
            
                        ## Get the property name
            if($counter -le $propertyNames.Length)
            {
                if($propertyName[$counter - 1])
                {
                    $propertyName = $propertyNames[$counter - 1] 
                }
            }

                        ## Get the property value
            if($counter -le $propertyTypes.Length)
            {
                if($types[$counter - 1])
                {
                    $propertyType = $propertyTypes[$counter - 1] 
                }
            }

        }
        ## Parse by delimiter
        else
        {
            $propertyName = "Property$($counter + 1)"
            
                        ## Get the property name
            if($counter -lt $propertyNames.Length) 
            {
                if($propertyNames[$counter])
                {
                    $propertyName = $propertyNames[$counter] 
                }
            }

                        ## Get the property value
            if($counter -lt $propertyTypes.Length)
            {
                if($propertyTypes[$counter])
                {
                    $propertyType = $propertyTypes[$counter] 
                }
            }
        }

                add-note $returnObject $propertyName ($matches[$counter] -as $propertyType)
    }
    
    $returnObject
}


## Create a new mshObject
function new-mshobject 
{
    new-object management.automation.PsObject
}

## Add a note to a mshObject
function add-note ($object, $name, $value) 
{
    $object | add-member NoteProperty $name $value
}

## Unit testing helper
function Assert
{
    $message = $args[0]
    $test = $args[1]

    if($test -eq $false)
    {
        write-host "`n$message"
    }
    else
    {
        write-host -NoNewLine "."
    }
}

## Unit tests
function UnitTest
{
    ## Mutually Exclusive
    $return = parse-textobject -Delimiter:"aoe" -ParseExpression:"oeu"
    Assert "Should have received a usage message" $($return[0].IndexOf("Usage:") -eq 0)
    
    ## Custom Split
    $return = "Hello World" | parse-textobject -ParseExpression:"(.*) (.*)" -PropertyName:First,Second
    Assert "return-First should be 'Hello'" $($return.First -eq "Hello")
    Assert "return-Second should be 'World'" $($return.Second -eq "World")

    ## Custom Split, PropertyName overflow
    $return = "Hello World" | parse-textobject -ParseExpression:"(.*) (.*)" -PropertyName:First,Second,Third
    Assert "return-First should be 'Hello'" $($return.First -eq "Hello")
    Assert "return-Second should be 'World'" $($return.Second -eq "World")

    ## Custom Split Single
    $return = "Hello" | parse-textobject -ParseExpression:"(.*)" -PropertyName:All
    Assert "return-All should be 'Hello'" $($return.All -eq "Hello")
    
    ## No Object Definition, parseExpression
    $return = "Hello World" | parse-textobject -ParseExpression:"(.*) (.*)"
    Assert "return-Property1 should be 'Hello'" $($return.Property1 -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")

    ## Insufficient Object Definition, parseExpression
    $return = "Hello World" | parse-textobject -ParseExpression:"(.*) (.*)" -PropertyName:Hello
    Assert "return-Hello should be 'Hello'" $($return.Hello -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")

    ## Insufficient Object Definition, parseExpression, with comma
    $return = "Hello World" | parse-textobject -ParseExpression:"(.*) (.*)" -PropertyName:Hello
    Assert "return-Hello should be 'Hello'" $($return."Hello" -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")
    
    ## Delimited split
    $return = "Hello World" | parse-textobject -Delimiter:"[ \t]+" -PropertyName:First,Second
    Assert "return-First should be 'Hello'" $($return.First -eq "Hello")
    Assert "return-Second should be 'World'" $($return.Second -eq "World")

    ## Delimited split, object definition overflow
    $return = "Hello World" | parse-textobject -Delimiter:"[ \t]+" -PropertyName:First,Second,Third
    Assert "return-First should be 'Hello'" $($return.First -eq "Hello")
    Assert "return-Second should be 'World'" $($return.Second -eq "World")
    
    ## No Object Definition, delimited
    $return = "Hello World" | parse-textobject -Delimiter:"[ \t]+"
    Assert "return-Property1 should be 'Hello'" $($return.Property1 -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")
    
    ## Insufficient Object Definition, delimited
    $return = "Hello World" | parse-textobject -Delimiter:"[ \t]+" -PropertyName:Hello
    Assert "return-Hello should be 'Hello'" $($return.Hello -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")

    ## Insufficient Object Definition, delimited, with comma
    $return = "Hello World" | parse-textobject -Delimiter:"[ \t]+" -PropertyName:Hello
    Assert "return-Hello should be 'Hello'" $($return.Hello -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")
    
    ## Header Examples
    $return = "Hello World" | parse-textobject
    Assert "return-Property1 should be 'Hello'" $($return.Property1 -eq "Hello")
    Assert "return-Property2 should be 'World'" $($return.Property2 -eq "World")
    
    $return = "Hello World" | parse-textobject -Delimiter "ll"
    Assert "return-Property1 should be 'He'" $($return.Property1 -eq "He")
    Assert "return-Property2 should be 'o World'" $($return.Property2 -eq "o World")

    $return = "Hello World" | parse-textobject -ParseExpression "He(ll.*o)r(ld)"
    Assert "return-Property1 should be 'llo Wo'" $($return.Property1 -eq "llo Wo")
    Assert "return-Property2 should be 'ld'" $($return.Property2 -eq "ld")

    $return = "Hello World" | parse-textobject -PropertyName FirstWord,SecondWord
    Assert "return-FirstWord should be 'Hello'" $($return.FirstWord -eq "Hello")
    Assert "return-SecondWord should be 'World'" $($return.SecondWord -eq "World")

        $return = "123 456" | parse-textobject -PropertyType $([string],[int])
    Assert "return-Property1 should be '123'" $($return.Property1 -eq "123")
    Assert "return-Property2 should be '456'" $($return.Property2 -eq 456)
        Assert "return-Property2 should be [int]" $($return.Property2 -is [int])


        ## Bug fix
    $return = $null | parse-textobject
    Assert "return-Property1 should be ''" $($return.Property1 -eq "")

        ## Parses with types
    $return = "Hello 1234" | parse-textobject -PropertyType $([string],[int])
    Assert "return-Property1 should be 'Hello'" $($return.Property1 -eq "Hello")
    Assert "return-Property2 should be '1234'" $($return.Property2 -eq 1234)
        Assert "return-Property2 * 2 should be '2468'" $(($return.Property2 * 2) -eq 2468)

        ## Type overflow, extras default to string
    $return = "1234 5678" | parse-textobject -PropertyType $([int])
    Assert "return-Property1 should be '1234'" $($return.Property1 -eq 1234)
    Assert "return-Property2 should be '5678'" $($return.Property2 -eq "5678")
        Assert "return-Property1 * 2 should be '2468'" $(($return.Property1 * 2) -eq 2468)
        Assert "return-Property2 * 2 should be '56785678'" $(($return.Property2 * 2) -eq "56785678")
}

Main $input $parseExpression $propertyType $propertyName $delimiter $unitTest

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

[Edit 2: Updated script to work with new builds]
[Edit 3: Updated script to add type constraints, and consistent parameter naming]

Somebody I work with recently tried to buy a camera from an internet retailer called “WaWaDigital.”  He was telling me about the low price they had it listed for, when the guy called him to confirm the order.

At this point, I recognized what was going on.  For some reason, there is an entire cottage industry (in the little cottage called Brooklyn, New York) that sells the camera for below cost, but cancels the order on you unless you buy hundreds of dollars of over-priced accessories.

As the phone call progresses, it’s like watching a really cheesy horror film play out.  It’s bad, but funny in its predictability.

“No, thanks, I have all of the storage cards I need.”
Mumble mumble on-the-other-end-of-a-cell-phone-mumble
“The batteries only last for 2 pictures, do they?  No thanks, I have all the batteries I need.”
Mumble mumble don’t be so damn cheap mumble mumble

At this point, he’s walking into my office, with the expression on his face of, “You wouldn’t believe the phone call I’m on right now.”  I can pretty much hear both sides of the conversation now, and the next thing I was expecting to hear was that the order was canceled.  Instead, it continues…

“Don’t be cheap?  Listen.  That is ridiculous customer service.  Cancel my order and goodbye.”
“Cancel your order?  You really want to pay the 30% restocking fee for canceling your order?”
“What restocking fee?  There is no order, and you’re not going to charge me one.”
“Oh yes I will.”
“Go ahead, and I’ll dispute the charges on my card and it’ll cost you even more.”

Then, my jaw literally drops, as I scramble to help him record the phone call.

“Don’t you even dare.  You do that, and I’ll break your neck.  You hear me?  I’ll come there and break your f-----g neck.  I’ll …”

“Goodbye.”, and he hangs up.

As we get ready to record the return call, he calls back, but we let it dump into voice mail.  Evidently, intelligence tests are not part of “new hire training,” as this guy leaves a death threat on voice mail with an unblocked number.  [41 seconds, 665 kb].  (Warning, contains swears.)

Transcript:

..(718) 627-7192
Received at 3:22 pm

You better not pick up, b---h.  I’m gonna to come down there and break your god damn neck.  You heard me, alright?  Kid, you better hear me, b---h.  Do you hear me, B---H?  Yes, you’d better believe it.  You’re in biiiig trouble, my friend.

<click>

Wawdigital.com (and its pseudonyms, Stop4Camera.com, Starlight Cameras, Stargate Photo, and probably half a dozen others) is a complete and utter scam.  Such a poorly-run scam that their websites often contain text from the last domain name they had to abandon:

“…  wawadigital is not amanufacturer, but we are willing to replace a defective item”
http://www.stargatephoto.com/shopping/Policies&Services.asp

“Stop4Camera expressly disclaims all warranties…”
http://www.stargatephoto.com/shopping/LegalDisclaimer.asp

“Stop4Camera reserves the right to refuse …”
http://www.wawadigital.com/shopping/LegalDisclaimer.asp

I’m still agape at the absurdity of this all.  It’s amazing that any of these places are still in existence.  I don’t know their owners can avoid jail.  When you search the internet for the various incarnations of WaWaDigital, people have already reported them to nearly every authority I can think of.

Something tells me that JD Power won’t come-a-callin’ for WaWaDigital.com.

To make matters worse, the guy at work said he found good reviews about them at this site: http://www.everyprice.com.  Unbelievably, that site is a scam, too.  A meta-scam that must turn a blind-eye to (or simply not include) negative reviews: http://www.everyprice.com/merchant.asp?merchant_id=36&view=reviews.  Most of their many categories simply go off to the equivalent Amazon category.  But navigate down to anything that stays on their site.  For example:

Home & Garden -> Massage Recliner -> (Pick any.)

The top merchants are:
- IBuyPlasma.com
- CentralDigital.com
- DBuys.com
- RealDealShop.com

Now, I do my standard searches for these retailers – include their name, along with sundry expletives:

- http://www.google.com/search?q=ibuyplasma+scam
- http://www.google.com/search?q=centraldigital.com+scam
- http://www.google.com/search?q=dbuys.com+scam
- http://www.google.com/search?q=realdealshop.com+scam

Not a single legitimate shop.  Yet they have good reviews on the site.  You find good reviews everywhere for these scam retailers – they must have a full-time staff that runs around the internet posting fake reviews.

Thank goodness for the trusty “sundry expletive” search I rely on, instead.

One very significant problem in computer security is the "Dancing Pigs" syndrome [1][2]:

If J. Random Websurfer clicks on a button that promises dancing pigs on his computer monitor, and instead gets a hortatory message describing the potential dangers of the applet — he's going to choose dancing pigs over computer security any day. If the computer prompts him with a warning screen like: "The applet DANCING PIGS could contain malicious code that might do permanent damage to your computer, steal your life's savings, and impair your ability to have children," he'll click OK without even reading it. Thirty seconds later he won't even remember that the warning screen even existed.

Today on an internal mailing list, the following brilliant suggestion came up proposing a solution:

From: <A person>
Sent: Tuesday, November 08, 2005 12:20 PM
To: <Lots of people>
Subject: <Some security topic>

Perhaps if we shipped Vista with Dancing Pigs, then users could watch the pre-installed Dancing Pigs instead of being enticed to download/install potentially malicious new Dancing Pigs.

- <A person>

-----Original Message-----
From: <Another person>
Sent: Tuesday, November 08, 2005 12:14 PM
To: <Lots of people>
Subject: <Some security topic>

The underlying problem is not the autorun functionality, although it doesn't help.  Mark Russinovich would have gotten infected with that rootkit no matter what because he wanted to listen to the Dancing Pigs CD.  Running as LUA wouldn't help if the user were allowed to elevate (using XP RunAs or Vista UAP).  The kids create CDs or carry thumb drives or whatever and then perform whatever actions are needed to listen to or watch the Dancing Pigs.

-– <Another person>

-----Original Message-----
<Several dozen pages of email snipped>

A fellow Microsoftie, “Jim of Seattle,” has been contributing pieces to songfight.org since 2003.  His most recent piece, “Welcome to ______,” adds another landslide win to his list.  (That’s 6 dashes, if you care.)  He takes the simple Windows XP startup theme, and then builds a nice classical melody around it.  Check it out – it’s truly cool.

He battles a powerful demon in this song – that being the strongly charged musical idiom.  In language, an idiom is “… an expression whose meaning is not compositional – that is, whose meaning does not follow from the meaning of the individual words of which it is composed” [Wikipedia]  A musical idiom is the musical equivalent – short advertising themes are probably the most common example.  He uses many of the default sounds from Windows – songs which are strongly tied to our computing instincts in many cases.  Without introducing the sounds carefully, it’s easy for a listener to get jarred from the mood of a piece because of an overwhelming urge to see what error dialog just popped up.

Opera companies staging Wagner’s “Lohengrin” need to be especially careful of this phenomenon.  The third act has the overly-familiar “Wedding March” – a song that brings so much more to the table now than it did when it was written.  In fact, many productions drop the song altogether to avoid having the 20th century associations invade the carefully crafted storyline.

As a side-note, this song finally helped me figure out that the theme is:

D# A# G# D# A#

This is a “life hack” I’ve been meaning to share since I designed my very first GeoCities homepage, and I figure that now’s a good time to get off of my laurels and do it.  That is, memory and memorization techniques that I’ve found helpful through time.  Not that I’ve invented them, but that I use them.

[ Apple, Pencil, Fox, Anvil ]
- a list -

One difficult recall task we are often faced with is memorizing a list of things.  Remembering a shopping list.  Retaining a “to do” list.  However, the only reason it is difficult is that you haven’t (yet!) found a way to make it concrete, and less abstract.

One of the things I was always impressed with, in my many years as a Starbucks supervisor, was how easy it was to memorize the drinks of regular customers.  Some people were bad at it – the people who didn’t really interact with the customers, or get to know them.  However, when you do take the time to learn more about your regulars, you start associating their favourite drink with them automatically.  It’s no longer an arbitrary mapping of “Fred” to “Double Tall Ristretto Vanilla Non-fat Extra Hot With Whip Latte.”  Instead, you recall making their drink as you talked about their troubles at work.  Or laughing about how the non-fat gives them permission for the whipped cream.  It’s a little story that builds up to something that you relate to almost unconsciously.  After awhile, you just start associating drinks to people, without even having drink-based stories underpinning your memory.

Now how does this relate to memorizing lists?

It relates because you use the same technique.  Turn the list from something abstract to something that you can relate to.  The secret is to link each item with the next item through a vivid and dynamic story.  With this technique, you’ll be able to memorize gigantic lists nearly effortlessly.  And as a cool parlour trick, you’ll memorize them in order – and be able to recite them both forward and backward.

Start with the first item. Paint a vivid picture of it in your mind, adding motion if possible.  Visualize all of the detail you can.  Then, incorporate the next item into your scene.  Form a strong association between the two items, again using vivid imagery and motion.  Then, drop the first item, and form a strong link between the second and third items.  Continue this pattern of associating two at a time until you reach the end of your list.

Let’s memorize the list above.

You’re standing in the produce section of Safeway, looking at the gently slanting apple trays.  You pick up an enormous apple, marveling at its deep red luster.  You turn it around in your hands, looking for any imperfections.  Your mouth waters in anticipation.  Satisfied that you’ve selected the best apple, you begin to put it away. 

Then, out of nowhere, you hear the singing zip of an arrow flying from a bow.  You feel your hand jerk, and look down in surprise.  Splashes of apple juice decorate your arms and shirt as a sharp yellow pencil settles deep into the apple.

You’re walking down a country road, marveling at the bright yellow pencil you got from school.  The smell of freshly cut grass wafts through the air.  The sound of clucking chickens draws your gaze to the right.  Hens flap wildly, as you notice a fox attempting to open their gate.  The gate is locked, so he’s not having much luck.  In desperation, the fox calls out to you, “Hey, mind if I borrow your pencil?  I’m trying to pick this lock.”

As you approach him, holding out your pencil, and anvil drops from the sky, smiting the fox for his evil ways.

I’ll bet you already have the list memorized.  And I bet you’ll still remember it next week.

As Adam points out in his most recent entry, our last release changed the default execution policy to a mode called "Restricted."  The first time you run a script in the new shell, you’ll see the following error message:

The file C:\my_script.msh cannot be loaded. The execution of scripts
is disabled on this system. Please see "get-help about_signing" for
more details.

This is probably not the mode you want to stay with, as it doesn’t run those awesome scripts you write.  Yeah, that’s right, YOUR awesome scripts.  Remember how nice I’m being to you, as I’m about to ask a favour :)

This has probably been one of the most difficult features we’ve implemented, but the reasons are not technical.  It’s difficult, because we know that it makes the shell nearly unusable out of the box.  It’s difficult, because we know we’ve essentially added a manual step to the automatic installation of Monad.

Why?  It’s for your sake, and for the safety of the ecosystem. 

In today’s malicious environment, certain software categories need to be held to a higher standard of security.  For these components, secure by default usually means unusable by default.  Or in other words, “some assembly required.” 

By default, firewalls block nearly all inbound traffic.  IIS serves only static HTML.  Internet Explorer’s “enhanced security configuration” visits only trusted sites.  Outlook disables all dynamic content, images, active links, and optionally even all HTML.  Taken to the extreme, consider the story of OpenBSD:

The open software development model has allowed the organisation to upgrade with an uncompromising view to security enhancement - anything that stands in the way of a more secure environment has been savagely pruned, the result being a clean, slippery system with a surface too tight for conventional breach.

How does this help you?  For one thing, we hope it makes it less likely that your computer administrator at work reflexively restricts you from using MSH altogether.

Getting back to the title of the post, how do you get out of this restricted mode, and which execution policy should you pick?  This is answered by a document included in the distribution zip file, “about_signing.help.txt.”  Although this is a help file for the product, we weren’t able to include it in the actual installer in time for release.  To make it work like the rest of the about_* help files, copy it to your MSH installation directory.

I personally set my execution policy to AllSigned, although most will probably want to run under the RemoteSigned policy that earlier drops were configured to use.  However, security decisions always involve tradeoffs.  Please read the about_signing documentation to understand the implications that your execution policy has.

As you install this new version of Monad, almost all of you will change your execution policy.  Some of you will also blog your installation experience.  A subset of you “installing bloggers” will change it to “Unrestricted,” based on your personal evaluation of its tradeoffs.  I honestly think you should run in RemoteSigned mode instead, but it’s not my job to convince you.  However, most of you ran under the RemoteSigned execution policy for the past several months without even realizing it existed.

Now, it’s time for me to ask that favour: please let your readership make their own decision about an execution policy, rather than suggest they run in Unrestricted mode.  They might not want to make the same security decision you have, but might not realize there is any other alternative.  The documentation provides the background they need to make an informed choice.

Also, I’d like to express my thanks to our Monad MVPs that provided feedback on an earlier piece I had written to summarize the Monad execution policies.  Since our documentation team didn’t have the time to write about_signing, it was written by yours truly.  Your feedback was extremely valuable in helping me craft the public documentation on the subject.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Many thanks to Cyrus for pointing this out.  Videos from the PDC are now available for streaming via http://microsoft.sitestream.com/PDC05/.  The site is running rather slowly right now (it's not being Slashdotted, that's for sure,) so bring a bag of patience with you.

I, of course, recommend you start with Jeffrey and Jim's Monad presentation (during which I was Proctoring at the hands-on-labs.)  As Jeffrey quotes, "This is not your Dad's filesystem".  Then, if you haven't already, check out this year's belle of the ball, Linq :)

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I'm sure you've noticed by now.  One of the things you end up doing a lot in shells and scripting languages is parsing text.  We've done it many times in articles on this blog.  We've seen it in many articles on other blogs.  We've all sat in quiet reverence of the wondrous power that is Regular Expressions.

And you want to be part of the Regex Illuminati.

Eric has just started a series of exercises on Learning Regular Expresions. In his words, this is

"The first in a series of exercises designed to teach you more about regular expressions, written by a guy who got partway through writing a regex book."

I find Eric's "Regular Expression Workbench" to be invaluable in working with regular expressions.  I think you'll enjoy (or at least benefit from!) this series of articles.

Pay close attention -- I'll be blogging a script in the near future that will turn your regular expression skills into Real Ultimate Power.  Regular expressions are cool; and by cool, I mean totally sweet.

Since the first Channel 9 interviews were so popular, Jeffrey recently gave another interview on Channel 9.  With the intentions of talking about the PDC and the CLR, he ended up talking about nearly everything :)

Check it out! http://channel9.msdn.com/Showpost.aspx?postid=127819

Marcel has been posting some interesting articles on using Monad to generate the MD5 hashes of files.  Now, an MD5 hash of a file is just an array of bytes.  Typical hashing programs display this in a more friendly manner:

MSH:15 C:\Temp >md5sum 71-59-B7.bmp
a05805e638741bb767f97c0e88962952 *71-59-B7.bmp

Although the output of Marcel’s scripts could definitely be crafted to display this output, they currently output the string representation of a byte array:

MSH:19 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp)
160 88 5 230 56 116 27 183 103 249 124 14 136 150 41 82

One of the comments in response to Marcel’s post was that Monad should, by default, output byte arrays as hex.  This is a good suggestion, and we can go even further with it.  Let’s write a script to give us a full hex editor-like view of a byte array:

MSH:20 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp) | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   A0 58 05 E6 38 74 1B B7 67 F9 7C 0E 88 96 29 52   X.æ8t.•gù|.??)R

Or even better, let’s use it to dump out a very small bitmap – 10 pixels of the colour (R=0x71 G=0x59 B=0xB7)

MSH:21 C:\Temp >get-content 71-59-B7.bmp -encoding byte | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   42 4D 5E 00 00 00 00 00 00 00 36 00 00 00 28 00  BM^.......6...(.
00000010   00 00 0A 00 00 00 01 00 00 00 01 00 20 00 00 00  ............ ...
00000020   00 00 00 00 00 00 C4 0E 00 00 C4 0E 00 00 00 00  ......Ä...Ä.....
00000030   00 00 00 00 00 00 B7 59 71 FF B7 59 71 FF B7 59  ......•Yq.•Yq.•Y
00000040   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF B7 59  q.•Yq.•Yq.•Yq.•Y
00000050   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF        q.•Yq.•Yq.•Yq.

To make it easier to determine byte offsets, files are usually broken down into 16-byte rows.  The left-hand section gives the offset of the 16-byte chunk.  The middle section gives the hex representation of the data at that location.  These pieces of data are aligned in columns also, corresponding to their location within the 16-byte chunk.  So column “E” in row 0x40 means a file offset of (0x40 + 0x0E) = 0x4E.  The last section gives an ASCII representation of the data.

In this representation, it becomes possible to see some of the underlying structure of the bitmap format:

Now, for the script:

## format-hex.msh
## Convert a byte array into a hexidecimal dump
##
## Example usage:
## get-content 'c:\windows\Coffee Bean.bmp' -encoding byte | format-hex | more

## Convert the input to an array of bytes.  This is a strongly-typed variable,
## so that we're not trying to iterate over strings, directory entries, etc.
[byte[]] $bytes = $(foreach($byte in $input) { $byte })

## Store our header, and formatting information
$counter = 0
$header = "            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F"
$nextLine = "{0}   " -f
    [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
$asciiEnd = ""

## Output the header
"`r`n$header`r`n"

foreach($byte in $bytes)
{
   ## Display each byte, in 2-digit hexidecimal, and add that to the left-hand
   ## side.  Notice the use of the '-f' operator here.  This provides access
   ## to the facilities offered by [String]::Format.
   $nextLine += "{0:X2} " -f $byte

   ## If the character is printable, add its ascii representation to
   ## the right-hand side.  Otherwise, add a dot to the right hand side.
   if(($byte -ge 0x20) -and ($byte -le 0xFE))
   {
      $asciiEnd += [char] $byte
   }
   else
   {
      $asciiEnd += "."
   }

   $counter++;

   ## If we've hit the end of a line, combine the right half with the left half,
   ## and start a new line.
   if(($counter % 16) -eq 0)
   {
      "$nextLine $asciiEnd"
      $nextLine = "{0}   " -f
        [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
      $asciiEnd = "";
   }
}

## At the end of the file, we might not have had the chance to output the end
## of the line yet.  Only do this if we didn't exit on the 16-byte boundary,
## though.
if(($counter % 16) -ne 0)
{
   while(($counter % 16) -ne 0)
   {
      $nextLine += "   "
      $asciiEnd += " "
      $counter++;
   }
   "$nextLine $asciiEnd"
}

""

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]