Precision Computing

One very significant problem in computer security is the "Dancing Pigs" syndrome [1][2]:

If J. Random Websurfer clicks on a button that promises dancing pigs on his computer monitor, and instead gets a hortatory message describing the potential dangers of the applet — he's going to choose dancing pigs over computer security any day. If the computer prompts him with a warning screen like: "The applet DANCING PIGS could contain malicious code that might do permanent damage to your computer, steal your life's savings, and impair your ability to have children," he'll click OK without even reading it. Thirty seconds later he won't even remember that the warning screen even existed.

Today on an internal mailing list, the following brilliant suggestion came up proposing a solution:

From: <A person>
Sent: Tuesday, November 08, 2005 12:20 PM
To: <Lots of people>
Subject: <Some security topic>

Perhaps if we shipped Vista with Dancing Pigs, then users could watch the pre-installed Dancing Pigs instead of being enticed to download/install potentially malicious new Dancing Pigs.

- <A person>

-----Original Message-----
From: <Another person>
Sent: Tuesday, November 08, 2005 12:14 PM
To: <Lots of people>
Subject: <Some security topic>

The underlying problem is not the autorun functionality, although it doesn't help.  Mark Russinovich would have gotten infected with that rootkit no matter what because he wanted to listen to the Dancing Pigs CD.  Running as LUA wouldn't help if the user were allowed to elevate (using XP RunAs or Vista UAP).  The kids create CDs or carry thumb drives or whatever and then perform whatever actions are needed to listen to or watch the Dancing Pigs.

-– <Another person>

-----Original Message-----
<Several dozen pages of email snipped>

A fellow Microsoftie, “Jim of Seattle,” has been contributing pieces to songfight.org since 2003.  His most recent piece, “Welcome to ______,” adds another landslide win to his list.  (That’s 6 dashes, if you care.)  He takes the simple Windows XP startup theme, and then builds a nice classical melody around it.  Check it out – it’s truly cool.

He battles a powerful demon in this song – that being the strongly charged musical idiom.  In language, an idiom is “… an expression whose meaning is not compositional – that is, whose meaning does not follow from the meaning of the individual words of which it is composed” [Wikipedia]  A musical idiom is the musical equivalent – short advertising themes are probably the most common example.  He uses many of the default sounds from Windows – songs which are strongly tied to our computing instincts in many cases.  Without introducing the sounds carefully, it’s easy for a listener to get jarred from the mood of a piece because of an overwhelming urge to see what error dialog just popped up.

Opera companies staging Wagner’s “Lohengrin” need to be especially careful of this phenomenon.  The third act has the overly-familiar “Wedding March” – a song that brings so much more to the table now than it did when it was written.  In fact, many productions drop the song altogether to avoid having the 20th century associations invade the carefully crafted storyline.

As a side-note, this song finally helped me figure out that the theme is:

D# A# G# D# A#

This is a “life hack” I’ve been meaning to share since I designed my very first GeoCities homepage, and I figure that now’s a good time to get off of my laurels and do it.  That is, memory and memorization techniques that I’ve found helpful through time.  Not that I’ve invented them, but that I use them.

[ Apple, Pencil, Fox, Anvil ]
- a list -

One difficult recall task we are often faced with is memorizing a list of things.  Remembering a shopping list.  Retaining a “to do” list.  However, the only reason it is difficult is that you haven’t (yet!) found a way to make it concrete, and less abstract.

One of the things I was always impressed with, in my many years as a Starbucks supervisor, was how easy it was to memorize the drinks of regular customers.  Some people were bad at it – the people who didn’t really interact with the customers, or get to know them.  However, when you do take the time to learn more about your regulars, you start associating their favourite drink with them automatically.  It’s no longer an arbitrary mapping of “Fred” to “Double Tall Ristretto Vanilla Non-fat Extra Hot With Whip Latte.”  Instead, you recall making their drink as you talked about their troubles at work.  Or laughing about how the non-fat gives them permission for the whipped cream.  It’s a little story that builds up to something that you relate to almost unconsciously.  After awhile, you just start associating drinks to people, without even having drink-based stories underpinning your memory.

Now how does this relate to memorizing lists?

It relates because you use the same technique.  Turn the list from something abstract to something that you can relate to.  The secret is to link each item with the next item through a vivid and dynamic story.  With this technique, you’ll be able to memorize gigantic lists nearly effortlessly.  And as a cool parlour trick, you’ll memorize them in order – and be able to recite them both forward and backward.

Start with the first item. Paint a vivid picture of it in your mind, adding motion if possible.  Visualize all of the detail you can.  Then, incorporate the next item into your scene.  Form a strong association between the two items, again using vivid imagery and motion.  Then, drop the first item, and form a strong link between the second and third items.  Continue this pattern of associating two at a time until you reach the end of your list.

Let’s memorize the list above.

You’re standing in the produce section of Safeway, looking at the gently slanting apple trays.  You pick up an enormous apple, marveling at its deep red luster.  You turn it around in your hands, looking for any imperfections.  Your mouth waters in anticipation.  Satisfied that you’ve selected the best apple, you begin to put it away. 

Then, out of nowhere, you hear the singing zip of an arrow flying from a bow.  You feel your hand jerk, and look down in surprise.  Splashes of apple juice decorate your arms and shirt as a sharp yellow pencil settles deep into the apple.

You’re walking down a country road, marveling at the bright yellow pencil you got from school.  The smell of freshly cut grass wafts through the air.  The sound of clucking chickens draws your gaze to the right.  Hens flap wildly, as you notice a fox attempting to open their gate.  The gate is locked, so he’s not having much luck.  In desperation, the fox calls out to you, “Hey, mind if I borrow your pencil?  I’m trying to pick this lock.”

As you approach him, holding out your pencil, and anvil drops from the sky, smiting the fox for his evil ways.

I’ll bet you already have the list memorized.  And I bet you’ll still remember it next week.

As Adam points out in his most recent entry, our last release changed the default execution policy to a mode called "Restricted."  The first time you run a script in the new shell, you’ll see the following error message:

The file C:\my_script.msh cannot be loaded. The execution of scripts
is disabled on this system. Please see "get-help about_signing" for
more details.

This is probably not the mode you want to stay with, as it doesn’t run those awesome scripts you write.  Yeah, that’s right, YOUR awesome scripts.  Remember how nice I’m being to you, as I’m about to ask a favour :)

This has probably been one of the most difficult features we’ve implemented, but the reasons are not technical.  It’s difficult, because we know that it makes the shell nearly unusable out of the box.  It’s difficult, because we know we’ve essentially added a manual step to the automatic installation of Monad.

Why?  It’s for your sake, and for the safety of the ecosystem. 

In today’s malicious environment, certain software categories need to be held to a higher standard of security.  For these components, secure by default usually means unusable by default.  Or in other words, “some assembly required.” 

By default, firewalls block nearly all inbound traffic.  IIS serves only static HTML.  Internet Explorer’s “enhanced security configuration” visits only trusted sites.  Outlook disables all dynamic content, images, active links, and optionally even all HTML.  Taken to the extreme, consider the story of OpenBSD:

The open software development model has allowed the organisation to upgrade with an uncompromising view to security enhancement - anything that stands in the way of a more secure environment has been savagely pruned, the result being a clean, slippery system with a surface too tight for conventional breach.

How does this help you?  For one thing, we hope it makes it less likely that your computer administrator at work reflexively restricts you from using MSH altogether.

Getting back to the title of the post, how do you get out of this restricted mode, and which execution policy should you pick?  This is answered by a document included in the distribution zip file, “about_signing.help.txt.”  Although this is a help file for the product, we weren’t able to include it in the actual installer in time for release.  To make it work like the rest of the about_* help files, copy it to your MSH installation directory.

I personally set my execution policy to AllSigned, although most will probably want to run under the RemoteSigned policy that earlier drops were configured to use.  However, security decisions always involve tradeoffs.  Please read the about_signing documentation to understand the implications that your execution policy has.

As you install this new version of Monad, almost all of you will change your execution policy.  Some of you will also blog your installation experience.  A subset of you “installing bloggers” will change it to “Unrestricted,” based on your personal evaluation of its tradeoffs.  I honestly think you should run in RemoteSigned mode instead, but it’s not my job to convince you.  However, most of you ran under the RemoteSigned execution policy for the past several months without even realizing it existed.

Now, it’s time for me to ask that favour: please let your readership make their own decision about an execution policy, rather than suggest they run in Unrestricted mode.  They might not want to make the same security decision you have, but might not realize there is any other alternative.  The documentation provides the background they need to make an informed choice.

Also, I’d like to express my thanks to our Monad MVPs that provided feedback on an earlier piece I had written to summarize the Monad execution policies.  Since our documentation team didn’t have the time to write about_signing, it was written by yours truly.  Your feedback was extremely valuable in helping me craft the public documentation on the subject.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Many thanks to Cyrus for pointing this out.  Videos from the PDC are now available for streaming via http://microsoft.sitestream.com/PDC05/.  The site is running rather slowly right now (it's not being Slashdotted, that's for sure,) so bring a bag of patience with you.

I, of course, recommend you start with Jeffrey and Jim's Monad presentation (during which I was Proctoring at the hands-on-labs.)  As Jeffrey quotes, "This is not your Dad's filesystem".  Then, if you haven't already, check out this year's belle of the ball, Linq :)

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I'm sure you've noticed by now.  One of the things you end up doing a lot in shells and scripting languages is parsing text.  We've done it many times in articles on this blog.  We've seen it in many articles on other blogs.  We've all sat in quiet reverence of the wondrous power that is Regular Expressions.

And you want to be part of the Regex Illuminati.

Eric has just started a series of exercises on Learning Regular Expresions. In his words, this is

"The first in a series of exercises designed to teach you more about regular expressions, written by a guy who got partway through writing a regex book."

I find Eric's "Regular Expression Workbench" to be invaluable in working with regular expressions.  I think you'll enjoy (or at least benefit from!) this series of articles.

Pay close attention -- I'll be blogging a script in the near future that will turn your regular expression skills into Real Ultimate Power.  Regular expressions are cool; and by cool, I mean totally sweet.

Since the first Channel 9 interviews were so popular, Jeffrey recently gave another interview on Channel 9.  With the intentions of talking about the PDC and the CLR, he ended up talking about nearly everything :)

Check it out! http://channel9.msdn.com/Showpost.aspx?postid=127819

Marcel has been posting some interesting articles on using Monad to generate the MD5 hashes of files.  Now, an MD5 hash of a file is just an array of bytes.  Typical hashing programs display this in a more friendly manner:

MSH:15 C:\Temp >md5sum 71-59-B7.bmp
a05805e638741bb767f97c0e88962952 *71-59-B7.bmp

Although the output of Marcel’s scripts could definitely be crafted to display this output, they currently output the string representation of a byte array:

MSH:19 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp)
160 88 5 230 56 116 27 183 103 249 124 14 136 150 41 82

One of the comments in response to Marcel’s post was that Monad should, by default, output byte arrays as hex.  This is a good suggestion, and we can go even further with it.  Let’s write a script to give us a full hex editor-like view of a byte array:

MSH:20 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp) | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   A0 58 05 E6 38 74 1B B7 67 F9 7C 0E 88 96 29 52   X.æ8t.•gù|.??)R

Or even better, let’s use it to dump out a very small bitmap – 10 pixels of the colour (R=0x71 G=0x59 B=0xB7)

MSH:21 C:\Temp >get-content 71-59-B7.bmp -encoding byte | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   42 4D 5E 00 00 00 00 00 00 00 36 00 00 00 28 00  BM^.......6...(.
00000010   00 00 0A 00 00 00 01 00 00 00 01 00 20 00 00 00  ............ ...
00000020   00 00 00 00 00 00 C4 0E 00 00 C4 0E 00 00 00 00  ......Ä...Ä.....
00000030   00 00 00 00 00 00 B7 59 71 FF B7 59 71 FF B7 59  ......•Yq.•Yq.•Y
00000040   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF B7 59  q.•Yq.•Yq.•Yq.•Y
00000050   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF        q.•Yq.•Yq.•Yq.

To make it easier to determine byte offsets, files are usually broken down into 16-byte rows.  The left-hand section gives the offset of the 16-byte chunk.  The middle section gives the hex representation of the data at that location.  These pieces of data are aligned in columns also, corresponding to their location within the 16-byte chunk.  So column “E” in row 0x40 means a file offset of (0x40 + 0x0E) = 0x4E.  The last section gives an ASCII representation of the data.

In this representation, it becomes possible to see some of the underlying structure of the bitmap format:

Now, for the script:

## format-hex.msh
## Convert a byte array into a hexidecimal dump
##
## Example usage:
## get-content 'c:\windows\Coffee Bean.bmp' -encoding byte | format-hex | more

## Convert the input to an array of bytes.  This is a strongly-typed variable,
## so that we're not trying to iterate over strings, directory entries, etc.
[byte[]] $bytes = $(foreach($byte in $input) { $byte })

## Store our header, and formatting information
$counter = 0
$header = "            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F"
$nextLine = "{0}   " -f
    [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
$asciiEnd = ""

## Output the header
"`r`n$header`r`n"

foreach($byte in $bytes)
{
   ## Display each byte, in 2-digit hexidecimal, and add that to the left-hand
   ## side.  Notice the use of the '-f' operator here.  This provides access
   ## to the facilities offered by [String]::Format.
   $nextLine += "{0:X2} " -f $byte

   ## If the character is printable, add its ascii representation to
   ## the right-hand side.  Otherwise, add a dot to the right hand side.
   if(($byte -ge 0x20) -and ($byte -le 0xFE))
   {
      $asciiEnd += [char] $byte
   }
   else
   {
      $asciiEnd += "."
   }

   $counter++;

   ## If we've hit the end of a line, combine the right half with the left half,
   ## and start a new line.
   if(($counter % 16) -eq 0)
   {
      "$nextLine $asciiEnd"
      $nextLine = "{0}   " -f
        [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
      $asciiEnd = "";
   }
}

## At the end of the file, we might not have had the chance to output the end
## of the line yet.  Only do this if we didn't exit on the 16-byte boundary,
## though.
if(($counter % 16) -ne 0)
{
   while(($counter % 16) -ne 0)
   {
      $nextLine += "   "
      $asciiEnd += " "
      $counter++;
   }
   "$nextLine $asciiEnd"
}

""

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to refresh our description data while we navigate the file system.  We hadn't yet integrated it to our directory listings, so let's do that now.

Step 3: Customize the format.mshxml for directory listings

Our final step is to customize the formatting information that Monad uses to output a directory listing.  This is a fairly small change that mimics the way we output the “LastWriteTime” column.

First, open “FileSystem.format.mshxml” from your Monad installation directory.  This file tells Monad how convert the raw .Net File System objects into a form more suitable for humans.  The tweaking we are about to do illustrates one of the fundamental powers of Monad.  Data in the pipeline retains its full object-oriented fidelity until it reaches your screen.  This final stage of the pipeline is called “Formatting and Output.”

Now, before we customize our directory listing, let’s go over the different parts of the formatting file to see what it does.

The formatting files are XML.  The root <Configuration> node contains the following high-level configuration nodes: SelectionSets, Controls, and ViewDefinitions.

<Configuration>
    <SelectionSets>
        <SelectionSet>
           ...
        </SelectionSet>
    </SelectionSets>

    <!-- # GLOBAL CONTROL DEFINITIONS # -->
    <Controls>
        <Control>
           ...
        </Control>
    </Controls>
   
    <!-- # VIEW DEFINITIONS # -->
    <ViewDefinitions>
       <View>
          ...
       </View>
       <View>
          ...
       </View>
    </ViewDefinitions>
</Configuration>

The SelectionSets node lists a set of .Net types.  Later, the custom views from this file use this SelectionSet to help Monad determine the object types to which Monad should apply this formatting information.

The Controls node defines a custom header.  Later, custom views from this file use this header on groups of file system objects.

The Views form the real core of this file:

<ViewDefinitions>
   <View>
      <Name>children</Name>
      <ViewSelectedBy>
         ...
      </ViewSelectedBy>
      <GroupBy>
         ...
      </GroupBy>
      <TableControl>
         <TableHeaders>
            ...
         </TableHeaders>
         <TableRowEntries>
            ...
         </TableRowEntries>
      </TableControl>
   </View>
   <View>
      ...
      <ListControl>
         ...
      </ListControl>
   </View>
   <View>
      ...
      <WideControl>
         ...
      </WideControl>
   </View>
</ViewDefinitions>

We start by naming the view, and then use the ViewSelectedBy node to reference the list of .Net objects to which this view applies.  This reference is the name of one of the SelectionSets nodes earlier in the file.  Next, the GroupBy node tells Monad how to group the objects.  For example, a directory listing is grouped by its parent.  Each grouping gets a header, as defined by one of the Controls earlier in the file.  This is most evident when you do a recursive directory listing, as in “dir –rec.”

Then, we define three views: a TableControl, a ListControl, and a WideControl.  These correspond to the following statements, respectively:

 dir | format-table
 dir | format-list
 dir | format-wide

We’ll focus on (and customize) only the TableControl.

A TableControl has a header row, followed by data rows.  We define the header format with a sequence of TableColumnHeader nodes, then the row format with a sequence of TableRowEntry nodes.

Headers have a label, width, and alignment.  In this file, row entries reference either a direct property of the underlying object, or a ScriptBlock.  As illustrated by the LastWriteTime entry, ScriptBlocks reference the current pipeline object using the “$_” automatic variable.

When Monad loads its formatting data, the first file to define a view wins.  Ours will define the TableControl for DirectoryInfo objects first, so it will override the view that the default FileSystem.format.mshxml defines.

To make Monad look at our file before the default, add the following lines to your profile.msh:

## Update the formatting XML, overriding one of the filesystem
## views with the one we define in FileSystem.Description.Format.mshxml
$formatFile = "$(parse-path $profile)\FileSystem.Description.Format.mshxml"
update-FormatData -prependpath $formatFile

Now, copy the FileSystem.format.mshxml into the same directory as your profile, but call it FileSystem.Description.format.mshxml.  After you copy, change its read-only bit so that you can edit it.  Open the file, now at “My Documents”\msh\FileSystem.Description.format.mshxml.  We’ll place our customizations in this file, rather than hacking on the one that ships with Monad. 

For a test, go to a temporary directory and create a file with a very long name:

MSH:7 C:\Temp >"hi" > ("a"*200)
MSH:8 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaa

In your custom formatting file, delete the <wrap /> node, reload MSH, and then do another listing in that directory:

MSH:2 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaa...

You can delete the long file now, as we’re done with it.

Let’s finally get our description column into directory listings.  In your custom formatting file, do the following:

1. Delete the SelectionSets node, and all of its children.  The defaults will serve us just fine.
2. Delete the Controls node, and all of its children.  The defaults will serve us just fine.
3. Delete the View node that contains the ListControl.
4. Delete the View node that contains the WideControl.
5. Change the width of the LastWriteTime header to 20.
6. After the last Table Column Header (labeled “<TableColumnHeader/>”,) create a new TableColumnHeader node:
   

   <TableColumnHeader>
      <Label>Description</Label>
      <Alignment>left</Alignment>
      <Width>30</Width>
   </TableColumnHeader>

7. After the last Table Column Item (the one for “Name,”) create a new TableColumnItem node:
   
   <TableColumnItem>
      <ScriptBlock>
         ${lee.holmes.descriptions}[$_.Name]
      </ScriptBlock>
   </TableColumnItem>
   
8. Your file should now look like this [FileSystem.Description.Format.mshxml]
9. Create a file called “DESCRIPT.ION” in your profile directory.  Place the following in it:
   
   DESCRIPT.ION Descriptions file
   FileSystem.Description.Format.mshxml Formatting customizations
   profile.msh MSH custom profile
   
10. Restart your MSH shell, navigate to your profile directory, and get a directory listing:
    
    

And there you have it.  DESCRIPT.ION support in Monad.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to parse DESCRIPT.ION files -- and entertained the whole family while we were at it.  This was a manual step, though, so we need to figure out how to make it happen as we use the shell.

Step 2: Automatically refresh the hashtable

In order for this feature to be of any use, we need to ensure that the descriptions hashtable is updated before the directory output hits the screen.  Our primary choices are:

• When we format the directory listing
• When we list the directory items
• When we change directories

The formatting files should primarily be used for formatting, not heavy procedural code, so that rules out the formatting option.  Directory listing (get-childitem) is really the best injection point, but has many commonly used parameters.  Because of that, shimming ourselves into that process is likely to be buggy and complex.  That rules out that option.  That leaves “when we change directories” as the best place to insert our code.

I personally use the ‘cd’ alias to change directories on my system.  Since we want to inject some code into this process, we’ll have to make a new function called ‘cd.’  This function changes to the new directory, and then refreshes the “lee.holmes.descriptions” hashtable with filenames and descriptions from the current directory.  One thing to note is that aliases take precedence over functions.  ‘cd’ is a built-in alias, so we’ll have to also delete the current ‘cd’ alias if it exists.  As a note, this will have a bug / missing feature.  By overriding only the cd functionality, we’ll only be able to get descriptions for items in the current directory.  Something like “dir c:\” won’t show us descriptions for files in other directories.

These are the last handful of lines in my profile-custom.msh:

## Remove the CD alias.  It will override our function if we don't.
if(test-path Alias:\cd) { remove-item Alias:\cd }

## Initialize our default 'descriptions' hashtable to hold
## a mapping of filename to description.
${GLOBAL:lee.holmes.descriptions} = @{}

## Change to a new location.  Once there, refresh the 'descriptions'
## hashtable with the filenames and descriptions of items in the
## current location.
function cd($location)
{
   ## We could use set-location here, but this will also let
   ## us use pop-location to go backwards in our directory navigation
   ## history.  
   push-location $location

   ## Clear the descriptions hashtable, and define the regular expression
   ## that parses a DESCRIPT.ION file
   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   ## Go through each line in DESCRIPT.ION, if it exists, and parse out the
   ## filename / description.
   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }
}

Stay tuned -- we'll wrap this up next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]