Precision Computing

As Adam points out in his most recent entry, our last release changed the default execution policy to a mode called "Restricted."  The first time you run a script in the new shell, you’ll see the following error message:

The file C:\my_script.msh cannot be loaded. The execution of scripts
is disabled on this system. Please see "get-help about_signing" for
more details.

This is probably not the mode you want to stay with, as it doesn’t run those awesome scripts you write.  Yeah, that’s right, YOUR awesome scripts.  Remember how nice I’m being to you, as I’m about to ask a favour :)

This has probably been one of the most difficult features we’ve implemented, but the reasons are not technical.  It’s difficult, because we know that it makes the shell nearly unusable out of the box.  It’s difficult, because we know we’ve essentially added a manual step to the automatic installation of Monad.

Why?  It’s for your sake, and for the safety of the ecosystem. 

In today’s malicious environment, certain software categories need to be held to a higher standard of security.  For these components, secure by default usually means unusable by default.  Or in other words, “some assembly required.” 

By default, firewalls block nearly all inbound traffic.  IIS serves only static HTML.  Internet Explorer’s “enhanced security configuration” visits only trusted sites.  Outlook disables all dynamic content, images, active links, and optionally even all HTML.  Taken to the extreme, consider the story of OpenBSD:

The open software development model has allowed the organisation to upgrade with an uncompromising view to security enhancement - anything that stands in the way of a more secure environment has been savagely pruned, the result being a clean, slippery system with a surface too tight for conventional breach.

How does this help you?  For one thing, we hope it makes it less likely that your computer administrator at work reflexively restricts you from using MSH altogether.

Getting back to the title of the post, how do you get out of this restricted mode, and which execution policy should you pick?  This is answered by a document included in the distribution zip file, “about_signing.help.txt.”  Although this is a help file for the product, we weren’t able to include it in the actual installer in time for release.  To make it work like the rest of the about_* help files, copy it to your MSH installation directory.

I personally set my execution policy to AllSigned, although most will probably want to run under the RemoteSigned policy that earlier drops were configured to use.  However, security decisions always involve tradeoffs.  Please read the about_signing documentation to understand the implications that your execution policy has.

As you install this new version of Monad, almost all of you will change your execution policy.  Some of you will also blog your installation experience.  A subset of you “installing bloggers” will change it to “Unrestricted,” based on your personal evaluation of its tradeoffs.  I honestly think you should run in RemoteSigned mode instead, but it’s not my job to convince you.  However, most of you ran under the RemoteSigned execution policy for the past several months without even realizing it existed.

Now, it’s time for me to ask that favour: please let your readership make their own decision about an execution policy, rather than suggest they run in Unrestricted mode.  They might not want to make the same security decision you have, but might not realize there is any other alternative.  The documentation provides the background they need to make an informed choice.

Also, I’d like to express my thanks to our Monad MVPs that provided feedback on an earlier piece I had written to summarize the Monad execution policies.  Since our documentation team didn’t have the time to write about_signing, it was written by yours truly.  Your feedback was extremely valuable in helping me craft the public documentation on the subject.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Many thanks to Cyrus for pointing this out.  Videos from the PDC are now available for streaming via http://microsoft.sitestream.com/PDC05/.  The site is running rather slowly right now (it's not being Slashdotted, that's for sure,) so bring a bag of patience with you.

I, of course, recommend you start with Jeffrey and Jim's Monad presentation (during which I was Proctoring at the hands-on-labs.)  As Jeffrey quotes, "This is not your Dad's filesystem".  Then, if you haven't already, check out this year's belle of the ball, Linq :)

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I'm sure you've noticed by now.  One of the things you end up doing a lot in shells and scripting languages is parsing text.  We've done it many times in articles on this blog.  We've seen it in many articles on other blogs.  We've all sat in quiet reverence of the wondrous power that is Regular Expressions.

And you want to be part of the Regex Illuminati.

Eric has just started a series of exercises on Learning Regular Expresions. In his words, this is

"The first in a series of exercises designed to teach you more about regular expressions, written by a guy who got partway through writing a regex book."

I find Eric's "Regular Expression Workbench" to be invaluable in working with regular expressions.  I think you'll enjoy (or at least benefit from!) this series of articles.

Pay close attention -- I'll be blogging a script in the near future that will turn your regular expression skills into Real Ultimate Power.  Regular expressions are cool; and by cool, I mean totally sweet.

Since the first Channel 9 interviews were so popular, Jeffrey recently gave another interview on Channel 9.  With the intentions of talking about the PDC and the CLR, he ended up talking about nearly everything :)

Check it out! http://channel9.msdn.com/Showpost.aspx?postid=127819

Marcel has been posting some interesting articles on using Monad to generate the MD5 hashes of files.  Now, an MD5 hash of a file is just an array of bytes.  Typical hashing programs display this in a more friendly manner:

MSH:15 C:\Temp >md5sum 71-59-B7.bmp
a05805e638741bb767f97c0e88962952 *71-59-B7.bmp

Although the output of Marcel’s scripts could definitely be crafted to display this output, they currently output the string representation of a byte array:

MSH:19 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp)
160 88 5 230 56 116 27 183 103 249 124 14 136 150 41 82

One of the comments in response to Marcel’s post was that Monad should, by default, output byte arrays as hex.  This is a good suggestion, and we can go even further with it.  Let’s write a script to give us a full hex editor-like view of a byte array:

MSH:20 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp) | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   A0 58 05 E6 38 74 1B B7 67 F9 7C 0E 88 96 29 52   X.æ8t.•gù|.??)R

Or even better, let’s use it to dump out a very small bitmap – 10 pixels of the colour (R=0x71 G=0x59 B=0xB7)

MSH:21 C:\Temp >get-content 71-59-B7.bmp -encoding byte | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   42 4D 5E 00 00 00 00 00 00 00 36 00 00 00 28 00  BM^.......6...(.
00000010   00 00 0A 00 00 00 01 00 00 00 01 00 20 00 00 00  ............ ...
00000020   00 00 00 00 00 00 C4 0E 00 00 C4 0E 00 00 00 00  ......Ä...Ä.....
00000030   00 00 00 00 00 00 B7 59 71 FF B7 59 71 FF B7 59  ......•Yq.•Yq.•Y
00000040   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF B7 59  q.•Yq.•Yq.•Yq.•Y
00000050   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF        q.•Yq.•Yq.•Yq.

To make it easier to determine byte offsets, files are usually broken down into 16-byte rows.  The left-hand section gives the offset of the 16-byte chunk.  The middle section gives the hex representation of the data at that location.  These pieces of data are aligned in columns also, corresponding to their location within the 16-byte chunk.  So column “E” in row 0x40 means a file offset of (0x40 + 0x0E) = 0x4E.  The last section gives an ASCII representation of the data.

In this representation, it becomes possible to see some of the underlying structure of the bitmap format:

Now, for the script:

## format-hex.msh
## Convert a byte array into a hexidecimal dump
##
## Example usage:
## get-content 'c:\windows\Coffee Bean.bmp' -encoding byte | format-hex | more

## Convert the input to an array of bytes.  This is a strongly-typed variable,
## so that we're not trying to iterate over strings, directory entries, etc.
[byte[]] $bytes = $(foreach($byte in $input) { $byte })

## Store our header, and formatting information
$counter = 0
$header = "            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F"
$nextLine = "{0}   " -f
    [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
$asciiEnd = ""

## Output the header
"`r`n$header`r`n"

foreach($byte in $bytes)
{
   ## Display each byte, in 2-digit hexidecimal, and add that to the left-hand
   ## side.  Notice the use of the '-f' operator here.  This provides access
   ## to the facilities offered by [String]::Format.
   $nextLine += "{0:X2} " -f $byte

   ## If the character is printable, add its ascii representation to
   ## the right-hand side.  Otherwise, add a dot to the right hand side.
   if(($byte -ge 0x20) -and ($byte -le 0xFE))
   {
      $asciiEnd += [char] $byte
   }
   else
   {
      $asciiEnd += "."
   }

   $counter++;

   ## If we've hit the end of a line, combine the right half with the left half,
   ## and start a new line.
   if(($counter % 16) -eq 0)
   {
      "$nextLine $asciiEnd"
      $nextLine = "{0}   " -f
        [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
      $asciiEnd = "";
   }
}

## At the end of the file, we might not have had the chance to output the end
## of the line yet.  Only do this if we didn't exit on the 16-byte boundary,
## though.
if(($counter % 16) -ne 0)
{
   while(($counter % 16) -ne 0)
   {
      $nextLine += "   "
      $asciiEnd += " "
      $counter++;
   }
   "$nextLine $asciiEnd"
}

""

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to refresh our description data while we navigate the file system.  We hadn't yet integrated it to our directory listings, so let's do that now.

Step 3: Customize the format.mshxml for directory listings

Our final step is to customize the formatting information that Monad uses to output a directory listing.  This is a fairly small change that mimics the way we output the “LastWriteTime” column.

First, open “FileSystem.format.mshxml” from your Monad installation directory.  This file tells Monad how convert the raw .Net File System objects into a form more suitable for humans.  The tweaking we are about to do illustrates one of the fundamental powers of Monad.  Data in the pipeline retains its full object-oriented fidelity until it reaches your screen.  This final stage of the pipeline is called “Formatting and Output.”

Now, before we customize our directory listing, let’s go over the different parts of the formatting file to see what it does.

The formatting files are XML.  The root <Configuration> node contains the following high-level configuration nodes: SelectionSets, Controls, and ViewDefinitions.

<Configuration>
    <SelectionSets>
        <SelectionSet>
           ...
        </SelectionSet>
    </SelectionSets>

    <!-- # GLOBAL CONTROL DEFINITIONS # -->
    <Controls>
        <Control>
           ...
        </Control>
    </Controls>
   
    <!-- # VIEW DEFINITIONS # -->
    <ViewDefinitions>
       <View>
          ...
       </View>
       <View>
          ...
       </View>
    </ViewDefinitions>
</Configuration>

The SelectionSets node lists a set of .Net types.  Later, the custom views from this file use this SelectionSet to help Monad determine the object types to which Monad should apply this formatting information.

The Controls node defines a custom header.  Later, custom views from this file use this header on groups of file system objects.

The Views form the real core of this file:

<ViewDefinitions>
   <View>
      <Name>children</Name>
      <ViewSelectedBy>
         ...
      </ViewSelectedBy>
      <GroupBy>
         ...
      </GroupBy>
      <TableControl>
         <TableHeaders>
            ...
         </TableHeaders>
         <TableRowEntries>
            ...
         </TableRowEntries>
      </TableControl>
   </View>
   <View>
      ...
      <ListControl>
         ...
      </ListControl>
   </View>
   <View>
      ...
      <WideControl>
         ...
      </WideControl>
   </View>
</ViewDefinitions>

We start by naming the view, and then use the ViewSelectedBy node to reference the list of .Net objects to which this view applies.  This reference is the name of one of the SelectionSets nodes earlier in the file.  Next, the GroupBy node tells Monad how to group the objects.  For example, a directory listing is grouped by its parent.  Each grouping gets a header, as defined by one of the Controls earlier in the file.  This is most evident when you do a recursive directory listing, as in “dir –rec.”

Then, we define three views: a TableControl, a ListControl, and a WideControl.  These correspond to the following statements, respectively:

 dir | format-table
 dir | format-list
 dir | format-wide

We’ll focus on (and customize) only the TableControl.

A TableControl has a header row, followed by data rows.  We define the header format with a sequence of TableColumnHeader nodes, then the row format with a sequence of TableRowEntry nodes.

Headers have a label, width, and alignment.  In this file, row entries reference either a direct property of the underlying object, or a ScriptBlock.  As illustrated by the LastWriteTime entry, ScriptBlocks reference the current pipeline object using the “$_” automatic variable.

When Monad loads its formatting data, the first file to define a view wins.  Ours will define the TableControl for DirectoryInfo objects first, so it will override the view that the default FileSystem.format.mshxml defines.

To make Monad look at our file before the default, add the following lines to your profile.msh:

## Update the formatting XML, overriding one of the filesystem
## views with the one we define in FileSystem.Description.Format.mshxml
$formatFile = "$(parse-path $profile)\FileSystem.Description.Format.mshxml"
update-FormatData -prependpath $formatFile

Now, copy the FileSystem.format.mshxml into the same directory as your profile, but call it FileSystem.Description.format.mshxml.  After you copy, change its read-only bit so that you can edit it.  Open the file, now at “My Documents”\msh\FileSystem.Description.format.mshxml.  We’ll place our customizations in this file, rather than hacking on the one that ships with Monad. 

For a test, go to a temporary directory and create a file with a very long name:

MSH:7 C:\Temp >"hi" > ("a"*200)
MSH:8 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaa

In your custom formatting file, delete the <wrap /> node, reload MSH, and then do another listing in that directory:

MSH:2 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaa...

You can delete the long file now, as we’re done with it.

Let’s finally get our description column into directory listings.  In your custom formatting file, do the following:

1. Delete the SelectionSets node, and all of its children.  The defaults will serve us just fine.
2. Delete the Controls node, and all of its children.  The defaults will serve us just fine.
3. Delete the View node that contains the ListControl.
4. Delete the View node that contains the WideControl.
5. Change the width of the LastWriteTime header to 20.
6. After the last Table Column Header (labeled “<TableColumnHeader/>”,) create a new TableColumnHeader node:
   

   <TableColumnHeader>
      <Label>Description</Label>
      <Alignment>left</Alignment>
      <Width>30</Width>
   </TableColumnHeader>

7. After the last Table Column Item (the one for “Name,”) create a new TableColumnItem node:
   
   <TableColumnItem>
      <ScriptBlock>
         ${lee.holmes.descriptions}[$_.Name]
      </ScriptBlock>
   </TableColumnItem>
   
8. Your file should now look like this [FileSystem.Description.Format.mshxml]
9. Create a file called “DESCRIPT.ION” in your profile directory.  Place the following in it:
   
   DESCRIPT.ION Descriptions file
   FileSystem.Description.Format.mshxml Formatting customizations
   profile.msh MSH custom profile
   
10. Restart your MSH shell, navigate to your profile directory, and get a directory listing:
    
    

And there you have it.  DESCRIPT.ION support in Monad.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to parse DESCRIPT.ION files -- and entertained the whole family while we were at it.  This was a manual step, though, so we need to figure out how to make it happen as we use the shell.

Step 2: Automatically refresh the hashtable

In order for this feature to be of any use, we need to ensure that the descriptions hashtable is updated before the directory output hits the screen.  Our primary choices are:

• When we format the directory listing
• When we list the directory items
• When we change directories

The formatting files should primarily be used for formatting, not heavy procedural code, so that rules out the formatting option.  Directory listing (get-childitem) is really the best injection point, but has many commonly used parameters.  Because of that, shimming ourselves into that process is likely to be buggy and complex.  That rules out that option.  That leaves “when we change directories” as the best place to insert our code.

I personally use the ‘cd’ alias to change directories on my system.  Since we want to inject some code into this process, we’ll have to make a new function called ‘cd.’  This function changes to the new directory, and then refreshes the “lee.holmes.descriptions” hashtable with filenames and descriptions from the current directory.  One thing to note is that aliases take precedence over functions.  ‘cd’ is a built-in alias, so we’ll have to also delete the current ‘cd’ alias if it exists.  As a note, this will have a bug / missing feature.  By overriding only the cd functionality, we’ll only be able to get descriptions for items in the current directory.  Something like “dir c:\” won’t show us descriptions for files in other directories.

These are the last handful of lines in my profile-custom.msh:

## Remove the CD alias.  It will override our function if we don't.
if(test-path Alias:\cd) { remove-item Alias:\cd }

## Initialize our default 'descriptions' hashtable to hold
## a mapping of filename to description.
${GLOBAL:lee.holmes.descriptions} = @{}

## Change to a new location.  Once there, refresh the 'descriptions'
## hashtable with the filenames and descriptions of items in the
## current location.
function cd($location)
{
   ## We could use set-location here, but this will also let
   ## us use pop-location to go backwards in our directory navigation
   ## history.  
   push-location $location

   ## Clear the descriptions hashtable, and define the regular expression
   ## that parses a DESCRIPT.ION file
   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   ## Go through each line in DESCRIPT.ION, if it exists, and parse out the
   ## filename / description.
   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }
}

Stay tuned -- we'll wrap this up next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In the BBS (Bulletin Board Service) days, one of the common past times was to connect to a board and browse long file listings in search of neat things to download.  When the names of files were restricted to 11 characters (8.3,) you didn’t have much to go by.  BBS software soon came up with a meta-file, called “descript.ion,” which mapped file names to longer, more verbose descriptions.  During a file listing, the BBS software read this file and displayed these detailed descriptions inline.  Fastidious SysOps (Sytem Operators, of which I was proudly one for many years) spent long hours painstakingly describing files for the benefit of both themselves, and their users.

It was enormously helpful, but is rarely supported by software in today’s new-fangled “long filename” era.  Let’s change that.

To add DESCRIPT.ION support to Monad, we’ll do the following:

1. Write some code that parses the file, and stores the filename-to-description mapping into a form more suitable for later consumption.  We’ll store this into a global hashtable, called “lee.holmes.descriptions.”
2. Automatically refresh the hashtable while we use the shell.  To do this, we’ll augment the “cd” function.
3. Create a new FileSystem.format.mshxml snippet to add a “Description” column to our directory listings.

Step 1: Parse the file

The format of descript.ion files is very simple.  It is a text file, with lines in the format of:
<filename><single space><arbitrary description>

For example:

DESCRIPT.ION Mapping of file names to descriptions

It is now possible for filenames to have spaces in them, so the format has evolved a little to additionally include:

<“quoted filename”><single space><arbitrary description>

To save you the work, a regular expression that parses a line of this file format is:
$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

In English:
 Find a quote.  Put anything before the next quote into the “file” capture.  Find a space.  Put anything after the space into the “text” capture.
 or
 Put anything before the first space into the “file” capture.  Find a space.  Put anything after the space into the “text” capture. 

This regular expression uses named captures to make our further work a little easier:

MSH:11 D:\Lee\MSH >$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"
MSH:12 D:\Lee\MSH >"DESCRIPT.ION Mapping of file names to descriptions" -match $descriptionRegex
True
MSH:13 D:\Lee\MSH >$matches

Key                            Value
---                            -----
text                           Mapping of file names to descriptions
file                           DESCRIPT.ION
0                              DESCRIPT.ION Mapping of file names to descriptions

MSH:14 D:\Lee\MSH >$matches.file
DESCRIPT.ION
MSH:15 D:\Lee\MSH >$matches.text
Mapping of file names to descriptions

So, the next step is to write some code that iterates over every line is DESCRIPT.ION (if it exists,) and store the file / text mappings to a hashtable:

   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }

I’m doing two weird things with my variable here.  First, I’m naming it “lee.holmes.descriptions.”  I do this because it’s a shell-wide variable, and I don’t want to 

• be interfered with
• interfere with others

Second, I put the variable name in curly braces.  Why?  Monad has very permissive rules for variable names, and curly braces save you from having to escape every special character – such as the period.

Try this, and become an instant life-of-the-party:

• Type a dollar sign
• Type an open curly-brace
• Sit on your keyboard, have your cat walk on it, and try to type your name with your forehead
• Type a closed curly brace, followed by “ = $true”
• Dir variable:

Anyways, back to business.  We'll continue with step 2 next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I mentioned to many of you during the PDC that we’d be working hard to make our printed hands-on labs available for download.  As Adam points out, get ‘em while they’re still hot!

We’ve even gone a bit further, and made the “Getting Started” guide and a few quick references available in the documentation pack, available via the Microsoft download center.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Although all of you are probably Monad junkies already, I thought I'd point out Thomas Lee's very helpful Introduction to Monad article.  Great stuff!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]