Precision Computing

Since the first Channel 9 interviews were so popular, Jeffrey recently gave another interview on Channel 9.  With the intentions of talking about the PDC and the CLR, he ended up talking about nearly everything :)

Check it out! http://channel9.msdn.com/Showpost.aspx?postid=127819

Marcel has been posting some interesting articles on using Monad to generate the MD5 hashes of files.  Now, an MD5 hash of a file is just an array of bytes.  Typical hashing programs display this in a more friendly manner:

MSH:15 C:\Temp >md5sum 71-59-B7.bmp
a05805e638741bb767f97c0e88962952 *71-59-B7.bmp

Although the output of Marcel’s scripts could definitely be crafted to display this output, they currently output the string representation of a byte array:

MSH:19 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp)
160 88 5 230 56 116 27 183 103 249 124 14 136 150 41 82

One of the comments in response to Marcel’s post was that Monad should, by default, output byte arrays as hex.  This is a good suggestion, and we can go even further with it.  Let’s write a script to give us a full hex editor-like view of a byte array:

MSH:20 C:\Temp >get-md5 (get-childitem 71-59-B7.bmp) | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   A0 58 05 E6 38 74 1B B7 67 F9 7C 0E 88 96 29 52   X.æ8t.•gù|.??)R

Or even better, let’s use it to dump out a very small bitmap – 10 pixels of the colour (R=0x71 G=0x59 B=0xB7)

MSH:21 C:\Temp >get-content 71-59-B7.bmp -encoding byte | format-hex

            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F

00000000   42 4D 5E 00 00 00 00 00 00 00 36 00 00 00 28 00  BM^.......6...(.
00000010   00 00 0A 00 00 00 01 00 00 00 01 00 20 00 00 00  ............ ...
00000020   00 00 00 00 00 00 C4 0E 00 00 C4 0E 00 00 00 00  ......Ä...Ä.....
00000030   00 00 00 00 00 00 B7 59 71 FF B7 59 71 FF B7 59  ......•Yq.•Yq.•Y
00000040   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF B7 59  q.•Yq.•Yq.•Yq.•Y
00000050   71 FF B7 59 71 FF B7 59 71 FF B7 59 71 FF        q.•Yq.•Yq.•Yq.

To make it easier to determine byte offsets, files are usually broken down into 16-byte rows.  The left-hand section gives the offset of the 16-byte chunk.  The middle section gives the hex representation of the data at that location.  These pieces of data are aligned in columns also, corresponding to their location within the 16-byte chunk.  So column “E” in row 0x40 means a file offset of (0x40 + 0x0E) = 0x4E.  The last section gives an ASCII representation of the data.

In this representation, it becomes possible to see some of the underlying structure of the bitmap format:

Now, for the script:

## format-hex.msh
## Convert a byte array into a hexidecimal dump
##
## Example usage:
## get-content 'c:\windows\Coffee Bean.bmp' -encoding byte | format-hex | more

## Convert the input to an array of bytes.  This is a strongly-typed variable,
## so that we're not trying to iterate over strings, directory entries, etc.
[byte[]] $bytes = $(foreach($byte in $input) { $byte })

## Store our header, and formatting information
$counter = 0
$header = "            0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F"
$nextLine = "{0}   " -f
    [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
$asciiEnd = ""

## Output the header
"`r`n$header`r`n"

foreach($byte in $bytes)
{
   ## Display each byte, in 2-digit hexidecimal, and add that to the left-hand
   ## side.  Notice the use of the '-f' operator here.  This provides access
   ## to the facilities offered by [String]::Format.
   $nextLine += "{0:X2} " -f $byte

   ## If the character is printable, add its ascii representation to
   ## the right-hand side.  Otherwise, add a dot to the right hand side.
   if(($byte -ge 0x20) -and ($byte -le 0xFE))
   {
      $asciiEnd += [char] $byte
   }
   else
   {
      $asciiEnd += "."
   }

   $counter++;

   ## If we've hit the end of a line, combine the right half with the left half,
   ## and start a new line.
   if(($counter % 16) -eq 0)
   {
      "$nextLine $asciiEnd"
      $nextLine = "{0}   " -f
        [Convert]::ToString($counter, 16).ToUpper().PadLeft(8, '0')
      $asciiEnd = "";
   }
}

## At the end of the file, we might not have had the chance to output the end
## of the line yet.  Only do this if we didn't exit on the 16-byte boundary,
## though.
if(($counter % 16) -ne 0)
{
   while(($counter % 16) -ne 0)
   {
      $nextLine += "   "
      $asciiEnd += " "
      $counter++;
   }
   "$nextLine $asciiEnd"
}

""

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to refresh our description data while we navigate the file system.  We hadn't yet integrated it to our directory listings, so let's do that now.

Step 3: Customize the format.mshxml for directory listings

Our final step is to customize the formatting information that Monad uses to output a directory listing.  This is a fairly small change that mimics the way we output the “LastWriteTime” column.

First, open “FileSystem.format.mshxml” from your Monad installation directory.  This file tells Monad how convert the raw .Net File System objects into a form more suitable for humans.  The tweaking we are about to do illustrates one of the fundamental powers of Monad.  Data in the pipeline retains its full object-oriented fidelity until it reaches your screen.  This final stage of the pipeline is called “Formatting and Output.”

Now, before we customize our directory listing, let’s go over the different parts of the formatting file to see what it does.

The formatting files are XML.  The root <Configuration> node contains the following high-level configuration nodes: SelectionSets, Controls, and ViewDefinitions.

<Configuration>
    <SelectionSets>
        <SelectionSet>
           ...
        </SelectionSet>
    </SelectionSets>

    <!-- # GLOBAL CONTROL DEFINITIONS # -->
    <Controls>
        <Control>
           ...
        </Control>
    </Controls>
   
    <!-- # VIEW DEFINITIONS # -->
    <ViewDefinitions>
       <View>
          ...
       </View>
       <View>
          ...
       </View>
    </ViewDefinitions>
</Configuration>

The SelectionSets node lists a set of .Net types.  Later, the custom views from this file use this SelectionSet to help Monad determine the object types to which Monad should apply this formatting information.

The Controls node defines a custom header.  Later, custom views from this file use this header on groups of file system objects.

The Views form the real core of this file:

<ViewDefinitions>
   <View>
      <Name>children</Name>
      <ViewSelectedBy>
         ...
      </ViewSelectedBy>
      <GroupBy>
         ...
      </GroupBy>
      <TableControl>
         <TableHeaders>
            ...
         </TableHeaders>
         <TableRowEntries>
            ...
         </TableRowEntries>
      </TableControl>
   </View>
   <View>
      ...
      <ListControl>
         ...
      </ListControl>
   </View>
   <View>
      ...
      <WideControl>
         ...
      </WideControl>
   </View>
</ViewDefinitions>

We start by naming the view, and then use the ViewSelectedBy node to reference the list of .Net objects to which this view applies.  This reference is the name of one of the SelectionSets nodes earlier in the file.  Next, the GroupBy node tells Monad how to group the objects.  For example, a directory listing is grouped by its parent.  Each grouping gets a header, as defined by one of the Controls earlier in the file.  This is most evident when you do a recursive directory listing, as in “dir –rec.”

Then, we define three views: a TableControl, a ListControl, and a WideControl.  These correspond to the following statements, respectively:

 dir | format-table
 dir | format-list
 dir | format-wide

We’ll focus on (and customize) only the TableControl.

A TableControl has a header row, followed by data rows.  We define the header format with a sequence of TableColumnHeader nodes, then the row format with a sequence of TableRowEntry nodes.

Headers have a label, width, and alignment.  In this file, row entries reference either a direct property of the underlying object, or a ScriptBlock.  As illustrated by the LastWriteTime entry, ScriptBlocks reference the current pipeline object using the “$_” automatic variable.

When Monad loads its formatting data, the first file to define a view wins.  Ours will define the TableControl for DirectoryInfo objects first, so it will override the view that the default FileSystem.format.mshxml defines.

To make Monad look at our file before the default, add the following lines to your profile.msh:

## Update the formatting XML, overriding one of the filesystem
## views with the one we define in FileSystem.Description.Format.mshxml
$formatFile = "$(parse-path $profile)\FileSystem.Description.Format.mshxml"
update-FormatData -prependpath $formatFile

Now, copy the FileSystem.format.mshxml into the same directory as your profile, but call it FileSystem.Description.format.mshxml.  After you copy, change its read-only bit so that you can edit it.  Open the file, now at “My Documents”\msh\FileSystem.Description.format.mshxml.  We’ll place our customizations in this file, rather than hacking on the one that ships with Monad. 

For a test, go to a temporary directory and create a file with a very long name:

MSH:7 C:\Temp >"hi" > ("a"*200)
MSH:8 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaaaaaaaaaaaa
                                             aaaa

In your custom formatting file, delete the <wrap /> node, reload MSH, and then do another listing in that directory:

MSH:2 C:\Temp >dir

    Directory: FileSystem::C:\Temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         10/9/2005   3:16 PM          4 aaaaaaaaaaa...

You can delete the long file now, as we’re done with it.

Let’s finally get our description column into directory listings.  In your custom formatting file, do the following:

1. Delete the SelectionSets node, and all of its children.  The defaults will serve us just fine.
2. Delete the Controls node, and all of its children.  The defaults will serve us just fine.
3. Delete the View node that contains the ListControl.
4. Delete the View node that contains the WideControl.
5. Change the width of the LastWriteTime header to 20.
6. After the last Table Column Header (labeled “<TableColumnHeader/>”,) create a new TableColumnHeader node:
   

   <TableColumnHeader>
      <Label>Description</Label>
      <Alignment>left</Alignment>
      <Width>30</Width>
   </TableColumnHeader>

7. After the last Table Column Item (the one for “Name,”) create a new TableColumnItem node:
   
   <TableColumnItem>
      <ScriptBlock>
         ${lee.holmes.descriptions}[$_.Name]
      </ScriptBlock>
   </TableColumnItem>
   
8. Your file should now look like this [FileSystem.Description.Format.mshxml]
9. Create a file called “DESCRIPT.ION” in your profile directory.  Place the following in it:
   
   DESCRIPT.ION Descriptions file
   FileSystem.Description.Format.mshxml Formatting customizations
   profile.msh MSH custom profile
   
10. Restart your MSH shell, navigate to your profile directory, and get a directory listing:
    
    

And there you have it.  DESCRIPT.ION support in Monad.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Last time, we wrote code to parse DESCRIPT.ION files -- and entertained the whole family while we were at it.  This was a manual step, though, so we need to figure out how to make it happen as we use the shell.

Step 2: Automatically refresh the hashtable

In order for this feature to be of any use, we need to ensure that the descriptions hashtable is updated before the directory output hits the screen.  Our primary choices are:

• When we format the directory listing
• When we list the directory items
• When we change directories

The formatting files should primarily be used for formatting, not heavy procedural code, so that rules out the formatting option.  Directory listing (get-childitem) is really the best injection point, but has many commonly used parameters.  Because of that, shimming ourselves into that process is likely to be buggy and complex.  That rules out that option.  That leaves “when we change directories” as the best place to insert our code.

I personally use the ‘cd’ alias to change directories on my system.  Since we want to inject some code into this process, we’ll have to make a new function called ‘cd.’  This function changes to the new directory, and then refreshes the “lee.holmes.descriptions” hashtable with filenames and descriptions from the current directory.  One thing to note is that aliases take precedence over functions.  ‘cd’ is a built-in alias, so we’ll have to also delete the current ‘cd’ alias if it exists.  As a note, this will have a bug / missing feature.  By overriding only the cd functionality, we’ll only be able to get descriptions for items in the current directory.  Something like “dir c:\” won’t show us descriptions for files in other directories.

These are the last handful of lines in my profile-custom.msh:

## Remove the CD alias.  It will override our function if we don't.
if(test-path Alias:\cd) { remove-item Alias:\cd }

## Initialize our default 'descriptions' hashtable to hold
## a mapping of filename to description.
${GLOBAL:lee.holmes.descriptions} = @{}

## Change to a new location.  Once there, refresh the 'descriptions'
## hashtable with the filenames and descriptions of items in the
## current location.
function cd($location)
{
   ## We could use set-location here, but this will also let
   ## us use pop-location to go backwards in our directory navigation
   ## history.  
   push-location $location

   ## Clear the descriptions hashtable, and define the regular expression
   ## that parses a DESCRIPT.ION file
   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   ## Go through each line in DESCRIPT.ION, if it exists, and parse out the
   ## filename / description.
   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }
}

Stay tuned -- we'll wrap this up next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In the BBS (Bulletin Board Service) days, one of the common past times was to connect to a board and browse long file listings in search of neat things to download.  When the names of files were restricted to 11 characters (8.3,) you didn’t have much to go by.  BBS software soon came up with a meta-file, called “descript.ion,” which mapped file names to longer, more verbose descriptions.  During a file listing, the BBS software read this file and displayed these detailed descriptions inline.  Fastidious SysOps (Sytem Operators, of which I was proudly one for many years) spent long hours painstakingly describing files for the benefit of both themselves, and their users.

It was enormously helpful, but is rarely supported by software in today’s new-fangled “long filename” era.  Let’s change that.

To add DESCRIPT.ION support to Monad, we’ll do the following:

1. Write some code that parses the file, and stores the filename-to-description mapping into a form more suitable for later consumption.  We’ll store this into a global hashtable, called “lee.holmes.descriptions.”
2. Automatically refresh the hashtable while we use the shell.  To do this, we’ll augment the “cd” function.
3. Create a new FileSystem.format.mshxml snippet to add a “Description” column to our directory listings.

Step 1: Parse the file

The format of descript.ion files is very simple.  It is a text file, with lines in the format of:
<filename><single space><arbitrary description>

For example:

DESCRIPT.ION Mapping of file names to descriptions

It is now possible for filenames to have spaces in them, so the format has evolved a little to additionally include:

<“quoted filename”><single space><arbitrary description>

To save you the work, a regular expression that parses a line of this file format is:
$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

In English:
 Find a quote.  Put anything before the next quote into the “file” capture.  Find a space.  Put anything after the space into the “text” capture.
 or
 Put anything before the first space into the “file” capture.  Find a space.  Put anything after the space into the “text” capture. 

This regular expression uses named captures to make our further work a little easier:

MSH:11 D:\Lee\MSH >$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"
MSH:12 D:\Lee\MSH >"DESCRIPT.ION Mapping of file names to descriptions" -match $descriptionRegex
True
MSH:13 D:\Lee\MSH >$matches

Key                            Value
---                            -----
text                           Mapping of file names to descriptions
file                           DESCRIPT.ION
0                              DESCRIPT.ION Mapping of file names to descriptions

MSH:14 D:\Lee\MSH >$matches.file
DESCRIPT.ION
MSH:15 D:\Lee\MSH >$matches.text
Mapping of file names to descriptions

So, the next step is to write some code that iterates over every line is DESCRIPT.ION (if it exists,) and store the file / text mappings to a hashtable:

   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }

I’m doing two weird things with my variable here.  First, I’m naming it “lee.holmes.descriptions.”  I do this because it’s a shell-wide variable, and I don’t want to 

• be interfered with
• interfere with others

Second, I put the variable name in curly braces.  Why?  Monad has very permissive rules for variable names, and curly braces save you from having to escape every special character – such as the period.

Try this, and become an instant life-of-the-party:

• Type a dollar sign
• Type an open curly-brace
• Sit on your keyboard, have your cat walk on it, and try to type your name with your forehead
• Type a closed curly brace, followed by “ = $true”
• Dir variable:

Anyways, back to business.  We'll continue with step 2 next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I mentioned to many of you during the PDC that we’d be working hard to make our printed hands-on labs available for download.  As Adam points out, get ‘em while they’re still hot!

We’ve even gone a bit further, and made the “Getting Started” guide and a few quick references available in the documentation pack, available via the Microsoft download center.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Although all of you are probably Monad junkies already, I thought I'd point out Thomas Lee's very helpful Introduction to Monad article.  Great stuff!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I’ve been watching Jeff’s progress on this shell for awhile, and he’s finally posted a download.  jaMSH is an alternative to the default msh.exe that we ship, and has some very promising features.

It uses the GNU Readline Library as its input mechanism, so you immediately get gobs and gobs and gobs of command-editing features.  It’s also implemented a fairly rich tab-completion model (filenames, variables, members,) and offers a facility to load cmdlets and providers during startup.  This ability to dynamically load cmdlets and providers is something that we had to remove from our default shell because of the versioning problems it introduces, but we’re introducing a stronger (and more usable model) in the next drop or two.

It’s not yet at a state where I can use it as my primary shell, but I’m looking forward to seeing progress on this project.  He’s licensed it under the GPL – why not see how you can contribute?

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Today, Pubsub brought me Jeff’s post, “In Defense of Verbosity.”  In it, he talks about (and praises) the fine expressive balance that Monad allows you to walk between terseness, and verbosity.
 
In an interactive session (where you care only about the output,) terseness is your friend.  Take, for example, the following command that finds the 10 most referenced DLLs in running processes:

gps | select  id -exp modules | group filename | sort count -des | select -f 10

That’s easy to type, but a bear to understand.  If this command will die along with your history buffer, then you’ve done no harm – and saved yourself typing to boot.  If you plan to enshrine this in a script, though, you need to be considerate of the poor sap that will have to eventually maintain that script.  More often than not, that poor sap is you.

So, yes, Monad supports verbosity:

## Get all of the processes
$processes = get-process

## For all of the processes, expand out their Modules.
## This works like a SQL join.  If you have a process object with
## ID=1234, Modules={Filename=ntdll.dll, Filename=gdi32.dll}
## then we get pseudo objects like this:
## ID=1234,Filename=ntdll.dll
## ID=1234,Filename=gdi32.dll
$allModules = $processes | select-object Id -expand Modules

## Group the results by Filename (module file name, that is)
$filenames = $allModules | group-object Filename

## Then pick the top ten.
$topTen = $filenames | sort Count -descending:$true | select -first 10
write-object $topTen

A similar philosophy holds for all programming languages, and even more importantly so.  In programming, your code doesn’t vanish along with your history buffer.  It becomes a maintenance task as soon as you finish writing it.  How difficult a maintenance task?  Well, that’s up to you.

By the way, Jeff:

[terse shorthand is often more difficult to read than terse code.  This is an intentional trade-off, though, as the time spent to capture information is far more precious than the time spent to post-process it.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Do you use Monad?  Do you want to be ever-present in the corridors that we walk down every day?  Here’s your chance!

Send a photo of you, your team, your company logo, or even your favourite Monad screenshot to &{ "monadphoto@leeAOEUholmes.com" -replace "AOEU","" }, and I’ll post it(*) on our “Shrine of the Customer” bulletin board.

(*) I do, of course, reserve the right to not post pictures that would get me fired.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]