Precision Computing

Last time, we wrote code to parse DESCRIPT.ION files -- and entertained the whole family while we were at it.  This was a manual step, though, so we need to figure out how to make it happen as we use the shell.

Step 2: Automatically refresh the hashtable

In order for this feature to be of any use, we need to ensure that the descriptions hashtable is updated before the directory output hits the screen.  Our primary choices are:

• When we format the directory listing
• When we list the directory items
• When we change directories

The formatting files should primarily be used for formatting, not heavy procedural code, so that rules out the formatting option.  Directory listing (get-childitem) is really the best injection point, but has many commonly used parameters.  Because of that, shimming ourselves into that process is likely to be buggy and complex.  That rules out that option.  That leaves “when we change directories” as the best place to insert our code.

I personally use the ‘cd’ alias to change directories on my system.  Since we want to inject some code into this process, we’ll have to make a new function called ‘cd.’  This function changes to the new directory, and then refreshes the “lee.holmes.descriptions” hashtable with filenames and descriptions from the current directory.  One thing to note is that aliases take precedence over functions.  ‘cd’ is a built-in alias, so we’ll have to also delete the current ‘cd’ alias if it exists.  As a note, this will have a bug / missing feature.  By overriding only the cd functionality, we’ll only be able to get descriptions for items in the current directory.  Something like “dir c:\” won’t show us descriptions for files in other directories.

These are the last handful of lines in my profile-custom.msh:

## Remove the CD alias.  It will override our function if we don't.
if(test-path Alias:\cd) { remove-item Alias:\cd }

## Initialize our default 'descriptions' hashtable to hold
## a mapping of filename to description.
${GLOBAL:lee.holmes.descriptions} = @{}

## Change to a new location.  Once there, refresh the 'descriptions'
## hashtable with the filenames and descriptions of items in the
## current location.
function cd($location)
{
   ## We could use set-location here, but this will also let
   ## us use pop-location to go backwards in our directory navigation
   ## history.  
   push-location $location

   ## Clear the descriptions hashtable, and define the regular expression
   ## that parses a DESCRIPT.ION file
   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   ## Go through each line in DESCRIPT.ION, if it exists, and parse out the
   ## filename / description.
   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }
}

Stay tuned -- we'll wrap this up next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In the BBS (Bulletin Board Service) days, one of the common past times was to connect to a board and browse long file listings in search of neat things to download.  When the names of files were restricted to 11 characters (8.3,) you didn’t have much to go by.  BBS software soon came up with a meta-file, called “descript.ion,” which mapped file names to longer, more verbose descriptions.  During a file listing, the BBS software read this file and displayed these detailed descriptions inline.  Fastidious SysOps (Sytem Operators, of which I was proudly one for many years) spent long hours painstakingly describing files for the benefit of both themselves, and their users.

It was enormously helpful, but is rarely supported by software in today’s new-fangled “long filename” era.  Let’s change that.

To add DESCRIPT.ION support to Monad, we’ll do the following:

1. Write some code that parses the file, and stores the filename-to-description mapping into a form more suitable for later consumption.  We’ll store this into a global hashtable, called “lee.holmes.descriptions.”
2. Automatically refresh the hashtable while we use the shell.  To do this, we’ll augment the “cd” function.
3. Create a new FileSystem.format.mshxml snippet to add a “Description” column to our directory listings.

Step 1: Parse the file

The format of descript.ion files is very simple.  It is a text file, with lines in the format of:
<filename><single space><arbitrary description>

For example:

DESCRIPT.ION Mapping of file names to descriptions

It is now possible for filenames to have spaces in them, so the format has evolved a little to additionally include:

<“quoted filename”><single space><arbitrary description>

To save you the work, a regular expression that parses a line of this file format is:
$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

In English:
 Find a quote.  Put anything before the next quote into the “file” capture.  Find a space.  Put anything after the space into the “text” capture.
 or
 Put anything before the first space into the “file” capture.  Find a space.  Put anything after the space into the “text” capture. 

This regular expression uses named captures to make our further work a little easier:

MSH:11 D:\Lee\MSH >$descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"
MSH:12 D:\Lee\MSH >"DESCRIPT.ION Mapping of file names to descriptions" -match $descriptionRegex
True
MSH:13 D:\Lee\MSH >$matches

Key                            Value
---                            -----
text                           Mapping of file names to descriptions
file                           DESCRIPT.ION
0                              DESCRIPT.ION Mapping of file names to descriptions

MSH:14 D:\Lee\MSH >$matches.file
DESCRIPT.ION
MSH:15 D:\Lee\MSH >$matches.text
Mapping of file names to descriptions

So, the next step is to write some code that iterates over every line is DESCRIPT.ION (if it exists,) and store the file / text mappings to a hashtable:

   ${GLOBAL:lee.holmes.descriptions} = @{}
   $descriptionRegex = "`"(?<file>[^`"]*)`" (?<text>.*)|(?<file>[^ ]*) (?<text>.*)"

   if(test-path descript.ion)
   {
      foreach($line in get-content descript.ion)
      {
         if($line -match $descriptionRegex)
         {
            ${lee.holmes.descriptions}[$matches.file] = $matches.text
         }
      }
   }

I’m doing two weird things with my variable here.  First, I’m naming it “lee.holmes.descriptions.”  I do this because it’s a shell-wide variable, and I don’t want to 

• be interfered with
• interfere with others

Second, I put the variable name in curly braces.  Why?  Monad has very permissive rules for variable names, and curly braces save you from having to escape every special character – such as the period.

Try this, and become an instant life-of-the-party:

• Type a dollar sign
• Type an open curly-brace
• Sit on your keyboard, have your cat walk on it, and try to type your name with your forehead
• Type a closed curly brace, followed by “ = $true”
• Dir variable:

Anyways, back to business.  We'll continue with step 2 next time.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I mentioned to many of you during the PDC that we’d be working hard to make our printed hands-on labs available for download.  As Adam points out, get ‘em while they’re still hot!

We’ve even gone a bit further, and made the “Getting Started” guide and a few quick references available in the documentation pack, available via the Microsoft download center.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Although all of you are probably Monad junkies already, I thought I'd point out Thomas Lee's very helpful Introduction to Monad article.  Great stuff!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

I’ve been watching Jeff’s progress on this shell for awhile, and he’s finally posted a download.  jaMSH is an alternative to the default msh.exe that we ship, and has some very promising features.

It uses the GNU Readline Library as its input mechanism, so you immediately get gobs and gobs and gobs of command-editing features.  It’s also implemented a fairly rich tab-completion model (filenames, variables, members,) and offers a facility to load cmdlets and providers during startup.  This ability to dynamically load cmdlets and providers is something that we had to remove from our default shell because of the versioning problems it introduces, but we’re introducing a stronger (and more usable model) in the next drop or two.

It’s not yet at a state where I can use it as my primary shell, but I’m looking forward to seeing progress on this project.  He’s licensed it under the GPL – why not see how you can contribute?

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Today, Pubsub brought me Jeff’s post, “In Defense of Verbosity.”  In it, he talks about (and praises) the fine expressive balance that Monad allows you to walk between terseness, and verbosity.
 
In an interactive session (where you care only about the output,) terseness is your friend.  Take, for example, the following command that finds the 10 most referenced DLLs in running processes:

gps | select  id -exp modules | group filename | sort count -des | select -f 10

That’s easy to type, but a bear to understand.  If this command will die along with your history buffer, then you’ve done no harm – and saved yourself typing to boot.  If you plan to enshrine this in a script, though, you need to be considerate of the poor sap that will have to eventually maintain that script.  More often than not, that poor sap is you.

So, yes, Monad supports verbosity:

## Get all of the processes
$processes = get-process

## For all of the processes, expand out their Modules.
## This works like a SQL join.  If you have a process object with
## ID=1234, Modules={Filename=ntdll.dll, Filename=gdi32.dll}
## then we get pseudo objects like this:
## ID=1234,Filename=ntdll.dll
## ID=1234,Filename=gdi32.dll
$allModules = $processes | select-object Id -expand Modules

## Group the results by Filename (module file name, that is)
$filenames = $allModules | group-object Filename

## Then pick the top ten.
$topTen = $filenames | sort Count -descending:$true | select -first 10
write-object $topTen

A similar philosophy holds for all programming languages, and even more importantly so.  In programming, your code doesn’t vanish along with your history buffer.  It becomes a maintenance task as soon as you finish writing it.  How difficult a maintenance task?  Well, that’s up to you.

By the way, Jeff:

[terse shorthand is often more difficult to read than terse code.  This is an intentional trade-off, though, as the time spent to capture information is far more precious than the time spent to post-process it.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Do you use Monad?  Do you want to be ever-present in the corridors that we walk down every day?  Here’s your chance!

Send a photo of you, your team, your company logo, or even your favourite Monad screenshot to &{ "monadphoto@leeAOEUholmes.com" -replace "AOEU","" }, and I’ll post it(*) on our “Shrine of the Customer” bulletin board.

(*) I do, of course, reserve the right to not post pictures that would get me fired.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

First off, I’d like to welcome the many of you who’ve just discovered Monad, and this blog.  The PDC was a great experience – I was glad to have had the chance to talk to so many of you in person.

One of the things I almost forget about, now that I basically live in the Monad shell, is how far it raises the bar for the command line, and how enthusiastic its users become.

Many of my conversations in the hands-on lab and track lounge started from a clean slate.  That is, talking to somebody that had never (or just barely) heard of Monad.  Extra points if the person gets misty-eyed as they talk of their 20 years of experience in the Unix environment.  You might think that would be a hostile crowd, but far from it.  Many of us on the team used to fit that description, as well.

After sitting down with somebody for a demo, they would soon start nodding their head in understanding and appreciation.  That would quickly progress to the “aha!” moment, where they realize how easily they can express themselves an object-oriented shell.

In text-only shells, people are often proud of their accomplishments because of how complex it was to express their intent – not because of how complex their intent was.  The rich composition model offered by an object-based pipeline truly empowers end-users.

As I took a break at the tables near the Starbucks, this infectious enthusiasm really hit home.  I saw two attendees walk by, one of them enthusiastically speaking to the other with waving hands, and excited eyes.  His evangelism brought a smile to my face: “… but it’s OBJECTS going down the pipeline!”

So where from here?  There are many great Monad resources to help you get settled into your new shell:

Monad Team Blog: http://blogs.msdn.com/monad/default.aspx
Adam Barr's Blog: http://www.proudlyserving.com/
My earlier posts: http://www.leeholmes.com/blog/
Reskit.net: http://www.reskit.net/monad/
Channel 9 Wiki: http://channel9.msdn.com/wiki/default.aspx/Channel9.MSHWiki
Newsgroup: nntp://microsoft.public.windows.server.scripting,
MSDN: http://winfx.msdn.microsoft.com/library/default.asp?url=/library/en-us/monad_gettingstarted/html/e72d9b1b-c1c0-41f0-83e7-d230ff3f9144.asp

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

If you love reading about Monad (and haven't yet subscribed to the newsgroup,) now is a good time to do so!

[Edit 01/25/06: Removed Beta 2 download links, and instead point to Thomas' continually updated download links.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In an earlier post, I introduced a download manager in PowerShell.  (I have since updated it.  If you use the script, you might want to download the update.)  The major pain with it, though, is getting the URLs into the text file required by the download manager.  “Right-click, copy link location, paste” just doesn’t cut it for more than a few links.  To resolve this problem, we’ll write another script to parse URLS out of the locally-saved HTML of a web page.

MSH:48 C:\Temp >$userAgent = "PowerShell User"
MSH:49 C:\Temp >$wc = new-object System.Net.WebClient
MSH:50 C:\Temp >$wc.Headers.Add("user-agent", $userAgent)
MSH:51 C:\Temp >$wc.DownloadString("http://channel9.msdn.com") > temp.html
MSH:52 C:\Temp >parse-urls temp.html http://channel9.msdn.com/ wmv$
mms://wm.microsoft.com/ms/msnse/0508/25408/bill_staples_iis7_2005_MBR.wmv
http://download.microsoft.com/download/c/3/9/c39e98c3-03b7-4fa1-959a-8116e3ceb1e3
/bill_staples_iis7_2005.wmv

Now, links are represented in HTML pages as anchor tags: usually something like

<a href=”url”>description</a>

However, there are many variables that get in the way of the simple parsing required by the example above: quote style, and CSS decorations, to name a few.  This calls for some heavy pattern matching in text; a problem usually solved by regular expressions.  (For an overview of regular expressions in PowerShell see my earlier post.)  In fact, almost all of the heavy lifting in this script is done through a single regular expression.

Now, regular expressions are notoriously fiddly things.  It’s hard enough to get them to work while you’re writing them – let alone fixing bugs in them weeks later.  The path out of this predicament lies in the tried and true (but surprisingly unpopular) technique called unit testing.

In unit testing, you write automated tests that exercise your code.  Ideally, you write the tests before you actually write the code, but any unit testing is better than none at all.

MSH:55 C:\Temp >parse-urls -unittest:$true
.................

<after breaking the script >

MSH:56 C:\Temp >parse-urls -unittest:$true
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 2.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: test2.  Actual: .  Test failed..

Although there are no bona-fide unit testing frameworks for MSH scripts, the concept is bleedingly simple.  We can implement the basic requirements by including only a few simple functions in our script:

## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert
     (
 [bool] $condition = $(Please specify a condition),
 [string] $message = "Test failed."
     )
{
 if(-not $condition)
 {
  write-host "FAIL. $message"
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     (
 $expected = $(Please specify the expected object),
 $actual = $(Please specify the actual object),
 [string] $message = "Test failed."
     )
{
 if(-not ($expected -eq $actual))
 {
  write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

Now, let’s see it in practice (along with the URL parser goodies I promised):

## parse-urls.ps1
## Parse all of the URLs out of a given file.

param(
        ## The filename to parse
    [string] $filename,
    
    ## The URL from which you downloaded the page.
    ## For example, http://www.microsoft.com/index.html
    [string] $base,
    
    ## The Regular Expression pattern with which to filter 
    ## the returned URLs
    [string] $pattern = ".*",
    
    ## Unit testing flag.
    [switch] $unitTest    
     )          
     
## Defines the regular expression that will parse an URL
## out of an anchor tag.  
$regex = "<\s*a\s*[^>]*?href\s*=\s*[`"']*([^`"'>]+)[^>]*?>"

## The main function isn't a built-in function, but can make
## your script easier to read.  Since functions need to be defined
## before you use them, complicated scripts tend to have their function 
## definitions obscure the main logic of the script.
## 
## To combat this, we define a function, "main," and then call it
## (or dot-source it) at the very end of the script.
function main
{
   if(-not $unitTest)
   {
      parse-file
   }
   else
   {
      unittest
   }
}

## Parse the file for links
function parse-file
{
   if(-not $filename) { throw "Please specify a filename." }
   if(-not $base) { throw "Please specify a base URL." }

   ## Do some minimal source URL fixups, by switching backslashes to
   ## forward slashes
   $base = $base.Replace("\", "/")

   if($base.IndexOf("://") -lt 0)
   { 
      throw "Please specify a base URL in the form of " +
        "http://server/path_to_file/file.html" 
   }

   ## Determine the server from which the file originated.  This will
   ## help us resolve links such as "/somefile.zip"
   $base = $base.Substring(0,$base.LastIndexOf("/") + 1)
   $baseSlash = $base.IndexOf("/", $base.IndexOf("://") + 3)
   $domain = $base.Substring(0, $baseSlash)


   ## Put all of the file content into a big string, and
   ## get the regular expression matches
   $content = [String]::Join('', (get-content $filename))
   $contentMatches = get-matches $content $regex

   foreach($contentMatch in $contentMatches)
   {
      if(-not ($contentMatch -match $pattern)) { continue }

      $contentMatch = $contentMatch.Replace("\", "/")

      ## Hrefs may look like:
      ## ./file
      ## file
      ## ../../../file
      ## /file
      ## url
      ## We'll keep all of the relative paths, as they will resolve.
      ## We only need to resolve the ones pointing to the root.
      if($contentMatch.IndexOf("://") -gt 0)
      {
         $url = $contentMatch
      }
      elseif($contentMatch[0] -eq "/")
      {
         $url = "$domain$contentMatch"
      }
      else
      {
         $url = "$base$contentMatch"
         $url = $url.Replace("/./", "/")
      }

      $url
   }
}

 

function get-matches
     (
    [string] $content = "",
    [string] $regex = ""
     )
{
   $returnMatches = new-object System.Collections.ArrayList

   $resultingMatches = [Regex]::Matches($content, $regex, "IgnoreCase")
   foreach($match in $resultingMatches) 
   { 
      [void] $returnMatches.Add($match.Groups[1].Value.Trim())
   }

   $returnMatches   
} 

 

function unittest
{
   ## A well-formed HREF
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Well-formed"
   AssertEquals "test1" $matches[0] "Well-formed"

   ## Case insensitive
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Insensitive"
   AssertEquals "test1" $matches[0] "Insensitive"

   ## Non-quoted attribute
   $matches = @(get-matches '<a href=test1>Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Non-quoted"
   AssertEquals "test1" $matches[0] "Non-quoted"

   ## Unbalanced quoted attribute
   $matches = @(get-matches "<a href=`"test1>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Unbalanced"
   AssertEquals "test1" $matches[0] "Unbalanced"

 

   ## Single ticks for quotes
   $matches = @(get-matches "<a href=`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Single-tick"
   AssertEquals "test1" $matches[0] "Single-tick"

 

   ## Lots of spaces
   $matches = @(get-matches `
    "<a     href =    `'test1`'    >Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Spaces"
   AssertEquals "test1" $matches[0] "Spaces"

   ## Class names
   $matches = @(get-matches `
    "<a class=`"test`" href =`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Classes"
   AssertEquals "test1" $matches[0] "Classes"

   ## Two URLs
   $matches = @(get-matches `
    "<a href=test1>test1</a><a href=`'test2`'>test2</a>" $regex)
   AssertEquals 2 $matches.Count "Two urls"
   AssertEquals "test1" $matches[0] "Two urls"
   AssertEquals "test2" $matches[1] "Two urls"

   write-host
}

 

## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert 
     ( 
    [bool] $condition = $(Please specify a condition),
    [string] $message = "Test failed." 
     )
{
    if(-not $condition)
    {
        write-host "FAIL. $message"
    }
    else
    {
        write-host -NoNewLine "."
    }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     ( 
    $expected = $(Please specify the expected object),
    $actual = $(Please specify the actual object),
    [string] $message = "Test failed." 
     )
{
    if(-not ($expected -eq $actual))
    {
        write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
    }
    else
    {
        write-host -NoNewLine "."
    }
}

main

 

 

Now, here’s the great thing about unit testing.  Let’s say I find some HTML link code that this script should be able to parse, but doesn’t.  In that case, I simply write a new unit test for that code, and edit the regular expression to make the test pass.  If all of the tests continue to pass, then I can be sure that I didn’t break anything that used to work.

Now, go forth, and write high quality scripts!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]
[Edit: Updated to PowerShell RC2]