Archives for the Month of September, 2005

jaMSH: Jeff’s Alternative Monad Shell Host

I’ve been watching Jeff’s progress on this shell for awhile, and he’s finally posted a download.  jaMSH is an alternative to the default msh.exe that we ship, and has some very promising features.

It uses the GNU Readline Library as its input mechanism, so you immediately get gobs and gobs and gobs of command-editing features.  It’s also implemented a fairly rich tab-completion model (filenames, variables, members,) and offers a facility to load cmdlets and providers during startup.  This ability to dynamically load cmdlets and providers is something that we had to remove from our default shell because of the versioning problems it introduces, but we’re introducing a stronger (and more usable model) in the next drop or two.

It’s not yet at a state where I can use it as my primary shell, but I’m looking forward to seeing progress on this project.  He’s licensed it under the GPL – why not see how you can contribute?

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In Defense of Verbosity

Today, Pubsub brought me Jeff’s post, “In Defense of Verbosity.”  In it, he talks about (and praises) the fine expressive balance that Monad allows you to walk between terseness, and verbosity.
 
In an interactive session (where you care only about the output,) terseness is your friend.  Take, for example, the following command that finds the 10 most referenced DLLs in running processes:

gps | select  id -exp modules | group filename | sort count -des | select -f 10

That’s easy to type, but a bear to understand.  If this command will die along with your history buffer, then you’ve done no harm – and saved yourself typing to boot.  If you plan to enshrine this in a script, though, you need to be considerate of the poor sap that will have to eventually maintain that script.  More often than not, that poor sap is you.

So, yes, Monad supports verbosity:

## Get all of the processes
$processes = get-process

## For all of the processes, expand out their Modules.
## This works like a SQL join.  If you have a process object with
## ID=1234, Modules={Filename=ntdll.dll, Filename=gdi32.dll}
## then we get pseudo objects like this:
## ID=1234,Filename=ntdll.dll
## ID=1234,Filename=gdi32.dll
$allModules = $processes | select-object Id -expand Modules

## Group the results by Filename (module file name, that is)
$filenames = $allModules | group-object Filename

## Then pick the top ten.
$topTen = $filenames | sort Count -descending:$true | select -first 10
write-object $topTen

A similar philosophy holds for all programming languages, and even more importantly so.  In programming, your code doesn’t vanish along with your history buffer.  It becomes a maintenance task as soon as you finish writing it.  How difficult a maintenance task?  Well, that’s up to you.

By the way, Jeff:

[terse shorthand is often more difficult to read than terse code.  This is an intentional trade-off, though, as the time spent to capture information is far more precious than the time spent to post-process it.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Want to Help us Decorate? Your Pictures Wanted!

Do you use Monad?  Do you want to be ever-present in the corridors that we walk down every day?  Here’s your chance!

Send a photo of you, your team, your company logo, or even your favourite Monad screenshot to &{ "monadphoto@leeAOEUholmes.com" -replace "AOEU","" }, and I’ll post it(*) on our “Shrine of the Customer” bulletin board.

(*) I do, of course, reserve the right to not post pictures that would get me fired.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Welcome to the Monad family, and thoughts on the PDC

First off, I’d like to welcome the many of you who’ve just discovered Monad, and this blog.  The PDC was a great experience – I was glad to have had the chance to talk to so many of you in person.

One of the things I almost forget about, now that I basically live in the Monad shell, is how far it raises the bar for the command line, and how enthusiastic its users become.

Many of my conversations in the hands-on lab and track lounge started from a clean slate.  That is, talking to somebody that had never (or just barely) heard of Monad.  Extra points if the person gets misty-eyed as they talk of their 20 years of experience in the Unix environment.  You might think that would be a hostile crowd, but far from it.  Many of us on the team used to fit that description, as well.

After sitting down with somebody for a demo, they would soon start nodding their head in understanding and appreciation.  That would quickly progress to the “aha!” moment, where they realize how easily they can express themselves an object-oriented shell.

In text-only shells, people are often proud of their accomplishments because of how complex it was to express their intent – not because of how complex their intent was.  The rich composition model offered by an object-based pipeline truly empowers end-users.

As I took a break at the tables near the Starbucks, this infectious enthusiasm really hit home.  I saw two attendees walk by, one of them enthusiastically speaking to the other with waving hands, and excited eyes.  His evangelism brought a smile to my face: “… but it’s OBJECTS going down the pipeline!”

So where from here?  There are many great Monad resources to help you get settled into your new shell:

Monad Team Blog: http://blogs.msdn.com/monad/default.aspx
Adam Barr's Blog: http://www.proudlyserving.com/
My earlier posts: http://www.leeholmes.com/blog/
Reskit.net: http://www.reskit.net/monad/
Channel 9 Wiki: http://channel9.msdn.com/wiki/default.aspx/Channel9.MSHWiki
Newsgroup: nntp://microsoft.public.windows.server.scripting,
MSDN: http://winfx.msdn.microsoft.com/library/default.asp?url=/library/en-us/monad_gettingstarted/html/e72d9b1b-c1c0-41f0-83e7-d230ff3f9144.asp

 

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

New Monad Download Available! (Monad Beta 2)

I used to have download links here, but they were for an older version of Monad.  But I'm still getting plenty of hits per day for it, and needlessly sending people on a wild goose chase.  Since Thomas has been doing a great job of keeping his Reskit.net up to date with the latest Monad Download information, I'll point you to him for the best Monad Download information.

Beta 2 signals our move out of BetaPlace. The discussion on the private newsgroups have been very helpful for all involved, and we're making that even better. We now have a public newsgroup available at nntp://microsoft.public.windows.server.scripting, that replaces the old one on BetaPlace. Please only use the new newsgroup.

If you love reading about Monad (and haven't yet subscribed to the newsgroup,) now is a good time to do so!

[Edit 01/25/06: Removed Beta 2 download links, and instead point to Thomas' continually updated download links.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Unit Testing in PowerShell – a Link Parser

In an earlier post, I introduced a download manager in PowerShell.  (I have since updated it.  If you use the script, you might want to download the update.)  The major pain with it, though, is getting the URLs into the text file required by the download manager.  “Right-click, copy link location, paste” just doesn’t cut it for more than a few links.  To resolve this problem, we’ll write another script to parse URLS out of the locally-saved HTML of a web page.

MSH:48 C:\Temp >$userAgent = "PowerShell User"
MSH:49 C:\Temp >$wc = new-object System.Net.WebClient
MSH:50 C:\Temp >$wc.Headers.Add("user-agent", $userAgent)
MSH:51 C:\Temp >$wc.DownloadString("
http://channel9.msdn.com") > temp.html
MSH:52 C:\Temp >parse-urls temp.html
http://channel9.msdn.com/ wmv$
mms://wm.microsoft.com/ms/msnse/0508/25408/bill_staples_iis7_2005_MBR.wmv
http://download.microsoft.com/download/c/3/9/c39e98c3-03b7-4fa1-959a-8116e3ceb1e3
/bill_staples_iis7_2005.wmv

Now, links are represented in HTML pages as anchor tags: usually something like

<a href=”url”>description</a>

However, there are many variables that get in the way of the simple parsing required by the example above: quote style, and CSS decorations, to name a few.  This calls for some heavy pattern matching in text; a problem usually solved by regular expressions.  (For an overview of regular expressions in PowerShell see my earlier post.)  In fact, almost all of the heavy lifting in this script is done through a single regular expression.

Now, regular expressions are notoriously fiddly things.  It’s hard enough to get them to work while you’re writing them – let alone fixing bugs in them weeks later.  The path out of this predicament lies in the tried and true (but surprisingly unpopular) technique called unit testing.

In unit testing, you write automated tests that exercise your code.  Ideally, you write the tests before you actually write the code, but any unit testing is better than none at all.

MSH:55 C:\Temp >parse-urls -unittest:$true
.................

<after breaking the script >

MSH:56 C:\Temp >parse-urls -unittest:$true
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 2.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: test2.  Actual: .  Test failed..

Although there are no bona-fide unit testing frameworks for MSH scripts, the concept is bleedingly simple.  We can implement the basic requirements by including only a few simple functions in our script:

## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert
     (
 [bool] $condition = $(Please specify a condition),
 [string] $message = "Test failed."
     )
{
 if(-not $condition)
 {
  write-host "FAIL. $message"
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     (
 $expected = $(Please specify the expected object),
 $actual = $(Please specify the actual object),
 [string] $message = "Test failed."
     )
{
 if(-not ($expected -eq $actual))
 {
  write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

Now, let’s see it in practice (along with the URL parser goodies I promised):

## parse-urls.ps1
## Parse all of the URLs out of a given file.

param(
        ## The filename to parse
    [string] $filename,
    
    ## The URL from which you downloaded the page.
    ## For example, http://www.microsoft.com/index.html
    [string] $base,
    
    ## The Regular Expression pattern with which to filter 
    ## the returned URLs
    [string] $pattern = ".*",
    
    ## Unit testing flag.
    [switch] $unitTest    
     )          
     
## Defines the regular expression that will parse an URL
## out of an anchor tag.  
$regex = "<\s*a\s*[^>]*?href\s*=\s*[`"']*([^`"'>]+)[^>]*?>"

## The main function isn't a built-in function, but can make
## your script easier to read.  Since functions need to be defined
## before you use them, complicated scripts tend to have their function 
## definitions obscure the main logic of the script.
## 
## To combat this, we define a function, "main," and then call it
## (or dot-source it) at the very end of the script.
function main
{
   if(-not $unitTest)
   {
      parse-file
   }
   else
   {
      unittest
   }
}

## Parse the file for links
function parse-file
{
   if(-not $filename) { throw "Please specify a filename." }
   if(-not $base) { throw "Please specify a base URL." }

   ## Do some minimal source URL fixups, by switching backslashes to
   ## forward slashes
   $base = $base.Replace("\""/")

   if($base.IndexOf("://") -lt 0)
   { 
      throw "Please specify a base URL in the form of " +
        "http://server/path_to_file/file.html" 
   }

   ## Determine the server from which the file originated.  This will
   ## help us resolve links such as "/somefile.zip"
   $base = $base.Substring(0,$base.LastIndexOf("/") + 1)
   $baseSlash = $base.IndexOf("/", $base.IndexOf("://") + 3)
   $domain = $base.Substring(0, $baseSlash)

   ## Put all of the file content into a big string, and
   ## get the regular expression matches
   $content = [String]::Join('', (get-content $filename))
   $contentMatches = get-matches $content $regex

   foreach($contentMatch in $contentMatches)
   {
      if(-not ($contentMatch -match $pattern)) { continue }

      $contentMatch = $contentMatch.Replace("\""/")

      ## Hrefs may look like:
      ## ./file
      ## file
      ## ../../../file
      ## /file
      ## url
      ## We'll keep all of the relative paths, as they will resolve.
      ## We only need to resolve the ones pointing to the root.
      if($contentMatch.Inde
xOf("://") -gt 0)
      {
         $url = $contentMatch
      }
      elseif($contentMatch[0] -eq "/")
      {
         $url = "$domain$contentMatch"
      }
      else
      {
         $url = "$base$contentMatch"
         $url = $url.Replace("/./""/")
      }

      $url
   }
}

 
function get-matches
     (
    [string] $content = "",
    [string] $regex = ""
     )
{
   $returnMatches = new-object System.Collections.ArrayList

   $resultingMatches = [Regex]::Matches($content, $regex, "IgnoreCase")
   foreach($match in $resultingMatches) 
   { 
      [void] $returnMatches.Add($match.Groups[1].Value.Trim())
   }

   $returnMatches   

 
function unittest
{
   ## A well-formed HREF
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Well-formed"
   AssertEquals "test1" $matches[0"Well-formed"

   ## Case insensitive
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Insensitive"
   AssertEquals "test1" $matches[0"Insensitive"

   ## Non-quoted attribute
   $matches = @(get-matches '<a href=test1>Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Non-quoted"
   AssertEquals "test1" $matches[0"Non-quoted"

   ## Unbalanced quoted attribute
   $matches = @(get-matches "<a href=`"test1>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Unbalanced"
   AssertEquals "test1" $matches[0"Unbalanced"

 

   ## Single ticks for quotes
   $matches = @(get-matches "<a href=`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Single-tick"
   AssertEquals "test1" $matches[0"Single-tick"
 
   ## Lots of spaces
   $matches = @(get-matches `
    "<a     href =    `'test1`'    >Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Spaces"
   AssertEquals "test1" $matches[0"Spaces"
   ## Class names
   $matches = @(get-matches `
    "<a class=`"test`" href =`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Classes"
   AssertEquals "test1" $matches[0"Classes"

   ## Two URLs
   $matches = @(get-matches `
    "<a href=test1>test1</a><a href=`'test2`'>test2</a>" $regex)
   AssertEquals 2 $matches.Count "Two urls"
   AssertEquals "test1" $matches[0"Two urls"
   AssertEquals "test2" $matches[1"Two urls"

   write-host
}

 
## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert 
     ( 
    [bool] $condition = $(Please specify a condition),
    [string] $message = "Test failed." 
     )
{
    if(-not $condition)
    {
        write-host "FAIL. $message"
    }
    else
    {
        write-host -NoNewLine "."
    }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     ( 
    $expected = $(Please specify the expected object),
    $actual = $(Please specify the actual object),
    [string] $message = "Test failed." 
     )
{
    if(-not ($expected -eq $actual))
    {
        write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
    }
    else
    {
        write-host -NoNewLine "."
    }
}

main

 

 

Now, here’s the great thing about unit testing.  Let’s say I find some HTML link code that this script should be able to parse, but doesn’t.  In that case, I simply write a new unit test for that code, and edit the regular expression to make the test pass.  If all of the tests continue to pass, then I can be sure that I didn’t break anything that used to work.

Now, go forth, and write high quality scripts!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]
[Edit: Updated to PowerShell RC2]