Precision Computing

I’ve been watching Jeff’s progress on this shell for awhile, and he’s finally posted a download.  jaMSH is an alternative to the default msh.exe that we ship, and has some very promising features.

It uses the GNU Readline Library as its input mechanism, so you immediately get gobs and gobs and gobs of command-editing features.  It’s also implemented a fairly rich tab-completion model (filenames, variables, members,) and offers a facility to load cmdlets and providers during startup.  This ability to dynamically load cmdlets and providers is something that we had to remove from our default shell because of the versioning problems it introduces, but we’re introducing a stronger (and more usable model) in the next drop or two.

It’s not yet at a state where I can use it as my primary shell, but I’m looking forward to seeing progress on this project.  He’s licensed it under the GPL – why not see how you can contribute?

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Today, Pubsub brought me Jeff’s post, “In Defense of Verbosity.”  In it, he talks about (and praises) the fine expressive balance that Monad allows you to walk between terseness, and verbosity.
 
In an interactive session (where you care only about the output,) terseness is your friend.  Take, for example, the following command that finds the 10 most referenced DLLs in running processes:

gps | select  id -exp modules | group filename | sort count -des | select -f 10

That’s easy to type, but a bear to understand.  If this command will die along with your history buffer, then you’ve done no harm – and saved yourself typing to boot.  If you plan to enshrine this in a script, though, you need to be considerate of the poor sap that will have to eventually maintain that script.  More often than not, that poor sap is you.

So, yes, Monad supports verbosity:

## Get all of the processes
$processes = get-process

## For all of the processes, expand out their Modules.
## This works like a SQL join.  If you have a process object with
## ID=1234, Modules={Filename=ntdll.dll, Filename=gdi32.dll}
## then we get pseudo objects like this:
## ID=1234,Filename=ntdll.dll
## ID=1234,Filename=gdi32.dll
$allModules = $processes | select-object Id -expand Modules

## Group the results by Filename (module file name, that is)
$filenames = $allModules | group-object Filename

## Then pick the top ten.
$topTen = $filenames | sort Count -descending:$true | select -first 10
write-object $topTen

A similar philosophy holds for all programming languages, and even more importantly so.  In programming, your code doesn’t vanish along with your history buffer.  It becomes a maintenance task as soon as you finish writing it.  How difficult a maintenance task?  Well, that’s up to you.

By the way, Jeff:

[terse shorthand is often more difficult to read than terse code.  This is an intentional trade-off, though, as the time spent to capture information is far more precious than the time spent to post-process it.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

Do you use Monad?  Do you want to be ever-present in the corridors that we walk down every day?  Here’s your chance!

Send a photo of you, your team, your company logo, or even your favourite Monad screenshot to &{ "monadphoto@leeAOEUholmes.com" -replace "AOEU","" }, and I’ll post it(*) on our “Shrine of the Customer” bulletin board.

(*) I do, of course, reserve the right to not post pictures that would get me fired.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

First off, I’d like to welcome the many of you who’ve just discovered Monad, and this blog.  The PDC was a great experience – I was glad to have had the chance to talk to so many of you in person.

One of the things I almost forget about, now that I basically live in the Monad shell, is how far it raises the bar for the command line, and how enthusiastic its users become.

Many of my conversations in the hands-on lab and track lounge started from a clean slate.  That is, talking to somebody that had never (or just barely) heard of Monad.  Extra points if the person gets misty-eyed as they talk of their 20 years of experience in the Unix environment.  You might think that would be a hostile crowd, but far from it.  Many of us on the team used to fit that description, as well.

After sitting down with somebody for a demo, they would soon start nodding their head in understanding and appreciation.  That would quickly progress to the “aha!” moment, where they realize how easily they can express themselves an object-oriented shell.

In text-only shells, people are often proud of their accomplishments because of how complex it was to express their intent – not because of how complex their intent was.  The rich composition model offered by an object-based pipeline truly empowers end-users.

As I took a break at the tables near the Starbucks, this infectious enthusiasm really hit home.  I saw two attendees walk by, one of them enthusiastically speaking to the other with waving hands, and excited eyes.  His evangelism brought a smile to my face: “… but it’s OBJECTS going down the pipeline!”

So where from here?  There are many great Monad resources to help you get settled into your new shell:

Monad Team Blog: http://blogs.msdn.com/monad/default.aspx
Adam Barr's Blog: http://www.proudlyserving.com/
My earlier posts: http://www.leeholmes.com/blog/
Reskit.net: http://www.reskit.net/monad/
Channel 9 Wiki: http://channel9.msdn.com/wiki/default.aspx/Channel9.MSHWiki
Newsgroup: nntp://microsoft.public.windows.server.scripting,
MSDN: http://winfx.msdn.microsoft.com/library/default.asp?url=/library/en-us/monad_gettingstarted/html/e72d9b1b-c1c0-41f0-83e7-d230ff3f9144.asp

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

If you love reading about Monad (and haven't yet subscribed to the newsgroup,) now is a good time to do so!

[Edit 01/25/06: Removed Beta 2 download links, and instead point to Thomas' continually updated download links.]

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

In an earlier post, I introduced a download manager in PowerShell.  (I have since updated it.  If you use the script, you might want to download the update.)  The major pain with it, though, is getting the URLs into the text file required by the download manager.  “Right-click, copy link location, paste” just doesn’t cut it for more than a few links.  To resolve this problem, we’ll write another script to parse URLS out of the locally-saved HTML of a web page.

MSH:48 C:\Temp >$userAgent = "PowerShell User"
MSH:49 C:\Temp >$wc = new-object System.Net.WebClient
MSH:50 C:\Temp >$wc.Headers.Add("user-agent", $userAgent)
MSH:51 C:\Temp >$wc.DownloadString("http://channel9.msdn.com") > temp.html
MSH:52 C:\Temp >parse-urls temp.html http://channel9.msdn.com/ wmv$
mms://wm.microsoft.com/ms/msnse/0508/25408/bill_staples_iis7_2005_MBR.wmv
http://download.microsoft.com/download/c/3/9/c39e98c3-03b7-4fa1-959a-8116e3ceb1e3
/bill_staples_iis7_2005.wmv

Now, links are represented in HTML pages as anchor tags: usually something like

<a href=”url”>description</a>

However, there are many variables that get in the way of the simple parsing required by the example above: quote style, and CSS decorations, to name a few.  This calls for some heavy pattern matching in text; a problem usually solved by regular expressions.  (For an overview of regular expressions in PowerShell see my earlier post.)  In fact, almost all of the heavy lifting in this script is done through a single regular expression.

Now, regular expressions are notoriously fiddly things.  It’s hard enough to get them to work while you’re writing them – let alone fixing bugs in them weeks later.  The path out of this predicament lies in the tried and true (but surprisingly unpopular) technique called unit testing.

In unit testing, you write automated tests that exercise your code.  Ideally, you write the tests before you actually write the code, but any unit testing is better than none at all.

MSH:55 C:\Temp >parse-urls -unittest:$true
.................

<after breaking the script >

MSH:56 C:\Temp >parse-urls -unittest:$true
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 1.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: 2.  Actual: 0.  Test failed..
FAIL.  Expected: test1.  Actual: .  Test failed..
FAIL.  Expected: test2.  Actual: .  Test failed..

Although there are no bona-fide unit testing frameworks for MSH scripts, the concept is bleedingly simple.  We can implement the basic requirements by including only a few simple functions in our script:

## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert
     (
 [bool] $condition = $(Please specify a condition),
 [string] $message = "Test failed."
     )
{
 if(-not $condition)
 {
  write-host "FAIL. $message"
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     (
 $expected = $(Please specify the expected object),
 $actual = $(Please specify the actual object),
 [string] $message = "Test failed."
     )
{
 if(-not ($expected -eq $actual))
 {
  write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
 }
 else
 {
  write-host -NoNewLine ".";
 }
}

Now, let’s see it in practice (along with the URL parser goodies I promised):

## parse-urls.ps1
## Parse all of the URLs out of a given file.

param(
        ## The filename to parse
    [string] $filename,
    
    ## The URL from which you downloaded the page.
    ## For example, http://www.microsoft.com/index.html
    [string] $base,
    
    ## The Regular Expression pattern with which to filter 
    ## the returned URLs
    [string] $pattern = ".*",
    
    ## Unit testing flag.
    [switch] $unitTest    
     )          
     
## Defines the regular expression that will parse an URL
## out of an anchor tag.  
$regex = "<\s*a\s*[^>]*?href\s*=\s*[`"']*([^`"'>]+)[^>]*?>"

## The main function isn't a built-in function, but can make
## your script easier to read.  Since functions need to be defined
## before you use them, complicated scripts tend to have their function 
## definitions obscure the main logic of the script.
## 
## To combat this, we define a function, "main," and then call it
## (or dot-source it) at the very end of the script.
function main
{
   if(-not $unitTest)
   {
      parse-file
   }
   else
   {
      unittest
   }
}

## Parse the file for links
function parse-file
{
   if(-not $filename) { throw "Please specify a filename." }
   if(-not $base) { throw "Please specify a base URL." }

   ## Do some minimal source URL fixups, by switching backslashes to
   ## forward slashes
   $base = $base.Replace("\", "/")

   if($base.IndexOf("://") -lt 0)
   { 
      throw "Please specify a base URL in the form of " +
        "http://server/path_to_file/file.html" 
   }

   ## Determine the server from which the file originated.  This will
   ## help us resolve links such as "/somefile.zip"
   $base = $base.Substring(0,$base.LastIndexOf("/") + 1)
   $baseSlash = $base.IndexOf("/", $base.IndexOf("://") + 3)
   $domain = $base.Substring(0, $baseSlash)


   ## Put all of the file content into a big string, and
   ## get the regular expression matches
   $content = [String]::Join('', (get-content $filename))
   $contentMatches = get-matches $content $regex

   foreach($contentMatch in $contentMatches)
   {
      if(-not ($contentMatch -match $pattern)) { continue }

      $contentMatch = $contentMatch.Replace("\", "/")

      ## Hrefs may look like:
      ## ./file
      ## file
      ## ../../../file
      ## /file
      ## url
      ## We'll keep all of the relative paths, as they will resolve.
      ## We only need to resolve the ones pointing to the root.
      if($contentMatch.IndexOf("://") -gt 0)
      {
         $url = $contentMatch
      }
      elseif($contentMatch[0] -eq "/")
      {
         $url = "$domain$contentMatch"
      }
      else
      {
         $url = "$base$contentMatch"
         $url = $url.Replace("/./", "/")
      }

      $url
   }
}

 

function get-matches
     (
    [string] $content = "",
    [string] $regex = ""
     )
{
   $returnMatches = new-object System.Collections.ArrayList

   $resultingMatches = [Regex]::Matches($content, $regex, "IgnoreCase")
   foreach($match in $resultingMatches) 
   { 
      [void] $returnMatches.Add($match.Groups[1].Value.Trim())
   }

   $returnMatches   
} 

 

function unittest
{
   ## A well-formed HREF
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Well-formed"
   AssertEquals "test1" $matches[0] "Well-formed"

   ## Case insensitive
   $matches = @(get-matches '<a href="test1">Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Insensitive"
   AssertEquals "test1" $matches[0] "Insensitive"

   ## Non-quoted attribute
   $matches = @(get-matches '<a href=test1>Test1_Text</a>' $regex)
   AssertEquals 1 $matches.Count "Non-quoted"
   AssertEquals "test1" $matches[0] "Non-quoted"

   ## Unbalanced quoted attribute
   $matches = @(get-matches "<a href=`"test1>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Unbalanced"
   AssertEquals "test1" $matches[0] "Unbalanced"

 

   ## Single ticks for quotes
   $matches = @(get-matches "<a href=`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Single-tick"
   AssertEquals "test1" $matches[0] "Single-tick"

 

   ## Lots of spaces
   $matches = @(get-matches `
    "<a     href =    `'test1`'    >Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Spaces"
   AssertEquals "test1" $matches[0] "Spaces"

   ## Class names
   $matches = @(get-matches `
    "<a class=`"test`" href =`'test1`'>Test1_Text</a>" $regex)
   AssertEquals 1 $matches.Count "Classes"
   AssertEquals "test1" $matches[0] "Classes"

   ## Two URLs
   $matches = @(get-matches `
    "<a href=test1>test1</a><a href=`'test2`'>test2</a>" $regex)
   AssertEquals 2 $matches.Count "Two urls"
   AssertEquals "test1" $matches[0] "Two urls"
   AssertEquals "test2" $matches[1] "Two urls"

   write-host
}

 

## A simple assert function.  Verifies that $condition
## is true.  If not, outputs the specified error message.
function assert 
     ( 
    [bool] $condition = $(Please specify a condition),
    [string] $message = "Test failed." 
     )
{
    if(-not $condition)
    {
        write-host "FAIL. $message"
    }
    else
    {
        write-host -NoNewLine "."
    }
}

## A simple "assert equals" function.  Verifies that $expected
## is equal to $actual.  If not, outputs the specified error message.
function assertEquals
     ( 
    $expected = $(Please specify the expected object),
    $actual = $(Please specify the actual object),
    [string] $message = "Test failed." 
     )
{
    if(-not ($expected -eq $actual))
    {
        write-host "FAIL.  Expected: $expected.  Actual: $actual.  $message."
    }
    else
    {
        write-host -NoNewLine "."
    }
}

main

 

 

Now, here’s the great thing about unit testing.  Let’s say I find some HTML link code that this script should be able to parse, but doesn’t.  In that case, I simply write a new unit test for that code, and edit the regular expression to make the test pass.  If all of the tests continue to pass, then I can be sure that I didn’t break anything that used to work.

Now, go forth, and write high quality scripts!

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]
[Edit: Updated to PowerShell RC2]

For those of you going to the PDC, make sure to check these two out:

http://commnet.microsoftpdc.com/content/sessionview.aspx?TopicID=1bbfa7e1-d4a7-4556-9e52-680bb143f8d6

http://commnet.microsoftpdc.com/content/sessionview.aspx?TopicID=53e82533-f513-405d-92b6-a09e52fc678f

In the first, Jeffrey and Jim will be presenting a detailed introduction to Monad.  These sessions usually end up being standing-room only, generating both spontaneous applause and cheers. 

In the second, Bruce will be part of the panel discussing scripting and dynamic languages on the CLR.

If you do plan to attend either presentation, please consider adding it to your PDC Session Scheduler soon.  This helps us plan for the appropriate audience size, and to gauge interest.

Also, Jeff Jones, Jim Truher, and I will be staffing Monad's "Hands on Labs."  In the hands on labs, you get to play with the shell, implement a C# cmdlet, and implement a cmdlet provider.  We'll also spend time in the track lounge, should you just want to chat.
 

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

As Adam and I mentioned before, Monad is now available in very public beta via the WinFX installer.

As Adam mentions in his post, "It's not installed by default (due to a last-minute setup integration problem), so you have to go to the Start menu, choose `"All Programs`", then `"Microsoft Windows SDK`", then `"Install Windows Command Shell`". That runs our standard installer (the same one you get from betaplace)." [Those back-ticks are my first Monad inside joke!]

Anyhow, part of the fun of SDK membership is SDK Documentation.  Feel free to browse our Getting Started Guide, our Conceptual Overview, or even our (mostly incomplete) API Documentation.

Of course, you can also do site-specific searches from search engines to help you glean the content quicker: site:winfx.msdn.microsoft.com MSH format-table .

(The equivalent MSN Search returns no results, sorry... )

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

One of the shames about our team is that not all of the brilliant members have blogs.  One could also argue that none of the brilliant members have blogs :)

In any case, we're looking to fix that -- as we now have a Monad Team blog, where other folks from Monad will have a chance to contribute.  You'll recognize many of the names from our BetaPlace newsgroups -- Jeff and Marcel have already posted a few articles.

Enjoy.

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

This is one of the lamer things I’ve done as a computer geek, and that’s saying a lot.  However, any self-respecting person does not compromise on their values.  One of my computer values happens to be “Run with Least Privilege.”

What’s a geek to do, then, when a cherished application fails to run under a non-admin account?  Often, the solution is more of a negotiation between your security goals, and the software reality.  A program scribbles all over its installation directory (in Program Files\<Program>,) so you grudgingly grant yourself full control over that specific directory.  Or, it performs a similar travesty by continually modifying data under HKLM\Software\<Vendor>\<Program>.  So, you grudgingly grant yourself full control over that specific registry key.

Now, the cherished software in question is very helpful: Macro Maker.  It’s a very useful macro program, but not very friendly when run as a non-Administrator.

After installing it (as Administrator,) you run into issues the first time you run it:

Temporarily(!) giving ourselves full access to this key makes the error go away, and the program loads fine.  However, that’s not something I’m willing to negotiate.  If I have full access to HKLM\Software, I can completely destroy my system, as can malware acting on my behalf.  I can add spyware to my startup folders, and much more.  So, hastily remove our elevated permissions from the Software key.

Next, we fire up RegMon from a MakeMeAdmin window to see what the problem is.  Set your filter to MacroMaker.exe, and try to run the program again.  The offending entry appears:

The requested access is 0xF003F (just off of the screen shot above.)  You can determine what this means by looking in MSDN, or by looking up the values for the registry access flags in the header files that ship with Visual Studio.

However, we intend to fix the program, so let’s try another approach.  Before we start, copy “MacroMaker.exe” to “MacroMaker.exe.bak.”

Now, open up OllyDbg from a MakeMeAdmin window, and then open the MacroMaker.exe program.  The main debugging window comes up, so right-click it.  Select “Search for | All referenced text strings.”

In the window that pops up, right-click and select “Search for text.”  Search for “Error opening Software key.”  OllyDbg shows a hit, so double-click on it.

Now, OllyDbg has some pretty awesome analysis capabilities.  It’s pinpointed the entrypoint to the assembler routine several lines above: “> 6A 00”.  Right-click that line, and select “Find references to | Selected command.”  OllyDbg opens a references window that shows two locations: “JNZ MacroMak.00410E71” (may be different on your system,) and “PUSH 0, (Initial CPU Selection).”  The first one is a jump into this routine, while the second is the routine itself.  Double click on the first address.

The culprit pops out almost immediately, 6 lines above the jump to the “Error Routine:”

MacroMaker opens the Software key (presumably to see if it exists,) but uses KEY_ALL_ACCESS as the access request.   Do you recognize the number in the assembly instruction for that line?  PUSH 0F0030F.

So how do we fix this?  Well, we’ll make it request only KEY_READ access.  After all, that’s all it should require.

MSDN documents the values for the registry access flags well in this article.  The value for KEY_READ is 0x20019.  Double-click the line with “Access = KEY_ALL_ACCESS,” and change “PUSH 0F003F” to “PUSH 020019”, then click “Assemble.”

Right-click anywhere in the disassembly window, and select “Analysis | Analyze Code.”  Voila, it now requests KEY_READ access.

Right-click the disassembly area again, select “Copy to executable | All modifications,”  then click “Copy All.”  In the window that opens, right-click the disassembly, select “Save file,” and name the program “MacroMakerFixed.exe”

Finally, select the “Debug” menu option, “Close,” and click “Yes.”  Close OllyDbg.

Now, run “MacroMakerFixed.exe” and enjoy your cherished least privilege account!  When you feel comfortable that you haven’t broken the program, copy it over MacroMaker.exe and delete MacroMakerFixed.exe.

Mission Accomplished.