Archives for the ‘Uncategorized’ Category

More Detecting Obfuscated PowerShell

In a recent post, we talked a little bit about detecting obfuscated PowerShell through the use of PowerShell's tokenizer - tackling, as an example, the highly irregular variable names generated by MetaSploit's PowerShell encoder.

Obfuscation has been around as long as computer programs have, so the rise of obfuscated PowerShell scripts shouldn't be much of a surprise. Obfuscated VBScript, Perl, Ruby, Python, and of course assembly language are very common.

A great example comes from this blog:, which relies heavily on dynamic evaluation through Invoke-Expression:

Symbolic PowerShell!

Enabling ScriptBlock logging in PowerShell v5 is an incredibly effective way to gain insight into this style of technique:


Obfuscation through Invoke-Expression or basic things like variable names are one thing, but what happens when people really start obfuscating the mechanics of the scripts themselves? At DerbyCon this year, Daniel Bohannon (Blue Teamer Par Excellence) recently gave a really great presentation: "Invoke-Obfuscation: PowerShell obFUsk8tion Techniques & How To (Try To) D""e`Tec`T 'Th'+'em' ".

In that presentation, he dropped gems like this that rely heavily on the Format operator:

Token Obfuscation

That obfuscation doesn't rely on the Invoke-Expression command, so would show up basically the same in script block logging. Along with the crazy Invoke-Expression stuff, these examples demonstrate, without a doubt, that relying on string matching alone to detect evil is a fool's errand.

But here's the thing.

Obfuscated scripts aren't normal. Anybody looking at an obfuscated script knows that they're not normal. They stick out like a sore thumb. This alone can be used as an incredibly rich signal that somebody is trying to avoid getting caught, in the same way that all self-respecting SOCs look for the events that come from an attacker turning off Antivirus.

In the previous post, we looked at the letter frequency of variable names, but what if we expanded on that approach to look at all characters in a script? The Invoke-Expression based script above relied entirely on 16 characters. The script that relied on PowerShell's Format operator relied heavily on quoting and brace characters.

But how do you know what's abnormal across "all PowerShell scripts"?

On the PowerShell team, one thing we often use for questions like this is a corpus that we created by downloading everything we could get our grubby little hands on. So, let's take a look at character frequencies using Measure-CharacterFrequency.

PS C:\PowerShellCorpus\PoshCode> $globalFrequency = Measure-CharacterFrequency *.ps1
PS C:\PowerShellCorpus\PoshCode> $globalFrequency | Select -First 20

Name Percent
---- -------
E      9.912
T      7.414
A      5.512
R       5.43
S      5.303
I      5.041
N      5.025
O      4.944
L      3.509
M        3.3
C      3.191
$      3.076
P      2.914
D      2.753
U       2.29
-      1.955
.      1.917
"      1.822
F      1.626
G      1.526

Now, compare that to some of these other ones:

## The Token-based obfuscation that relies on the Format operator
PS > Measure-CharacterFrequency C:\temp\tokenall.ps1 | Select -First 10

Name Percent
---- -------
'     20.175
{      7.456
}      7.456
,      5.702
E      3.947
T      3.509
N      3.509
"      3.509
(       3.07
)       3.07

## The one that relies on Invoke-Expression
PS > Measure-CharacterFrequency C:\temp\symbolic.ps1 | Select -First 10

Name Percent
---- -------
$     21.808
{     21.659
}     21.659
+     13.313
"      7.452
=      2.832
[      2.086
(      1.689
;       1.54
)      1.341


The difference is huge, and unmistakable. But how do we compare these sets in a robust and reliable way? We steal from the field of Information Retrieval, that's how!

The field of Information Retrieval has long used a technique called vector similarity / cosine similarity to compare two sets of things. For example, the similarity of two documents, or how closely a search matches a document.

Cosine Similarity

That's a lot of complicated math-looking symbols - but it turns out it's very easy to calculate. Here's an example, using Measure-VectorSimilarity.

PS > Measure-VectorSimilarity @(1..10) @(4..15)


So, let's automate a vector similarity comparison. Take random selection of scripts from PoshCode, dump in some obfuscated ones, and see if anything sticks out based on the vector similarity score:

PS > md c:\temp\randomscripts
PS > dir | Get-Random -Count 20 | Copy-Item -Destination C:\temp\randomscripts
PS > copy C:\temp\symbolic.ps1 C:\temp\randomscripts
PS > copy C:\temp\tokenall.ps1 C:\temp\randomscripts
PS > dir C:\temp\randomscripts\ | % {
>>>     $scriptFrequency = $_ | Measure-CharacterFrequency.ps1
>>>     $sim = Measure-VectorSimilarity $globalFrequency $scriptFrequency 
>>>             -KeyProperty Name -ValueProperty Percent
>>>     [PSCustomObject] @{ Name = $_.Name; Similarity = $sim }
>>> }

Name                                     Similarity
----                                     ----------
43a28a15-5023-4feb-a71f-abe95aa0f2a6.ps1      0.957
Export-PSCredential_4.ps1                     0.979
Get-BogonList_1.ps1                           0.925
Get-Netstat _1.9.ps1                           0.89
Get-Parameter_8.ps1                           0.959
group-byobject_4.ps1                          0.939
IADsDNWithBinary Cmdlet_1.ps1                 0.924
Import-ExcelToSQL_2.ps1                       0.961
Invoke-Sql_2.ps1                              0.979
List AddRemovePrograms.ps1                    0.961
Lock-WorkStation.ps1                          0.905
Monitor-FileSize_1.ps1                        0.974
symbolic.ps1                                  0.157
Reverse filename sequenc.ps1                  0.874
scriptable telnet client_2.ps1                0.967
Set Active Sync DeviceID.ps1                  0.955
SharePoint Large Lists_1.ps1                  0.944
Show-Sample_1.ps1                             0.919
Start-Verify.ps1                              0.923
tokenall.ps1                                  0.379


In fact, if you graph the similarity scores of the nearly 3500 scripts from PoshCode, only 2% of them have a similarity score less than 80%. And almost all of them are legitimately obfuscated for fun.


The difference is unmistakable. If you're currently trying to detect malicious script-based content, be sure to also look for indicators of script obfuscation. Reliable techniques exist, and you're likely running blind without them.

Fortune 500 PowerPoint Fodder

In preparation for some upcoming presentations, I wanted to make some images of the current Fortune 500 logos. No such resource existed, so now it does.


The attached ZIP has:

  • Logos for the Fortune 500, numbered by their position
  • A PowerPoint deck with a slide where they are composed together
  • An image of the logos in a grid
  • An image of the logos in a grid, with a “Fortune 500” logo on top.


Fixing carriage jam and 0x61011beb error on HP Photosmart Premium

I recently had my HP Photosmart Premium stop working. Initially, it failed with an error message requesting that I clear the carriage jam. This was caused by the carriage being stuck at the far right of the printer where it normally goes to clean the print head.

The carriage was stuck enough that moving it with my hands didn’t work. When I took the sides of the printer off, there was a movable plate stuck below the carriage that I was able to slide away using a bamboo skewer.

open side

After resetting the printer, the carriage jam message went away – only to be replaced with a generic complaint about “error 0x61011beb.”

Error 0x61011beb appears to be a “catch all” error message indicating physical problems. Online resources helpfully suggest doing a factory reset of the printer, which didn’t resolve my issue.

One thing I did notice before the printer would give the error message was the central rod to the left of the big white gear spinning for a long time.

gear assembly

During the printer reset, the white gear would sometimes spin, thereby turning the black gear meshed to it. But when the rod spun, very little happened. If I rotated the white gear manually, the movable plate that caused the carriage jam would move.

Another thing I noticed was that the whole assembly attached to the central rod wasn’t on very tight. I thought perhaps it somehow got knocked loose, so tried a bunch of printer resets after re-seating it, or holding it against the rod manually. None of those worked.

The rod in the gear assembly goes through a little black plastic gear. After taking the assembly off the rod, I was able to look at the gear. It had a small crack, which I assume was preventing it from gripping the rod properly (which was preventing it from having the power to turn the white gear, which was preventing the plate from moving, which was breaking my printer).

I initially tried to put some glue on the inside of the gear, put it back into the rod assembly, put it all back onto the rod, and let it sit overnight. Unfortunately, that didn’t stick well enough and the printer acted like nothing had changed.

Fortunately, I had access to a 3d printer, and was able re-print the gear. The original is on the left.


I used Parametric Involute Bevel and Spur Gears by GregFrost, with the following dimensions:

gear (
    gear_thickness = 3.8,
    rim_thickness = 3,
    hub_thickness = 3,
    bore_diameter = 4.62,

I’ve shared this gear here: The inner bore diameter is intentionally thin. I used a needle file to open the gap until it was JUST able to squeeze onto the raised ridges on the rod.

If you’re running into the same issue (slipping of the gear) but don’t have access to a 3d printer, some other things might work:

  • Finding another gear with the same number of teeth, inner bore diameter, and outer bore diameter. It has 15 teeth, an outer diameter of 14.38mm, and an inner bore of about 4.62mm.
  • Filling the central bore of the gear with moldable plastic (like Instamorph), and then drilling out a new central bore.
  • Printing the gear out at Shapeways or a similar service.
  • Asking a local hacker space if they can help you print out the gear.

In the end, I was able to rescue a $300 printer with 2.4 cents of plastic.


Detecting Obfuscated PowerShell

I was recently looking at a sample that was encoded using MSF’s basic template obfuscation (stolen without attribution from Matt Graeber of course):

For example:


function mtKZ {

       Param ($l7PpJu1SE4VO, $qhnBBk5lHo)            

       $pcE6VKGt = ([AppDomain]::CurrentDomain.GetAssemblies() | Where-Object { $_.GlobalAssemblyCache -And $_.Location.Split('\\')[-1].Equals('System.dll') }).GetType('Microsoft.Win32.UnsafeNativeMethods')


       return $pcE6VKGt.GetMethod('GetProcAddress').Invoke($null, @([System.Runtime.InteropServices.HandleRef](New-Object System.Runtime.InteropServices.HandleRef((New-Object IntPtr), ($pcE6VKGt.GetMethod('GetModuleHandle')).Invoke($null, @($l7PpJu1SE4VO)))), $qhnBBk5lHo))



For detection purposes, attempted obfuscation like this (i.e.: the variable names) are themselves an indicator to malicious activity.


PowerShell’s AST APIs make detection of stuff like this a breeze. For example, here’s a way to get all of the variables in $Path:


$tokens = @()

$null = [System.Management.Automation.Language.Parser]::ParseFile($Path, [ref] $tokens, [ref] $null)

$tokens | ? VariablePath | % { $_.VariablePath.UserPath }


With that, we can start to do some variable analysis. Basic entropy is a pretty good start. When you combine that with letter frequency distribution, this creates a pretty good obfuscation metric:

14 [C:\temp]
>> dir *.ps1 | % { Measure-VariableObfuscation.ps1 $_.FullName } | sort ObfuscationMetric

Path                                                  Entropy    TopFourLetters ObfuscationMetric
----                                                  -------    -------------- -----------------
C:\temp\hello.ps1                                           0                 0                 0
C:\temp\foo2.ps1                                            0                 1                 0
C:\temp\2.ps1                                               0                 0                 0
C:\temp\1.ps1                                               0                 0                 0
C:\temp\3.ps1                                               0                 0                 0
C:\temp\verbose.ps1                          3.17281073351987 0.666666666666667  1.05760357783996
C:\temp\msf_template.ps1                     3.07084709362252 0.631578947368421  1.13136471870303
C:\temp\foo.ps1                              3.65719253292414 0.588235294117647  1.50590280767465
C:\temp\pester.temp.tests.ps1                3.47972685963298            0.5625  1.52238050108943
C:\temp\configtest.ps1                       3.75004181130572 0.495934959349594  1.89026497805654
C:\temp\sendmailmessagetest.ps1              3.79012121177685  0.48780487804878  1.94128159627595
C:\temp\Repro.ps1                            4.16910660776366  0.46583850931677  2.22697620042034
C:\temp\sttest.ps1                           4.27537906345103 0.469194312796209  2.26939552183183
C:\temp\TranscriptTest.ps1                   4.17394102071541 0.425629290617849  2.39738946498757
C:\temp\Invoke-ActiveScriptEventConsumer.ps1 4.21710521416516 0.415384615384615  2.46538458674271
C:\temp\mywatch-command.ps1                   4.2973293816282  0.42159383033419  2.48560182741991
C:\temp\Burn-Console.ascii.ps1               4.45029315016471 0.352501867064974  2.88155650574519
C:\temp\Burn-Console.ps1                     4.45029315016471 0.352501867064974  2.88155650574519
C:\temp\Invoke-TokenManipulation.ps1         4.91011096435002 0.384693390598902  3.02122372925736
C:\temp\Invoke-TokenManipulationNonAdmin.ps1 4.91011096435002 0.384693390598902  3.02122372925736
C:\temp\stager.ps1                           5.32866566677047 0.244131455399061  4.02777076220679

MSF could of course adapt to this, but its algorithm would continue to have predictable and detectable output. All you’ve got to do is look J

And of course, Measure-VariableObfuscation:





























#requires -Module PowerShellArsenal






$tokens = @(); $null = [System.Management.Automation.Language.Parser]::ParseFile($Path, [ref] $tokens, [ref] $null)

$bytes = [byte[]][char[]]-join ($tokens | ? VariablePath | % { $_.VariablePath.UserPath })

$entropy = 0

$top4 = 0



    $entropy = Get-Entropy $bytes

    $letterFrequency = Measure-LetterFrequency (-join ($tokens | ? VariablePath | % { $_.VariablePath.UserPath })) -Raw

    $top4 = $letterFrequency[1..4] | Measure-Object -Sum Percent | % Sum


[PSCustomObject] @{

    Path = $Path

    Entropy = $entropy

    TopFourLetters = $top4

    ObfuscationMetric = $entropy * (1 - $top4)


Launching Modern Applications from the Command Line

We had an interesting discussion at work the other day about how to launch modern Windows applications from the command line.

There are a few solutions out there (Tome’s is close), although few of them are happy with their results 🙂

Many of them rely on protocol handlers (i.e.: “start bingnews://”), but that means memorizing a bunch of protocol handler prefixes.

Tome’s blog mentions the Get-AppxPackage cmdlet – the real workhorse of a proper solution.

Modern applications are placed in Appx packages. These packages may contain one or more applications. If an AppxPackage contains several applications, these will be listed in its package manifest – which you can retrieve through Get-AppxPackageManifest. The manifest is XML, which PowerShell’s XML support makes crazy easy to navigate.

Here’s an example of getting all of the modern applications that you can launch – place it in Get-AppxPackageEx.ps1:

param($Name = "*")

foreach($package in Get-AppxPackage)
    foreach($appId in ($package | Get-AppxPackageManifest).Package.Applications.Application.Id)
        if(($package.Name -like $Name) -or ($appId -like $Name))


There’s an example of getting all the apps you might want to launch. When it comes to launching one, the secret is to use the Shell handler to do it. Here’s a script that accomplishes it:


foreach($package in Get-AppxPackage)
    foreach($appId in ($package | Get-AppxPackageManifest).Package.Applications.Application.Id)
        if(($package.Name -like $Name) -or ($appId -like $Name))
            $commandLine = "shell:AppsFolder\$($package.PackageFamilyName)!$appId"
                start $commandLine



204 [C:\temp]
>> Start-AppxPackage *twitter* -whatif
What if: Performing the operation "Start-AppxPackage.ps1" on target "shell:AppsFolder\9E2F88E3.Twitter_wgeqdkkx372wm!App".

Saving yourself from Sender’s Remorse with Outlook

tl;dr summary: Create an Outlook rule to “delay outgoing mail by <5> minutes”, “unless body contains: е”, where the character used for the body exclusion comes from typing ALT+1077.

“When it comes to email, it seems that I always do my best proof reading immediately after I press the Send button.”

Perhaps you’ve been in the situation before, where you immediately regret sending an email in anger. Or realize after a few more moments of reflection that the email you just sent was well and truly wrong. Or you accidentally hit the “Send” button, and now have an incomplete thought floating around in everybody’s mailbox.

Fortunately, you’re not the first to run into this issue. And if you’re using Outlook, there’s a simple way to prevent it.

The first time somebody told me how they solved the problem, they pointed out that Outlook Rules Wizard lets you set the following condition on all outgoing mail:


In that, they select “Delay sending by 5 minutes”. The next screen in that wizard is the “Except when” conditions, where they said, “Unless sent with high importance”:


That sounds really handy: high-importance mail goes out immediately, while all the rest is delayed for 5 minutes. And it seemed to work – I never noticed hasty / incomplete mails from him, but he did seem like kind of a rager because you’d sometimes get mails marked urgent for totally non-urgent things.

Unfortunately, the issue with that system was that mails are sometimes time sensitive but not urgent. If you take a look at the other exclusion alternatives, nothing seems reasonable. Message size? Subject? (Who wants to tag subjects with things like: “FAST: “).

In search of a solution, I found another useful feature: configuring Outlook to send and receive every 10 minutes, even when connected to a fast network. That way, your mails wait for 5 minutes on average before leaving your outbox. You still get your incoming mail immediately, so it seems like it’s the best of all worlds.

Unfortunately, there were plenty of situations where I went to fish a mail out of my Outbox, only to find it missing.

The issue with this solution is that 5 minutes is only an average. 10% of your mails will leave your Outbox in less than a minute. 5% will vanish in less than 30 seconds. And then of course there’s the 5-10% of mails that feel like they take an entire working day to leave – so basically you’re unhappy about 20% of your outgoing mail 🙂

After some more fiddling, I finally landed on a pretty useful solution, and have been using it ever since. This one goes back to the “Delay sending by 5 minutes” approach, but with one twist. Since the main issue with exclusions in the subject / body of your mail is that externalizes your rules onto your readers, what if they didn’t notice?

Fortunately, there are a handful of characters that look like spaces, but aren’t. To type a special character in Windows, you hold down the ALT key, type the character code on the number pad of your keyboard (not the numbers on the top row of your keyboard), and then let go of the ALT key.

The one I use is ALT+1077. It is еasy to remеmbеr and rеminds me of my BBS days. It rеprеsеnts the Cyrillic letter ‘e’. I used it 10 times in the previous sentence 🙂

You have two main options here:

"Except if the subject contains specific words"


When you click on "specific words", press ALT+255, and now every mail you send with this in the subject goes out immediately. This looks like a blank space.

"Except if the body contains specific words"

This is the best - you can replace an 'e' with ALT+1077 (е) pretty much anywhere in the content of the mail, and nobody will know.


If you use ALT+255 in that dialog (like with the subject), the rule doesn't seem to work. It works fine for plain-text mails, but HTML mails don't seem to pick this up properly. I think that perhaps Outlook auto-correct replaces it with a space.

So instead, ALT+1077 is a good substitute for an 'e'.


All in all, I only have two minor gripes with these techniques for Sender's Remorse:

  1. (In the 'Body' case): ALT+1077 doesn't work on a laptop that has no number pad. If this is a big issue, you can copy and paste it from another mail, a "snippets" file, or add an international keyboard that lets you do this without the number pad.
  2. (In the 'Subject' case): If you use this technique when responding to an existing mail thread, the subject line change makes Outlook treat your mail as a fork of the thread. This is why I prefer the 'Body' approach.
  3. Outgoing mail rules are applied when you press ‘Send’ the first time. If you decide that a mail is time-sensitive after sending it, you can’t add ALT+1077 to the content to make it leave your outbox any quicker. If you decide that your mail is time sensitive after pressing send, you need to create a new mail. You can take all the content from the one in your Outbox so it’s not a huge problem, just a slight annoyance.

ScanSnap ix500 Handwriting Recognition

I got a question today about the handwriting recognition capabilities of my ScanSnap ix500. I'm still madly in love with the ScanSnap for paperwork and automatic document filing, but handwriting isn't its strong point. Here's an example document I scanned:


Here's what was recognized:

e//o vjon 'anreo/ia//po/n/
0 V/or d reqw ar 3a
eilo yor,
c 0 Worio
/C Ol2
"po/yi //e[[o is/oMiiaftc chsel Hello regularcbhcl
iiellc W(^rU ^Ic^iYcksel //e//o wr/c/^m/ec///ey
reouiar lex

Here is the actual PDF: 2015_04_09_19_03_15

Adding custom confirmation to commands

We recently had a customer question where they were concerned that some commands might be typed accidentally and end up causing significant disruption. In general, commands that fit that classification include a confirmation message to warn you, but sometimes you just don’t agree with what the cmdlet author thought was a high-impact action. While Restart-Computer might be a day-to-day operation for some servers, it might be certain doom for others.

So let’s pretend you want to make Stop-Process prompt you whenever you run it.

In PowerShell V3, the solution is amazing with PSDefaultParameterValues. In this case, it will opt in to the ‘WhatIf’ behaviour by default, which you can override if you want:

4 [C:\]
>> $PSDefaultParameterValues["Stop-Process:WhatIf"] = $true

5 [C:\]
>> Stop-Process -name notepad
What if: Performing the operation "Stop-Process" on target "notepad (5936)".

6 [C:\]
>> Stop-Process -name notepad -WhatIf:$false

(Notepad stops)                                                                                                                                                         

In PowerShell V2, the solution is still pretty handy. You can use PowerShell’s proxy function APIs insert your own logic into a cmdlet. In this case, we can have it confirm:

19 [C:\]
>> Set-HighImpactCommand Stop-Process

20 [C:\]
>> Stop-Process -Name Notepad

Invoke Stop-Process?
Stop-Process has high impact. Invoke?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): N
Invocation of Stop-Process was discontinued.
At line:25 char:106
+ ... -Process?')) { throw 'Invocation of Stop-Process was discontinued.' }
+                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Invocation of S...s discontinued.:String) [], RuntimeException
    + FullyQualifiedErrorId : Invocation of Stop-Process was discontinued.

21 [C:\]
>> Stop-Process -Name Notepad

Invoke Stop-Process?
Stop-Process has high impact. Invoke?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y

22 [C:\]                                                                                                                                                        

And the function that does the magic?












function Set-HighImpactCommand




    $MetaData = New-Object System.Management.Automation.CommandMetaData (Get-Command $CommandName -CommandType Cmdlet)

    $functionContent = ([System.Management.Automation.ProxyCommand]::Create($MetaData))

    $updatedFunction = $functionContent -replace "begin`r`n{",

        "begin`r`n{`r`n if(-not `$PSCmdlet.ShouldContinue('$CommandName has high impact. Invoke?', 'Invoke ${CommandName}?')) { throw 'Invocation of $CommandName was discontinued.' }`r`n"

    Set-Item function:\GLOBAL:$CommandName $updatedFunction


The Wonderful World of PowerShell Filtering and Globbing

If you’ve been using PowerShell for long, you are probably familiar with the concept of wildcards. At the very least, you’ve done something like this:

PS C:\temp> dir *.txt

    Directory: C:\temp

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         1/21/2015  10:01 AM        664 test.txt                                                                                                                                                   

Or perhaps you’ve taken a lap or two around about_wildcards and now type things like this in your sleep:

PS C:\temp> dir C:\win*\*.N[a-f]?\F*\v2*\csc.exe

    Directory: C:\Windows\Microsoft.NET\Framework\v2.0.50727

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         5/26/2014   9:39 PM      77960 csc.exe

    Directory: C:\Windows\Microsoft.NET\Framework64\v2.0.50727

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         5/26/2014   9:39 PM      88712 csc.exe                                                                                                                                                    

While wildcarding in the Path parameter is both powerful and useful, you might have seen another parameter: Filter.

In the PowerShell documentation, we describe the –Filter parameter in Get-ChildItem as:


Specifies a filter in the provider's format or language. The value of this parameter qualifies the Path parameter. The syntax of the filter, including the use of wildcards, depends on the provider. Filters are more efficient than other parameters, because the provider applies them when retrieving the objects, rather than having Windows PowerShell filter the objects after they are retrieved.

In a SQL provider, the –Filter parameter might offer SQL syntax (like: –Filter “WHERE Name LIKE %pattern%”). Or, the AD provider might offer LDAP syntax. In the FileSystem provider, PowerShell’s wildcard syntax (dir *.txt) is very similar to the NTFS “format or language” which also looks like: *.txt. In the Filesystem Provider’s case, the Win32 API (FindFirstFile) takes a pattern parameter that is then processed by the API itself.

When you use wildcards in cmd.exe, file resolution and wildcarding is done directly by this Win32 API.

What’s the difference?

Now you might wonder about Filesystem: if both wildcards and filters are so similar, why does PowerShell need its own? Why not just call the Win32 API like cmd.exe does?

The primary distinction is around power. As about_wildcards mentions, PowerShell offers the character and character range operators. The native Win32 API does not.

Wildcard Description        Example  Match             No match

-------- ------------------ -------- ----------------- --------

*        Matches zero or    a*       A, ag, Apple      banana

         more characters

?        Matches exactly    ?n       an, in, on        ran

         one character in

         the specified


[ ]      Matches a range    [a-l]ook book, cook, look  took

         of characters

[ ]      Matches specified  [bc]ook  book, cook        hook


There’s also a surprising distinction around correctness. Try these examples in your System32 directory.

Should return all files with three-letter extensions:

$r1 = dir *.???

$r2 = dir –Filter *.???

Compare-Object $r1 $r2 –Property FullName

(Oops! –Filter returns directories, as well as files with 1 or 2 letter extensions!)

Should return all files with “2” in the name

$r1 = dir *2*

$r2 = dir –Filter *2*

Compare-Object $r1 $r2 –Property FullName

(Oops! –Filter returns a ton of stuff without “2” in the name.)

For the last example, this is because native wildcard filters ALSO work against the 8.3 filename representation!

PS C:\windows\system32> cmd /c dir /x *2* | sls SqlServerSpatial.dll

04/03/2010  10:57 AM           459,104 SQLSER~2.DLL SqlServerSpatial.dll                                                                                                                                

But what about performance?

Native Filesystem filters are unquestionably faster than PowerShell doing all of the wildcard matching on its own. However, PowerShell doesn’t do all of the matching on its own. In Version 2, we added support for partial filtering to the Filesystem provider (and of course, to any other provider that wants to implement it). When the Filesystem provider applies this partial filtering, it offloads as much of the filtering work as it can to the raw Win32 APIs – and then does more powerful (and correct) wildcard matching on the smaller set of results.

So now you know – when it comes to the Filesystem provider, you probably don’t want or need the –Filter parameter!


If you want to know more about wildcarding in the Filesystem provider, this is covered in Recipe 20.6 in the PowerShell Cookbook, which you can preview for free here: Find Files that Match a Pattern.

Extracting Tables from PowerShell’s Invoke-WebRequest

If you’ve ever wanted to extract tables from a web page in PowerShell, the Invoke-WebRequest cmdlet is exactly what the doctor ordered.

Once you’ve invoked the cmdlet, the ‘ParsedHtml’ property gives you access to the Internet Explorer DOM of that page. From there, you can get elements by tag name (“TABLE”), ID, and more.

One neat application of this technique is to automatically parse data out of tables on the web page. I recently needed to do this, and the PowerShell script really wasn’t that complicated. In true PowerShell style, each row of the table is output as an object – that way, you can access the data as you would with any other PowerShell cmdlet. Even better - if the table uses the TH tag (“Table Heading”), it uses those headings as property names for the output objects.

Here’s an example of it in action:

1 [C:\Users\leeholm]
>> $url = ''

2 [C:\Users\leeholm]
>> $r = Invoke-WebRequest $url

3 [C:\Users\leeholm]
>> Get-WebRequestTable.ps1 $r -TableNumber 0 | Format-Table -Auto

P1              P2         P3                   P4
--              --         --                   --
Gardiner Number Hieroglyph Description of Glyph Details
Q1                         Seat                 Phono. st, ws, . In st ?seat, place,? wsir ?Osiris,? ?tm ?perish.?
Q2                         Portable seat        Phono. ws. In wsir ?Osiris.?
Q3                         Stool                Phono. p.
Q4                         Headrest             Det. in wrs ?headrest.?
Q5                         Chest                Det. in hn ?box,? ?fdt ?chest.?
Q6                         Coffin               Det. or Ideo. in qrs ?bury,? krsw ?coffin.?
Q7                         Brazier with flame   Det. of fire. In ?t ?fire,? s?t ?flame,? srf ?temperature.?

4 [C:\Users\leeholm]                                                                                                    

And the script:


















































    [Parameter(Mandatory = $true)]

    [Microsoft.PowerShell.Commands.HtmlWebResponseObject] $WebRequest,


    [Parameter(Mandatory = $true)]

    [int] $TableNumber


## Extract the tables out of the web request

$tables = @($WebRequest.ParsedHtml.getElementsByTagName("TABLE"))

$table = $tables[$TableNumber]

$titles = @()

$rows = @($table.Rows)

## Go through all of the rows in the table

foreach($row in $rows)


    $cells = @($row.Cells)


    ## If we've found a table header, remember its titles

    if($cells[0].tagName -eq "TH")


        $titles = @($cells | % { ("" + $_.InnerText).Trim() })



    ## If we haven't found any table headers, make up names "P1", "P2", etc.

    if(-not $titles)


        $titles = @(1..($cells.Count + 2) | % { "P$_" })


    ## Now go through the cells in the the row. For each, try to find the

    ## title that represents that column and create a hashtable mapping those

    ## titles to content

    $resultObject = [Ordered] @{}

    for($counter = 0; $counter -lt $cells.Count; $counter++)


        $title = $titles[$counter]

        if(-not $title) { continue }


        $resultObject[$title] = ("" + $cells[$counter].InnerText).Trim()


    ## And finally cast that hashtable to a PSCustomObject

    [PSCustomObject] $resultObject