Archives for the Month of October, 2009

Mathematical Pumpkins

Over the past few years, Pumpkin carving in my family has somehow ended up focusing on two themes: Math, and Knitting.

A Sierpinski Triangle -- which surprisingly only took a toothpick or two to repair isolated triangles:

Sierpinski Carpet (along with a wee bit of evil, of course:)

 

Not being one to cut 64 of the level-three squares by hand, a cordless drill came in extremely handy.

Mandelbrot, and Koch snowflake:

030 112

 

Knitting randoms:

106 109

And just a cool cat in a window:

Picture 636

Unfortunately, when I'm out trick-or-treating, there's nobody around to give candy to the little monsters.  I leave a note above a bowl on a chair – and now I finally know why my calligraphy pens include red in the set!

 

Removing Deleted Items Left by the iPhone

One thing you might notice if you have an iPhone connecting to an Inbox via the IMAP protocol is that messages you delete tend to stick around when viewed from other devices (such as Outlook, Outlook Web Access, etc.)

This is caused by an out-of-date view of mail management, where your Inbox handles everything. When you delete an item, some IMAP clients (such as the iPhone) mark them as deleted, but don’t actually remove the item from the server. Some clients hide these deleted items. Some show them with a line through them. Some ignore the deleted flag altogether.

Most email clients (including iPhone) move deleted items to their own sub-folder, so marking items as deleted just ends up being an annoyance:

http://discussions.apple.com/thread.jspa?messageID=4934598
http://blogs.freshlogicstudios.com/Posts/View.aspx?Id=44a01293-3b32-4ee0-b23c-fac99348e1cd
http://www.robichaux.net/blog/2007/07/iphone_vs_windows_mobile_part_3_mail.php

The solution is the IMAP ‘EXPUNGE’ command. It permanently deletes items that have been marked for deletion (while of course leaving everything in Deleted Items untouched.) The iPhone has an option to do this once a day, but it doesn’t seem to work very well (and a day lag feels like an eternity, anyways.)

The folks at freshlogic have a fine application to run the command against your mail server. We can improve on it in a few ways by writing it in PowerShell. Our version:

  • Enables support for securely cached credentials
  • Doesn’t require an always-running application
  • Runs in the background

To do this, our script imports our saved password, and then uses Send-TcpRequest to simply connect to the server and send the EXPUNGE command. Finally, we use schtasks.exe to automate it.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
# 1) Get your password into a securestring:
# $ss = Read-Host -AsSecureString
# 2) Export it to disk
# $ss | ConvertFrom-SecureString |
# Out-File (Join-Path (Split-Path $profile) Sync-MailboxDeletedState.ps1.credential)
# 3) Create the scheduled task
# schtasks /Create /TN MailboxUpdater /SC MINUTE /MO 5 /TR
# "powershell -NoProfile -WindowStyle Hidden -File 'c:\path\to\Sync-MailboxDeletedState.ps1'"

$username = "[email protected]"
$server = "imap.example.com"
$port = 993

$passwordPath = Join-Path (Split-Path $profile) Sync-MailboxDeletedState.ps1.credential
$password = Get-Content $passwordPath | ConvertTo-SecureString

$cred = New-Object System.Management.Automation.PsCredential $username,$password

$nc = $cred.GetNetworkCredential()
$commands = "A1 LOGIN $($nc.UserName + '@' + $nc.Domain) $($nc.Password)",
            'A2 SELECT INBOX',
            'A3 EXPUNGE',
            'A4 LOGOUT'

$commands | Send-TcpRequest $server $port -UseSSL

Scripting Network / TCP Connections in PowerShell

[Edit: This function has been improved and added to the PowerShell Cookbook module. Get it from the PowerShell Gallery.]

Awhile back, I introduced a script that allows you interact with remote TCP ports (such as Telnet.) While useful, it worked only interactively. It would be even more useful if you were able to script a network or TCP connection.

Let me introduce Send-TcpRequest.ps1 v2, which allows exactly that:

First, a simple scripted HTTP session:

$http = @"
GET / HTTP/1.1
Host:search.msn.com
`n`n
"@

$http | Send-TcpRequest search.msn.com 80

Second, a scripted POP3 session (Parse-TextObject comes from here: http://www.leeholmes.com/blog/parsetextObjectAWKWithAVengeance.aspx):

if(-not (test-path Variable:\mailCredential))
{
$mailCredential = Get-Credential
}
$address = $mailCredential.UserName
$password = $mailCredential.GetNetworkCredential().Password
$pop3Commands = "USER $address","PASS $password","STAT","QUIT"
$output = $pop3Commands | Send-TcpRequest mail.leeholmes.com 110
$inbox = $output.Split("`n")[3]
$status = $inbox | Parse-TextObject -PropertyName "Response","Waiting","BytesTotal","Extra"
"{0} messages waiting, totaling {1} bytes." -f $status.Waiting,$status.BytesTotal

 

Now, here is Send-TcpRequest.ps1

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
##############################################################################
## Send-TcpRequest.ps1
##
## From Windows PowerShell Cookbook (O'Reilly)
## by Lee Holmes (http://www.leeholmes.com/guide)
##
## Send a TCP r
equest to a remote computer, and return the response.

## If you do not supply input to this script (via either the pipeline, or the
## -InputObject parameter,) the script operates in interactive mode.
##
## Example:
##
## $http = @"
## GET / HTTP/1.1
## Host:search.msn.com
## `n`n
## "@
##
## $http | Send-TcpRequest search.msn.com 80
##############################################################################
param(
        [string] $remoteHost = "localhost",
        [int] $port = 80,
        [switch] $UseSSL,
        [string] $inputObject,
        [int] $commandDelay = 100
     )[string] $output = ""

## Store the input into an array that we can scan over. If there was no input,
## then we will be in interactive mode.
$currentInput = $inputObject
if(-not $currentInput)
{
    $SCRIPT:currentInput = @($input)
}
$scriptedMode = [bool] $currentInput

function Main
{
    ## Open the socket, and connect to the computer on the specified port
    if(-not $scriptedMode)
    {
        write-host "Connecting to $remoteHost on port $port"
    }

    trap { Write-Error "Could not connect to remote computer: $_"; exit }
    $socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port)

    if(-not $scriptedMode)
    {
        write-host "Connected. Press ^D followed by [ENTER] to exit.`n"
    }

    $stream = $socket.GetStream()
   
    if($UseSSL)
    {
        $sslStream = New-Object System.Net.Security.SslStream $stream,$false
        $sslStream.AuthenticateAsClient($remoteHost)
        $stream = $sslStream
    }

    $writer = new-object System.IO.StreamWriter $stream

    while($true)
    {
        ## Receive the output that has buffered so far
        $SCRIPT:output += GetOutput

        ## If we're in scripted mode, send the commands,
        ## receive the output, and exit.
        if($scriptedMode)
        {
            foreach($line in $currentInput)
            {
                $writer.WriteLine($line)
                $writer.Flush()
                Start-Sleep -m $commandDelay
                $SCRIPT:output += GetOutput
            }

            break
        }
        ## If we're in interactive mode, write the buffered
        ## output, and respond to input.
        else
        {
            if($output) 
            {
                foreach($line in $output.Split("`n"))
                {
                    write-host $line
                }
                $SCRIPT:output = ""
            }

            ## Read the user's command, quitting if they hit ^D
            $command = read-host
            if($command -eq ([char] 4)) { break; }

            ## Otherwise, Write their command to the remote host
            $writer.WriteLine($command)
            $writer.Flush()
        }
    }

    ## Close the streams
    $writer.Close()
    $stream.Close()

    ## If we're in scripted mode, return the output
    if($scriptedMode)
    {
        $output
    }
}

## Read output from a remote host
function GetOutput
{
    ## Create a buffer to receive the response
    $buffer = new-object System.Byte[] 1024
    $encoding = new-object System.Text.AsciiEncoding

    $outputBuffer = ""
    $foundMore = $false

    ## Read all the data available from the stream, writing it to the
    ## output buffer when done.
    do
    {
        ## Allow data to buffer for a bit
        start-sleep -m 1000

        ## Read what data is available
        $foundmore = $false
        $stream.ReadTimeout = 1000

        do
        {
            try
            {
                $read = $stream.Read($buffer, 0, 1024)

                if($read -gt 0)
                {
                    $foundmore = $true
                    $outputBuffer += ($encoding.GetString($buffer, 0, $read))
                }
            } catch { $foundMore = $false; $read = 0 }
        } while($read -gt 0)
    } while($foundmore)

    $outputBuffer
}

. Main

[Edit: Thanks to Marco for pointing out a few issues that come up when you try to retrieve massive amounts of data (such as a newsgroup listing). I've updated the script to fix those.]
[Edit2: Updated to call it Send-TcpRequest, and support SSL]

PowerShell Activity Tracker

This work has progressed. Please see the updated version at: http://www.leeholmes.com/blog/2017/02/10/timejournal-time-profiling-for-humans/

PowerShell Cryptography Library

When playing with cryptography challenges (don’t we all?,) you end up leaning on a bunch of common tasks. For example, substituting all text in a string with a set of replacements (substitution ciphers,) XORing strings together, applying dictionary-based algorithms, investigating word frequency, and more.

PowerShell lends itself really well to these challenges, and I’ve developed a small cryptography library along the way. Here it is. Your job, as a cryptographer, is to uncover the hidden comments 🙂 (Hint: don’t spend very long. Some problems are unbreakable.)

For example, here’s the script I used (interactively) to solve an “XOR Cipher” at http://cryptolib.com/challenges.php.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
$cipherHex1 = @"
3526532A122A183D4D0E4A2E3713101954013E2B3E2246235728
573553064832781F164A481F2A2B396350341E2B5E3A5E1C4861
301C11544F1D22363863413F192A4A36520A0D2D2D13044A001A
252025335E2F142F5A355A
"@

$cipherHex2 = @"
32335734163A5136510E41612818114A4912252C2722462F1820
4B7956014E283C180D4D001522312724573416225C794C074C2C
3D0E435A4F063B29383012251B214C315A1C41283618435E5212
2531342D556612205B2B461F592837131019481225212F225B2A
"@

$words = @{}
Get-Content dictfull.txt | % { $words[$_] = $true }

## Turn the cipher text into a single string
$cipherText1 = $cipherHex1 -replace "`r`n",""
$cipherText2 = $cipherHex2 -replace "`r`n",""

## Convert hex to decimal
$cipherBytes1 = $cipherText1 -split '(..)' | ? { $_ } | % { [Convert]::ToInt32($_, 16) }
$cipherBytes2 = $cipherText2 -split "(..)" | ? { $_ } | % { [Convert]::ToInt32($_, 16) }

## Drop the key by XORing the two cipher texts together
$plainTextCombined = XORBytes $cipherBytes1[0..19] $cipherBytes2[0..19]

## Initialize the key
$key = ,0x20*20

## First break comes by scanning through the entire dictionary, guessing the
## first word of one plaintext, then seeing if that corresponds to a key that makes
## the second plaintext look reasonable
$output = $(foreach($word in $words.GetEnumerator() | % { $_.Key })
{
    $testString = XORBytes $plainTextCombined[0..($word.Length - 1)] ($word.ToCharArray())
    (-join [char[]]$testString) + " : $word"
}
)

## Output the list to a file, then clean the file to include only alpha-numeric
## characters.
$output > first_word.txt
$alphaOnly = gc first_word.txt | select-string '^[a-zA-Z ]+ : .*$'
$alphaOnly > first_word_alpha.txt

## The first break comes from seeing that "healed dr" corresponds to "operation".
## Make a guess of that as the first 9 elements of the key.
$k
ey
[0],$key[1],$key[2],$key[3],$key[4],$key[5],$key[6],$key[7],$key[8] = XORBytes ("healed dr".ToCharArray()) $cipherBytes1[0..8]

## These other magic values come from looking at the output and making educated guesses
$key[9],$key[10] = XORBytes ("se".ToCharArray()) $cipherBytes1[49..50]
$key[11],$key[12],$key[13] = XORBytes ("ion".ToCharArray()) $cipherBytes2[91..93]
$key[14],$key[15],$key[16] = XORBytes ("gs ".ToCharArray()) $cipherBytes1[74..76]
$key[17],$key[18],$key[19] = XORBytes ("Fit".ToCharArray()) $cipherBytes2[37..39]

## Display the currently-guessed plain texts alongside eachother,
## including the key and index into the cipher text.
for($counter = 0; $counter -lt $cipherBytes2.Count; $counter++)
{
    $keyChar = $key[$counter % 20]
    "{0} {1} {2} {3}" -f $counter,([char] $keyChar),([char] ($cipherBytes1[$counter] -bxor $keyChar)),([char] ($cipherBytes2[$counter] -bxor $keyChar))
}

-join [char[]] (XORBytes $key $cipherBytes1)
-join [char[]] (XORBytes $key $cipherBytes2)
-join [char[]] $key

 

And the library:

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
function Get-SubstitutedText($cipherText, $substitutions)
{
    $content = $(foreach($char in [char[]] $cipherText)
    {
        if($substitutions["$char"])
        {
            $substitutions["$char"]
        }
        else
        {
            "."
        }
    }
    )
   
    -join $content
}

function XORBytes($passwordBytes, $cipherBytes2)
{
    $combinedBytes = @()
    for($counter = 0; $counter -lt $cipherBytes2.Length; $counter++)
    {
        $combinedBytes += $passwordBytes[$counter % $passwordBytes.Length] -bxor $cipherBytes2[$counter]
    }
   
    $combinedBytes
}

function Search-DictionaryForWord($pattern, $floating)
{
    if(-not $floating) { $pattern = "^$pattern`$" }
    Select-String $pattern wordlist.txt
}

function Search-DictionaryForPattern($pattern, $floating)
{
    $substituted = ""
    $toFind = ""
   
    if(-not $floating) { $toFind += "^" }
   
    foreach($char in $pattern.ToLower().ToCharArray())
    {
        if($char -eq ".")
        {
            $toFind += "."
            continue
        }
       
        $subIndex = $substituted.IndexOf($char)
       
        if($subIndex -ge 0)
        {
            $toFind += "\" + ($subIndex + 1)
        }
        else
        {
            if(-not $substituted)
            {
                $toFind += "(.)"
            }
            else
            {
                $toFind += "([^$substituted])"
            }
            $substituted += $char
        }
    }
   
    if(-not $floating) { $toFind += '$' }
    Write-Verbose $toFind
    Select-String $toFind wordlist.txt
}

function Measure-LetterFrequency($cipherText)
{
    $frequentLetters = " etnoriasfindhdlcumwfgypbvkjxqz"

    "`nLetter Frequencies:`n"
    $groups = [char[]] $cipherText.ToLower() | group | Sort -Desc Count

    $groupNumber = 0
    $(foreach($group in $groups)
    {
        $group | Select Name,
            @{
                Name = "Replacement";
                Expression = { $frequentLetters[$groupNumber] } },
            Count,
            @{
                Name = "Percent";
                Expression = { "{0:..%}" -f ($_.Count / $cipherText.Length) } }
        $groupNumber++  
    }) | ft -auto | out-string | % { $_.Trim() }
}

function Measure-BigraphFrequency($cipherText)
{
    $frequentBigraphs = "th","he","an","in","er","on","re",
        "ed","nd","ha","at","en","es","of","nt","ea","ti",
        "to","io","le","is","ou","ar","as","de","rt","ve"
       
    $cipherText = $cipherText -replace " ",""
    $cipherText = $cipherText -replace "\W",""

    "`nBigraph Frequencies:`n"
    $groups = 
        @(($cipherText -split '(..)') +
          ($cipherText.Remove(0,1) -split '(..)')) |
          ? { $_.Trim().Length -eq 2 } | group | Sort -Desc Count

    $groupNumber = 0
    $(foreach($group in $groups)
    {
        $group | Select Name,
            @{
                Name = "Replacement";
                Expression = { $frequentBigraphs[$groupNumber] } },
            Count
        $groupNumber++  
    }) | ft -auto | out-string | % { $_.Trim() }
}

function Measure-DoubleLetterFrequency($cipherText)
{
    $frequentDoubles = "ss","ee","tt","ff","ll","mm","oo"
       
    "`nDouble Frequencies:`n"
    $groups = [Regex]::Matches($cipherText, '(.)\1') | % { $_.Value } |
        group | Sort -Desc Count

    $groupNumber = 0
    $(foreach($group in $groups)
    {
        $group | Select Name,
            @{
                Name = "Replacement";
                Expression = { $frequentDoubles[$groupNumber] } },
            Count
        $groupNumber++  
    }) | ft -auto | out-string | % { $_.Trim() }
   
    "`nOther groups:"
    "$($frequentDoubles[$groupNumber..($frequentDoubles.Length - 1)])"
}

function Measure-InitialFrequency($cipherText)
{
    $frequentInitialLetters = "toawbcdsfmrhiyeglnoujk"

    "`nInitial letters:`n"
    $groups = -split $cipherText.ToLower() | % { ([char[]] $_)[< /span>0] } | group | Sort -Desc Count

    $groupNumber = 0
    $(foreach($group in $groups)
    {
        $group | Select Name,
            @{
                Name = "Replacement";
                Expression = { $frequentInitialLetters[$groupNumber] } },
            Count,
            @{
                Name = "Percent";
                Expression = { "{0:..%}" -f ($_.Count / $groups.Count) } }
        $groupNumber++  
    }) | ft -auto | out-string | % { $_.Trim() }
}

function combo($string)
{
    if($string.Length -eq 2)
    {
        $string
        $string[-1] + $string[0]
    }
    else
    {
        foreach($element in $string.ToCharArray())
        {
            combo ($string -replace $element,'') | foreach { $element + $_ }
            combo ($string -replace $element,'') | foreach { $_ }
        }
    }
   
}

PowerShell RPN Calculator

Adam Barr blogged bits and pieces of a PowerShell RPN calculator a few years ago: first the basics, and then some tweaks to clean it up. An RPN calculator, if you haven’t played with one before, flips the way you enter data. Rather than type “2 + 2”, you type “2 2 +”. RPN-style calculation supposedly has lots of great benefits. While I understand it and can do it, I wouldn’t say I “get it.”

Anyways.

One of the things he runs into with the last version is the PowerShell’s (version one) inability to dynamically invoke static methods:

… you should be able to write:

$add = [Decimal]::Add
$add.Invoke(2,3)

but it doesn't work. That's not exactly the same as getting the value of a static property, but maybe there's a related issue, or maybe I just can't figure out how to do it right.

Well, in version two it does work, and that makes the RPN calculator a whole lot cleaner.

Another interesting aspect about the latest implementation is that it has a lot of very similar code segments. It has tables for operations on the Double class that take one argument, operations on the Double class that take two arguments, operations on the Math class that take one argument, and operations on the Math class that take two arguments. Afterward, there are four nearly identical blocks of code that perform the operation and store the result.

Note: This of course isn’t a slag on the implementation, this is just continuing the tinkering on an interesting concept.

We can simplify this in two ways:

  1. Don’t create hard-coded lists of operators. Instead, we’ll look at all methods in the Math and Double class to see if one matches. This approach gives us 30-something operators for free, and perhaps more in future versions of the .NET Framework.
  2. Don’t create hard-coded lists of arity: the number of arguments consumed by an operator. Instead, we’ll look at the method overloads that match the operator name, and see how many arguments they consume.

Now, there are some subtleties to both points:

  1. The method names in the Decimal and Math classes get kind of long. You don’t want to have to write “2 3 Multiply” in an RPN calculator. To work around that, we’ll define a hashtable of shortcuts that simply map operators to their names.
  2. Some operators have multiple overloads. For example, the one-argument Round method rounds a number to zero decimal points. The two-argument Round method rounds it to the specified number of decimal points.

By leveraging PowerShell’s built-in support for introspection and dynamic method invocation, we now have a script that is both much shorter, and much more powerful.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
##############################################################################
##
## rpn.ps1
##
##############################################################################

<#

.SYNOPSIS
Evaluates a statement as an RPN calculator. Supports all operations from
System.Math and System.Decimal.

.EXAMPLE
rpn 2 2 +

.EXAMPLE
rpn 2 3 + 1.3 / 2 Round2 Negate

#>

$s = new-object System.Collections.Stack
$n = 0d

$shortcuts = @{
    "+" = "Add"; "-" = "Subtract"; "/" = "Divide"; "*" = "Multiply";
    "%" = "Remainder"; "^" = "Pow"; "||" = "Abs"
}

:ARGLOOP foreach ($a in $args) {
    if($shortcuts[$a]) { $a = $shortcuts[$a] }

    ## First, see if it's a number. If so, push it.
    try { $s.Push( [Decimal] $a ); continue } catch {}

    ## It's an operation. Extract the operation name
    ## (such as Floor, Round, etc.) It may also represent a
    ## specific operation (such as Round2 - Round to specified precision).
    $argCountList = $a -replace "(\D+)(\d*)",'$2'
    $op = $a.Substring(0, $a.Length - $argCountList.Length)

    ## We support any static operations from the Decimal or Math classes
    foreach($type in [Decimal],[Math])
    {
        if($definition = $type::$op)
        {
           
## If they haven't specifically given the number of arguments,
            ## see how many this method supports. We go through each overload
            ## definition, and see how many commas it has.
            if(-not $argCountList)
            {
                $argCountList = $definition.OverloadDefinitions |
                    Foreach-Object { ($_ -split ", ").Count } |
                    Sort-Object -Unique
            }

            ## Now, for each overload, see if we can call it.
            foreach($argCount in $argCountList)
            {
                try
                {
                    $methodArguments = $s.ToArray()[($argCount-1)..0]
                    $result = $type::$op.Invoke($methodArguments)

                    ## If we were able to call the method, pop all of its
                    ## arguments off of the stack.
                    $null = 1..$argCount | % { $s.Pop() }

                    ## Then push the result
                    $s.Push($result)
                    continue ARGLOOP

                }
                catch
                {
                    ## If we catch an error, try with the next number of
                    ## arguments
                }
            }
        }
    }
}

$s