NODENEXUS

Average Time to Check If an Array Is Sorted

Sat, 29 Oct 2016 00:00:00 +0000

It’s always a pleasure to find interesting problems hiding in seemingly mundane places. Recently a colleague of mine posed the following twist on a basic algorithms question. It’s easy to see that the worst-case performance to check if a random $n$-length array is sorted is $O(n)$. But what is the average performance? Is it still linear? Sublinear? Is it a function of the length at all?

Throughout this post, we’ll assume that all arrays are drawn uniformly at random from the set of $n$-length arrays containing the numbers $1$ through $n$ with no duplicates. Of course, if an adversary is providing the input arrays, you should expect the average and worst cases to be the same.

Let’s start out by writing out exactly what we’re trying to compute. We check if a random array is sorted by performing comparisons between each adjacent pair of elements, returning either when we find an inversion or reach the end. The average performance is equal to the expected number of comparisons needed to find the first inversion in a random array. That is,

$E[C] = \sum_{i=1}^{n-1} P(c_i) * i$

where $C$ is the random variable for the number of comparisons performed, $c_i$ is the index (starting at one) of the first inversion, $P(c_i)$ is the probability of seeing it at that location, and $n$ is the array length.

That’s the simple expression, but for reasons that I’ll explain momentarily, we’ll rewrite it using conditional probabilities:

$\small E[C] = P(c_1) + P(\overline{c_1}) * \bigg( P(c_2|\overline{c_1}) * 2 + P(\overline{c_2}|\overline{c_1}) * \Big(... + P(\overline{c_{n-2}}|\overline{c_1},...,\overline{c_{n-3}}) * (n-1)\Big)\bigg)$

where $P(\overline{c_i})$ is the complement of $P(c_i)$, or $1 - P(c_i)$. We can see that quations $(1)$ and $(2)$ are equal by distributing the complementary terms, working from the outside in.

Now we need to find a way to compute the values for the probabilities. If the array were small, say four elements long, we could just enumerate all sorted subarrays and count how many of the remaining choices of next element introduce inversions. The subarrays are sorted because, as the conditional probabilities represent, we are interested in the likelihood of seeing the first inversion when we append the next element. The conditional probabilities save us from having to enumerate subarrays beyond their first inversions, saving a little time and memory in a program.

$\notag \begin{array}{c|c|c} \text{Subarray} & \text{# Choices} & \text{# Causing Inversions} \\ \hline [1] & 3\ (2, 3, 4) & 0 \\ [2] & 3\ (1, 3, 4) & 1\ (1) \\ [3] & 3\ (1, 2, 4) & 2\ (1, 2) \\ [4] & 3\ (1, 2, 3) & 3\ (1, 2, 3) \\ \hline \text{Section Total} & 12 & 6 \\ \hline [1,2] & 2\ (3, 4) & 0 \\ [1,3] & 2\ (2, 4) & 1\ (2) \\ [1,4] & 2\ (2, 3) & 2\ (2, 3) \\ [2,3] & 2\ (1, 4) & 1\ (1) \\ [2,4] & 2\ (1, 3) & 2\ (1, 3) \\ [3,4] & 2\ (1, 2) & 2\ (1, 2) \\ \hline \text{Section Total} & 12 & 8 \\ \hline [1,2,3] & 1\ (4) & 0 \\ [1,2,4] & 1\ (3) & 1\ (3) \\ [1,3,4] & 1\ (2) & 1\ (2) \\ [2,3,4] & 1\ (1) & 1\ (1) \\ \hline \text{Section Total} & 4 & 3 \\ \end{array}$

Of course, we want to answer the question for much larger arrays, so let’s write a quick and dirty script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70 #!/usr/bin/env python

#
# inversion_count.py
#

from collections import deque
import sys


def main():
    arr_length = int(sys.argv[1])

    # Use a deque like a linked list
    states = deque()
    for i in xrange(arr_length):
        choices = set()

        for j in xrange(arr_length):
            if i != j:
                choices.add(j)

        states.append({
            "progress": [i],
            "choices": choices
        })

    results_by_length = {}

    while len(states) > 0:
        state = states.popleft()
        state_length = len(state["progress"])

        # If we haven't seen a subarray of this length, add category to results
        if state_length not in results_by_length:
            results_by_length[state_length] = {
                "total": 0,
                "inverted": 0
            }

        for choice in state["choices"]:
            results_by_length[state_length]["total"] += 1

            if state["progress"][-1] > choice:
                results_by_length[state_length]["inverted"] += 1
            else:
                # We only add a new state if our progress in sorted...
                choices = set(state["choices"])
                choices.remove(choice)

                if choices:
                    # And there are still choices available to the new state
                    new_progress = list(state["progress"])
                    new_progress.append(choice)

                    new_state = {
                        "progress": new_progress,
                        "choices": choices
                    }
                    states.append(new_state)

    for length, result in results_by_length.iteritems():
        print("%d: %d / %d" % (length, result["inverted"], result["total"]))


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print "Usage error: Expects desired array length as an argument"
    else:
        main()

$ chmod +x inversion_count.py
$ ./inversion_count.py 3
1: 3/6
2: 2/3
$ ./inversion_count.py 5
1: 10 / 20
2: 20 / 30
3: 15 / 20
4: 4 / 5
$ ./inversion_count.py 10
1: 45 / 90
2: 240 / 360
3: 630 / 840
4: 1008 / 1260
5: 1050 / 1260
6: 720 / 840
7: 315 / 360
8: 80 / 90
9: 9 / 10

We can see see that the probability of seeing the first inversion on the $i$th comparison follows the sequence

$\notag \frac{1}{2}, \frac{2}{3}, \frac{3}{4}, \frac{4}{5}...$

regardless of the length of the array. So we can plug back into equation $(2)$:

$E[C] = \frac{1}{2} + \frac{1}{2} \bigg( \frac{2}{3} * 2 + \frac{1}{3} \Big( \frac{3}{4} * 3 + \frac{1}{4} \big(... + \frac{1}{n-2} * (n - 1) \big) \Big) \bigg)$

Now we distribute the fraction multiplication across the nested expressions and get the coefficient sequence

$\notag \frac{1}{2}, \frac{1}{3}, \frac{1}{8}, \frac{1}{30}, \frac{1}{144}, \frac{1}{840} ...$

Luckily enough for people like myself who don’t see the pattern immediately, the Online Encyclopedia of Integer Sequences has an entry for this sequence¹:

$a(n) = \frac{n}{(n+1)!}$

Thus we have everything we need to rewrite equation $(2)$ cleanly:

$E[C] = \sum_{i=1}^{n-1} \frac{n^2}{(n+1)!}$

Expanding out the terms, we can see the sum converges to a familiar and unexpected value.

$E[C] = \frac{1}{2} + \frac{2}{3} + \frac{3}{8} + \frac{4}{30} + \frac{5}{144} + \frac{6}{840} + ... \approx 1.718281 = e - 1$

I don’t claim to understand why Euler’s Number is involved, but I find it both curious and convenient that the expected time it takes to check for array sortedness is constant. Even as your arrays grow extremely large, you can rest easy knowing your function isSorted(arr) is unlikely to take anywhere close to linear time.

Note that OEIS has the equation for the sequence in the numerator, so we use the inverse. ↩

Shell Fu in zsh

Thu, 09 Apr 2015 00:00:00 +0000

Programmers are a lot like exotic car owners. We live for the cutting edge and the most efficient solutions. We’re not afraid of a little (or a lot) of DIY. Some of us are actually crazy enough to cool our rigs with liquid nitrogen in the name of clock-speed records. I’m sure supercar enthusiasts would understand.

Whereas auto modding is all in the physical world, however, computer performance is a function of hardware and software tuning. Despite being the proud builder of my own (air-cooled) gaming rig, I’ll be the first to admit that I don’t really know too much about hardware hackery. I do, however, have some tips when it comes to fine-tuning your development environment. Let me share with you some secrets of shell fu.

At a university where many (most?) classes use Java, shell fu technique seems Bernoulli-distributed: you either customize to your heart’s desire or not at all. That means that many people could increase their productivity in the shell with just a couple of aliases and other simple tricks. I’ll start with the low-hanging fruit and then move on to some more advanced tweaks for those of you who have seen some things before.

Oh. My. Zsh.

bash (the Bourne-again shell) is probably the world’s most popular shell. Unless you’ve changed it yourself, your machine’s default shell is almost certainly bash. As the de facto standard, bash is a time-tested option with good documentation. That being said, if you’re not afraid of new things, I highly suggest that you make a leap of faith and make the Z shell your default.

What’s the difference between zsh and bash, you ask? zsh was built to be a csh replacement with support for many ksh features, particularly autocompletion. Its scripting language is similar to bash’s with various syntactic quirks.¹ Unless you have a lot of experience with many different shells, you probably won’t notice the behind-the-scenes distinctions. What you will notice, however, is the incredible out-of-the-box user experience that zsh provides over bash.

Simply put, zsh makes it incredibly easy to execute commands and navigate directories. With built in features like global aliases (put aliases inside other expressions!) and filepath completion (press tab to toggle through each potential match), you’ll wonder how you ever tolerated using a terminal without them. You’ll cringe the next time you try to use your friend’s shell and it’s bash.

There are so many excellent features packed into zsh that I can’t possibly cover them all here. In fact, I still haven’t discovered everything zsh has to offer. The slides below cover the absolute essentials you should know when you upgrade from bash.

Why Zsh is Cooler than Your Shell from jaguardesignstudio

In the event that you skipped the slides above (for shame!), let me draw your attention to Oh My Zsh. In their own words:

Oh My Zsh is an open source, community-driven framework for managing your zsh configuration. That sounds boring. Let's try this again. Oh My Zsh is a way of life! Once installed, your terminal prompt will become the talk of the town or your money back!

Once you install Oh My Zsh, you’ll be able to use dozens of premade shell prompts (“themes”) and functions (“plugins”), all located in ~/.oh-my-zsh. Want a flashy shell befitting a true hacker? One of the 138 included themes in ~/.oh-my-zsh/themes is bound to suit your needs. Are you tired of navigating back and forth between the shell and browser to create a new GitHub repository? There’s a plugin for that. In fact, there are so many plugins that somebody made a list. Swapping themes or plugins is as easy as changing a line in your .zshrc file.

Oh My Zsh is also easy to extend yourself; just put your own scripts, themes, and plugins in ~/.oh-my-zsh/custom. Oh My Zsh will automatically source the scripts, though you’ll still have to change your .zshrc to specify the use of custom themes and plugins. Note that custom plugins and themes go in ~/.oh-my-zsh/custom/plugins and ~/.oh-my-zsh/custom/themes, respectively.

Now let’s see some custom enhancements. While everything that follows is written from the perspective of a zsh user, the lessons (and even most of the code) can be adapted for bash.

The Pearl Within

The first thing most people notice about zsh is its support for better user prompts than what bash can provide. Whereas bash allows prompt customization via the PS1 variable, zsh provides two variables: PROMPT and RPROMPT. As you might guess, PROMPT is the replacement for PS1, while RPROMPT is a second, right-justified prompt. That’s right, now you can have your prompt display information on both sides of the window!

An awful lot has been already been written about shell prompt customization. For example, Steve Losh has a wonderful blog post detailing how to create a prompt complete with a battery indicator. My own prompt is adapted from Losh’s example. It looks like this:

cstedman@nebulose ➤  LVL 1 -------------------------------------------------------- Tue Apr 07 2015 ➤  17:00
~ ± 蛇 ✓ █                                                                                          ▸▸▸▸▸▸▹▹▹▹

Here is the corresponding .zsh-theme file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38 # Variables for precmd
LONG_HOST=$(hostname)
SHORT_HOST=${LONG_HOST[(ws:.:)1]}
HOST_LEN=${#SHORT_HOST}
LSCOLORS=hxfxcxdxHxegedabagacad
BAT_CHARGE='/Users/cstedman/Code/Shell/config/batcharge.py'

# Precommand which runs before redrawing the prompt
function precmd() {
  RPROMPTGOAL=${(z)$($BAT_CHARGE 2>/dev/null)}
  RPROMPT="%{$RPROMPTGOAL[1]%}$RPROMPTGOAL[2]%{$RPROMPTGOAL[3]%}"
  LINENUM=""
  for ((i = 0; i < $COLUMNS - 45 - $HOST_LEN; i++)); do LINENUM="${LINENUM}-"; done
  if test -n "$VIRTUAL_ENV" ; then
    PROMPT=$_OLD_VIRTUAL_PS1
  fi
}

# Indicate that the CWD is a git or mercurial repository
function vcsCheck {
    git branch >/dev/null 2>/dev/null && echo '± ' && return
    hg root >/dev/null 2>/dev/null && echo '☿ ' && return
}

# Indicate that the CWD contains a python virtualenv (white snake symbol),
# or that a virtualenv is active (yellow snake symbol)
function virtualenvCheck {
  if [[ -a bin/activate ]] || test -n "$VIRTUAL_ENV" ; then
    if test -n "$VIRTUAL_ENV" ; then
      echo "$fg_bold[yellow]蛇 $reset_color"
    else
      echo "$fg[white]蛇 $reset_color"
    fi
  fi
}

PROMPT='%(!.%{$fg_bold[red]%}.%{$fg_bold[white]%})%n%{$reset_color%}@%{$fg_bold[white]%}%m%{$reset_color%} ➤  %(5L.%{$fg[red]%}.%(3L.%{$fg[yellow]%}.%{$fg[white]%}))LVL $SHLVL%{$reset_color%} $LINENUM %{$fg[white]%}%D{%a %b %d %Y}%{$reset_color%} ➤  %(2T.%{$fg_bold[red]%}.%(1T.%{$fg_bold[red]%}.%{$fg[white]%}))%T%{$reset_color%}
%(2j.%{$fg_bold[red]%}%j .%(1j.%{$fg_bold[yellow]%}%j .))%{$reset_color%}%~ %{$fg_bold[red]%}$(vcsCheck)%{$reset_color%}$(virtualenvCheck)%(?.%{$fg_bold[green]%}✓.%{$fg_bold[red]%}✗)%{$reset_color%} '

This file looks pretty ugly, and unfortunately that tends to be true of most shell scripts. In particular, the syntax for PROMPT and RPROMPT is completely inscrutable to human beings. To make matters worse, search engines generally will not return relevent results if you make queries like “zsh %{“. Here are two cheatsheets to help you decipher the prompt. Again, I adapted my prompt from Losh’s example, so I recommend reading his post if you’re confused by anything you see here. I’m only going to focus on the precmd and virtualenvCheck functions.

precmd is a special function in zsh.² If defined, zsh will execute the function immediately before it draws the shell prompt. I use it to update the battery indicator and scale my prompt to different pane or terminal widths. I use RPROMPT for the battery indicator only; everything else is part of PROMPT. Therefore, in order to get the date and time to appear right-aligned in the pane, I change the length of the line of dashes with precmd. To get the target length for the line, I use the COLUMNS environment variable to get the width of the pane, subtract the number of characters that never change (such as the date and time), and then subtract the length of the hostname (which can change).

Notice that I do not define RPROMPT in the same way that Losh does. For some reason, his method did not work for me, so I had to modify his Python script to return the indicator and color separately. Here’s the new code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42 #!/usr/bin/env python
# coding=UTF-8

import math, subprocess

p = subprocess.Popen(["ioreg", "-rc", "AppleSmartBattery"], stdout=subprocess.PIPE)
output = p.communicate()[0]

o_max = [l for l in output.splitlines() if 'MaxCapacity' in l][0]
o_cur = [l for l in output.splitlines() if 'CurrentCapacity' in l][0]
o_chr = [l for l in output.splitlines() if 'IsCharging' in l][0]

b_max = float(o_max.rpartition('=')[-1].strip())
b_cur = float(o_cur.rpartition('=')[-1].strip())
b_chr = o_chr.rpartition('=')[-1].strip()

charge = b_cur / b_max
charge_threshold = int(math.ceil(10 * charge))

# Output

total_slots, slots = 10, []
filled = int(math.ceil(charge_threshold * (total_slots / 10.0))) * u'▸'
empty = (total_slots - len(filled)) * u'▹'

out = (filled + empty).encode('utf-8')
import sys

color_green = "$fg_bold[green]"
color_yellow = "$fg_bold[yellow]"
color_red = "$fg_bold[red]"
color_cyan = "$fg_bold[cyan]"
color_reset = "$reset_color"
color_out = (
        color_cyan if b_chr == 'Yes'
            else color_green if len(filled) > 6
                else color_yellow if len(filled) > 4
                    else color_red
                    )

out = color_out + ' ' + out + ' ' + color_reset
sys.stdout.write(out)

Next, let’s look at virtualenvCheck. This function checks for two different conditions. First, if a Python virtualenv is currently active, it represents this fact with 蛇 (the Chinese character for “snake”) in yellow. Second, if the CWD contains an inactive virtualenv, the 蛇 character is white instead of yellow. The function itself is very easy to understand; it just checks for the presence of environment variables created by virtualenv. precmd also checks for one of these environment variables to prevent virtualenv from modifying the prompt. By default, an active virtualenv will prepend the name of the directory in which it is contained. I decided to disable that behavior because it ruined the spacing of the prompt, and I am able to infer the location of the virtualenv from context.

That’s pretty much everything to say about my zsh theme. The main lesson is to use shell functions to make the prompt dynamic and expressive. If you’re dead-set on bash, the same lesson applies, but unfortunately you miss out on RPROMPT. Next up, we’ll look at how we can use functions to simplify common tasks and “overload” system binaries to change their behavior.

What’s Your Function?

As I mentioned before, Oh My Zsh provides dozens of helpful plugins for Django, Ruby, Node, Sublime, and many more tools. If you’ve already loaded some plugins in your .zshrc file, you can run functions | less to see all available functions (but note that functions starting with underscores are not meant to be called by users directly).

Sometimes, however, you just won’t be able to find the right tool for the job. No worries! As programmers, we’ll make our own solution. Just write a function in ~/.oh-my-zsh/custom/ARBITRARY-NAME.zsh, and zsh will pick it up automatically.³ Type the name of the function into the shell (no parentheses) and PRESTO! Let’s look at two examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24 #
# Open the .gitignore in the current repository, or prompt the user to create
# one if it does not already exist
#
function gitignore() {
  setopt local_options no_case_match
  local GIT_CHECK=$(git rev-parse --show-toplevel 2> /dev/null) 
  if ! [[ -n $GIT_CHECK ]]; then
    echo "${fg_bold[red]}Error:${reset_color} Directory is not a git repository"
    false
  else
    if [[ -a $GIT_CHECK/.gitignore ]]; then
      $EDITOR $GIT_CHECK/.gitignore
    else
      local tmp
      vared -p 'No .gitignore found. Create one? (y/n) ' tmp
      if [[ $tmp =~ "^(y|yes)$" ]]; then
        echo ".DS_Store\n*.swp" >> $GIT_CHECK/.gitignore
      else
        false
      fi
    fi
  fi
}

Every time I start a new project, one of the first things I do is create a .gitignore file to prevent myself from checking in project dependencies, swap files, and application credentials. I occasionally need to make changes to this file, which entails navigating to the root directory of the application and typing vim .gitignore. Not so bad, right? Maybe not, but we can do better. Currently, we can’t tab-complete the file until we’ve typed vim .giti, because otherwise zsh doesn’t know if we mean .gitignore or the .git directory. Fix the completion issue, add in the ability to open or create the file from anywhere in the project directory, and we have a perfect use-case for a zsh function.

If you understood the functions in the .zsh-theme file, this new example should be relatively easy to follow. A few notes:

setopt local_options no_case_match makes the regex match on line 17 case-insensitive.
I use local variables using the local keyword in order to avoid polluting the environment unnecessarily.
The command git rev-parse --show-toplevel returns the root of the git repository, assuming we are within one. Otherwise it throws an error that we pipe to /dev/null.
vared prints the string following the -p option, waits for user input, and stores the value in the variable given.
I assume that the user has specified the path to the binary of their favorite editor in the EDITOR variable. We want to avoid hard-coding, of course!

While writing this post, I found out that Oh My Zsh already has its own gitignore plugin. From what I can tell, it uses gitignore.io to generate language-specific .gitignore files. I haven’t used it, and I’ll probably just stick to my own solution, but it may be worth a look before you use my code.

Our first example demonstrates how to use zsh functions to simplify basic actions and save time. The function we created had a name that didn’t conflict with any other executables in our PATH. Avoiding name conflicts is important because zsh actually tries to resolve command names with functions first, before searching the PATH. So what happens if we make a function called gcc? If done carelessly, the result will be the disruption of C program compilation. However, the ability to override executables also gives us a way to make transparent wrapper functions that augment default behavior. Let’s see one more example:

1
2
3
4
5
6
7
8
9
10 #
# Open multiple files in vim with vertical split
# 
function vim() {
  if [[ $# -ge 2 ]]; then
    $EDITOR $@ -O
  else
    $EDITOR $@
  fi
}

This function is obviously very simple, but its implications are vast. Every time I type the word vim in my shell, this function is called instead of the real vim binary. It looks at the number of arguments provided to the function and decides how to invoke the binary depending on the result. If one argument is found, the function calls vim normally. Otherwise, it instructs vim to open the files in parallel vertical windows. Normally, vim handles multiple file arguments by opening one buffer at a time, starting with the first argument, and opening each subsequent buffer when the previous one is closed. I dislike the default behavior because I either provided the second filename by mistake (and I’d like to solve the problem immediately) or I actually did want to see both files at the same time (in which case I’d prefer the parallel window option to be the default).

Again, notice that I’m using the EDITOR variable as a stand-in for the full filepath to the vim binary. If you decide to override other executables using this pattern, you’ll need to hardcode the full paths or create your own environment variables. You cannot just use the original command name, as zsh will again perform a lookup in the list of functions first, causing an infinite loop.

While the code examples above are all zsh scripts, similar programs may be written for bash. However, while Oh My Zsh provides a safe place to put custom scripts in ~/.oh-my-zsh/custom, bash users will have to make their own local scripts directory and prepend it to their PATH.

From A to Zsh

If you’re just beginning to learn the ways of shell fu, I hope I’ve convinced you to at least give zsh a try. Whether you’re building complex themes or autocompleting git commands, zsh is a blast to use. Personally, I’m most excited about executable overloading, and I plan on doing much more of it. Have any ideas for executables to extend with scripting? Think you can give a better example? Have the best prompt in town? I’d love to see what you come up with. Happy hacking!

Though, really, all shell scripting languages are pretty quirky. ↩
Bash provides an alternative called PROMPT_COMMAND. ↩
If you plan on making something a little more robust, or you have multiple functions, consider creating a plugin. Put your work in ~/.oh-my-zsh/custom/plugins/YOUR-PLUGIN/YOUR-PLUGIN.plugin.zsh, and make sure to include the plugin in your .zshrc: plugins=(..... YOUR-PLUGIN ....). ↩

A Shark on the Network

Fri, 28 Nov 2014 00:00:00 +0000

DISCLAIMER: I do not condone monitoring network traffic with intent to spy or otherwise cause harm to people. My interest in the techniques described below is academic. I write in the hope that I can improve Internet security, or at least increase public awareness of vulnerabilities. If you are attempting to replicate my work, please be careful to do it in a controlled setting, such as your own home. Despite court rulings like this one, the legality of packet sniffing on public networks is very murky, and you may get in a lot of trouble.

I’ve spent a lot of time this year thinking about networking, the web, and security on the Internet. Since the Snowden leaks, revelations about the scale and sophistication of government cyberweapons have the public talking about the danger of metadatacollection. In response, I began to wonder how easy it would be for any would-be adversary to perform a practical collection attack on a standard WiFi network. In order to judge the difficulty for myself, I decided it was time to dabble in the Dark Arts.

Starting to Sniff

Enter Wireshark, a popular “packet sniffer”. Wireshark records, or “captures”, the various messages sent and received by your computer’s network card. If you’ve taken a class on computer networking, you’ve probably already downloaded the program; if not, you should get it now (for free) if you want to follow along.

Normally, Wireshark only captures the packets whose source or destination are your own computer. However, it can also be used to configure your network card to enter “promiscuous mode”, allowing you to capture all network traffic indiscriminately. The metaphorical equivalent is a mailbox that, instead of receiving letters addressed to a single house, receives a copy of every letter addressed to any house on the street. If you’re wondering why the network card has access to all messages on the network, consider that you need to see every message in order to determine which ones you are supposed to receive.

Entering promiscuous mode turns out to be remarkably easy; just go to ‘Capture Options’, double-click on the interface you want to capture on, and select the options to capture packets in promiscuous mode as well as monitor mode.¹ I find it odd that entering promiscuous mode alone is not enough to perform the capture. Perhaps one of you can tell me why it works that way.²

Directions for capturing on an interface in promiscuous mode.

Your computer should now capture all packets observed at the interface, instead of only the packets going to or from your own machine. To confirm that you’re capturing all traffic on the network, type http.host == xkcd.com into Wireshark’s filter bar and go to xkcd.com from another device. If you capture the packets, you’re in business.

The traffic destined for xkcd.com should look something like this.

You should take some time to play around with Wireshark and understand exactly how much power you now have. Just by clicking two checkboxes, you can observe every message sent to and from your local router; you can collect the IP and MAC addresses of all nearby hosts; you can map every request for a web page back to its source, know what was requested, and read all unencrypted data. All you did to access this information was instruct your machine not to throw away messages destined for other clients. You didn’t have to tamper with network hardware or other machines, you didn’t hurt network performance, and nobody can detect what you’re doing.

The implications should concern you.

Look Ma, No GUI

Of course, Wireshark isn’t the easiest tool to use when it comes to actually analyzing the data you collect. You’ll still have to inspect packets manually in order to extract the information you’re looking for. What if you just want a list of all MAC address pairs which exchanged messages? Wouldn’t it be nice if you could programmatically collect, filter, and print only the packets or header fields you’re interested in?

Assuming you downloaded Wireshark, You already have just the tool for the job, called Tshark. Tshark provides CLI (command line interface) support for most of the actions you can do in Wireshark, making it possible to integrate network sniffing into your programs. The only catch is that, depending on how you installed Wireshark, You may have to find the Tshark binary and add its location to your PATH in order to use it.³

Let me demonstrate just how powerful Tshark can be. A few weeks ago at HackPrinceton, I presented a tool I called HostShark. HostShark sniffs the network for outgoing web traffic, both HTTP and HTTPS, and visualizes the hostnames of the traffic destinations using D3. I used Tshark to capture packets, extract the necessary header fields, and pipe the data to a Python program for further processing. Incredibly, that entire series of operations was performed in a single line of bash:

$ tshark -lIY "tcp.port==443 or http.request" -T fields -E separator="|" -e ip.dst -e http.host 2> /dev/null | python process.py

I won’t explain what each of these command line options is for (tshark help is your friend), but I’ll give you the gist. This command promiscuously captures outgoing HTTP and HTTPS packets. HTTP traffic may be filtered directly via http.request, but HTTPS traffic is a little bit more complicated because Tshark and Wireshark don’t provide a built-in filter. However, we can create a de facto filter with tcp.port==443, taking advantage of the convention for HTTPS traffic to use port 443. We now extract the destination IP and HTTP hostname from the packet, separating the values with a vertical line (‘|’). For example, a request to xkcd.com might result in the output “107.6.106.82|xkcd.com”. Note that HTTPS traffic will not have an HTTP hostname header, so the returned value will be the empty string. The extracted data is finally piped to the Python program below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35 import json
from socketIO_client import SocketIO
import sys
import time
import urllib2

ips = {}

with SocketIO('localhost', 7717) as socketIO:
    try:
        buff = ''
        while True:
            buff += sys.stdin.read(1)
            if buff.endswith('\n'):
                parts = buff[:-1].split('|')
                if (parts[1] is ''):
                    if parts[0] not in ips:
                        # make the API call here
                        data = json.load(urllib2.urlopen('http://stat.ripe.net/data/reverse-dns-ip/data.json?resource=' + parts[0]))
                        if data['data']['result'] is not None:
                            ips[parts[0]] = data['data']['result'][0]
                            socketIO.emit('host', { 'host': data['data']['result'][0] });
                        else:
                            ips[parts[0]] = None
                    else:
                        lookup = ips[parts[0]]
                        if lookup is not None:
                            socketIO.emit('host', { 'host': lookup });
                else:
                    if parts[1] != "stat.ripe.net":
                        socketIO.emit('host', { 'host': parts[1] });
                buff = ''
    except KeyboardInterrupt:
       sys.stdout.flush()
       pass

Again, I won’t go into detail about how this program works (partly because it doesn’t matter, and partly because I’m not a Pythonista and don’t want to teach bad practices). It takes each incoming line, splits it at the ‘|’ and checks if the hostname is non-empty. If so, the hostname is forwarded to a server via Socket.io. Otherwise, the hostname must first be determined through a reverse DNS lookup on the IP address. The ips dictionary stores the results of the lookups to prevent duplicate requests to the lookup API.

Hostbusters

Now that I’ve revealed the gooey center of HostShark, let’s see it in action. I’m not going to bother with explaining the frontend or backend code here, since that isn’t the point of this blog post. You are, of course, encouraged to check out the rest of the code on my GitHub, and pull requests are welcome.

A few notes on how HostShark works. The visualizer refreshes every fifteen seconds and shows the hostnames of all requests made in the past minute. The hostnames are displayed heirarchically as a series of concentric arcs. The innermost arcs represent top-level domains (TLDs). Every subsequent arc is a subdomain of the one within.

Here’s a screencast of HostShark in action:

An example of a real capture at HackPrinceton.

You’re Gonna Need a Bigger Boat

When you eavesdrop on connections without interfering with them, you are performing what is known as a “passive” attack. HostShark is just one example of a passive attack. Other possible passive attacks include mapping all communicating pairs of hosts or simply analyzing the contents of HTTP packets.

Passive attacks take advantage of three fundamental limitations of modern networking:

Network protocols leak metadata. HTTPS and SSL encryption may protect your data from prying eyes, but the encrypted payload must still be wrapped in TCP, IP, and ethernet headers in order to be transported to its destination. Those headers expose IP addresses, MAC addresses, and other transport details. The headers themselves cannot be encrypted because routers and switches must be able to read them in order to forward the traffic; this is an inherent limitation of the design of the Internet.
The destination of your traffic may reveal your intent. This is really a corollary to #1. Given that the source and destination of a message is public knowledge, the corresponding payload is only truly private if a third party cannot guess the contents with greater than negligable probabilty. In the case of requests made to well-known services on the Internet, it might be possible to make quite educated guesses about what is being exchanged. For example, if I know you’re sending messages to united.com, I can probably guess that you’re booking a trip. If you’re visiting WikiLeaks or freedom.press, I know you’re interested in political activism. The danger increases as websites use subdomains to divide services. If you watch the screencast above, you will see that Facebook uses different subdomains for their various frontend proxies. By analyzing the traffic, I might be able to determine that edge-star-shv-04-atl1.facebook.com is used to serve requests for images, while channel-proxy-shv-04-frc3.facebook.com handles comments or chat. Even if I can’t read the traffic, I have fine-grained knowledge of what you are doing.
Passive attacks are undetectable. As I said before, there is no change to any machine except your own, and nobody else’s network service is negatively impacted by your actions. As long as nobody is looking over your shoulder, you are completely invisible.

Unfortunately, until a solution to these limitations is devised (and widely deployed!), there is no way to prevent passive attacks from occurring. You can only hope to mitigate the damage by exercising discretion on public Internet. For now, the sharks roam free.

Next Steps

In this post, I describe a family of attacks based on passive observation of the traffic on the network. Passive attacks are not the end of the story, however. If an attacker is able to insert herself along the “path” between two hosts, becoming one of the computers to receive and forward the packets, she can perform an “active” attack by modifying packets mid-transit. Such an attacker is known as a man-in-the-middle (MitM). MitM are extremely dangerous because they may cause an end host to reveal information which would have otherwise remained secret. For instance, a MitM could change a webpage to transmit login credentials insecurely. The only disadvantage of an active attack over a passive attack is the possibility of being detected. Fortunately, Princeton detects (some) active attacks.

I don’t have any plans to write about active attacks right now. I’m reluctant to teach such a powerful and harmful technique. That being said, the knowledge is out there for those who are willing to do a little digging.

Note that your computer may not come with a network card capable of promiscuous network monitoring. I have no problems on a 2013 Macbook, and I’d expect even older Macs to work as well. With PCs, it depends on the hardware vendor. ↩
Since It is possible that the behavior is dependent on the hardware or operating system. ↩
On Mac, I found the Tshark binary at /Applications/Wireshark.app/Contents/Resources/bin/tshark. I added it to my PATH by making a softlink from my /usr/local/bin directory to the location of the binary. ↩