Tuesday, December 12, 2017

Listening to Wi-Fi probe requests

Most Wi-Fi capable devices broadcast probe requests to figure out what networks are available in the area. When an access point receives a probe request it will answer with information about the network it is running. The probe request contains the device's MAC address, which is usually static, so by listening to the probe requests you can determine if a specific device is in the vicinity of your Wi-Fi receiver. How often probe requests are sent is device specific but most devices send them every now and then, even when they are already connected to a network.

To listen to the probe requests perform the following steps. Note that not all Wi-Fi cards allow you to see the probe requests. If you are not seeing any probe requests it's probably because your Wi-Fi card hides them. I'm using a "D-Link DWA-111 Wireless G USB Adapter" on Windows.
  1. Install python 3.6.3 (https://www.python.org/downloads/)
  2. Download my script from here
  3. Install Npcap 0.94 (https://nmap.org/npcap/). Make sure you click "Support raw 802.11 traffic (and monitor mode) for wireless adapters" during installation. I had to re-boot my computer after the installation to get everything to work well.
  4. Install Wireshark 2.4.2 (https://www.wireshark.org/#download)
  5. Set the Wi-Fi card to monitoring mode by running the following command as administrator. "Wi-Fi 2" is the interface name and should be changed to whatever interface you wish to use for the sniffing.

    C:\Windows\System32\Npcap\WlanHelper.exe "Wi-Fi 2" mode monitor
  6. Run my script. You may have to change the interface name and tshark path.

    python wifi_monitor.py "Wi-Fi 2" "C:\Program Files\Wireshark\tshark.exe"

    If everything works you should now see the MAC addresses of devices doing probe requests in your vicinity, something like this:

    Date, Time, Number of previous sightings, Is local address?, MAC address
    Local addresses are usually not static over time!
    2017-11-06 21:13   6312   f4:f5:d8:ff:e0:b9
    2017-11-06 21:17      0 X 72:a3:98:af:12:c3
    2017-11-06 21:17    200  
    70:70:0d:ce:21:77
    2017-11-06 21:20    201  
    70:70:0d:ce:21:77
The MAC addresses are a bit hard to remember so you can create aliases for them by adding a line in aliases.txt.

70:70:0d:ce:21:77 Suspicious device

70:70:0d:ce:21:77 will now show up as Suspicious device in the script output. Note that it may take a while before the script re-loads aliases.txt and the change takes effect.

The script saves all the probe requests in log.txt. I have implemented a few scripts that extract statistics from the log file. histogram_of_sighting_times.py creates a histogram of the sighting times for a device.

python histogram_of_sighting_times.py 1 4 12 "Suspicious device"


The script uses Matplotlib so you need to install that before using it (https://matplotlib.org/users/installing.html).

device_graph.py creates a device graph based on how well device sightings correlate temporally. If two devices always show up at the same time they will have a very strong connection in the graph, if two devices have never appeared close to each other in time they will have no connection at all. The created graph (device_graph.gdf) can be analyzed in Gephi (https://gephi.org/) or any similar program. For more information about Gephi check out this post.

Thursday, December 7, 2017

Roboreader

Edited 20180121

Staying aware of what's going on around you in a large software project is a bit of a chore. Meetings, reading various information sources like mail lists and wikis, maintaining queries and watches in support systems like gerrit, and so on. It would be nice if this could be automated, at least to some extent, so I gave it a try.

Roboreader continuously fetches new texts from various sources, reads them, and shows you only the texts that you are interested in. Currently it supports two sources, imap (email) and gerrit, but it's easy to add more sources. To figure out which texts that are of interest to you there are four match engines.

Pattern search
This match engine just applies a number of regular expressions you have created to the texts. If a text matches any of the regular expressions it is considered interesting.

Word frequency
This match engine figures out your interests by looking at which words you overuse. It looks at all the mails, patches, bug reports, etc that you have written and compares them to all the mails, patches, bug reports, etc that other people have written. If you overuse for example "anti-banding" it will assume you are interested in that topic and all texts containing "anti-banding" will be marked as interesting to you.

ANN
This match engine uses an ANN (artificial neural network) classifier to classify texts as "interesting" or "not interesting". The training data for the "interesting" class are the texts you have authored and the training data for the "not interesting" class are the texts that other people have authored. During training the ANN will learn patterns that are unique to you and use them to figure out which texts you might be interested in. Much like the "word frequency" match engine, but patterns instead of words, which makes this match engine a lot more flexible.

Carrot2
This match engine uses the open-source text clustering engine Carrot2. It clusters all new texts and all the texts you have authored. If a new text ends up in the same cluster as one of the texts you have authored it is considered interesting.

I am currently not working with large scale software development so I couldn't test how well it all worked myself, instead I set it loose on AOSP's (Android) gerrit and told it I was "Jakub Pawlowski", a software engineer at Google working with bluetooth. This approach is a bit flawed. All the texts are patches, which is unlikely in a real scenario. It is also hard for me to assess whether the found texts are of interest to Jakub or not as I'm not Jakub. Better than nothing though.

The results where a bit so-so. Pattern search works as expected but since you have to write the regular expressions yourself it doesn't add much value. Word frequency and ANN find interesting texts without too much noise, but I've got a feeling they miss a lot of interesting texts. I was unable to get anything useful out of Carrot2, way too much noise or no matches at all, could be a configuration issue though.

If you want to try it out yourself you can download the script here. To get going just edit config.xml and run the script. Current directory is used when looking for or storing files (like configuration and state).

python3 roboreader.py