Misc - 2

I am going to attend the Chaos Communication Congress 2008 in two weeks. People who want to meet me there to talk about BinNavi, Hexer, static code analysis, reverse engineering in general, or why the movie Hackers is more realistic than most people think, please contact me using one of the options you can find on the navigation bar on the right side of this page.

Here's something amusing. I spent the first half of the day writing a short Haskell program which generates x86 instructions in MASM syntax. The program generates all variants of the non-privileged instructions from the opcodes.chm file of the MASM32 package. This means that the instruction generator is not complete at all. FPU, MMX, SSE and other newer-than-x486 instructions are not covered. Nevertheless the generator already generates nearly 150,000 different x86 instructions.

When assembled with MASM32 the resulting file is more than 600 KB big. Trying to disassemble this thing with a few standard disassemblers turns out to be a problem. IDA fails to disassemble an instruction after maybe 5% of the executable and never manages to recover afterwards. Lots of manual help is necessary to convince IDA to go on. OllyDBG manages to disassemble that instruction but has huge gaps at many, many other points of the disassembly. The created file is an interesting test file for x86 disassemblers I'd say.

The Haskell program is just about 300 lines long. 280 of those lines are the definitions of  the instructions and what operands they can take. The generation of the instructions from the instruction definitions is just 20 lines and all but 8 lines are not even strictly necessary. I love Haskell's expressiveness.

Anyway, click here to see the Haskell source or click here to download the whole package including the Haskell program (source + EXE), the generated output of the Haskell program, a MASM32 source file that can be used to assemble the test file, and the test file EXE itself.

Next week I'm going to attend this year's hack.lu conference in Luxembourg. So if anybody else who attends hack.lu wants to talk about the new version of BinNavi, Hexer, static code analysis, reverse engineering in general, or some kind of other topic contact me please (see the right side bar for contact options).

Yesterday I saw a talk given by Frank Boldewin where he mentioned the FreeIconList trick to fool code emulators. At this point I started to wonder what other Win32 API functions are basically unused. Using Ero Carrera's Python library pefile to parse PE files I wrote a small Python script that tries to find out what Win32 API are basically unused.

The modus operandi was simple. I read the exported functions of all DLL files in WindowsDir and WindowsDir/system32 and compared them to the functions imported by all EXE/DLL files in WindowsDir, WindowsDir/system32 and my entire Program Files directory.

The first result is that most exported functions are apparently basically never used. My script managed to find 127569 exported functions in 1225 DLL files. 104608 of those are never used by the 6615 EXE/DLL files which import functions ("used" is liberally defined as "imported through the import directory" here, of course). That leaves 22961 functions which are actually used.

Here are some output files which show the exported DLL functions sorted by their usage. The numeric column contains the number of PE files which import the function statically. That means that 3475 of the 6615 files I tested import GetLastError for example.

• Click here to see the Top 2000 most used API functions
• Click here to see the usage statistics of all advapi32.dll functions
• Click here to see the usage statistics of all gdi32.dll functions
• Click here to see the usage statistics of all kernel32.dll functions
• Click here to see the usage statistics of all msvcrt.dll functions
• Click here to see the usage statistics of all ole32.dll functions
• Click here to see the usage statistics of all oleaut32.dll functions
• Click here to see the usage statistics of all shell32.dll functions
• Click here to see the usage statistics of all user32.dll functions

Random notes

• kernel32.dll is surprisingly dominant while gdi32.dll is surprisingly "unused"
• pefile is extremely awesome and easy to use
• Don't be confused that API functions like lstrlen are imported 0 times, check lstrlenA and lstrlenW

Click here to download the Python script.

Unless something goes spectacularly wrong I'll be in Montreal from next Thursday to Sunday to attend RECON 2008. If you wanna do one or more of the following things, make sure to meet me at some point:

• Talk/Complain about BinNavi
• Talk/Complain about BinDiff
• Talk/Complain about VXClass
• Talk/Complain about Hexer
• Talk/Complain about Zynamics
• Talk/Complain about any other topic
• Get an awesome VXClass t-shirt
• Have some poutine with me

I'll be the guy with the yellow Sabre Security bag. Alternatively you can shoot me an email or ask around.

Two days ago I had the pleasant experience to participate in some kind of informal reverse engineering session with three other guys. Between dinner and way too long after midnight we debugged a popular piece of malware that is floating around the internet right now. The first guy already reverse-engineered an earlier version of the malware. He was the guy in charge who did most of the debugging. The second guy was the author of a program that monitors and logs the behaviour of processes, especially malware processes. The goal of the session was to find out why the malware sample worked perfectly in VMWare (after we patched out the VMWare check, at least) but crashed as soon as second guy's monitoring tool was active. The third guy was very familiar with the malware too but on a higher level (behaviour, network activity, how it spreads, its historic development and usage, ...). I was the fourth guy. Without a direct interest in the malware or the malware monitoring tool I just wanted to see what goes wrong. Furthermore I was the guy for snarky comments from the background like "see, I told you take a VMWare snapshot before stepping over that call".


Continue reading "Pair Reverse Engineering"

Over the last days some group released passwords to nearly 45000 MySpace accounts and they announced to release another 30000 passwords in the next few days. I used a few hours before Saturday's lunch to write a small program that analyzes the passwords that were released so far.

At worst the results of this are a useless time-filler, at best it's a case study of what happens if a website forces their users to choose passwords with certain minimum requirements. MySpace demands that every password contains at least one non-alphabetical character (like 0, 1, 2, or !, ?, @). How the users adhered to this requirement can be seen in the tables below.

It is my understanding that the 43713 passwords that were leaked so far come from fishing sites that trapped people to enter their password. This makes the passwords less reliable than a password list hacked straight from the MySpace servers. People could have misspelled their MySpace passwords or they could have entered fake information after they noticed that someone was trying to steal their password. A quick analysis has shown that probably less than 1% of the leaked passwords suffer from these problems. Continue reading "A brief analysis of 40,000 leaked MySpace passwords"

If you have ever wondered about things like "gee, why does that sp guy never update his site anymore" I have some good news. I was busy being a grad student. At least until 27 minutes ago when I finished the presentation of my thesis and answered the last pesky question I was asked about it by a professor.

Being a grad student turned out to be way more demanding than being an undergrad. Less time to slack off led to less site updates. A shocking concept, I know.

Anyway, unless some kind of nightmare happens like

Administration Guy: Hey sp, what was your second elective?
sp: Erm, a second elective?

I will soon receive a nifty little Master of Science (in Computer Science) diploma. It will be my second most important formal proof of qualification, topped only by my beloved Windows 95 Power User certificate I received from Brainbench like a decade ago (it looks like this but I don't have a pic of my own one here).

Well, that gave me an excuse to write a site update. Time to slack off a bit now. Between moving out of my student appartment, doing a lot of administrative stuff at college, visiting a bunch of people (potentially) for the last time, and finding a job, I probably won't make another site update in the next 2-3 weeks. But after I'm done with all of that, my goal is to write more site updates again.

Here are some more details about the program I used to create some graphs from Wikipedia two days ago. The C# source code is now available. The program takes five command-line parameters.

• The name of the Wikipedia XML input file. You can download it here. It's the 1.8 GB pages-articles file.
• A number that specifies how many nodes you want in the final graph. 25 - 50 are reasonable values.
• The keyword to search for (like Egypt for example). Note that this parameter is case-sensitive.
• A number that specifies how many sentences are considered when searching articles for the keyword.
• The name of the output file. This file is a graph definition file that can be turned into a graph using dot.exe from the GraphViz package (like dot.exe -Tsvg output.txt > graph.svg for example).

I toyed around with the parameters for a while and here are a few more things I noticed:

• Searching through the first three sentences of all articles seems to produce very nice results for most keywords.
• If the keyword is relatively rare (for example "Australian rules football") it's OK to search through the entire article (set the sentences parameter to 0). Don't do this for popular keywords though or you'll end up with a graph that shows articles that are objectively important but only tangentially related to the keyword. If you do a full-article search for "Germany" for example you end up with a graph full of nodes containing the names of other European countries that played a role in Germany's history. That's because articles about countries have a high importance and all European countries were somehow important to Germany in the last few thousand years.
• Trying to use the size of articles as an indicator of their relative importance didn't work out. Look at this 4,000 words treatise on the Goomba and compare it to the page for Niels Bohr which is only half as long. This should give you a first idea about the potential problems. There's still legacy code in the app from where I tried that idea. It's easy to enable again it but you need to recompile the app.
• Trying to use the position of the keyword compared to the size of the article didn't work either. Long articles still have too much weight. A better idea might be to give an article weight 10 if the keyword appeared in the first 10% of the article, weight 9 if it appeared in the next 10%, and so on. I didn't try that though.
• Different parameters lead to different meanings of the resulting graphs. If you do a 3-sentences search for LSD the graph shows information about the drug itself and its history. If you do a full-text search one half of the graph is dominated by rock stars.

Here are a few other graphs which turned out particularly well:

• Chile: PNG - SVG
• Nintendo: PNG - SVG
• Ferrari: PNG - SVG
• Star Trek: PNG - SVG
• Intelligent Design: PNG - SVG
• Australian Rules Football: PNG - SVG

The key to cool graphs is to choose a keyword that has lots of articles which nevertheless belong closely together. An example for a bad keyword is "Mathematics". There are thousands of math-related articles in Wikipedia but they don't belong closely together because math is a huge and fragmented field. The resulting graphs of keywords like math degenerate into trees or unconnected subgraphs.

Generating a graph takes approximately 5 minutes on my computer. In most cases nearly all the time is spent on parsing the 8 GB XML file. Generating the actual graph is nearly always a matter of seconds. Only for keywords like Germany or America which have some ten-thousand relevant articles generating the graph takes a few more minutes.

Recently I finished Ian Shaw's book The Oxford History of Ancient Egypt. It was pretty interesting but it introduces more names per page than your average Tolkien book. I really could have used a chronologically ordered graph that shows names of important people and places and how they are connected. Creating a graph like that manually is obviously too much work so I tried to use the power of Wikipedia to create a graph automatically. My original plan didn't quite work out. Apparently the problem of data-mining from texts written in a natural language like the English language can't be solved in a few hours on a lazy saturday afternoon. At least not by me. Nevertheless I managed to get some interesting partial results.

Continue reading "Data-mining Wikipedia"

Here's a new version of my Delphi obfuscator Pythia. The main reason for the update was a necessary bugfix. Version 1.0 incorrectly obfuscated some files with forms that contained complex controls (that is controls like Panels which consist of several subcontrols). I've added two additional command-line options too. The first option prints information about the input file (Click to see an example), the second option prints information about the obfuscation process (Click to see an example).

Here's a short example of what Pythia does. Click here to see a screenshot of the information that can be gathered about a typical Delphi file. Click here to see the same file after obfuscation. Curiously enough the obfuscated file is actually a Borland C++ Builder file. Pythia can handle BCB++ files too because the underlying VCL model is the same in BCB++ and in Delphi.

Click here to download Pythia 1.1

It's the little surprises that make programming so interesting. In chapter 14 ( Pragmatic Paranoia ) of the Pragmatic Programmer the authors Andrew Hunt and David Thomas teach the reader that assuming things about code is a bad idea (one example they give is about minutes which have more or less than 60 seconds). You should double and triple check every assumption you make and then let the program check your assumption again with an assert statement.

Yesterday one of these assumptions cost me about 90 minutes. What do you think is the result of the expression 0xFFFFFFFF12345678 & 0xFFFFFFFF where & is the operator of the bit-wise AND operation? Continue reading "It's the little surprises ..."

My college career is slowly coming to an end. If everything is going according to the plan I am going to graduate with a Master of Science degree in Computer Science at the end of the next semester. During my time at college I worked as a TA for five classes and I will probably work as a TA for a sixth class next semester. Four of the five classes were Intro to C++ classes taught to 4th semester students. The other class was an Intro to Programming class taught to freshmen.

During the last few years I was in the role of the student and I was in the role of the teacher in programming classes. In that time I developed opinions about the curriculum and pretty much everything that went on in class. I already promised some of my teachers that I am going to write down everything once I hold my degree in my hands. Every complaint and every suggestion will be neatly categorized in a visually appealing LaTeX document. That's the plan at least.

One of the suggestions I'm planning to write down is the way introductory programming classes are taught and especially the programming language used in class. The style of introductory programming classes and the programming languages taught in college are certainly popular topics to complain about on the Internet. Joel Spolsky decries JavaSchools where graduates know neither pointers nor recursion. Dan Zambonini touches on the topic as part of his idea of a decent CS curriculum (the article started a Slashdot discussion with 654 comments too). On Digg someone raised a comparable question and received 154 answers. And once in a while you can hear surprising news like MIT getting rid of Scheme and replacing it with Python in undergraduate classes. Continue reading "Some thoughts on freshmen programming classes"

While I wrote my last blog entry I remembered another curiosity about localization which I like to mention once in a while in various exciting places on the Internet. Converting strings between upper case and lower case is difficult. It is in fact so difficult that I don't think software will ever get it perfectly right.

Here is the example I like to give. The German word for street is Straße. The ß character (called Eszett or sharp S) is a curious character. It only exists in lower case form. There is no corresponding upper case character for it. So how do we capitalize the word Straße? The grammatically correct capitalization of the word Straße is STRASSE. The lower case ß character is mapped to two upper case S characters.

Continue reading "Converting strings to upper case is tricky"

A few days ago Elliotte Rusty Harold blogged about The Downside of Localization. He gave the example of the Xerces XML parser and how error messages are now available in various languages. Mr Harold correctly recognized that this could introduce a new problem: Developers working with localized error messages are less likely to find a solution for their problem on Google. "Paste an English exception message into Google, and you’ll probably find 10 people who have already had and solved your problem. But the same message in Greek or Croatian? Maybe not."

I wrote a comment on his blog mentioning that the problem could easily be solved if only error messages contained a unique error code. That error code could then be used in Google queries. In a reply to my comment someone pointed out that ideas like that are rarely talked about. Now I'm not an expert on localization, I've localized exactly one commercial application in my life (a small web application based on JSP/Servlets I wrote during a summer internship three years ago). Nevertheless I like to read blogs and websites about localization efforts and therefore I know that proper use of Unicode characters is shockingly difficult, not all cultures read and write from left to right and therefore you better watch out how you design your GUI, and that not all countries agree on the exact position of national borders and people might boycott your product if the localized version of your software includes the wrong map. I think this qualifies me to write something about the topic on my site.

Continue reading "A plea for unique error codes"