Since Binary Ninja 4.2, users have been able to enable WARP, and now with Binary Ninja 5.1 we are finally enabling it by default! WARP has received many features users have been asking for, and a healthy amount of polish on top of that as well. In this blog we’ll describe what WARP is, hot it works, and how you can use it to improve your reverse engineering experience.
WARP Is This?
WARP (and similarly SigKit) was designed to identify known functions in unknown binaries, increasing the ability to reason about what a binary does (or doesn’t) do. An addition goal of WARP is to provide the groundwork for function similarity lookups (fuzzy matching) and a format that can be used to transfer analysis information passively.
Because of the statically linked nature of many modern binaries (especially Rust and Go) we often end up with quantities of library code far exceeding that of “user” code. It is the job of WARP to provide the facilities for you as a user to create signatures of these common functions to match and apply information automatically.
WARP works by first creating a GUID for each unique basic block in a given function, from which a “Function GUID” is derived. This GUID is then used to look up possible matching functions. The GUID itself is stable and should be identical for the same function in any given binary. This poses some challenges given that a function might be referencing data or code at a different address. To address this we zero out “variant” instructions before generating the basic block GUID. A more detailed description is available over on the WARP GitHub repository.
WARP works directly on the raw bytes of the file, and we use our BNIL representations to identify the instructions to mask off. This means that all architectures that are supported by Binary Ninja will work, including custom ones. Additionally, where other reverse engineering tools have similar capabilities, they can also generate compatible GUIDs, allowing cross-tool information sharing!
What Can I Expect?
When you open a new binary in 5.1, after function analysis finishes, we will then take all functions and their respective GUID and attempt to automatically match against relevant WARP data, applying matched information. As part of applying information we will tag the function. This makes it easy for you as a user to know when WARP information has been applied.
After the function is matched, the following information will be transferred (if available):
- function symbol
- function type
- calling convention
- comments and variables
Stats for Nerds
For those who like numbers (I do not) we compared a few different binaries and use cases from both the previous development version of WARP and the 5.1 release. This way we can prove we’ve made improvements with… science?
For each binary we simply open it up in Binary Ninja (either 5.0 or 5.1) and observe the number of tagged WARP functions. The below chart details a few binaries, while the information is self-evident, the data we are matching on is not, so I will detail that below:
- Binary A-D: The binaries are matching against our bundled MSVC signatures.
- Binary E: This binary is testing our ability to transfer functions from a separate version of the same application, specifically version A to version E.
- Binary F: This binary is testing our ability to match on RISC like architectures, this showcases improvements to our variant instruction detection, as we are generating signatures from one binary with symbols and applying them to another binary of the same microcontroller.
The keen-eyed among you will notice a small dip in matched functions for Binary B. This brings up an important point about accuracy: the reason we are matching fewer functions in that binary is because our dataset of functions was increased, causing some collisions on functions which previously had none. While the local result might seem worse, this is a good thing!
The functions were all named “Concurrency::X”. If you’ve reversed windows binaries, you likely already know that these are bogus names that often show up in some function matchers. It’s common when analyzing C++ binaries to see your tool mark up some small constructors with the aforementioned names. This can easily cause confusion and provide no value during reverse engineering.
The premise of WARP is that we do not need to hand hold the creation of signatures or prematurely prune problematic signatures, which is a big problem with SigKit and other similar tools. For example, previously with SigKit, Binary Ninja would consistently annotate operator new() in place of operator delete() (see the below images), and this was happening automatically with little feedback to the user that the source was SigKit.
Knowing this, we consider the dip in matched functions for Binary B to be an improvement. As we add more data to the bundled signatures, incorrect matches on signatures will continue to go down.
New UI and API
One of the major improvements of this release was the addition of an interactive UI, giving you control over what information is applied and access to additional information such as the list of possible functions for any given function.
This new UI is written in C++ as a separate bundled plugin which of course means we had to add an API to WARP. With 5.1 you can now write scripts to create, query and apply function information. For anyone wanting to know more, we include a section detailing the API and an example python script in the user documentation.
Improved Signature Creation
While the process for consuming WARP information has been streamlined with the UI, we still needed to improve the workflow for producing the WARP information. While processing headlessly is still supported with the API, headless is not required. Previously, SigKit required the headless API for creating signatures of library files.
To start with, we overhauled the processing of individual files. From the UI you can now process static libraries directly. This critical feature was not readily available to users with a personal license. Now you just open any view and run the command WARP - Create - From File(s) and pass in your static libraries (.a, .lib, .rlib). Upon completion, you’ll have a single .warp file with all the function signatures. This also works for binaries with multiple architectures contained in them like a universal Mach-O.
Creating and managing the set of files to make the signature file can be very tedious. We added support for creating a signature file from a project to help with this. For commercial users and above, you can now easily process multiple files at the same time all from the UI! This uses the same process as above, so static libraries will also work. You can additionally merge separate .warp files using this same command to make distribution easier and to save on storage.
For those working with an enterprise server, this means you can have users push new libraries to a single collection and create signatures for all of them at once, pruning all duplicate functions in the process. Internally, we use this exact workflow to create the bundled signatures we ship with the product.
You can read more about creating signatures in the user documentation.
Network Support
One major reason why we decided to move away from the SigKit format is so we can easily look up possible functions over the network. 5.1 includes early functionality for accessing an online service, however, it is currently disabled, and we plan to enable it in a subsequent development build.
When enabled and approved (even when enabled by default we plan to require the user to confirm a dialog before using the network server), this server will provide you with possible matches as you navigate. When querying data from the server, only the GUID of the function is sent; no sensitive information and no license information is shared. This is important because an explicit goal of WARP is to be a usable repository of type information for all reverse engineering tooling.
Future Work
We still have many areas of potential improvement, and we intend to add support for additional use cases for WARP. Some of those include:
- Expand the public WARP server to support private, on-premises deployments for enterprise customers pre-populated with data
- Build out fuzzy matching: compare functions by similarity scores, not just exact GUID matches
- Increase the quantity/quality of function constraints to identify more functions automatically
- Integrate WARP in other tools
- Continue growing the bundled signature database, also ship signatures in the extension manager
- Improvements to main function detection by utilizing symbolicated WARP functions
Sunsetting SigKit
Binary Ninja 5.1 is likely the last stable that will ship with SigKit enabled. We plan to move the current SigKit functionality from Binary Ninja to an external plugin to allow users to build SigKit themselves if they still have old signatures they depend on.
Format Improvements
While we went over some of the feature improvements, there were many performance, heuristic, and general improvements and bug fixes in this release. With Binary Ninja 5.1, the WARP format has been stabilized. One of the key improvements in this area was the file size, which has been more than halved following breaking changes described here.
While integration with Binary Ninja is our current focus, we are going to be providing more documentation and code for integration of WARP in other tools. That progress is tracked in the following issues (1, 2, 3, 4) for several popular reverse engineering tools.
Wrapping Up
For anyone who has previously tried WARP, we hope this release makes your workflow with WARP more streamlined and useful. For those who have not tried WARP, we hope this release gives you what you need to start cutting down on the more tedious parts of binary analysis and help you identify interesting functions more quickly! For more information on using WARP, see the online documentation.
While we work towards the release of the public server and more bundled signatures, we are hoping you will join our slack to provide feedback and share your own signatures with the community!