Predicting Buffer Overflows Using Shimming Technology

            A buffer overflow (or buffer overrun) is a well-known class of security vulnerability wherein a program tries to write more data than will fit into a buffer of a particular size.  A malicious user can crash an application by passing specially crafted data to the application.  The malicious user can also cause the application to execute unauthorized instructions.  For instance, if an application that executes with super user privileges fails to ensure that the length of input data entered is equal to or smaller than a buffer that has been allocated to store the input data, then any overflow data will simply be written over memory locations that happen to be next to the buffer.  If this post-buffer memory area happens to hold executable instructions for the application, the malicious user can overwrite the application instructions with a different set of instructions that can perform unauthorized tasks such as altering or deleting important system files.

            Properly written programs ought to check the length of input data to ensure that the input data is not larger than the buffer that has been allocated for it, but this precaution is frequently overlooked or implemented incorrectly.  There are numerous tools for detecting buffer overflows.  However, these tools can detect buffer overflows only by actually causing them first.  An application often crashes or malfunctions each time after a buffer overflow is induced, thereby interfering significantly with testing the application.  If buffer overflows can be predicted without having to cause them first, potential causes of buffer overflows can be uncovered without getting in the way of testing the application.  For instance, testers may be able to identify sources of buffer overflows without having to alter their testing logic to search for them.

            Buffer overflows happen most commonly when an application invokes a buffer-overflow prone application programming interface (API).  FIG.1 illustrates the basic operations for predicting buffer overflows.  In operation 101, applications that need to be tested are started.  Next, in operation 102, buffer-overflow prone APIs are identified based on a set of commonly observed characteristics of input parameters.  In operation 103, the maximum size of the input buffer of the identified APIs can be estimated.  Finally, in operation 104, the estimated maximum size of the input buffer is validated against the size of the input buffer that is declared by the input parameters.

            A method for predicting buffer overflows without having to cause them can be implemented using a shimming tool.  When an API is shimmed, a call that is made to the API by an application is intercepted and directed to execute a different function.  For instance, snprintf function can be shimmed so that when an application invokes snprin...