Debugging memory leaks
Click here to read more about SpamStopsHere, the e-mail security company that brings you this blog.
Debugging memory leaks is important not only for a well working system, but for system security.
Today we were testing a new platform for our filter servers. The Linux server that it’s hosted on showed over five gigabytes of memory being used when running the free command, not including the buffers and cache, yet adding up the total size of resident memory being used by running processes only equaled about two gigabytes of memory.
After rebooting the server, the memory usage was back to normal. This is a good indicator of a memory leak. There are really only a few software applications that run on this server, and only one that we coded, so I focused my attention there.
There are many memory debugging applications out there, but they’re mostly not just memory debuggers but code debuggers and require some linking and custom compiling of your application. This may be good if you are the software developer, but if you’re just doing some quality control like me, you just want to be able to run a separate program to try to detect any problems.
The tool that I chose is called Valgrind, which is licensed with a GNU General Public License, making it free software. You’ll want to compile valgrind on the computer that you’re testing it on, so it’s not available in any precompiled binary format, and you’ll need the appropriate compilers installed.
Similar to using strace, Valgrind is used by simply adding it to your comamnd line to run your program under its supervision. For example, if you have a program named myprogram:
valgrind –leak-check=yes –trace-children=yes myprogram
Valgrind will detect memory management and threading bugs. Running the software against our filter server application which runs as a daemon, resulted in some useful results:
==28482== Conditional jump or move depends on uninitialised value(s)
==28482== at 0×80496EF: DecodeAllHTML (URL_Filter.c:718)
==28482== by 0×80510D1: preProcessBuffer (tcpServer.c:222)
==28482== by 0×80513AF: processBuffer (tcpServer.c:309)
==28482== by 0×805275C: hin (tcpServer.c:918)
==28482== by 0×8053FC4: stream2 (tcpServer.c:1474)
==28482== by 0×8054D69: processClient (tcpServer.c:1844)
==28482== by 0xBE43CB: start_thread (in /lib/tls/libpthread-2.3.4.so)
==28482== by 0xB4F1AD: clone (in /lib/tls/libc-2.3.4.so)
Valgrind only shows you memory leak information at the end of the program, as it’s deallocating memory, so if you’re using it on a daemon, you’ll likely want a way to signal it to exit.
==28482== LEAK SUMMARY:
==28482== definitely lost: 0 bytes in 0 blocks.
==28482== possibly lost: 952 bytes in 7 blocks.
==28482== still reachable: 109,357,906 bytes in 3,460,664 blocks.
==28482== suppressed: 0 bytes in 0 blocks.
==28482== Reachable blocks (those to which a pointer was found) are not shown.
The “possibly lost” indicates a likely memory leak. Not being the developer, this information was enough for me to send the new platform back to the developers for another look at whatever they missed. If you are the developer, the output gives you the function names and tells you where the memory is being allocated.
Valgrind can also show you memory problems like heap buffer overruns, which are common vectors for attackers to compromise security through privilege escalation. Valgrind will also warn when the program tries to write in another program’s stack, showing possible stack buffer overrun problems, which have been typically exploited for remote command execution. Valgrind won’t magically show you if your software is vulnerable to these types of exploits, because you still have to run the proper test cases, but Valgrind will likely be a valuable tool for detecting these vulnerabilities.
April 2nd, 2008 at 4:48 pm
I’d like to point out that processes often use more memory than is reported by a tool such as Top or Task Manager. Those tools typically report only the real (or system) memory for each process. You should always look at the amount of virtual memory that a process is holding for a true report of the amount of memory that a process is using.
A process may be using 1 GB of memory, even though you might only have .5 GB of RAM. Top or Task Manager may show that the process is using .25 GB because that is how much real (system) memory that the OS is letting the process have. However, the process may in fact be swapping out memory frequently with the virtual memory. If you look at the virtual memory usage, you’ll see the true 1 GB of memory being used.
Use the “ps -aux” command to view the virtual memory of processes. The VSZ column shows the virtual memory, and the RSS column shows the real memory.
For Windows, add the “Virtual Memory Size” column to Task Manager’s Processes tab. Mem Usage is the real memory. VM Size is the virtual memory.
I’d also like to explain what a memory leak is for those who don’t know.
First a little background.
At runtime, a process frequently requests memory from the operating system’s memory manager. The process uses the memory to temporarily store data. When the data is no longer needed, the process must “free” the memory to return it back to the operating system’s memory manager.
Now the definition.
If the process fails to return the memory to the operating system before the register that holds the memory falls out of scope, the process effectively loses track of the memory, and the memory is unable to be returned to the operating system. This is called a memory leak.
If this occurs repeatedly, a process will consume more and more memory from the operating system, and you’ll see the virtual memory usage grow and grow. The memory can be returned to the operating system by ending the process.
A memory leak is usually caused by a logic error, or bug, in a program, which should be fixed.
It’s interesting to note that Java and JavaScript automatically free memory through a process called automatic “garbage collection”, so typically Java and JavaScript are immune to memory leaks, but there is a performance penalty for automatic garbage collection.
April 23rd, 2008 at 12:59 am
You can prevent memory leaks by watching for some common problems. Collection classes, such as hash tables and vectors, are common places to find the cause of a memory leak. This is particularly true if the class has been declared static and exists for the life of the application. . The prevalence of memory leak bugs has led to the development of a number of debugging tools to detect unreachable memory. Coverity Prevent is one of the tool that you can use for fixing these kinds of bugs. Coverity Prevent is also used by the Department of Homeland security to scan many open source projects. You can get more info at http://www.Coverity.com