Debug symbols allow you to have better debugging sessions. They have information about the functions contained in executables and dynamic libraries and provide you with information to get clean call stacks. A Symbol Server allows the debugger to load the correct symbols, binaries and sources automatically.
In this article, I describe how to set up a Symbol Server for your programming team to store symbols (from the operating system, third party libraries and from your own libraries), binaries and source code to improve the productivity of the team and the service given to your clients.
1. Configuring the Symbol Server
You will need the latest version of Debugging Tools for Windows. Download it and install (install the source server support under SDK/Source Server; it is not installed by default and we will need it later). After installing add the installation directory (C:\Program Files\Debugging Tools for Windows) and source server directory (C:\Program Files\Debugging Tools for Windows\sdk\srcsrv) to the PATH environment variable.
Now, you have to decide where you will store the symbols. For a team, having a shared repository is a good idea. Here we will store the symbols in \\BuildServer\Symbols. The only service needed in BuildServer machine is sharing the Symbols folder. Reserve enough space for this purpose (I recommend a minimum of 30Gb).
We will start using symbols downloaded from Microsoft Symbol Server (http://msdl.microsoft.com/download/symbols). This server stores all the symbols from Microsoft products (released operating systems, service packs, directx …)
If you are using Visual Studio 2005, options for symbols are under Tools/Options/Debugging/Symbols
If you are not using Visual Studio 2005, you must set the _NT_SYMBOL_PATH environment variable as follows:
_NT_SYMBOL_PATH = SRV*\\BuildServer\Symbols*http://msdl.microsoft.com/download/symbols
The next time you start debugging, you will start to populate the symbol server with symbols through HTTP from Microsoft. It will only download the symbols needed and cache them for your next debugging session (and for the debugging session of your teammates if you are using a shared directory).
I show you here the stack trace from a session without a Symbol Server and from a session with a Symbol Server installed.
Great difference, isn’t it?
This is only the beginning of the history. Continue reading to discover the amazing things you can do with your new Symbol Server.
2. Populating the server with your own symbols and binaries
You can store your own symbols and binaries into the symbol store using symtore.exe. For a detailed description of the command-line options for symstore consult the documentation given in the SDK or in MSDN.
A good idea is to have your daily build integrated with your Symbol Server, adding all the exes, dlls and pdbs resulting from your builds (daily, stable, beta, etc). For example, the following command stores all pdbs and binary files found under c:\build:
symstore add /r /f c:\build\*.* /s \\BuildServer\Symbols /t “Prueba” /v “Build 0.5.730″ /c “05/05/06 Daily Build”
Inside the directory 000Admin of your shared symbol server directory you will find a file called history.txt with all the transactions that have ocurred in the Symbol Server. You will need this information when you want to delete symbols from the database. For more information consult the documentation.
The next thing you must do is set up your MODPATH environment variable. It says where to get binaries (exes and dlls) from the symbols. It must point to your symbol server cache directory:
You don’t have to do this if you are using Visual Studio 2005. It will automatically retrieve binaries from your Symbol Server.
And now, for the funny part: generate a dump of your program (you can create one while debugging in Debug/Save Dump As…) and copy it to another computer. Now, open that dump (with Visual Studio) and press F5 to start debugging. Visual Studio will automatically download the symbols and the binaries associated to that dump. Magic isn’t it?
But we can do it better. We can retrieve proper source codes too, continue reading.
3. Retrieving source files
To use this feature, you need Visual Studio 2005 (it is not supported by previous versions) and activate the option under Tool/Options/Debugging/General/Enable source server support. This is a new option that has not been publicized by Microsoft and I don’t understand why.
To store sources in your Symbol Server you have to source index your symbols files (pdbs). Inside the symbols you have to store a reference to the source that was used to generate each symbol. The source code is not saved inside the symbols, only a reference to the file in your source server control. Read the srcsrv.doc document provided by the SDK to understand how this feature exactly works.
Inside the SDK of Debugging Tools for Windows you have scripts for Perforce, Visual SourceSafe and SubVersion. They give you enough documentation to extend the support to other source control servers. Perforce will be used here. The scripts are written in perl (yes, Microsoft giving perl scripts) so you will need to install a perl interpreter (you can find one here).
Before storing your symbols in the server (and generally you will be doing this as a phase of your build process) you source index them using p4index (for perforce) located in C:\Program Files\Debugging Tools for Windows\sdk\srcsrv.
You can check that your symbols have been properly indexed using the srctool command.
Prueba2\Debug\Prueba.pdb: 3 source files are indexed – 241 are not
And now you are ready to repeat the experiment, open the dump in a machine with no source, no binaries and no symbols. As soon as you start debugging, the source will be automatically downloaded from your source server (perforce in this example)
Don’t you find this exciting?
4. How to get maximum productivity
We have seen how Visual Studio connect with the Symbol Server to get the needed information. Your program can do the same to get extra information. These are some guidelines:
- Store all your builds in the Symbol Server (daily and stable at least). Clients should be using the binaries produced by your build system.
- Provide stack traces in asserts, error dialogs, loggers. Get those traces using your symbol server.
- Generate minidumps in critical errors. Those minidumps should be sent to your team for analyzing.
- Use all this information (traces, dumps) when publishing reports in your bugs database.
I’ll write more articles about these points and how to access the Symbol Server from your code in future posts. For now, this is enough. I hope you enjoyed this.