This page is about using a debugger with FreeSWITCH. This is an advanced topic. For an introduction to troubleshooting please see Troubleshooting FreeSWITCH
Handy Troubleshooting Links
The update in the code to enable a UUID Stamp at each debug line was kindly provided by Mathieu Rene and is referred to in the "Finding session pointers" section below. You can enable this by a simple option at the
<param name="uuid" value="true" />
In a message like "error near line X" the line number refers to the concatenation of all the individual configuration files named freeswitch.xml.fsxml and is kept in memory by FreeSWITCH™. The default location is ~freeswitch/log/freeswitch.xml.fsxml
Do not edit this file. Edit the individual configuration file, then issue the reloadxml command in fs_cli or the FS console.
firstname.lastname@example.org>reloadxml API CALL [reloadxml()] output: +OK [[error near line 1622]: unexpected closing tag </extension>] email@example.com>
In the example above, line 1622 refers to the offending line number in freeswitch.xml.fsxml so open that file to determine where the error lies, but correct the problem in the source file, whether it be dialplan/default.xml or directory/default/1000.xml or wherever. Do not edit the freeswitch.xml.fsxml file directly.
Doing it Better
Some of these can be hard to track down especially if one has update a number of XML files. In such situation an external tool such as 'xmllint' (part of libxml2-utils on Debian/Ubuntu) should be used. It will narrow down the error to exact line with a more information error message.
Example of cryptic FreeSWITCH output:
Example of xmllint output find the error on line 4465; please note that you will also have to filter through errors generated by some of the configuration included in FreeSwith XML files assuming you haven't deleted much of those configs.
Recompiling with debug symbols on
Note: FreeSWITCH is compiled with debug symbols on by default !
export CFLAGS="-g -ggdb" export MOD_CFLAGS="-g -ggdb" ./configure
Creating core files
For core files to be created when a crash occurs, set the core ulimit to unlimited before starting FreeSWITCH
$ ulimit -c unlimited $ ./freeswitch
The core file should be located in the directory where FreeSWITCH was started (i.e., if you were in the /tmp directory when you typed the command to start FreeSWITCH, then there should be a file called something like /tmp/core.xxx).
NOTE: On OS X, core files are dumped to a hidden directory called /cores by default, not the current directory!
Loading FreeSWITCH in GDB
$ gdb bin/freeswitch
Once you have reproduced the error you run the following command:
Getting a Backtrace
Once you load the core file into GDB you can collect a backtrace typing
Default output to the file gdb.txt
Collect the output when reporting bugs to Jira.
If you have a FS source tree available, please run instead:
Finding session pointers
Sometimes hunting stuff down can be a royal pain. The following code, submitted by Mathieu Rene, will print out all the session information (pointer information, etc.) for each session in a core file. After loading gdb with a core file (see above) do this in gdb:
set $x=session_manager.session_table->table->first while($x != 0x0) printf "uuid %s is at %p\n", $x->pKey, $x->data set $x = $x->next end
NB: $x->data is the switch_core_session_t*, you can dereference the channel with ->channel and can print some other stuff you might be looking for. This also work to print the content of a hashtable (use hash_table_name->table->first on the first set)
Simple bash script to make debug easy
In the source directory there is a support-d sub-directory. Inside this directory is a handy shell script called fscore_pb. The script will collect the appropriate trace (and some other information), post it to pastebin.freeswitch.org, and then give you the URL to give to the developers.
support-d/fscore_pb gcore [user] support-d/fscore_pb [user]
If gcore is given it will take a gcore of a running instance of FreeSWITCH (this will pause the process during the dump). If gcore is not given it will use the most recent core.* file in the directory.
User is optional, and is the name given when submitting the pastebin. If it is omitted it will use your local username.
On Ubuntu (Dapper/Hardy/Jaunty) gdb does not show symbols for some modules (e.g. mod_portaudio), while CentOS 5.3 haven't got such issue.
The reason of the issue is that gdb in Ubuntu (probably Debian is affected too) is buggy. As a quick fix you can install development version of gdb from sources (install flex, bison and texinfo and get sources from git repository) which solves the issue. gdb trunk 20090626 is proven to work.