Debugging Freeswitch
About
This page is about using a debugger with FreeSWITCH.
If you are new to troubleshooting, asking for help, or filing bug reports, please read this first. It will save you a lot of time.
For an introduction to please see troubleshooting see Troubleshooting FreeSWITCH
Troubleshooting
Handy Troubleshooting Links
UUID Stamp at each DEBUG line
The update in the code to enable a UUID Stamp at each debug line (provided kindly by Mathieu Rene and referred to in the "Finding session pointers" section below) has been committed to the GIT head (as of 19/Nov/2009), and you can enable this by a simple option at the logfile.conf.xml:
<param name="uuid" value="true" />
Chasing down XML errors
So you have FreeSWITCH with all of those included files and you get the "error near line X". With all of the included files where does this line number come from.
By default it is ~freeswitch/log/freeswitch.xml.fsxml
Do not edit this file while FreeSWITCH is running. It is memory mapped.
freeswitch@localhost.localdomain>reloadxml
API CALL [reloadxml()] output:
+OK [[error near line 1622]: unexpected closing tag </extension>]
freeswitch@localhost.localdomain>
Doing it Better
Some of these can be hard to track down especially if one has update a number of XML files. In such situation an external tool such as 'xmllint' (part of libxml2-utils on Debian/Ubuntu) should be used. It will narrow down the error to exact line with a more information error message.
Example of cryptic FreeSWITCH output:
root@shared-hosting-1:/usr/local/freeswitch/log# ../bin/freeswitch
Error: stacksize 8192 is not optimal: run ulimit -s 240 from your shell before starting the application.
auto-adjusting stack size for optimal performance...
2013-01-12 23:41:29.410698 [INFO] switch_event.c:589 Activate Eventing Engine.
2013-01-12 23:41:29.421120 [WARNING] switch_event.c:563 Create additional event dispatch thread 0
Cannot Initialize [[error near line 4455]: unexpected <]
Example of xmllint output find the error on line 4465; please note that you will have to also filter through errors generated by some of the vanilla included FreeSwith XML files. In t:
root@shared-hosting-1:/usr/local/freeswitch/log# xmllint freeswitch.xml.fsxml
freeswitch.xml.fsxml:224: parser error : Unescaped '<' not allowed in attributes values
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:224: parser error : attributes construct error
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:224: parser error : Couldn't find end of Start Tag api line 224
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:225: parser error : Opening and ending tag mismatch: exten line 224 and apis
</apis>
^
freeswitch.xml.fsxml:226: parser error : Opening and ending tag mismatch: apis line 223 and configuration
</configuration>
^
freeswitch.xml.fsxml:428: parser error : EntityRef: expecting ';'
am name="url" value="http://query.voipcnam.com/query.php?api_key=MYAPIKEY&number
^
freeswitch.xml.fsxml:4450: parser error : Opening and ending tag mismatch: configuration line 222 and section
</section>
^
freeswitch.xml.fsxml:4465: parser error : StartTag: invalid element name
<! -- this is custom added -->
^
freeswitch.xml.fsxml:11591: parser error : Opening and ending tag mismatch: section line 221 and document
</document>
^
freeswitch.xml.fsxml:11592: parser error : Premature end of data in tag document line 19
^
Recompiling with debug symbols on
Note: FreeSWITCH is compiled with debug symbols on by default !
export CFLAGS="-g -ggdb"
export MOD_CFLAGS="-g -ggdb"
./configure
Creating core files
For core files to be created when a crash occurs, set the core ulimit to unlimited before starting FreeSWITCH
$ ulimit -c unlimited
$ ./freeswitch
The core file should be located in the directory where FreeSWITCH was started (i.e., if you were in the /tmp directory when you typed the command to start FreeSWITCH, then there should be a file called something like /tmp/core.xxx).
NOTE: On macOS, core files are dumped to a hidden directory called /cores by default, not the current directory!
Loading FreeSWITCH in GDB
From /usr/local/freeswitch:
$ gdb bin/freeswitch
Once you have reproduced the error you run the following command:
$ gdb bin/freeswitch core.xxx
Getting a Backtrace
Once you load the core file into GDB you can collect a backtrace typing
set logging file /tmp/backtrace.log
set logging on
set pagination off
bt
bt full
info threads
thread apply all bt
thread apply all bt full
quit
Collect the output when reporting bugs to Jira
If you have a FS source tree available, please run instead:
./scripts/backtrace-from-core /path/to/corefile
Finding session pointers
Sometimes hunting stuff down can be a royal pain. The following code, submitted by Mathieu Rene, will print out all the session information (pointer information, etc.) for each session in a core file. After loading gdb with a core file (see above) do this in gdb:
set $x=session_manager.session_table->table->first
while($x != 0x0)
printf "uuid %s is at %p\n", $x->pKey, $x->data
set $x = $x->next
end
NB: $x->data is the switch_core_session_t*, you can dereference the channel with ->channel and can print some other stuff you might be looking for. This also work to print the content of a hashtable (use hash_table_name->table->first on the first set)
Simple bash script to make debug easy
In the source directory there is a support-d sub-directory. Inside this directory is a handy shell script called fscore_pb. The script will collect the appropriate trace (and some other information), post it to pastebin.freeswitch.org, and then give you the URL to give to the developers.
Usage:
support-d/fscore_pb gcore [user]
support-d/fscore_pb [user]
If gcore is given it will take a gcore of a running instance of FreeSWITCH (this will pause the process during the dump). If gcore is not given it will use the most recent core.* file in the directory.
User is optional, and is the name given when submitting the pastebin. If it is omitted it will use your local username.
gdb issues
On Ubuntu (Dapper/Hardy/Jaunty) gdb does not show symbols for some modules (e.g. mod_portaudio), while CentOS 5.3 haven't got such issue.
The reason of the issue is that gdb in Ubuntu (probably Debian is affected too) is buggy. As a quick fix you can install development version of gdb from sources (install flex, bison and texinfo and get sources from git repository) which solves the issue. gdb trunk 20090626 is proven to work.