[FS-9600] memory corruption in mod_kazoo Created: 01/Oct/16  Updated: 21/Mar/17  Resolved: 08/Oct/16

Status: Closed
Project: FreeSWITCH
Component/s: mod_kazoo
Affects Version/s: 1.7
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Sergey Safarov Assignee: Luis Azedo
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS 7

Issue Links:
Related
relates to FS-10025 Deglobalize global symbol globals in ... Resolved
CPU Architecture:
x86-64
Kernel:
Linux
uname: Linux fs2.voip.rcsnet.ru 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Userland:
GNU/Linux
Distribution:
CentOS
Distribution Version:
CentOS 7
lsb_release: LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core
Compiler:
gcc
Compiler Version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
FreeSWITCH GIT Revision: 2bd7cfdf9ab74f0e1d1e9bba8c60578ac7dd5401
GIT Master Revision hash:: 2bd7cfdf9ab74f0e1d1e9bba8c60578ac7dd5401
Target Version:
1.9

 Description   
When function "create_socket_with_port" called from mod_kazoo.c:523 then "globals" struct is corrupted.

As example GDB output

Backtrace
(gdb) bt
#0 create_acceptor () at mod_kazoo.c:523
#1 0x00007f077cc79320 in mod_kazoo_load (module_interface=0x7f07a40ad2a0, pool=0x7f07480e86d8) at mod_kazoo.c:651
#2 0x00007f07ab1867ba in switch_loadable_module_load_file (path=0x2489190 "/usr/lib64/freeswitch/mod/mod_kazoo.so", filename=0x2489180 "mod_kazoo", global=SWITCH_FALSE, new_module=0x7f07a40ad338)
    at src/switch_loadable_module.c:1485
#3 0x00007f07ab186c7e in switch_loadable_module_load_module_ex (dir=0x2229120 "/usr/lib64/freeswitch/mod", fname=0x7f0748016100 "mod_kazoo", runtime=SWITCH_TRUE, global=SWITCH_FALSE, err=0x7f07a40ad3e8)
    at src/switch_loadable_module.c:1593
#4 0x00007f07ab186a2f in switch_loadable_module_load_module (dir=0x2229120 "/usr/lib64/freeswitch/mod", fname=0x7f0748016100 "mod_kazoo", runtime=SWITCH_TRUE, err=0x7f07a40ad3e8)
    at src/switch_loadable_module.c:1547
#5 0x00007f079d761580 in load_function (cmd=0x7f0748016100 "mod_kazoo", session=0x0, stream=0x7f07a40add00) at mod_commands.c:2581
#6 0x00007f07ab189e67 in switch_api_execute (cmd=0x7f0748000cd0 "load", arg=0x7f0748000cd5 "mod_kazoo", session=0x0, stream=0x7f07a40add00) at src/switch_loadable_module.c:2565
#7 0x00007f07ab0fdf05 in switch_console_execute (xcmd=0x7f0748001134 "load mod_kazoo", rec=0, istream=0x7f07a40add00) at src/switch_console.c:395
#8 0x00007f079dfab798 in api_exec (thread=0x0, obj=0x7f07a40ae1c0) at mod_event_socket.c:1521
#9 0x00007f079dfae4a9 in parse_command (listener=0x7f0764007448, event=0x7f07a40ae748, reply=0x7f07a40ae750 "", reply_len=512) at mod_event_socket.c:2300
#10 0x00007f079dfafe34 in listener_run (thread=0x7f077cf61868, obj=0x7f0764007448) at mod_event_socket.c:2731
#11 0x00007f07ab434de4 in dummy_worker (opaque=0x7f077cf61868) at threadproc/unix/thread.c:151
#12 0x00007f07a863edc5 in start_thread () from /usr/lib64/libpthread.so.0
#13 0x00007f07a836bced in clone () from /usr/lib64/libc.so.6
(gdb)

Before calling "create_socket_with_port"
(gdb) f 0
#0 create_acceptor () at mod_kazoo.c:523
523 if (!(globals.acceptor = create_socket_with_port(globals.pool, globals.port))) {
(gdb) p globals
$1 = {pool = 0x7f07480e86d8, threads = 0, acceptor = 0x0, ei_cnode = {thishostname = '\000' <repeats 64 times>, thisnodename = '\000' <repeats 128 times>, thisalivename = '\000' <repeats 63 times>,
    ei_connect_cookie = '\000' <repeats 512 times>, creation = 0, self = {node = '\000' <repeats 1020 times>, num = 0, serial = 0, creation = 0}}, ei_nodes_lock = 0x0, ei_nodes = 0x0,
  config_fetch_binding = 0x0, directory_fetch_binding = 0x0, dialplan_fetch_binding = 0x0, chatplan_fetch_binding = 0x0, channels_fetch_binding = 0x0, event_filter = 0x7f07480e8590, epmdfd = 0,
  num_worker_threads = 10, nat_map = SWITCH_FALSE, ei_shortname = SWITCH_FALSE, ei_compat_rel = 0, ip = 0x7f0748023280 "::", ei_cookie = 0x7f07480e68a0 "change_me",
  ei_nodename = 0x7f07480e68c0 "freeswitch", kazoo_var_prefix = 0x7f07480ef0a0 "variable_ecallmgr*", var_prefix_length = 17, flags = 0, send_all_headers = 0, send_all_private_headers = 1,
  connection_timeout = 500, receive_timeout = 1, receive_msg_preallocate = 2000, event_stream_preallocate = 4000, send_msg_batch = 10, event_stream_framing = 2, port = 8031, config_filters_fetched = 0}
(gdb)

When entering to "create_socket_with_port"
(gdb) s
create_socket_with_port (pool=0x7f07480e86d8, port=8031) at kazoo_utils.c:140
140 if(switch_sockaddr_info_get(&sa, globals.ip, SWITCH_UNSPEC, port, 0, pool)) {
(gdb) p globals
$2 = {pool = 0x22307d8, threads = 35896768, acceptor = 0x7f07a482fb70, ei_cnode = {thishostname = '\000' <repeats 64 times>, thisnodename = '\000' <repeats 128 times>,
    thisalivename = '\000' <repeats 63 times>, ei_connect_cookie = '\000' <repeats 422 times>..., creation = 0, self = {
      node = "\330\a#\002\000\000\000\000HWr\244\a\177\000\000\001\000\000\000\001\000\000\000@Vr\244\a\177\000\000\360Ur\244\a\177\000\000\220Vr\244\a\177\000\000@\335\001H\a\177\000\000\002\000\000\000\001", '\000' <repeats 11 times>, "@e8\002", '\000' <repeats 20 times>, "`\342\"\002\000\000\000\000 \221\"\002\000\000\000\000@\231\"\002\000\000\000\000P\235\"\002\000\000\000\000`\241\"\002\000\000\000\000\260\265\"\002\000\000\000\000\300\271\"\002\000\000\000\000@\332\"\002\000\000\000\000н\"\002\000\000\000\000\340\301\"\002\000\000\000\000\220\255\"\002\000\000\000\000\240\261\"\002\000\000\000\000p\245\"\002\000\000\000\000\200"..., num = 2876386352, serial = 32519, creation = 5}}, ei_nodes_lock = 0x0, ei_nodes = 0x0, config_fetch_binding = 0x0, directory_fetch_binding = 0x0,
  dialplan_fetch_binding = 0x1, chatplan_fetch_binding = 0x0, channels_fetch_binding = 0x0, event_filter = 0x0, epmdfd = 0, num_worker_threads = 0, nat_map = SWITCH_TRUE, ei_shortname = SWITCH_FALSE,
  ei_compat_rel = 0, ip = 0x0, ei_cookie = 0x200000000 <Address 0x200000000 out of bounds>, ei_nodename = 0x0, kazoo_var_prefix = 0xffffffff00004e20 <Address 0xffffffff00004e20 out of bounds>,
  var_prefix_length = 1476397248, flags = 32519, send_all_headers = 0, send_all_private_headers = 1079566336, connection_timeout = 0, receive_timeout = 0, receive_msg_preallocate = 0,
  event_stream_preallocate = 1633906540, send_msg_batch = 1936681068, event_stream_framing = 13940, port = 27694, config_filters_fetched = 1818321775}
(gdb)


 Comments   
Comment by Sergey Safarov [ 07/Oct/16 ]
struct global in ".c" files declared as "extern". But in fact CentOS 7 with "gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)" for different source files global struct use different memory regions.

To work arround this is implemented patch.
https://freeswitch.org/stash/users/safarov/repos/freeswitch2/commits/948eed9cab58752cd45f48877fefff3df83c3d6e

If patch may merged after merging PR #1001
Comment by Luis Azedo [ 07/Oct/16 ]
Hi Sergey,
you seem to be the only one having this issue, maybe your problem lies elsewhere.
Comment by Sergey Safarov [ 07/Oct/16 ]
Hi Luis
May be. But to verify I have done:
1) created new host from scratch, and compile latest source - issue is reproduced;
2) upgraded gcc, libtool and binutils to gcc-6.1.1, libtool-2.4.6 and binutils-2.27 and issue is resolved.

I can provide access to build host and you can look it.
Comment by Luis Azedo [ 08/Oct/16 ]
Hi Sergey,

so, if the issue is gone by upgrading "gcc, libtool and binutils to gcc-6.1.1, libtool-2.4.6 and binutils-2.27" , then we should close this ?
Best
Comment by Sergey Safarov [ 08/Oct/16 ]
If build environment of FreeSwitch team will be updated then, yes.
But think it is not simply process.
Comment by Luis Azedo [ 08/Oct/16 ]
we are using builds for centos7 from freeswitch repo and we cannot reproduce this ? (1.6.11)
Comment by Sergey Safarov [ 08/Oct/16 ]
Ok.
I will close it.
Comment by Sergey Safarov [ 21/Mar/17 ]
Think this is fixed in commit a3ee7595bb4de3de6fe00b1b48b31b9aefe3a2f6
Generated at Wed Sep 19 02:50:38 CDT 2018 using JIRA 7.3.3#73014-sha1:d5be8da522213be2ca9ad7b043c51da6e4cc9754.