aMule Forum
English => en_Bugs => Topic started by: zekkerj on December 27, 2010, 05:32:08 AM
-
Hi,
Sorry to dig up the thread but a few months back here my amule has caused memory leak. My system has 2GiB of RAM, and 8GiB swap.
amule runs fine for hours, sometimes days, then suddenly starts to allocate memory desperately, until it wipes out all memory and make the system hang. To lessen the problem, I set a 3GiB limit per process. Now at least the system survives the failure of amule.
I would really like to fix this.
aMule 2.2.6 using wxGTK2 v2.8.10 (OS: Linux)
Kubuntu 10.04
Kernel 2.6.32-27-49-generic x86_64
I suppose I should gather some backtraces of failure, where should I start?
Oh, one detail: the failure began to appear after I changed my 1Mbps connection by another of 15Mbps (God bless my ISP). Not sure if it didn't happened before, but now it's much more frequent.
-
That's not related, so I made a new thread for it.
Sounds like the RRM problem. Please try the SVN version. There is a fix for a memleak on high bandwidth in it.
-
OK. Downloaded "aMule-SVN-r10381". Compiled OK.
I'm giving it a try, right now.
Thanks.
-
Terminated after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
backtrace:
[2] ) [0x3647ccad16] in /usr/lib/libstdc++.so.6[0x3647ccad16]
[3] ) [0x3647ccad43] in /usr/lib/libstdc++.so.6[0x3647ccad43]
[4] __cxa_rethrow in /usr/lib/libstdc++.so.6[0x3647ccadc6]
[5] ?? in /usr/local/bin/amule[0x5e3ff2]
[6] ?? in /usr/local/bin/amule[0x5e22f4]
[7] ?? in /usr/local/bin/amule[0x5e0806]
[8] ?? in /usr/local/bin/amule[0x5deb7c]
[9] ?? in /usr/local/bin/amule[0x5dcdea]
[10] ?? in /usr/local/bin/amule[0x5db126]
[11] ?? in /usr/local/bin/amule[0x5d98a0]
[12] wxThreadInternal::PthreadStart(wxThread*) in /usr/lib/libwx_baseu-2.8.so.0[0x3640ee8111]
[13] ?? in /lib/libpthread.so.0[0x7f99b6a129ca]
[14] clone in /lib/libc.so.6[0x7f99b64ec70d]
-
So it's not that. (You should compile with debug symbols to get a useful backtrace, but with bad_alloc it's useless anyway.) Hm.
Kubuntu 10.04
Kernel 2.6.32-27-49-generic x86_64
Is that a recent kernel (a few weeks old)? Can you boot from an older kernel instead? We are having oom problems with recent Ubuntu kernels, mostly in 10.10 though.
-
Yes, it's recent, but the problem ocurred in very same way in the other kernels (2.6.32 22-25).
By the way, was this BT useful? Or should I change the compile directives to make it better?
-
Well, when the program runs out of memory, the problem is where the memory is wasted, not where it finally runs out.
Otoh, chances are you can catch the actual waster. Probably the upload thread I think.
Yes, please compile with debug infos enabled and run it from gdb, as described in the sticky in the backtrace board.
-
Im archlinux 64bit user. I tried last svn tarball and last stable release. There is a memory problem it s certain. After running for a while, amule starts trashing hdds and uses more and more ram till amule crash when reaching ram's hardware limit.
I recompile amule's versions with debug option but there s nothing to share about problem.
And indeed this is important problem to make amule useless. I read 10417's "Fix possible memleak in CKademliaUDPListener::ProcessSearchResponse()" log but there are another problem about memory leak.
-
is there any chance that the problem be related to search? I have a feeling (i.e. I never confirmed this reasonably) that amule only crashes if I use the search (global and/or kad).
-
I thought same thing, and I left amule working without any searching and crashed again.
-
In last try I was some sleepy and to be sure, again I tried to use amule without any search and crashed again.
-
There we go...
Program received signal SIGSEGV, Segmentation fault.
__pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
50 pthread_mutex_lock.c: Arquivo ou diretório não encontrado.
in pthread_mutex_lock.c
(gdb) bt
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x0000003644c3e5cf in ?? () from /lib/libglib-2.0.so.0
#2 0x0000003644c40dfe in g_source_remove () from /lib/libglib-2.0.so.0
#3 0x000000364b9e4c36 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#4 0x0000003641a24b67 in GSocket::Detected_Write() () from /usr/lib/libwx_baseu_net-2.8.so.0
#5 0x000000364b425d3f in ?? () from /usr/lib/libgdk-x11-2.0.so.0
#6 0x0000003644c3e8c2 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#7 0x0000003644c42748 in ?? () from /lib/libglib-2.0.so.0
#8 0x0000003644c42c55 in g_main_loop_run () from /lib/libglib-2.0.so.0
#9 0x000000364f93bbb7 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
#10 0x000000364b9e3558 in wxEventLoop::Run() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#11 0x000000364ba682eb in wxAppBase::MainLoop() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#12 0x0000003640e8f47c in wxEntry(int&, wchar_t**) () from /usr/lib/libwx_baseu-2.8.so.0
#13 0x000000000064eb38 in main (argc=1, argv=0x7fffffffe258) at amule-gui.cpp:94
(gdb) bt full
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#1 0x0000003644c3e5cf in ?? () from /lib/libglib-2.0.so.0
No symbol table info available.
#2 0x0000003644c40dfe in g_source_remove () from /lib/libglib-2.0.so.0
No symbol table info available.
#3 0x000000364b9e4c36 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) () from /usr/lib/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#4 0x0000003641a24b67 in GSocket::Detected_Write() () from /usr/lib/libwx_baseu_net-2.8.so.0
No symbol table info available.
#5 0x000000364b425d3f in ?? () from /usr/lib/libgdk-x11-2.0.so.0
No symbol table info available.
#6 0x0000003644c3e8c2 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
No symbol table info available.
#7 0x0000003644c42748 in ?? () from /lib/libglib-2.0.so.0
No symbol table info available.
#8 0x0000003644c42c55 in g_main_loop_run () from /lib/libglib-2.0.so.0
No symbol table info available.
#9 0x000000364f93bbb7 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
No symbol table info available.
#10 0x000000364b9e3558 in wxEventLoop::Run() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#11 0x000000364ba682eb in wxAppBase::MainLoop() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#12 0x0000003640e8f47c in wxEntry(int&, wchar_t**) () from /usr/lib/libwx_baseu-2.8.so.0
No symbol table info available.
#13 0x000000000064eb38 in main (argc=1, argv=0x7fffffffe258) at amule-gui.cpp:94
No locals.
(gdb) thread apply all bt
Thread 6 (Thread 0x7ffff3369700 (LWP 16812)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:211
#1 0x0000003640ee6a16 in wxConditionInternal::WaitTimeout(unsigned long) () from /usr/lib/libwx_baseu-2.8.so.0
#2 0x0000003640ee79e7 in wxSemaphoreInternal::WaitTimeout(unsigned long) () from /usr/lib/libwx_baseu-2.8.so.0
#3 0x0000000000796d64 in CTimerThread::Entry (this=0x7fffec0008f0) at Timer.cpp:66
#4 0x0000003640ee8111 in wxThreadInternal::PthreadStart(wxThread*) () from /usr/lib/libwx_baseu-2.8.so.0
#5 0x00007ffff7bc69ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#6 0x00007ffff76a070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7ffff436b700 (LWP 16809)):
#0 0x00007ffff7bcf11d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1 0x0000003640eeda5c in wxMicroSleep(unsigned long) () from /usr/lib/libwx_baseu-2.8.so.0
#2 0x00000000005d95ae in UploadBandwidthThrottler::Entry (this=0x1cb33f0) at UploadBandwidthThrottler.cpp:323
#3 0x0000003640ee8111 in wxThreadInternal::PthreadStart(wxThread*) () from /usr/lib/libwx_baseu-2.8.so.0
#4 0x00007ffff7bc69ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#5 0x00007ffff76a070d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7ffff7fbe800 (LWP 16799)):
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x0000003644c3e5cf in ?? () from /lib/libglib-2.0.so.0
#2 0x0000003644c40dfe in g_source_remove () from /lib/libglib-2.0.so.0
#3 0x000000364b9e4c36 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#4 0x0000003641a24b67 in GSocket::Detected_Write() () from /usr/lib/libwx_baseu_net-2.8.so.0
#5 0x000000364b425d3f in ?? () from /usr/lib/libgdk-x11-2.0.so.0
#6 0x0000003644c3e8c2 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#7 0x0000003644c42748 in ?? () from /lib/libglib-2.0.so.0
#8 0x0000003644c42c55 in g_main_loop_run () from /lib/libglib-2.0.so.0
#9 0x000000364f93bbb7 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
#10 0x000000364b9e3558 in wxEventLoop::Run() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#11 0x000000364ba682eb in wxAppBase::MainLoop() () from /usr/lib/libwx_gtk2u_core-2.8.so.0
#12 0x0000003640e8f47c in wxEntry(int&, wchar_t**) () from /usr/lib/libwx_baseu-2.8.so.0
#13 0x000000000064eb38 in main (argc=1, argv=0x7fffffffe258) at amule-gui.cpp:94
(gdb)
Interestingly, now I've got a "Segmentation Violation". Is this error wxGTK related?
-
People, anything bad can happen when you are running out of memory.
OK. Can you try a few things to help track the problem down:
- Try latest SVN. I have plugged a memory leak in 10417, though I doubt it was The Big One.
- Try amuled with amulegui instead of amule, to see if it makes a difference. If it crashes - which of the two uses up the memory?
- Try amule but disable Kad, use only ED2K.
-
I tried KAD disabled and it crashed again. Now amuled with amulegui is working for a while. I ll report what ll happen.
-
still both(amuled and amuleguis) is running and still no crash. Normally it should crashed several times in that time.
-
Very interesting.
Is your up/download speed still the same as it was with amule?
Did you compile wxWidgets yourself, or did it come with your distro?
-
Well, I was in front of my computer when aMule started going eating all the available memory in a similar fashion than it is described here.
I attach amule to gdb and tried to debug, but nothing of value was gained. In all cases, the backtrace pointed to WxSocket::Clone being called again and again from an event. :(
No repeat yet.
-
That backtrace would have been really interesting. :-(
I did suspect some networking problem. amuled uses different network functions (at least when built with wx 2.8 ), which could explain why only the monolith is involved.
-
It happened again, but I wasn't in front of my computer and couldn't interrupt amule.
I'm using this shell hack in order to attach gdb to aMule next time it happens, it could be useful for some of you. I'll post the backtrace again.
while true ; do if `test \`ps axl | grep amule | grep -v grep | awk '{ print $8 ;}'\` -gt 2000000`; then gdb -p `pidof amule`; else sleep 10; fi; done
This checks whether amule's RSS is bigger than 2Gb and if so, it launches gdb (and stops amule) In Ubuntu it should be run as root.
[I'm using svn head]
-
Disabling KAD seems to have worked for me. Maybe you have a different problem, as you're using a different system?
-
Well, it happened again, aMule did eat all the available memory, but this time the script did save my gnome-session and allowed me to debug.
IMVHO this is the same case than in RRM, it happened when I did set up a hard limit on speed and a lot of connections started queuing without being served, or something like that.
Some backtraces:
(gdb) thread apply all bt
Hilo 3 (Thread 0x7fc3f6bb4700 (LWP 22872)):
#0 0x00007fc401f5036d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007fc40058b3ec in wxMicroSleep (microseconds=<value optimized out>) at ../src/unix/utilsunx.cpp:191
#2 0x0000000000574bab in UploadBandwidthThrottler::Entry (this=<value optimized out>)
at UploadBandwidthThrottler.cpp:323
#3 0x00007fc400584dfa in wxThreadInternal::PthreadStart (thread=0x2cd7c00) at ../src/unix/threadpsx.cpp:766
#4 0x00007fc401f48971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#5 0x00007fc3ff81692d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Hilo 2 (Thread 0x7fc3f5bb1700 (LWP 22874)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00007fc401f4a849 in _L_lock_953 () from /lib/libpthread.so.0
#2 0x00007fc401f4a66b in __pthread_mutex_lock (mutex=0x244c7d0) at pthread_mutex_lock.c:61
#3 0x00007fc400583216 in wxMutexInternal::Lock (this=0x244c7d0) at ../src/unix/threadpsx.cpp:248
#4 0x00007fc400586652 in Enter (this=0x23d41e0, event=<value optimized out>) at ../include/wx/thread.h:271
#5 wxEvtHandler::AddPendingEvent (this=0x23d41e0, event=<value optimized out>) at ../src/common/event.cpp:1152
#6 0x00000000006b95d1 in CTimerThread::Entry (this=0x2e2ad00) at Timer.cpp:70
#7 0x00007fc400584dfa in wxThreadInternal::PthreadStart (thread=0x2e2ad00) at ../src/unix/threadpsx.cpp:766
#8 0x00007fc401f48971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#9 0x00007fc3ff81692d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#10 0x0000000000000000 in ?? ()
Hilo 1 (Thread 0x7fc402337940 (LWP 22867)):
#0 wxNodeBase::wxNodeBase (this=0x7fc37bb534b0, list=0x7fc3f00010b0, previous=0x7fc37bb533d0, next=0x0,
data=0xc1e500, key=...) at ../src/common/list.cpp:76
#1 0x00007fc400517cf5 in wxObjectListNode (this=0x7fc3f00010b0, prev=0x7fc37bb533d0, next=0x0, data=0xc1e500,
key=...) at ../include/wx/list.h:1185
#2 wxObjectList::CreateNode (this=0x7fc3f00010b0, prev=0x7fc37bb533d0, next=0x0, data=0xc1e500, key=...)
at ../include/wx/list.h:1185
#3 0x00007fc40052b001 in wxListBase::Append (this=0x7fc3f00010b0, object=0x0) at ../src/common/list.cpp:244
#4 0x00007fc40058666d in Append (this=0xc1e500, event=<value optimized out>) at ../include/wx/list.h:1185
#5 wxEvtHandler::AddPendingEvent (this=0xc1e500, event=<value optimized out>) at ../src/common/event.cpp:1156
#6 0x00007fc40082fa84 in wxSocketBase::OnRequest (this=0x485c990, notification=wxSOCKET_OUTPUT)
at ../src/common/socket.cpp:1006
#7 0x00007fc40083442e in GSocket::Detected_Write (this=0x6459110) at ../src/unix/gsocket.cpp:1836
#8 0x00007fc3fec8399f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>,
data=<value optimized out>) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#9 0x00007fc3fd0ec342 in g_main_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#10 g_main_context_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#11 0x00007fc3fd0f02a8 in g_main_context_iterate (context=0x23d4780, block=<value optimized out>,
dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#12 0x00007fc3fd0f07b5 in g_main_loop_run (loop=0x2c3b4e0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#13 0x00007fc3ff0423e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#14 0x00007fc400c15a28 in wxEventLoop::Run (this=0x2e2a640) at ../src/gtk/evtloop.cpp:76
#15 0x00007fc400ca5f18 in wxAppBase::MainLoop (this=0x23d41e0) at ../src/common/appcmn.cpp:312
#16 0x00007fc40051f025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>)
at ../src/common/init.cpp:448
#17 0x00000000005d0892 in main (argc=1, argv=0x7fc3f00010b0) at amule-gui.cpp:93
(gdb) thread apply all bt full
Hilo 3 (Thread 0x7fc3f6bb4700 (LWP 22872)):
#0 0x00007fc401f5036d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
No locales.
#1 0x00007fc40058b3ec in wxMicroSleep (microseconds=<value optimized out>) at ../src/unix/utilsunx.cpp:191
tmReq = {tv_sec = 0, tv_nsec = 253000000}
#2 0x0000000000574bab in UploadBandwidthThrottler::Entry (this=<value optimized out>)
at UploadBandwidthThrottler.cpp:323
timeSinceLastLoop = 1
minFragSize = 1300
doubleSendSize = 2600
sleepTime = 254
thisLoopTick = 1572013229
bytesToSpend = <value optimized out>
extraSleepTime = 1
lastLoopTick = 1572013229
allowedDataRate = 10240
rememberedSlotCounter = 6
sendLock = {m_isOk = 16, m_mutex = @0x7fc40052d7d8}
#3 0x00007fc400584dfa in wxThreadInternal::PthreadStart (thread=0x2cd7c00) at ../src/unix/threadpsx.cpp:766
pthread = 0x2aa7110
rc = <value optimized out>
dontRunAtAll = false
__FUNCTION__ = "PthreadStart"
#4 0x00007fc401f48971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
__res = <value optimized out>
pd = 0x7fc3f6bb4700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140479634818816, -4108199948263213771, 140734433312144,
140479634819520, 140479827537984, 3, 4141992929088386357, 4140142868344747317}, mask_was_saved = 0}},
priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#5 0x00007fc3ff81692d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#6 0x0000000000000000 in ?? ()
No symbol table info available.
Hilo 2 (Thread 0x7fc3f5bb1700 (LWP 22874)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
No locales.
#1 0x00007fc401f4a849 in _L_lock_953 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00007fc401f4a66b in __pthread_mutex_lock (mutex=0x244c7d0) at pthread_mutex_lock.c:61
ignore1 = <value optimized out>
ignore2 = 38062032
ignore3 = -512
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#3 0x00007fc400583216 in wxMutexInternal::Lock (this=0x244c7d0) at ../src/unix/threadpsx.cpp:248
err = <value optimized out>
__FUNCTION__ = "Lock"
#4 0x00007fc400586652 in Enter (this=0x23d41e0, event=<value optimized out>) at ../include/wx/thread.h:271
No locales.
#5 wxEvtHandler::AddPendingEvent (this=0x23d41e0, event=<value optimized out>) at ../src/common/event.cpp:1152
eventCopy = <value optimized out>
__FUNCTION__ = "AddPendingEvent"
#6 0x00000000006b95d1 in CTimerThread::Entry (this=0x2e2ad00) at Timer.cpp:70
now = <value optimized out>
delta = <value optimized out>
---Type <return> to continue, or q <return> to quit---
evt = {<wxEvent> = {<wxObject> = {_vptr.wxObject = 0x7dfa10, static ms_classInfo = {
m_className = 0x7fc4005b5050 L"wxObject", m_objectSize = 16, m_objectConstructor = 0,
m_baseInfo1 = 0x0, m_baseInfo2 = 0x0, static sm_first = 0x7fc40137b180, m_next = 0x7fc40080dd20,
static sm_classTable = 0x2393010}, m_refData = 0x0}, m_eventObject = 0x0, m_eventType = 10245,
m_timeStamp = 0, m_id = 6128, m_callbackUserData = 0x0, m_propagationLevel = 0, m_skipped = false,
m_isCommandEvent = false, static ms_classInfo = {m_className = 0x7fc4005c71c0 L"wxEvent",
m_objectSize = 64, m_objectConstructor = 0, m_baseInfo1 = 0x7fc40080dc20, m_baseInfo2 = 0x0,
static sm_first = 0x7fc40137b180, m_next = 0x7fc40080fbc0,
static sm_classTable = 0x2393010}}, <No data fields>}
lastEvent = 1572010891
#7 0x00007fc400584dfa in wxThreadInternal::PthreadStart (thread=0x2e2ad00) at ../src/unix/threadpsx.cpp:766
pthread = 0x2e71ec0
rc = <value optimized out>
dontRunAtAll = false
__FUNCTION__ = "PthreadStart"
#8 0x00007fc401f48971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
__res = <value optimized out>
pd = 0x7fc3f5bb1700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140479618029312, -4108199948263213771, 140734433313232,
140479618030016, 140479827537984, 3, 4141999526695023925, 4140142868344747317}, mask_was_saved = 0}},
priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#9 0x00007fc3ff81692d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#10 0x0000000000000000 in ?? ()
No symbol table info available.
Hilo 1 (Thread 0x7fc402337940 (LWP 22867)):
#0 wxNodeBase::wxNodeBase (this=0x7fc37bb534b0, list=0x7fc3f00010b0, previous=0x7fc37bb533d0, next=0x0,
data=0xc1e500, key=...) at ../src/common/list.cpp:76
No locales.
#1 0x00007fc400517cf5 in wxObjectListNode (this=0x7fc3f00010b0, prev=0x7fc37bb533d0, next=0x0, data=0xc1e500,
key=...) at ../include/wx/list.h:1185
No locales.
#2 wxObjectList::CreateNode (this=0x7fc3f00010b0, prev=0x7fc37bb533d0, next=0x0, data=0xc1e500, key=...)
at ../include/wx/list.h:1185
No locales.
#3 0x00007fc40052b001 in wxListBase::Append (this=0x7fc3f00010b0, object=0x0) at ../src/common/list.cpp:244
__FUNCTION__ = "Append"
node = <value optimized out>
#4 0x00007fc40058666d in Append (this=0xc1e500, event=<value optimized out>) at ../include/wx/list.h:1185
No locales.
#5 wxEvtHandler::AddPendingEvent (this=0xc1e500, event=<value optimized out>) at ../src/common/event.cpp:1156
eventCopy = <value optimized out>
__FUNCTION__ = "AddPendingEvent"
#6 0x00007fc40082fa84 in wxSocketBase::OnRequest (this=0x485c990, notification=wxSOCKET_OUTPUT)
at ../src/common/socket.cpp:1006
event = aviso: can't find linker symbol for virtual table for `wxSocketEvent' value
{<wxEvent> = {<wxObject> = {_vptr.wxObject = 0xc11470, static ms_classInfo = {
m_className = 0x7fc4005b5050 L"wxObject", m_objectSize = 16, m_objectConstructor = 0,
m_baseInfo1 = 0x0, m_baseInfo2 = 0x0, static sm_first = 0x7fc40137b180, m_next = 0x7fc40080dd20,
static sm_classTable = 0x2393010}, m_refData = 0x0}, m_eventObject = 0x485c990, m_eventType = 10002,
m_timeStamp = 0, m_id = 6123, m_callbackUserData = 0x0, m_propagationLevel = 0, m_skipped = false,
m_isCommandEvent = false, static ms_classInfo = {m_className = 0x7fc4005c71c0 L"wxEvent",
m_objectSize = 64, m_objectConstructor = 0, m_baseInfo1 = 0x7fc40080dc20, m_baseInfo2 = 0x0,
static sm_first = 0x7fc40137b180, m_next = 0x7fc40080fbc0, static sm_classTable = 0x2393010}},
m_event = wxSOCKET_OUTPUT, m_clientData = 0x0, static ms_classInfo = {
m_className = 0x7fc400836e58 L"wxSocketEvent", m_objectSize = 80,
m_objectConstructor = 0x7fc40082e850 <wxSocketEvent::wxCreateObject()>, m_baseInfo1 = 0x7fc40080fc00,
---Type <return> to continue, or q <return> to quit---
m_baseInfo2 = 0x0, static sm_first = 0x7fc40137b180, m_next = 0x7fc400a40b40,
static sm_classTable = 0x2393010}}
flag = <value optimized out>
#7 0x00007fc40083442e in GSocket::Detected_Write (this=0x6459110) at ../src/unix/gsocket.cpp:1836
No locales.
#8 0x00007fc3fec8399f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>,
data=<value optimized out>) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
closure = 0x7fc3e81c7c70
gdk_cond = GDK_INPUT_WRITE
#9 0x00007fc3fd0ec342 in g_main_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
dispatch = 0x7fc3fd130a90 <g_io_unix_dispatch>
user_data = 0x7fc3e81c7c70
callback = 0x7fc3fec83950 <gdk_io_invoke>
cb_funcs = 0x7fc3fd38c610
cb_data = 0x7fc3e80659e0
current_source_link = {data = 0x7fc3e81a8110, next = 0x0}
source = 0x7fc3e81a8110
current = 0x2c22a60
i = 1
#10 g_main_context_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
No locales.
#11 0x00007fc3fd0f02a8 in g_main_context_iterate (context=0x23d4780, block=<value optimized out>,
dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
max_priority = 110
timeout = 0
some_ready = 1
nfds = 38
allocated_nfds = 12565584
fds = <value optimized out>
__PRETTY_FUNCTION__ = "g_main_context_iterate"
#12 0x00007fc3fd0f07b5 in g_main_loop_run (loop=0x2c3b4e0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
self = 0x23d5570
__PRETTY_FUNCTION__ = "g_main_loop_run"
#13 0x00007fc3ff0423e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
tmp_list = 0x2e2a640
functions = 0x0
init = 0x7fc40108d588
loop = <value optimized out>
#14 0x00007fc400c15a28 in wxEventLoop::Run (this=0x2e2a640) at ../src/gtk/evtloop.cpp:76
__FUNCTION__ = "Run"
exitcode = <value optimized out>
#15 0x00007fc400ca5f18 in wxAppBase::MainLoop (this=0x23d41e0) at ../src/common/appcmn.cpp:312
mainLoop = {<wxEventLoopPtr> = {m_ptr = 0x2e2a640}, m_pp = 0x23d4258, m_pOld = 0x0}
#16 0x00007fc40051f025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>)
at ../src/common/init.cpp:448
No locales.
#17 0x00000000005d0892 in main (argc=1, argv=0x7fc3f00010b0) at amule-gui.cpp:93
No locales.
-
Some more backtraces (after continuing and pressing Ctrl-C)
(gdb) bt
#0 __pthread_mutex_unlock_usercnt (mutex=0x23a9840) at pthread_mutex_unlock.c:41
#1 __pthread_mutex_unlock (mutex=0x23a9840) at pthread_mutex_unlock.c:290
#2 0x00007fc400583066 in wxMutexInternal::Unlock (this=0x23a9840) at ../src/unix/threadpsx.cpp:297
#3 0x00007fc40082fa84 in wxSocketBase::OnRequest (this=0x485c990, notification=wxSOCKET_OUTPUT)
at ../src/common/socket.cpp:1006
#4 0x00007fc40083442e in GSocket::Detected_Write (this=0x6459110) at ../src/unix/gsocket.cpp:1836
#5 0x00007fc3fec8399f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>,
data=<value optimized out>) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#6 0x00007fc3fd0ec342 in g_main_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#7 g_main_context_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#8 0x00007fc3fd0f02a8 in g_main_context_iterate (context=0x23d4780, block=<value optimized out>,
dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#9 0x00007fc3fd0f07b5 in g_main_loop_run (loop=0x2c3b4e0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#10 0x00007fc3ff0423e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#11 0x00007fc400c15a28 in wxEventLoop::Run (this=0x2e2a640) at ../src/gtk/evtloop.cpp:76
#12 0x00007fc400ca5f18 in wxAppBase::MainLoop (this=0x23d41e0) at ../src/common/appcmn.cpp:312
#13 0x00007fc40051f025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>)
at ../src/common/init.cpp:448
#14 0x00000000005d0892 in main (argc=1, argv=0x0) at amule-gui.cpp:93
(gdb) bt
#0 sYSMALLOc (av=0x7fc3f0000020, bytes=48) at malloc.c:3432
#1 _int_malloc (av=0x7fc3f0000020, bytes=48) at malloc.c:4747
#2 0x00007fc3ff7ac38e in __libc_malloc (bytes=48) at malloc.c:3660
#3 0x00007fc400005ded in operator new(unsigned long) () from /usr/lib/libstdc++.so.6
#4 0x00007fc400517cdb in wxObjectList::CreateNode (this=0x7fc3f00010b0, prev=0x7fc35c080020, next=0x45, data=0x4,
key=...) at ../include/wx/list.h:1185
#5 0x00007fc40052b001 in wxListBase::Append (this=0x7fc3f00010b0, object=0x4) at ../src/common/list.cpp:244
#6 0x00007fc40058666d in Append (this=0xc1e500, event=<value optimized out>) at ../include/wx/list.h:1185
#7 wxEvtHandler::AddPendingEvent (this=0xc1e500, event=<value optimized out>) at ../src/common/event.cpp:1156
#8 0x00007fc40082fa84 in wxSocketBase::OnRequest (this=0x485c990, notification=wxSOCKET_OUTPUT)
at ../src/common/socket.cpp:1006
#9 0x00007fc40083442e in GSocket::Detected_Write (this=0x6459110) at ../src/unix/gsocket.cpp:1836
#10 0x00007fc3fec8399f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>,
data=<value optimized out>) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#11 0x00007fc3fd0ec342 in g_main_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#12 g_main_context_dispatch (context=0x23d4780) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#13 0x00007fc3fd0f02a8 in g_main_context_iterate (context=0x23d4780, block=<value optimized out>,
dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#14 0x00007fc3fd0f07b5 in g_main_loop_run (loop=0x2c3b4e0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#15 0x00007fc3ff0423e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#16 0x00007fc400c15a28 in wxEventLoop::Run (this=0x2e2a640) at ../src/gtk/evtloop.cpp:76
#17 0x00007fc400ca5f18 in wxAppBase::MainLoop (this=0x23d41e0) at ../src/common/appcmn.cpp:312
#18 0x00007fc40051f025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>)
at ../src/common/init.cpp:448
#19 0x00000000005d0892 in main (argc=1, argv=0x7fc35c080020) at amule-gui.cpp:93
It seems the culprit is clear, However I'm afraid I don't have time to investigate right now, I hate being so busy, I want some fun.
-
Can you try to build wx yourself and see if that makes a difference?
-
No problem, I'll build wx. Some specific version?
Anyways IMVHO I think wx has nothing to do with that bug, it is a case similar to RRM, aMule cannot handle overload. I can see a correspondence about aMule's load and OOM occurences. Well, it could be the wx network code, I'm not familiar enough with it, but somehow aMule should control the rate of new connections.
In fact, right now I've dramatically increased the load (I have 1000 downloads) and aMule cannot be stable even for an hour. It helps I have know a much faster internet connection.
Unded that kind of load, amule either OOM or crashes, I'll modify my script in order to run aMule directly under GDB and stop it in case it starts eating memory.
Regards, BTK.
-
No problem, I'll build wx. Some specific version?
2.8.11 please.
Did you set your number of internet connections and you up/down speed to something your machine can handle, or did you set speed to zero ( == max) ?
-
Ok building 2.8.11. Anyways I guess the result will be identical to the binary Ubuntu version.
I always run a reasonable upload limit. Using a low download limit shortens the time to crash. Right now I'm running with upload limit set to 0.
I guess my machine can handle it ok (it is a pretty powerful machine), given than aMule runs in this setup about 100/200 minutes before going out of memory or crashing. However I didn't get yet a backtrace of the crashes, only for the OOM.
I'll report back.
-
Well it seems that disabling the download speed limit (i,e. setting it to 0) enhances aMule's stability. I'll let it running overnight and see what happens.
amule build with local wx is done, but I didn't replace yet my running one.
-
Please also apply the logging patch (http://forum.amule.org/index.php?topic=18543.msg100748#new).
-
Ok patch applied. I guess if another one should be used for the downloads.
Right now amule is running flawlessly with download limit set to zero and very high load (more than reasonable). I'll wait overnight and then I'll set up a download limit and see it that crashes amule. Given that previously (with the limit set) it was crashing very quickly and the backtrace showed an OOM when OnConnect, there must be some flaw in the way the upload throttlers handle the very high load.
-
Well I got bored after 10 hours of stable amule and did set up a download limit and guess what happened? amule crashed in 5 minutes.
The crash happens earlier given the difference of the limit with the real maximum possible download.
That is to say, my mule was downloading at 800Kb/s without limit, then if I setup the limit at 600 the crash will occur but take much to happen that if I set up the limit at 200Kb/s.
When amule got to 2Gb of RSS I got the following backtrace:
Program received signal SIGINT, Interrupt.
0x00007ffff5c21370 in __gnu_debug::_Safe_iterator_base::_M_get_mutex () from /usr/lib/libstdc++.so.6
(gdb) bt
#0 0x00007ffff5c21370 in __gnu_debug::_Safe_iterator_base::_M_get_mutex () from /usr/lib/libstdc++.so.6
#1 0x00007ffff5c2cc4e in __gnu_debug::_Safe_iterator_base::_M_detach() () from /usr/lib/libstdc++.so.6
#2 0x00000000004b268c in CDownloadQueue::CopyFileList (this=0x13c01e0, out_list=...,
includeCompleted=<value optimized out>) at DownloadQueue.cpp:193
#3 0x000000000063213e in CTransferWnd::UpdateCategory (this=<value optimized out>, index=4,
titleChanged=<value optimized out>) at TransferWnd.cpp:162
#4 0x00000000006322d0 in CTransferWnd::UpdateCatTabTitles (this=0x15dada0) at TransferWnd.cpp:404
#5 0x000000000064d64b in MuleNotify::HandleNotification (ntf=...) at GuiEvents.cpp:50
#6 0x000000000047e0ed in MuleNotify::DoNotify (func=<value optimized out>) at GuiEvents.h:393
#7 0x0000000000660a59 in CPartFile::Process (this=0x17abf10, reducedownload=<value optimized out>,
m_icounter=<value optimized out>) at PartFile.cpp:1574
#8 0x00000000004b638d in CDownloadQueue::Process (this=<value optimized out>) at DownloadQueue.cpp:445
#9 0x000000000044d8a7 in CamuleApp::OnCoreTimer (this=0xc68280) at amule.cpp:1176
#10 0x00007ffff6203fe0 in wxEvtHandler::ProcessEventIfMatches (entry=<value optimized out>, handler=0x7fffffffd5a0,
event=...) at ../src/common/event.cpp:1239
#11 0x00007ffff6205333 in wxEventHashTable::HandleEvent (this=<value optimized out>, event=..., self=0xc68280)
at ../src/common/event.cpp:906
#12 0x00007ffff62054a8 in wxEvtHandler::ProcessEvent (this=0xc68280, event=...) at ../src/common/event.cpp:1301
#13 0x00007ffff620451d in wxEvtHandler::ProcessPendingEvents (this=0xc68280) at ../src/common/event.cpp:1196
#14 0x00007ffff615e3e9 in wxAppConsole::ProcessPendingEvents (this=<value optimized out>)
at ../src/common/appbase.cpp:294
#15 0x00007ffff69241bb in wxAppBase::ProcessIdle (this=0x7fffffffd660) at ../src/common/appcmn.cpp:435
#16 0x00007ffff687a8e9 in wxapp_idle_callback () at ../src/gtk/app.cpp:213
#17 0x00007ffff2d6a342 in g_main_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#18 g_main_context_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#19 0x00007ffff2d6e2a8 in g_main_context_iterate (context=0xc68820, block=<value optimized out>,
dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#20 0x00007ffff2d6e7b5 in g_main_loop_run (loop=0x7fffe405b380) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#21 0x00007ffff4cc03e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#22 0x00007ffff6893a28 in wxEventLoop::Run (this=0x7fffe41054b0) at ../src/gtk/evtloop.cpp:76
#23 0x00007ffff6923f18 in wxAppBase::MainLoop (this=0xc68280) at ../src/common/appcmn.cpp:312
#24 0x00007ffff619d025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>)
at ../src/common/init.cpp:448
#25 0x00000000005d0892 in main (argc=1, argv=0x7fffffffd5a0) at amule-gui.cpp:93
(gdb) c
It is useless, but then I tried to continue and got a segfault :)
#0 0x00007ffff53e1ba5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007ffff53e56b0 in abort () at abort.c:92
#2 0x00007ffff6208d61 in wxFatalSignalHandler () at ../src/unix/utilsunx.cpp:1112
#3 <signal handler called>
#4 __pthread_mutex_lock (mutex=0x545345445f45435a) at pthread_mutex_lock.c:50
#5 0x00007ffff2d69b8a in g_source_unref_internal (source=0x7fffa943b870, context=0x545345445f454352, have_lock=0)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1502
#6 0x00007ffff2d617c0 in g_io_add_watch_full (channel=<value optimized out>, priority=<value optimized out>,
condition=<value optimized out>, func=0x7ffff4901950 <gdk_io_invoke>, user_data=0x7fffe4223260,
notify=0x7ffff4901930 <gdk_io_destroy>) at /build/buildd/glib2.0-2.26.1/glib/giochannel.c:671
#7 0x00007ffff49018e6 in IA__gdk_input_add_full (source=<value optimized out>, condition=0,
function=<value optimized out>, data=0x29ec2b0, destroy=0) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1129
#8 0x00007ffff68953b6 in GSocketGUIFunctionsTableConcrete::Install_Callback (this=<value optimized out>, socket=
0x29ec2b0, event=<value optimized out>) at ../src/gtk/gsockgtk.cpp:98
#9 0x00007ffff64b167e in GSocket::Write (this=0x29ec2b0, buffer=<value optimized out>, size=<value optimized out>)
at ../src/unix/gsocket.cpp:1257
#10 0x00007ffff64ad388 in wxSocketBase::_Write (this=0x38856e0, buffer=0x1f66290, nbytes=46)
at ../src/common/socket.cpp:539
#11 0x00007ffff64ad4dc in wxSocketBase::Write (this=0x545345445f45435a, buffer=0x1, nbytes=1)
at ../src/common/socket.cpp:507
#12 0x00000000006818ce in CSocketClientProxy::Write (this=0x38856e0, buffer=0x1f66290, nbytes=46) at Proxy.cpp:1309
#13 0x00000000004e3d08 in CEncryptedStreamSocket::Write (this=0x38856e0, lpBuf=0x1f66290, nBufLen=46)
at EncryptedStreamSocket.cpp:210
#14 0x00000000004e0ec2 in CEMSocket::Send (this=0x38856e0, maxNumberOfBytesToSend=<value optimized out>,
minFragSize=<value optimized out>, onlyAllowedToSendControlPacket=<value optimized out>) at EMSocket.cpp:572
#15 0x00000000004999a2 in CEMSocket::SendControlData (this=0x545345445f45435a, maxNumberOfBytesToSend=1,
minFragSize=1) at EMSocket.h:69
#16 0x000000000048e21b in CClientTCPSocket::SendControlData (this=0x545345445f45435a, maxNumberOfBytesToSend=1,
overchargeMaxBytesToSend=1) at ClientTCPSocket.cpp:2124
#17 0x00000000005747b4 in UploadBandwidthThrottler::Entry (this=<value optimized out>)
at UploadBandwidthThrottler.cpp:381
#18 0x00007ffff6202dfa in wxThreadInternal::PthreadStart (thread=0x16d7c50) at ../src/unix/threadpsx.cpp:766
#19 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#20 0x00007ffff549492d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()
Umm interesting but the bug should be easy to reproduce. STU, just add lots of donwloads so you will download very fast and then set a small speed limit.
Now starting with my local wx version with the logging patch applied. BTW, I tried to look for DownloadBandwithTrottler and I found nothing :)
Regards,
BTK
-
OK, I can see I've got some more room to maneuver. I'll have to do some code reading while you are watching all your new porn. ;D
(Your first backtrace is pointing at a very silly piece of code btw.)
I'll get back to you when I find something. It may take a while.
-
(Your first backtrace is pointing at a very silly piece of code btw.)
I have a script that sends a INT signal when aMule reaches 2Gb of RSS, of course it is a random bt.
I'll get back to you when I find something. It may take a while.
Perfect, I'm also having a look. Keep in mind the all my statments are inferred from occassional crashes, so not very conclusive. Indeed it could be a build problem or whatever else.
-
Well, I'll keep all my posts in this thread.
Situation: svn amule (10424) + oomlog patch + local wx. All built with debug mode and optimization enabled. Very high load, capable of donwloading about 1Mb/s, limited to 100Kb/s
I get a Sigsev, normal memory consumption.
2011-01-10 23:52:33: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 215
2011-01-10 23:52:37: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 208
2011-01-10 23:52:40: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 3 TCQF 0 SO 15 EMS 220
2011-01-10 23:52:44: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 220
2011-01-10 23:52:47: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 214
2011-01-10 23:52:50: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 222
2011-01-10 23:52:54: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 230
2011-01-10 23:52:58: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 229
2011-01-10 23:53:02: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 212
2011-01-10 23:53:05: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 220
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:26: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 39 CQF 0 TCQ 1 TCQF 0 SO 15 EMS 215
2011-01-10 23:53:27: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 38 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 186
2011-01-10 23:53:29: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 3 CQF 10 TCQ 0 TCQF 0 SO 15 EMS 204
2011-01-10 23:53:32: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 4 TCQF 0 SO 15 EMS 204
2011-01-10 23:53:35: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 194
2011-01-10 23:53:39: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 3 TCQF 0 SO 15 EMS 212
2011-01-10 23:53:42: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 3 TCQF 1 SO 15 EMS 213
2011-01-10 23:53:45: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 220
2011-01-10 23:53:51: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 2 SO 15 EMS 219
2011-01-10 23:53:53: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 3 TCQF 5 SO 15 EMS 226
2011-01-10 23:53:56: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 3 CQF 0 TCQ 1 TCQF 12 SO 15 EMS 235
2011-01-10 23:53:59: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 8 CQF 23 TCQ 0 TCQF 31 SO 15 EMS 237
2011-01-10 23:54:03: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 1 TCQF 84 SO 15 EMS 242
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 243
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 243
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 244
2011-01-10 23:54:21: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 3 TCQF 124 SO 15 EMS 211
2011-01-10 23:54:23: UploadBandwidthThrottler.cpp(311): UBT: t 17 CQ 0 CQF 0 TCQ 1 TCQF 134 SO 15 EMS 211
2011-01-10 23:54:26: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 184 SO 15 EMS 207
2011-01-10 23:54:32: UploadBandwidthThrottler.cpp(311): UBT: t 6 CQ 0 CQF 0 TCQ 0 TCQF 194 SO 15 EMS 209
2011-01-10 23:54:35: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 302 SO 15 EMS 209
2011-01-10 23:54:38: UploadBandwidthThrottler.cpp(311): UBT: t 3 CQ 0 CQF 0 TCQ 0 TCQF 386 SO 15 EMS 195
2011-01-10 23:54:44: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 1 TCQF 460 SO 15 EMS 208
2011-01-10 23:54:45: UploadBandwidthThrottler.cpp(311): UBT: t 4 CQ 3 CQF 0 TCQ 1 TCQF 518 SO 15 EMS 205
2011-01-10 23:54:51: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 552 SO 15 EMS 220
2011-01-10 23:54:53: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 563 SO 15 EMS 227
2011-01-10 23:55:02: UploadBandwidthThrottler.cpp(311): UBT: t 9 CQ 0 CQF 0 TCQ 0 TCQF 695 SO 15 EMS 231
2011-01-10 23:55:02: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 695 SO 15 EMS 231
2011-01-10 23:55:06: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 705 SO 15 EMS 232
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 4 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 9 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:36: UploadBandwidthThrottler.cpp(311): UBT: t 14 CQ 0 CQF 0 TCQ 1 TCQF 732 SO 15 EMS 207
2011-01-10 23:55:38: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 759 SO 15 EMS 156
2011-01-10 23:55:44: UploadBandwidthThrottler.cpp(311): UBT: t 11 CQ 0 CQF 0 TCQ 0 TCQF 829 SO 15 EMS 163
(gdb) bt
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabb0, socket=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/gtk/gsockgtk.cpp:119
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2b2eda0) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2b2eda0, source=82, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67ed0, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1cb17a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x1c9ca80) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67930) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67930) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56dd0) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
#18 0x00000000005d04c2 in main (argc=1, argv=0x1) at amule-gui.cpp:93
(gdb) bt full
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
No locales.
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
source = <value optimized out>
__PRETTY_FUNCTION__ = "g_source_remove"
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabb0, socket=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/gtk/gsockgtk.cpp:119
m_id = 0x27c6d00
c = 1
__PRETTY_FUNCTION__ = "virtual void GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent)"
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
No locales.
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2b2eda0) at ./src/unix/gsocket.cpp:1836
No locales.
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2b2eda0, source=82, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
socket = 0x2b2eda0
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
closure = 0x7fffe40f0fb0
gdk_cond = GDK_INPUT_WRITE
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
dispatch = 0x7ffff2a14a90 <g_io_unix_dispatch>
user_data = 0x7fffe40f0fb0
callback = 0x7ffff478e950 <gdk_io_invoke>
cb_funcs = 0x7ffff2c70610
cb_data = 0x7fffe40fe170
current_source_link = {data = 0x7fffe61c4170, next = 0x0}
source = 0x7fffe61c4170
current = 0x1771950
i = 45
#9 g_main_context_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
No locales.
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67ed0, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
max_priority = 300
timeout = 0
some_ready = 1
nfds = 99
allocated_nfds = -159537912
fds = <value optimized out>
__PRETTY_FUNCTION__ = "g_main_context_iterate"
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1cb17a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
self = 0xc67870
__PRETTY_FUNCTION__ = "g_main_loop_run"
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
tmp_list = 0x16be450
functions = 0x0
init = 0x7fffffffdfd0
loop = <value optimized out>
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x1c9ca80) at ./src/gtk/evtloop.cpp:76
__FUNCTION__ = "Run"
activate = {m_evtLoopOld = 0x0}
exitcode = 32767
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67930) at ./src/common/appcmn.cpp:312
mainLoop = {<wxEventLoopPtr> = {m_ptr = 0x1c9ca80}, m_pp = 0xc679a8, m_pOld = 0x0}
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67930) at ./src/common/appcmn.cpp:367
No locales.
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56dd0) at ./src/common/init.cpp:448
callOnExit = {<No data fields>}
---Type <return> to continue, or q <return> to quit---
initializer = {m_ok = true}
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
No locales.
#18 0x00000000005d04c2 in main (argc=1, argv=0x1) at amule-gui.cpp:93
-
No locales.(gdb) thread apply all bt
Hilo 4 (Thread 0x7fffeb6bc700 (LWP 9659)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:212
#1 0x00007ffff60dbebd in wxConditionInternal::WaitTimeout (this=0x1c9c5d0, milliseconds=100) at ./src/unix/threadpsx.cpp:405
#2 0x00007ffff60dee0f in wxCondition::WaitTimeout (this=0x1c8bc68, milliseconds=100) at ./include/wx/thrimpl.cpp:256
#3 0x00007ffff60dc427 in wxSemaphoreInternal::WaitTimeout (this=0x1c8bc60, milliseconds=100) at ./src/unix/threadpsx.cpp:552
#4 0x00007ffff60df191 in wxSemaphore::WaitTimeout (this=0x178f5d8, milliseconds=100) at ./include/wx/thrimpl.cpp:320
#5 0x00000000006b91dd in CTimerThread::Entry (this=0x178f5a0) at Timer.cpp:66
#6 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x178f5a0) at ./src/unix/threadpsx.cpp:766
#7 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x178f5a0) at ./src/unix/threadpsx.cpp:718
#8 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#9 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#10 0x0000000000000000 in ?? ()
Hilo 2 (Thread 0x7fffec6bf700 (LWP 9654)):
#0 0x00007ffff526ed3e in __libc_sigaction (sig=<value optimized out>, act=<value optimized out>, oact=0x7fffec6be400)
at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:67
#1 0x00007ffff526eb49 in __bsd_signal (sig=-328473600, handler=<value optimized out>) at ../sysdeps/posix/signal.c:49
#2 0x00007ffff63c3d45 in GSocket::Send_Stream (this=0x293d520, buffer=0x7fffdc3be670 "\234\016\b\204\256\004", size=6) at ./src/unix/gsocket.cpp:1684
#3 0x00007ffff63c2eae in GSocket::Write (this=0x293d520, buffer=0x7fffdc3be670 "\234\016\b\204\256\004", size=6) at ./src/unix/gsocket.cpp:1233
#4 0x00007ffff63bcfa5 in wxSocketBase::_Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at ./src/common/socket.cpp:539
#5 0x00007ffff63bcedd in wxSocketBase::Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at ./src/common/socket.cpp:507
#6 0x00000000006814ee in CSocketClientProxy::Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at Proxy.cpp:1309
#7 0x00000000004e3718 in CEncryptedStreamSocket::Write (this=0x2713e70, lpBuf=0x7fffdc3be670, nBufLen=6) at EncryptedStreamSocket.cpp:210
#8 0x00000000004e08a2 in CEMSocket::Send (this=0x2713e70, maxNumberOfBytesToSend=<value optimized out>, minFragSize=<value optimized out>,
onlyAllowedToSendControlPacket=<value optimized out>) at EMSocket.cpp:577
#9 0x0000000000499382 in CEMSocket::SendControlData (this=0xd, maxNumberOfBytesToSend=3966493520, minFragSize=3966493360) at EMSocket.h:69
#10 0x000000000048dbfb in CClientTCPSocket::SendControlData (this=0xd, maxNumberOfBytesToSend=3966493520, overchargeMaxBytesToSend=3966493360)
at ClientTCPSocket.cpp:2124
#11 0x00000000005741f4 in UploadBandwidthThrottler::Entry (this=<value optimized out>) at UploadBandwidthThrottler.cpp:401
#12 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x1563050) at ./src/unix/threadpsx.cpp:766
#13 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x1563050) at ./src/unix/threadpsx.cpp:718
#14 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#15 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#16 0x0000000000000000 in ?? ()
Hilo 1 (Thread 0x7ffff7fb4940 (LWP 9635)):
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabb0, socket=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/gtk/gsockgtk.cpp:119
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2b2eda0) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2b2eda0, source=82, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67ed0, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1cb17a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x1c9ca80) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67930) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67930) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56dd0) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
#18 0x00000000005d04c2 in main (argc=1, argv=0x1) at amule-gui.cpp:93
(gdb) thread apply all bt full
Hilo 4 (Thread 0x7fffeb6bc700 (LWP 9659)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:212
No locales.
#1 0x00007ffff60dbebd in wxConditionInternal::WaitTimeout (this=0x1c9c5d0, milliseconds=100) at ./src/unix/threadpsx.cpp:405
curtime = {m_ll = 1294700147551}
temp = {m_ll = 551}
sec = 1294700147
millis = 551
tspec = {tv_sec = 1294700147, tv_nsec = 551000000}
err = 32767
#2 0x00007ffff60dee0f in wxCondition::WaitTimeout (this=0x1c8bc68, milliseconds=100) at ./include/wx/thrimpl.cpp:256
__FUNCTION__ = "WaitTimeout"
#3 0x00007ffff60dc427 in wxSemaphoreInternal::WaitTimeout (this=0x1c8bc60, milliseconds=100) at ./src/unix/threadpsx.cpp:552
elapsed = {m_ll = 0}
remainingTime = 100
locker = {m_isOk = true, m_mutex = @0x1c8bc60}
startTime = {m_ll = 1294700147451}
#4 0x00007ffff60df191 in wxSemaphore::WaitTimeout (this=0x178f5d8, milliseconds=100) at ./include/wx/thrimpl.cpp:320
__FUNCTION__ = "WaitTimeout"
#5 0x00000000006b91dd in CTimerThread::Entry (this=0x178f5a0) at Timer.cpp:66
now = 4294966780
delta = 15271
evt = {<wxEvent> = {<wxObject> = {_vptr.wxObject = 0x7dfa30, static ms_classInfo = {m_className = 0x7ffff611acc8 L"wxObject", m_objectSize = 16,
m_objectConstructor = 0, m_baseInfo1 = 0x0, m_baseInfo2 = 0x0, static sm_first = 0x7ffff6ff9ba0, m_next = 0x7ffff6390980, static sm_classTable =
0xc27010}, m_refData = 0x0}, m_eventObject = 0x0, m_eventType = 10245, m_timeStamp = 0, m_id = 6128, m_callbackUserData = 0x0, m_propagationLevel = 0,
m_skipped = false, m_isCommandEvent = false, static ms_classInfo = {m_className = 0x7ffff61326c0 L"wxEvent", m_objectSize = 64, m_objectConstructor = 0,
m_baseInfo1 = 0x7ffff6390880, m_baseInfo2 = 0x0, static sm_first = 0x7ffff6ff9ba0, m_next = 0x7ffff6392880,
static sm_classTable = 0xc27010}}, <No data fields>}
lastEvent = 1914991355
#6 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x178f5a0) at ./src/unix/threadpsx.cpp:766
__clframe = {__cancel_routine = 0x7ffff60dc84f <wxPthreadCleanup(void*)>, __cancel_arg = 0x178f5a0, __do_it = 1, __cancel_type = 0}
pthread = 0xd1d5c0
rc = 0
dontRunAtAll = false
__FUNCTION__ = "PthreadStart"
#7 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x178f5a0) at ./src/unix/threadpsx.cpp:718
No locales.
#8 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
__res = <value optimized out>
pd = 0x7fffeb6bc700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737143097088, 2056249433585625392, 140737488346992, 140737143097792, 140737354125376, 3, -2056292213641959120,
-2056266445640685264}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#9 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#10 0x0000000000000000 in ?? ()
No symbol table info available.
Hilo 2 (Thread 0x7fffec6bf700 (LWP 9654)):
#0 0x00007ffff526ed3e in __libc_sigaction (sig=<value optimized out>, act=<value optimized out>, oact=0x7fffec6be400)
at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:67
result = <value optimized out>
kact = {k_sa_handler = 0x1, sa_flags = 335544320, sa_restorer = 0x7ffff526ec20 <__restore_rt>, sa_mask = {__val = {4096, 0 <repeats 15 times>}}}
koact = {k_sa_handler = 0x1, sa_flags = 335544320, sa_restorer = 0x7ffff526ec20 <__restore_rt>, sa_mask = {__val = {4096, 140737159882048, 22425832,
140737159882976, 5737169, 140736884375584, 140736884375672, 0, 140736946012096, 120, 140737306651560, 36, 128, 140736884375584, 133, 140736946012224}}}
#1 0x00007ffff526eb49 in __bsd_signal (sig=-328473600, handler=<value optimized out>) at ../sysdeps/posix/signal.c:49
act = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {4096, 0 <repeats 15 times>}}, sa_flags = 268435456,
---Type <return> to continue, or q <return> to quit---
sa_restorer = 0x15630e8}
oact = {__sigaction_handler = {sa_handler = 0x7ffff478e930 <gdk_io_destroy>, sa_sigaction = 0x7ffff478e930 <gdk_io_destroy>}, sa_mask = {__val = {
140737294952752, 0, 140737306653582, 140737055919760, 13008592, 0, 140737055919760, 140737266519648, 43696720, 0, 140737263762544, 140737055919664,
13008385, 1798918, 1798918, 140737294952784}}, sa_flags = 5724141, sa_restorer = 0x7fffec6be7b0}
#2 0x00007ffff63c3d45 in GSocket::Send_Stream (this=0x293d520, buffer=0x7fffdc3be670 "\234\016\b\204\256\004", size=6) at ./src/unix/gsocket.cpp:1684
old_handler = 0x7fffec6be7b0
ret = 0
#3 0x00007ffff63c2eae in GSocket::Write (this=0x293d520, buffer=0x7fffdc3be670 "\234\016\b\204\256\004", size=6) at ./src/unix/gsocket.cpp:1233
ret = 0
__PRETTY_FUNCTION__ = "int GSocket::Write(const char*, int)"
#4 0x00007ffff63bcfa5 in wxSocketBase::_Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at ./src/common/socket.cpp:539
total = 0
ret = -1
#5 0x00007ffff63bcedd in wxSocketBase::Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at ./src/common/socket.cpp:507
No locales.
#6 0x00000000006814ee in CSocketClientProxy::Write (this=0x2713e70, buffer=0x7fffdc3be670, nbytes=6) at Proxy.cpp:1309
lock = {m_isOk = true, m_mutex = @0x2713fb8}
#7 0x00000000004e3718 in CEncryptedStreamSocket::Write (this=0x2713e70, lpBuf=0x7fffdc3be670, nBufLen=6) at EncryptedStreamSocket.cpp:210
__FUNCTION__ = "Write"
#8 0x00000000004e08a2 in CEMSocket::Send (this=0x2713e70, maxNumberOfBytesToSend=<value optimized out>, minFragSize=<value optimized out>,
onlyAllowedToSendControlPacket=<value optimized out>) at EMSocket.cpp:577
tosend = 6
result = <value optimized out>
bWasLongTimeSinceSend = false
sentControlPacketBytesThisCall = 0
__FUNCTION__ = "Send"
lock = {m_isOk = true, m_mutex = @0x2714328}
anErrorHasOccured = false
sentStandardPacketBytesThisCall = 0
#9 0x0000000000499382 in CEMSocket::SendControlData (this=0xd, maxNumberOfBytesToSend=3966493520, minFragSize=3966493360) at EMSocket.h:69
No locales.
#10 0x000000000048dbfb in CClientTCPSocket::SendControlData (this=0xd, maxNumberOfBytesToSend=3966493520, overchargeMaxBytesToSend=3966493360)
at ClientTCPSocket.cpp:2124
returnStatus = {success = 192, sentBytesStandardPackets = 32767, sentBytesControlPackets = 3828357696}
#11 0x00000000005741f4 in UploadBandwidthThrottler::Entry (this=<value optimized out>) at UploadBandwidthThrottler.cpp:401
socketSentBytes = {success = false, sentBytesStandardPackets = 0, sentBytesControlPackets = 0}
socket = 0x2714290
slots = <value optimized out>
spentBytes = 0
spentOverhead = <value optimized out>
sendLock = {m_isOk = true, m_mutex = @0x1563070}
timeSinceLastLoop = <value optimized out>
minFragSize = 1300
doubleSendSize = 2600
sleepTime = <value optimized out>
thisLoopTick = 1914991441
bytesToSpend = <value optimized out>
extraSleepTime = 1
step = 46
lastLoopTick = 0
allowedDataRate = <value optimized out>
rememberedSlotCounter = 2
sendLock = {m_isOk = false, m_mutex = @0x7fffffffdb30}
#12 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x1563050) at ./src/unix/threadpsx.cpp:766
__clframe = {__cancel_routine = 0x7ffff60dc84f <wxPthreadCleanup(void*)>, __cancel_arg = 0x1563050, __do_it = 1, __cancel_type = 0}
pthread = 0x152a740
rc = 0
dontRunAtAll = false
__FUNCTION__ = "PthreadStart"
#13 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x1563050) at ./src/unix/threadpsx.cpp:718
No locales.
---Type <return> to continue, or q <return> to quit---
#14 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
__res = <value optimized out>
pd = 0x7fffec6bf700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737159886592, 2056249433585625392, 140737488345904, 140737159887296, 140737354125376, 3, -2056285613887837904,
-2056266445640685264}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#15 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#16 0x0000000000000000 in ?? ()
No symbol table info available.
Hilo 1 (Thread 0x7ffff7fb4940 (LWP 9635)):
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
No locales.
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
source = <value optimized out>
__PRETTY_FUNCTION__ = "g_source_remove"
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabb0, socket=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/gtk/gsockgtk.cpp:119
m_id = 0x27c6d00
c = 1
__PRETTY_FUNCTION__ = "virtual void GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent)"
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
No locales.
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2b2eda0) at ./src/unix/gsocket.cpp:1836
No locales.
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2b2eda0, source=82, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
socket = 0x2b2eda0
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
closure = 0x7fffe40f0fb0
gdk_cond = GDK_INPUT_WRITE
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
dispatch = 0x7ffff2a14a90 <g_io_unix_dispatch>
user_data = 0x7fffe40f0fb0
callback = 0x7ffff478e950 <gdk_io_invoke>
cb_funcs = 0x7ffff2c70610
cb_data = 0x7fffe40fe170
current_source_link = {data = 0x7fffe61c4170, next = 0x0}
source = 0x7fffe61c4170
current = 0x1771950
i = 45
#9 g_main_context_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
No locales.
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67ed0, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
max_priority = 300
timeout = 0
some_ready = 1
nfds = 99
allocated_nfds = -159537912
fds = <value optimized out>
__PRETTY_FUNCTION__ = "g_main_context_iterate"
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1cb17a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
self = 0xc67870
__PRETTY_FUNCTION__ = "g_main_loop_run"
---Type <return> to continue, or q <return> to quit---
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
tmp_list = 0x16be450
functions = 0x0
init = 0x7fffffffdfd0
loop = <value optimized out>
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x1c9ca80) at ./src/gtk/evtloop.cpp:76
__FUNCTION__ = "Run"
activate = {m_evtLoopOld = 0x0}
exitcode = 32767
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67930) at ./src/common/appcmn.cpp:312
mainLoop = {<wxEventLoopPtr> = {m_ptr = 0x1c9ca80}, m_pp = 0xc679a8, m_pOld = 0x0}
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67930) at ./src/common/appcmn.cpp:367
No locales.
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56dd0) at ./src/common/init.cpp:448
callOnExit = {<No data fields>}
initializer = {m_ok = true}
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
No locales.
#18 0x00000000005d04c2 in main (argc=1, argv=0x1) at amule-gui.cpp:93
No locales.
-
Another crash, same pattern, as soon as TCQF reached 800 it got a sigsev/sigabort.
Now switching to self-build amule using stock wx. I'm running amule inside gdb, next try will be running it without and watching the logger.
-
Hey, switching builds had the effect, I'm getting an OOM now! Same situation as before but with Ubuntu stock WX.
2011-01-11 00:33:50: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 232
2011-01-11 00:33:54: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 231
2011-01-11 00:33:58: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 1 TCQF 0 SO 16 EMS 231
2011-01-11 00:34:01: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 219
2011-01-11 00:34:04: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 241
2011-01-11 00:34:08: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 4 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 245
2011-01-11 00:34:12: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 5 CQF 0 TCQ 1 TCQF 0 SO 16 EMS 244
2011-01-11 00:34:16: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 243
2011-01-11 00:34:19: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 4 CQF 0 TCQ 1 TCQF 0 SO 16 EMS 243
2011-01-11 00:34:22: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 14 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 258
2011-01-11 00:34:25: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 263
2011-01-11 00:34:29: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 2 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 262
2011-01-11 00:34:41: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 5 TCQF 0 SO 16 EMS 257
2011-01-11 00:34:41: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 5 TCQF 0 SO 16 EMS 257
2011-01-11 00:34:41: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 5 TCQF 0 SO 16 EMS 257
2011-01-11 00:34:43: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 10 TCQF 0 SO 16 EMS 232
2011-01-11 00:34:46: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 232
2011-01-11 00:34:49: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 16 EMS 236
2011-01-11 00:34:53: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 28 SO 16 EMS 220
2011-01-11 00:34:56: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 95 SO 16 EMS 239
Umm, we see a pattern with TCQF here, but no so much as before and aMule is eating 2Gb of memory. Anyways, it seems if TCQF increases, something bad is bound to happen.
Now I have amule interrupted eating 2Gb of memory, I mean I may operate in GDB with it. The backtrace is the same as previous OOM.
(gdb) bt
#0 0x00007ffff7bc9c9d in __pthread_mutex_unlock_usercnt (mutex=0xc3d840) at pthread_mutex_unlock.c:52
#1 __pthread_mutex_unlock (mutex=0xc3d840) at pthread_mutex_unlock.c:290
#2 0x00007ffff6201066 in wxMutexInternal::Unlock (this=0xc3d840) at ../src/unix/threadpsx.cpp:297
#3 0x00007ffff64ada84 in wxSocketBase::OnRequest (this=0x20e2ba0, notification=wxSOCKET_OUTPUT) at ../src/common/socket.cpp:1006
#4 0x00007ffff64b242e in GSocket::Detected_Write (this=0x1fc6610) at ../src/unix/gsocket.cpp:1836
#5 0x00007ffff490199f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#6 0x00007ffff2d6a342 in g_main_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#7 g_main_context_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#8 0x00007ffff2d6e2a8 in g_main_context_iterate (context=0xc68820, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#9 0x00007ffff2d6e7b5 in g_main_loop_run (loop=0x1c9ade0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#10 0x00007ffff4cc03e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#11 0x00007ffff6893a28 in wxEventLoop::Run (this=0x1c8e2a0) at ../src/gtk/evtloop.cpp:76
#12 0x00007ffff6923f18 in wxAppBase::MainLoop (this=0xc68280) at ../src/common/appcmn.cpp:312
#13 0x00007ffff619d025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>) at ../src/common/init.cpp:448
#14 0x00000000005d0ae2 in main (argc=1, argv=0x0) at amule-gui.cpp:93
OnRequest, my friend. Running it a little more (until it eats some more memory) shows our friend
(gdb) bt
#0 _int_malloc (av=0x7ffff572ce40, bytes=48) at malloc.c:4709
#1 0x00007ffff542a38e in __libc_malloc (bytes=48) at malloc.c:3660
#2 0x00007ffff5c83ded in operator new(unsigned long) () from /usr/lib/libstdc++.so.6
#3 0x00007ffff6195cdb in wxObjectList::CreateNode (this=0x7fffe40010b0, prev=0x3, next=0x0, data=0x77537030, key=...) at ../include/wx/list.h:1185
#4 0x00007ffff61a9001 in wxListBase::Append (this=0x7fffe40010b0, object=0x77537030) at ../src/common/list.cpp:244
#5 0x00007ffff620466d in Append (this=0xc1e500, event=<value optimized out>) at ../include/wx/list.h:1185
#6 wxEvtHandler::AddPendingEvent (this=0xc1e500, event=<value optimized out>) at ../src/common/event.cpp:1156
#7 0x00007ffff64ada84 in wxSocketBase::OnRequest (this=0x20e2ba0, notification=wxSOCKET_OUTPUT) at ../src/common/socket.cpp:1006
#8 0x00007ffff64b242e in GSocket::Detected_Write (this=0x1fc6610) at ../src/unix/gsocket.cpp:1836
#9 0x00007ffff490199f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#10 0x00007ffff2d6a342 in g_main_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#11 g_main_context_dispatch (context=0xc68820) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#12 0x00007ffff2d6e2a8 in g_main_context_iterate (context=0xc68820, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#13 0x00007ffff2d6e7b5 in g_main_loop_run (loop=0x1c9ade0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#14 0x00007ffff4cc03e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#15 0x00007ffff6893a28 in wxEventLoop::Run (this=0x1c8e2a0) at ../src/gtk/evtloop.cpp:76
#16 0x00007ffff6923f18 in wxAppBase::MainLoop (this=0xc68280) at ../src/common/appcmn.cpp:312
#17 0x00007ffff619d025 in wxEntry (argc=<value optimized out>, argv=<value optimized out>) at ../src/common/init.cpp:448
#18 0x00000000005d0ae2 in main (argc=1, argv=0x3) at amule-gui.cpp:93
So we are adding a lot of events which cannot be served. This is why I'm againts of event based programming for everything.
And we finally end with a SIGSEGV :) nice and complete pack. The backtrace is identical to the previous one.
A wild guess, could the sockets or connections expire somehow in the queue and cause the SIGSEGV. I'm saying this because I find a correlation among the amount of time I stop amule inside a gdb and the crash happening. If I don't wait time, amule keeps eating memory and doesn't crash. Maybe some debug message for when a socket is deleted by timeout could be useful.
Well, I guess is hard to find something new, I'll study the code and play with debug options.
Regards,
BTK
-
Using svn wx -> the crash persists.
I did some malloc debugging and my guess is that the sockets get thousand of pending events, I'll have a closer look.
-
Indeed, with wx svn amule's behaviour is much worse, I'll try with a previous known wx version, something is wrong in the socket handling. (Well, everything, there's a reason no one uses event-based programming for sockets and networds, but poll and queues, and trying to cast epoll into events, uff, not a good idea.)
-
I did modify wx and amule in order to confirm my suspitions on where the memory is spent. Indeed I confirmed that memory is exhausted allocating new events like crazy for EMSockets.
I did modify STU patch in order to see total events for sockets (TEVS), see:
2011-01-12 20:01:12: UploadBandwidthThrottler.cpp(326): UBT: t 0 CQ 0 CQF 0 TCQ 1 TCQF 63 SO 16 EMS 226 LEMS 226 TEVS 0
2011-01-12 20:01:16: UploadBandwidthThrottler.cpp(326): UBT: t 2 CQ 14 CQF 0 TCQ 1 TCQF 128 SO 16 EMS 237 LEMS 237 TEVS 0
2011-01-12 20:01:19: UploadBandwidthThrottler.cpp(326): UBT: t 3 CQ 0 CQF 0 TCQ 1 TCQF 196 SO 16 EMS 229 LEMS 229 TEVS 0
2011-01-12 20:01:23: UploadBandwidthThrottler.cpp(326): UBT: t 4 CQ 0 CQF 0 TCQ 2 TCQF 277 SO 16 EMS 224 LEMS 224 TEVS 0
2011-01-12 20:01:27: UploadBandwidthThrottler.cpp(326): UBT: t 3 CQ 0 CQF 0 TCQ 2 TCQF 350 SO 16 EMS 224 LEMS 224 TEVS 0
2011-01-12 20:01:35: UploadBandwidthThrottler.cpp(326): UBT: t 5 CQ 0 CQF 0 TCQ 3 TCQF 376 SO 16 EMS 230 LEMS 230 TEVS 0
2011-01-12 20:01:35: UploadBandwidthThrottler.cpp(326): UBT: t 7 CQ 0 CQF 0 TCQ 3 TCQF 376 SO 16 EMS 230 LEMS 230 TEVS 0
2011-01-12 20:01:38: UploadBandwidthThrottler.cpp(326): UBT: t 6 CQ 0 CQF 0 TCQ 0 TCQF 447 SO 16 EMS 231 LEMS 231 TEVS 230
2011-01-12 20:13:25: UploadBandwidthThrottler.cpp(326): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 455 SO 16 EMS 231 LEMS 231 TEVS 45217650
2011-01-12 20:13:25: UploadBandwidthThrottler.cpp(326): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 455 SO 16 EMS 231 LEMS 231 TEVS 187067970
At this point I did stop amule, it was eating some 2Gb of RAM. See that we have more than 187 million events pending for 231 sockets. Not good. I guess a good understanding of wxSocketClient and friends is needed in order to avoid this.
Unfortunately I don't understand yet the complex handing of amule download limits so I cannot help more, but I'll try.
I did patch wx and amule, it you are interested say so and I'll post it.
-
Indeed, with wx svn amule's behaviour is much worse, I'll try with a previous known wx version, something is wrong in the socket handling.
Last time I tried wx SVN it even failed in Windows, where it worked before (better than in Linux at least). Kad was firewalled. Who knows what they screwed up again. Problem is we are probably the only wx app in existance using network in such a way.
Unfortunately I don't understand yet the complex handing of amule download limits so I cannot help more, but I'll try.
Welcome to the club. :(
The whole EMSocket stuff is nightmarish. Lot of verbatim eMule code, but with some deviations (which aren't explained). Virtual functions, several layers for crypto and proxy (is anybody using proxy?!?), and so on. I did fix the download limit function some time ago (because it was off roundly by factor two), but only had to fix a calculation, not go through the whole path.
Anyway, thanks to your work we have made a big progress from "runs out of memory" where we were some days before. We'll just keep beating it till it's dead. :)
-
Well for one forcing sockets to process all their pending events or disabling download in that cases could help. Unfortunately the event stuff is private to wxEvtHandler and wxSocket, so having a fine grain control over events is not possible in aMule.
Anyways we have a hint, sockets eat all available memory in pending events, and given that we (aMule) have no control whatsoever over events, we could make a case to wx developers in the sense that the library and event handling code shouldn't allow this to happen.
Regards,
BTK
-
Sounds like a job for... me?
-
Well, I'd very much like to see you showing off and fixing the bug with a "duh, I have to pamper everybody here". :)
-
Well, the obvious suspects to investigate are CEMSocket::OnReceive and specially CEMSocket::SetDownloadLimit.
-
My guess at what's going on:
Lets revisit how wx/glib work with sockets. When we create a CEMSocket, wx adds it to the list of glib's sockets to check. (see GSocketGUIFunctionsTableConcrete::Install_Callback). Modulo error handling, the callback calls Detected_{Read,Write}, functions which perform some basic error handling and call the callback. socket.cpp establishes an unique event handler (see wxSocketBase::OnRequest), which then checks the settable mask for event in wxSocketClient and proceeds to create the appropriate wx events unless they are masked.
Creating an event in wx is just appending it to a list, so nothing special happens here. Well, we have seen half the story, so glib is detecting that a socket is receiving data and wx is adding events to a list, causing an OOM condition. We must understand now why the created events are not served.
wx processes events using glib's idle handling. This means that the wx event handler will only run if glib runs its idle handlers. This means that no more events should be happening. I have checked that when amule starts eating all the available memory and no wx event handling is taking place, so what is causing the idle handler to not be run?
Indeed, in glib's main loop, sockets are checked for new data. If some socket has new data the above sequence will happen, which is pretty cpu intensive, as it must create a new event, (must call new, etc...). As soon as the rate of new data is great enough to use 100% CPU, boom!! wx event handling will not take place and pending events will eat all the available memory.
What has this to do with disabling the download limit helping the crash not to occur. Well, a lot. If we read from some socket but we don't eat all the available data, a new event will be generated by wx, kind of a spurious event given that a new event will be generated indicating there's data remaining. Ideally we'd like to have events only in the case new data was actually received. If we disable the limit, amule will read as much data as possible and indeed we'll get only "real" events when new network data arrives. Picture this, if we have 1000 bytes waiting in the socket's buffer and we read 1 byte each time, we will have to process 1000 events, although no new data arrived.
Umm, that's ugly, but what could be the fix? IMHO the only way to stop this behaviour is to force CEMSocket to mask socket events when it knows that there's more data in the buffer. This will fix the OOM, and wx event handling starvation should require much higher load.
Summary: Every time we don't read in full a socket buffer or new data arrives from network a new wx event is created. Download limit makes partial reads a frequent case. When the socket event handling manages to eat 100% CPU we can't process the wx events, so amule dies in OOM creating millions of unprocessable events.
This is my view, of course I'm not expert in glib/wx so take all my finding with a grain of salt, I could have missed something important. I'll keep working on that, I want to think more about the issue before writing a patch, of course ideas and comments are appreciated :) Don't hesitate to request for more information in case some of my writing is not clear.
Regards,
BTK
-
Very well explained btkaos, but I'm afraid you are wrong.
- There is a sigsegv that is only visible when amule runs in gdb. Up to this everything is fine, only after this aMule creates events like mad and eats up all the mem.
- Your theory says in short, network events are still created, but not processed anymore. This would mean incoming data (plus a little overhead) would be stored in RAM instead of being flushed to disk. Well, the oom happens within a very short time after things start going wrong. Memory is eaten up much faster than you could download. And even if you are on a high speed line - if processing stops, no new blocks are requested from sources, and everything comes to a screeching halt.
- Where does the influence of the Kernel change fit into the picture?
-
Well this is a proof of concept completely untested patch:
@@ -249,6 +265,10 @@ void CEMSocket::OnReceive(int nErrorCode)
// Update limit
if (ret >= downloadLimit) {
downloadLimit = 0;
+ // We are above the read limit, avoid getting inputs until next turn.
+ this->SetNotify(wxSOCKET_OUTPUT_FLAG |
+ wxSOCKET_CONNECTION_FLAG |
+ wxSOCKET_LOST_FLAG);
} else {
downloadLimit -= ret;
}
@@ -293,11 +313,16 @@ void CEMSocket::OnReceive(int nErrorCode)
void CEMSocket::SetDownloadLimit(uint32 limit)
{
downloadLimit = limit;
downloadLimitEnable = true;
// CPU load improvement
if(limit > 0 && pendingOnReceive == true){
- OnReceive(0);
+ // Reenable the socket
+ this->SetNotify(wxSOCKET_INPUT_FLAG |
+ wxSOCKET_OUTPUT_FLAG |
+ wxSOCKET_CONNECTION_FLAG |
+ wxSOCKET_LOST_FLAG);
+ OnReceive(0);
}
}
I will apply fuzzily due to all my modifications, but it should apply. I'm testing it right now :)
-
If it fails please try:
Add the following in wx/socket.h to wxSocketBase (as public):
void * GetGSocket() const { return m_socket; }
Add a log line in CEMSocket::Destroy() and print GetGSocket() (which is available for CEMSocket since it derives from wxSocketBase).
AddLogLineN(CFormat(wxT("Destroyed GSocket %p")) % GetGSocket());
When you get the next crash, check the log if the GSocket involved was destroyed recently.
-
- There is a sigsegv that is only visible when amule runs in gdb. Up to this everything is fine, only after this aMule creates events like mad and eats up all the mem.
I'm not seeing this sigsegv, and I'm running gdb all the time. I did see it but was due some bad interaction of connections getting dropped
Try it yourself, get a lot of downloads so you are able to download at very high speed, then limit amule to 10Kb/s download. See what happens.
- Your theory says in short, network events are still created, but not processed anymore. This would mean incoming data (plus a little overhead) would be stored in RAM instead of being flushed to disk.
My debugging shows that millions of events are being created and that no wx event processing takes place. The memory is being eaten by the events itself (see OnRequest in wx sources, socket.cpp and the code of AddPendingEvent). Indeed, a wxSocketEvent has quite a small memory footprint and uses not network data. The network data is store in kernel buffers.
Well, the oom happens within a very short time after things start going wrong. Memory is eaten up much faster than you could download.
Memory is eaten by wxSocketEvent objects. From my last crash: number of pending events: 1326118478.
And even if you are on a high speed line - if processing stops, no new blocks are requested from sources, and everything comes to a screeching halt.
Of course, when the OOM starts amule is not responsive, the app is trapped a loop. Indeed as you said, the OOM takes place in a very short time, it fills my 8Gb RAM in about 20/30 seconds. Indeed, log stops responding, etc...
Let me clarify that I have verified with debugging all what I said. What I'm not so sure is why glib never calls their idle handler anymore, I guess that is because 100% cpu use, but I'm not sure. However I have checked that this is what happens, my debugging shows g_main_iterate_context polling and creating new sockets events continuosly.
- Where does the influence of the Kernel change fit into the picture?
I can reproduce this with old kernels and all wx versions.
BTW, the patch doesn't work :(
-
If it fails please try:
Add the following in wx/socket.h to wxSocketBase (as public):
void * GetGSocket() const { return m_socket; }
Add a log line in CEMSocket::Destroy() and print GetGSocket() (which is available for CEMSocket since it derives from wxSocketBase).
AddLogLineN(CFormat(wxT("Destroyed GSocket %p")) % GetGSocket());
When you get the next crash, check the log if the GSocket involved was destroyed recently.
STU, I'm not getting crashes, but OOM.
-
Indeed, let me clarify that all my backtraces come from attaching gdb to a process going OOM. I have a "watcher" script to do this. I didn't get a crash "per se"
The crashes some other users seem to have could be attributed to two things:
- Glib/GTK are not safe with respect to out of memory conditions. In words of their main developer, handling gracefully OOM would have been "too complicated" so instead let the program crash. X windows can do this, and of course is very expensive, given that you have cover every call to a function which may allocate memory. Nevermind techniques to do that transparently exist from more than 30 years, but it seems some developers were not very well versed.
- When this particular bug happens, wx event handling won't run, so socket cleanup, and some code which relies on events start malfunctioning.
-
I get a Sigsev, normal memory consumption.
2011-01-10 23:52:33: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 215
2011-01-10 23:52:37: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 208
2011-01-10 23:52:40: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 3 TCQF 0 SO 15 EMS 220
2011-01-10 23:52:44: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 220
2011-01-10 23:52:47: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 214
2011-01-10 23:52:50: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 222
2011-01-10 23:52:54: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 230
2011-01-10 23:52:58: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 229
2011-01-10 23:53:02: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 212
2011-01-10 23:53:05: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 220
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:24: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 7 TCQF 0 SO 15 EMS 216
2011-01-10 23:53:26: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 39 CQF 0 TCQ 1 TCQF 0 SO 15 EMS 215
2011-01-10 23:53:27: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 38 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 186
2011-01-10 23:53:29: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 3 CQF 10 TCQ 0 TCQF 0 SO 15 EMS 204
2011-01-10 23:53:32: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 4 TCQF 0 SO 15 EMS 204
2011-01-10 23:53:35: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 2 TCQF 0 SO 15 EMS 194
2011-01-10 23:53:39: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 3 TCQF 0 SO 15 EMS 212
2011-01-10 23:53:42: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 3 TCQF 1 SO 15 EMS 213
2011-01-10 23:53:45: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 0 TCQF 0 SO 15 EMS 220
2011-01-10 23:53:51: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 2 SO 15 EMS 219
2011-01-10 23:53:53: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 0 CQF 0 TCQ 3 TCQF 5 SO 15 EMS 226
2011-01-10 23:53:56: UploadBandwidthThrottler.cpp(311): UBT: t 0 CQ 3 CQF 0 TCQ 1 TCQF 12 SO 15 EMS 235
2011-01-10 23:53:59: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 8 CQF 23 TCQ 0 TCQF 31 SO 15 EMS 237
2011-01-10 23:54:03: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 1 TCQF 84 SO 15 EMS 242
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 243
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 243
2011-01-10 23:54:17: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 106 SO 15 EMS 244
2011-01-10 23:54:21: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 3 TCQF 124 SO 15 EMS 211
2011-01-10 23:54:23: UploadBandwidthThrottler.cpp(311): UBT: t 17 CQ 0 CQF 0 TCQ 1 TCQF 134 SO 15 EMS 211
2011-01-10 23:54:26: UploadBandwidthThrottler.cpp(311): UBT: t 1 CQ 0 CQF 0 TCQ 0 TCQF 184 SO 15 EMS 207
2011-01-10 23:54:32: UploadBandwidthThrottler.cpp(311): UBT: t 6 CQ 0 CQF 0 TCQ 0 TCQF 194 SO 15 EMS 209
2011-01-10 23:54:35: UploadBandwidthThrottler.cpp(311): UBT: t 2 CQ 0 CQF 0 TCQ 0 TCQF 302 SO 15 EMS 209
2011-01-10 23:54:38: UploadBandwidthThrottler.cpp(311): UBT: t 3 CQ 0 CQF 0 TCQ 0 TCQF 386 SO 15 EMS 195
2011-01-10 23:54:44: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 1 TCQF 460 SO 15 EMS 208
2011-01-10 23:54:45: UploadBandwidthThrottler.cpp(311): UBT: t 4 CQ 3 CQF 0 TCQ 1 TCQF 518 SO 15 EMS 205
2011-01-10 23:54:51: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 552 SO 15 EMS 220
2011-01-10 23:54:53: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 563 SO 15 EMS 227
2011-01-10 23:55:02: UploadBandwidthThrottler.cpp(311): UBT: t 9 CQ 0 CQF 0 TCQ 0 TCQF 695 SO 15 EMS 231
2011-01-10 23:55:02: UploadBandwidthThrottler.cpp(311): UBT: t 7 CQ 0 CQF 0 TCQ 0 TCQF 695 SO 15 EMS 231
2011-01-10 23:55:06: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 705 SO 15 EMS 232
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 4 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 5 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 9 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:33: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 724 SO 15 EMS 228
2011-01-10 23:55:36: UploadBandwidthThrottler.cpp(311): UBT: t 14 CQ 0 CQF 0 TCQ 1 TCQF 732 SO 15 EMS 207
2011-01-10 23:55:38: UploadBandwidthThrottler.cpp(311): UBT: t 8 CQ 0 CQF 0 TCQ 0 TCQF 759 SO 15 EMS 156
2011-01-10 23:55:44: UploadBandwidthThrottler.cpp(311): UBT: t 11 CQ 0 CQF 0 TCQ 0 TCQF 829 SO 15 EMS 163
(gdb) bt
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabb0, socket=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/gtk/gsockgtk.cpp:119
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2b2eda0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2b2eda0) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2b2eda0, source=82, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67ed0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67ed0, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1cb17a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x1c9ca80) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67930) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67930) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56dd0) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
#18 0x00000000005d04c2 in main (argc=1, argv=0x1) at amule-gui.cpp:93
I didn't get a crash "per se"
???
-
For the fun of it - does it have an influence if you disable "show extended info on category tabs" ?
-
Yes I'm aware of this post but quicly saw that this was an exceptional instance. I guess I should have posted about it. Note that this was an isolated crash.
Now I understand the problem, look at the value of the glib's context. Indeed it seems like an overflow of events glib's loop or something like that:
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffe6398db0, context=0xffffffffffffffff, have_lock=0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
static void
g_source_destroy_internal (GSource *source,
GMainContext *context,
gboolean have_lock)
{
if (!have_lock)
LOCK_CONTEXT (context);
Boom here, as context is incorrect. As context is derived from source this seems like a glib bug (not the first time) or some bad interaction of wveventloop not running. Anyways the context returned by g_main_context_default was very wrong here.
I didn't get a crash for the last days.
Sorry for the confusion, STU. I guess you should try to reproduce it. Get a lots of sources and limit the download.
Regards,
BTK
-
For the fun of it - does it have an influence if you disable "show extended info on category tabs" ?
Ok I'll try it, I see how that setting may interact weirly.
Patch I'm trying right now, changing line 307 of app.cpp from
wxTheApp->m_idleTag = g_idle_add_full(G_PRIORITY_LOW, wxapp_idle_callback, NULL, NULL);
wxTheApp->m_idleTag = g_idle_add_full(G_PRIORITY_HIGH, wxapp_idle_callback, NULL, NULL);
This should make the event loop of wx higher priority than the events coming from sockets. Too early to tell if it works.
-
Patch I'm trying right now, changing line 307 of app.cpp from
wxTheApp->m_idleTag = g_idle_add_full(G_PRIORITY_LOW, wxapp_idle_callback, NULL, NULL);
wxTheApp->m_idleTag = g_idle_add_full(G_PRIORITY_HIGH, wxapp_idle_callback, NULL, NULL);
This should make the event loop of wx higher priority than the events coming from sockets. Too early to tell if it works.
Well, as I suspected that change didn't work, it starves the rest of the application with the timer events once aMule eats a lot of CPU.
Rebuilding wx and I'll try STU suggestion next.
-
btkaos, I don't believe the problem is the events are not processed. Problem is rather they are generated at an insane, inexplicable rate.
Your patch disabling events didn't work. Does it work if you disable events altogether using Notify(false) instead of SetNotify() ?
-
btkaos, I don't believe the problem is the events are not processed. Problem is rather they are generated at an insane, inexplicable rate.
I have confirmed that once the rate of events starts going up, wx event handing is starved. I started suspecting an event problem when AddLogLineN stopped working. You may believe it, but I have experimental data that shows otherwise.
Once you reproduce the bug, stop aMule when it has just eaten some hundredths of Mb. Then place a breakpoint in wxapp_idle_callback. You'll see that it is never called.
You may want to add some printf to wxapp_idle_callback instead. Keep in mind that AddLogLine doesn't work once the bug shows, as we have no event processing.
Your patch disabling events didn't work. Does it work if you disable events altogether using Notify(false) instead of SetNotify() ?
With that patch we will lose loss of connection events and so on, so it is not feasible. The problem is not how to disable events, but where and when. Anyways given the nature of the problem a more intrusive fix may be needed. It would be great if some developer could reproduce it. So far we have 3 users confirming that disabling the donwload limit solves the OOM.
-
Problem is rather they are generated at an insane, inexplicable rate.
I believe I did explain that. Lots of data is coming and aMule doesn't read it because of download limit. Also if we read from a socket and there's data remaining another event will occur.
What is difficult to understand is why glib generates events for a socket if there's an event pending to be handled. Well, wx always accepts it, that's the problem. So from glib POV the event has been handled, from wx POV not yet, but glib feels free to emit another event. Once we reach an event rate enough to starve wx event processing, BOOM.
It seems the bug is then in wx socket code. IMHO when accepting a glib event, wx should tell glib to not report that kind of events again. When the wx event is served, it should enable it again. This is standard in event based programming.
-
BTW, disabling "show extended info on category tabs" didn't have any effect.
-
I am having an awesome weekend, but I will go back home monday and check this out.
Promise.
-
Just a note, wxSockets in svn is very different than it is now, although it seems to suffer from the same problem
Kry, it may be a wxWidget bug or an amule bug, depends on whether it is fixable from amule.
I'd say it is not going to be fixable from within aMule, of course I'm not sure at all. Given the state of wx svn writing a patch may prove difficult until wx change rate slows down.
-
More info for debug. I modified wx event handler to print once a second if it is called. You can see the moment it is never called back.
The other output is all the EMSocket we have with the first number being the download limit of the socket, between braces the number of events the socket has pending.
0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 2 [0] 14 [0] 0 [0] 1024 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 1024 [0] 0 [0] 10 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 32 [0] 0 [0] 49 [0] 27 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 54 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 79 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 1024 [0] 0 [0] 1024 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] 0 [0] ---------------------------------------------------
wx event Callback!!!
wx event Callback!!!
wx event Callback!!!
0 [0] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 2 [1] 13 [1] 0 [1] 1024 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 1024 [1] 0 [1] 10 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 31 [1] 0 [1] 49 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 75 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 1024 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] 0 [1] ---------------------------------------------------
TEVS 164
wx event Callback!!!
wx event Callback!!!
0 [0] 0 [66470] 0 [66470] 0 [66470] 0 [66470] 0 [66472] 0 [66472] 2 [66472] 13 [66472] 0 [66474] 1024 [66474] 0 [66474] 0 [66474] 0 [66476] 0 [66476] 0 [66476] 0 [66476] 0 [66478] 0 [66478] 0 [66478] 0 [66478] 0 [66480] 0 [66480] 0 [66480] 0 [66480] 1024 [66482] 0 [66482] 10 [66482] 0 [66482] 0 [66484] 0 [66484] 0 [66484] 0 [66484] 31 [66486] 0 [66486] 49 [66486] 0 [66486] 0 [66488] 0 [66488] 0 [66490] 0 [66490] 0 [66490] 0 [66490] 0 [66492] 0 [66492] 0 [66492] 0 [66492] 0 [66494] 0 [66494] 0 [66494] 0 [66494] 0 [66494] 0 [66496] 0 [66496] 0 [66496] 0 [66496] 0 [66496] 0 [66498] 0 [66498] 0 [66498] 0 [66498] 0 [66498] 0 [66500] 0 [66500] 0 [66500] 0 [66500] 0 [66502] 0 [66502] 0 [66502] 0 [66502] 0 [66504] 0 [66504] 0 [66504] 0 [66504] 0 [66506] 0 [66506] 0 [66506] 0 [66506] 0 [66507] 0 [66508] 0 [66508] 0 [66508] 0 [66509] 0 [66510] 0 [66510] 0 [66510] 0 [66511] 0 [66512] 0 [66512] 0 [66512] 0 [66513] 0 [66514] 0 [66514] 0 [66514] 0 [66515] 0 [66516] 0 [66516] 0 [66516] 0 [66517] 0 [66518] 0 [66518] 0 [66518] 0 [66519] 0 [66520] 0 [66520] 0 [66520] 0 [66521] 0 [66522] 0 [66522] 0 [66522] 0 [66523] 0 [66524] 0 [66524] 0 [66524] 0 [66524] 0 [66526] 0 [66526] 0 [66526] 0 [66526] 0 [66528] 0 [66528] 0 [66528] 0 [66528] 0 [66530] 0 [66530] 0 [66530] 0 [66530] 0 [66532] 0 [66532] 0 [66532] 0 [66533] 0 [66534] 0 [66534] 0 [66534] 0 [66535] 0 [66536] 0 [66536] 0 [66536] 0 [66537] 0 [66538] 0 [66538] 1024 [66538] 0 [66539] 0 [66540] 0 [66540] 0 [66540] 0 [66541] 0 [66542] 0 [66542] 0 [66542] 0 [66542] 0 [66544] 0 [66544] 0 [66544] 0 [66544] 0 [66545] 0 [66546] 0 [66546] 0 [66546] 0 [66547] 0 [66548] 0 [66548] 0 [66548] 0 [66549] 0 [66550] 0 [66550] 0 [66550] 0 [66551] 0 [66552] 0 [66552] 0 [66552] 0 [66553] 0 [66554] 0 [66554] 0 [66554] 0 [66555] 0 [66556] ---------------------------------------------------
TEVS 11697450
0 [0] 0 [434748] 0 [434749] 0 [434752] 0 [434752] 0 [434753] 0 [434756] 2 [434756] 13 [434756] 0 [434760] 1024 [434760] 0 [434760] 0 [434764] 0 [434764] 0 [434764] 0 [434766] 0 [434768] 0 [434768] 0 [434770] 0 [434772] 0 [434772] 0 [434773] 0 [434776] 0 [434776] 0 [434776] 1024 [434777] 0 [434780] 10 [434780] 0 [434780] 0 [434783] 0 [434784] 0 [434784] 0 [434784] 31 [434788] 0 [434788] 49 [434788] 0 [434792] 0 [434792] 0 [434792] 0 [434796] 0 [434796] 0 [434796] 0 [434800] 0 [434800] 0 [434800] 0 [434802] 0 [434804] 0 [434804] 0 [434804] 0 [434808] 0 [434808] 0 [434808] 0 [434812] 0 [434812] 0 [434812] 0 [434816] 0 [434816] 0 [434816] 0 [434819] 0 [434820] 0 [434820] 0 [434821] 0 [434824] 0 [434824] 0 [434828] 0 [434828] 0 [434828] 0 [434832] 0 [434832] 0 [434832] 0 [434836] 0 [434836] 0 [434836] 0 [434838] 0 [434840] 0 [434840] 0 [434840] 0 [434844] 0 [434844] 0 [434844] 0 [434848] 0 [434848] 0 [434848] 0 [434852] 0 [434852] 0 [434852] 0 [434855] 0 [434856] 0 [434856] 0 [434857] 0 [434860] 0 [434860] 0 [434860] 0 [434864] 0 [434864] 0 [434864] 0 [434868] 0 [434868] 0 [434868] 0 [434872] 0 [434872] 0 [434872] 0 [434874] 0 [434876] 0 [434876] 0 [434876] 0 [434880] 0 [434880] 0 [434880] 0 [434884] 0 [434884] 0 [434884] 0 [434888] 0 [434888] 0 [434888] 0 [434892] 0 [434892] 0 [434892] 0 [434892] 0 [434896] 0 [434896] 0 [434896] 0 [434900] 0 [434900] 0 [434900] 0 [434904] 0 [434904] 0 [434904] 0 [434908] 0 [434908] 0 [434908] 0 [434910] 0 [434912] 0 [434912] 0 [434912] 0 [434916] 0 [434916] 0 [434916] 0 [434920] 0 [434920] 0 [434920] 1024 [434924] 0 [434924] 0 [434924] 0 [434928] 0 [434928] 0 [434928] 0 [434929] 0 [434932] 0 [434932] 0 [434932] 0 [434936] 0 [434936] 0 [434936] 0 [434940] 0 [434940] 0 [434940] 0 [434944] 0 [434944] 0 [434944] 0 [434946] 0 [434948] 0 [434948] 0 [434948] 0 [434952] 0 [434952] 0 [434952] 0 [434956] 0 [434956] 0 [434956] 0 [434960] 0 [434960] 0 [434960] 0 [434964] 0 [434964] 0 [434964] 0 [434965] ---------------------------------------------------
TEVS 76513101
0 [0] 0 [1139230] 0 [1139230] 0 [1139235] 0 [1139241] 0 [1139241] 0 [1139241] 2 [1139246] 13 [1139252] 0 [1139252] 1024 [1139252] 0 [1139256] 0 [1139263] 0 [1139263] 0 [1139263] 0 [1139267] 0 [1139273] 0 [1139274] 0 [1139274] 0 [1139276] 0 [1139284] 0 [1139285] 0 [1139285] 0 [1139288] 0 [1139292] 1024 [1139296] 0 [1139296] 10 [1139296] 0 [1139303] 0 [1139307] 0 [1139307] 0 [1139307] 0 [1139312] 31 [1139318] 0 [1139318] 49 [1139318] 0 [1139324] 0 [1139329] 0 [1139329] 0 [1139329] 0 [1139332] 0 [1139340] 0 [1139340] 0 [1139340] 0 [1139344] 0 [1139350] 0 [1139351] 0 [1139351] 0 [1139353] 0 [1139361] 0 [1139362] 0 [1139362] 0 [1139363] 0 [1139370] 0 [1139373] 0 [1139373] 0 [1139374] 0 [1139381] 0 [1139384] 0 [1139384] 0 [1139384] 0 [1139390] 0 [1139395] 0 [1139395] 0 [1139395] 0 [1139397] 0 [1139403] 0 [1139406] 0 [1139406] 0 [1139407] 0 [1139415] 0 [1139417] 0 [1139417] 0 [1139418] 0 [1139424] 0 [1139428] 0 [1139428] 0 [1139428] 0 [1139435] 0 [1139439] 0 [1139439] 0 [1139439] 0 [1139445] 0 [1139450] 0 [1139450] 0 [1139450] 0 [1139454] 0 [1139461] 0 [1139461] 0 [1139461] 0 [1139466] 0 [1139472] 0 [1139472] 0 [1139472] 0 [1139475] 0 [1139483] 0 [1139483] 0 [1139483] 0 [1139486] 0 [1139491] 0 [1139494] 0 [1139494] 0 [1139494] 0 [1139502] 0 [1139505] 0 [1139505] 0 [1139506] 0 [1139511] 0 [1139516] 0 [1139516] 0 [1139516] 0 [1139522] 0 [1139527] 0 [1139527] 0 [1139527] 0 [1139531] 0 [1139538] 0 [1139538] 0 [1139538] 0 [1139543] 0 [1139548] 0 [1139549] 0 [1139549] 0 [1139552] 0 [1139560] 0 [1139560] 0 [1139560] 0 [1139563] 0 [1139569] 0 [1139571] 0 [1139571] 0 [1139572] 0 [1139581] 0 [1139582] 0 [1139582] 0 [1139582] 0 [1139590] 0 [1139593] 0 [1139593] 0 [1139593] 0 [1139600] 1024 [1139604] 0 [1139604] 0 [1139604] 0 [1139610] 0 [1139615] 0 [1139615] 0 [1139615] 0 [1139620] 0 [1139626] 0 [1139626] 0 [1139626] 0 [1139631] 0 [1139637] 0 [1139637] 0 [1139637] 0 [1139641] 0 [1139648] 0 [1139648] 0 [1139648] 0 [1139652] 0 [1139659] 0 [1139659] 0 [1139661] 0 [1139669] 0 [1139670] 0 [1139670] 0 [1139672] 0 [1139678] 0 [1139681] 0 [1139681] 0 [1139682] 0 [1139690] 0 [1139692] 0 [1139692] 0 [1139692] 0 [1139699] ---------------------------------------------------
TEVS 200497642
0 [0] 0 [2220620] 0 [2220621] 0 [2220626] 0 [2220633] 0 [2220637] 0 [2220639] 2 [2220639] 13 [2220639] 0 [2220647] 1024 [2220654] 0 [2220658] 0 [2220658] 0 [2220658] 0 [2220666] 0 [2220672] 0 [2220677] 0 [2220677] 0 [2220677] 0 [2220685] 0 [2220691] 0 [2220696] 0 [2220696] 0 [2220696] 0 [2220703] 1024 [2220709] 0 [2220715] 10 [2220715] 0 [2220715] 0 [2220722] 0 [2220728] 0 [2220734] 0 [2220734] 31 [2220734] 0 [2220740] 49 [2220746] 0 [2220753] 0 [2220753] 0 [2220753] 0 [2220759] 0 [2220765] 0 [2220772] 0 [2220772] 0 [2220772] 0 [2220777] 0 [2220782] 0 [2220790] 0 [2220791] 0 [2220791] 0 [2220795] 0 [2220801] 0 [2220809] 0 [2220810] 0 [2220810] 0 [2220813] 0 [2220819] 0 [2220827] 0 [2220829] 0 [2220829] 0 [2220832] 0 [2220838] 0 [2220846] 0 [2220848] 0 [2220848] 0 [2220850] 0 [2220856] 0 [2220865] 0 [2220867] 0 [2220867] 0 [2220869] 0 [2220875] 0 [2220884] 0 [2220886] 0 [2220886] 0 [2220886] 0 [2220889] 0 [2220895] 0 [2220903] 0 [2220905] 0 [2220905] 0 [2220907] 0 [2220913] 0 [2220922] 0 [2220924] 0 [2220924] 0 [2220926] 0 [2220932] 0 [2220940] 0 [2220943] 0 [2220943] 0 [2220944] 0 [2220950] 0 [2220959] 0 [2220962] 0 [2220962] 0 [2220963] 0 [2220969] 0 [2220978] 0 [2220981] 0 [2220981] 0 [2220982] 0 [2220988] 0 [2220997] 0 [2221000] 0 [2221000] 0 [2221000] 0 [2221007] 0 [2221015] 0 [2221019] 0 [2221019] 0 [2221019] 0 [2221026] 0 [2221034] 0 [2221038] 0 [2221038] 0 [2221038] 0 [2221043] 0 [2221051] 0 [2221057] 0 [2221057] 0 [2221057] 0 [2221061] 0 [2221070] 0 [2221076] 0 [2221076] 0 [2221076] 0 [2221080] 0 [2221088] 0 [2221094] 0 [2221095] 0 [2221095] 0 [2221099] 0 [2221107] 0 [2221111] 0 [2221114] 0 [2221114] 0 [2221116] 0 [2221124] 0 [2221130] 0 [2221133] 0 [2221133] 1024 [2221135] 0 [2221143] 0 [2221147] 0 [2221152] 0 [2221152] 0 [2221152] 0 [2221160] 0 [2221166] 0 [2221171] 0 [2221171] 0 [2221171] 0 [2221179] 0 [2221185] 0 [2221190] 0 [2221190] 0 [2221190] 0 [2221198] 0 [2221204] 0 [2221209] 0 [2221209] 0 [2221209] 0 [2221217] 0 [2221223] 0 [2221228] 0 [2221228] 0 [2221228] 0 [2221236] 0 [2221242] 0 [2221247] 0 [2221247] 0 [2221247] 0 [2221255] 0 [2221261] 0 [2221266] 0 [2221266] 0 [2221266] ---------------------------------------------------
TEVS 390822299
All this is weird. But at least you may see where the memory is spent and when wx event handling is starved.
Below is a backtrace of event creation. How silly it calls WakeUpIdle! You can see that the cause is GDK_INPUT_WRITE event.
(gdb) bt
#0 0x00007ffff67da26c in wxApp::WakeUpIdle (this=0x7ffff602465d) at ./src/gtk/app.cpp:139
#1 0x00007ffff602465d in wxWakeUpIdle () at ./src/common/appbase.cpp:647
#2 0x00007ffff60e0f69 in wxEvtHandler::AddPendingEvent (this=0xc1e620, event=...) at ./src/common/event.cpp:1164
#3 0x00007ffff63bdb40 in wxSocketBase::OnRequest (this=0x1a1d8f0, notification=wxSOCKET_OUTPUT) at ./src/common/socket.cpp:1006
#4 0x00007ffff63bd94f in wx_socket_callback (notification=GSOCK_OUTPUT, cdata=0x1a1d8f0 "\260Xv") at ./src/common/socket.cpp:942
#5 0x00007ffff63c4323 in GSocket::Detected_Write (this=0x1a1df60) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fcb1c in _GSocket_GDK_Input (data=0x1a1df60, source=65, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67f10, block=<value optimized out>, dispatch=<value optimized out>,
self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x1681110) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67fab5f in wxEventLoop::Run (this=0x167f110) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896dc8 in wxAppBase::MainLoop (this=0xc67970) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896f42 in wxAppBase::OnRun (this=0xc67970) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56e10) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe0bc, argv=0x7fffffffe1a8) at ./src/common/init.cpp:460
#18 0x00000000005d0b82 in main (argc=1, argv=0x0) at amule-gui.cpp:93
Just a note, aMule is running, just stopped in gdb
0 1000 3592 3590 20 0 2892712 2623280 ptrace tl pts/1 1:33 /usr/local/bin/amule
-
LOG analysis, for some reason, it seems like glib went crazy and started firing events without control.
Ops, I forgot, download limit is set to 10Kbps.
-
My patches to wx-2.8.11 and amule in case anyone wants to investigate.
-
BTW, the patch doesn't work :(
I have modified your patch, please retry.
I have changed the way events are reenabled and added a flag for events disabled, that's the part that counts. The socket counting is also in the patch. You can keep your other debug stuff of course.
-
Ok trying the patch, however I don't yet understand fully how the download limiting works. I guess I disabled the events in the wrong place.
Note that this seems like it should be fixed in wxwidgets, waiting for them to finish the overhaul and i'll send a bug.
-
With your patch again a corrupted context, damm glib this should never happen. Anyways I did see some interesting things, but I have to double check them.
(gdb) bt
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffdc2fe130, context=0xffffffffffffffff, have_lock=0)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabd0, socket=0x2218aa0, event=GSOCK_OUTPUT)
at ./src/gtk/gsockgtk.cpp:119
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2218aa0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2218aa0) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2218aa0, source=19, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67f10, block=<value optimized out>, dispatch=<value optimized out>,
self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x7fffe43371a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x7fffe4396a40) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67970) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67970) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56e10) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe09c, argv=0x7fffffffe188) at ./src/common/init.cpp:460
#18 0x00000000005d0b02 in main (argc=1, argv=0x1) at amule-gui.cpp:93
-
Well the patch didn't help, an OOM again, caused by zillions of wxSocketEvent queued.
Anyways glib's behaviour is really weird here, what could corrupt glib's context. The code involved seems to be thread safe (was my number 1 suspicion). Anyways glib/gtk was once infamous as single-threaded toolkits, and indeed gtk used to have very embarrassing limitations regarding threading.
Maybe the context shouldn't be manipulated from different threads, but it doesn't seem so. Who could overwrite the context?
It is only accessed one time, and it is surrounded by:
G_LOCK_DEFINE_STATIC (main_loop);
static GMainContext *default_main_context;
static GSList *main_contexts_without_pipe = NULL;
Regards,
BTK
-
Just for reference:
:~/tmp/amule$ objdump -x /usr/lib/debug/lib/libglib-2.0.so.0.2600.1 | grep 2e0b
00000000002e0b28 l O .bss 0000000000000004 counter.11490
00000000002e0b44 l O .bss 0000000000000008 child_watch_wake_up_pipe
00000000002e0b40 l O .bss 0000000000000004 child_watch_init_state
00000000002e0b98 l O .bss 0000000000000004 depth_private.9210
00000000002e0b9c l O .bss 0000000000000004 thread_context_stack
00000000002e0b60 l O .bss 0000000000000030 g__main_context_list_lock
00000000002e0b90 l O .bss 0000000000000008 main_context_list
00000000002e0bd0 l O .bss 0000000000000008 main_contexts_without_pipe
00000000002e0ba0 l O .bss 0000000000000030 g__main_loop_lock
00000000002e0bd8 l O .bss 0000000000000008 default_main_context
00000000002e0bf0 l O .bss 0000000000000008 profile_data
00000000002e0bf8 l O .bss 0000000000000008 profile_allocs
00000000002e0be8 l O .bss 0000000000000008 gmem_profile_mutex
00000000002e0be0 g O .bss 0000000000000004 g_mem_gc_friendly
-
I don't yet understand fully how the download limiting works.
You don't read everything from the socket in case of an event, only as much as needed to fill your bandwidth. Then, in the next cycle, you read again (you don't need another event for this).
Well the patch didn't help, an OOM again, caused by zillions of wxSocketEvent queued.
I was afraid of that. If the events aren't processed anymore, OnReceive isn't called and event generation isn't turned off.
What wx should do:
- two flags in socket for "read event posted", "write evenet posted"
- flag is set true when event is posted
- flag is set false on Read/Write
- when flag is true no Read/Write event is posted.
Guess you can hack it in. That might cure it.
-
I don't yet understand fully how the download limiting works.
You don't read everything from the socket in case of an event, only as much as needed to fill your bandwidth. Then, in the next cycle, you read again (you don't need another event for this).
Note that indeed wx *will* generate another event in that case.
From http://docs.wxwidgets.org/stable/wx_wxsocketbase.html:
The wxSOCKET_INPUT event will be issued whenever there is data available for reading. This will be the case if the input queue was empty and new data arrives, or if the application has read some data yet there is still more data available. This means that the application does not need to read all available data in response to a wxSOCKET_INPUT event, as more events will be produced as necessary.
What my patch tried to do is to disable those events, as you said aMule knows that it has data available. Picture this, aMule reads from the socket but more data is available, wx generates another event, then aMule receives the event but reads no data as it exhausted the limit, another event is generated (maybe), etc...
Of course we also have the wx
Well the patch didn't help, an OOM again, caused by zillions of wxSocketEvent queued.
I was afraid of that. If the events aren't processed anymore, OnReceive isn't called and event generation isn't turned off.
What wx should do:
- two flags in socket for "read event posted", "write evenet posted"
- flag is set true when event is posted
- flag is set false on Read/Write
- when flag is true no Read/Write event is posted.
Guess you can hack it in. That might cure it.
Yes basically we must tell glib not to generate more events until the wx event is processed. This is missing now. That is easy and no flag is needed, when receiving the glib event tell glib to mask that event, and in the wx event handler reenable it unconditionally. The event handler must assume than it is always called with the underlying socket masked for that operation. Similar to asyncronous interrupt processing, the interruption manager that particular interruption and the IH enables it again.
I was going to write a patch, but note that the wxSockets is completely different from 2.8 to 2.9, so I don't want to target 2.9 yet given that it seems very unstable.
Maybe I'll write one for wxSockets 2.8, just for checking that it solves the problem. I wouldn't rule out a bug in glib, the crash points to that.
-
I don't have a working linux installed, I found out today. So, tomorrow I'll reinstall one and test all this.
You crash sounds like one I found a long time ago in wxWidgets that I didn't get to backport to them because I lost a harddrive. If that's it, I'll surely be able to fix it again.
As for the events... we'll see.
-
With your patch again a corrupted context, damm glib this should never happen. Anyways I did see some interesting things, but I have to double check them.
(gdb) bt
#0 __pthread_mutex_lock (mutex=0x7) at pthread_mutex_lock.c:50
#1 0x00007ffff29d004f in g_source_destroy_internal (source=0x7fffdc2fe130, context=0xffffffffffffffff, have_lock=0)
at /build/buildd/glib2.0-2.26.1/glib/gmain.c:942
#2 0x00007ffff29d290e in g_source_remove (tag=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1717
#3 0x00007ffff67fcc7d in GSocketGUIFunctionsTableConcrete::Uninstall_Callback (this=0x7ffff6ccabd0, socket=0x2218aa0, event=GSOCK_OUTPUT)
at ./src/gtk/gsockgtk.cpp:119
#4 0x00007ffff63c37ef in GSocket::Disable (this=0x2218aa0, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1527
#5 0x00007ffff63c42f8 in GSocket::Detected_Write (this=0x2218aa0) at ./src/unix/gsocket.cpp:1836
#6 0x00007ffff67fca1c in _GSocket_GDK_Input (data=0x2218aa0, source=19, condition=GDK_INPUT_WRITE) at ./src/gtk/gsockgtk.cpp:41
#7 0x00007ffff478e99f in gdk_io_invoke (source=<value optimized out>, condition=<value optimized out>, data=<value optimized out>)
at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1082
#8 0x00007ffff29d0342 in g_main_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2149
#9 g_main_context_dispatch (context=0xc67f10) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2702
#10 0x00007ffff29d42a8 in g_main_context_iterate (context=0xc67f10, block=<value optimized out>, dispatch=<value optimized out>,
self=<value optimized out>) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2780
#11 0x00007ffff29d47b5 in g_main_loop_run (loop=0x7fffe43371a0) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:2988
#12 0x00007ffff4b4d3e7 in IA__gtk_main () at /build/buildd/gtk+2.0-2.22.0/gtk/gtkmain.c:1237
#13 0x00007ffff67faa5f in wxEventLoop::Run (this=0x7fffe4396a40) at ./src/gtk/evtloop.cpp:76
#14 0x00007ffff6896cc8 in wxAppBase::MainLoop (this=0xc67970) at ./src/common/appcmn.cpp:312
#15 0x00007ffff6896e42 in wxAppBase::OnRun (this=0xc67970) at ./src/common/appcmn.cpp:367
#16 0x00007ffff606ae5b in wxEntry (argc=@0x7ffff638f4b0, argv=0xc56e10) at ./src/common/init.cpp:448
#17 0x00007ffff606af31 in wxEntry (argc=@0x7fffffffe09c, argv=0x7fffffffe188) at ./src/common/init.cpp:460
#18 0x00000000005d0b02 in main (argc=1, argv=0x1) at amule-gui.cpp:93
Well the backtrace is strange but is is almost clear what is going on.
In gsocket.cpp:1836 is a usual callback for detected data operation.
CALL_CALLBACK(this, GSOCK_OUTPUT);
CALL_CALLBACK then calls to Disable which in turn just calls to g_source_remove. Why g_source_remove fails with a valid socket is indeed a cause of mistery. Some kind of overflow?
-
Picture this, aMule reads from the socket but more data is available, wx generates another event, then aMule receives the event but reads no data as it exhausted the limit, another event is generated (maybe), etc...
Since we read nothing in this case, no other event should be generated.
Did you notice how all of your sockets simultaneously started piling up events? Looks like something seriously breaks in the underlying system.
What I keep thinking is, maybe we should kick out all of the wx socket usage (along with the underlying GTK networking) and use some other socket lib. Maybe Boost.Asio. It would also solve (some of) the problems with 2.9 . I seriously doubt wx devs test against aMule or anything else driving a socket lib to its limits. They probably send a few packets from A to B and think it works.
(expecting some serious flaming for this... :-\ )
-
Picture this, aMule reads from the socket but more data is available, wx generates another event, then aMule receives the event but reads no data as it exhausted the limit, another event is generated (maybe), etc...
Since we read nothing in this case, no other event should be generated.
We may read very few bytes (2 bytes) or something like that and another event will pop up.
Did you notice how all of your sockets simultaneously started piling up events? Looks like something seriously breaks in the underlying system.
Yes, I'm testing a patch that should solve the socket issue and I'm getting the very strange glib context corruption crashes. It may well be that Ubuntu glib's is broken. I'll build glib/gtk from scratch (this is going to be a lot of work :) ) and see. Could be a compiler bug.
For now, it would be really helpful if someone could reproduce the bug. All you need is add some files so your download speed is very high, then limit it to 10Kbps. For quick reproduction, the bigger is the possible download speed faster aMule will crash.
Specially interesting would be to know if previous ubuntu versions or other linux distributions. Some Fedora or Suse feedback in this case would be great.
What I keep thinking is, maybe we should kick out all of the wx socket usage (along with the underlying GTK networking) and use some other socket lib. Maybe Boost.Asio. It would also solve (some of) the problems with 2.9 . I seriously doubt wx devs test against aMule or anything else driving a socket lib to its limits. They probably send a few packets from A to B and think it works.
(expecting some serious flaming for this... :-\ )
I completely agree, but that's because IMHO gtk is a library which leaves a lot to be desired and very unprofessional, at least when compared to high quality libraries like qt. wx is so so.
Anyways I wouldn't blame wx or amule yet, the bug could well be in the kernel, glib, libc, who knows. We need more testers who are able to reproduce the bug. Switching to Boost is a lot of work.
-
This crash is interesting. The context gets corrupted again, this time to the weird value of 0x45445f454352554f. Google says nothing for this value.
Really really weird and fishy, this couldn't if the code is compiled correctly. I guess the event overflow is just another instance of the bug, the context is corrupted in such a way that glib gets in an infinite loop sending events. As you noted all sockets receive the same number of events.
I'd forget for now about blaming wxEventHandling (though theoretically it could be starved in the manner I described) .
(gdb) bt full
#0 __pthread_mutex_lock (mutex=0x45445f4543525557) at pthread_mutex_lock.c:50
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#1 0x00007ffff29ceec0 in g_source_set_callback_indirect (source=0x7fffdc2b4160, callback_data=0x7fffdc0ce920,
callback_funcs=0x7ffff2c70610) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1149
context = 0x45445f454352554f
old_cb_data = <value optimized out>
old_cb_funcs = <value optimized out>
__PRETTY_FUNCTION__ = "g_source_set_callback_indirect"
#2 0x00007ffff29c77ac in g_io_add_watch_full (channel=<value optimized out>, priority=0, condition=<value optimized out>,
func=0x7ffff478e950 <gdk_io_invoke>, user_data=0x7fffdc062e20, notify=0x7ffff478e930 <gdk_io_destroy>)
at /build/buildd/glib2.0-2.26.1/glib/giochannel.c:668
source = 0x7fffdc2b4160
id = <value optimized out>
__PRETTY_FUNCTION__ = "g_io_add_watch_full"
#3 0x00007ffff478e8e6 in IA__gdk_input_add_full (source=<value optimized out>, condition=0, function=<value optimized out>,
data=0x25f9940, destroy=0) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1129
result = <value optimized out>
channel = 0x7fffdc2b40e0
cond = 12
#4 0x00007ffff67fcbb1 in GSocketGUIFunctionsTableConcrete::Install_Callback (this=0x7ffff6ccabd0, socket=0x25f9940, event=GSOCK_OUTPUT)
at ./src/gtk/gsockgtk.cpp:98
m_id = 0x23773b0
c = 1
#5 0x00007ffff63c379d in GSocket::Enable (this=0x25f9940, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1521
No locales.
#6 0x00007ffff63c2f1f in GSocket::Write (this=0x25f9940, buffer=0x7fffe41e9840 "\3176\246", size=6) at ./src/unix/gsocket.cpp:1257
ret = -1
__PRETTY_FUNCTION__ = "int GSocket::Write(const char*, int)"
#7 0x00007ffff63bcfa5 in wxSocketBase::_Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at ./src/common/socket.cpp:539
total = 0
ret = -1
#8 0x00007ffff63bcedd in wxSocketBase::Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at ./src/common/socket.cpp:507
No locales.
#9 0x0000000000681b8e in CSocketClientProxy::Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at Proxy.cpp:1310
lock = {m_isOk = true, m_mutex = @0x1da8e88}
#10 0x00000000004e3bf8 in CEncryptedStreamSocket::Write (this=0x1da8d40, lpBuf=0x7fffe41e9840, nBufLen=6) at EncryptedStreamSocket.cpp:210
__FUNCTION__ = "Write"
#11 0x00000000004e0ad2 in CEMSocket::Send (this=0x1da8d40, maxNumberOfBytesToSend=<value optimized out>,
minFragSize=<value optimized out>, onlyAllowedToSendControlPacket=<value optimized out>) at EMSocket.cpp:611
tosend = 6
result = <value optimized out>
bWasLongTimeSinceSend = false
sentControlPacketBytesThisCall = 0
__FUNCTION__ = "Send"
lock = {m_isOk = true, m_mutex = @0x1da9200}
anErrorHasOccured = false
sentStandardPacketBytesThisCall = 0
#12 0x000000000049959f in CEMSocket::SendFileAndControlData (this=0x45445f4543525557, maxNumberOfBytesToSend=3691833632, minFragSize=1)
at EMSocket.h:71
No locales.
#13 0x000000000048dd5b in CClientTCPSocket::SendFileAndControlData (this=0x45445f4543525557, maxNumberOfBytesToSend=3691833632,
overchargeMaxBytesToSend=1) at ClientTCPSocket.cpp:2138
returnStatus = {success = true, sentBytesStandardPackets = 0, sentBytesControlPackets = 18}
#14 0x000000000057481d in UploadBandwidthThrottler::Entry (this=<value optimized out>) at UploadBandwidthThrottler.cpp:480
socketSentBytes = {success = true, sentBytesStandardPackets = 0, sentBytesControlPackets = 0}
data = 69633
socket = 0x45445f4543525557
slotCounter = 18
slots = <value optimized out>
spentBytes = 0
spentOverhead = <value optimized out>
sendLock = {m_isOk = true, m_mutex = @0x18d6820}
timeSinceLastLoop = <value optimized out>
minFragSize = 1300
doubleSendSize = 2600
sleepTime = <value optimized out>
thisLoopTick = 2600814829
bytesToSpend = <value optimized out>
extraSleepTime = 1000
step = 45
lastLoopTick = 0
allowedDataRate = <value optimized out>
rememberedSlotCounter = 15
sendLock = {m_isOk = 200, m_mutex = @0x7ffff7fcca48}
#15 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x18d6800) at ./src/unix/threadpsx.cpp:766
__clframe = {__cancel_routine = 0x7ffff60dc84f <wxPthreadCleanup(void*)>, __cancel_arg = 0x18d6800, __do_it = 1, __cancel_type = 0}
pthread = 0x16a6320
rc = 0
dontRunAtAll = false
__FUNCTION__ = "PthreadStart"
#16 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x18d6800) at ./src/unix/threadpsx.cpp:718
No locales.
#17 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
__res = <value optimized out>
pd = 0x7fffec6bf700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737159886592, -7788523305851881642, 140737488345856, 140737159887296,
140737354125376, 3, 7788486849677817686, 7788540326228588374}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0},
data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#18 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#19 0x0000000000000000 in ?? ()
No symbol table info available.
-
I built the whole stack: git glib, gtk 2.22.1 (gtk in git is gtk3 and proved to be too difficult to build with wx) an wx 2.8.11.
Nothing changed :( I got an OOM, waiting so check for a crash.
-
I am having this issue too. So ran aMule through GDB and just when it started eating all my RAM I paused aMule.
By that time 3 aMule threads were living: the main application loop, the timer running in the background and the uploadbandwith-throttler.
So I switched between the three and made them run step-by-step. The main application thread was the one allocating all my RAM.
-
I am having this issue too. So ran aMule through GDB and just when it started eating all my RAM I paused aMule.
By that time 3 aMule threads were living: the main application loop, the timer running in the background and the uploadbandwith-throttler.
So I switched between the three and made them run step-by-step. The main application thread was the one allocating all my RAM.
Yes, it is called from glib main loop. I think we already know the cause of the OOM. glib main loops generates a event for a socket watch, this calls wx callbacks and queues and event.
This sequence is never stopped for all socket, so wx event handling won't never run, resulting in a OOM.
Right now I'm pretty much sure that this kind of event-spree is due to a corruption of glib's main context (which sometimes causes a crash) but I'm not able to see what causes it, it is very difficult to stop amule *just* before the problem.
-
btkaos, at last I found my old patch, which was never applied to wx trunk branches for various reasons.
Please test attached patch over an up-to-date, clean version of the branch 2.8 of wxWidgets, then tell me if you see a different beaviour, or even more problems! I adapted it from a different branch, so I might have messed up somewhere.
Edit (Stu): This is not the patch that fixes the problem!
-
Ok thanks, I'll try, but not today.
-
YOU WILL TRY WHENEVER I DAMN WELL TELL YOU TO.
I mean, take your time.
-
Any news?
-
Trying the patch right now. Interesting thing it that amule did crash with low activity and download limit, but it took much longer.
-
Sadly the patch had no effect, amule got stuck again in an endless glib loop of event creation.
Regards,
BTK
-
Funny. Give me a couple more days to continue investigating.
For the record - I haven't been able to reproduce it locally, so I'm just fixing blindly. You have any SURE way to reproduce it? I know it's somewhere in the thread, but... can you give me the exact specs (distro, version, kernel, window manager, number of CPUs and versions of relevant libraries) so I can try and clone them locally?
-
For the record - I haven't been able to reproduce it locally, so I'm just fixing blindly. You have any SURE way to reproduce it? I know it's somewhere in the thread, but... can you give me the exact specs (distro, version, kernel, window manager, number of CPUs and versions of relevant libraries) so I can try and clone them locally?
A current Ubuntu should do, (some users reported hapenning on Fedora too)
Just add whatever enough downloads so you will have big download speed, then limit it to, say 10 Kbps. Wait a while and aMule should crash/OOM.
The important point is to manage to have more download speed available than the limit for a bit of time. Let me know how it goes.
-
Any luck reproducing the bug?
So far I've only seen reports from users, but no dev could reproduce it. I can't completely eliminate we are doing something weird.
It would be really helpful if another dev could reproduce the bug and confirm it is reproducible indeed.
Regards,
BTK
-
It would be really helpful if another dev could reproduce the bug and confirm it is reproducible indeed.
Setting the download limit to 0 doesn't really solve the issue on my side. Instead it seems to limit the probability of occuring. Ever since the OOM issue arised I kept the download limit at 0 and even so it still crashes from time to time. On much fewer occasions, but still it does.
-
It would be really helpful if another dev could reproduce the bug and confirm it is reproducible indeed.
Setting the download limit to 0 doesn't really solve the issue on my side. Instead it seems to limit the probability of occurring. Ever since the OOM issue arised I kept the download limit at 0 and even so it still crashes from time to time. On much fewer occasions, but still it does.
Well that's one sample but it seems not the exact same bug, as I couldn't reproduce the bug when DL limit == 0. Still waiting for ^marcell^ to send me the exact conditions for reproducing that case.
I'll be more comfortable if either STU or Kry could reproduce it also.
-
One question, are you using 32 or 64bits ubuntu?
-
Ok, you have reaaaaaaaaally long addresses, so I'll assume 64.
-
Ok, you have reaaaaaaaaally long addresses, so I'll assume 64.
Correct, running Ubuntu 10.10 64bits:
Linux xxx 2.6.35-26-generic #46-Ubuntu SMP Sun Jan 30 06:59:07 UTC 2011 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 10.10
Release: 10.10
Codename: maverick
-
It happens with 32 too.
-
This is a weird bug Kry, in my testing the one behaving badly is glib, AFAICS amule does nothing wrong. glib gets in an infinite loop. The point is that a thing called "glib main context" gets corrupted. Of course neither wx or amule know that it even exists. How that can happen is really weird, as it is a global variable in glib.
Puzzling.
-
The thing is, I have actually seen it in a test application for a different bug I'm working on fixing inside wxWidgets/wxSocket. Infinite triggering of INPUT events from GTK. But I haven't been able to reproduce it in aMule yet.
I wouldn't worry too much about the glib main context being corrupted, to be honest with you. Who fucking knows what happens when the buildup of events starts breaking havok everywhere.
We may be looking at a glib/gtk bug, or a wxWidgets bug, or a combination of the two. But isn't it fun to investigate?
-
btkaos, I fixed another wx bug that was causing GTK events to be fired multiple times under certain conditions, or even after the socket is destroyed (causing a crash).
It's not exactly your problem, but please tell me if it makes a difference for you. Do not use my old patch from the previous post. Just apply this one, and tell me if it makes any difference.
The patch is GTK-specific and may break the ABI, so please recompile the entire wxWidgets after patching.
-
Ok testing it now, in some minutes we'll know.
-
Looking good if it is running and hasn't crashed yet. :-\
-
Well, I let it run for one hour and it didn't crash (the cpu usage the donwload limit caused was huge) :)
Now I switched back to the unpatched library and amule did crash in 5 minutes, did another run and it went oom quite fast (it took about 15 minutes).
So far the patch seems to solve (or hide) the problem quite well, another question is how to propagate the fix to distribution users. Anyways I will test the patch for the whole before declaring it OK.
Now we should understand why glib failed without the patch, I can't tell whether glib or wx is to blame. From my understanding of glib code g_source_destroy should be thread safe, although this UNLOCK_CONTEXT in g_source_destroy_internal seems very suspicious.
if (old_cb_funcs)
{
UNLOCK_CONTEXT (context);
old_cb_funcs->unref (old_cb_data);
LOCK_CONTEXT (context);
}
-
Anyways I will test the patch for the whole before declaring it OK.
whole night I mean. BTW, why can't I edit my posts anymore?
-
Editing was disabled so that people can't slip in spam later. There was a time limit of like 20 min involved first which Kry has disabled it seems - can't say if intentionally.
-
You're a moderator now, you can edit posts. Use it carefully as you can edit other people's posts. Please don't, unless necessary.
There are a couple of things that wx without patch could cause, one of them is removing the same event token twice, which is obviously wrong. I don't know why that could cause problems. Also, two threads could call gtk_* functions at the same time, causing a crash as well.
The patch is not ready to be distributed to anyone. It has to be changed to wxMutex instead of pthread, and made compatible with the other ports to fix across the board. Once I ahve that done and in wx's svn, we can think of pestering distributions.
-
There are a couple of things that wx without patch could cause, one of them is removing the same event token twice, which is obviously wrong. I don't know why that could cause problems. Also, two threads could call gtk_* functions at the same time, causing a crash as well.
The patch is not ready to be distributed to anyone. It has to be changed to wxMutex instead of pthread, and made compatible with the other ports to fix across the board. Once I ahve that done and in wx's svn, we can think of pestering distributions.
Well after some small analysis I think both wx and glib are buggy. Under no circumstance should glib corrupt its context, and that indeed can happen if you look at the snippet of code I posted before. Of course if wx removes the same event token twice this is also a wx bug. Anyways the bug needs more investigation.
You're a moderator now, you can edit posts. Use it carefully as you can edit other people's posts. Please don't, unless necessary.
Ah ok thanks :) I don't mind doing some forum housekeeping in the crash/backtraces subforum like merging and moving topics, etc... but of course only if you allow it.
-
wx 2.8.12 is scheduled for the end of the month, so I'll try to squeeze the change in.
-
of course only if you allow it.
You're a moderator now
;)
-
Finally - thanks to btkaos's script - here attached you find a full backtrace with downloadlimit set to 0 on a linux machine.
-
Finally - thanks to btkaos's script - here attached you find a full backtrace with downloadlimit set to 0 on a linux machine.
Hi ^marcell^, the backtrace looks similar to some of the one I got, but it is not informative as it doesn't show the allocation path.
No symbol table info available.
Would you mind to use debug versions of the libraries (in Debian those are the -dbg stuff and --with-wxdebug in amule's config)
Maybe next time you should press "c" and "Ctrl-C" in gdb to get the backtrace of the allocation path.
The bug was not exclusive of donwloadlimit, that just made it more likely to happen. Have you tried the wx patch? Did it help?
-
Would you mind to use debug versions of the libraries (in Debian those are the -dbg stuff and --with-wxdebug in amule's config)
Will do so.
Have you tried the wx patch? Did it help?
I am linking with wx svn and do not have a "gsockgtk.cpp" file. So the patch is of no use to me.
-
Have you tried the wx patch? Did it help?
I am linking with wx svn and do not have a "gsockgtk.cpp" file. So the patch is of no use to me.
Ok, sorry, I didn't realize this.
wx svn has a full socket rewrite and indeed needs a completely different patch.
Also in my own limited testing wx svn had some other issues that made it unusable with amule. I plan to try again once wx can be compiled with GTK 3.
-
wx 2.8.12 is scheduled for the end of the month, so I'll try to squeeze the change in.
What month?
-
Just a note, it seems that high loads may keep causing problems [see RRM's thread], although Kry patchs helps a lot.
-
Fix commited for the imminent wxWidgets 2.8.12, see http://trac.wxwidgets.org/ticket/13055
-
I'm pretty sure I'm getting the exact same issue
amule runs fine then all of a sudden it start using memory quickly until it crashed
I'm using current stable Genoo to create this issue
I have experienced the same issue on 3 different machines
Your saying if I update to 2.8.12 it should solve this issue?
-
Your saying if I update to 2.8.12 it should solve this issue?
Yes.
There is other related crash that is yet unsolved, but it is much harder to trigger.
-
I'm either extremely lucky or the wx update fixed the problem