Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[alex-m1234]

I've been following the other forum posts about amule consuming all memory and crashing recently.  The crash is caused something that changed in either amule, cryptopp wxwidgets or something.  I have been experiencing the following:
1) amule starts normally
2) amule runs for a while (1 or 2 hours)
3) amule starts consuming all memory
4) amule starts consuming all swap
5) system is unresponsive
6) eventually the kernel kills the amule process
7) system is back to normal.

The following is the memory usage of amule during that time:

Code: [Select]

  PID USER      PR  NI  VIRT  RES  SHR S %CPU   %MEM    TIME+  COMMAND                                         
 5662 alex      20   0  499m  58m  23m S   4.8   0.7   1:53.31 amule                                           
... [steady normally running state for 1-2 hours]...
 5662 alex      20   0  499m  58m  23m S   4.8   0.7   1:53.31 amule                                           
 5662 alex      20   0  499m  58m  23m S   4.8   0.7   1:53.55 amule                                           
 5662 alex      20   0  512m  71m  23m R  18.3   0.9   1:55.28 amule                                           
 5662 alex      20   0  798m 357m  23m R 100.4   4.5   2:00.32 amule                                         
 5662 alex      20   0 1344m 903m  23m R 100.5  11.3   2:05.36 amule                                         
 5662 alex      20   0 1889m 1.4g  23m R 100.5  18.1   2:10.40 amule                                         
 5662 alex      20   0 2438m 2.0g  23m R 100.5  25.0   2:15.44 amule                                         
 5662 alex      20   0 2984m 2.5g  23m R 100.7  31.8   2:20.49 amule                                         
 5662 alex      20   0 3596m 3.1g  23m R 100.3  39.4   2:25.52 amule                                         
 5662 alex      20   0 4236m 3.7g  23m R 100.5  47.4   2:30.56 amule                                         
 5662 alex      20   0 4831m 4.3g  23m R  98.4  54.9   2:35.50 amule                                           
 5662 alex      20   0 5444m 4.9g  22m R 100.3  62.5   2:40.54 amule                                         
 5662 alex      20   0     0    0    0 D  47.2   0.0   2:42.97 amule                                           
This was generated by extracting the amule details from the "top" command every 10 or 20 seconds.  Clearly you can see that it is consuming all memory.

I tried running amule through valgrind, but it doesn't ever get to the same state.  I think this is because valgrind slows down amule so much that timeouts occur and then it doesn't behave the same.  I could try it again if necessary.  Also, can someone who uses valgrind regularly post a suppression file that I could look at and compare to mine.

My machine is fedora 14.
uname -a:
Linux alex.builderlynx.com 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

WxWidgets:
wx-config --version-full
2.8.11.0

I compiled my own version of cryptopp using the most recent subversion snapshot available from the website.  This has the patch applied that was mentioned in the forum post:
http://forum.amule.org/index.php?topic=18222.msg100743#msg100743

I am working with amule 10424 snapshot.  I compile amule with:
./configure --prefix=/usr --disable-optimize --disable-ccache --enable-debug --enable-geoip --enable-webserver
make
make install

I have run amule through gdb.  Here is the backtrace information that I get.  Hopefully it is some use to you:

Code: [Select]

Program received signal SIGSEGV, Segmentation fault.
0x000000359be09284 in pthread_mutex_lock () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install amule-2.2.6-3.fc14.x86_64
(gdb) bt
#0  0x000000359be09284 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x000000359d63e939 in ?? () from /lib64/libglib-2.0.so.0
#2  0x000000359d64074e in g_source_remove () from /lib64/libglib-2.0.so.0
#3  0x0000003a03be86d6 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) ()
   from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#4  0x00000035a6e234a7 in GSocket::Detected_Write() () from /usr/lib64/libwx_baseu_net-2.8.so.0
#5  0x00000035a262537f in ?? () from /usr/lib64/libgdk-x11-2.0.so.0
#6  0x000000359d641e33 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#7  0x000000359d642610 in ?? () from /lib64/libglib-2.0.so.0
#8  0x000000359d642c82 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#9  0x00000035a4f4b0b7 in gtk_main () from /usr/lib64/libgtk-x11-2.0.so.0
#10 0x0000003a03be74f8 in wxEventLoop::Run() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#11 0x0000003a03c6499b in wxAppBase::MainLoop() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#12 0x00000035a56972ca in wxEntry(int&, wchar_t**) () from /usr/lib64/libwx_baseu-2.8.so.0
#13 0x000000000062318a in main (argc=1, argv=0x7fffffffe0a8) at amule-gui.cpp:93
(gdb) bt full
#0  0x000000359be09284 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000000359d63e939 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#2  0x000000359d64074e in g_source_remove () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3  0x0000003a03be86d6 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) ()
   from /usr/lib64/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#4  0x00000035a6e234a7 in GSocket::Detected_Write() () from /usr/lib64/libwx_baseu_net-2.8.so.0
No symbol table info available.
#5  0x00000035a262537f in ?? () from /usr/lib64/libgdk-x11-2.0.so.0
No symbol table info available.
#6  0x000000359d641e33 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#7  0x000000359d642610 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#8  0x000000359d642c82 in g_main_loop_run () from /lib64/libglib-2.0.so.0
No symbol table info available.
#9  0x00000035a4f4b0b7 in gtk_main () from /usr/lib64/libgtk-x11-2.0.so.0
No symbol table info available.
#10 0x0000003a03be74f8 in wxEventLoop::Run() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#11 0x0000003a03c6499b in wxAppBase::MainLoop() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
No symbol table info available.
#12 0x00000035a56972ca in wxEntry(int&, wchar_t**) () from /usr/lib64/libwx_baseu-2.8.so.0
No symbol table info available.
#13 0x000000000062318a in main (argc=1, argv=0x7fffffffe0a8) at amule-gui.cpp:93
No locals.
(gdb) thread apply all bt

Thread 6 (Thread 0x7fffeaffd700 (LWP 4759)):
#0  0x000000359be0bc2e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000035a56e7e56 in wxConditionInternal::WaitTimeout(unsigned long) () from /usr/lib64/libwx_baseu-2.8.so.0
#2  0x00000035a56e8282 in wxSemaphoreInternal::WaitTimeout(unsigned long) () from /usr/lib64/libwx_baseu-2.8.so.0
#3  0x000000000073e0fc in CTimerThread::Entry (this=0x1bad110) at Timer.cpp:66
#4  0x00000035a56e8fe1 in wxThreadInternal::PthreadStart(wxThread*) () from /usr/lib64/libwx_baseu-2.8.so.0
#5  0x000000359be06d5b in start_thread () from /lib64/libpthread.so.0
#6  0x000000359b6e4a7d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fffebfff700 (LWP 4756)):
#0  0x000000359b63421e in __libc_sigaction () from /lib64/libc.so.6
#1  0x000000359b634029 in ssignal () from /lib64/libc.so.6
#2  0x00000035a6e23b0f in GSocket::Send_Stream(char const*, int) () from /usr/lib64/libwx_baseu_net-2.8.so.0
#3  0x00000035a6e246e8 in GSocket::Write(char const*, int) () from /usr/lib64/libwx_baseu_net-2.8.so.0
#4  0x00000035a6e1fb68 in wxSocketBase::_Write(void const*, unsigned int) () from /usr/lib64/libwx_baseu_net-2.8.so.0
#5  0x00000035a6e1fb9c in wxSocketBase::Write(void const*, unsigned int) () from /usr/lib64/libwx_baseu_net-2.8.so.0
#6  0x00000000006fdc38 in CSocketClientProxy::Write (this=0x332db20, buffer=0x2981970, nbytes=46) at Proxy.cpp:1309
#7  0x00000000004fad42 in CEncryptedStreamSocket::Write (this=0x332db20, lpBuf=0x2981970, nBufLen=46) at EncryptedStreamSocket.cpp:210
#8  0x00000000004f813a in CEMSocket::Send (this=0x332db20, maxNumberOfBytesToSend=1300, minFragSize=1300, onlyAllowedToSendControlPacket=true)
    at EMSocket.cpp:572
#9  0x000000000049e766 in CEMSocket::SendControlData (this=0x332db20, maxNumberOfBytesToSend=20, minFragSize=1300) at EMSocket.h:69
#10 0x000000000049e356 in CClientTCPSocket::SendControlData (this=0x332db20, maxNumberOfBytesToSend=20, overchargeMaxBytesToSend=1300)
    at ClientTCPSocket.cpp:2124
#11 0x00000000005b2dae in UploadBandwidthThrottler::Entry (this=0x1c6bed0) at UploadBandwidthThrottler.cpp:381
#12 0x00000035a56e8fe1 in wxThreadInternal::PthreadStart(wxThread*) () from /usr/lib64/libwx_baseu-2.8.so.0
#13 0x000000359be06d5b in start_thread () from /lib64/libpthread.so.0
#14 0x000000359b6e4a7d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fc4960 (LWP 4745)):
#0  0x000000359be09284 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x000000359d63e939 in ?? () from /lib64/libglib-2.0.so.0
#2  0x000000359d64074e in g_source_remove () from /lib64/libglib-2.0.so.0
#3  0x0000003a03be86d6 in GSocketGUIFunctionsTableConcrete::Uninstall_Callback(GSocket*, GSocketEvent) ()
   from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#4  0x00000035a6e234a7 in GSocket::Detected_Write() () from /usr/lib64/libwx_baseu_net-2.8.so.0
#5  0x00000035a262537f in ?? () from /usr/lib64/libgdk-x11-2.0.so.0
#6  0x000000359d641e33 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#7  0x000000359d642610 in ?? () from /lib64/libglib-2.0.so.0
#8  0x000000359d642c82 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#9  0x00000035a4f4b0b7 in gtk_main () from /usr/lib64/libgtk-x11-2.0.so.0
#10 0x0000003a03be74f8 in wxEventLoop::Run() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#11 0x0000003a03c6499b in wxAppBase::MainLoop() () from /usr/lib64/libwx_gtk2u_core-2.8.so.0
#12 0x00000035a56972ca in wxEntry(int&, wchar_t**) () from /usr/lib64/libwx_baseu-2.8.so.0
#13 0x000000000062318a in main (argc=1, argv=0x7fffffffe0a8) at amule-gui.cpp:93

Let me know if there are other tests you would like me to perform, or if I can do something else to help figure out this problem.  Thanks for a great program!  Alex.

[^marcell^]

Let me know if there are other tests you would like me to perform, or if I can do something else to help figure out this problem.

Could you try running amule with KAD disabled and see if it still eats up all of your RAM?

[alex-m1234]

I unchecked the preference Connection > Networks > Kademlia.  It is running in GDB now.

Just a bit more about this... When I run through GDB then it gives the seg fault and returns control to GDB.  I can then generate the stack traces.  When I then "continue" in GDB, it starts the process of eating up all the memory.  Then I have to kill amule.  So, I'm pretty sure the error is the start (or close to) the start of the problem.

Thanks. Alex.

[alex-m1234]

I ran it twice with KAD turned off.  It does the same thing.  Uses up all the RAM, then swapping and then crashing.  Let me know what else I can try.  Does the stack trace show useful information?

[Stu Redman]

Alex, thank you for your valuable help!

Does the stack trace show useful information?

Hmm. All it shows is that no code of aMule is directly involved.
Am I understanding you correctly:
- the top trace was done at 10-20s steps WITHOUT gdb
- with gdb it segfaults BEFORE consuming the memory
- after continuing in gdb (continue from a segfault?!? wonders never cease...) it does what it did before: grab 100% cpu to  get busy using up all memory

This would mean:
- wx segfaults in GSocket::Detected_Write() (maybe an invalid socket)
- instead of crashing clean aMule somehow catches the exception and continues
- corruption caused by the crash has put things off track and cause a memory waste

And good to know it's not Kad. One more thing to rule out.

Are you also getting these crashes since a recent Kernel update? Because that's what is happening in Ubuntu.

[btkaos]

Well, IMVHO the kernel update issue may expose a bug in aMule.

[Stu Redman]

I never said it is not a bug in aMule.  (Though I wouldn't rule out wx either. Or some wacky build problem, that's why I keep asking about a test with a self compiled wx, compiled with the actual environment, headers and all causing the problems.) I actually hope for an aMule bug, because this would be the easiest place to fix. (After finding the sucker.) But the visible consequences are in some way tied to the Kernel update.
If it is a "silent crash" the consequences of accessing a broken socket object may very well change with some Kernel change of course.

So you can leave the VH out. I'm open to any ideas.

[btkaos]

The kernel has had some changes in the scheduler, this of course impacts socket allocation and most of the networking code.

This seems a bug very hard to track, but we will manage to it

[alex-m1234]

Sorry, my original post was combining the details of different runs of amule into a single post. 
Run #1: Run amule normally, use top to monitor the memory usage:
1) amule runs for a while (an hour or so).
2) amule starts to consume all memory.
3) the top output I originally posted shows the memory consumption increasing.  The top output is only showing the last minute or so of activity.  Before that it is totally stable memory amount.

Run #2: Run amule within GDB:
1) amule runs for a while (an hour or so).
2) amule encounters problem.  control is given back to GDB.  I produce the back trace info (bt, bt full, etc).
3) I type "continue" in GDB.  Control goes back to amule.
4) amule starts to consume all the memory.

I am pretty sure that is what is happening.

I don't do much native linux debugging, but you are correct about amule continuing after GDB detects the segfault.  I don't even know if that is possible, but that is what seems to be happening. 

My Fedora updates itself regularly.  I am guessing this amule problem happened after some update or another.  I don't know specifically which update caused the problem though.  This problem started happening a couple of months ago.  I kinda figured it would fix itself in the following updates, but that hasn't happened yet, so I started digging in a lilttle more and posting this message.

I hope I answered your questions.  Let me know what else I can try.  Would compiling my own version of wxWidgets be useful?

[btkaos]

Hi alex,

Sorry, my original post was combining the details of different runs of amule into a single post. 
Run #1: Run amule normally, use top to monitor the memory usage:
1) amule runs for a while (an hour or so).
2) amule starts to consume all memory.

Can you reproduce the crash realiabily? If so, do you have a download limit set?

If the crash happens regularly, try disabling the donwload limit (set it to 0) and see what happens.

I hope I answered your questions.  Let me know what else I can try.  Would compiling my own version of wxWidgets be useful?

IMVHO compiling your own wxWidgets is not useful ATM.

[alex-m1234]

Can you reproduce the crash realiabily? If so, do you have a download limit set?

If the crash happens regularly, try disabling the donwload limit (set it to 0) and see what happens.

In my previous runs of amule (previous posts), I had a download speed limit set (30Kbs down and 15Kbs up).  I have changed the down limit to 0 and restarted amule.  Amule has been running now for 3 hours without any problems.  I'll let it go for a while longer.  Changing the limit definitely seems to have helped/fixed the problem.

I can also reproduce the out of memory problem regularly when I have the download limit set.  That is what my first few posts were saying.

[alex-m1234]

So I let it run for about 8 hours in total with the download limit set to 0.  It worked fine for that entire time.  I guess that means the issue has something to do with the bandwidth limiting in amule.  Let me know if I can test something else.

[btkaos]

So I let it run for about 8 hours in total with the download limit set to 0.  It worked fine for that entire time.  I guess that means the issue has something to do with the bandwidth limiting in amule.  Let me know if I can test something else.

Nothing that I can think for the moment. I'm researching the issue but it is a very complex one.