aMule Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

We're back! (IN POG FORM)

Pages: 1 ... 4 5 [6] 7 8

Author Topic: Re: upredictable memory leak  (Read 53929 times)

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #75 on: January 18, 2011, 11:31:24 PM »

Picture this, aMule reads from the socket but more data is available, wx generates another event, then aMule receives the event but reads no data as it exhausted the limit, another event is generated (maybe), etc...
Since we read nothing in this case, no other event should be generated.
We may read very few bytes (2 bytes) or something like that and another event will pop up.
Quote
Did you notice how all of your sockets simultaneously started piling up events? Looks like something seriously breaks in the underlying system.
Yes, I'm testing a patch that should solve the socket issue and I'm getting the very strange glib context corruption crashes. It may well be that Ubuntu glib's is broken. I'll build glib/gtk from scratch (this is going to be a lot of work :) ) and see. Could be a compiler bug.

For now, it would be really helpful if someone could reproduce the bug. All you need is add some files so your download speed is very high, then limit it to 10Kbps. For quick reproduction, the bigger is the possible download speed faster aMule will crash.

Specially interesting would be to know if previous ubuntu versions or other linux distributions. Some Fedora or Suse feedback in this case would be great.
Quote
What I keep thinking is, maybe we should kick out all of the wx socket usage (along with the underlying GTK networking) and use some other socket lib. Maybe Boost.Asio. It would also solve (some of) the problems with 2.9 . I seriously doubt wx devs test against aMule or anything else driving a socket lib to its limits. They probably send a few packets from A to B and think it works.
(expecting some serious flaming for this...  :-\ )
I completely agree, but that's because IMHO gtk is a library which leaves a lot to be desired and very unprofessional, at least when compared to high quality libraries like qt. wx is so so.

Anyways I wouldn't blame wx or amule yet, the bug could well be in the kernel, glib, libc, who knows. We need more testers who are able to reproduce the bug. Switching to Boost is a lot of work.
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #76 on: January 18, 2011, 11:35:55 PM »

This crash is interesting. The context gets corrupted again, this time to the weird value of 0x45445f454352554f. Google says nothing for this value.

Really really weird and fishy, this couldn't if the code is compiled correctly. I guess the event overflow is just another instance of the bug, the context is corrupted in such a way that glib gets in an infinite loop sending events. As you noted all sockets receive the same number of events.

I'd forget for now about blaming wxEventHandling (though theoretically it could be starved in the manner I described) .
Code: [Select]
(gdb) bt full
#0  __pthread_mutex_lock (mutex=0x45445f4543525557) at pthread_mutex_lock.c:50
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = <value optimized out>
#1  0x00007ffff29ceec0 in g_source_set_callback_indirect (source=0x7fffdc2b4160, callback_data=0x7fffdc0ce920,
    callback_funcs=0x7ffff2c70610) at /build/buildd/glib2.0-2.26.1/glib/gmain.c:1149
        context = 0x45445f454352554f
        old_cb_data = <value optimized out>
        old_cb_funcs = <value optimized out>
        __PRETTY_FUNCTION__ = "g_source_set_callback_indirect"
#2  0x00007ffff29c77ac in g_io_add_watch_full (channel=<value optimized out>, priority=0, condition=<value optimized out>,
    func=0x7ffff478e950 <gdk_io_invoke>, user_data=0x7fffdc062e20, notify=0x7ffff478e930 <gdk_io_destroy>)
    at /build/buildd/glib2.0-2.26.1/glib/giochannel.c:668
        source = 0x7fffdc2b4160
        id = <value optimized out>
        __PRETTY_FUNCTION__ = "g_io_add_watch_full"
#3  0x00007ffff478e8e6 in IA__gdk_input_add_full (source=<value optimized out>, condition=0, function=<value optimized out>,
    data=0x25f9940, destroy=0) at /build/buildd/gtk+2.0-2.22.0/gdk/gdkevents.c:1129
        result = <value optimized out>
        channel = 0x7fffdc2b40e0
        cond = 12
#4  0x00007ffff67fcbb1 in GSocketGUIFunctionsTableConcrete::Install_Callback (this=0x7ffff6ccabd0, socket=0x25f9940, event=GSOCK_OUTPUT)
    at ./src/gtk/gsockgtk.cpp:98
        m_id = 0x23773b0
        c = 1
#5  0x00007ffff63c379d in GSocket::Enable (this=0x25f9940, event=GSOCK_OUTPUT) at ./src/unix/gsocket.cpp:1521
No locales.
#6  0x00007ffff63c2f1f in GSocket::Write (this=0x25f9940, buffer=0x7fffe41e9840 "\3176\246", size=6) at ./src/unix/gsocket.cpp:1257
        ret = -1
        __PRETTY_FUNCTION__ = "int GSocket::Write(const char*, int)"
#7  0x00007ffff63bcfa5 in wxSocketBase::_Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at ./src/common/socket.cpp:539
        total = 0
        ret = -1
#8  0x00007ffff63bcedd in wxSocketBase::Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at ./src/common/socket.cpp:507
No locales.
#9  0x0000000000681b8e in CSocketClientProxy::Write (this=0x1da8d40, buffer=0x7fffe41e9840, nbytes=6) at Proxy.cpp:1310
        lock = {m_isOk = true, m_mutex = @0x1da8e88}
#10 0x00000000004e3bf8 in CEncryptedStreamSocket::Write (this=0x1da8d40, lpBuf=0x7fffe41e9840, nBufLen=6) at EncryptedStreamSocket.cpp:210
        __FUNCTION__ = "Write"
#11 0x00000000004e0ad2 in CEMSocket::Send (this=0x1da8d40, maxNumberOfBytesToSend=<value optimized out>,
    minFragSize=<value optimized out>, onlyAllowedToSendControlPacket=<value optimized out>) at EMSocket.cpp:611
        tosend = 6
        result = <value optimized out>
        bWasLongTimeSinceSend = false
        sentControlPacketBytesThisCall = 0
        __FUNCTION__ = "Send"
        lock = {m_isOk = true, m_mutex = @0x1da9200}
        anErrorHasOccured = false
        sentStandardPacketBytesThisCall = 0
#12 0x000000000049959f in CEMSocket::SendFileAndControlData (this=0x45445f4543525557, maxNumberOfBytesToSend=3691833632, minFragSize=1)
    at EMSocket.h:71
No locales.
#13 0x000000000048dd5b in CClientTCPSocket::SendFileAndControlData (this=0x45445f4543525557, maxNumberOfBytesToSend=3691833632,
    overchargeMaxBytesToSend=1) at ClientTCPSocket.cpp:2138
        returnStatus = {success = true, sentBytesStandardPackets = 0, sentBytesControlPackets = 18}
#14 0x000000000057481d in UploadBandwidthThrottler::Entry (this=<value optimized out>) at UploadBandwidthThrottler.cpp:480
        socketSentBytes = {success = true, sentBytesStandardPackets = 0, sentBytesControlPackets = 0}
        data = 69633
        socket = 0x45445f4543525557
        slotCounter = 18
        slots = <value optimized out>
        spentBytes = 0
        spentOverhead = <value optimized out>
        sendLock = {m_isOk = true, m_mutex = @0x18d6820}
        timeSinceLastLoop = <value optimized out>
        minFragSize = 1300
        doubleSendSize = 2600
        sleepTime = <value optimized out>
        thisLoopTick = 2600814829
        bytesToSpend = <value optimized out>
        extraSleepTime = 1000
        step = 45
        lastLoopTick = 0
        allowedDataRate = <value optimized out>
        rememberedSlotCounter = 15
        sendLock = {m_isOk = 200, m_mutex = @0x7ffff7fcca48}
#15 0x00007ffff60dc705 in wxThreadInternal::PthreadStart (thread=0x18d6800) at ./src/unix/threadpsx.cpp:766
        __clframe = {__cancel_routine = 0x7ffff60dc84f <wxPthreadCleanup(void*)>, __cancel_arg = 0x18d6800, __do_it = 1, __cancel_type = 0}
        pthread = 0x16a6320
        rc = 0
        dontRunAtAll = false
        __FUNCTION__ = "PthreadStart"
#16 0x00007ffff60dc5a5 in wxPthreadStart (ptr=0x18d6800) at ./src/unix/threadpsx.cpp:718
No locales.
#17 0x00007ffff7bc6971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
        __res = <value optimized out>
        pd = 0x7fffec6bf700
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737159886592, -7788523305851881642, 140737488345856, 140737159887296,
                140737354125376, 3, 7788486849677817686, 7788540326228588374}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0},
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        robust = <value optimized out>
        freesize = <value optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#18 0x00007ffff532192d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locales.
#19 0x0000000000000000 in ?? ()
No symbol table info available.
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #77 on: January 19, 2011, 02:07:03 AM »

I built the whole stack: git glib, gtk 2.22.1 (gtk in git is gtk3 and proved to be too difficult to build with wx) an wx 2.8.11.

Nothing changed :( I got an OOM, waiting so check for a crash.
Logged

^marcell^

  • Developer
  • Hero Member
  • *****
  • Karma: 28
  • Offline Offline
  • Posts: 524
Re: upredictable memory leak
« Reply #78 on: January 23, 2011, 12:14:22 PM »

I am having this issue too. So ran aMule through GDB and just when it started eating all my RAM I paused aMule.
By that time 3 aMule threads were living: the main application loop, the timer running in the background and the uploadbandwith-throttler.
So I switched between the three and made them run step-by-step. The main application thread was the one allocating all my RAM.
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #79 on: January 23, 2011, 03:38:58 PM »

I am having this issue too. So ran aMule through GDB and just when it started eating all my RAM I paused aMule.
By that time 3 aMule threads were living: the main application loop, the timer running in the background and the uploadbandwith-throttler.
So I switched between the three and made them run step-by-step. The main application thread was the one allocating all my RAM.
Yes, it is called from glib main loop. I think we already know the cause of the OOM. glib main loops generates a event for a socket watch, this calls wx callbacks and queues and event.

This sequence is never stopped for all socket, so wx event handling won't never run, resulting in a OOM.

Right now I'm pretty much sure that this kind of event-spree is due to a corruption of glib's main context (which sometimes causes a crash) but I'm not able to see what causes it, it is very difficult to stop amule *just* before the problem.
Logged

Kry

  • Ex-developer
  • Retired admin
  • Hero Member
  • *****
  • Karma: -665
  • Offline Offline
  • Posts: 5795
Re: upredictable memory leak
« Reply #80 on: January 28, 2011, 01:00:23 AM »

btkaos, at last I found my old patch, which was never applied to wx trunk branches for various reasons.

Please test attached patch over an up-to-date, clean version of the branch 2.8 of wxWidgets, then tell me if you see a different beaviour, or even more problems! I adapted it from a different branch, so I might have messed up somewhere.


Edit (Stu): This is not the patch that fixes the problem!
« Last Edit: February 17, 2011, 08:03:03 PM by Stu Redman »
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #81 on: January 28, 2011, 02:33:02 AM »

Ok thanks, I'll try, but not today.
Logged

Kry

  • Ex-developer
  • Retired admin
  • Hero Member
  • *****
  • Karma: -665
  • Offline Offline
  • Posts: 5795
Re: Re: upredictable memory leak
« Reply #82 on: January 28, 2011, 03:49:22 AM »

YOU WILL TRY WHENEVER I DAMN WELL TELL YOU TO.

I mean, take your time.
Logged

Kry

  • Ex-developer
  • Retired admin
  • Hero Member
  • *****
  • Karma: -665
  • Offline Offline
  • Posts: 5795
Re: Re: upredictable memory leak
« Reply #83 on: February 05, 2011, 10:32:09 PM »

Any news?
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #84 on: February 05, 2011, 11:52:09 PM »

Trying the patch right now. Interesting thing it that amule did crash with low activity and download limit, but it took much longer.
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #85 on: February 06, 2011, 01:47:41 AM »

Sadly the patch had no effect, amule got stuck again in an endless glib loop of event creation.

Regards,
BTK
Logged

Kry

  • Ex-developer
  • Retired admin
  • Hero Member
  • *****
  • Karma: -665
  • Offline Offline
  • Posts: 5795
Re: Re: upredictable memory leak
« Reply #86 on: February 06, 2011, 04:13:50 AM »

Funny. Give me a couple more days to continue investigating.

For the record - I haven't been able to reproduce it locally, so I'm just fixing blindly. You have any SURE way to reproduce it? I know it's somewhere in the thread, but... can you give me the exact specs (distro, version, kernel, window manager, number of CPUs and versions of relevant libraries) so I can try and clone them locally?
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: Re: upredictable memory leak
« Reply #87 on: February 07, 2011, 12:29:19 AM »

For the record - I haven't been able to reproduce it locally, so I'm just fixing blindly. You have any SURE way to reproduce it? I know it's somewhere in the thread, but... can you give me the exact specs (distro, version, kernel, window manager, number of CPUs and versions of relevant libraries) so I can try and clone them locally?
A current Ubuntu should do, (some users reported hapenning on Fedora too)

Just add whatever enough downloads so you will have big download speed, then limit it to, say 10 Kbps. Wait a while and aMule should crash/OOM.

The important point is to manage to have more download speed available than the limit for a bit of time. Let me know how it goes.
Logged

btkaos

  • Global Moderator
  • Sr. Member
  • *****
  • Karma: 110
  • Offline Offline
  • Posts: 486
  • Kaos is infinite!
Re: upredictable memory leak
« Reply #88 on: February 08, 2011, 05:02:20 PM »

Any luck reproducing the bug?

So far I've only seen reports from users, but no dev could reproduce it. I can't completely eliminate we are doing something weird.

It would be really helpful if another dev could reproduce the bug and confirm it is reproducible indeed.

Regards,
BTK
Logged

Stu Redman

  • Administrator
  • Hero Member
  • *****
  • Karma: 214
  • Offline Offline
  • Posts: 3739
  • Engines screaming
Re: upredictable memory leak
« Reply #89 on: February 08, 2011, 07:52:07 PM »

It would be really helpful if another dev could reproduce the bug and confirm it is reproducible indeed.

Setting the download limit to 0 doesn't really solve the issue on my side. Instead it seems to limit the probability of occuring. Ever since the OOM issue arised I kept the download limit at 0 and even so it still crashes from time to time. On much fewer occasions, but still it does.
Logged
The image of mother goddess, lying dormant in the eyes of the dead, the sheaf of the corn is broken, end the harvest, throw the dead on the pyre -- Iron Maiden, Isle of Avalon
Pages: 1 ... 4 5 [6] 7 8