I have also noticed progressive memory leaking from Mac aMule. In addition to top and Activity Monitor indicating a large memory footprint for the process, aMule will thrash for a couple of minutes (showing the spinning beachball cursor) if I try to open the preferences or file details dialog. It doesn't have such problems immediately after launch, only after it's been running for hours. Running sample on aMule while it's thrashing shows lots of memory allocation and deallocation calls on the stack.
I have made some half-assed attempts to identify the source of the leaks, but don't have any definitive information. One can use the leaks command (see "man leaks" for info) to scan an application's memory space for allocated blocks that are no longer referenced anywhere. This is a bit like the first half of a garbage collection routine. After aMule has run for some hours, the report is very, very long with many small allocations having been leaked. If memory serves, I have seen leaks amounting to as much as 70 megabytes in hundreds of thousands of allocations.
The leaks command can report stack traces for the leaked allocations if you run aMule with the environment variable MallocStackLogging set to 1. I have never managed to get such a report because I always forgotten some simple step in the process. (I can be stupid, sometimes.) You want to use a build of aMule that hasn't been stripped of its symbols. You have to let aMule run for a while, but you can't let it run for too long or the stack logging will fill memory and it will crash. I think the leaks are related to GUI events, so you want to be active in the GUI while running the test. I keep meaning to go back and try again.
The leaks report can contain a snippet from the allocated block, especially if it recognizes the type (string, OS primitive, etc.). Often these snippets contain text which I recognize as four-character-codes used by the Mac OS to tag certain kinds of data structures like event records, window records, GUI controls, etc. Some other leaks show their type directly as CFSocket objects and the like. I'm pretty sure wxMac is failing to release or dispose of OS-supplied data structures after it's done with them.
As mentioned in the leaks man page, there is also /Developer/Applications/Performance Tools/MallocDebug.app which performs a similar function in a GUI app. Annoyingly, each program can miss leaks that the other will catch. MallocDebug can attach to an app which was launched with MallocStackLogging set, or it can directly launch an application itself.