I must say that I have seen similar behaviour with amuled on Solaris. I have traced it to failed connectivity to the aMule GUI: If the connection fails a few times somehow amuled starts to use the entire memory available to the user.
Luckily Solaris has advanced resource management features so the hogging can be limited to a few hundred megabytes. But it is very reproducable.