Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Kry]

AICH does't happen during the final hashing, as far as I know.

[wuischke]

You are correct, Kry, I excuse for misinforming you, p.r.

   // We can only create the AICH hashset if the file is a knownfile or
   // if the partfile is complete, since the MD4 hashset is checked first,
   // so that the AICH hashset only gets assigned if the MD4 hashset
   // matches what we expected. Due to the rareity of post-completion
   // corruptions, this gives us a nice speedup in most cases.

[freddy77]

Mmm... my patch is already in SVN... however I don't understand! My router have 64mb but still hangs when start upload... perhaps the problem is that I don't have swap Yes, you should think that if you have 29MB of swap used + 32Mb of ram you have 29+32 = 61Mb used... but this is not true... you have more ram. The reason is that unaccessed anonymous memory don't take physical ram and are only allocated in swap. I have about 40mb free (cache+free) but taking into account process size (about 10mb... really huge!), 20mb (see below) and needed cache to not slow down too much are not that much...

I ran my SVN version for a day on my intel testing machine and memory (with Kad enabled!) stay under 23mb... I really don't understand... I'll try to enable overcommit of memory (I know it's a crazyness but can help). Perhaps I have a problem in mmap code using mips that cause more memory allocation? I'll add some logging. I have also to test a .so I wrote to override allocation function and put allocations in a mmaped are so to emulate swap (but only for amuled ).

Perhaps using another SHA/MD4/whatever from different implementation (like OpenSSL) could help. Is there a test program to compare hashing speeds??

[p.r.]

I'm going to use cpulimit ( http://cpulimit.sourceforge.net ).

 I  don't know how I'll use it in right way because I have 5  same named ("amuled") processes but only one of them is  bad behaved.

Amuled performs final hashing (pid  1152) : 

Code: [Select]

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
 1152 p2p      R      11832   761 58.2 38.9 amuled
  704 p2p      R       4908   701 14.2 16.1 rtorrent
    4 root     SW         0     1  1.6  0.0 kswapd
  760 p2p      D      11832     1  0.8 38.9 amuled
 1242 root     R        196  1203  0.6  0.6 exe
  762 p2p      S      11832   761  0.2 38.9 amuled
  764 p2p      S      11832   761  0.0 38.9 amuled
  761 p2p      S      11832   760  0.0 38.9 amuled
  410 root     S        112     1  0.0  0.3 chkbutton
  700 root     S         68     1  0.0  0.2 dtach
  667 root     S         68     1  0.0  0.2 busybox
  411 root     S         68     1  0.0  0.2 webs
    1 root     S         48     0  0.0  0.1 init
  588 root     S         48     1  0.0  0.1 ftpd
  533 root     S         44     1  0.0  0.1 sh
 1203 root     S         28   667  0.0  0.0 sh
  701 p2p      S         28   700  0.0  0.0 sh
  364 root     SW         0     1  0.0  0.0 kjournald
    3 root     SWN        0     1  0.0  0.0 ksoftirqd_CPU0
   34 root     SW         0     1  0.0  0.0 loop0
    8 root     SW         0     1  0.0  0.0 mtdblockdTake a look at Kernel Swap Daemon (kswapd)  - its activity is not heavy for CPU and takes only 1.6%.


For comparison - Amuled in normal operation:

Code: [Select]

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
 704 p2p      S       7284   701 16.3 24.0 rtorrent
  760 p2p      R       5804     1  0.8 19.1 amuled
 1242 root     R        196  1203  0.2  0.6 exe
  667 root     S         68     1  0.2  0.2 busybox
    4 root     SW         0     1  0.2  0.0 kswapd
  762 p2p      S       5804   761  0.0 19.1 amuled
  764 p2p      S       5804   761  0.0 19.1 amuled
  761 p2p      S       5804   760  0.0 19.1 amuled
 1203 root     S        220   667  0.0  0.7 sh
  410 root     S        132     1  0.0  0.4 chkbutton
    1 root     S         80     0  0.0  0.2 init
  700 root     S         68     1  0.0  0.2 dtach
  411 root     S         68     1  0.0  0.2 webs
  588 root     S         48     1  0.0  0.1 ftpd
  533 root     S         44     1  0.0  0.1 sh
  701 p2p      S         28   700  0.0  0.0 sh
  364 root     SW         0     1  0.0  0.0 kjournald
    3 root     SWN        0     1  0.0  0.0 ksoftirqd_CPU0
   34 root     SW         0     1  0.0  0.0 loop0
    8 root     SW         0     1  0.0  0.0 mtdblockd
  686 root     SW         0     1  0.0  0.0 dropbear
freddy77
How could I get your patch?

[Stu Redman]

http://amule.uw.hu/tarballs/

[freddy77]

News.... I still don't know if good or bad
I enabled overcommit in kernel (/proc/sys/vm/overcommit_memory) and amuled with my patch and it started working. The problem is that after a while (some minutes) memory got exhausted 
I saw that my mmap patch do the job (now hashing don't stop the router) but I cannot run amuled.
Perhaps I'll manage a modified kernel with swap support (not that easy as it seems).
Well... I have still detected two way to reduce memory usage:
- do not buffer downloaded files (I have a partial patch that still use mmap to avoid double memory use... I'll try to fix it or I'll save files directly)
- use utf8 for string coding... not that easy

[Stu Redman]

Have you tried to disable Kad ?

[freddy77]

Yes, Kad disabled 

[Stu Redman]

IP filter disabled?

There are several lists (sources, uploadqueue) that are not very dynamic and take a lot of space (I'm just guessing here, I didn't measure). Maybe you can move some of the constant data (strings, hashes,...) to a mmap file, possibly stored on a USB flash drive? I'd let the file simply grow over time (what are a few GB nowadays) and clean it up by restarting the app all 24h or so.

[p.r.]

Well, torrent uses sha1 on small subsets (64kB to 4MB according to a large online encyclopaedia). ed2k (i.e. aMule) uses bigger chunks of 9500 kB which are hashed using MD4. aMule uses additionally AICH which uses sha1 to hash small parts of 180 kB, i.e. it creates 52+1 hashes per chunk. (Furthermore aMule copies the whole file after verification whereas torrent usually does not. That's what Kry's question is targetting.)

Therefore it's safe to assume that aMule is doing more work when hashing. This does, however not explain the slowdown. If, however, your computer is low on RAM this might be caused by the higher RAM usage of aMule when hashing. (It does currently load the whole 9.28 MB chunk into RAM, iirc.) This might cause swapping which is always a lot slower. freddy77 is working on patches to lower the RAM usage, so this might be improved in the future.

1. But  MD4 isn't  significant slower (in 3 times), it's  even a little bit faster than sha1.

2. Are any changes in 2.2.4 concerning memory managment during procedure of final file hashing?

[Stu Redman]

2. Are any changes in 2.2.4 concerning memory managment during procedure of final file hashing?

No. Freddy's mmap patch is in trunk only.

[p.r.]

And what about data alignment?
Do amule try to perform aligned memory access with powerpc or don't care about it?
And why powerpc is not mentioned in ArchSpecific.h?

Sorry if I'm asking stupid questions, but I've googled  this info , and I supposed that it could be a reason of my problem too.

[Stu Redman]

Should have nothing to do with final hashing.
There is no Cyptopp.cc file in this project, and now rawpoke in cryptopp at all.

You're right about PowerPC missing in ArchSpecific.h (though all RawPokes are used on already aligned data as far as I see). What's the predefine for it - __powerpc__ ?

[p.r.]

If I knew, may be  __powerpc__.

[Stu Redman]

Just try it.

Code: [Select]

#ifdef __powerpc__
ARRGH
#endif
If you get a compile error it's correct.