aMule Forum
English => en_Bugs => Topic started by: anp on June 24, 2008, 07:45:16 PM
-
i have share consisting 4K+ directories and 15K+ files;
amule start takes about 40 minutes (hashing files took 5 hours) on AMD Athlon64 3000+ linux/i686 binary.
and before this process finishes, amuled do not connect to any servers or KAD.
why amuled does not hash files in background? and what it does with files? (about 10-20 new files created between executions, 1-2 files moved)
-
What version, what platform ?
-
aMule 2.2.1, linux/i686 self-compiled binary (gcc 4.0.3)
-
in addition, amuled stuck with ~90% CPU usage while new file is downloaded for long time (at least 10 minutes), while time, needed to hash such file is 10-15 seconds (with 10GB share)
-
See also:
http://forum.amule.org/index.php?topic=15329.0
did it happen with previous versions of amule?
-
sorry, i dont know, i started using amule with big shares from version 2.2.1
-
I advise to share less files for the network's sake, althought this is surely not the solution to the problem. I can't say anything about the high CPU usage problems, I'm afraid.
-
15k files is, very much probably, more than the servers are allowing indexing when you publish them.
-
only about 10% of servers not allowing shares with more than 1000 files.
i'm not using it.
and KAD clients can search all my files - it is not problem.
but why do we need rehashing all files at start and every time when file download completed?
-
It doesn't do that.
-
long time, 90% CPU usage, and hdd activity....
if not hashing, what it does?
-
I don't know. You tell me. But it's not hashing, or at least it shouldn't be. Also hashing is done on a separate lower prio thread.
You need to provide us with more information if you want us to find out what's going on in your aMule. Compile it with debug information, enable some relevant debug categories, run it to reproduce the problem and then give us the log with all the debug information. I am not going to create 15k files in my hard drive explicitely to reproduce it, specially when I ahve told you that most servers will never index that amount of files.
-
enable some relevant debug categories
section "Debug" in amule.conf?
what categories i should enable?
Compile it with debug information
--with-wxdebug Force building against a debug build of wxWidgets,
even if --disable-debug is given (default is auto)
is it enought?
specially when I ahve told you that most servers will never index that amount of files
maybe, but files can be accessed via KAD...
-
Honestly you should try to reduce the number of files that you are sharing.
But my guess is, that aMule is publishing the files over KAD, before you can start.
-
log says, that server connection (and after some time KAD connection, "firewalled" and then "ok")
goes some time after the amule ends its HDD and CPU activity
log:
2008-06-24 23:52:51: ClientCreditsList.cpp(168): Creditfile loaded, 1108 clients are known
2008-06-24 23:52:51: IPFilter.cpp(109): Loading IP-filters 'ipfilter.dat' and 'ipfilter_static.dat'.
2008-06-24 23:52:51: IPFilter.cpp(333): Loaded 0 IP-ranges from '/home/p2p/.aMule/ipfilter.dat'. 0 malformed lines were disca
rded.
2008-06-24 23:52:51: IPFilter.cpp(333): Loaded 0 IP-ranges from '/home/p2p/.aMule/ipfilter_static.dat'. 0 malformed lines wer
e discarded.
2008-06-24 23:52:51: ExternalConn.cpp(160): *** TCP socket (ECServer) listening on 0.0.0.0:4712
2008-06-24 23:52:51: MuleUDPSocket.cpp(81): Created Server UDP-Socket at port 41299
2008-06-24 23:52:51: MuleUDPSocket.cpp(81): Created Client UDP-Socket at port 41296
2008-06-24 23:52:51: ServerList.cpp(83): Loading server.met file: /home/p2p/.aMule/server.met
2008-06-24 23:52:51: ServerList.cpp(168): 242 servers in server.met found
2008-06-24 23:52:51: ServerList.cpp(817): No server list address entry in 'addresses.dat' found. Please paste a valid server
list address into this file in order to auto-update your server list
2008-06-24 23:52:51: DownloadQueue.cpp(176): Found 36 part files
2008-06-25 00:18:10: SharedFileList.cpp(358): Found 29266 known shared files, 59 unknown
2008-06-25 00:18:10: amule.cpp(801): Connecting
2008-06-25 00:18:10: ServerSocket.cpp(621): Servers: Trying to connect
2008-06-25 00:18:10: ServerSocket.cpp(754): Connecting to Razorback 3.3 (92.48.193.14 - 92.48.193.14:5000) using protocol obf
uscation.
high CPU and HDD usage happens between this messages:
2008-06-24 23:52:51: DownloadQueue.cpp(176): Found 36 part files
2008-06-25 00:18:10: SharedFileList.cpp(358): Found 29266 known shared files, 59 unknown
in this start files are fully unchanged (shared filesystem locked in read-only mode, except "ed2k-incoming" directory, which resides in another filesystem)
59 unknown files are zero-length files:
2008-06-25 00:18:11: RoutingZone.cpp(188): Read 200 Kad contacts
2008-06-25 00:18:11: ThreadTasks.cpp(111): Hasher: Warning, 0-size file, skipping: /mnt/fs3/games/mixed/linuxcenter-games-dvd
/Installers/Daikatana/daikatana_1.2-english.uk.run
2008-06-25 00:18:11: ThreadScheduler.cpp(308): ThreadScheduler: Completed task 'Hashing - /mnt/fs3/games/mixed/linuxcenter-ga
mes-dvd/Installers/Daikatana/daikatana_1.2-english.uk.run', 58 tasks remaining.
after downloading file it gets loss of network connection, and same CPU and HDD usage for a long time:
2008-06-27 09:29:54: PartFile.cpp(2268): Finished downloading: Peremotka.2008.L2.DVDRip_INTERFILM.avi
2008-06-27 09:29:55: ThreadScheduler.cpp(308): ThreadScheduler: Completed task 'AICH Syncronizing', 0 tasks remaining.
2008-06-27 09:30:58: ClientTCPSocket.cpp(2066): Packet Parsing Errors: Caught EOF exception: SafeIO::EOF: Attempt to read pas
t end of file.
On packet with protocol e3, opcode 4c, size 156 ClientData: Client Unknown on IP:Port 84.109.142.171:4662 using Unknown Unkno
wn
2008-06-27 09:51:52: ClientTCPSocket.cpp(2066): Packet Parsing Errors: Caught EOF exception: SafeIO::EOF: Attempt to read pas
t end of file.
On packet with protocol e3, opcode 4c, size 156 ClientData: Client Unknown on IP:Port 84.109.142.171:4662 using Unknown Unkno
wn
2008-06-27 09:59:16: Connected to Kad (firewalled)
2008-06-27 09:59:17: Connected to Kad (ok)
and shared filesystem is not so slow...
[root@ebp3 /]# time ls -Rrltsh /mnt >/dev/null
real 0m1.466s
user 0m0.730s
sys 0m0.410s
-
The known.met file is not a relational database. So it processes secuentially all the entries, 30 THOUSAND of them.
When aMule iterates your shared folder, for every file it finds it compares it to the known file list. Which has 30 thousand entries.
I'm sure you can see the problem there.
About your comments on Kad indexing your files: Yes, Kad will index all your files. It will however take a long long, LONG time to do so, so I hope you leave aMule open for several days if you plan the network to index all those files.
-
The known.met file is not a relational database. So it processes secuentially all the entries, 30 THOUSAND of them.
is it protocol limitation or program limitation? (at same share running microdc2 for directconnect network, and it works fine)
When aMule iterates your shared folder, for every file it finds it compares it to the known file list. Which has 30 thousand entries.
I'm sure you can see the problem there.
it is not problem. problem is losing of network connection to kad/servers and losing control during this time (amule not downloading files, not responding to commands)
About your comments on Kad indexing your files: Yes, Kad will index all your files. It will however take a long long, LONG time to do so, so I hope you leave aMule open for several days if you plan the network to index all those files.
yes, server designed to run in 24x7 mode