Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[stoatwblr]

I'll give it a ty when it shows up on the SVN tarball page

[Stu Redman]

http://code.google.com/p/amule/source/checkout

[stoatwblr]

Ta.

The switch to backgrounding should solve another issue I complained about a long time ago (slow startup with lots of shares enabled)

[stoatwblr]

10601 fixes the problem for me (by updating known.met only after 300Mb and not after 30 files anymore). Please try it.

Known.met now seems to be growing to about 500kB before being truncated.

[stoatwblr]

So far so good, thanks.

One other thing I noticed but forgot to mention.

If amule(d) is told to shutdown, it insists on finishing the entire rehashing list first instead of just closing off at the file it's working on and closing off known.met cleanly, then resuming wher it left off when next started.

If there are a lot of files left to rehash, that can result in an apparent "runaway" state.

[Stu Redman]

The switch to backgrounding should solve another issue I complained about a long time ago (slow startup with lots of shares enabled)

No. Shared files are searched in foreground still. aMule takes a good minute for this for me with 18000 shares.
There is a performance boost in 10602, but apparently only in Windows. Not in Ubuntu though. Maybe in other platforms too.

Known.met now seems to be growing to about 500kB before being truncated.

So far so good, thanks.

Huh? known.met should never be truncated. It gets the appearance however if you check its size while it's being rewritten.

[stoatwblr]

That's what I'm doing.

Anyway... after applying the update, same problem (known.met being reported as corrupted and scanning starting over from scratch)

I've crosschecked by dropping the shares down to ~1800 files, waited until it finished scanning them and issued a shutdown command via amulcmd.
Amuled's been sitting there for about 45 mins so far eating 100% CPU. I'm pretty sure that if I HUP it, known.met will be reported as corrupt again.

Yes, it's Ubuntu (Natty) and I usually run amuled, not amule.

One other thing I spotted - while hashes are being generated, downloads and  speed drops to virtually zero, even if there was pretty good throughput before the hashing started. Uploads don't seem to be affected as much.

[stoatwblr]

Amuled finally exited 2 hours after being given the shutdown command - during that period no files in the share area were opened but it was using 100+% cpu and constantly writing to known.met

Is it possible that known.met file is being run through a second time after hash generation for some reason?

[Stu Redman]

Please make a debug build of amuled and activate Verbose Debug-Logging for
General
Hasher
AICH-Hasher
KnownFileList
ThreadScheduler

That's in amule.conf:

Code: [Select]

[eMule]
VerboseDebug=1
[Debug]
Cat_General=1
Cat_Hasher=1
Cat_AICH-Hasher=1
Cat_KnownFileList=1
Cat_ThreadScheduler=1

Try if you find something useful in the log. You can also mail the log to me if you trust me with your shares. Don't post it here however.

[stoatwblr]

Not much being logged at all It seems that logging stops after a while.

Still getting "Known file list corrupted, contains invalid header."

How does amule attempt rto recover from this? known2_64.met hasn't been touched in several days but it still seems to want to rehash every single file in known.met

[GonoszTopi]

known2_64.met hasn't been touched in several days but it still seems to want to rehash every single file in known.met

That's because known.met contains the eD2k, known2_64.met the AICH hashsets for the known files. Neither can be recovered from the other.

[stoatwblr]

There must be a better way  of doing this. What about using sqlite or mysql, or some form of atomic updates?

[stoatwblr]

Even as a starting point, doing the rehashing in a new file and then renaming it into place would solve a lot of issues.

ie:

cp known*.met known*.met.new
rehash the */new files
cp known*.met known.met.old
mv known*.met.new  known*.met.new

These files are read-only for the most part during normal operation.

This would be a pig on FAT32 systems but it's fairly painless on *nixen and NTFS and it gets around any issues caused by system crashes or SIGKILL  during writes.

 

[Stu Redman]

Rehashing is done in memory. The met files are then written in a single step.

[stoatwblr]

I know, but the fact remains that there is a race condition.

Better to write the file out and then rename it into position.

 I had similar problems with sendmail db files with a few hundred domains  to handle. In that instance it resultes in spam filters failing for a few seconds or mail not being delieverd properly - only a few seconds, but enough to cause problems.