Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[stoatwblr]

SVN 10594

If amule is killed for any reason, the known_met (or known64_met) seems to lose a lot of AICH/MD4 entries it previously had (but usually not all)...

This results in a lot of unnecessary rehashing. Not good if there are a lot of files on the shared side of things.

[Kry]

Funny that, if you throw your harddrive out of a window, it loses them all.

[Stu Redman]

That's why we should store that kind of stuff in a database instead of a met.

[stoatwblr]

That's why we should store that kind of stuff in a database instead of a met.

Alternatively, create new mets and copy them in

Even just shutting down amule seems to result in a few entries being trashed.

Currently known2 is 733Mb while known is about 200k

[Kry]

That's why we should store that kind of stuff in a database instead of a met.

You do realize that the .met format is a database?

[Stu Redman]

Kind of. Technically any file with data in it can be considered to be a "database". I had something more robust and sophisticated in mind, like SQLite. With an index. So you don't have to go sequentially through the whole thing every time in CAICHHashSet::SaveHashSet() . And transactions.

And I don't like that CAICHSyncTask::Entry() opens the met for read/write. It should read only. Writing is only for
a) creating the header if it's empty (useless, already happens in CAICHHashSet::SaveHashSet()  )
b) truncating on error - I'd rather close and reopen it in that special case!

Maybe this is causing the problem?

Anyway, if known2 is 733Mb it's gone way beyond what it was intended for.  I'd delete it and start all over from scratch. Let it rehash everything shared for once and be done.

[stoatwblr]

Anyway, if known2 is 733Mb it's gone way beyond what it was intended for.  I'd delete it and start all over from scratch. Let it rehash everything shared for once and be done.

I suspected you might suggest that.

How big was it intended to grow?

There are only a couple hundred files exported at the moment - I've cut things right back but the exports keep getting trashed.

[stoatwblr]

Even just shutting down amule seems to result in a few entries being trashed.

Specifically: Shutting down Amuled (either with HUP or a shutdown command via amulecmd) results in corruption of hashes of files that were downloading at the instant of the shutdown, resulting in them having to be rehashed before downloads can resume.

On the side of hashing of files being shared, I noticed that known2's mtime  seems to only be updated every 60 minutes. It looks like there needs to be better clean closing of the file at shutdown - and as Stu suggested, only open read/write when absolutely needed.

[Stu Redman]

I don't think rehashing of downloading files is related to known.met at all.

Look at the logfile after shutdown. Is there "aMule shutdown completed." at the end?
What about the timestamps? The .part.met must all be same or newer as the .part .

[stoatwblr]

Look at the logfile after shutdown. Is there "aMule shutdown completed." at the end?

[

Yes.

What about the timestamps? The .part.met must all be same or newer as the .part .

I'll have to check this next time I shutdown.

[stoatwblr]

What about the timestamps? The .part.met must all be same or newer as the .part .

Yes, they all are.

[stoatwblr]

fwiw after a few days it's back up to ~260Mb and back to the same problem.

[stoatwblr]

known2 eventually grew to around 550MB, then amuled sat there spinning its wheels at 100% cpu for several hours, with known.met growing to 5-7Mb then shrinking back to zero.

Everything exited apparently cleanly on a HUP, however when restarting the following mesage was logged.

$ grep -i known logfile
!2011-08-27 18:24:31: WARNING: Known file list corrupted, contains invalid header.
 2011-08-27 18:24:31: Creditfile loaded, 7986 clients are known
 2011-08-27 18:26:41: Found 384 known shared files, 18066 unknown

Amuled is again sitting at 100% with known.met cycling in size while known2 stays static.

Yes I really do have that many files on share (distros and suchlike)

As Stu says, it appears the met format can't cope with stuff this big. Perhaps some sort of run/compile time alternative such as a mysql interface needs to be considered.

Opinions?

[stoatwblr]

After a little more digging...

It may not be changing known2, but it _IS_ recalculating all the hashes on shared files.

Is this expected behaviour if it thinks the header is corrupt?

[Stu Redman]

You know, sharing 18000 files makes no sense. But what happens here shouldn't happen of course.

I just shared two wxWidgets folders with 18000 files. Yes, behavior is silly. Hashing is supposed to be a background task, but every 30 files known.met is rewritten. Completely. This looks like its size is growing/shrinking. And the writing is done in the foreground, which makes the whole hashing-in-background pointless. GUI is completely blocked.

10601 fixes the problem for me (by updating known.met only after 300Mb and not after 30 files anymore). Please try it.

(And I see I'm tired - I also slipped in my patch to open known2_64.met in read only mode for reading, which was supposed to be an extra revision.  )