aMule Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

We're back! (IN POG FORM)

Author Topic: Don't stack "reload shared"  (Read 2613 times)

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Don't stack "reload shared"
« on: April 15, 2012, 02:44:34 AM »

Every time adjustments are made to the categories, amule(d) reloads its sharelist.

This means if a bunch of tweaks are being made, quite a few can get stacked into the command queue - lots of CPU taken and it can take quite a while to run if there are a lof of files shared.

I think it would be best to discard any new reload commands if there's already one being executed.

Logged

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #1 on: April 15, 2012, 02:52:04 AM »

On a similar note - "reload shared" can easily generate so much load that all existing transfers (in and out) are dropped - even when not rehashing files which may have changed names (I usually rename files whilst moving 'em to their final location)

That's one of the reasons having several stacked is a bad thing.

Dropping the thread's priority might help a bit. It's not critical that the share list is available "now, this very instant" most of the time.
Logged

Vollstrecker

  • Global Moderator
  • Hero Member
  • *****
  • Karma: 65
  • Offline Offline
  • Posts: 1471
  • Unofficial Debian Packager
    • http://vollstreckernet.de
Re: Don't stack "reload shared"
« Reply #2 on: April 15, 2012, 12:54:14 PM »

Maybe the other way around would be better. I new reload is discarded, and the old thread is already finished with the file you renamed, it can cause confusion. I think it's better to just abort running thread, and start a new one.
Logged
Homefucking is killing prostitution

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #3 on: April 15, 2012, 03:07:45 PM »

The problem with that way round is that impatient users may never allow a reload to complete.

Some feedback would be essential if "abort running one" is the default.

When renaming files, I tend to rename/move entire blocks - 10-50 files at a time, then "reload shared" when all that is done.


That option isn't offered when tweaking categories (Why is a reload shared needed when changing categories around anyway?)

Logged

Stu Redman

  • Administrator
  • Hero Member
  • *****
  • Karma: 214
  • Offline Offline
  • Posts: 3830
  • Engines screaming
Re: Don't stack "reload shared"
« Reply #4 on: April 15, 2012, 04:23:44 PM »

(Why is a reload shared needed when changing categories around anyway?)
Because you could have changed the category directory.

Please estimate how much time you spend changing cats, how much time aMule is just running on its own, and thus how much overall gain is to be expected from such a change.
Logged
The image of mother goddess, lying dormant in the eyes of the dead, the sheaf of the corn is broken, end the harvest, throw the dead on the pyre -- Iron Maiden, Isle of Avalon

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #5 on: April 15, 2012, 07:10:21 PM »

Amount of time: not much, BUT it tends to be in batches, so having 5-15 reloads stacked is bad news.

Even if changing the cat directory, that's only going to affect files saved in from that point - and if it needs rescanning why not scan just that directory instead of the whole lot?

$ wc -l /amule/shareddir.dat
1852 /amule/shareddir.dat

 2012-04-15 02:11:58: SharedFileList.cpp(377): Found 24403 known shared files, 2 unknown

Scanning that lot  (~7Tb) adds up - especially when the storage is on NFS, not directly attached - current completion time runs to about 5-7 minutes per run, more if rehashing is required.

Logged

Stu Redman

  • Administrator
  • Hero Member
  • *****
  • Karma: 214
  • Offline Offline
  • Posts: 3830
  • Engines screaming
Re: Don't stack "reload shared"
« Reply #6 on: April 16, 2012, 08:51:53 PM »

I don't say your idea doesn't make sense. I just say it's complicated to implement and nobody except you with your lots of shared files will notice a difference (unless I introduce some new bug in the process). That's why I'm against it.
What do you change when you adjust your cats?

Even if changing the cat directory, that's only going to affect files saved in from that point - and if it needs rescanning why not scan just that directory instead of the whole lot?
You would have to remove the files from the old directory first from your batch of shared files, which is not as easy as it sounds. You have to consider duplicates and all that.
Logged
The image of mother goddess, lying dormant in the eyes of the dead, the sheaf of the corn is broken, end the harvest, throw the dead on the pyre -- Iron Maiden, Isle of Avalon

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #7 on: April 17, 2012, 12:16:59 PM »

Quote
What do you change when you adjust your cats?

Adding, removing, renaming, loading files into them, or transferring some of the content to another cat.

I don't generally change the directory they use away from the default.

Logged

Stu Redman

  • Administrator
  • Hero Member
  • *****
  • Karma: 214
  • Offline Offline
  • Posts: 3830
  • Engines screaming
Re: Don't stack "reload shared"
« Reply #8 on: April 17, 2012, 10:00:54 PM »

All that shouldn't require a reload. I'll take a look.
Logged
The image of mother goddess, lying dormant in the eyes of the dead, the sheaf of the corn is broken, end the harvest, throw the dead on the pyre -- Iron Maiden, Isle of Avalon

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #9 on: April 26, 2012, 08:12:16 PM »

How about checking on cat directory changes to see if the dir is already in sharedir.dat and not reloading if it is?

Logged

Stu Redman

  • Administrator
  • Hero Member
  • *****
  • Karma: 214
  • Offline Offline
  • Posts: 3830
  • Engines screaming
Re: Don't stack "reload shared"
« Reply #10 on: April 27, 2012, 10:19:37 PM »

That won't do. If cata is in aaa and catb is in bbb and you change catb to aaa, then aaa is already shared, but you still have to reload to unshare bbb.
Logged
The image of mother goddess, lying dormant in the eyes of the dead, the sheaf of the corn is broken, end the harvest, throw the dead on the pyre -- Iron Maiden, Isle of Avalon

stoatwblr

  • Sr. Member
  • ****
  • Karma: 12
  • Offline Offline
  • Posts: 317
Re: Don't stack "reload shared"
« Reply #11 on: June 17, 2012, 01:12:11 AM »

Hmm. I was hoping there would be an easier way.

It's a bit like sendmail  db admin stuff. When there are only a few rules in place then reloading works well, but as the number of rules (or files in this case) increases, there's more sense in being able to reload in a sidethread or to add/remove rules on the fly.

There are major performance penalties for sharing lots of files (I reduced what I'm sharing by 90% over the last few days and got noticeable improvement in outbound/inbound bandwidth stability (it was all over the place), as well as significantly higher averages (mostly caused by outbound b/w not falling to zero for 30 seconds at a time - or 4-6 minutes during a reload)

Logged