Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Stu Redman]

Stu's non-compressed EC mod dropped cpu usage AND sped up synchronisation by 90%++

What exactly do you mean? The hack I posted as a zip? That also had zlib compression.
If you want to try without compression, just raise EC_MAX_UNCOMPRESSED in ECSocket.cpp to like 10000000 and recompile amuled.

(Yes, I know I'm abusing the program by driving this hard. Call it "stress testing" )

You are generating valuable feedback. 

Switching the progress bar off made no difference to CPU.

Good, so it's not that dumb. But there is still an O(n^2) somewhere in the download view I believe.
As I said - I'll investigate, but only after the big GUI update. Maybe the problem has already been fixed there (increasing the hype...).

[lfroen]

One important point: Stu's non-compressed EC mod dropped cpu usage AND sped up synchronisation by 90%++ (seconds vs minutes)

I see. Apparently zlib calls are that costly.

(Yes, I know I'm abusing the program by driving this hard. Call it "stress testing" )

Stress testing is important, but tricky to do. Sometimes it's difficult to tell where limit is hit.

Switching the progress bar off made no difference to CPU.

So problem is probably not in drawing itself.

But there is still an O(n^2) somewhere in the download view I believe.

Not necessary. It can be O(n) with big multiplier (bad implementation).

[stoatwblr]

What exactly do you mean? The hack I posted as a zip? That also had zlib compression.

Whatever you did in that hack, it was a lot faster to sync up.

I'll try your suggestion

[stoatwblr]

I see. Apparently zlib calls are that costly.

Yikes! They are when you set this:

#define EC_COMPRESSION_LEVEL    Z_BEST_COMPRESSION

in ECSocket.cpp (Thanks for the pointer Stu.)

That's equivalent to gzip -9 - which has a _massive_ CPU overhead for a small size gain over gzip's default level 6.

Here's what SSH's man page has to say about session compression:

      -C      Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP connections).  The com‐
             pression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol
             version 1.  Compression is desirable on modem lines and other slow connections, but will only slow down things on fast net‐
             works.  The default value can be set on a host-by-host basis in the configuration files; see the Compression option.

Gzip's man page says:

       -# --fast --best
              Regulate  the  speed of compression using the specified digit #, where -1 or --fast indicates the fastest compression method
              (less compression) and -9 or --best indicates the slowest compression method (best compression).   The  default  compression
              level is -6 (that is, biased towards high compression at expense of speed).

After years of fiddling with long (slow)  links, compression level 1 or 2 are about all I bother with even on transoceanic SSH sessions (we use SSH+rsync a lot at $orkplace to keep multi-Tb data archives up to date as well as to pull in stuff from observatories worldwide) .

There's a clear  throughput penalty on local LAN with compression enabled - rsync initial sessions max out at 25-30Mb/s with it on (using i7 cpus and 8Gb+ ram) vs 60-90Mb/s with it off (disk speed is the limit with compression off on local links, vs CPU overhead  with compression on)

I'm not totally dissing client-server compression - it's useful on slow links but it's always a juggling act to balance the latency+CPU overhead caused by compression vs link latency caused by uncompressed data - and care has to be taken to not try compressing data running across an already compressed link (such as a SSH tunnel, or a modem link (It's rare for modems to not include compression these days)). That's why I keep suggesting it be switchable.

Meantime I'll try  setting compression to minimum (fastest) as a starting point and see how it behaves.

[Stu Redman]

That's why I keep suggesting it be switchable.

So why don't you just try it out and switch it off and post the result here?

[stoatwblr]

I have been.

It's better with min/no compression but works better still with your suggestion of increasing the min packet size - mainly because there's no latency in/out of zlib even though that's  doing nothing (with compression disabled)

The CPU usage is still high but I suspect that's because I still have debugging dialled right up. Despite that it's still behaving better than on the stock compression/min packet settings - but not performing  as well as your hack.

[Stu Redman]

You know you can (and should in your case) use optimize and debug together.
The hack just replaced a large and complicated piece of code with a few wx calls. But the problem was elsewhere.

Anyone against that I add a setting to disable EC compression?

[lfroen]

Anyone against that I add a setting to disable EC compression?

Not at all. And when enabled, it probably should use less aggressive setting.

P.S.
That's interesting discussion - _some_ time ago, I argued here that reworking code to save bandwidth is useless idea.

[Stu Redman]

So why was zlib EC compression implemented in the first place (and with much effort looking at the code), if saving bandwidth is useless?
My approach is: what you don't transmit you don't have to compress. Maybe the effort to compress the data outweighs the bandwidth gain. Current amulegui EC transmits much less without compression than it did before with compression.

[lfroen]

So why was zlib EC compression implemented in the first place (and with much effort looking at the code), if saving bandwidth is useless?

For 2 reasons - once upon a time, IIRC, EC was text based. So, zlib was really useful, them. I kept it, because without some kind of profiling cost of compression wasn't obvious.
Second reason is people with slow links, who want to control amule over DSL.

My approach is: what you don't transmit you don't have to compress.

You already have this idea implemented. Look for CObjTagMap in EC source. I guess it can be improved.

Maybe the effort to compress the data outweighs the bandwidth gain. Current amulegui EC transmits much less without compression than it did before with compression.

What you (or me at a time) do, is ALSO compression. Which cost both in memory (O(n) at least) and CPU. On embedded system (I tried amule on NSLU2 and AppleTV) this will be painful.

[GonoszTopi]

once upon a time, IIRC, EC was text based. So, zlib was really useful, them.

IIRC we didn't have any compression on ECv1.

[Kry]

But it certainly was text based.

[stoatwblr]

You know you can (and should in your case) use optimize and debug together.

Have been playing with that. It's a helluvalot faster (similar 90-95% reduction in sync up time), but your hack is even faster still when compiled with optimization.

(Using -O2 --unroll-loops)

The latest incarnation (svn 10089)  is partially losing contact with the server after a while - eg, throughput speeds don't update and stopping/resuming a file doesn't show any change, but restarting the GUI shows that the commands worked.

[Stu Redman]

I'm preparing a version that allows to cleanly disable zlib compression for EC.

Anybody against permanently using a wxstream instead of the complicated buffering ECSocket is doing now? That's what my hack did. It appears to speed up things under load and greatly simplifies things.

If you're still losing contact with 10089 that's frustrating.  Please enable EC debug logging and try to figure out at which point things start breaking.

[stoatwblr]

It was ok with EC contact up to 10087 if that helps.

Debugging's on but nothing's being logged when contact is lost. Do you have any  particular flag you want enabled in remote.conf other than these?

VerboseDebug=1
Cat_General=1
Cat_External\ Connect=1