Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Mr Faber]

When I was using eMule I used USS to manage the upload of it so eMule uses the maximum possible upload without killing my connection. I could still play online or dowloading something without problems. It wasn't perfect but it works in most cases for me.
After I switched to Linux and with it to aMule I decided very early to use traffic shaping. From theory traffic shaping is much more better than USS.
But it doesn't work together with aMule . I think that my config is correct but it doesn't work. The upload of aMule gets high to 50, which is the limit (for aMule and HTB), for some seconds and goes slowly down to zero or a little higher than zero. After some seconds/minutes it get high again. If I remove the uploadlimit and put zero in the preferences the upload gets up to 70 which can't be possible according to my connection and my traffic shaping limit and then fast down to zero for a long time and up and so on.
Others seems to have the same problem as I have had readed in the german section so it seems to have something to do with aMule?!
Does anyone know a solution or have the same problem? Traffic shaping is so great if it would work as expected

cu
Mr Faber

[GonoszTopi]

Mr Faber,

I also use traffic shaping, and also have this kind of "problem" (quotation marks because I don't really care). The thing is, it's not a problem, it's just your side of the HTB queue. Delays and spikes are in a close relation to queue length. Say you have a queue length of 100 - i.e. the shaper will queue 100 packets for you. aMule will fill those 100 packets in no time - therefore a spike in the upload speed, and won't be able to send any more data (will be dropped, if it does) until a certain percentage of the packets get dequeued. This will show in your graph as 0.0.
On the other side of the shaper there's a smooth flow of traffic - you can see it, if you put another machine between the one with the shaper and the internet connection, as a gateway. You'll see a constant steady upload speed.

You may be able to reduce those "incative" times and lower your spikes by reducing queue length on your HTB shaper. But keep in mind that shorter queues also affect badly the shaper performance itself, i.e. it won't be able to shape so nicely.

[lfroen]

T

But keep in mind that shorter queues also affect badly the shaper performance itself, i.e. it won't be able to shape so nicely.

That is not exactly true. Size of queue you choose (in packets) is related to size of packets and to rtt you going to overcome. All linux shapers works in bps, not packets. But generally you don't choose internal size of htb queue.

What you do control, is size of bursts and inter-class priority. Those can highly affect amule performance, due to kind of packets it sends.

Mr.Faber - don't trust amule in statistics. Take ethereal and measure real bandwidth for each class coming out of eth0 (or ppp0 for adsl). Since amule instead of talking to network, talks to htb arbiter, values you see are almost meaningless.

You also can request htb own stats, which reflect exactly what's going on.

[Mr Faber]

Thank you for your replies. The problem is that my upload is in most cases much more lower than before shaping and my download can't come high because of the upload of zero over some time.
@lfroen
Yes, I thought the same but after let aMule running for two or three hours and no improvement and very bad download I disabled HTB.
I will just post my configuration. Maybe I can tune something.

I use shorewall as firewall and traffic shaping configurator.
/etc/shorewall/tcrules

Code: [Select]

#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
1 $FW 0.0.0.0/0 all
2 192.168.0.2 0.0.0.0/0 all

/etc/shorewall/tcstart

Code: [Select]

run_tc qdisc add dev eth0 root handle 1: htb default 40

run_tc class add dev eth0 parent 1: classid 1:1 htb rate 500kbit
#server
run_tc class add dev eth0 parent 1:1 classid 1:10 htb rate 80kbit ceil 500kbit prio 0
#pc
run_tc class add dev eth0 parent 1:1 classid 1:20 htb rate 128kbit ceil 500kbit prio 1
#amuled
run_tc class add dev eth0 parent 1:1 classid 1:30 htb rate 96kbit ceil 500kbit prio 2
#net
run_tc class add dev eth0 parent 1:1 classid 1:40 htb rate 128kbit ceil 128kbit prio 1

run_tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
run_tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
run_tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10
run_tc qdisc add dev eth0 parent 1:40 handle 40: sfq perturb 10

run_tc filter add dev eth0 protocol ip parent 1:0 handle 1 fw flowid 1:10
run_tc filter add dev eth0 protocol ip parent 1:0 handle 2 fw flowid 1:20
run_tc filter add dev eth0 protocol ip parent 1:0 handle 3 fw flowid 1:30

To realize aMule packets I mark them with this command in the file /etc/shorewall/start

Code: [Select]

############################################################################
# Shorewall 2.2 -- /etc/shorewall/start
#
# Add commands below that you want to be executed after shorewall has
# been started or restarted.
#
iptables -t mangle -A OUTPUT -m owner --cmd-owner amuled -j MARK --set-mark 3

Today I have reenabled HTB and find something interesting. I didn't change the internal upload limit of 40 of aMule. I wait some time and the upload was constant 40 because no other application stealing the bandwith and aMule never gets the highest point. After disabling the upload limit I get the same problem as before like GonoszTopi says.
But I want to give more if I don't use the connection so that isn't a real solution for me. Maybe the aMule uploadalgorithm is to strict. As far as some packets blocked or dropped it directly slow down the upload very much. Maybe there can be in future an option for more aggressive upload for traffic shapers

cu
Mr Faber

[Mr Faber]

A solution can be as described before a new upload algorithm which doesn't stop the transfer if some packets doesn't send. I should slow down in maybe 1 KB parts until no more packets dropped. The same algorithm to upload more. Something like the eMule USS would be good but of course without pings. It seems to have a stochastic algorithm.

cu
Mr Faber

[GonoszTopi]

A solution can be as described before a new upload algorithm which doesn't stop the transfer if some packets doesn't send. I should slow down in maybe 1 KB parts until no more packets dropped.

There's no need to reinvent the wheel. It's implemented in TCP level, and you can do nothing about it, except rewriting the kernel. What you said, is actually happening, just you don't see it, bacuse it's all low-level thing. aMule is a high level application (the highest in OSI model), and it does not know, and does not have to know what's happening in lower levels. What it only knows, is when he sends a packet, it will arrive (most probably, network errors cannot be excluded).

[Mr Faber]

It seems to make sense but bitorrent haven't this problem (on windows, I haven't checked it with linux) so it can't be only a tcp problem. Another point is that if I set the limits to zero the upload stats seems to be wrong in nearly all cases.
I will check it on linux too.

cu
Mr Faber

[lfroen]

Mr.Faber: you seem to misunderstand what traffic shaping is. Also you seem to have absolutly no idea about "how network works. and how tcp works in particular". Please, take some time to educate yourself by reading appropriate literature before "do-it-yourself". I seriosly mean it.
Tcp together with shaping is quite complicated topic. In few words: it doesn't work as you think. Not at all.

[Mr Faber]

I know that traffic shaping isn't the Holy Grail. I am not as stupid as it seems .
Ok, maybe I have no idea of tcp/traffic shaping and so on but why bittorrent can upload constant (without going to zero up and back to zero again) and aMule not? I don't want to blame you I just want to upload as much I can. Through traffic shaping I have realized that my connection can offer more than 60 KB (like the isp says) without high ping results because the tcp overhead is included and aMule can use 55 KB of it. Thats more than on windows.
By the way I have read much about traffic shaping of course not enough Linux and everything based on it is a NeverEnding Story if I want to read all availabel docs

cu
Mr Faber

[lfroen]

I am not as stupid as it seems

Nobody calls you stupid. Lack of education is not stupidity. There's nothing wrong to realize that you wrong because subject is more complicated that you thing.

why bittorrent can upload constant (without going to zero up and back to zero again) and aMule not?

I'm also using traffic shaping and never have this problem. But I'm using SFQ and not HTB. If you really want to understand what's going on:
1. Make sure you understand http://www.amule.org/wiki/index.php/FAQ_network
2. measure traffic right - use ethereal with correct filter, not amule stats
3. compare measurement with stats from tc. Check shaper algorithm.
4. Now you can make calculation and compare them to amule stats

[Mr Faber]

I would say that I know how the ed2k networks work, not in every detail but the theory. Thats why I want to upload more
Yeah, thats what I have to do.
Ok, maybe I made a big mistake but as far as I know SFQ is not a shaping algorithm It is an queue algorithm which I use too. If you would check my second post in this thread you can see my tc config. I use for every queue SFQ which means as far as I know Stochastic Fairness Queueing. Maybe you can give me some tipps whats wrong.

Can you please test something too? Can you remove your upload limit in aMule and see what it does? Does it work stable for you with traffic shaping? Do you use traffic shaping to give aMule as much bandwith as it can use or just to slow down aMule if you need the upload bandwith. aMule upload with limits of 40 or 50 works pretty fine (constant upload) for me if no other use the bandwith so tc have to drop or queue nothing but as far as bandwith it needed aMule shows the described behaviour.
How much upload you can offer? Maybe it has something to do with upload.

cu
Mr Faber

[lfroen]

Ok, maybe I made a big mistake but as far as I know SFQ is not a shaping algorithm It is an queue algorithm which I use too. If you would check my second post in this thread you can see my tc config. I use for every queue SFQ which means as far as I know Stochastic Fairness Queueing. Maybe you can give me some tipps whats wrong.

Don't let the names to confuse you. All traffic shaping is about 2 things:
1. Queue the packet to some queue (may be different queue for different packets)
2. When algorithm tell "ready" - choose some not-empty queue, take packet from there and send it.
3. Repeat 2 untill there're at least 1 not-empty queue

Do you use traffic shaping to give aMule as much bandwith as it can use or just to slow down aMule if you need the upload bandwith

The whole purpose of shaping is to"slow down". How exactly slow is determined by algorithm settings.

My uplink speed is limited by ISP to 90kbps. My tc distinguish between "interactive traffic" (http, ssh, netmeeting, etc) and "bulk" (like amule, ftp). Interactive traffic get strict priority over "bulk". You should do the same, basically. Unless you have other needs I don't know about.

[Mr Faber]

Can you post your config? Maybe I find my mistake or can tune my config.
Hey, the wiki is very great, thanks. It's more technical than other information about ed2k.
Maybe it has something to do with ACK and Co packets. I am going to try some optimizations with this packets.

cu
Mr Faber

[lfroen]

Code: [Select]

# root CBQ
tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit

# shape everything @ upload speed
tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit allot 1500 prio 5 bounded isolated

# high prio class 1:10:
tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit allot 1600 prio 1 avpkt 1000

# bulk and default class 1:20 - gets slightly less traffic,
# and a lower priority:
tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit allot 1600 prio 2 avpkt 1000

# both get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
Take a look here http://lartc.org/howto/ to find out what it means. I'm not posting filters (packet classification) section, which is present, but has no relevance to this discussion.

[Mr Faber]

I have changed my tc config. I use still a separate class for aMule but this is separated in two parts too (one for bulk and one for ack packets). But aMule still shows the same behaviour. What can I do?
By the way I have used the wondershaper config for the aMule class but it doesn't work. Only the the bulk class was used.
I use for all other important classes the wondershaper config with success.

Code: [Select]

#amuled
iptables -t mangle -A OUTPUT -m owner --cmd-owner amuled -d 0.0.0.0/0 -j MARK --set-mark 231
#amuled connection attemp packets
iptables -t mangle -A OUTPUT -m owner --cmd-owner amuled -m length --length :64 -d 0.0.0.0/0 -j MARK --set-mark 232

VER}kbit burst 6k prio 2
tc class add dev eth0 parent 1:2 classid 1:23 htb rate 88kbit ceil $(($UP_CEIL_SERVER*9/10))kbit burst 6k prio 3

tc class add dev eth0 parent 1:23 classid 1:231 htb rate 80kbit ceil $(($UP_CEIL_SERVER*9/10))kbit burst 6k
tc class add dev eth0 parent 1:23 classid 1:232 htb rate 8kbit ceil $(($UP_CEIL_SERVER*9/10))kbit burst 6k

tc qdisc add dev eth0 parent 1:231 handle 231: sfq perturb 9
tc qdisc add dev eth0 parent 1:232 handle 232: sfq perturb 11

# amuled
tc filter add dev eth0 parent 1: protocol ip prio 15 handle 231 fw flowid 1:231
tc filter add dev eth0 parent 1: protocol ip prio 10 handle 232 fw flowid 1:232
cu
Mr Faber