aMule Forum

English => en_Bugs => Topic started by: bill_bool on January 31, 2005, 10:57:34 PM

Title: ip_conntrack: table full, dropping packet.
Post by: bill_bool on January 31, 2005, 10:57:34 PM
I think that it is since I upgraded to rc8 and 2.6.10, I've got those kind of messages, my connection is getting down, and my CPU usage grows to 99.x%

And those weird reactions :
cat: /proc/net/ip_conntrack: No space left on device

although cat /proc/sys/net/ipv4/ip_conntrack_max gives me 32760

Any Idea ?

Thanks.

PS : System is a mdk cooker, up to date.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: bill_bool on February 02, 2005, 07:54:38 AM
I finally rebooted, and the number of connections given by cat /proc/net/ip_conntrack | wc -l is back to normal (from around 14000 previously to ~700 now). I'll monitor if it stays at this correct level or continues to increase to anormal levels.

edit : after one hour : ~5000
edit2 : after 12 hours : ~15000
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Kry on February 02, 2005, 03:31:36 PM
ya, seems we're not closing connections.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: bill_bool on February 02, 2005, 10:54:36 PM
Right.

I've shutdown aMule 2 hours ago, and I still have hundreds connections shown as established...

tcp      6 419598 ESTABLISHED src=MyIP dst=XXX sport=36583 dport=4662 packets=2 bytes=112 src=XXX dst=MyIP sport=4662 dport=36583 packets=1 bytes=64 [ASSURED] mark=0 use=1

What's strange is that I get the same dest IP thousands of time, I explain :
cat /proc/net/ip_conntrack | grep XXX | wc -l
4310
cat /proc/net/ip_conntrack | grep YYY | wc -l
2881
cat /proc/net/ip_conntrack | wc -l
13307
cat /proc/net/ip_conntrack | grep MyIP | wc -l
13307
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Xaignar on February 02, 2005, 11:48:53 PM
Kry, I'm pretty sure that the retry-code is at least part of the problem. Because AFAICS, we will keep trying to connect, even after we should have dropped the connection. It seems to me that the only valid case in which we should retry is WOULD_BLOCK, and that seems to be a tiny minority of the actual cases.
Title: A me too post
Post by: hgg on February 04, 2005, 01:14:31 PM
Same thing here, I'm using:

SuSE 9.2
kernel 2.6.8-24.11-default (the stock SuSE kernel with updates)
aMule-cvs-20050130

Relevant configure options:

--enable-amulecmd --disable-cas --disable-wxcas --disable-gsocket

With an uptime of 2 days and IP filtering enabled I get:

cat /proc/net/ip_conntrack | wc -l
54733

Max number is:
cat /proc/sys/net/ipv4/ip_conntrack_max
65528

Up until now the only way I've found to avoid this is to reboot every 2 or 3 days...
Title: possible solution
Post by: Ralf1108 on February 05, 2005, 05:24:54 PM
Hi guys,


i had the same problem with the ip_conntrack. I also got thousands of connections and the ip_conntrack list was full.

My solution was :

I have an 1000/128kbit DSL  and set the upload speed to 10 KB. With this setting I got the problems with the high ip_conntrack and very little download. I lowered the value to 7 KB upload... and now I have only 28 download, but the problem with ip_conntrack has gone !!
And my download speed is also very good, at average 20-25 KB. Maybe you give it a try....


Ralf
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Xaignar on February 05, 2005, 07:49:06 PM
That's interesting. Thanks for the info.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: skolnick on February 06, 2005, 05:29:36 AM
Hi!

Just tried to see if I could reproduce the problem, but all I get is this:

[root@neptune ~]# cat /proc/net/ip_conntrack | wc -l
cat: /proc/net/ip_conntrack: No such file or directory
0
[root@neptune ~]#
2.6.9-1.667
I don't know why the file does not exist on my system. I'm using Fedora Core 3, kernel 2.6.9-1.667

Thanks.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: skolnick on February 06, 2005, 05:31:17 AM
Quote
Originally posted by Kry
ya, seems we're not closing connections.

Is this a bug? if so, could you please inform us when it is fixed? I would like to try the CVS version with that fix on it.

Thanks.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: freakyc on February 22, 2005, 08:15:27 PM
Ya, the next time using aMule after saying it was working for me in this thread (http://forum.amule.org/thread.php?threadid=3095&sid=), it did it to me again :(

I've currently switched to a packaged router now from my linux router though, at least for the moment.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: hgg on February 23, 2005, 11:27:37 AM
I've been studding this problem and these are my conclusions so far:

Code: [Select]
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A INPUT -m state --state INVALID -j DROP
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

iptables -t raw -A OUTPUT -p TCP -s 0/0 --sport 4662 -j NOTRACK
iptables -t raw -A OUTPUT -p TCP -s 0/0 --dport 4662 -j NOTRACK
iptables -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -I INPUT 1 -m state --state UNTRACKED -j ACCEPT

The last four lines are the important part because they disable the conntrack to or from port 4662. Using these rules my number o established connections remains stable. BUT I'm only  able to get low ID...
[/list]

Just as a little rant, I must say that I find this behavior very strange because this way anybody in a linux system can write a program that if runned locally will disable the network interface, effectively making DoS in the machine. Note that my machine is a standard SuSE 9.2 box without great modifications, so we are talking about a standard system...  This shouldn't be possible! :-(
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Kry on February 23, 2005, 04:03:56 PM
Interesting.

I'll start writing a local DoS tool!


Wait, I already did, it's called aMule :P
Title: Re: ip_conntrack: table full, dropping packet.
Post by: hgg on February 23, 2005, 11:41:33 PM
Some interesting info about this connection tracking business:

http://www.sns.ias.edu/~jns/security/iptables/iptables_conntrack.html

BTW: Since my last post my connection count is:

Code: [Select]
     2 CLOSE
      1 CLOSE_WAIT
    735 ESTABLISHED
      1 FIN_WAIT
      1 SYN_SENT
    139 TIME_WAIT

Much better than the 10000+ connections I would get in 8~10 hours...

And the entries in /proc/net/ip_conntrack look like this:

Code: [Select]
tcp      6 392796 ESTABLISHED src=213.22.231.55 dst=213.13.224.144 sport=13996 dport=70 src=213.13.224.144 dst=213.22.231.55 sport=70 dport=13996 [ASSURED] use=1 mark=0
tcp      6 30 TIME_WAIT src=83.28.208.239 dst=213.22.231.55 sport=2701 dport=4662 src=213.22.231.55 dst=83.28.208.239 sport=4662 dport=2701 [ASSURED] use=1 mark=0
tcp      6 431999 ESTABLISHED src=213.22.231.55 dst=81.84.12.106 sport=27475 dport=4661 src=81.84.12.106 dst=213.22.231.55 sport=4661 dport=27475 [ASSURED] use=6 mark=0
tcp      6 392786 ESTABLISHED src=213.22.231.55 dst=213.13.224.144 sport=13865 dport=70 src=213.13.224.144 dst=213.22.231.55 sport=70 dport=13865 [ASSURED] use=1 mark=0
tcp      6 392798 ESTABLISHED src=213.22.231.55 dst=213.13.224.144 sport=14021 dport=70 src=213.13.224.144 dst=213.22.231.55 sport=70 dport=14021 [ASSURED] use=1 mark=0
tcp      6 392805 ESTABLISHED src=213.22.231.55 dst=213.13.224.144 sport=14093 dport=70 src=213.13.224.144 dst=213.22.231.55 sport=70 dport=14093 [ASSURED] use=1 mark=0
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Vollstrecker on February 24, 2005, 02:43:50 AM
Without contrack, I would miss IRC support. I've just set Queue to 1000 and the problem was gone.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: bill_bool on February 27, 2005, 01:35:52 PM
Quote
Originally posted by Vollstrecker
Without contrack, I would miss IRC support. I've just set Queue to 1000 and the problem was gone.

I tried with queue set to 500, and I still have the same problem... around 5000 connections on my PC and my router stalling with approx. 3800 connections :(
Title: Re: ip_conntrack: table full, dropping packet.
Post by: hgg on February 28, 2005, 11:32:12 AM
I've managed to optimize my iptables filters:

Code: [Select]
iptables -t raw -A OUTPUT -p TCP -s 0/0 --dport 4662 -j NOTRACK
iptables -t raw -A OUTPUT -p TCP -s 0/0 --sport 4662 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --sport 4662 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --dport 4662 -j NOTRACK

iptables -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT
iptables -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT

With this rules I do not track port 4662 and I can get high ids! The number of established connections keeps growing but at a very slow pace, slow enough to time out in five days without filling ip_conntrack.

It's still annoying, but now I do not have to reboot every 2, 3 days... Of course this will not solve the problem with the routers. :-(

I'm using CVS from 20050219, should I upgrade?
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Vollstrecker on March 12, 2005, 10:36:57 PM
Are these rules set in Addition to the nromal rules, or in place of them?
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Vollstrecker on March 13, 2005, 10:17:45 PM
I have now the following setuo, and get an Low ID, Without the last 8 lines I have an High one, but the contrack problem. Whats wrong with these rules?

$IPT -t nat -A POSTROUTING -o $EXTIF -p TCP -s $Werner1 --sport 4662 -j MASQUERADE
$IPT -t nat -A POSTROUTING -o $EXTIF -p TCP -s $Werner1 --dport 4662 -j MASQUERADE
$IPT -t nat -A POSTROUTING -o $EXTIF -p UDP -s $Werner1 --dport 4672 -j MASQUERADE
$IPT -t nat -A POSTROUTING -o $EXTIF -p TCP -s $Werner1 --dport 4661 -j MASQUERADE
$IPT -t nat -A POSTROUTING -o $EXTIF -p UDP -s $Werner1 --dport 4665 -j MASQUERADE
$IPT -t nat -i $EXTIF -I PREROUTING -p tcp --dport 4662 -j DNAT --to-destination $Werner1:4662
$IPT -t nat -i $EXTIF -I PREROUTING -p udp --dport 4665 -j DNAT --to-destination $Werner1:4665
$IPT -t nat -i $EXTIF -I PREROUTING -p udp --dport 4672 -j DNAT --to-destination $Werner1:4672
$IPT -I FORWARD -p TCP --sport 4662 -j ACCEPT
$IPT -I FORWARD -p TCP --dport 4662 -j ACCEPT
$IPT -I FORWARD -p UDP --dport 4672 -j ACCEPT
$IPT -I FORWARD -p TCP --dport 4661 -j ACCEPT
$IPT -I FORWARD -p UDP --dport 4662 -j ACCEPT
$IPT -I FORWARD -p UDP --dport 4662 -j ACCEPT
$IPT -I FORWARD -p UDP --dport 4665 -j ACCEPT
$IPT -I FORWARD -p UDP --dport 4672 -j ACCEPT

$IPT -t raw -A OUTPUT -p TCP -s 0/0 --dport 4662 -j NOTRACK
$IPT -t raw -A OUTPUT -p TCP -s 0/0 --sport 4662 -j NOTRACK
$IPT -t raw -A PREROUTING -p TCP -s 0/0 --sport 4662 -j NOTRACK
$IPT -t raw -A PREROUTING -p TCP -s 0/0 --dport 4662 -j NOTRACK
$IPT -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
$IPT -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT
$IPT -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
$IPT -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
Title: Re: ip_conntrack: table full, dropping packet.
Post by: GonoszTopi on March 14, 2005, 08:19:49 PM
$IPT -t raw -A OUTPUT -p TCP -s 0/0 --dport 4662 -j NOTRACK
$IPT -t raw -A OUTPUT -p TCP -s 0/0 --sport 4662 -j NOTRACK
$IPT -t raw -A PREROUTING -p TCP -s 0/0 --sport 4662 -j NOTRACK
$IPT -t raw -A PREROUTING -p TCP -s 0/0 --dport 4662 -j NOTRACK
$IPT -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
$IPT -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT
$IPT -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
$IPT -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT

Vollstrecker, give it a try without the red lines.

Note: and you have
$IPT -I FORWARD -p UDP --dport 4665 -j ACCEPT
twice ...
Title: Re: ip_conntrack: table full, dropping packet.
Post by: skolnick on March 14, 2005, 09:52:30 PM
Quote
Originally posted by GonoszTopi
Note: and you have
$IPT -I FORWARD -p UDP --dport 4665 -j ACCEPT
twice ...
In fact the repeated line is this:

$IPT -I FORWARD -p UDP --dport 4662 -j ACCEPT

Regards.
Title: Re: ip_conntrack: table full, dropping packet.
Post by: Vollstrecker on March 14, 2005, 09:57:06 PM
I commented out the red ones, but nothing changed.