I've managed to optimize my iptables filters:
iptables -t raw -A OUTPUT -p TCP -s 0/0 --dport 4662 -j NOTRACK
iptables -t raw -A OUTPUT -p TCP -s 0/0 --sport 4662 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --sport 4662 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --dport 4662 -j NOTRACK
iptables -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT
iptables -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
With this rules I do not track port 4662 and I can get high ids! The number of established connections keeps growing but at a very slow pace, slow enough to time out in five days without filling ip_conntrack.
It's still annoying, but now I do not have to reboot every 2, 3 days... Of course this will not solve the problem with the routers. :-(
I'm using CVS from 20050219, should I upgrade?