aMule Forum
English => Feature requests => Topic started by: hopelessone on December 24, 2008, 01:22:02 PM
-
Hi Ya,
I would like to see:
to be able to load more than 600 + files in paused mode without aMule crashing
...i have 30 downloads going and every time i get above 600+ downloads waiting on paused mode ...i get the error 24 random cannot write... i been waiting for a feature like this..
Thanks...
-
Raise your system's ulimits for open files.
-
amuled will likely to crash in this case anyway (when fd number will pass 1023)
-
Then raise it.
-
I have had more than 800 files queued on my aMule and it works fine. Using it under fedora 10 with default ulimit settings.
Regards.
BTW: Why is the 1023 limit you talk about lfroen? I have loaded like 1300 torrents on azureus and it works, so if aureus can, i guess amule can do it also. Is it something to do with the C++ structure or something?
Regards.
-
No, the aMuled code. amuleD
-
:D It seems I didn't read properly the post. However I do not get why amule can but amuleD can't if AFAIK they share tons of code.
Regards.
-
But not the low-level network one. A simple change would suffice to increase it, of course. Or even make it as big as ulimit's
-
Raise your system's ulimits for open files.
We could also make a pool of, say, 50 file handles for the downloads instead of permanently assigning one handle to each. And then assign the handles at need.
But not the low-level network one.
Can we change that when we switch to wx 3.0 (when wx-sockets become available for non-GUI apps) ?
-
If we ever do.
-
Mumble mumble... I didn't understand the problem. I though was sufficient a number of file descriptors like
upload connections + download connections + uploading/hashing files
but I noted that partial files require always an open file descriptor, from PartFile.h
CFile m_hpartfile; //permanent opened handle to avoid write conflicts
which conflicts refer this comment??
-
It's just a lazy implementation imho. It would of course be possible to make a pool of filehandles for the downloads and release the longest unused one to save handles. Well - it had been possible at least until you came up with the mapping code. ;)
-
It's just a lazy implementation imho. It would of course be possible to make a pool of filehandles for the downloads and release the longest unused one to save handles. Well - it had been possible at least until you came up with the mapping code. ;)
Oh my god :) Well... believe or not is instead a way to support more than 1024 open files!! Well... I have still to test this idea but open a file with open, mmap it, close file descriptor, now you have mmap-ed area which you can write. Then you have remapXXX (I don't remember XXX) which is Linux extension which allow you to move the area without using file descriptor :) However I think it would be better (if not already implemented) to have some sort of queuing long processes (like hashing) in order to limit disk seeking (wait to finish an hashing before starting with another file) and file opening.
I still don't understand write conflicts problems... a PartFile represent a partial file and real file is accesses using the PartFile object. Obviously you can't handle two tasks on same PartFile at the same time so you use m_hpartfile serialized so I don't see any problem if I open file before operation and close it just after read/write.
About cache I would suggest a LazyClose/LazyOpen either in PartFile or in CFile that don't close file if there are less than X file "lazy opened". A simple counter and a fixed array suffice.
-
Before you start digging into it:
- The 1024 is only a amuled issue, not an amule issue. With amule, file handles are unlimited (if they are unlimited in your OS at least). For amuled there is a chance to get rid of the limit and the legacy code when wx 3.0 comes out and sockets/events for non-gui apps become available.
- 600 downloads at a time is overdoing things anyway. A machine capable of that can also run amule instead of amuled and so work around the problem.
- I don't want to know what happens if you open a partfile, memory-map part of it for reading, close it (still mapped), reopen it, and then write to it. Especially on the multiple platforms we support. No, thanks (shiver).
- Full hashing is done on background thread (single part hashing is done in the main thread). Download is finished then anyway, and upload is suspended meanwhile. Otherwise access is single-threaded, so there are no write conflicts. Don't trust any ole comment lying around. ;)
Bottom line: I would just leave it like it is. The risk outweights the gain imho.
-
The 1024 is only a amuled issue, not an amule issue. With amule, file handles are unlimited (if they are unlimited in your OS at least). For amuled there is a chance to get rid of the limit and the legacy code when wx 3.0 comes out and sockets/events for non-gui apps become available.
?? I though 1024 was a OS system limit so it would be a limit even for amule GUI... I think that even wx will use a file descriptor for sockets and files... I'm not digging this stuff that much... just I would like to know some aMule detail...
-
@Stu: wxBase-2.x does have sockets.
@freddy77: The limitation is that wxBase doesn't support events so we had to write our own event loop using system functions like select(2).
-
@Stu: wxBase-2.x does have sockets.
So why are we using different socket code in amuled than in aMule? I'm refering to WoWoW30 (http://svn.wxwidgets.org/viewvc/wx/wxWidgets/trunk/docs/publicity/WoWoW30.html?view=co):
support to wxBase was added for events loops, timers and sockets for writing event-based client or server apps with wxWidgets 3.0
-
Because it doesn't have an event loop and we don't support 3.0 (and won't switch the code to it until a) it's out and b) it's been out for a while and everyone uses it).
-
Before you start digging into it:
- The 1024 is only a amuled issue, not an amule issue. With amule, file handles are unlimited (if they are unlimited in your OS at least). For amuled there is a chance to get rid of the limit and the legacy code when wx 3.0 comes out and sockets/events for non-gui apps become available.
- 600 downloads at a time is overdoing things anyway. A machine capable of that can also run amule instead of amuled and so work around the problem.
I have a _lot_ more than 600 requests open (mostly old/rare stuff that's been in the queue for months and only shows up for a few hours at a time).
Above about 1500 entries (Stopped!) in the queue, Amule is highly unstable and drops out with Wx errors at the slightest provokation - anything such as changing views, or opening the preferences window.
Switching from Stopped to Waiting makes things even more unstable.
This is happening in 2.2.4 and in SVN
(Linux, 100k file handle limt)
There are definitely stability issues that need looking at. While the developers may feel that I'm crazy to push things to breaking point, it's usually at breaking point that security issues and trivially fixed bugs may show up
Holding filehandles open unecessarily is counterproductive - AND it means that disk space isn't being released when files are deleted out of the Incoming directory (ie, moved to another partition).
-
Above about 1500 entries (Stopped!) in the queue, Amule is highly unstable and drops out with Wx errors at the slightest provokation - anything such as changing views, or opening the preferences window.
AMule or aMuled?
Can you post a backtrace?
Holding filehandles open unecessarily is counterproductive
Not if 99000 are still left.
- AND it means that disk space isn't being released when files are deleted out of the Incoming directory (ie, moved to another partition).
Only temp files get a permanent handle, not completed shared files. So you're wrong.
I admit that it makes no sense assigning a file handle for paused downloads without completed parts, and I'd like to change that.
-
I admit that it makes no sense assigning a file handle for paused downloads without completed parts, and I'd like to change that.
There was an interesting approach by some dude here long time ago that closed the handles when they weren't used for a while. You can search for it.
-
Above about 1500 entries (Stopped!) in the queue, Amule is highly unstable and drops out with Wx errors at the slightest provokation - anything such as changing views, or opening the preferences window.
AMule or aMuled?
Amule. Amuled is already borked at 1024.
Can you post a backtrace?
Here? or in the Backtrace area forum?
I'll have to load up a version with debugging symbols. Is SVn preferred?
Holding filehandles open unecessarily is counterproductive
Not if 99000 are still left.
Not many people are going to go and tweak their ulimits.
- AND it means that disk space isn't being released when files are deleted out of the Incoming directory (ie, moved to another partition).
Only temp files get a permanent handle, not completed shared files. So you're wrong.
OK.
I admit that it makes no sense assigning a file handle for paused downloads without completed parts, and I'd like to change that.
Even in "waiting" state, there's no need to keep the filehandle open. Please try not to emulate Windows resource-hogging operational models. It makes things difficult to run on older hardware (I just updated from a 1Gb P41.8 to a 2Gb P4 3.0. In general it's all I need for Linux and this comes from $orkplace junkbox...)
-
You realize you upload in addition to download, right?
EDIT: Other than files with no complete parts, of course.
-
You realize you upload in addition to download, right?
Of course - but the program should only need to open filehandles when upload requests come in (or when downloading, of course)
Again: Why hold filehandles open if they're not needed? It's wasteful.
-
Because it's easy and makes it faster to access the files when the requests come. OTOH of course we shouldn't do it, and again, we have already sad it should be fixed.
-
This is the backtrace I'm seeing in the version without debugging symbols enabled.
----------------------------=| BACKTRACE FOLLOWS: |=----------------------------
Current version is: aMule 2.2.4 using wxGTK2 v2.8.9
Running on: Linux 2.6.28-11-generic i686
[2] wxString::~wxString() in amule [0x8086311]
[3] wxFatalSignalHandler in /usr/lib/libwx_baseu-2.8.so.0[0xb75bef26]
[4] ?? in [0xb7ee6400]
[5] wxStringTokenizer::~wxStringTokenizer() in amule [0x8096396]
[6] CryptoPP::FileSource::~FileSource() in amule [0x80bd2f3]
[7] wxStringTokenizer::~wxStringTokenizer() in amule [0x80962b6]
[8] wxTimer::IsRunning() const in amule [0x81e88e8]
[9] std::vector<bool, std::allocator<bool> >::_M_fill_insert(std::_Bit_iterator, unsigned int, bool) in amule [0x80c7fc3]
[10] wxString::~wxString() in amule [0x808adf8]
[11] wxAppConsole::HandleEvent(wxEvtHandler*, void (wxEvtHandler::*)(wxEvent&), wxEvent&) const in /usr/lib/libwx_baseu-2.8.so.0[0xb751b2a1]
[12] wxEvtHandler::ProcessEventIfMatches(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&) in /usr/lib/libwx_baseu-2.8.so.0[0xb75ba60a]
[13] wxEventHashTable::HandleEvent(wxEvent&, wxEvtHandler*) in /usr/lib/libwx_baseu-2.8.so.0[0xb75bb824]
[14] wxEvtHandler::ProcessEvent(wxEvent&) in /usr/lib/libwx_baseu-2.8.so.0[0xb75bb92b]
[15] wxEvtHandler::ProcessPendingEvents() in /usr/lib/libwx_baseu-2.8.so.0[0xb75baa89]
[16] wxAppConsole::ProcessPendingEvents() in /usr/lib/libwx_baseu-2.8.so.0[0xb751b601]
[17] wxAppBase::ProcessIdle() in /usr/lib/libwx_gtk2u_core-2.8.so.0[0xb781f58f]
[18] ?? in /usr/lib/libwx_gtk2u_core-2.8.so.0 [0xb776e123]
[19] ?? in /usr/lib/libglib-2.0.so.0 [0xb6b5ac81]
[20] g_main_context_dispatch in /usr/lib/libglib-2.0.so.0[0xb6b5cb88]
[21] ?? in /usr/lib/libglib-2.0.so.0 [0xb6b600eb]
[22] g_main_loop_run in /usr/lib/libglib-2.0.so.0[0xb6b605ba]
[23] gtk_main in /usr/lib/libgtk-x11-2.0.so.0[0xb6fa07d9]
[24] wxEventLoop::Run() in /usr/lib/libwx_gtk2u_core-2.8.so.0[0xb7787cb5]
[25] wxAppBase::MainLoop() in /usr/lib/libwx_gtk2u_core-2.8.so.0[0xb781f3fe]
[26] wxAppBase::OnRun() in /usr/lib/libwx_gtk2u_core-2.8.so.0[0xb781efd1]
[27] wxEntry(int&, wchar_t**) in /usr/lib/libwx_baseu-2.8.so.0[0xb7554eea]
[28] wxEntry(int&, char**) in /usr/lib/libwx_baseu-2.8.so.0[0xb75550e7]
[29] std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf() in amule [0x815fff0]
[30] __libc_start_main in /lib/tls/i686/cmov/libc.so.6[0xb723b775]
[31] ?? in amule [0x8085121]
-
There was an interesting approach by some dude here long time ago that closed the handles when they weren't used for a while. You can search for it.
You mean this one (http://www.amule.org/amule/index.php?topic=11655)? Too bad the patch is not attached anymore. :(
Does someone still have it?
I like the approach of a CFileAutoClose class that handles opening/closing internally. And a simple policy like closing after 10min of inactivity should do. Really, if a client has waited 2 hours for a download slot he won't notice 20ms required to open a file. ;)
-
Ooh! That does sound tasty. Please implement. Meanwhile, I'll just continue to jack up my maxfilesperproc to some ridiculous number ;)
-
Weird how the attachments are gone, this might have been before the forum migration...
... wait, since when do I have this awesome memory?
-
BTW stu, careful with such a class. Functions don't get called magically, so you would need to poll the classes for status/timeout/etc...
-
Got it running. I'm downloading 1436 files using amuled right now. 8)
The downside is, the gui (amule or amulegui) is behaving poor with such a number of files. The DownloadListCtrl hogs all CPU showing a cat with >1000 files. We really need to something about that.
-
Like make it virtual.
-
Of course we're fucked becuae I have to finish my changes first.
So how about that's what I do this weekend.
-
Deja vu... ;)
-
You never know when it's true until it happens.
-
Done in 9585. :)
-
You never know when it's true until it happens.
No, not even then.
-
Amuled is running now happily with more than 2000 files in "waiting" state.
It still gives Wx errors occaisonally.
-
Moved this issue here (http://www.amule.org/amule/index.php?topic=16926).