Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[sirius]

Hi, i'm downloading a large file (2.64 GB) and got some strange readings.

Please have a look at the picture ...

A while ago, it read:

2,64GB Size
3,21GB Downloaded
1,92GB Finished

The download continued and now it reads:
2,64GB Size
0,09GB Downloaded (99,32 MB)
2,02GB Finished

Still, it reads 2,59 GB lost through errors
530 MB gained through compression

So for me it appears that there is a buffer overrun problem.

Might be in the area of nearly exactly 4 GB (2,59GB+2,02GB - 99,32+530,03 MB = 4091,29 MB which could be 4096 MB including rounding errors) if my calculations are correct. Or somewhere else, I can of course only assume on how the numbers works together ... still, its damned close to 4 GB.

But if I'm right, this could be a build in limit that should be removed ...

(Addition: Kry, I'm sorry, I nearly forgot the colours for you, but in the little time  I had, I at least managed to put some red in for you to read :-)

[phoenix]

sirius,

Didn't you stop your download in the mean time? Like restarted aMule or something? I am not sure now, would have to check the code, but I believe that the "Downloaded" part is how many bytes you downloaded at this session, the finished part is how much "good parts" you have downloaded, good in the sense of good hash. So, if for some reason you stopped downloading, maybe this counter has been reset.

You see, your finished part has increased from 1,92GB to 2,02, which is about 100 MB (99.32MB someone? ), and besides the coincidence, this means that the file download is progressing

Cheers!

[GonoszTopi]

phoenix, "Downloaded" == "Transferred", which is never reset until the file completes.

sirius, you're right, this counter is an uint32 which will overflow at 4GB. This is a heritage from eMule, and not so straightforward to fix. We should fix saving the data into .part.met file too (otherwise it would overflow when you restart aMule the next time), but do this without breaking compatibility with older .met files and eMule's .met files. The bug will be fixed, but we have to consider carefully each move we take.

[Kry]

wtf ~ colours

[sirius]

Originally posted by Kry
wtf ~ colours

don't cry kry :-)

Didn't you stop your download in the mean time?

No, luckily amile-cvs is stable again and the download of the 4GB happend in about a day ... so no.

We should fix saving the data into .part.met file too (otherwise it would overflow when you restart aMule the next time)

Well, that would explain why i got some strange error messages when i first downloaded another version of the file wih a similar size. Amule was crashing every 30 minutes or so at this time a few days ago and I can remember having seen strange hash errors and file i/o errors with this file only. Now I have the strange occourence that the file has a status of "finished - completing", a size of several GB donloaded but for whatever reason a size of finished with 0.0 %. It was never completing, sitting there for a day and hashing before it went to 0% and now i have been able stop it (becuse i was trying o restart it) but it can't be started again, it can't be deleted either and it's not finishing so eventuall on the next restart I will resort to simply removing the part file. But this could be another bug (or bugs) related to the overflow issue combined with the restart.

which also reminds me of the 4 gb issue on fat32 partitions where i would recommend (if not inbuild already) on a check if the filesystem of the temp and  incoming directories can handle large files and putting a safeguard in. It might otherwise lead to all sorts of issues ...

I will see that I don't restart amule until he file is finished. luckily it on my linux server now :-) Thank to amule developed by you guy's!

Here are the latest stats. Say, can this above problem be the reason why i have quite a large downstream of 100 kb and more, downloading GB by GB and only getting slight increases in the %? I mean luckily i have a flatrate, but it definatle is strange tht with all other files (mostly several 100 MB) i only loose a few MB or even win a few due to compression and with this file it's double the fileszie already!?

Also, there is another 2.x GB file wich is now at 1.5 GB and the lost counter is already at 25% and rising .... will watch and keep you posted, but seems to e a general issue.

will post some screenshots now ...

[sirius]

Screenshot of the 2nd file with huge losses of 25% currently

[sirius]

Screenshot of the finished file with 0%

it's the last file in green (*grin*) and you can also see the other 2 files listed (and a 3rd unimportant one)

some other details. amule running for 2 day's now and i added the files about that same time. (yes got a big pipe at home:-) total download 26GB, overhead 200 MB.

hope this info helps

[sirius]

Hi, i came closer to the error after turning the debuggin on.

I get countless errors where the hash is a long 00000000 while expecting something else. This happens for both the large files. And I can't think of a fs error because I had just run a fsck.reiserfs --fix-fixable 2 days ago because I did want to make sure hat that the filesystme is ok. And I only get the error for the Files of more than 2 GB.

And another minor bug: wanted to copy the log from the log window, but the selection keept being reset. When I selected "select all" from the ricght-click menue amule was hung with 100% for a minute or two .... it's fine now again, but you can see he high load in the screen shot and nothing was selected. Maybe a check would be beneicial if someone is selecting and for that time buffer the log entries but don't write them to the log window? Oherwise it might not be possible to copy a log if many messages are written.

Oh and btw: the lost area has now reached 6.3 GB while the % is oscilating between 84,xx and 85,xx %. As this is the same for a few hours now, i guess this is a limit and the file will not grow from here one .... but I'll close amule and do anothher fsck, just in case ...

Comment added 1 Day later: ok, the file slowly growed to 93 percent, but the many 0000000000 errors continue ...

sirius

[sirius]

have a look at this, I did a run of fsck and had a similar result, a size error is reported with exactly the large file only ... can this be a coincidence? I don't think so. Also, AFAIK I'm using sparse files for temp, hope the bug is not in this area ... would be a shame (and waste of Hd space) if one could not use that feaure ... then again I'm not sure about sparse files, a I can't find an option in amule and ls and du show he same filesizes, Could be that I remember that from emule and mixed it with the knowledge that reiserfs can handle sparse files too. I'm not an expert there ...

Results:

Code: [Select]

vdr01 ~ # umount /dev/hda5
umount: /mnt/hda5: device is busy
umount: /mnt/hda5: device is busy

vdr01 ~ # umount -l /dev/hda5

vdr01 ~ # fsck.reiserfs --fix-fixable /dev/hda5
reiserfsck 3.6.17 (2003 [URL]www.namesys.com[/URL])

Will check consistency of the filesystem on /dev/hda5
and will fix what can be fixed without --rebuild-tree
Will put log info to 'stdout'

###########
reiserfsck --fix-fixable started at Tue Aug 23 18:15:14 2005
###########
Replaying journal..
Trans replayed: mountid 73, transid 4954288, desc 6111, len 4, commit 6116, next trans offset 6099
Trans replayed: mountid 73, transid 4954289, desc 6117, len 4, commit 6122, next trans offset 6105
Trans replayed: mountid 73, transid 4954290, desc 6123, len 4, commit 6128, next trans offset 6111
Trans replayed: mountid 73, transid 4954291, desc 6129, len 4, commit 6134, next trans offset 6117
Trans replayed: mountid 73, transid 4954292, desc 6135, len 7, commit 6143, next trans offset 6126
Trans replayed: mountid 73, transid 4954293, desc 6144, len 40, commit 6185, next trans offset 6168
Trans replayed: mountid 73, transid 4954294, desc 6186, len 14, commit 6201, next trans offset 6184
Trans replayed: mountid 73, transid 4954295, desc 6202, len 6, commit 6209, next trans offset 6192
Trans replayed: mountid 73, transid 4954296, desc 6210, len 14, commit 6225, next trans offset 6208
Trans replayed: mountid 73, transid 4954297, desc 6226, len 20, commit 6247, next trans offset 6230
Trans replayed: mountid 73, transid 4954298, desc 6248, len 9, commit 6258, next trans offset 6241
Trans replayed: mountid 73, transid 4954299, desc 6259, len 1, commit 6261, next trans offset 6244
Trans replayed: mountid 73, transid 4954300, desc 6262, len 68, commit 6331, next trans offset 6314
Trans replayed: mountid 73, transid 4954301, desc 6332, len 12, commit 6345, next trans offset 6328
Trans replayed: mountid 73, transid 4954302, desc 6346, len 1, commit 6348, next trans offset 6331
Reiserfs journal '/dev/hda5' in blocks [18..8211]: 15 transactions replayed
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
/tmp/track-01.isovpf-10670: The file [2 100] has the wrong size in the StatData (4294434816) - corrected to (4294434816)
/amule/temp/002.partvpf-10670: The file [111 231] has the wrong size in the StatData (2832269312) - corrected to (2832269312)
/003.partvpf-10670: The file [111 371] has the wrong size in the StatData (2618664581) - corrected to (2618667008)
/014.partvpf-10670: The file [111 338] has the wrong size in the StatData (2832236544) - corrected to (2832236544)
finished
No corruptions found
There are on the filesystem:
        Leaves 24964
        Internal nodes 180
        Directories 56
        Other files 926
        Data block pointers 25064981 (2489863 of them are zero)
        Safe links 14
###########
reiserfsck finished at Tue Aug 23 18:16:42 2005
###########
vdr01 ~ #
Note the 3 large files with problems 002, 003 and 014.part
Output of previous command 2 day's ago only shows the 0% problem file (014.part)

Code: [Select]

###########
reiserfsck --fix-fixable started at Sun Aug 21 05:46:01 2005
###########
Replaying journal..
Trans replayed: mountid 73, transid 4489188, desc 4368, len 8, commit 4377, next trans offset 4360
Trans replayed: mountid 73, transid 4489189, desc 4378, len 8, commit 4387, next trans offset 4370
Reiserfs journal '/dev/hda5' in blocks [18..8211]: 2 transactions replayed
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
/tmp/track-01.isovpf-10670: The file [2 100] has the wrong size in the StatData (4294434816) - corrected to (4294434816)
/amule/temp/014.partvpf-10670: The file [111 338] has the wrong size in the StatData (2832236544) - corrected to (2832236544)
finished
No corruptions found
There are on the filesystem:
        Leaves 31942
        Internal nodes 221
        Directories 57
        Other files 1448
        Data block pointers 32036977 (7306137 of them are zero)
        Safe links 14
###########
reiserfsck finished at Sun Aug 21 05:48:18 2005
###########

also note that the "lost" value is now down to 1.6 GB .... might be the described problem with the .met files ... how about you simply add the values twice, the original one with the 4GB limit and one bigger? Would of course only work if emule is ignoring unknown values in .met files ... but would keep backwards compatibility ...

[GonoszTopi]

which also reminds me of the 4 gb issue on fat32 partitions where i would recommend (if not inbuild already) on a check if the filesystem of the temp and incoming directories can handle large files and putting a safeguard in. It might otherwise lead to all sorts of issues ...

The eD2k protocol cannot handle files larger than 4Gb

Also, AFAIK I'm using sparse files for temp, hope the bug is not in this area ... would be a shame (and waste of Hd space) if one could not use that feaure ... then again I'm not sure about sparse files, a I can't find an option in amule and ls and du show he same filesizes, Could be that I remember that from emule and mixed it with the knowledge that reiserfs can handle sparse files too. I'm not an expert there ...

If ls and du shows the same size, then[list=1]

• your file system doesn't support sparse files, or
• the file is "complete", i.e. there are no empty blocks inside it, or
• ls or du is cheating you.
  [/list=1]

[sirius]

Originally posted by GonoszTopi

The eD2k protocol cannot handle files larger than 4Gb

If ls and du shows the same size, then[list=1]

• your file system doesn't support sparse files, or
• the file is "complete", i.e. there are no empty blocks inside it, or
• ls or du is cheating you.
  [/list=1]

Well, thanks that answers that question.