[imagineers] wget flawed on HD900?

[fortean]

While using the HD900 wget (part of busybox) to fetch a rather large binary, I noticed that the resulting file differed from the original.

Perhaps some of you can try this, to see if you get the same results:

cd /bin
wget http://www.imagineers.nl/hd900/tmp/smbclient

it should result in a file /bin/smbclient with size 3225296 bytes.

What are your results?

(BTW: yes, that's a working smbclient, but it requires libpopt and some symlinks, more about that later, first let's establish if we found another bug).

 

[Boldor]

The result is ... 3200720 bytes the first time and 3204816 the second time.

/bin > wget http://www.imagineers.nl/hd900/tmp/smbclient
Connecting to www.imagineers.nl
smbclient 100% |********************************************| 3149 KB 00:00 ETA

3149 KB stays the same

 

[Boldor]

Sorry, starting smbclient gives "Segmentation fault" as result. So I think it gets broken before arrival.

 

[bazinshine]

fine here mate

/tmp/tmp > wget http://www.imagineers.nl/hd900/tmp/smbclient
Connecting to www.imagineers.nl[80.89.228.10]:80
smbclient 100% |****************************| 3149 KB 00:00 ETA
/tmp/tmp > ls -l
-rw-r--r-- 1 root root 3225296 Sep 3 21:47 smbclient
/tmp/tmp >

 

[Boldor]

In /tmp its works correct. Not in the /bin directory ?

/tmp > wget http://www.imagineers.nl/hd900/tmp/smbclient
Connecting to www.imagineers.nl
smbclient 100% |***************************************************************************************************| 3149 KB 00:00 ETA

-rw-r--r-- 1 root root 3225296 Sep 3 23:12 smbclient

 

[bazinshine]

pmsl, so it would seem, well spotted fortean,
solution for now must be then wget to /tmp and mv to bin

 

[fortean]

Sorry, starting smbclient gives "Segmentation fault" as result. So I think it gets broken before arrival.

Well, actually it seems to work overhere - but it requires libpopt.0.0.0 -> libpopt.0 in /usr/lib en some fiddling with symlinks.

(Which, BTW, if you promise not to talk about it :grin:, can be fetched from the same location, as can a little script named 'goforit' that contains instructions on how to set it up so it works. Now, no guarantees, as I only compiled it to enable me to help somebody to find out why his NAS can't be connected to the 900).

 

[fortean]

pmsl, so it would seem, well spotted fortean,
solution for now must be then wget to /tmp and mv to bin

Perhaps because /tmp is actually a RAM disk and /bin is in flash..?

Mm. Why then, would a mv work, while direct writing in flash (MTD) does not?

I will try this myself tomorrow, can't spend much time here today, have to run..

 

[bazinshine]

Perhaps because /tmp is actually a RAM disk and /bin is in flash..?

Mm. Why then, would a mv work, while direct writing in flash (MTD) does not?

Would seem a good explanation, I will test it under a script and see if the mv works too mate, but think it should really,,,,,
but then again, so should the wget.

 

[fortean]

I just did a quick try and it seems that bazanshine is correct: if you do a wget to /tmp and do a 'local' mv, it works allright. /tmp is a tmpfs, so it is actually in RAM.

I don't have the busybox source here, but I guess that wget does a regular open(), and does not use O_SYNC and/or O_DIRECT flags. This means that the operating system will cache the blocks initially, and write them to the "disk" (flash/MTD drive) in the background.

Perhaps the cache simply fills up during the download, as it takes the jffs fs quite a while to write the 'blocks' to the MTD. I haven't got a clue what happens when the write-cache (in memory) is full and the kernel does not have swapspace to write too. May be that some blocks are simply discarded..? That would explain why small files are written perfectly.

Well, as both 'mv' (fileutils) and 'wget' (busybox) are Open Source code DGS (or we..) must be able to figure it out, and fix it.

Next question then: how many other standard utilities that do file I/O behave like this, for example, can we trust 'tar', or 'dd' or ..??

I reported this to DGS, I will point them to this thread so they can read about the nature of the problem.

BTW: I just received an init script from DGS, which might solve the RTC problem, will check on that next.

 

[bazinshine]

I just did a quick try and it seems that bazanshine is correct: if you do a wget to /tmp and do a 'local' mv, it works allright. /tmp is a tmpfs, so it is actually in RAM.

I don't have the busybox source here, but I guess that wget does a regular open(), and does not use O_SYNC and/or O_DIRECT flags. This means that the operating system will cache the blocks initially, and write them to the "disk" (flash/MTD drive) in the background.

Perhaps the cache simply fills up during the download, as it takes the jffs fs quite a while to write the 'blocks' to the MTD. I haven't got a clue what happens when the write-cache (in memory) is full and the kernel does not have swapspace to write too. May be that some blocks are simply discarded..? That would explain why small files are written perfectly.

Well, as both 'mv' (fileutils) and 'wget' (busybox) are Open Source code DGS (or we..) must be able to figure it out, and fix it.

Next question then: how many other standard utilities that do file I/O behave like this, for example, can we trust 'tar', or 'dd' or ..??

I reported this to DGS, I will point them to this thread so they can read about the nature of the problem.

BTW: I just received an init script from DGS, which might solve the RTC problem, will check on that next.

Busybox sources arent a problem I will try do some digging, and compare to other levels of busybox i use on other boxes and see if the problem exists in there compiled busybox or problem outwith that.. but from logical thinking its seems very like its not doing a sync on the wget to mtd memory.( testing other things with sync, when doing this wee test it seems to miss only the last few kb, which mirrors the amount of data it misses if you do a dd backup and forget to do sync )
dd command should be fine,,,, i have used a lot in last few weeks ,, long as you do a sync manually after dd backup it is fine then to use with mtdutils, so would seem thats commands ok,
but i do see your point with this, and that there may just be other small apps inside busybox not quite performing like they should... and so hows about we try complile own busybox

 

[fortean]

Well, yes, as we could compile our own kernels, our own gnu tools etc. - but in that process we will loose the support of DGS, I guess. Hence I say, _they_ have to do the recompiling and redistributing. But I am very willing to help them.