Something Windows _really_ sucks at

[andrewducker]

I'm copying files from a my networked hard drive to a USB stick.

It tells me it's going to take about 7 hours. For 3 Gig of data.

I copy it from the networked drive to my C drive. This takes about 4 minutes.

I then start copying it to the USB stick. This is varying between 8 minutes and 45 minutes, depending on whether it's copying a big file (fast) or lots of little files (slow).

Now, what the hell is the problem with copying files from the network to a USB stick that suddenly adds all of that time on? Why do the latencies multiply?

Comments

[gonzo21.livejournal.com]

Are you copying files that have a complicated file hierarchy?

I find copying smaller batches of files goes a lot quicker than if I copy them all at once.

[andrewducker]

Nope. A couple of subfolders, but nothing unusual.

[spacelem.livejournal.com]

Is there some caching going on there?

[andrewducker]

Why would that slow it down?

[daveon.livejournal.com]

USB driver throughput I suspect.

[andrewducker]

Why would that make it go from 3 minutes + 8 minutes to seven hours?

[momentsmusicaux.livejournal.com]

Also, why would it affect anything?

Network -> PC is faster. So the USB driver should never be kept waiting.

[chuma.livejournal.com]

Is this 1.0 or 2.0?

[andrewducker]

2.0

[marrog.livejournal.com]

USB 2.0 shouldn't take that long for 3G of data anyway, should it? I'm sure it doesn't take that long for mine... maybe you need a new USB stick? They do wear out eventually, don't they?

Although I have noticed that it's usually faster for me to pull files off my external onto the harddrive and then on from there to a stick than it is to pull them directly from the external onto the stick - as in, combined times, still faster to do two transfers. But that's with two USB connected thingies.

[steer.livejournal.com]

OK. I do not know the underlying reason but windows is super rubbish at scheduling copies to usb. In particular you will notice this if you drag a large file to the USB stick and while it is copying drag a second. Two files whuch would otherwise take 2 minutes each can take hours.
This is apparently to be fixed in win 8.
While I do not know the root cause I suspect it is an issue called "pipelining". This is when a copying program fills buffers inefficiently and the "pauses" come to take up more than the "copying".

[the-fetket.livejournal.com]

ARGH!!!! basics of computing...

copying to your harddrive is i/o used once
copying from your harddrive is i/o used once
Copying to you stick from the network is i/o being used simultaneously for two tasks.

Same basic I/O connects most of your peripheries. This isn't just windows this is all computers... *sigh*

[andrewducker]

And the reason why 9 minutes of task+4 minutes of task = 7 hours of task is...?

[the-fetket.livejournal.com]

And if you want to do a first year software engineering experiment take two pads and four pages of two separate books and three people calls abigail, bob and conner.

Experiment 1: A tells B contents of page 1. B writes it down and checks it with A. Then B tells C contents of page 2. C writes it down and checks it with B.

Experiment 2: A tells B contents of page 3, meanwhile B is also telling C the contents of page 4. B is writing page 3 down, C is writing down page 4. Both are checking with their sources.

Compare the times of the experiment... Experiment 2 usually ends with people yelling at each other and cheating just to get the page copied down before the end of the tutorial.

[steer.livejournal.com]

wow whoever taught you basics of computing architecture should be blushing now. seriously. This depends whether the transfer is bounded by the speed of a shared bus or by some other non-shared queue (hint - it is the other queue). Still, you can test your "computer has run out of IOs" theory - try what andrew did and see if it also makes memory or output to graphic card run slowly. I'll wait here until you feel stupid.

[andrewhickey.livejournal.com]

I don't think that's necessarily a windows-specific thing. I've had similar problems (though not *quite* so bad) when copying files from a USB hard drive to a USB stick on Debian (though that's also on a very old, not very fast, desktop).

[steer.livejournal.com]

You get similar problems on linux but not nearly so bad in newer versions in my experience. The difference is extremely marked. On my ubuntu system I can queue such copies and the copy time is slightly more than linear (copying files to USB which would take a, b and c seconds takes slightly more than a+b+c but on windows it can be 10 times what it should take). I know ubuntu is somehow "cleverer" in the copy as in gui the copy windows for the files merge into the same window (did not a few years back). This feature is scheduled for win 8.

What i do not know is the underlying mechanism as the memory in USB should theoretically be very good at random access.

[recycled-sales.livejournal.com]

Am I right in thinking that USB, LAN and SATA all go through the Southbridge/ PCH? Is this maybe a limitation there?

[steer.livejournal.com]

no way surely. That is comparatively fast even if they are limited by that it should be able to handle 1000s of times speeds andrew quotes. If it is limited there it is nothing so simple as lack of bus bw.

[strawberryfrog.livejournal.com]

This post is on hacker news, and there is commentary there: http://news.ycombinator.org/item?id=3056457

[andrewducker]

Submitted by me. And I got 26 points out of it so far!

Turn on write caching, and copy from local disk not network

[barrkel.livejournal.com]

Make sure you have write caching enabled for the drive (Google for it if necessary). Windows optimizes for filesystem consistency rather than performance by default for removable drives.

Re: Turn on write caching, and copy from local disk not network

[andrewducker]

Why would that cause a large discrepancy between NAS->USB over HD->USB?

[sean underwood (from livejournal.com)]

My guess would be that the straight-to-USB scenario somehow results in everything literally being copied one file at a time, so a whole lot of time is spent waiting around on blocking I/O calls.

Meanwhile, copying to disk and then to USB both use buffered I/O, so the actual reads and writes are happening in a smaller number of larger chunks, and a lot less time is wasted on latency.

[khbrown.livejournal.com]

Also timeouts / failures when attempting to copy files from a Windows share to Linux.

Various 1GB AVIs that simply will not copy this way but which were fine the other way round from SAMBA to Windows.