Brain-dead design

[andrewducker]

FTP on thousands of tiny files is incredibly slow because it's creating a new connection* for each one.  This seems very, very silly.

How should I be doing this?  Or is this really the best the internet can manage?

*Not a completely new connection, of course, but it goes into and out of file-transfer for each one, and I'm watching the commands scrolling up the screen endlessly, rather than it just uploading a 5MB block of "stuff" and a series of commands for what to do with it at the other end...

Comments

[pseudomonas.livejournal.com]

Can you zip/tgz the files together at one end and then extract at the other?

[andrewducker]

Don't have any kind of command-line access. It's all through cpanel.

[pseudomonas.livejournal.com]

CGI script ;)

[drdoug.livejournal.com]

tar if you're old school, zip if you're a Microsoftie, StuffIt if you're a Machead.

Or some FTP servers offer 'download entire folder as zip' as an option - may require reversing the polarity of your FTP flux.

[andrewducker]

I'm uploading, not downloading...

[matgb]

I have never managed to understand how to unzip stuff server side, but would also be very appreciative of a bit of software and/or explanation of how it's done.

[kurosau.livejournal.com]

I don't know of any, but I figured I'd say this. Have you ever heard of an FTP program that handles a queued upload without creating a new connection? I haven't, but that might be what you need.

[channelpenguin.livejournal.com]

write one?? Just a crazy kooky suggestion, given what you do all dat and all that....

[channelpenguin.livejournal.com]

all *day* even.

[andrewducker]

The problem here is that I'm moving files onto a unix box on the other side of the planet, which I don't have shell access on...

[hirez.livejournal.com]

mget/mput? Or is this disallowed on the likes of vsftpd due to random security malarkey?

[autopope.livejournal.com]

Does the server support rsync? If so, rsync is the way to go.

[andrewducker]

Looks like it would if I had SSH set up - which I don't, but would be worthwhile if I was doing this a lot. Cheers!

[call-waiting.livejournal.com]

It's the best that FTP can manage. That's just the way the protocol was designed, in the days when it was assumed that even a 1k text file would take a lot longer to transfer than to establish the data connection.

I don't know if its' possible to pipeline the connection process: afterall, the control channel in a FTP connection isn't *doing* anything while the data connection is active; if the server would let you, it ought to be possible to send the next PORT/PASV;RECV/STOR while the last one is still processing.

Failing that, of course, you could always just have multiple complete FTP connections.

It's clunky 'cause it's old.