Best Practice for Large Uploads

[wwarby]

Hi all,

I'm developing a project at the moment using Ext where I need the ability for users to upload very large files - quite possibly hundreds of megabytes but definitely in excess of my host's upload limit in PHP (7MB). Obviously given that I am using Ext, aesthetics and ease-of-use for the customer are high priorities for me.

The potential size of files I want to receive makes a standard "file" type input control a non-starter, however pretty I might be able to make the interface (which is to say very pretty, if using Ext.ux.UploadPanel from Jozef Sakalos). Research has shown me that SWF-based uploads are riddled with platform problems due to buggy versions of Flash from Adobe, so I started looking at Java-based systems that allow the data by "chunked" at the client end, enabling very large uploads. So far everything I have found has been buggy, hideously ugly (some of them enough to make my eyes sting) or ridiculously overpriced. I don't mind spending a little cash on getting this right (say up to about $30) but obviously I'd rather use free or open source software, and in an ideal world, I'd rather not show a Java UI at all - I want something that I can put in the background and wrap with an Ext UI. I wondered if FTP was perhaps another way to go if it meant I didn't have the upload size limitation, but I presume I would still need Java on the front end to handle the FTP protocol which isn't native to the browser, at least for uploads.

Grateful for any thoughts on the best approach to this problem.

[wwarby]

Urgh. I've posted this in the wrong forum - it was intended for the ExtJS thread, not GWT. Should I re-post, or will it get moved?

[jaime_]

What you are trying to accomplish is not easy. Uploading large files by the traditional method will lead you to problems both in the client and the server side. While you seem to be focusing now on the client part (showing progress, multiple file upload at the same time and so on), the server side is equally complex.

Here is an interesting post from someone who did it without any Flash nor Java. They set up a server called NodeJS which is based on Google V9 and therefore is able to run Javascript code on the server side. You can see it on the wild at http://www.transload.it

This is interesting because due to the event-driven nature of Javascript they are not wasting a server thread on every file upload. Also, they are able to do cool things like to process the file as it comes, so it's possible to abort the upload if the file is not in the format they expect it to be. Everything with low resource consumption.

Check it out at http://debuggable.com/posts/streaming-file-uploads-with-node-js:4ac094b2-b6c8-4a7f-bd07-28accbdd56cb

[wwarby]

Jaime,

Thanks very much for taking the time to respond - I read the transcode.it source example with much fascination - it never would have occurred to me to attempt to run JavaScript on the server side but for sure it looks like an elegant solution. My concern is whether I'll be able to do this on shared web hosting, but I'll certainly look into it. My fallback plan is a rather unattractive but very functional Java applet I found that permits multiple file uploads, progress, and most importantly, chunking of the data for re-assembly at the other end, and seems to work nicely with a test file of 1.6GB. It's called JumpLoader.