[MarkLogic Dev General] Can app server handle file uploads >2GB?
geert.josten at dayon.nl
Tue Feb 7 11:04:29 PST 2012
My guess at it is that MarkLogic would try to maintain the binary file as
request param within memory, which is limited to some size for good
reason. With 10+ concurrent requests, all uploading 2 Gb, memory would
build up very rapidly. I'm guessing the limits are related to the limits
within xdmp:document-get, which does not allow reading files larger than
16Mb or 64Mb (depending on 32/64-bit install).
I think I wouldn't advice to use form upload to send large files across.
Other protocols like (S)FTP are better suited for that. You could allow
such files to be uploaded to file locations through a separate (S)FTP
server on the same host. You can let MarkLogic monitor that directory
using a Scheduled Task, and perhaps leveraging Information Studio flows.
In these processes you can use different function to load such large
files, though I am not sure they are able to handle 2+Gb files..
> -----Oorspronkelijk bericht-----
> Van: general-bounces at developer.marklogic.com [mailto:general-
> bounces at developer.marklogic.com] Namens Demian Hess
> Verzonden: dinsdag 7 februari 2012 19:45
> Aan: general at developer.marklogic.com
> Onderwerp: [MarkLogic Dev General] Can app server handle file uploads
> We would like to store large binary files (many EPS and high
> resolution PDFs) in MarkLogic. It would be easiest for users to create
> zips with all their files and then upload via an HTML form
> (enctype=multipart/form-data), but the size of the zip archives would
> be more than 2GB. Indeed, some individual files can get to 2GB by
> I tried POSTing a 2GB file to an app server and got a "413 Request
> Entity Too Large" error. I don't see any settings in MarkLogic to
> control max size of uploads for HTTP app servers.
> How efficient would ML be at handling large posts of 100MB and larger?
> Would it buffer everything in memory or would it stream to a temporary
> file on disk? Would ML 5.0 be more efficient than 4.0 at handling
> large file uploads?
> General mailing list
> General at developer.marklogic.com
More information about the General