tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Chaney <a...@compulsivecreative.com>
Subject Re: managing user uploads best practices
Date Mon, 11 Feb 2008 17:38:26 GMT
Totally agree with everything brien says below.

I also run a web site with a large number of media objects. I've been 
involved in filing system and media projects for many years both in 
research and production capacity.

The slight increase in complexity of the solution to maintain the media 
objects outside the database is completely offset by the huge 
improvement in performance of the database.

Also it is particularly worth stressing the backup advantages. If you 
have 10,000 users (not a large number these days) each with 5 images 
and/or one podcast you are going to have very, very large monolithic 
database backups which are either going to be in a binary form and thus 
very opaque or simply huge as sql dumps unless you separate the binary data.

Another advantage is that it allows you to have a very scalable media 
server. The media server does not have to participate in the session. 
You can have a number of independent media servers who simply download 
files identified by an ID. The media server(s) don't have to be on the 
same box, platform or even in the same building. The issues of 
clustering simply go away (a media download is a single operation and 
thus does not benefit from clustering.) If you are running a large 
number of concurrent downloads, you only need a trivial load balancing
front end server to direct requests at the media servers. If you require
authentication to restrict access to the media then you may have a bit 
more work to do but it is not too difficult.

So if you have a site with a lot of media, you can tune the application 
component to be small and responsive and the media delivery components 
to be simple and fast and optimizied for continuous download.





brien colwell wrote:
> On the topic of DB versus filesystem for media, I prefer storing media
> in a filesystem and meta data in a DB. The advantages of storing large
> binary files outside of the DB are
> 
> * Reduce contention in the DB -- it's doing so much
> * You have more control with a filesystem where your data goes -- e.g.
> you can have a disk for one type of data, and another disk for a
> different type
> * Filesystems are optimized to stream out data ... which I think makes sense
> 
> That first point is really key. If you're serving data to a UI from
> the same database you're storing large files into, the whole
> experience is going to suffer. The best article I've read on this
> topic is
> 
> http://mysqldatabaseadministration.blogspot.com/2008/01/i-will-not-blob.html
> 
> The main scale issue I'm careful of is how many files I put per
> directory. For large amounts of files, I sometimes link directories
> into trees. I might be a little paranoid about this, but it seems to
> scale well.
> 
> In terms of backup and security, I think a FS is easier to manage --
> e.g. chmod and rsync on a cron job -- though you'll have to write more
> code and scripts. You also have to be careful of things like shell
> injection if you're running shell commands from your webserver.
> 
> 
> Anyway, hope that gives some help. I was also confused with this when
> I started writing DB backed apps.
> 
> 
> 
> On Feb 9, 2008 4:58 PM,  <mbeards@comcast.net> wrote:
>> Thanks for the suggestions!
>>
>> I like using the database for storage for it's easy maintenance and security, but
marshalling a lot of binary data (like a large image library) adds a bit of overhead to the
application.
>>
>> I'll look into pursuing a storage directory external to the webapp.
>>
>>
>>
>>  -------------- Original message ----------------------
>> From: "Johnny Kewl" <john@kewlstuff.co.za>
>>> ---------------------------------------------------------------------------
>>> HARBOR: http://coolharbor.100free.com/index.htm
>>> The most powerful application server on earth.
>>> The only real POJO Application Server.
>>> Making the Java dream come true.
>>> ---------------------------------------------------------------------------
>>> ----- Original Message -----
>>> From: <mbeards@comcast.net>
>>> To: <users@tomcat.apache.org>
>>> Sent: Friday, February 08, 2008 11:13 PM
>>> Subject: managing user uploads best practices
>>>
>>> Yes... outside.
>>> Its been a long long time now, vaguely remember struggling with Apache
>>> uploader then eventually getting all to work...
>>> Anyway... what I did is store the files in an Apache httpD folder, so I
>>> could spy on the uploads, and they available for viewing again.
>>> And whats cool is because Apache is also the load balancer in my case... can
>>> have lots of TC's doing their thing.
>>> I was making a kind of wiki thing for an estate agency... thats how I did
>>> it.... way back when...
>>>
>>>
>>>> What's the current wisdom on managing user uploaded files to a web app
>>>> that's
>>>> deployed via a WAR?
>>>>
>>>> In other words, when the WAR is updated, the directory containing uploaded
>>>> files
>>>> would be wiped out.
>>>>
>>>> Do people save uploaded files outside of the web app root directory?
>>>> Security
>>>> issues with this?
>>>> Do people not use auto-expanding WAR files and manage the deployment by
>>>> hand?
>>>> Do you not include the directory for uploaded files in the WAR (but create
>>>> it at
>>>> runtime) and then trust that the expanded WAR won't overwrite it on
>>>> deployment?
>>>>
>>>> Any pointers greatly appreciated!
>>>>
>>>> ---------------------------------------------------------------------
>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 
> !DSPAM:47aed67d224182136417547!
> 

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message