[sf-lug] Dynamically Scalable Data?

[Ernest De Leon]

For some reason, I'm perfectly ok with the scalability from the server and
application side.  (That may or may not be a bad thing)  :)  I guess what
I'm more after is tackling the issue of dynamic data availability (say due
to a spike in search traffic for a particular video or image).  Even within
that, I'm not really concerned with bandwidth per se, but more with the
ability of the data storage device (a SAN in this case) to scale it's output
on a certain finite data set within the larger warehouse.  With new tools
available in the storage world like dynamic data optimization, it is now
possible to redistribute data across the spindles in a SAN to increase
throughput out of the device itself onto the fiber network.  I am fleshing
out a way to take a dynamic event like a surge in search queries for a given
finite data set and translate that into an automated process or system than
can integrate with the storage management platform and dynamically scale the
throughput on the fly.  Then perhaps when the spike in interest subsides,
redistribute the data 'normally' to allow for the next surge to take
precedence.

Ernest

On Fri, Apr 18, 2008 at 11:27 AM, Kristian Erik Hermansen <
kristian.hermansen at gmail.com> wrote:

> 2008/4/18 Ernest De Leon <edeleonjr at gmail.com>:
> > Let me preface this question by saying that very often I have these
> insane
> > ideas which I then attempt to architect and flush out in my mind despite
> the
> > fact that I may never actually implement them or I may not have intended
> to
> > even before I thought through it.  This is probably one of those...
>
> You are not insane :-)  Amazon does something very similar with EC2
> and AWS.  Google's MapReduce and GFS are known public details of
> internal operational infrastructure...
>
> http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud
> http://en.wikipedia.org/wiki/Amazon_Web_Services
> http://en.wikipedia.org/wiki/MapReduce
> http://en.wikipedia.org/wiki/Google_File_System
> --
> Kristian Erik Hermansen
> --
> "Clever ones don't want the future told. They make it."
>



-- 
Ernest de Leon
http://www.smbtechadvice.com

"They who can give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety." - A common 18th Century sentiment
voiced by Benjamin Franklin

"A patriot must always be ready to defend his country against his
government." - Edward Abbey

"All that is necessary for evil to triumph is for good men to do nothing." -
Edmund Burke, English statesman and political philosopher (1729-1797)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://linuxmafia.com/pipermail/sf-lug/attachments/20080418/b353ae28/attachment.html>