Filer is a REST based client/server tier for image processing and storage.  Like the rest of RGB Daily its implemented in PHP.  I have branched it from the core and plan to develop it as a separate project; potentially as an open source project.  The system is design to offload heavy weight image processing tasks asynchronously to clustered LAMP services such as those provided by Mosso or Media Temple.  So far RGB Daily is composed of the following horizonatally scalable tiers:

• Web Tier (dedicated nodes)
  
• Apache with mod_php5
• PHP with APC
• Memcached for session storage & DB query caching
• CDN for application static resources
  
• DB Tier (dedicated nodes)
• InnoDB storage engine
  
• Application Level Partitioning (still needs work)
• Filer (LAMP cloud service)
• File serving and image processing
  
• SmartStream client disconnects after URI generation but before image processing
• Intelligent file spreading for intelligent/scalable directory structure

More to come...  

JQuery

So code cleanup is going really well.  In the process I have really polished up my CSS/XHTML/Javascript skills.  The core software for RGB Daily, Pligg, is written with Prototype and  Script.aculo.us on the front end.  A couple of the custom plugins that where written use JQuery though. Working my way thorugh bugs I became really impressed with jQuery and decided to find out more about its author, John Resig.  Turns out he lives in Boston and also works for Mozilla now.  He is definitely my kinda architect.  In the pocess of two days I pretty much taught myself jQuery.

I started out just wanting to get rid of the ugly dialogs that the alert/confirm javascript functions produced.  I learned a lot from a plugin called Impromptu, written by Trent Richardson.  After figuring out how Prototype was stomping over the “$” shorthand of jQuery I was on my way (I’m a Java guy, not use to that kind of wild west stuff).  I hacked the CSS and added some convenience wrapper classes to implement replacements for the native prompt functions.  Here is a quick pic of what I was able to achieve:

PR

So I’m showing a little more than just the confirm dialog.  In the coming weeks I’ll be giving a little more guidance on what RGB Daily actually does and for what communities it serves.  For now you can have a look and speculate

As for updates on soft launch…  Still looks like September is reasonable.  At this point to be honest I just want to make the core production ready.  That means a bug free UI (address known bugs from internal testing) and a server side that is built for scale on clustered LAMP.

There are big problems with the code base concerning security and performance.  Much of the custom code for RGB Daily is not leveraging caching properly.  Most of the inputs into the system are not even properly escaped for querying against the DB.  Why am I saying all of this?  The take home is not to sit back and relax while you think your dream system is being built.  You still need to dive in on your own if you are an engineer and know what your doing.  If you’re not the programming type you need to hire a third party auditor before the final release of the code is delivered.  It’s actually better if you can have the code reviewed in milestones.

Status of Launch

So launch is really late, but the code is ten times better for it.  I also know pretty much every corner of the codebase and that will benefit me as the founder /architect of this puppy.  If a potential partner of investor has a question I have the answers and that’s invaluable.  August is the last major month of coding and a formal launch will happen towards the end of the month.  This is a drop dead date.

Originally posted August 7, 2008

So I have been implementing a couple of performance improvements across the codebase.  RGB Daily is written in PHP, CSS/XHTML, and Javascript.  I was able to speed up the performance on the client side by integrating Minify to serve us the CSS and Javascript.  Processing on the server side was still taking around 2-3 seconds on average.  It turns out that the bottleneck was the templating engine.  There was a long standing bug which cleared previously loaded config constants when a new config file was appended. Consequently the main config file that was used in the application was reloaded after every sub module that makes up the page.

I refactored the code so that it only loads a single config file once per template instantiation.  I will go further and cache the parsed config files to get even more speed gains.  Localized files will only cost as much as a simple include.

Originally posted July 28, 2008

Performance & Uptime

Raw performance and uptime are the two most important features of application hosting.  Ease of use shouldn't even be on the chart, but that is exactly what happen when I chose Mosso.  Mosso has been going through continuous growing pains as they progress towards Internet scale.  Mosso is great for blogs and simple web sites, however currently it is not a competent platform for Web 2.0 style applications.  I used them for during the entire development phase of RGB Daily and still experience outages that kept me from committing and testing code.  The rest of the team at the time (Moonrank), also found the lack of SVN support cumbersome. 

There were definitely things that I liked about Mosso.  I think the top thing on the list was the SAN, maintenance and headache free storage growth on the SAN was a critical requirement for RGB.  I tin the thing that wasn't so much fun was the lack of partitioning.  In the end you want to know that you have a guaranteed amount of CPU and RAM.  Over the past week we have been working with a new host that provides the following:

• Xen Virtualization and Shell Access
  
• Dedicated CPU and RAM
• Scalable SAN storage via Left Hand Networks
• VLAN and management via Interworx

We are pleased, but we have learned the hard way that talking about hosting doesn't buy you anything.  The bottom line is that there are a ton of solutions and what matters is if they meet your needs.  Also you will have to be comfortable with getting you're hands dirty if you want to get the most bang for your buck.  I should have taken to heart from Kevin Rose's advice, "Own your own hardware."  I would revise that slightly and say that it is important for you to have control of both vertical and horizontal scalability.  There are a few players in the market that allow you to do that.  Which one you choose will depend on your particular requirements and architecture.  I will share that list:

• GoGrid
• Linode
• Slicehost
• Joyent
  
• Voxel

I leave you to do your own research.


Original post from May 7, 2008 ( Was previously hosted on Wordpress @ Mosso)

Choosing a Host

We are feverishly trying to wrap up the first alpha release of RGB Daily.  We will be starting a private preview release once we get the latest code settled on our new host.  We evaluated several options for hosting including both dedicated/virtual servers and managed solutions.  In the end we decided that the best architectural solution for our needs would be a clustered LAMP solution.  Building out own multi-tiered clustered solution is not something we are interested in doing, so managed was the only choice for us.  Also we wanted a solution that would be able to grow with the demands of a growing user base and their data.  We needed a system that had a manageable fixed price, but also was able to grow with us as our storage and bandwidth costs grew.

Potential Hosts

In terms of managed hosting with clustered architectures there are only really three choices.  Media Temple, Joyent, and Mosso.   I host quite a few of my personal projects on Media Temple and I am fairly satisfied with the low price and set and forget nature of everything.  At the same time there are quite a few deficiencies in the way they do things.  I experimented with Joyent for less than a month and they are not really a managed solution.  You really only get vertical scalability and you are responsible for keeping the lights on.  With most true managed solutions you really only have to worry about data integrity over time.  Services are stateless and come up and down, but the hosting company for the most part takes care of availability of your service.  At Joyent I found myself tweaking settings for an individual Apache instance.  This was going to scale into a ton of headache.

Mosso

In the end we went with Mosso. I’m not 100% sure of their uptime or overall performance, but I know that we can make the appropriate changes to out application code to take advantage of their solid architecture.  The fact that we have high speed SAN that is mounted across the web tier means that we can take advantage of aggressive file based object caching.  Also the MySQL layer is clustered - this means that we will be able to handle high load and can get away with expensive queries as long as we cache the results.  There are a few hot spots in the code base right now that we will be cleaning up overtime.  To keep things simple though we will only be caching for right now.

Key benefits of Mosso:

• $100 base hosting with cost effective paths to growth
• Rich platform for applications (LAMP/.Net)
• SAN storage that can grow without the complexities of managing storage
• Clustered Apache & MySQL