Message Queue & Real-Time Web

This tittle may look old/odd for some people and may look very techie for some people but if you want to know about the secret-sauce behind Twitter or better yet, if you want to know about the next gen web technology(aka real-time web), then you may want to read this post.

Message Queue is not new. When did Tibco go public? In the enterprise world, specifically in java-land, IBM MQ & Tibco MQ are very well known products. JMS provided a thin wrapper to connect MQ products to J2EE App Servers in late 90's. The combination of JMS API + MQ product worked fine for last 10 years or so. Remember, MQ products were working fine even before JMS!

Then came SOA & followed by bunch of alphabet soups. ESP, SOA BUS, Internet BUS and etc which basically had NO meaning at all...The underlying architecture is the same. Some more connectors...that's it.

This should give you a good overview of MQ. Also, you may think that MQ is a solved problem. The question is that why the hell Twitter went & built it's own MQ? First it built a version using Ruby and failed. The 2nd version runs on JVM w/ Scala.

Are you puzzled now?
[1] Is Twitter folks are naive?
or
[2] Is there a problem with current MQ products?

Simple answer is "no" to both! Then why the heck Twitter built a new MQ? Is too much VC $ sitting in the bank? There is a technical reason!

Let me tell you, Twitter folks are pretty smart. The exiting MQ products are bulky, expensive and complex to cluster them for scale & reliability. Simply it's an overkill for web applications. On top of it, there is no guarantee that these products which are primarily designed to run inside the firewall can work on internet. These MQ products are good for enterprise scale! I am sure that these guys looked around and found NOTHING for web apps. So, they built using a technology invented 30mins back(Ruby) and Garbage Collection didn't work well. Then they switched to JVM but not Java. I very much hope that 2.0 of Twitter MQ works fine for them. (I wish they should have taken 100% Java approach for building MQ)

Ok, good. The story looks good so far. Why I am ranting about all these?

The reason is that recently I was looking for a light weight MQ with HTTP/REST interface built using stable Java platform and found NOTHING! Yes, NOTHING. The closest i found was Apache ActiveMQ but it has it's own bucket full of problems.(if you are interested, look at my questions in the users archive) For infrastructure products, I don't trust if it's build using some newer technologies with no mileage. I am aware of Rabbit MQ and all...I don't want to commit the same mistake Twitter folks have done!

Well, now you may ask why MQ is important for next gen web technology? The answer is "real time". Twitter started as a CMS architecture but they changed it to message-oriented async architecture for real time update & scale. Writing app directly talking to DB will fall flat in the face, if you want to move to real-time update with millions of transactions or users!

Twitter is just the start. Now you can see that next gen web technology misses a key infrastructure product! If you are a Java shop, you have some options. But for others, it's a question! OR build one like Twitter if you still have VC $ left in the bank!

I couldn't able to find a stable REST based MQ written in Java. If you know one, pl tell me. [Hint: If any one can make AWS SQS in a box, you have a business!]