notice a trend?

Configure wicket to use HttpSessionStore to keep all Pagemap data in the session aka your memcachedb storage. This enables you to kill off wicket application servers at will without any expired sessions.

Configure wicket to use ONE_PASS_RENDER RenderStrategy so it does not buffer response data locally, so essentially you can bounce users around from server to server without expiring the session.

Keeping two jetty instances on a single server can easily support a full blown diggig without any problems. You have the added benefit of stability by redundancy and ability to roll out rolling updates without interruption to your service.

Stateful web applications do scale!

Concerning state size, I have a jetty file session manager written. It lets me keep an eye on session size via ‘du -hs *’ command. Nice and simple.

Some J2EE/hibernate stuff does not vertically scale! so dropping down to SQL is a must. For example, user.getFriends().add(foo); is highly inefficient. It fetches all your friends just to add a single entity to the collection. Works great when you just have 2-5 friends, not so much when you have 10000-30000 friends.

off to bed, pulled an all nighter. Sorry if I have any grammar or typos in my post.

Sorry to drag this on, but you’re tantalizing the CS-degree in me… is fab40 all done in Wicket/Spring/Hibernate?

I’m a big Wicket fan, haven’t decided on ORM prefs yet and never had an “ah-ha!” moment with Spring where I FINALLY understood what all the hub-bub is about.

I’m curious what you had to do on a Wicket site to survive a Digg; namely with keeping the session sizes down by carefully crafting the model/detachable models for everything.

That is one issue I had with ShoutStuff (Wicket), I was either being sloppy with my models or forgetting that the state can get versioned unless you use IDetachableModel for every single model except those that cannot re-attach.

Anyways I digress… like I said, you got my programming-brain all excited

If it was up to me, I would love an Activerecord HQL/ejb-ql compatible ORM. This might actually be possible with jruby; merge of the HQL parser with activerecord. The one critical thing missing in activerecord is lack of distributed sql cache. Sure you can use page fragment caching, but then you have to man handle it manually

Anyway good luck with digg, you have a nice site here.

Great tips, really appreciate you taking the time to write all that up.

Out of curiosity have you guys dabbled with JPA or other persistence technologies like iBatis, or has Hibernate always given you what you needed and done it well enough that you never had to stray?

Thanks for replying to my comment

More than enough room to cache hot spots on the site. Since Hibernate caches queries it knows when to expire those caches automatically. So you get free cache expiration without any intervention.

Offload as much into ram as possible. Do the obvious database optimizations, indexes for large tables where you run constraints or joins on. 8kb filesystem recordsize for efficient database writes for postgres and 16kb for mysql innodb. ZFS really kicks ass in this department, you are able to create partitions and set recordsizes on the fly.

I had two jetty application server instances load balancing the site on the server. To avoid heavy garbage collections I would kill off the process when RSS ram got high ~510megs. This was done via crontab script that ran at 5 minute intervals. Solaris’s SMF would restart the process automatically.

All in all it worked great after the first two complete meltdowns from the digg effect. Code base optimization and configuring the right settings does wonders. RAM is your best friend.

@victori, that’s insanely nice man… server load of 1.0 serving up that many pages. Do you just stick an unholy amount of ram in that machine and set the heap space up to like 4GB so Hibernate’s 2nd level cache can get as big as it wants?

Lessons learned:

Keep max keep alives to ~10-15 seconds and make sure your process has enough rlimit/maxfile count to support enough concurrent clients. Slow clients kill your httpd.

pfiles `pgrep nginx` for example will give you your maxfile descriptor count.

Can’t stress enough how much nginx kicks ass under load.

I can imagine with a highly customized solution like you were talking about you could really eek out some awesome performance underload, especially with some intelligent caching techniques.

John,
I’m glad you found the article helpful, thanks for posting! As far as abusing the readership (certainly an bad idea for any publication), I think you meant our response to James’s post?

It’s really hard to stomach internet know-it-alls after a long day… someone that drops into a genuinely interesting story and instead of contributing or just not saying anything, takes the time to fill out the comment form *just* to leave some asinine remark about how unimpressed they are and some indirect reference to how smart they are.

I try and discourage that with sarcasm… I can see how my response was a bit more biting than it should have been though. I’ll tone it down in the future.

Despite that, this is a good article – thanks for posting it.

We have hit front page on digg about 5 or 6 times, and have easily pulled 90k page views on dynamic pages without any problems. (Highly optimized Java/Hibernate backend / Async writes for view counter).

-Victor

Thanks guys. Really caught us off guard but we are very thankful to Justin for the story idea as well as all the readers checking it out and leaving feedback.

Also glad this article was helpful too, I was writing it *as* I had a few terminal windows up watching load, mucking with the plugins keeping track of which ones did what and how to keep things running.

Was pretty interesting. I just assumed we would have to host static-pages only for a day or so, I had no idea that some of those handy meta-information type plugins (most popular, related, etc.) could cause so much ancillary DB traffic. In hindsight it makes sense, but at the time I was bamboozled.

Cheers matey skipper you old sailor!