Web Tools for WoW Nerds
I went ahead and started the process for getting an honest-to-god dedicated server for raidbots. Lots more disk than I currently have, a little more RAM, and a decent dedicated processor. Once I have access, I’ll be starting to set up the new machine and migrating the code and data over. It’ll probably take a week or two to find the time to get it all set up and working (not to mention handling DNS and all that fun with minimum downtime). This is going to take precedence over any other development but once it’s done I’ll be able to dive headlong into making sense of the huge data dump WOL is providing me. First task once the migration is done – playing around with the Top DPS Bot and possibly creating the Average DPS Bot (to answer the age old question “what is the ACTUAL average DPS across all levels of raiding players (that log to WOL)?”
Nerdy Server Stuff
Onto the details! I’m moving from Linode to Server Beach. I love Linode and am sad to leave but they just don’t offer disk space at a price I can afford. I’m using Mongo as the database for raidbots and my usage require a ton of disk space – I could probably reduce the size of each document by compressing keys more intelligently but the rate of growth would likely still require moving to a server with proportionally more disk space than RAM and CPU. Also, at this point, I’d rather keep the keys in the documents as readable as possible to make it easier to develop against.
Turns out that most cloud hosting tends to have really limited disk options. Most of the disk sizes scale in lock step with the amount of RAM on the node. Elastic Hosts was one exception with reasonable prices but their one datacenter in San Antonio has a pretty high latency to Boston (where I live and develop). In addition, their bandwidth was pretty expensive. Amazon’s EC2 tended to be far too expensive for me as well – also, any solution that requires a price calculator that is more complicated than raidbots itself is probably more than I want to deal with.
The new server is a Core 2 Duo 6750, 2GB RAM, 2×160 GB drives, and essentially unlimited bandwidth (10Mbit NIC). Hopefully I should actually get access to it in a day or two and can start the process of configuring Debian, setting up Mongo/Apache/PHP/etc/etc/etc, and start migrating the code and data.
I’m also hoping that a few downtime blips might have been caused by being on a shared machine. There was a 5-10 minute downtime this week that seemed to be related to Mongo taking a long time to flush data to disk. I’m not sure if this is just going to end up happening with a single Mongo instance or if the disk my nodes were using were being saturated by another machine on the same physical host. We shall see! At least with a dedicated server I’ll know that it’s my own fault if there’s a massive performance problem. I’m a little worried that running everything on one box might end up causing some RAM contention issues (Mongo likes to slurp up every bit of available RAM but theoretically should let go of it if needed) but there’s only one way to find out.
Anyway, I’m excited if you couldn’t tell. Once this is set up, I should be able to jump back into developing code and doing cool stuff with the new data. It’ll be a huge relief not to have to spend a ton of time working on maintenance tools to make sure I’m not overstepping my minimal resources.
Update: I love it when companies under-promise and over-deliver. They said it would be ready in 24-48 hours but it was ready in about 4. I love the smell of a new server in the evening.
Running out of disk space on the raidbots database host (~80k parses aren’t small!) so I’m going to be doing some maintenance in the hope that it can compact some space. This could make the site less responsive or be down completely. Thank you for your patience!
Update: Scratch that. Don’t have enough disk space on my little VPS slice to repair my database. Will have to search for other solutions.
4.0.6 is out! The numbers for specs will start being gathered tonight (data is pulled every night at about 2AM EST and it takes 2-3 hours for the full run). DPS increases will probably show and be clear within the next few days as guilds get their kills going this week and new records are set. Specs that got hurt might take a little while longer to see a clear effect as new records may not be set as quickly for them. Regardless, it’ll be fun to watch over the next few days and weeks to see how 4.0.6 shapes up.
Raidbots is getting a decent amount of traffic from an MMO-Champion forum thread. It definitely looks like some requests aren’t going through at all and people are just getting errors.
I’m playing with apache configs right now to see if I can actually make the site work for all the visitors.
Thanks for coming! I’m working as fast as I can to make sure you can see the site.
Update: I think everything is back to normal. Requests have gone down and I made a few configuration changes to my web server that should allow it to handle more traffic at once. I’ll keep watch to make sure it all stays stable though.
I’ve made the move to using git to control the live and dev environments as well. Hopefully this should make it easier to make sure I don’t make mistakes when pushing new code to the live site. I’m new to git so I’ll probably make some mistakes as I go (like I did briefly tonight and took the site down).
Raidbots.com is now live!
The new historical Top DPS stuff is all there. The one launch caveat is that there is not yet a lot of data for heroics, 10 mans, or the summaries. These are gathering data daily and should start making more sense once there’s 2+ weeks of parses seen.
Let me know if you run into any problems or if you see any old links that don’t properly redirect!
Got a bunch of exciting stuff that’s so close to being ready I can taste it. Here’s what’s coming:
I’ll probably be doing all the migration work this weekend so keep on eye out for the new stuff!
Also, I’m getting closer to finishing the historical view of the Top DPS. It’s coming along nicely and should provide a pretty nice way to watch classes gain/lose relative power after patches or as higher level gear becomes available to the top ranking players.
So yeah, got linked from WoW Insider and traffic has soared. Let me know if you like what you see, if there are any problems, or anything else that floats your boat.
Given the new interest, I will probably be adding 10 man specific information to the CompareBot since a lot of people have been running compares on 10 man content. I’ve been running 25 man so haven’t had my own logs to look at for that information but I’ll go scour other guilds and get that up and running.
Next step after that is to actually style up and finish UI for the historical DPS averages for the Top DPS Tool. Data is being collected every day when the pages get refreshed but there’s no place to actually see that info right now.