Raidbots Blog

Web Tools for WoW Nerds

Category Archives: General

4/1 – New Server Inc

I went ahead and started the process for getting an honest-to-god dedicated server for raidbots.  Lots more disk than I currently have, a little more RAM, and a decent dedicated processor.  Once I have access, I’ll be starting to set up the new machine and migrating the code and data over.  It’ll probably take a week or two to find the time to get it all set up and working (not to mention handling DNS and all that fun with minimum downtime).  This is going to take precedence over any other development but once it’s done I’ll be able to dive headlong into making sense of the huge data dump WOL is providing me.  First task once the migration is done – playing around with the Top DPS Bot and possibly creating the Average DPS Bot (to answer the age old question “what is the ACTUAL average DPS across all levels of raiding players (that log to WOL)?”

Nerdy Server Stuff

Onto the details!  I’m moving from Linode to Server Beach.  I love Linode and am sad to leave but they just don’t offer disk space at a price I can afford.  I’m using Mongo as the database for raidbots and my usage require a ton of disk space – I could probably reduce the size of each document by compressing keys more intelligently but the rate of growth would likely still require moving to a server with proportionally more disk space than RAM and CPU.  Also, at this point, I’d rather keep the keys in the documents as readable as possible to make it easier to develop against.

Turns out that most cloud hosting tends to have really limited disk options.  Most of the disk sizes scale in lock step with the amount of RAM on the node.  Elastic Hosts was one exception with reasonable prices but their one datacenter in San Antonio has a pretty high latency to Boston (where I live and develop).  In addition, their bandwidth was pretty expensive.  Amazon’s EC2 tended to be far too expensive for me as well – also, any solution that requires a price calculator that is more complicated than raidbots itself is probably more than I want to deal with.

The new server is a Core 2 Duo 6750, 2GB RAM, 2×160 GB drives, and essentially unlimited bandwidth (10Mbit NIC).  Hopefully I should actually get access to it in a day or two and can start the process of configuring Debian, setting up Mongo/Apache/PHP/etc/etc/etc, and start migrating the code and data.

I’m also hoping that a few downtime blips might have been caused by being on a shared machine.  There was a 5-10 minute downtime this week that seemed to be related to Mongo taking a long time to flush data to disk.  I’m not sure if this is just going to end up happening with a single Mongo instance or if the disk my nodes were using were being saturated by another machine on the same physical host.  We shall see!  At least with a dedicated server I’ll know that it’s my own fault if there’s a massive performance problem.  I’m a little worried that running everything on one box might end up causing some RAM contention issues (Mongo likes to slurp up every bit of available RAM but theoretically should let go of it if needed) but there’s only one way to find out.

Anyway, I’m excited if you couldn’t tell.  Once this is set up, I should be able to jump back into developing code and doing cool stuff with the new data.  It’ll be a huge relief not to have to spend a ton of time working on maintenance tools to make sure I’m not overstepping my minimal resources.

Update: I love it when companies under-promise and over-deliver.  They said it would be ready in 24-48 hours but it was ready in about 4.  I love the smell of a new server in the evening.

Advertisements

2/27 Updates

CompareBot

  • ‘Targets’ and ‘Healing Taken’ tabs have relevant data for new compares.  The ‘Targets’ tab compares information from the ‘Damage by Actor’ tabs from World of Logs.
  • Found some bugs with totals and crit %s in the Healing tab (was double counting crits in some cases).  This should all be fixed up.
  • Heroic Magmaw and Heroic Omnotron Defense System have a few more debuffs and mistakes tracked.

General

  • Reduced whitespace in a few places to get more information above the fold.  Overall, things are a little busier but I think the slightly higher information density is good.  Let me know if you disagree!

Things Might Be Slow

Running out of disk space on the raidbots database host (~80k parses aren’t small!) so I’m going to be doing some maintenance in the hope that it can compact some space.  This could make the site less responsive or be down completely.  Thank you for your patience!

Update: Scratch that.  Don’t have enough disk space on my little VPS slice to repair my database.  Will have to search for other solutions.

4.0.6 Numbers

4.0.6 is out!  The numbers for specs will start being gathered tonight (data is pulled every night at about 2AM EST and it takes 2-3 hours for the full run).  DPS increases will probably show and be clear within the next few days as guilds get their kills going this week and new records are set.  Specs that got hurt might take a little while longer to see a clear effect as new records may not be set as quickly for them.  Regardless, it’ll be fun to watch over the next few days and weeks to see how 4.0.6 shapes up.

High Traffic Flakiness

Raidbots is getting a decent amount of traffic from an MMO-Champion forum thread.  It definitely looks like some requests aren’t going through at all and people are just getting errors.

I’m playing with apache configs right now to see if I can actually make the site work for all the visitors.

Thanks for coming!  I’m working as fast as I can to make sure you can see the site.

Update: I think everything is back to normal.  Requests have gone down and I made a few configuration changes to my web server that should allow it to handle more traffic at once.  I’ll keep watch to make sure it all stays stable though.

2/6 Updates

CompareBot

  • Compares can now be marked as “Unlisted” – right now this only controls whether or not the compare will show up in the Recent Comparisons list on the front page.  In time I would also use it to control whether or not parses come back in search results if I get to that.

Top DPS

  • Patch dates (or other notable game changes) can be marked on the timeline chart.  Given that 4.0.6 should come out on Tuesday, that date should be marked for a nice indication to see how DPS is affected.  Note that it’ll probably take a some days/weeks to see the full effects of the changes in the charts since I pull the last 2 weeks of data.

I’ve made the move to using git to control the live and dev environments as well.  Hopefully this should make it easier to make sure I don’t make mistakes when pushing new code to the live site. I’m new to git so I’ll probably make some mistakes as I go (like I did briefly tonight and took the site down).

Raidbots.com! More Stuff!

Raidbots.com is now live!

The new historical Top DPS stuff is all there.  The one launch caveat is that there is not yet a lot of data for heroics, 10 mans, or the summaries.  These are gathering data daily and should start making more sense once there’s 2+ weeks of parses seen.

Let me know if you run into any problems or if you see any old links that don’t properly redirect!

New Site Almost Ready

Got a bunch of exciting stuff that’s so close to being ready I can taste it.  Here’s what’s coming:

  • New domain that should hopefully be easier to remember/find
  • Top DPS Timeline – this is a historical view of DPS that should be particularly interesting in the days/weeks after patches.  I’m really happy with how this tool is turning out.
  • Overall DPS breakdowns for all formats (10N, 10H, 25N, 25H)
  • DPS data for 10 mans
  • My own servers.  I’m hugely appreciative of the hosting Kyth offered on StratFu but it’s time to move to my own hardware so she can spin down some of her own.  It’ll just be running on 2 fairly small Linodes but I think it should be able to handle the traffic.  I’ll be watching carefully to see if it all works nicely.

I’ll probably be doing all the migration work this weekend so keep on eye out for the new stuff!

1/13 Update

Top DPS

  • Hopefully fixed a nagging issue where the World of Logs link was incorrectly using ICC as the raid in all links.  This should be resolved over night as the pages are generated.

Also, I’m getting closer to finishing the historical view of the Top DPS.  It’s coming along nicely and should provide a pretty nice way to watch classes gain/lose relative power after patches or as higher level gear becomes available to the top ranking players.

Welcome New and Old Visitors!

So yeah, got linked from WoW Insider and traffic has soared.  Let me know if you like what you see, if there are any problems, or anything else that floats your boat.

Given the new interest, I will probably be adding 10 man specific information to the CompareBot since a lot of people have been running compares on 10 man content.  I’ve been running 25 man so haven’t had my own logs to look at for that information but I’ll go scour other guilds and get that up and running.

Next step after that is to actually style up and finish UI for the historical DPS averages for the Top DPS Tool.  Data is being collected every day when the pages get refreshed but there’s no place to actually see that info right now.