Armory Musings...

On the topic of Gear Scoring

2009-02-09T20:11:00.001Z

A few recent comments have pointed out that the reports as they stand at the moment include everyone at level 80 regardless of gear level - from players that 'just dinged' seconds before my spider saw them all the way through to those who have been raiding since the WotLK launch-week. Gear scoring is something I wanted to use to extract different groups of players in the past, but I've finally got around to implementing it.

All items in the game have an internal 'item level' attribute as well as its more widely known rarity value (uncommon, rare, epic etc.), generally the higher value the item the higher the 'item level'. There are even achievements[1,2] linked to the 'item level' of your gear. The simplest way to create a 'gear score' is just to add up the 'item level's, if you want to make it more fair/accurate include a factor/multiplier based on rarity - giving an epic item a higher score than a rare item at the same 'item level'.

I was curious as to how the fansites and addons calculate their scores, so here is a quick summary:

Be Imba! - Their score includes Gear, Gems and Enchants. I'm not particularly interested in Gems & Enchants; I'm looking for a way to discriminate on Gear-only.

Warcrafter - Similar to Be Imba! their score appears to include Gems and Enchants as well as Gear.

GearScore (addon) - A pretty interesting spin on gear scoring, it works by adding up at the stats (+sta, +agi, +spell power) provided by the gear regardless of the 'item level'.

FuBar_GearRating (addon) - An inactive project, but still works with the latest FuBar on patch 3.0.8. It multiplies the 'item level' with a few factors including the rarity, slot and class of the character (e.g. a Hunters ranged slot item is more important than an idol/relic/wand slot item).

The method used by FuBar_GearRating turned out to be my favourite approach. I am not interested in Gems or Enchants, they are more indicators of the gold the characters are willing to spend than the gear level they have access to.

I implemented the same algorithm used by FuBar_GearRating in my character database, a quick sanity check proved the score I came up with matched the addon's in-game results. Next, I ran it on 249,882 level 80 characters and I got the following results.

The raw data behind this graph is available as a simple CSV file here:
http://armory-musings.appspot.com/download/gearScore_20090209.csv

There are 64,676 characters (25.8% of the sample) with a score of 200 or more, only 3,463 of those have a score of over 215. The long tail of characters with a score of less than 100 appears to be those who are practically naked or wearing RP outfits and the like.

Overall I am quite happy with the results, I'll be using this new data to partition the reports I generate. Also a recent brainwave concluded that anyone with a Resilience of over 200 is probably spec'd/geared for PvP and should also be in a separate report.

Comments / questions / suggestions welcomed :)

Class & Realm reports, updated automagically every 24 hours:
http://armory-musings.appspot.com/reports

The return of the Realm balance reports...

2008-12-18T12:01:00.004Z

My spider software is still hurtling through lists of guilds and whatnot looking specifically for level 80 characters. Currently there are 260,389 level 80 characters used to generate the reports but this is ticking up by the second. I still have 50,464 guilds in the 'pending' queue waiting to be checked. Eventually the 71-79 characters will be refreshed as I expect a large number will have levelled by the time I return to them.

In other news, I have released a lot more information on the reports page. Each realm now has a separate 'Realm Balance' report showing the balance of classes for both Alliance and Horde on that realm. There is also a complete list of Guilds known on that realm. Don't panic if your realm report is a little on the empty side, your realm and its guilds are likely still somewhere in the 50,000+ TODO queue.

Lastly, I have released some sample data from 3,000 level 80 characters in both CSV and XML formats on the Developers page for you to play with. Limits on the size of files in Google App Engine has made it tricky to release larger data-sets but I have something planned.

Refreshed automagically every 6 hours:
http://armory-musings.appspot.com/reports

Armory XML: permanentenchant attribute decoded

2008-12-04T09:47:00.002Z

Something that really bothered me with the previous reports was the mysterious "permanentenchant" attribute on each 'slot' element in the Armory's XML. It didn't correlate to anything useful on the usual fan sites, as far as I could tell it wasn't an 'item', 'spell' or 'object' ID as far as WowHead was concerned.

This morning I wrote a really simple script that does the following:

Collate a list of all the permanentenchant attribute values from my database.

For each permanentenchant ID:

- Locate a character (realm & name) with that enchant in the correct slot in my database

- Fetch the item tooltip for that character's enchanted slot and parse the 'enchant' element

Now I had a list mapping the permanentenchant ID to the text displayed in the tooltip '2669' -> '+40 Spell Power'. Pretty neat start, but not quite what I wanted - I needed to figure out that '+40 Spell Power' on a Weapon actually means 'Enchant Weapon - Major Spellpower'.

Next, I try a pretty dumb lookup on WowHead '+40 Spell Power' didn't work but others like 'Unholy Weapon' were correctly identified.

For anything that's left, I compare the '+40 Spell Power' to the huge enchantment list on WoWWiki and then return to WowHead with the name of the enchant as seen on WowWiki.

One clever thing I did add is the distinction between enchantments that are spells (by Enchanters) and enchantments that are items (Spellthread).

Thirty seconds later the script has finished and my list now maps:
permanentenchant:'2669' -> tooltip:'+40 Spell Power' -> spell:'27975'

The result of all this is I now publish an XML file from my database with all the permanentenchant ID's mapped either to a 'spell' or 'item' ID suitable for WowHead. There are a few in the list where both 'spell' and 'item' are 0 - these haven't been figured out and will require manual attention to update.

http://armory-musings.appspot.com/developers/files/permanentenchant.xml

This XML File is published under a Creative Commons Attribution 3.0 License, you are free to copy, distribute and adapt the data provided credit is given to the source of the data (i.e. a link to my site).

New Dataz!
http://armory-musings.appspot.com/

New Website, New Data, Daily Updates

2008-12-01T12:27:00.000Z

I'm pleased to announce the launch of my new Armory Musings... project website. Information is updated (at least) daily direct from my spider database. The format of the reports will be improved over the next few days but all the data is under a Creative Commons license meaning you can copy, distribute and adapt the data in my reports as long as you link back to the original source (my site).

http://armory-musings.appspot.com/

Armory updates, and how to get at the new data

2008-11-27T11:02:00.007Z

On the 25th of November a major Armory update added a whole host of new information with the introduction of the Statistics tab. Unlike the other tabs on the Armory where all the information is present in the XML this new Statistics tab fetches more data asynchronously.

Firstly if you have a look at the character-statistics.xml you will see something like this:

http://eu.wowarmory.com/character-statistics.xml?r=(realm)&n=(name)

  <statistics>
    <summary>
      <statistic name="Total gold acquired" quantityCopper="43" quantityGold="828" quantitySilver="89"/>
      <statistic name="Gold looted" quantityCopper="94" quantityGold="61" quantitySilver="3"/>
      <statistic name="Total kills that grant experience or honor" quantity="932"/>
      <statistic name="Total Killing Blows" quantity="997"/>
      <statistic name="Creatures killed" quantity="1907"/>
    </summary>
    <rootCategories>
      <category id="130" name="Character">
        <category name="Wealth"/>
        <category name="Consumables"/>
        <category name="Reputation"/>
        <category name="Gear"/>
      </category>
      <category id="141" name="Combat"/>
      <category id="128" name="Kills">
        <category name="Creatures"/>
        <category name="Honorable Kills"/>
        <category name="Killing Blows"/>
      </category>
      <category id="122" name="Deaths">
        <category name="Arenas"/>
        <category name="Battlegrounds"/>
        <category name="Dungeons"/>
        <category name="World"/>
        <category name="Resurrection"/>
      </category>
      <category id="133" name="Quests"/>
      <category id="14807" name="Dungeons & Raids">
        <category name="Classic"/>
        <category name="The Burning Crusade"/>
        <category name="Wrath of the Lich King"/>
      </category>
      <category id="132" name="Skills">
        <category name="Secondary Skills"/>
        <category name="Professions"/>
      </category>
      <category id="134" name="Travel"/>
      <category id="131" name="Social"/>
      <category id="21" name="Player vs. Player">
        <category name="Arenas"/>
        <category name="Battlegrounds"/>
        <category name="World"/>
      </category>
    </rootCategories>
  </statistics>

The <summary> section will be different for everyone but the category id's appear to be fixed. When you click the Character category (id 130 as you can see above) an XML call fires off. Low and behold an XML document comes back looking like this:

http://eu.wowarmory.com/character-statistics.xml?r=(realm)&n=(name)&c=130

<category name="Character" requestUrl="/character-statistics.xml">
  <statistic name="Talent tree respecs" quantity="--"/>
  <statistic name="Badges of Justice acquired" quantity="--"/>
  <statistic name="Emblems of Heroism acquired" quantity="--"/>
  <statistic name="Emblems of Valor acquired" quantity="--"/>
  <category name="Wealth">
    <statistic name="Total gold acquired" quantityCopper="43" quantityGold="828" quantitySilver="89"/>
    <statistic name="Average gold earned per day" quantityCopper="96" quantityGold="46" quantitySilver="4"/>
    <statistic name="Gold looted" quantityCopper="94" quantityGold="61" quantitySilver="3"/>
    <statistic name="Gold from quest rewards" quantityGold="473" quantitySilver="54"/>
    <statistic name="Gold earned from auctions" quantity="--"/>
    <statistic name="Auctions posted" quantity="--"/>
    <statistic name="Auction purchases" quantity="5"/>
    <statistic name="Most expensive bid on auction" quantityGold="110"/>
    <statistic name="Most expensive auction sold" quantity="--"/>
    <statistic name="Gold from vendors" quantityCopper="49" quantityGold="294" quantitySilver="31"/>
    <statistic name="Gold spent on travel" quantityCopper="90" quantityGold="7" quantitySilver="87"/>
    <statistic name="Gold spent at barber shops" quantityCopper="50" quantityGold="7" quantitySilver="29"/>
    <statistic name="Gold spent on postage" quantityCopper="30" quantitySilver="27"/>
    <statistic name="Gold spent on talent tree respecs" quantity="--"/>
    <statistic name="Most gold ever owned" quantityCopper="14" quantityGold="601" quantitySilver="99"/>
    <statistic name="Items pickpocketed" quantity="108"/>
  </category>
  <category name="Consumables">
    <statistic name="Bandages used" quantity="--"/>
    <statistic name="Bandage used most" quantity="--"/>
    <statistic name="Different bandage types used" quantity="--"/>
    <statistic name="Health potions consumed" quantity="1"/>
    <statistic highest="Super Healing Potion" name="Health potion used most" quantity="1"/>
    <statistic name="Different health potions used" quantity="1"/>
    <statistic name="Mana potions consumed" quantity="22"/>
    <statistic highest="Super Mana Potion" name="Mana potion used most" quantity="22"/>
    <statistic name="Different mana potions used" quantity="1"/>
    <statistic name="Elixirs consumed" quantity="--"/>
    <statistic name="Elixir consumed most" quantity="--"/>
    <statistic name="Different elixirs used" quantity="--"/>
    <statistic name="Flasks consumed" quantity="--"/>
    <statistic name="Flask consumed most" quantity="--"/>
    <statistic name="Different flasks consumed" quantity="--"/>
    <statistic name="Beverages consumed" quantity="126"/>
    <statistic highest="Pungent Seal Whey" name="Beverage consumed most" quantity="98"/>
    <statistic name="Different beverages consumed" quantity="9"/>
    <statistic name="Food eaten" quantity="10"/>
    <statistic name="Different foods eaten" quantity="3"/>
    <statistic highest="Poached Bluefish" name="Food eaten most" quantity="8"/>
    <statistic name="Healthstones used" quantity="--"/>
  </category>
  <category name="Reputation">
    <statistic name="Most factions at Exalted" quantity="4"/>
    <statistic name="Most factions at Revered or higher" quantity="9"/>
    <statistic name="Most factions at Honored or higher" quantity="18"/>
    <statistic name="Most Northrend factions at Exalted" quantity="--"/>
    <statistic name="Most Outland factions at Exalted" quantity="3"/>
    <statistic name="Most Alliance factions at Exalted" quantity="1"/>
    <statistic name="Total factions encountered" quantity="39"/>
  </category>
  <category name="Gear">
    <statistic name="Equipped epic items in item slots" quantity="2"/>
    <statistic name="Epic items acquired" quantity="1"/>
    <statistic name="Epic items looted" quantity="--"/>
    <statistic name="Legendary items acquired" quantity="--"/>
    <statistic name="Mounts owned" quantity="3"/>
    <statistic name="Vanity pets owned" quantity="3"/>
    <statistic name="Extra bank slots purchased" quantity="7"/>
    <statistic name="Greed rolls made on loot" quantity="1"/>
    <statistic name="Need rolls made on loot" quantity="--"/>
  </category>
</category>

You can retrieve any of the Statistic sub-pages independently of the rest of the character information. I haven't spent a lot of time looking into it but the category id's appear to be fixed and everyone seems to have the same set of categories on display (for the moment).

Blessing of Salvation

2008-11-24T19:43:00.000Z

It’s been almost a year since my last post… With the recent launch of WotLK I have decided to resurrect this project and flood it with new data.

In a few days I expect to have the first batch of new data covering 100,000’s of level 70-80 characters. I am collecting data on lower level characters too, but mainly to check on them later when they have levelled up. I am concentrating the information on the level 70-80 range.

All the data will be hosted on a new website and updated daily directly from my database. The link to this new site will be posted later this week.

All data I publish from now on will be under the Creative Commons Attribution 3.0 License meaning you can copy, distribute and adapt my data provided you link back to the original source (i.e. my site).

In the mean time if you have any questions or suggestions, you can email at the usual address: okoloth (at) gmail (dot) com

Linking to World of Warcraft items from Blogger (Revisited)

2007-11-27T12:42:00.000Z

Some time ago I wrote a simple piece of HTML/Javascript to produce item tooltips driven by Allakhazam. I'm pleased to retire this script in favor of some new code from Wowhead.

The simple instructions are here and are summarised below:

1). Add this piece of HTML code anywhere on your page:
<script src="http://www.wowhead.com/widgets/power.js"></script>

2). Add a regular HTML link to an item in Wowhead, example:
<a class="q5" href="http://www.wowhead.com/?item=19019">Thunderfury, Blessed Blade of the Windseeker</a>

Questions & Answers (Updated)

2007-11-13T12:00:00.000Z

To answer as many questions as possible I decided to write a new post, I will update and re-post this page as I answer more questions.

Q. How do you collect this data?
A. I’ve alluded to the way I collect the data in previous posts. I have written a specialised piece of software specifically to data mine the Armory and other sites. For example, to get it started I can point it at the Armory Arena Ladder pages where all participating teams are listed. For each Arena Team the team member characters are fetched, if they are guilded then the guild list is fetched and all guild members are retrieved. This technique has some problems; players that are not guilded and not part of an arena team are ‘invisible’. I have thought about different ways to counter this but I’m pretty satisfied that the data is representative anyway.

Q. Do you use the ‘last modified’ or ‘last logged in’ parameters to determine if the character is still used?
A. I record all that information but I don’t discriminate against ‘stale’ characters, if you look at the ‘Level Count’ in my earlier post you will see a huge bump at level 60 denoting people who haven’t yet upgraded to The Burning Crusade but also characters that haven’t been played in some time.

Q. Can you produce reports for levels 20, 40, 60 as well as level 70?
A. Absolutely, this is something I really want to get to do. Presenting it is my only sticking point, I will be including levels 1-70 in charts where possible. See the two graphs below for an example of what I am working on producing in the future.

Note: These charts are produced from a tiny subset of data just to test out my charting software. Interestingly, the 'Talent Spec Balance' chart illustrates that for Priests, Shadow is the best levelling spec but the balance quickly switches over to Holy from level 68/69 as Priests become more in demand as healers.

Q. Can you segment your data by Arena Rating / Gear Rating / PvP Kills?
A. Yes, I probably can but at the moment I am concentrating on updating my basic class-specific reports. Arena Rating and PvP Kills are easy to build discriminators on and I have worked on mechanism to segment level 70’s by their ‘gear rating’ (just dinged 70, casual & raider).

Q. Why is realm (name) missing from your stats?
A. I have not included all realms yet, in Europe for example I have not picked up any of the non-English realms.

Q. How are you getting data from levels 1-9, Armory don’t show that?
A. The Armory *used* to show low level characters, I noticed them disappear mid-October. So I still have a lot of 1-9 character data, during each refresh I try to fetch these character pages anyway should they reach level 10 or higher then I will get updated data for them.

Q. Can you show a breakdown of levels/classes by battlegroup, realm, realm type, and Horde / Alliance balance at 70?
A. Absolutely, it has officially been added to my TODO list.

Q. Can you profile most popular classes in arena teams and group them by their rating?
A. Absolutely, this too has officially been added to my TODO list.

Q. Can you breakdown your class-specific stats by faction to see if racials make a difference in class choice for each faction?
A. Yes, this is doable too I’ve kept as much data as possible with a view to do ever more sophisticated reports with it.

UPDATE 13th November 2007

Q. Can you track population changes over time, for example talent balances as patches change dynamics in the game?
A. Yes, originally I wanted to keep full revision history for each character, each time my spider would fetch their character page an 'xml diff' of changes would be stored. Instead, I chose to export my reports in an XML format - not the abstracted percentages you see here but raw numbers, the idea being I could compare reports to show the high-level changes over time. I have this in a limited form for patch 2.1 (May 2007), much richer information since patch 2.2 (Sept 2007) and eventually I will be able to publish the shifts from patch 2.2 to 2.3.

Q. What hardware do you run this on?
A. The spider software, database server, file-storage and report generation is all running from one of my old servers at the moment. Spidering is not particularly intensive as its a fairly slow process. To be specific the machine is a Dual AMD MP 2000+ (1.67Ghz), 2Gb PC2100 DDR RAM, 2x18Gb SCSI discs and 250Gb IDE disc, its running Redhat Linux 9 the only non-standard thing is that the XML repositiory is on a ReiserFS partition. Backup-wise, I have a full mirror of all the data on my home RAID5 partition.

Q. Do your computations take forever?
A. It depends on the type of report, for example the Realm Balance reports are all done using information in the database and take around 5 minutes to run. The reports that breakdown details on characters obviously require the XML character sheets and take much longer; these are done in phases.
   1). Select a list of desired characters from the database by class, level etc.
   2). Iterate through each character, fetch the XML sheet and extract all the required information and add it to a temporary 'report' table in the database.
   3). Use the 'report' table to produce stats, this step can be repeated without re-doing the first two allowing me to develop the reports without having to re-fetch all the data.
This process runs at about 250,000 characters per hour.

Q. Have Blizzard contacted you about the load / bandwidth you are generating?
A. No, Blizzard have not contacted me about the load my spider generates. I invested more time in the 'collector' aspect of the spider than any other part; there is a lot of timing / delays / caching which prevent it from producing a crippling load. The spider supports a huge range of HTTP/1.1 features including things like If-Modified-Since and Accept-Encoding compression if the Armory servers choose to honor them, its User-Agent string clearly identifies this project and a link to this blog and lastly, all traffic comes from a single IP.
Interestingly, over at The Build Mine Kuroshiro spotted a notice on the Armory updates page saying "... third-party sites that mine Armory data may need to make adjustments to account for the new file configurations ..." which implies they are aware of mining projects but for the moment haven't explicitly banned them.

Q. Why do you use an apostrophe to pluralize your class names in the chart titles?
A. Because at the time the apostrophe provided an easy way to separate my PHP variable name from the "s" (see code) which I have recently fixed.

Q. Can you show the Realm Balance reports but only for level 70 characters?
A. Yes, I just posted it.

Q. Can you determine how many Arena Teams are going to be included / disqualified by the tighter rules for titles "20 games played and at least one person with 20% of the total games played" ?
A. Yes, this too has officially been added to my TODO list.

If you have any questions / comments, feel free to post here or email me :)