Vault Hunting 101

Discussions of any Borderlands game invariably include a reference to the
capricious whims of the Random Number Generator (RNG) at some point. This is usually in the context of how hard or easy it is to get a specific item, or the
standard waiver that “random is random” (RiR). Such discussions equally
invariably involve claims and counter-claims regarding how the game RNG
works, or how it can itself be “gamed” to encourage the desired result.

This guide is not wholly original, but will attempt to cover the key points in a clear and mostly non-mathematical manner for those who don’t spend their working days dealing with statistics. It draws heavily from previous articles, primarily:

• @HeyCarNut’s massive Guide to RNG and Guide to drop rate reference
• Paul Hellquist’s Inside the Box Loot article

To start with, though, here’s a rare actual photograph of Maya discussing
random number generation with CL4P-TP, courtesy of Heycarnut:

#

Let’s get this out of the way first: Any computer is incapable of generating truly random numbers; technically, they all use a pseudo-random number generator (PRNG). Given how vital it is to scientific research for PRNGs to generate numbers that are as truly random as possible, considerable efforts have gone into figuring out how to do this. Both Borderlands 2 (BL2) and The Pre-Sequel (TPS) is built using the Unreal engine, which provides a handy default random function.

As HeyCarNut found, this uses a simple recursive calculation that generates an integer value based on the value from the preceding calculation. Obviously, the very first time this calculation is performed it needs a number to start with: this is called the initial seed value. The Unreal engine uses the current CPU clock cycle count in order to give a different initial seed every time the game is started, meaning that the actual sequence of pseudo-random numbers generated will be
different each time. So how good is this approach in terms of generating random numbers? To quote HeyCarNut:

One might ask, “How good is such a generator?” Well, it’s decent, certainly good enough for use in games (as stated and it should be obvious, it’s also trivial to implement and very fast)… Games typically use LCGs* as their source of random numbers. These are not perfect, but well-implemented ones provide output that is indistinguishable from “true” randomness as far as the requirements for game logic and behavior are concerned.

* Linear-Congruential Pseudo-Random Number Generator. Now you
know…

TL,DR: BL2 and TPS may ‘only’ use a PRNG, but it’s good enough that you won’t notice a difference from true random behavior unless you spend an unhealthy amount of time collecting in-game statistics.

So how does the game decide what to give you when you kill something, or open a container? Game developer Paul Hellquist described the process pretty well in an [amazingly detailed developer blog post][2]. Every loot source in the game (container, chump, badass, super badass, all the way up to raid boss) has an associated list defining the number of different categories (‘item pools’) of loot that it can provide, along with a weighting factor (or probability) for obtaining an item from each pool.

The weights are expressed as a decimal number between zero and one, zero obviously meaning “never” and one meaning “guaranteed”. When you open a box or kill an enemy, the game looks up the definition list and, for each entry in that list, uses the PRNG to generate a random number between 0 and 1. If the result is more than the weight, you get nothing from that pool; if it is less than the weight, the game repeats this process with a fresh PRNG result for each item within the pool. The overall process is shown nicely in a [flow diagram from Paul’s article][2]:

In Paul’s example, we killed some basic bandit (or scav) who has eight entries in their list definition, meaning they could drop up to 8 items of loot. These are guns and gear; vehicle skins; money; an eridium bar (single moonstone); four eridium bars (moonstone cluster); health; ammo weighted according to the number of players in the game; and one additional chance of extra ammo.

Paul tells us that the weight for the first category in this table is 0.085 (8.5%, or between 1 in 11 and 1 in 12 odds.) So, if you killed an infinite number of chumps, 8.5% of those kills would result in an item from the gear and weapons pool. Note that other classes of loot source (e.g. ultimate badasses, raid bosses, …) have there own list definitions, meaning that they can drop different numbers of items using either the same or modified loot pools.

Let’s pause to address a common misconception: It is often claimed that scaling the output from a PRNG and using a lookup table makes the result less random. While it certainly is possible to manipulate the output of the PRNG in order to make “more random” while having the opposite effect, this isn’t true of simple linear scaling. Neither does comparing the result to a specified value make the outcome (drop or no drop) any more or less random. This is because when we play the game, we are merely sampling the results we would have obtained from an infinite number of monkeys, er, chumps. Our brains are hard-wired to look for patterns (it avoids thinking too hard), and limited samples of truly random outcomes tend to look like they contain a pattern.

This is demonstrated easily by playing the game and keeping track. (We can also do what HeyCarNut did, and simulate playing the game continuously for several weeks or months.) Following the drop rate increase for example, Savage Lee has 1 in 10 odds (10% or a probability of 0.010) of dropping a legendary Unkempt Harold of some flavor. Let’s do 100 farming runs indicating whether we did (1) or did not (0) succeed. Here’s the outcome, in groups of 10 trials each (because I don’t like farming that much!):

• Day 1: 0 1 0 1 0 0 0 0 0 0
• Day 2: 0 0 0 1 0 0 0 0 0 0
• Day 3: 0 0 0 0 0 0 0 0 0 0
• Day 4: 0 0 0 0 0 0 0 0 0 0
• Day 5: 0 0 0 0 0 1 0 0 0 0
• Day 6: 0 0 0 1 1 0 0 0 0 0
• Day 7: 0 0 0 0 0 0 0 1 0 0
• Day 8: 0 0 0 0 0 0 0 0 0 0
• Day 9: 0 0 0 0 0 0 0 0 0 0
• Day 10: 0 1 0 0 0 0 0 0 0 0

Just looking at that data, it’s hard to not notice that the 1st, 3rd, 7th, 9th and 10th columns are all zero. This doesn’t, however, mean every 1st, 3rd, 7th, 9th and 10th time you spawn in Three Horns you should simply quit and restart to save time! After all, if you’d only considered the first 50 tries (first five rows), you’d conclude by the same logic that the 5th, 8th, 15th, 18th, … times were not worth continuing either, but you would have been wrong.

Now suppose I told you that runs 1-10 and 51-60 (rows 1 and 6) were obtained
driving a bandit technical to Savage Lee’s spawn area, while 11-20 and 61-70 were done driving a racer. Notice how there are twice as many successful drops when driving a technical versus a racer? Sweet! It looks like we’ve found a way of
gaming the PRNG by selecting a particular type of vehicle! Except, it just looks
that way, because runs 31-40 and 71-80 were also done driving a technical…

It’s also interesting to see how the observed drop rate varies as the number of runs increases. For our 100 trials, it worked out to 8%, or closer to 1 in 12, but it sure looked higher over the first 20 trials! Here’s how the overall observed drop rate varies for each additional set of ten runs – eventually, we’d expect the observed rate to converge on the actual probability, but it might take a while:

Back Paul’s loot article. Let’s say that the scaled PRNG value came out to 0.079. This is less than the weight of 0.085, so we get one item from the Gear and Weapons pool. This is further broken down into the item sub-categories: weapons; shields; grenade mods; class mods; and relics (Oz kits in TPS). The weights for each item type are again either predefined values, or predefined values multiplied by some specified modifier. In fact there are two predefined weights used (with or without modifiers) in this pool: a value of 100 for “common” and one of 10 for “uncommon” items.

What’s different here is that the resulting weights are used to determine which type of item the drop will be, so a single item from the list. In this case, the individual weights (taking into account their modifiers) add up to 207. Relics are weighted as most rare, so if the scaled result from the PRNG call is less than 0.0483 (=10/207) the item will be a relic. If the result is greater than 0.5169 (=107/207) it will be a weapon of some sort:

(Note: these numbers date to before the drop rate increase; it’s unknown if they are slightly different now.)

So about half the time, our second PRNG call will give us a weapon, but roughly 1 time in 5 we will get a shield, and roughly 1 time in 20 it will be a relic. This process continues in order to determine the item rarity, the specific type of item, its manufacturer, and even its parts and stats.

Item rarity is determined in a similar fashion: there is a specific table defining the probability of getting each rarity from common chumps. Again, Paul gives us the actual numbers from the game as released:

Note that with the drop rate increase, that last entry is now 3 in 10000 – better, but still pretty long odds.

I said earlier that players can’t game the PRNG to favour a particular outcome. Indeed, Paul confirms the earlier assertion regarding the PRNG:

For the purposes of Borderlands 2, all you need to know is that whenever the game rolls these imaginary dice to determine what to drop it has absolutely no knowledge of the previous roll or the previous results and every number has an equal chance of being rolled every time the die is cast. For all intents and purposes it is “random” as laymen use the term.

However, he also states certain very specific circumstances where the game’s logic rules change what loot can drop. Note that this does not require any changes to the PRNG (random is still random); it simply modifies the entries in the loot tables.

We’ve already seen that more ammo drops from basic enemies with more players, but the process also provides an increased chance of finding the ammo you need most. (Tip: open all the ammo crates in an area before picking any of it up; that way, you keep the odds of finding the type you need in your favour.) Similarly, the probability of finding health is increased when your health is low, and you get an additional chance when it’s very low. Other examples are class mods (more likely to get ones for the characters in the party.

For weapons specifically, the table is altered depending on the game stage (i.e. the level set for the current map when you entered it for the first time on your current mission, or the highest level amongst the party in UVHM.) This is why you only see pistols drop at level 2, you won’t get shotguns or sniper rifles before level 5, and if you see someone wielding a level 9 Norfleet you can be certain that it didn’t drop in the game…

It’s pretty common to see forum posts that ask questions like:

• Did GBX secretly nerf [loot source]’s drop rates?
• Did GBX secretly buff [loot source]’s drop rates?
• Is [game feature] glitched? I’m not getting my loot…
• Is [game feature] glitched? I’m getting tons of loot…

While some of these can be attributed to missing key announcements about updates and hotfixes (to be fair, they’re often quite dense!), special events (Loot Hunt, drop rate tests), or are a result of known long-standing issues (Boll, Son of Mothrakk), most of the time they simply reflect a run of either bad or good luck.

Part of this is the fact that true randomness is counter-intuitive: an extremely common misconception is that if an event has a 1 in 10 chance of occurring, then it will always happen once in 10 tries. On average it will but, as we saw [earlier][#RLIR], sometimes you get different outcomes. Indeed, one of the surest signs that the BL2/TPS random loot generator is based on a good PRNG is that you do get runs of good and bad luck.

If you want to know whether your observed drop rate is what it should be, or the result of other factors (secret hotfix or glitch), it turns out that size really does matter – sample size, that is. I’ve already given the example of a dedicated legendary drop from one of the game’s mini-bosses. Here’s a larger example taken from my [survey of grinder success rates][11] in TPS. I wanted to get an idea of the probability of grinding an item of higher rarity without using moonstones. With almost 600 trials at this point, the observed rate for grinding three blues into a purple is 40%, or odds of 2 in 5. Here are the first 20 trials, where 0 indicates no deal and 1 indicates success:

1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0

You’ll notice that I got 4 purples in 20 grinds, which is half the rate I finally arrived at. Each group of 5 trials, however, contains 0, 1, or 2 successful grinds.

So how many times do you have to farm a particular loot source if you want to know either (a) the drop rate, (b) if the rate has been changed, or (c) if you have discovered a new glitch? We could go the statistical route, but I’m going to take a largely practical (and less mathematical) one. The primary guideline I use in deciding when I’ve tested “enough” is: Will one more trial outcome substantially change my observed drop rate?

Suppose, for example, I take the first 20 results with an observed rate of 4/20 (=20%). One more successful trial would change that to 5/21 (= 24%), while a failed trial would give 4/21 (=19%). You can see why the numbers vary so much at the beginning of the run! Ideally, I’d like that variation to be less than ±0.5 percentage points, meaning a minimum of 200 trials; if my observed rate is low – 5% or lower, say – I want to see that variation be even smaller, and that requires substantially more trials.

More relevant to actual gameplay experience is the length of each run of successful and unsuccessful outcomes since that’s what we remember about the game. Here’s the summary for the first few run lengths, where positive values count successive purples and negative ones count successive blues. Notice that there are several runs of quite bad luck, along with some pretty decent runs of good luck:

+1 -7 +1 -8 +2 -1 +1 -6 +1 -1 +2 -1 +3 -2 +4 -1 +1 -4 +4 -2

I don’t know if that’s what you expected to see or not; unless you’ve done a fair bit of statistics, my suspicion is probably the “not” category! As Paul Hellquist stated and HeyCarNut demonstrated, the game’s loot system does not keep track of prior outcomes when determining the next one: each grind (in this case) occurs independently of what happened before.

As a consequence, the probability of getting a purple from three blues in any single grind is always the same (p = 2/5 = 0.40); since this is quite high, the probability of getting purples on two consecutive grinds is still pretty good, at 16% (p(2,2)= (2/5)2 = 0.16). Similarly, for three purples in exactly three grinds, p(3,3) = 0.40^3 = 0.064 (6.4%), and so on. However, as we do more trials, the odds of seeing runs of 2, 3, 4, etc consecutive purples also go up. So although the odds of 9 purples in exactly 9 grinds are pretty slim (p(9,9) = 0.00026 or 0.026%), it’s not an unexpected event over the course of 600 trials; in fact, we can estimate the probability of at least one run of 9 purples in 600 grinds is about 9%.

Barrel Drops Accessory Drops Element Drops Type 1 37 None 73 None 74 Type 2 31 Action 3 Acid 9 Type 3 32 Core 2 Fire 4 Dynamic 4 Shock 6 Earnest 4 Slag 7 Maximized 7 Redundant 4 Win-Win 3