In this situation, a Calc III class might be more helpful. Even without a stats class, you had good intuition. There may be other useful ways to solve your problem, but a triple integral (one for each RNG draw) is one approach:
If you care, the value of x3 is what's being averaged/integrated. The terms inside the parentheses are the uniform PDFs (probabilty density functions). Because the interval is narrower for x3 than for x1, the density (or height) is larger -for a given triplet, 1/x2 >= 1/x1 >= 1/100.
If you think better in code, here's the first equation drawn in R:
rm(list=ls(all=TRUE)) #Clear any existing variables
replicationCount <- 1000000
sum <- 0
for( i in 1:replicationCount ) {
x1 <- runif(n=1, min=0, max=100) #Draw a random uniform score from [0, 100]
x2 <- runif(n=1, min=0, max=x1) #Draw a random uniform score from [0, x1]
x3 <- runif(n=1, min=0, max=x2) #Draw a random uniform score from [0, x2]
sum <- sum + x3
}
print(sum / replicationCount)
The second equation corresponds to your excel code; as you can see, the expected value nicely matches your empirical "13.3-13.4ish" estimate. The first equation (with lower bounds of zeros) might be what you were thinking during the initial post. Because the pdfs are uniform, you're initial assumption was probably correct -the terms can be multiplied independently. Be careful though: that won't work if the distributions aren't uniform. If you don't mind me asking, what are you modeling with those uniform distributions?
Tell me if I didn't explain something well. Do you still believe that your excel code was the correct way to express the problem? Was your only concern the discrepency between 12.5 and "13.3-13.4". If so, you're a tenacious tester.