Problems with the original code
Your strategy appears to be to pre-roll a bunch of extra d6s in RR, and then "pop" them whenever you need to process a reroll or explosion, replacing or appending to the defense pool respectively. (I'm also taking this to mean you intend for each explosion to be a separate die rather than adding onto the initial die -- the opposite interpretation as Ilmari Karonen's answer.)
This would work well if you were processing a single roll rather than computing the full distribution. Unfortunately, there are several problems with the original code:
Sequences expanded from dice pool arguments are always sorted
When you send e.g. 20d6 to the RR parameter, RR comes out sorted inside the function. So you will incorrectly always see the 6s first. That's also why 25d6 is affecting the results more than expected, since that's 5 more opportunities to frontload additional high rolls.
"Pre-rolling" large numbers of dice in their original order is too costly
Unfortunately, there's no easy way in AnyDice to get RR in their original order. Even if you could, it would make the calculation much more expensive. For 20d6, this would increase the number of possibilities for RR
$$
\text{from} \qquad \binom{20 + 6 - 1}{20} = 53,130 \qquad \text{to} \qquad 6^{20} = 3,656,158,440,062,976
$$
So pre-rolling 20 dice was never going to work in terms of computational cost.
You're appending too many dice
When you do {DROLLS, RR}, you're appending the entire RR sequence to DROLLS rather than a single die at a time. You probably meant something like {DROLLS, RRINDEX@RR}. The handling of RRINDEX appears off as well; remember that AnyDice starts indexing at 1, not 0.
There's no reason to continue past the first kept attack die
Since the highest remaining dice in each pool are always the ones being compared, if the attack die beats the defense die, it will beat all remaining defense dice as well. So as soon as we see an uncancelled attack die, that die is the result, and there is no reason to consider the rest of the dice.
"Binning" defense dice
A more feasible way to handle this random-sized dice pool is to first determine how many defense dice (including explosions) fell into each of three categories:
- Those that rolled a 6.
- Those that rolled 2-5 ("mids"). We don't immediately determine exactly what number these dice rolled, only how many dice rolled within this range. These mids could come from either the initial level or the post-explosion levels.
- Those that rolled a 1. 1s can only occur post-explosion, since all initial-level 1s are rerolled. If the initial level were the same as the exploding levels, we wouldn't need to go through this whole binning process.
Note that we choose to split between 2-5s and 1s instead of splitting between initial-level 2-5s and exploding-level 1-5s. This is because a pool of flat 1s doesn't expand into multiple possibilities. Minimizing the number of non-trivial pools is important for efficiency; the "pre-rolling" problem above can be seen as the difference between one pool of 20 dice and twenty pools of 1 die each.
Using digits of a die to represent a vector
This strategy does require a joint distribution of the three. It is possible, though a bit cumbersome, to represent a joint distribution in AnyDice using digits. We start with
d{1, 10:4, 100}
Here we place the ones in the ones place, the mids in the tens place, and the sixes in the hundreds and thousands places.
Explosions and rerolls
function: reroll N:n on ROLL:n {
if ROLL != N { result: ROLL }
}
DEFDIE: [reroll 1 on [explode d{1, 10:4, 100}]]
Here we don't return anything if the die is to be rerolled, which in AnyDice eliminates it from the probability space as desired. Note that the reroll goes outside the explode so that it only rerolls the initial level.
From a single die to a pool
At this point, the digits of DEFDIE represent the sixes, mids, and ones produced by a single defense die. To find the combined result of several defense dice, we can simply add them together using the d operator since each individual digit adds together:
function: attack ATK:n defense DEF:n {
result: [attack ATK defense counts 10000 + DEFdDEFDIE]
}
Here we add 10000 in order to make sure the result always has the same number of digits. AnyDice indexes digits starting from the most significant by default, so if we have a varying number of digits, we could get the wrong digits when we go to extract each component. If you want, you can index from the least significant digit instead by using
set "position order" to "lowest first"
but note that this also affects the order of sorting.
Extracting the digits
function: attack ATK:n defense counts DEFCOUNTS:n {
SIXES: {2:10,3}@DEFCOUNTS
if (SIXES >= ATK) {result: 0}
result: [attack ATKd6 defense SIXES and (4@DEFCOUNTS)d(d4+1) and 5@DEFCOUNTS]
}
Here, {2:10,3}@DEFCOUNTS takes the thousands place, multiplies by 10, and adds the hundreds place. This extracts the number of sixes. We exit early if there are enough sixes to cancel all the attack dice; the inner function call is fairly expensive since it is expanding two pools at the same time.
The last two arguments likewise extract the number of mids and ones, producing a dice pool for the mids.
We need the separate call level here because we want to extract all three bin counts from the same roll of DEFCOUNTS rather than from three independent rolls.
Computing the final result
Finally, we can compare elements and compute the final result. We don't need an extra sort because the elements are already sorted.
function: attack ATK:s defense SIXES:n and MIDS:s and ONES:n {
DEF: {6:SIXES, MIDS, 1:ONES}
loop I over {1..#ATK} {
if I@ATK > I@DEF {
result: I@ATK
}
}
result: 0
}
Here is the full script on AnyDice.
Icepool
Unfortunately, AnyDice times out beyond around 4 dice on each side. Here's the same strategy in my own Icepool Python probability package, which can compute larger pools in a reasonable amount of time.
First, we compute the distribution resulting from a single initial defense die. Starting with a d6, we create a Vector-valued die whose elements are the counts of 1s, 2-5s, and 6s.
from icepool import Pool, Reroll, d, map, vectorize
depth = 3
def make_def_die(roll, reroll_ones, depth):
if depth == 0:
return vectorize(0, 0, 0)
if roll == 1:
if reroll_ones:
return Reroll
else:
return vectorize(0, 0, 1)
elif roll == 6:
# Explode to the next level. Post-explosion 1s are not rerolled.
return vectorize(1, 0, 0) + d(6).map(make_def_die, False, depth-1)
else:
return vectorize(0, 1, 0)
def_die = d(6).map(make_def_die, True, depth)
depth = 3 is the most similar to the AnyDice code provided before, but you can try higher if you like.
Then, based on the binning, we can construct a corresponding mixed pool of 1s, (d4 + 1)s, and 6s using another function:
def make_def_pool(sixes, mids, ones):
return Pool([6] * sixes + [1] * ones + [d(4) + 1] * mids + [1] * ones)
From here we can make the final computation:
def compute_attack(atk_count, def_count):
atk_pool = d(6).pool(atk_count)
def_pool = (def_count @ def_die).map_to_pool(make_def_pool)
return atk_pool.sort_pair('>', def_pool, extra='keep').highest(1).sum()
atk_pool is just a pool of d6s.
- Since
Vectors add component wise, we can roll our desired number of def_die and add them together to get the total numbers of 1s, 2-5s, and 6s across them. This is what def_count @ def_die does. Note that depth effectively applies to each individual defense die.
- Then we use
map_to_pool and make_def_pool to convert this binning into a Pool.
sort_pair effectively sorts both pools, pairs them up 1:1 in descending order like RISK, keeps the elements from atk_pool that beat their corresponding defense die, and keeps the single highest. (If all attacker dice were eliminated, nothing gets summed and the result is 0.) Internally, this line uses a dynamic programming algorithm instead of actually enumerating all possible sorted rolls of the two pools, but the details are a much longer story.
Example result of output(compute_attack(4, 4)):

You can try this in your browser here.
When a result of 1 or 2 is impossible
Ilmari Karonen's answer, which took the interpretation that explosions are added to the initial die, noted that when the number of defense dice is equal to or greater than the number of attack dice, a result of 1 or 2 is impossible. It turns out this is also true if the explosions instead produce separate dice: No attack die will go unpaired, so for an attack die of 2 to win it must be paired with a 1; however, since the defense can only roll a 1 after exploding, each 1 on defense implies at least one additional 6 was also added to the defense pool, which would have pushed the 1s out of consideration in the first place.