Mfcttrf

I'm looking at the depth of trees in a random forest model, using the randomForest and randomnForestExplainer package in R.

The model I'm using is a basic linear regression model where there are 3 important predictor variables (p) and the rest are noise (q).

The first test I ran I set p = 3 and q = 10 and found that the mean minimal depth of the variables was never over 7 trees.

However, for the second test I set p = 3 and q = 100 and found that the mean minimal depth of the variables was 17 trees.

this can be seen in the plots below for both tests, where the colour-coded bar on the right displays the minimal depth of each variable.

So, my question is: why does adding more noise variables to my model mean deeper trees?

Splitting on noise features sometimes yields information gain (reduction in entropy, reduction in gini) purely by chance. When more noise features are included, it's more likely that the subsample of features selected at each split will be entirely composed of noise features. When this happens, no signal features are available. By the same token, the split quality is probably lower than it would be had a signal feature been used, so more splits (i.e. a deeper tree) will be required to attain the termination criteria (usually leaf purity).

Typically, random forest is set up to split until leaf purity. This can be a source of overfitting; indeed, the fact that your random forest is pulling in noise features for splits indicates that some amount of overfitting is taking place. To regularize the model, you can impose limitations on tree depth, the minimum information gain required to split, the number of samples to split, or leaf size to attempt to forestall spurious splits. The improvement in model quality from doing so is usually modest. These claims draw from the discussion in Hastie et al, Elements of Statistical Learning, p. 596

More information about how spurious features behave in random forest can be found in https://explained.ai/rf-importance/

Random forests sample variables at each split.

The default is to sample $sqrtp$ variables each time.

If you add more noise variables, the chance of the good variables being in the sample decreases. Hence they tend to appear first, on average, at a deeper level than before.

With 4+10 variables, there is about a 30% chance of each good variable being picked. With 4+100, the chance is only 10.6%. the chance of at least one good variable being available at height 1 are 1-(1-p)⁴, so 83% resp. 36% that one of the good variables is available and otherwise the first split has to use a noise variable etc.