Predicting Relative Shop Performance in Latin America

01Introduction

Small shops differ in the services they offer, the equipment they own, their payment options, and their owners’ attitudes toward the business. These differences may help explain why some shops perform better than others. However, a characteristic that appears related to performance in a descriptive comparison may not help predict the performance of a shop that the model has not seen.

The aim was therefore to test predictive value directly. Three questions guided the study: could the survey variables improve on a simple baseline, would more complex algorithms outperform regularized linear models, and which types of information contributed most to prediction? The study also examined whether the final predictions were accurate enough to be useful at the individual-shop level.

02Data and preparation

The sample and outcome

The cleaned sample contained 1,291 shops from Argentina, Brazil, Colombia, Mexico, and Peru. It covered bakeries, grocery stores, liquor stores, minimarts, and pharmacies. Approximately 80% of the shops were used for development, while the remaining 20% formed a protected test sample.

The outcome was a reconstructed Shop Performance Index (SPI), based on estimated commercial activity rather than verified profit. Weekly activity was estimated by multiplying reported daily customers, days open per week, and a country-specific estimate of spending per customer. Spending estimates came from the survey’s spending bands.

This quantity was converted to a natural logarithm. A separate regression then estimated expected log activity from country, store type, store size, and the logarithm of the number of workers. Subtracting this expectation produced the modeling target: a positive value meant activity above the structural benchmark and a negative value meant activity below it. For communication, the residual could be converted to an index using 100 × exp(residual), where 100 represents the benchmark.

For development shops, each benchmark prediction was produced without using that shop to fit the benchmark. For the test sample, the benchmark was fitted on all development shops and then applied to the test shops. This distinction allowed the test outcome to be constructed without fitting the benchmark on test observations.

Predictor groups

The analysis used 165 approved predictors drawn from the 257 cleaned variables. Identifiers, outcome ingredients, and variables reserved for structural adjustment were excluded from the predictor set. Three very rare binary candidates had also been excluded during predictor preparation. The four feature sets below tested the value of different sources of information.

Table 1. Predictor sets and their purpose

“Intervention” is a grouping label for possible business changes, such as payment services, internet access, and business bank accounts. It does not mean that the study established their causal effects.

Missing values and transformations

Missing responses were treated according to the questionnaire. For example, business-origin questions that did not apply to managers and supplier questions that did not apply to self-purchasing shops received explicit “not applicable” categories. Other missing numeric values were filled with training-fold medians. Missing categorical values received an explicit unknown category.

The three ranked response slots in the profitability question were replaced with 16 rank-weighted scores: a proposed profitability factor received three points if ranked first, two if second, one if third, and zero otherwise. This increased the all-primary set from 165 source variables to 178 engineered columns. These were divided into 168 numeric and 10 categorical columns.

For Ridge, Elastic Net, and support vector regression, numeric columns were standardized using training-fold means and standard deviations. Years open, years in the business, aisle counts, and cooler counts also received a log(1 + value) transformation. Tree models used the numeric values without this scaling or logarithmic transformation. Categorical variables were one-hot encoded for every model, with unfamiliar validation categories handled safely. The fitted all-primary design contained 204 model features.

03Model building and tuning

The analysis was implemented in Python using scikit-learn. A shared pipeline structure combined preprocessing with a regression estimator. The preprocessing rules were consistent within model families, while each fit learned its own medians, scaling values, and category encoding from training observations only.

Why these models were chosen

Ridge regression was the transparent starting point. It keeps all predictors but shrinks coefficients, which is helpful when many survey variables overlap. Elastic Net adds a penalty that can reduce some coefficients to zero, testing whether a smaller set of predictors can perform similarly. [1]

Support vector regression (SVR) with a radial basis function (RBF) kernel tested a smooth nonlinear relationship between shop profiles and the outcome. [2] Random Forest tested patterns built from many decision trees, while Histogram Gradient Boosting built trees sequentially to improve the current fit. These models allowed thresholds and interactions that were not specified in the linear models. [3]

K-nearest neighbors was not included because distances across many mixed survey variables may be difficult to interpret. A single decision tree was expected to be less stable than a forest. Neural networks were not prioritized given the small development sample and the large number of inputs. These were choices about the scope of this study; the excluded methods were not tested and cannot be declared inferior from these results.

Nested cross-validation

Five outer folds provided development predictions for shops excluded from each predictor-model fit. These folds followed the country-by-store-type assignments used in target construction. Each outer training sample contained 825 or 826 shops; its validation sample contained 206 or 207 shops.

Within each outer training sample, four inner folds were used to choose model settings. These inner folds were balanced by country and development SPI quartile. For each setting, the complete preprocessing-and-model pipeline was fitted on the inner training observations and scored on the inner validation observations. The setting with the lowest average mean absolute error was then fitted on the complete outer training sample and used to predict its outer validation fold. This separates parameter tuning from the outer model evaluation. [4]

All algorithms used the same folds and feature-set definitions. Combining the outer predictions gave one out-of-fold (OOF) prediction per shop for each algorithm and predictor set. The main comparison therefore contained 20 algorithm–feature-set combinations and 100 outer model fits, in addition to the many inner tuning fits.

Which settings were tuned?

A hyperparameter is a setting chosen before fitting a model. For example, Ridge’s alpha controls how strongly coefficients are shrunk. Grid search evaluated combinations of these settings. The following table shows the final search ranges used in development.

Table 2. Final development search grids

The Elastic Net and SVR searches were expanded after development checks showed that settings were often selected at a grid boundary. Their results were then updated. Boosting allowed up to 400 iterations, with early stopping using a 15% subset of the data available to each fit. Random seeds were fixed for reproducibility. Model selection used MAE, although Ridge, Elastic Net, and the tree models were fitted with squared-error-based objectives.

How performance was measured

MAE is the average absolute gap between predicted and observed log-SPI; lower values are better. RMSE gives extra weight to large errors. R² measures improvement in squared error relative to predicting the evaluation sample’s mean; it can be negative when a model performs poorly. Spearman correlation measures whether predicted rankings agree with observed rankings.

The main baseline predicted the target mean from the relevant training data for every validation shop. A fixed prediction of zero provided a second reference. Uncertainty in MAE differences was examined by resampling paired shop errors 5,000 times, preserving outer-fold sizes for development comparisons and country-by-store-type group sizes for test comparisons.

04Development results and final selection

Using all primary predictors, every model improved on the development mean baseline of 0.7314 MAE. RBF SVR gave the lowest error in the original comparison, followed closely by Random Forest and Ridge. The difference between SVR and Ridge was only 0.0013 log units.

Table 3. All-primary development OOF performance

Random Forest here uses the original 250-tree search. Arrows show the preferred direction for each metric. Values are pooled across outer validation predictions rather than averaged R² values from individual folds.

Stability and the choice of Ridge

Ridge, Elastic Net, SVR, and Random Forest each achieved positive R² in all five outer folds. Boosting did so in four folds and had the largest fold-to-fold variation in MAE. Random Forest was also refitted using 1,000 trees and five random seeds, retaining the settings selected in each outer fold. MAE ranged from 0.7063 to 0.7085, and predictions across seeds correlated above 0.990. The fixed-seed 1,000-tree forest used in the final comparison achieved MAE 0.7085 and R² 0.0621.

Paired comparisons did not resolve a clear winner among the three finalists. The 95% bootstrap intervals for MAE differences were −0.00936 to 0.00662 for SVR minus Ridge, and −0.00976 to 0.00700 for the stabilized forest minus Ridge. Both included zero. This supports treating their observed differences cautiously; it does not prove that their true performance is identical.

At final selection, a practical tolerance of 0.005 MAE was used to identify near-best models. Ridge was within this tolerance and was selected because it was easier to explain and reproduce. A final development search chose alpha = 3,000, with average validation MAE 0.7086; alpha = 1,000 was close at 0.7091. The chosen Ridge pipeline was then fitted on all 1,032 development shops. Only this model was taken forward to the protected test.

Which predictor sets helped?

Context-only models had R² values close to zero or negative. Practices and capabilities carried more information: every algorithm performed better with intervention-only predictors than with context alone. Adding interventions to context also reduced error across all five algorithms.

Adding context to practices was less helpful. It improved Ridge and Elastic Net slightly but worsened the other three models. Adding perceptions and beliefs then improved every algorithm. For Ridge, MAE fell from 0.7313 with context alone to 0.7174 with context and practices, and to 0.7097 with all primary predictors. This suggests that the survey’s predictive information was spread across practices, capabilities, and beliefs rather than concentrated in background characteristics.

05Held-out test results

The final Ridge model was evaluated on the 259 reserved shops after its predictor set and settings had been fixed. Its MAE was 0.7042, compared with 0.7445 for the development-mean baseline. The absolute reduction was 0.0403 log units, or 5.41%. The paired bootstrap interval for model-minus-baseline MAE was −0.0619 to −0.0182, supporting lower error in this sample.

Table 4. Final evaluation and baseline comparison

A constant prediction has no ranking variation, so its Spearman correlation is undefined. Test values near zero for baseline R² are slightly negative before rounding.

Test R² was 0.0914, indicating that the model accounted for about 9% of the variation in the constructed target. Spearman correlation was 0.3059, showing modest ranking ability. All four main metrics were slightly better on the test sample than in development. This was reassuring for this split, but the test remained part of the same survey: it was an internal held-out validation, not validation in a new country, time period, or independently collected study.

The observed standard deviation of log-SPI was 0.913, compared with 0.202 for predictions. The fitted calibration slope was 1.437 and its intercept was −0.019, suggesting that predictions were too compressed on this test sample. A narrower prediction range can also reflect limited information in the predictors, so matching the full observed spread is not itself a modeling goal.

Average predicted log-SPI was −0.063 in the lowest observed quartile and 0.091 in the highest, compared with observed means of −1.058 and 1.131. The model therefore struggled to represent unusually weak or strong performance. The exponentiated MAE was 2.02, which is the geometric mean of the larger-to-smaller ratio between predicted and observed SPI scores. This is an error summary, not a prediction interval or a percentage error. The 90th-percentile multiplicative error was 4.08.

Country and store-type differences

Table 5. Country-level test performance

Country results varied. Brazil had weak ranking performance in this split, while Mexico had a stronger correlation. Argentina and Peru each contributed only 15 test shops, making their estimates especially uncertain. Country MAE also depends on how widely the target varies within the country, so these values should not be read as a simple ranking of model quality.

By store type, MAE ranged from 0.627 for minimarts to 0.775 for pharmacies. Grocery stores showed a mean underprediction of 0.363 log units, despite the small overall mean error. These subgroup results identify questions for future validation, but they were not used to revise the locked model.

06What the models learned

Predictor importance was assessed on development validation folds by shuffling variables and measuring the change in error. If shuffling increased MAE, the fitted model had been using that information. Related variables were also shuffled together as domains, using 20 repetitions per domain and five repetitions per individual intervention candidate. This gives a measure of predictive contribution, not a treatment effect. Correlated variables can share information and make individual importance look small. [5]

The leading domains were products, services and payments; store and financial capabilities; owner motivation; and supplier evaluations. Their mean MAE increases across finalists were approximately 0.0106, 0.0092, 0.0075, and 0.0053. Individual contributions were much smaller. Utility-payment and cash-withdrawal services ranked highest in the consensus comparison.

Table 6. Eight leading individual candidates in the consensus ranking

¹ Number of outer folds in which Elastic Net retained the source variable. ² Direction of the average standardized Ridge coefficient across development folds. Neither selection frequency nor coefficient direction demonstrates a causal effect.

Elastic Net retained only 20–56 of the 204 processed features across the five folds, illustrating how a sparse model can select different variables from correlated groups. Evidence tiers combined importance rankings and consistency: 9 candidates received the highest tier, 12 the next tier, 13 mixed evidence, and 44 weak or inconsistent evidence. These were descriptive screening rules, not formal significance categories. Elastic Net selection provided additional context rather than determining the tiers.

The difference between importance and direction is visible for grocery vouchers: they had positive permutation importance but a negative Ridge coefficient. This means the variable helped prediction while being associated with lower predicted log-SPI after accounting for other included variables. It does not show that accepting vouchers harms a business.

07Discussion and limitations

The main lesson was that a more complex algorithm did not provide a clear practical advantage. The best models had very similar errors, and the transparent Ridge model remained competitive on the reserved sample. Regularization was useful because the data contained many overlapping predictors relative to the number of shops. The results are consistent with a collection of small signals rather than one dominant predictor.

Several limits affect the interpretation. First, the outcome was estimated from self-reported traffic, opening days, and spending bands. Errors in those inputs and the choice of structural benchmark affect what SPI represents. Predicting this score is not the same as predicting verified profit.

Second, the development target scores were constructed before predictor tuning. The cross-validation procedure did not rebuild the complete target-construction process inside every training split; some development target values can depend indirectly on observations in other folds. The nested results should therefore be understood as evaluation against a previously constructed target, with the reserved test providing the stronger final check. Grid expansions and final model choice also used development results, so the entire study was not one fully prespecified search.

Third, the bootstrap intervals resampled existing errors. They describe uncertainty conditional on the fitted models and constructed scores, without repeating training or estimating uncertainty in the spending assumptions and structural benchmark. The many exploratory predictor comparisons also mean that isolated high ranks should be interpreted cautiously.

Fourth, the test sample came from the same survey and included small country groups. The study did not establish performance for future shops, new populations, or operational decisions. A mean baseline and a zero baseline were evaluated; a training-median baseline, which is a natural reference when optimizing MAE, was not included.

Finally, the data were observational. Higher-performing shops may be more able to afford equipment and services, while local demand may influence both shop practices and performance. Predictive rankings can guide further investigation, but they cannot justify advice to adopt a product, payment method, or business practice.

08Conclusion

This study built and compared five regression approaches using common predictor groups, fold-based preprocessing, and separate tuning and validation steps. The full predictor set performed best for every algorithm. Ridge was selected because its accuracy was close to the best development result and its structure was easier to explain.

The final model reduced test MAE by 5.41% compared with the development-mean baseline and achieved R² of 0.0914. These results support a modest predictive contribution from the survey information. However, large individual errors and difficulty predicting extremes limit practical use. The evidence is best viewed as a basis for further research and cautious exploration of relative rankings; operational usefulness and the effects of changing business practices would require separate studies.

Methodological references

These references support the methods described above. Numerical findings are from the completed shopkeeper study.

1. Scikit-learn developers. Linear models: Ridge and Elastic Net.
2. Scikit-learn developers. Support vector regression.
3. Scikit-learn developers. Ensemble methods: Random Forest and Gradient Boosting.
4. Scikit-learn developers. Nested versus non-nested cross-validation.
5. Scikit-learn developers. Permutation feature importance.