Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbations

Overview

A model’s architecture tells us how to explore its weights.
Use that structure to build stronger ensembles from fewer candidates.

RandOpt adds random Gaussian noise to pretrained weights, selects the best K candidates on a task, and combines their answers by voting.

Modular Norm RandOpt shapes these perturbations using each module’s natural norm, its place in the architecture, and calibrated sensitivity. The selection and voting steps stay the same.

Repeat independently for i = 1, …, 9

One independent noise draw per candidate. Schematic example: N = 9, K = 3.

Scale each tensor in its own geometry.

RandOpt uses one noise scale across tensors. Modular Norm RandOpt gives each tensor its own size, measured in its own norm.

Modular Norm RandOpt

Inputs $\theta,\ D_{\mathrm{sel}},\ N,\ K,\ R$

Fix before search $s_p=\frac{m_{\mathcal M}}{m_p}\,\rho_p$
1
$\text{for }i=1,\ldots,N$
2
$Z_{i,p}\sim\mathcal N(0,I)$
3
$n_{i,p}\gets\lVert Z_{i,p}\rVert_{M_p}$
4
$\Delta\theta_{i,p}\gets\frac{R}{s_p}\,\frac{Z_{i,p}}{n_{i,p}}$
5
$\theta'_i\gets\theta+\Delta\theta_i$
6
$\mathrm{score}_i\gets\operatorname{Evaluate}(\theta'_i,D_{\mathrm{sel}})$
7
$\text{end for}$
8
$I_{\mathrm{top}}\gets\operatorname{TopK}(\{\mathrm{score}_i\},K)$
9
$\text{for test input }x$
10
$A(x)\gets\{\operatorname{Generate}(\theta'_i,x)\}_{i\in I_{\mathrm{top}}}$
11
$\hat y(x)\gets\operatorname{PluralityVote}(A(x))$
12
$\text{end for}$
13
$\text{return }\hat y$

Where the tensor scales come from.

Computed once, then shared by every candidate.

1 / 5 · Start with the model

Mass is an architectural weight, independent of parameter count. Only active tensors contribute to the total.

2 / 5 · Allocate to module groups

Use the fixed group weights in Table A2. Longer colored marks mean more mass.

3 / 5 · Split across repeated layers

Attention and MLP each share their mass equally across 28 layers in Qwen2.5-1.5B. Expand one layer.

4 / 5 · Divide among tensors

QKV counts as three shares; gate–up as two. Within each logical module, split its share equally over its physical tensors.

5 / 5 · Apply sensitivity correction

$\rho_p$ is set once by a Jacobian-based calibration on 64 Countdown prompts and bounded to $[\tfrac12,\,2]$ . See Section 3.2 and Appendix A.

Worked example · Q, layer 12

Tensor mass

m_p=\dfrac{M_g}{L_g}\times\dfrac{w_u}{\sum_v w_v}\times\dfrac1{n_u}

m_{\mathrm Q}=\dfrac{1/2}{28}\times\dfrac14\times1=\dfrac1{224}

s_{\mathrm Q}=\dfrac{m_{\mathcal M}}{m_{\mathrm Q}}\,\rho_{\mathrm Q}

=3.2\times224\times\rho_{\mathrm Q}

=716.8\,\rho_{\mathrm Q}

All six groups: $m_{\mathcal M}=1+\tfrac12+\tfrac12+1+\tfrac1{10}+\tfrac1{10}=3.2$

01Start with the model
Mass is an architectural weight, independent of parameter count. Only active tensors contribute to the total.
02Allocate to module groups
Use the fixed group weights in Table A2. Longer colored marks mean more mass.
Embedding1
Attention½
MLP½
Head1
Norm1/10
Other1/10
03Split across repeated layers
Attention and MLP each share their mass equally across 28 layers in Qwen2.5-1.5B. Expand one layer.
Attention · 28 equal shares
Layer 12 highlighted · mass 1/56
MLP · 28 equal shares
Layer 12 highlighted · mass 1/56
04Divide among tensors
QKV counts as three shares; gate–up as two. Within each logical module, split its share equally over its physical tensors.
Attention · QKV : O = 3 : 1
QKVO
MLP · gate/up : down = 2 : 1
gateupdown
05Apply sensitivity correction
$\rho_p$ is set once by a Jacobian-based calibration on 64 Countdown prompts and bounded to $[\tfrac12,\,2]$ . See Section 3.2 and Appendix A.

Worked example · Q, layer 12

Tensor mass
$m_p=\dfrac{M_g}{L_g}\times\dfrac{w_u}{\sum_v w_v}\times\dfrac1{n_u}$
$m_{\mathrm Q}=\dfrac{1/2}{28}\times\dfrac14\times1=\dfrac1{224}$

$s_{\mathrm Q}=\dfrac{m_{\mathcal M}}{m_{\mathrm Q}}\,\rho_{\mathrm Q}$ $=3.2\times224\times\rho_{\mathrm Q}$ $=716.8\,\rho_{\mathrm Q}$
All six groups: $m_{\mathcal M}=1+\tfrac12+\tfrac12+1+\tfrac1{10}+\tfrac1{10}=3.2$

Independent noise, every time.

Draw fresh Gaussian noise for each candidate and each tensor. Every candidate starts from the same pretrained weights.

Z_1,p

Z_2,p

Z_3,p

A fresh draw for each candidate and each tensor.

Measure what the tensor does.

Match the norm to the tensor’s role: the strongest linear direction, the largest embedding row, or the largest coordinate.

Linear mapsSpectral normThe strongest amplification direction.

\lVert Z\rVert_2=\sigma_{\max}(Z)

=\max_{\lVert v\rVert_2=1}\lVert Zv\rVert_2

EmbeddingsMaximum-row ℓ₂The largest Euclidean norm of any row.

\max_r\lVert Z_{r,:}\rVert_2

=\max_r\sqrt{\textstyle\sum_c Z_{rc}^{\,2}}

1-D parametersℓ∞ normThe largest absolute coordinate.

\lVert z\rVert_\infty=\max_j|z_j|

Normalize the direction. Set its size.

Normalize to unit natural norm, then set the size to R/sₚ. The direction is preserved; the scale follows the architecture.

NoiseRaw noise

NormalizeUnit norm

ScaleSize R/sₚ

The direction stays the same; its natural-norm size changes.

RandOpt

\Delta\theta_{i,p}=\sigma Z_{i,p}

One coefficient across tensors.

Keep the same selection and voting.

Evaluate N candidates, keep the top K, and combine their answers by plurality voting. Only the perturbation geometry changes.

The same selection and voting procedure as RandOpt.

Stronger ensembles. From fewer candidates.

Qwen2.5-1.5B-Instruct

Mean ± sample SD over 3 seeds. Each N uses a nested prefix of the same 300 candidates per run.
Candidate counts are equally spaced on the x-axis. Highlighted comparisons are differences in mean accuracy.

Performance across seven tasks.

N = 100 · K = 25 · Mean over 3 seeds · Scores (%)

Comparison with iterative baselines

Qwen2.5-1.5B-Instruct

Countdown

1,500 test examples

GSM8K

1,319 test examples

Total model–prompt evaluations

One evaluation is one model generating and scoring one prompt. Totals include search, checkpoint selection, and final evaluation (Appendix D.2).
Points show mean accuracy over 3 seeds. Iterative baselines use K=1.

Modular Norm RandOpt

Overview

Modular Norm RandOpt

Scale each tensor in its own geometry.

01Allocate

Where the tensor scales come from.

01Start with the model

02Allocate to module groups

03Split across repeated layers

04Divide among tensors

05Apply sensitivity correction

02Sample

Independent noise, every time.

03Measure

Measure what the tensor does.

04Scale

Normalize the direction. Set its size.

05Unchanged

Keep the same selection and voting.

Stronger ensembles. From fewer candidates.

Performance across seven tasks.

Comparison with iterative baselines

Countdown

GSM8K