On the Limits of Small Language Models in Vectorized Code Synthesis

“Can small language models reliably synthesize deterministic, high-throughput code without falling into the Python Loop Tax?”

1. Abstract & Motivation

Large frontier models (30B–70B+ parameters) demonstrate remarkable code synthesis abilities, but their deployment in production data systems is constrained by high inference latency, prohibitive cloud API costs, and privacy requirements.

In domain-specific tasks such as synthetic data compilation, the target code space has strict, well-defined mathematical properties. We explore whether compact models (0.5B $\rightarrow$ 7B parameters) can be specialized to serve as deterministic, low-latency code compilers capable of running locally on consumer hardware.

2. Theoretical Formulation: Strict AST Invariants

To rigorously test small model capabilities, we establish a deterministic static analysis boundary. Generated Python code must satisfy three formal Abstract Syntax Tree (AST) invariants to be considered valid:

graph TD
    A[Natural Language Specification] --> B[Gnoril Synthesizer Model: 0.5B - 7B]
    B --> C[Generated Python Code]
    C --> D{Static AST Validator}
    D -- "Contains for / while / .apply()" --> E[❌ Static AST Rejection]
    D -- "100% Vectorized Array Math" --> F[Sandboxed Execution Worker]
    F -- "Runtime Exception" --> G[1-Pass Surgical Self-Correction]
    G --> B
    F -- "Successful DataFrame Output" --> H[✅ Verified Execution]

Formal Invariants

1. Loop Elimination: All dynamic row-iteration nodes ($\text{ast.For}$, $\text{ast.While}$, $\text{ast.ListComp}$, $\text{ast.DictComp}$, $\text{ast.GeneratorExp}$) are forbidden. Only static schema meta-iteration over column names is permitted.
2. Pandas Method Ban: Eliminates slow Pandas iterative methods (.apply(), .iterrows(), .itertuples(), .applymap()).
3. Tensor Primitives: All categorical, numerical, and conditional logic must be expressed through vectorized array operations (np.random.choice, np.where, np.select, pd.to_datetime).

3. Empirical Evaluation: The “Syntax vs. Semantics” Execution Gap

Evaluating our models on the Gnoril-Bench v1.0 test suite (15 diverse single-table and relational multi-table generation tasks) revealed a fundamental dichotomy in small model code synthesis:

flowchart TD
    subgraph Syntax ["1. Structural Syntax Layer (Static AST)"]
        A["AST Vectorization Compliance (V-Score: 60.0% - 73.3%)"]
        A --> A1["✅ SFT easily eliminates loops, while-blocks, and .apply()"]
    end

    subgraph Semantics ["2. Runtime Execution Layer (Dynamic Sandbox)"]
        B["Sandbox Execution Yield (E-Score: 0.0% - 13.3%)"]
        B --> B1["❌ Severe bottleneck: Array broadcasting, string types, & probability sums"]
    end

    Syntax -.->|"Exposes the Execution Gap"| Semantics
    style Syntax fill:#eff6ff,stroke:#3b82f6,stroke-width:2px
    style Semantics fill:#fef2f2,stroke:#ef4444,stroke-width:2px

Benchmark Summary

4. Taxonomy of Execution Failure Modes

Analyzing the 60 total generation logs captured by the sandbox worker revealed five distinct failure categories:

graph LR
    A[Execution Failure Modes] --> B[1. Floating-Point Entropy]
    A --> C[2. Matrix Shape Broadcasting]
    A --> D[3. Implicit Type Casting]
    A --> E[4. Cross-Table Key Drift]
    A --> F[5. Index Boundary Errors]

    B --> B1["ValueError: probabilities do not sum to 1"]
    C --> C1["ValueError: operands could not be broadcast (N,) vs (N,1)"]
    D --> D1["TypeError: np.char.add() requires string arrays"]
    E --> E1["KeyError: 'user_id' vs 'account_id' mismatch"]
    F --> F1["IndexError: index N is out of bounds"]

Failure Dynamics Breakdown:

1. Floating-Point Entropy (36.7%): Small models cannot perform exact mental floating-point summation to $1.0$ (e.g. [0.5, 0.3, 0.3] sums to 1.1), triggering strict NumPy validation exceptions.
2. Return Type Contract Violations (23.3%): Returning raw numpy.ndarray arrays instead of wrapping dictionaries inside pd.DataFrame(...).
3. Matrix Shape Broadcasting (18.3%): Attempting element-wise operations between 1D vectors of shape (1000,) and 2D column matrices (1000, 1).
4. Relational Key Inconsistency (13.3%): Cross-table foreign key inconsistency (naming the parent primary key user_id and the child foreign key account_id).
5. Implicit Type Casting (8.3%): Calling string concatenation functions (np.char.add) on integer arrays without explicit .astype(str) conversion.

5. Closing the Execution Gap: Direct Preference Optimization (DPO)

To bridge the execution yield gap without increasing parameter count, we propose Execution-Feedback DPO. The execution sandbox automatically generates preference pairs without human annotation:

sequenceDiagram
    autonumber
    participant Model as Small Model (0.5B - 3B)
    participant Sandbox as Execution Sandbox
    participant DPO as DPOTrainer

    Model->>Sandbox: Generates Vectorized Code (e.g. unnormalized probabilities)
    Sandbox-->>Sandbox: Runs code in isolated Python worker
    alt Execution Fails
        Sandbox->>DPO: Record faulty code as REJECTED sample
        Sandbox->>Model: Inject surgical line-level repair prompt
        Model->>DPO: Record verified working code as CHOSEN sample
    else Execution Passes
        Sandbox->>DPO: Record as positive ground-truth sample
    end
    DPO->>Model: Optimize weights directly against runtime execution errors

6. Key Findings & Contributions

1. Structural Constraints Are Parameter-Efficient: Models as small as 0.5B can reliably acquire strict AST rules and eliminate row-by-row loops with fewer than 1,500 training examples.
2. The Execution Gap Requires Targeted Alignment: Syntactic compliance ($V\text{-Score}$) does not guarantee numerical execution success ($E\text{-Score}$). Array dimensions and floating-point normalization represent the core failure boundary for sub-3B models.
3. Automated Feedback Over Scale: Combining compact models with deterministic compiler sandboxes and Execution-Feedback DPO provides a viable architectural blueprint for fast, local code synthesis.

Citation

@article{ahmed2026limits,
  title   = {On the Limits of Small Language Models in Vectorized Code Synthesis},
  author  = {Ahmed},
  journal = {Research Notes},
  year    = {2026},
  url     = {https://thlurte.github.io/research/2026-08-15-limits-of-small-models-in-vectorized-code-synthesis/}
}