Introduction

Testing a SyntaxNet model trained on Spanish Universal Dependencies requires more than running one parse command. You need consistent preprocessing, deterministic evaluation data, and metric checks such as UAS and LAS. A repeatable evaluation pipeline helps you compare model versions and avoid false confidence from ad hoc manual tests.

Prepare a Clean Evaluation Split

Use a held-out Spanish UD file that was not part of training. Keep tokenization and sentence boundaries identical to your training pipeline. In parser evaluation, tiny preprocessing differences can shift scores significantly.

A practical folder layout keeps artifacts organized:

• data/es-ud-test.conllu
• models/spanish/
• predictions/es-test.pred.conllu

Before running inference, confirm file encoding is UTF-8 and lines are normalized.

file -I data/es-ud-test.conllu
wc -l data/es-ud-test.conllu

Run SyntaxNet Inference on Test Data

Exact binary names vary by build, but the flow is stable. Load the trained model, parse the test set, and write predictions in CoNLL-U compatible format.

# Example command shape. Adjust paths and binary names for your build.
syntaxnet_parser_eval   --model_path=models/spanish/parser-params   --input=data/es-ud-test.conllu   --output=predictions/es-test.pred.conllu

If your environment uses Docker, put this in a script to ensure consistent paths and runtime options.

1#!/usr/bin/env bash
2set -euo pipefail
3
4docker run --rm   -v "$PWD":/workspace   syntaxnet-image   bash -lc "cd /workspace && syntaxnet_parser_eval --model_path=models/spanish/parser-params --input=data/es-ud-test.conllu --output=predictions/es-test.pred.conllu"

Automating execution avoids drift between local runs and CI runs.

Compute UAS and LAS with a Python Evaluator

After inference, compute metrics from gold and predicted CoNLL-U files. The script below is lightweight and runnable with plain Python.

1from pathlib import Path
2
3def read_arcs(path):
4    arcs = []
5    for line in Path(path).read_text(encoding='utf-8').splitlines():
6        if not line or line.startswith('#'):
7            continue
8        cols = line.split('	')
9        if '-' in cols[0] or '.' in cols[0]:
10            continue
11        head = cols[6]
12        deprel = cols[7]
13        arcs.append((head, deprel))
14    return arcs
15
16def score(gold_path, pred_path):
17    gold = read_arcs(gold_path)
18    pred = read_arcs(pred_path)
19    if len(gold) != len(pred):
20        raise ValueError('Gold and prediction token counts differ')
21
22    uas_hits = 0
23    las_hits = 0
24    for (g_head, g_rel), (p_head, p_rel) in zip(gold, pred):
25        if g_head == p_head:
26            uas_hits += 1
27            if g_rel == p_rel:
28                las_hits += 1
29
30    total = len(gold)
31    uas = uas_hits / total
32    las = las_hits / total
33    return uas, las
34
35if __name__ == '__main__':
36    uas, las = score('data/es-ud-test.conllu', 'predictions/es-test.pred.conllu')
37    print(f'UAS: {uas:.4f}')
38    print(f'LAS: {las:.4f}')

This makes model comparison straightforward and scriptable.

Add Regression Checks in CI

Store a baseline metric file and fail CI if scores drop beyond an agreed threshold. This protects against accidental model regressions caused by preprocessing changes or wrong checkpoints.

python eval_parser.py > metrics/latest.txt
cat metrics/latest.txt

Then compare against baseline with a small threshold script. Keep the threshold realistic to avoid flaky failures.

Compare Model Versions with the Same Harness

Model testing becomes more useful when results from two checkpoints are compared with the same script and dataset. Keep one evaluation harness and pass model path as a parameter so the process remains identical.

1#!/usr/bin/env bash
2set -euo pipefail
3
4MODEL_PATH="$1"
5OUT_FILE="$2"
6
7syntaxnet_parser_eval   --model_path="$MODEL_PATH"   --input=data/es-ud-test.conllu   --output="$OUT_FILE"

Then run this script for each candidate model and compare UAS and LAS outputs. Small consistent improvements are usually more reliable than one-off large jumps caused by preprocessing mistakes.

Qualitative Error Review

Metrics are necessary but not sufficient. Review a sample of bad parses and classify error types such as attachment errors, relation confusion, or punctuation handling. This helps decide whether to improve data quality, model architecture, or feature preprocessing.

1from pathlib import Path
2
3def first_mismatch(gold_path, pred_path, limit=20):
4    gold = Path(gold_path).read_text(encoding='utf-8').splitlines()
5    pred = Path(pred_path).read_text(encoding='utf-8').splitlines()
6    shown = 0
7    for g, p in zip(gold, pred):
8        if not g or g.startswith('#'):
9            continue
10        gc = g.split('	')
11        pc = p.split('	')
12        if len(gc) < 8 or len(pc) < 8:
13            continue
14        if gc[6] != pc[6] or gc[7] != pc[7]:
15            print('gold:', g)
16            print('pred:', p)
17            print('---')
18            shown += 1
19            if shown >= limit:
20                break
21
22first_mismatch('data/es-ud-test.conllu', 'predictions/es-test.pred.conllu')

This targeted review makes improvement work practical and focused.

Common Pitfalls

• Evaluating on training data and overestimating real quality.
• Using mismatched tokenization between training and testing.
• Comparing models without fixed evaluation scripts and paths.
• Ignoring token count mismatches between gold and prediction files.
• Tracking only one metric and missing dependency-label regressions.

Summary

• Use a clean held-out Spanish UD test split.
• Run inference with consistent scripts and paths.
• Compute UAS and LAS from CoNLL-U outputs.
• Add metric regression checks in CI.
• Keep preprocessing and evaluation reproducible across environments.