Hybrid model pins X-ray landmarks under 1.5 mm for surgery plans

Res-SwinFusion, a parallel ResNet and Swin Transformer network from West China Hospital researchers, posted mean radial errors of 1.04 mm and 1.37 mm on two public cephalogram test sets, 0.63 mm on a public hand X-ray set, and 1.44 mm on an internal pelvic X-ray set. The work, published online 17 August 2026 in BIO Integration, targets automatic landmark placement that still fails when anatomy is complex or abnormal.

The practical gain sits one step past the error numbers. Once localization stays inside roughly 1-1.5 mm even on post-operative pelvis films, the harder clinical questions become how the model plugs into existing imaging systems and whether the same numbers hold across hospitals.

What the dual-branch network actually does

Pure convolutional networks lose global spatial context as they downsample, which can erase the fine cues needed to tell similar bony landmarks apart. Pure Transformers capture long-range dependencies but can blur local texture. Res-SwinFusion runs both in parallel: an ImageNet-pretrained ResNet branch keeps hierarchical local structure while a Swin Transformer branch models long-range relations through shifted-window attention.

Two custom pieces join the streams. The feature interactive aggregation (FIA) module enhances each branch before fusion so positional cues from the Transformer and structural detail from the CNN reinforce each other. The discrimination feature guidance (DFG) module feeds pixel-level cues across Swin blocks to reduce confusion among look-alike structures. Heatmap regression with Gaussian peaks then yields the final coordinates. The authors also examined how different Gaussian settings affect training convergence.

The design is end-to-end. Earlier two-stage pipelines first propose rough regions then refine each landmark on cropped patches; that multiplies training and inference steps. Res-SwinFusion aims for a single forward pass on the full resized radiograph.

Errors across three very different X-ray tasks

Performance was measured by mean radial error (MRE) in millimetres and success detection rate (SDR) inside fixed distance thresholds. The table below summarises the headline figures reported for Res-SwinFusion.

On the cephalogram Test1 and Test2 sets the team reported SDR gains of roughly 1.2 and 1.9 percentage points inside the 2.0 mm threshold relative to a strong prior heatmap method by Thaler and colleagues. On the hand set the MRE sits within a few hundredths of a millimetre of that same prior work while SDR edges higher. The internal pelvic number matters most for surgical relevance: these are coronal films after total hip arthroplasty, many complicated by osteoarthritis, fracture or femoral-head necrosis.

• Cephalograms: 400 public images (ISBI 2015 Grand Challenge), 0.1 mm pixel spacing, original 1935 × 2400, resized to 768 × 768 with padding; ground truth is the mean of two expert annotations.
• Hand radiographs: 910 images, 37 landmarks on joints and fingertips, wrist width normalised to 50 mm, resized to 512 × 512; 610 / 300 train-test split.
• Pelvic films: 326 de-identified postoperative images from West China Hospital (ethics No. 2023-1975), average ~4000 × 3200 at 0.1 mm, 10 landmarks labelled by a radiologist and an orthopaedic surgeon; first 200 for training.

Ablation runs showed that removing the Transformer global path, the FIA fusion, or the DFG guidance each raised error, confirming every piece contributed.

Why similar bones still fool older networks

Landmarks are small, often share local appearance (similar curvature, density, neighbouring soft tissue), and sit in fixed spatial relationships. CNN downsampling can discard the very pixels that distinguish one from another. When disease or surgery deforms the scene, the problem worsens. The authors list three concrete design answers:

• Dual encoder that keeps both local residual features and long-range Swin attention.
• DFG pathway that injects discriminative spatial signals into the Transformer stages so ambiguous structures separate more cleanly.
• FIA module that performs branch-specific enhancement before interactive fusion, rather than simple concatenation or weighted sum.

Training used Smooth L1 loss on the heatmaps, Adam at 1e-4 (decayed every five epochs), batch size 2, 30 epochs on Tesla V100 GPUs, no heavy augmentation beyond resize. The full training run took about 45 minutes in their setup. Code, configs and checkpoints sit on GitHub under an open licence so other groups can retrain on local data.

Who feels the difference first

Orthodontists already treat automated cephalometric tracing as a consistency tool rather than a replacement. One practising orthodontist put the attitude plainly on X: AI-assisted cephalometry is “a tool, not an autonomous dentist. The clinician still validates everything.” That framing matches the paper’s own language of assistance for diagnosis and surgical planning.

Orthopaedic teams planning or reviewing total hip work gain a second benefit. Manual placement of even ten pelvic landmarks is slow and varies between readers. An automatic first pass that stays near 1.4 mm on real post-operative films can cut that labour and flag outliers for human review. The same logic extends to registration, parameter measurement and abnormality quantification once the points are reliable.

The internal pelvic set already contains the hard cases (deformity, metal, necrosis) that break purely local models. That is where the second-order effect appears: once error is low enough on the difficult distribution, hospitals can start measuring end-to-end time saved and inter-observer reduction instead of arguing about millimetres.

How earlier pipelines hit their ceiling

Coordinate regression maps an entire image straight to (x, y) pairs and struggles with visual ambiguity. Heatmap methods improved matters by turning locations into spatial probability peaks, yet most stayed CNN-only. Two-stage global-then-refine systems improved precision at the cost of separate models per landmark and extra training loops. Hybrid CNN-Transformer work (TransFuse, Swin-UNet and relatives) proved complementary features help segmentation; landmark work needs the extra discrimination steps because the targets look alike and occupy tiny regions.

Res-SwinFusion sits in that lineage but adds the FIA and DFG pieces expressly for spatial disambiguation. The authors note that simply bolting a Transformer onto a CNN is not enough when the clinical failure mode is “which of these two similar ridges is the true landmark?”

Open weights change the next six months

The full open-access paper in BIO Integration includes architecture diagrams, full result tables and the ablation breakdowns. The corresponding public Res-SwinFusion training and test code already carries data-prep scripts for the head and hand sets plus a released checkpoint. The article itself is distributed under Creative Commons Attribution 4.0 terms, so reuse and commercial exploration are straightforward provided attribution is kept.

What remains open is multi-centre testing, prospective operating-room timing studies, and integration work with PACS and surgical planning workstations. The authors report parameters, FLOPs, memory and latency in a supplementary table to argue clinical feasibility, yet those figures are not reproduced in the main abstract. Further validation is required before any hospital treats the outputs as primary measurements.

For now the numbers stand: sub-millimetre to low-millimetre errors on three anatomies, including the deformed pelvises that matter for hip surgery, produced by a single hybrid network whose code anyone can run. The accuracy floor has moved; the integration work can start.