ChartNet Lets a Small Model Beat GPT-4o on Charts

A 2 billion parameter Granite vision model trained on ChartNet scored 70.3% at pulling tables from plots, against GPT-4o’s 46.7% on the same extraction test. MIT and IBM built the set so vision-language models can interpret charts from the code, the numbers, and the caption at once, not from pixels alone.

Jovana Kondic, an MIT EECS graduate student and lead author, framed the bottleneck as examples, not size. Dhiraj Joshi, a senior scientist at IBM Research, pointed at finance desks that live on trend charts and need those workflows to run without a frontier API bill.

A 2B Model Beats GPT-4o on Chart Extraction

The paper, accepted at CVPR 2026, states that the best ChartNet-tuned model beats GPT-4o across the four tasks they score: reconstruction, data extraction, summarization, and question answering with chain-of-thought. The fine print is less tidy. GPT-4o still leads one reconstruction check, code execution, at 95.9 against 90.4 for Granite Vision 3.3 2B after ChartNet training.

On the jobs a desk actually pays for, the 2B model is ahead. Data extraction is 70.3 against 46.7. Summarization is 83.9 against 77.1. Chart QA is 65.0 against 61.1. LLaVA 7B, also tuned on ChartNet, reaches 70.3 on that QA test, from a 55.1 baseline.

We developed ChartNet to be a one-stop shop for chart understanding, covering basically anything that an AI model and a practitioner who is training that model might need. We hope our work motivates researchers to achieve state-of-the-art performance with smaller models that don’t require infinite amounts of computation.

Jovana Kondic, MIT EECS graduate student and lead author

A 72B Qwen2-VL checkpoint, left untuned, sits at 50.3 on extraction and 75.9 on summaries. Mistral Small 24B is at 53.2 and 79.8. Extra parameters did not close the gap that labeled charts closed.

Every Chart Comes With Code, a Table, and a Quiz

Kondic’s group, with IBM Research, the MIT-IBM Watson AI Lab, and Abaka AI, treated a chart as five files that have to agree. The CVPR paper describes 1.5 million chart tuples across 24 chart types and six plotting libraries, including matplotlib, seaborn, and plotly. Each sample carries the same five parts.

THE FIVE FILES BEHIND EACH PLOT

• Plotting code: Executable code that rebuilds the figure, so the model can learn how marks map to values.
• Rendered image: The chart as a reader would see it, from bars and lines to heatmaps, violin plots, and tornado charts.
• Data table: The underlying CSV, which is the target for extraction.
• Summary: A natural-language writeup of trends, comparisons, and visual style.
• QA with reasoning: A hard question plus a think trace and an answer, used for chain-of-thought training.

ChartQA, still the common public test, has 14K images and three types, bar, line, and pie. UniChart has 611K. TinyChart has 680K. PlotQA has 224K. None of those sets ticks every column ChartNet does: real charts, code, tables, text, QA, reasoning traces, human labels, grounding boxes, and safety pairs.

Kondic said a vision-language model, unlike a person, may need thousands of examples before it reliably knows a line chart when it sees one. Prior training sets, she said, mostly taught simple questions. ChartNet is built to cover reconstruction, extraction, captions, and longer reasoning in one place.

They Grow New Charts by Editing Code

The generation trick is to leave image space. A seed chart goes into a vision-language model that writes approximate plotting code. A language model then rewrites that code, changing type, values, topic, and colors, and the pipeline renders the new figure. Kondic said one seed can yield hundreds of variants, which is how the set reached more than a million images.

Quality control is mechanical. The code has to run. The picture has to match the table. “We don’t want to just be generating diverse samples. We also want the information to be presented in a meaningful way,” Kondic said. A human-verified slice sits beside the synthetic bulk so a lab can fine-tune on labels that carry a validity check.

Kondic presents the pipeline in a short CVPR talk, including the seed-to-augmentation loop and the five-way alignment.

The paper also adds a safety slice: 7,000 training pairs and 600 test pairs. Sensitive charts in health, finance, and social topics get adversarial questions on bias and harm, each with a safe and an unsafe answer for preference training. Grounding boxes tag which marks support an answer, so a model can be scored on where it looked, not only on what it said.

How Small Open Models Moved the Numbers

The authors fine-tuned five open models on one ChartNet task at a time, then scored a held-out set of 2,000 synthetic tuples. Reconstruction uses four checks (code execution, two code-quality scores, and image match). Extraction, summarization, and most reconstruction scores are judged by GPT-4o. Chart QA uses RapidFuzz on the final answer. On 100 chart-table pairs, human raters lined up with that judge at Pearson r=0.86 for the best model, with Krippendorff’s alpha=0.81 between two people on GPT-4o’s own tables.

CHARTNET EVAL SCORES VERSUS GPT-4O

The 2B Granite run is the extraction leader in that grid, 23.6 points above GPT-4o. Its extraction gain from the untuned 2B checkpoint is +16.5, from 53.8. Summaries rise +19.9, from 64.0. LLaVA’s extraction jump is the largest in the paper, +41.8 from 17.0 to 58.8. Even SmolVLM at 256 million parameters moves extraction from 22.0 to 36.4. ChartGemma 3B, a chart specialist left untuned, scores 69.5 on QA, just under the tuned LLaVA 7B.

Those figures are in-family: the test is drawn from ChartNet’s own synthetic pool. Transfer is the harder ask. On ChartMimic-v2 code generation, Granite Vision 3.3 2B goes from 30.84 to 58.42 on the v2-direct split after ChartNet. On ChartCap summaries, its BLEU_4 goes from 1.60 to 12.40. The lift survives off the home test, if not at the same scale.

IBM Put ChartNet Into Granite Vision

The dataset is not a free-floating academic dump. IBM already trained product models on it. A March 31, 2026 IBM writeup says Granite 4.0 3B Vision is a LoRA adapter on Granite 4.0 Micro, aimed at tables, charts, and key-value fields, not generic chat. On the human-verified ChartNet test, judged by another language model, it posts an 86.4% Chart2Summary score, the top mark in that comparison. Chart2CSV is 62.1%, a hair behind Qwen3.5-9B at 63.4%.

GRANITE 4.0 3B VISION, IBM’S OWN TESTS

• Chart2Summary: 86.4% on the human-verified ChartNet benchmark, ahead of larger peers in that writeup.
• Chart2CSV: 62.1%, second to Qwen3.5-9B at 63.4% on a model more than twice the size.
• Tables: PubTables-v2 cropped 92.1 TEDS and full-page 79.3, OmniDocBench 64.0, TableVQA 88.1.
• Forms: 85.5% exact match zero-shot on VAREX, 1,777 U.S. government forms.

The weights are Apache 2.0. IBM says the same stack can sit on the Docling document processing pipeline, which crops figures from multi-page PDFs and hands clean chart images to chart2csv and chart2code. That is the small-firm path MIT described, and it does not require a 72B host. The Hugging Face card later lists Granite-Vision-4.1-4B as a second ChartNet-trained model in the same family.

The Download Firms Can Train On Came Later

MIT’s June 3, 2026 announcement sold ChartNet as an open set that could let a tight budget skip a commercial vision API. The files on Hugging Face are more lawyerly than that sentence. The original core and reasoning splits, posted March 31, 2026, sit under a notice that grants no license and limits use to checking the paper. The default that IBM now points trainers at is a later permissive cut.

CHARTNET RELEASE DATES

1. March 28, 2026: The paper goes on arXiv as 2603.27064, citing 1.5 million tuples; v2 follows on April 14.
2. March 31, 2026: Initial Hugging Face drop of core, reasoning, and human_verified, plus Granite 4.0 3B Vision.
3. April 29, 2026: IBM adds a 2.5 million row core_permissive split under CDLA Permissive 2.0 and makes it the default.
4. May 15, 2026: Abaka AI and 2077AI post 30K real-world charts with captions as a companion set.
5. June 3, 2026: Grounded QA lands; MIT publicly describes the project the same day.

The public ChartNet training set now lists a 1.7 million sample core, 4.2 million synthetic charts in total, 94,643 human-verified examples, and a 2,000-chart human-verified test that is disjoint from core and reasoning. Those four counts are not the same pile. The paper’s 1.5 million is the CVPR corpus. The 1.7 million is the public core. The 2.5 million is the CDLA split. The 4.2 million is the later synthetic total. A shop that wants to train, not just reproduce Table 2, needs the permissive default, not the original core.

Dense Time Series Still Trip These Models

A language model still sees a plot as a grid of patches unless someone shows it the code and the table. ChartNet attacks that flattening. It does not turn every Bloomberg screenshot into a clean CSV. The synthetic renderer produces tidy axes, honest legends, and executable source. A scanned 10-K slide, a watermarked terminal export, or a 250-day candle chart with overlapping series is a different object.

WHAT WE KNOW

• Home-test win: Granite Vision 3.3 2B beats GPT-4o on ChartNet extraction, summaries, and QA, and loses only the code-execution slice of reconstruction.
• Some transfer: ChartMimic-v2 and ChartCap both rise after ChartNet training, so the gain is not only in-distribution memorization.
• A runnable model: Granite 4.0 3B Vision is already posted under Apache 2.0 with chart2csv, chart2code, and chart2summary heads.

WHAT IS UNCONFIRMED

• Filing clutter: There is no public score in this paper on noisy, multi-panel pages from live financial PDFs.
• Point-level series: Whether a tuned 2B model recovers every coordinate on a dense market line, rather than the trend, is still an open measurement.
• Judge bias: Most ChartNet metrics use GPT-4o as a judge, which the authors check against people on tables but not on every task.

Joshi’s downstream-workflow claim still holds as a bet, not a deployment report. If the model can name the trend, a lot of research notes get cheaper. If it drops every fifth point on a long series, a trading desk will not retire its rules. The authors say they will add harder charts and take community patches. Until those land, the honest product is a small open reader that is now better than a giant API at clean plots, shipped with a license maze that only cleared in late April.

The 2B checkpoint that beat GPT-4o on extraction is on Hugging Face, and so is the later permissive chart pile it learned from. That is the part a shop with a GPU can use without waiting for the next frontier model.

Frequently Asked Questions

What License Covers ChartNet Training Use?

The default Hugging Face config, core_permissive, is released under the Community Data License Agreement, Permissive, Version 2.0, after IBM’s April 29, 2026 licensing update. The original core and reasoning files remain posted only so labs can reproduce the paper, and the card tells users not to treat those older splits as a stand-in for the default release.

How Large Is the Human-Verified Test Set?

The human_verified test split holds 2,000 charts whose images, summaries, and tables were checked by people, and that split is fully disjoint from the core and reasoning subsets. A separate 2,000-tuple synthetic holdout is what the CVPR tables use for the GPT-4o comparison, so the two “2,000” figures are different tests.

Did ChartNet Include Safety Alignment Data?

Yes. The paper releases 7,000 training samples and 600 test samples of adversarial chart questions paired with safe and unsafe answers for direct preference optimization, covering topics such as discrimination, violence, and political bias. The Hugging Face card still listed the safety subset as coming soon at the June 3, 2026 update, so the counts live in the paper even if that config is not the default download.

Which Granite Models Were Trained on ChartNet?

The dataset card names Granite-4.0-3B-Vision and Granite-Vision-4.1-4B. The CVPR experiments separately fine-tune granite-vision-3.3-2B and granite-docling-258M, plus SmolVLM 256M, Qwen2.5-VL-3B, and LLaVA 7B, so the Granite 4 product line is the shipping family and the 3.3 2B run is the paper’s extraction leader.

How Do You Load ChartNet in the Datasets Library?

Call load_dataset(“ibm-granite/ChartNet”, “core_permissive”) for the CDLA default. The paper’s original files load as configs named core, reasoning, human_verified, and grounded_qa, and a single core parquet shard is about 1.66GB for 10,000 rows, which is the size the card recommends pulling if you do not want the full multi-hundred-gigabyte dump.