Show HN: Parsimony – an open-source benchmark for how much code AI models add
parsimony.lasu.devMethod: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtracted, so it measures net units.
GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on 3–10 tasks on my own subscriptions (shown as faded circles, not ranked).
Any feedback or model contributions are welcome, especially public patch exports for newer models.
GitHub: https://github.com/La5u/Parsimony
Not affiliated with any of the benchmarks.
On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.
I wonder if the two will diverge enough that this becomes the tab-vs-spaces/vim-vs-emacs of the LLM era and they'll manage to keep users moving over at some point.
Astra codes like it has to pay by the keystroke (although in a way, it indeed does)
One approach, skill's etc for good clean readable understandable code.
i bet there's a better one involving something with metrics and scoring etc
Anything that can increase code quality would help.
For now that seems to be Qwen 3.8 27B that requires ~28GB of RAM (Q4) which is pretty doable for anyone with a beefy GPU or slightly-above-average mac.
I want to know how close open weights (open models) are to all these leading commercial models. Yes, yes, local models aren't as good, but that distance is closing as local models are already better than what we had just over a year ago in many cases.
Same with Text-to-Image generation (Flux 2, Z-Image-Turbo, etc..) or text-to-speech (Parakeet, Whisper, etc..).