We’re releasing Jev 1.14.0 along with experimental features for dreamers
typesafe.aiI guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.
Given that benchmarks are useful, why not give them economic value?
OTOH, I trust the benchmarks from Apolinario's HF space that you've linked to.
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
It's also open source, https://github.com/nicobrenner/jeffy it's in the roadmap to build something for the browser. Do you want to take a stab at it and submit a PR?
I have found a couple of problems Opus could fix that Sol couldn't (smoothing a three way join between curved edges and a bezier curve face is one I can remember). But it's much slower than Sol and generally doesn't produce as nice results.
Assuming I'm not just misunderstanding something critical (happens often!) and you haven't explored this already (very possible!), pointing your agent at this link and asking for some prototypes might be illuminating: https://huggingface.co/models?pipeline_tag=image-segmentatio...
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
What I had before was basically:
- 1 classifier question
- if ok, do some heavy computation
- after heavy computation, another classifier
So now what I do is:
- Jev with the 1st and 3rd classifier
- if ok, heavy computation
I can classify many things at once, and it's cheaper and faster than 1 for sure and 1 maybe classification that I had before.
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Still waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
It's essentially, "predict the outcome of a coin flip", "what is the result?", and "heads" and "tails" as the answer options. With the Jev website, they do not show the "this is not a real probability" disclaimer until after you get the answer (misleading imo, should be upfront and more prominent, eg. by the input form like the gemini app)
When I give this to an LLM, it will generally respond with something like 50/50, 100% confidence.
iirc, Jev has a "jaggedness" page that lists known problem areas with the model, and plain mathematics (which I'd argue this is) is one of them.
If they cannot get something so simple right, while at the same time offered as a decision with likelihoods machine, why should I trust their other answers with "probability" they produce?
Here's another post that shows how fragile or inconsistent, even dangerous, these decision models can be: https://zachlandes.com/trolley-problem/
Don't use decision models with important decisions, especially those that involve human lives or irreversible choices
You should never trust the machine. ;) But you can have a level of confidence with it, and you can know where it's strong and weak.
This is a language association machine, not a math and probability machine. It is specifically weak with any real math.
As simple as a coin flip is -- as simple as a child may be able to understand it -- the machine is specifically not designed to produce an accurate mathematical probability.
That's just not how LLMs work. It's not how they're designed. Nor these decision models which use identical principals.
All that aside... agreed regardless. Irreversible choices should never be finalized by such a volatile technology, regardless of how well we benchmark it. If there's even a chance it could be wrong, that is unacceptable.
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
I've had better luck just running a few of my own simple tests.
Very refreshing.
> For the builders AND dreamers
> Enter the danger zone at your own risk.
> Size does matter, apparently
> We’re trying something new out here: we’re going to treat you guys like adults
> Lots of people love to ask what kind of sci-fi we’re releasing next
That said, their docs are great and don't seem to include all the Reddit cringe.
I've yet to see any company match the copy writing skills of Valve though.
Videos will be next.
You could also pay for a ML product, but not so easily and cheap. This enables PoCs and small products.
- Inference for a bunch of open weights architectures
- Diffusion image generation fir Z ai turbo
- Vision input into LLM for inference
- Vulkan/fast CPU, optimized kernels
- Playground for testing
- Last but not least, system one fast inference mode with same API semantics as Jev
Decision models work by using a backbone model like qwen 3.5 9b, along with a special linear head that provides the readout for the forward pass.
- add to your README links to download the relevant .gguf files – I assume from HuggingFace?
- or even implement your own equivalent of `ollama pull`/`list`/`rm`/etc
- add some CI using Github Actions (or an alternative if you prefer)
- as well as `go install github.com/computerex/dlgo/cmd/dlgo@latest`, also document `go install github.com/computerex/dlgo/cmd/dlgo-vision@latest`, `go install github.com/computerex/dlgo/cmd/dlgo-image@latest`, etc (obvious to anyone experienced with Go, not to many inexperienced with it)
actually, I try running `go install github.com/computerex/dlgo/cmd/...@latest` on my Mac and I get a lot of errors; some of them are because `cmd/gpu-watchdog` is trying to use Windows-specific APIs (e.g. `syscall.NewLazyDLL`) even on macOS
It really feels like they're just admitting that they had it disabled because they needed investors to subsidize turning it on.
You can define tasks like sentiment analysis, support triage, or claim support, tweak the questions and criteria, and inspect the answers and probability distributions. You can also import existing recipes, save examples, and share recipes with recorded results.
An OpenRouter API key is required to use Jev.
1. Adding bounding boxes only compounds the biggest issue with Jev, which is that it's using a massive model to do what we've been using purpose-built[1] models to do for years(/decades), only a little worse and a lot more expensive. This is fine --great, even-- if presented as a tradeoff.[2] And a product that enables "smart if statements" that are "sci-fi" does not sound like a tradeoff at all. In that light, ending the post with facetiously-labeled graphs that just show it beating a bunch of random names (including models with twenty-seven BILLION parameters) on unknown metrics feels borderline dishonest.
2. "We love the community's desire to help make models better!" feels like Manufacturing Consent but with way lower stakes. I turn on various experimental features and pick nightly builds and all that like many of us do, but this feels... well, at best it's kinda offputting rhetoric for a practice I otherwise condone. IMHO.
3. Their job postings go beyond groan-inducing to vaguely insulting. Take this "Data Demon" posting[3], which is presumably for a data scientist (data engineer...?) but is branded around mocking the Turing-award-winning scientists trying to warn the world about superintelligence. Yes, I had the same reaction: what? Why?? It's giving a corporate culture where A) executives feel entitled to shoot from the hip and inject their personal crusades on a whim, and where B) scientists/science isn't appreciated (or, worse: understood). Criticizing the concept of measurement isn't helping, nor is their choice to label output percentages as confidence scores.
I guess I hope I'm misunderstanding, cause they're gonna make millions regardless, I'm sure. I just know that I can't think of when I'd reach for Jev 1.*[4] for a prod usecase, and this is an awfully big company for a prototyping shortcut.
[1]: https://huggingface.co/models
[2]: of dev speed for cost and accuracy, which is lowkey another way to say efficiency and efficacy. Two pretty massive things to trade!
[3]: https://jobs.ashbyhq.com/typesafe-ai/d315d436-e517-4f17-82bb...
[4]: Which, btw, what happened to semver? Did we stop doing semver cause python did? Did I miss a memo/blog? Not a rhetorical question, I swear. Deprecating minor versions just seems unholy, much less minor version thirteen!
And i already sub to jev and i think this launch is going to make me a much more loyal user now lol
> We’ve heard you loud and clear on context lengths. Jev actually supports a much longer context length than we’ve enabled in production. We chose to limit it intentionally because Jev, like all models, is less reliable on longer contexts, and this constraint encourages more effective usage patterns in prod.
> We aren’t your parents though, so while we strongly recommend limiting context length, you no longer have to:
> TypeSafe-Danger-Zone-Long-Context: true
Remember the relatioship incentive is skewed, you are giving in your precious data inexchange for random logprobs, that they offer to "cheap".
Eg: Longer context has long been known to reduce response accuracy, so it makes perfect sense why they’d limit it especially at launch. Personal evals are in fact more useful than public benchmarks so it makes sense (and is refreshing) that they’re cautioning against them…
"...there's no way that our team has a full understanding of what is needed to spark delight in the world and build sick sci-fi."
"Size does matter, apparently "
Too bad they seem too busy to answer these questions, IMO that feels like a missed early leverage opportunity for them. We benchmarked a bit self hosted alternative, and nothing reach quite the cost/performance of Jev, yet.
Maybe it's possible via their Digital Ocean offering https://ideas.digitalocean.com/changelog/now-available-jev-f...
If you are sending data over the internet to someone else's computer, who you can't or won't audit, you should keep you expectations realistic.
You can be sure that comapnies will only do ZDR if your data is more expensive to keep than its utility to them, even if they claim something else.
Why does this thing have to be another Web API?
The main benefit of Jev seems to be broader knowledge? But local models (especially the ones actually trained specifically for decisions, rather than ones that "support" it like embeddinggemma) are probably good enough for what most people use it for. They also seem to be quite easy to fine tune (I haven't tried, but seems to be what people say).
I mean, yes, that works at dev time; but going from zero to nonzero local inference components makes an agent implementation suddenly much more complex to ship to production / drastically narrows the set of hosts such an implementation can be deployed onto.
I see box is now a thing but no span?
So render text to image then use box right?