BERT on steroids, or a new chapter in AI engineering

Just the other day, a mentee and I were discussing whether it makes sense to use generative models for classification. Think SPAM/NOT_SPAM email filtering or classifying tickets in Jira. My take: quite often it does, especially when your set of classes is not fixed and every new criterion or category would otherwise mean retraining BERT.

To simplify: BERT was trained to fill [MASKED] positions in text with what fits best, using context from both sides:

For example:

I like [MASKED] coffee

GPT works autoregressively. That’s a fancy way of saying that the probability of the next token depends on all the tokens generated BEFORE it:

I like drinking → ?

Congratulations, you have completed the young ML engineer’s course. Now you know the two halves of the Transformer architecture.

BERT is an encoder: it turns input into a vector, or an embedding if you like. A small neural network is trained on top of those text representations. It uses the embeddings to classify the text fed into BERT according to the categories you chose when training that little network. That’s where the classification ability comes from: BERT + a mini neural network.

You can probably guess from the example above that BERT can predict all masked positions in parallel. That’s waaaay faster than generating tokens one by one and classifying based on the resulting reasoning.

So why do we use ChatGPT rather than ChatBERT? A few reasons:

1. Generative models do not need fine-tuning for every task; in-context learning handles that for us.
2. GPT is easier to train on internet text.
3. The GPT-only architecture scaled better.
4. …

If you’ve made it this far, congratulations: you’ve been warmed up to learn what JEV is. Jev is a new class of models from an obscure company called Typesafe. Its founder came up with InstructGPT, a concept I explained here.

We do not know JEV’s real architecture. But, to simplify aggressively, imagine BERT + a mini neural network that learned to understand class descriptions in a prompt, so you no longer have to retrain it every time something changes.

It even returns a real confidence_score. No more asking GPT to “return a number from 1 to 10 for how confident you are in your answer,” if anyone is still doing that 😅

JEV does not hallucinate; the way it works makes that impossible. Like BERT, it takes a JSON schema and fills the empty [MASKED] positions in parallel. It does not have to rewrite your entire schema from scratch to return an answer, which is why it is two orders of magnitude faster.

Let’s compare Jev’s pricing with models we used for similar tasks before:

As you can see, there is not much to debate here. BUT that does not mean JEV will rule everything. We were using generative models where something like JEV was clearly a better fit. Naturally, we will move to Jev once it reaches a stable release and we confirm that its classification abilities are as good as those of generative models.

This really is a big step in the right direction. I expect to see hybrid models: a Jev-like classifier + a GPT-like reasoner. JEV is not designed for long, sequential reasoning. TypeSafe explicitly recommends breaking complex decisions into atomic questions and assembling the subsequent logic in code.

Also, if you use it to decide which tool to call, it will not tell you which arguments to pass. You’ll still have to ask a generative model for those.

I already have access to Jev; you can apply here. Here is the first example I tried:

r = TypeSafeClient().system_one(
    state={
        "state": "Я не хочу верить результатам электорального процесса, я устал, босс, мне очень больно!"
    },
    questions={
        "urgency": Noul(
            instructions="Насколько срочно человеку нужна помощь?"
        ),
        "mood": Choice(
            instructions="В каком настроении пребывает человек?",
            criteria={
                "sadness": "чувство утраты, скорби или разочарования",
                "desperate": "чувство безнадёжности",
                "neutral": "нейтрально, ни рыба ни мясо",
            },
        ),
    },
)

print(r.nouls["urgency"].noul)
print(r.choices["mood"].choice)

In short, the hype is deserved. Finally, something is happening; I was running out of things to do… What I’m looking forward to most is open models of this kind from China. I’m very curious what’s inside: is there any BERT in there at all, or is it something entirely new?

The entire English-speaking internet is talking about JEV, and for good reason. If you’re into AI and this is the first JEV explainer you’ve read, forward it to your AI chats and appreciate the timing: everyone else’s JEV post will arrive later; my Bugshop has it already.