AI lab – AI in Action | Episode 03: AI Tokenization

Let’s talk about AI tokenization in this third episode of our AI in Action series.

The world throws a lot of crazy stuff at us these days, right? From self-driving cars to robots taking our jobs, it’s enough to make your head spin. But today, I want to talk about something a little less scary and a whole lot more… well, let’s just say “technical.” It’s called tokenization.

Tokenization is actually pretty interesting, especially if you ever wondered how these fancy machines understand the stuff we type and say and produce things when we give them prompts.

Imagine you’re teaching a kid to read. You wouldn’t just shove a copy of Moby Dick at them, right? No, you start with the alphabet, then syllables, then words. Tokenization is kind of like that for computers. It’s about breaking down that big, scary text into bite-sized pieces computers can actually process. Think of tokens as the building blocks of language. They’re like tiny lego blocks that fit together to form words, sentences, and whole paragraphs. 

And they are obtained by dissecting a sentence. That’s tokenization: breaking down sentences into smaller pieces—words, characters, subwords, you name it. It’s the AI equivalent of cutting up your food so you don’t choke on your metaphors.

With word tokenization, you take a sentence and chop that long string of words up into individual ones, like “the,” “quick,” “brown,” “fox.” This makes it way easier for the computer to identify patterns and, you know, actually understand what we’re trying to say.

But it’s not always about word tokenization. Sometimes, these computers gotta get real granular and break things down into individual letters, especially for languages that don’t have clear word boundaries, like, well, French. That’s called character tokenization. And then there’s even subword tokenization where you break a text into units that might be larger than a single character but smaller than a full word. For instance, “Chatbots” could be tokenized into “Chat” and “bots”.

Now let’s look at this in the context of generative AI. Imagine these AI models as super-powered word wizards. They’re all about predicting the next word, the next phrase, the next sentence, based on what they’ve already seen. And the secret ingredient? You guessed it: tokens.

These AI wizards are trained on a massive collection of content, learning the patterns, the relationships, the whole dance of how these tokens fit together.

Now, the real magic happens when you feed these models a sequence of tokens. They use their wizardry to predict the next token, and the next, and the next. It’s like watching a domino effect of words unfold. 

But you can imagine that there are challenges here. Language is messy. A sentence can mean one thing or its complete opposite depending on how you chop it up. Just think of a sentence like ‘Flying planes can be dangerous’: that can mean totally different things depending on how you slice those tokens. Are we talking about the dangers of flying planes, or about planes flying overhead that are dangerous? It’s enough to make your head spin, right? And let’s not even think about punctuation and the famous “Let’s eat (,) grandma” quote.

Plus, some languages just don’t play nice with spaces between words, like Chinese.

Moreover, from a technical point of view, Large Language Models or LLMs have what is called “a fixed context window”: that means that they can only process a certain amount of information at once. Just imagine the LLM as someone reading a book, but they can only see a few sentences at a time through a small window. Every time they move forward, previous sentences are completely lost from sight, replaced by new ones. That explains why generative AI might sometimes struggle to keep track of the entire conversation or to generate coherent text. 

On top of that, not all tokens in a pre-training dataset are necessarily used by an AI application. These things are picky and not every token makes the cut. Just like in your favorite reality shows, some words are voted off the island early. Cleaning up, curating and filtering! It’s all about tossing out the useless bits that might throw off our AI overlord’s learning curve—too long, too short, too weird.

Now, the survivors, our chosen tokens, get to go to training camp. Here’s where sequences of tokens—think conga lines of vocabulary—shuffle into the AI model that tries to make sense of human nonsense, finding patterns and figuring out how words like to hang out together.

But wait, there’s more. Things like evaluation and fine-tuning. Those are like the AI’s final exams, where some tokens might get a second chance to prove they’re not just fluff. And depending on how they perform, our AI might go back, tweak some dials, and shift its focus. It’s a never-ending cycle of pop quizzes and study sessions.

And what affects which tokens get to graduate? Oh, a whole series of factors. The AI’s architecture—basically, its brain structure—decides a lot. Whether it’s a fan of old-school hits or new wave beats, or if it’s training for something specific like translating languages or spotting spam in your email. Plus, the quality of data matters. Feed it garbage, and well, it will produce garbage out. But we’ll talk more about that in another episode.

Oh, and let’s not forget about hyperparameters, the AI’s personal trainer, setting the pace and intensity of its workout. Too harsh, and you burn out your tokens; too lenient, and they laze around, adding nothing to the AI’s smarts.

But how does the model decide which token to choose next? Well, that’s where things get a little complicated. It’s a balancing act between randomness and determinism. Sometimes, the model takes a wild guess, just to see where it leads. Other times, it sticks to the safer, more predictable path. It’s like a game of chance and strategy, all rolled into one.

So, next time you’re marveling at an AI-generated text, remember it’s all about those tiny tokens, dancing together in a complex symphony of language and prediction.

Alright, that’s all I got for this episode. Stay curious, folks, and maybe next time I’ll explain the blockchain… nah, just joking!

Written by Caroline De Cock, LL.M. , Head of Research.