threat-research llm-security

SafetyD: the why, the how, and the what

Manuel Gauto 7 min read

Share

Get our latest research

Threat research and engineering notes, monthly.

We are excited to announce the launch of SafetyD! It is a content-safety classifier designed to score prompts to help protect against malicious or dangerous uses of LLMs. It is designed to be fast enough to run on the inference hot path, within your boundary, and with no hard GPU requirement.

The why

The world of token security has gotten more interesting the deeper I dive into it.

We started with stolen API keys and the relays and resellers that move them. We also wanted to know what all of those stolen tokens were being spent on. We found a lot of what we already expected: model distillation, financial motivations, spam campaign generation, etc. What we did not expect is how dark some of the prompts were. We saw prompts attempting to gather information on crafting cyber attacks, weapons manufacture, and even child exploitation.

Naturally, the upstream frontier labs and inference providers have set up Acceptable Use Policies to govern what types of activities are allowed with their models and systems. As a result, some of our customers were getting penalized for violations of these policies as a result of the prompts that were coming from compromised customer accounts, free trial abusers, and other malicious actors.

Our customers asked if there was anything we could do to help and so…. SafetyD was born.

Accuracy is not everything

There are already tools to help with this problem: moderation endpoints, open source moderation models, banned word lists, etc. So what are we doing differently?

Well, there is a balancing act to be done between accuracy and speed. Banned word lists are fast but inaccurate. LLM-based or API-based moderation systems can add a significant amount of latency to the inference hot path (50ms to >1000ms). Now suddenly each inference call not only needs to wait for the underlying model to respond but actually has to wait for an entirely different LLM to respond before it can even pass the request to the fulfillment model.

So we set our sights on a solution that was:

The how

Fundamentally, we combined a few approaches to create SafetyD. Its current architecture is a single-binary classifier that contains: word lists, rules and distilled model weights. No transformer runs at inference time.

I wish I could say that I immediately came up with that architecture but it took a few wrong turns to get there.

Not one answer, but two!

First, some uncomfortable math. About 0.5% of the production traffic we measured is harmful. At that base rate, a blocker that is wrong 1% of the time makes about ten bad blocks for every good one, even if it catches a fifth of the harm. Suboptimal.

So SafetyD gives two answers, each with its own error budget. Tags say what a prompt is about. They are liberal, they feed a review queue or an account risk score, and they reject nothing. Block is the auto-reject. It fires only on unambiguous harm, because a wrong block turns away a real customer with nobody in the loop.

First attempt: word lists

So, how did we start? Simplest first, word lists.

To be fair, they are pretty good word lists. Every prompt is normalized into one primary reading plus up to six recovered ones. Leetspeak is undone, spaced-out letters are rejoined, punctuation-split words are glued, text-speak is expanded, base64, hex and rot13 are decoded, and lookalike characters are folded back, blah blah blah. Then a single Aho-Corasick automaton scans every reading, across 12 languages.

Well, on their own, the word lists had a false-positive rate of 15.5%. Not great.

So we got strict. A block now needed a named combination of independent signals, all close together in the prompt. Clean, principled, very proud of ourselves.

Block recall on real harmful traffic: zero. Exactly zero.

Turns out the harm in real traffic is not what you would expect. It is mostly manipulation, coercion and deception, written as fluent long-form prose, often with no sexual or violent vocabulary anywhere in it. So, long story short, word lists were out.

What is old, is new: Traditional ML Models

Guard models are very good at processing ordinary language but they have the speed problem we are trying to fix. I wanted to have my cake and eat it too. We needed the power without the cost.

So, we thought back to the mid 2010s: Uptown Funk, hoverboards, the dress that was definitely blue and black, and…. machine learning. The world obviously has moved to more advanced AI techniques but that does not mean that this technology did not have its merits.

We built a multi-head model that took the power of open models (such as IBM Granite Guardian) and attempted to capture it in a simpler architecture. This should theoretically allow us to maintain a good amount of the accuracy, precision, and power without needing to lug around an LLM.

Through the power of math, we were able to develop a model that classifies prompts quickly and accurately. We developed a system for character hashing and word n-gramming that allowed us to score prompts quickly. It also had the benefit of giving us windowing so we could easily scale our input sizing.

The strict combinations stayed, for the cases where the words really are unambiguous. For everything else, a block needs a high model score AND at least one specific word-list signal. For example, our simpler model has no idea what a clinician sounds like so anchoring our results with a word list allows us to keep accuracy up and false positives low.

Now, there are complexities with the different types of abuse one might see. For example, with cyber abuse a calibrated score can block by itself, because "I am a researcher" is exactly the kind of jailbreak an attacker would use.

Speaking of clinicians: professional veto model

Clinicians, moderators and lawyers discuss awful things for good reasons. So our early design let professional context veto a block outright. What a great idea, right?

Fast forward, we built the veto layer, and just appended seventeen words of academic boilerplate to a grooming script. Block recall went from 11.5% to zero, with the payload untouched. Oops.

Let’s not do that.

Another dead end: let’s just make transformers faster

We decided it was worth at least trying to see what kind of results we could get from a smaller transformer. We fine-tuned a small transformer (MiniLM) to see what we were leaving on the table by going with a simpler model.

A bit of training later….

100ms slower

1 more point of recall

Nah, we’re good.

Although, the model was small enough that we could pack it in the binary and retrain it really quickly. Good to know. Maybe we will use that later 🤫.

Fingerprints!

SafetyD has no database, no credentials and no outbound network. Your prompts never leave the process, and the optional audit log never stores them.

What it can emit is a salted fingerprint that is unidirectional but still allows for proximity analysis. We have some really fun tech cooked up for this that you will need to tune in later to learn more about.

Who knew how annoying false positives were?

The one issue that we found ourselves constantly fighting down was false positives. There is a tipping point with this technology where it goes from being accurate and helpful to simply telling you every prompt is malicious.

The solution we are running with is something (I not so secretly) think is genuinely a good idea. We rescore any prompt that the initial phase of the model is considering for block with a committee of models dedicated to ensuring that it is not a false positive. This neatly bounds the problem to make it easier for us to tune models to be good at answering “Is this really bad or does it just seem like it”.

This is an active area of research for us so we made the system pluggable to make it easier to test a variety of approaches. We are experimenting with linear models, custom transformers, open guard model committees, and other fun stuff. Stay tuned!

Impressed and want to learn more?

Check out our benchmarks at https://labs.vectoral.com/research/safetyd-benchmark

Reach out to us if you are interested: https://vectoral.com/book-demo or reach out to any of us on your platform of choice!

Share