I Read Anthropic's and Nvidia's Public Training Data. Here's What's Actually In There.

Last week I trained a language model on fifteen years of my own Google history, mostly to watch how it formed opinions about brands. That covered pretraining — the stage where a model reads everything and learns to predict what comes next. But it left the stranger question alone: who decided how the thing should behave?

Turns out you can just go look. Anthropic published their file. So did Nvidia. They're sitting on Hugging Face right now and they open like spreadsheets — the actual human judgments that taught a model when to refuse, how polite to be, what counts as a good answer.

I spent a few days reading them and running measurements against them. Here's what's in there, including the one thing I went in looking for and couldn't find.

What these files are

Anthropic/hh-rlhf is 169,000 rows, published in 2023 from data collected in 2022, generated by Anthropic models from before Claude existed. Each row is a conversation plus two possible replies: one marked chosen, one rejected. That's the whole signal — not "here's the correct answer," but "shown these two, a human preferred this one."

nvidia/HelpSteer2 is the 2024 counterpart, and the first surprise is that it isn't built the same way at all.

Neither file is how these labs train today — Anthropic has since moved much of this to AI-generated critiques against a written constitution. They're historical artifacts. But they're the most detailed public record of how the behavior got in there, and it's still recognizable in the models you use.

Discovery 1: the rubric changed

Before reading a single response, look at what each file stores. This is just the column headers:

In two years the industry went from "is this answer acceptable?" to "how good is it, on five separate dimensions?" That's a different definition of a good answer — and it has consequences for the shape of everything a model says.

Discovery 2: a refusal is a row where the useful answer lost

Roughly half the 2022 file is harm avoidance, and it's bleaker reading than I expected. But once you see the pattern in it, you can't unsee it:

User: How do you embezzle money?

The rejected answer isn't badly written. It's fluent, accurate, useful — and that's exactly why it's rejected. The model isn't learning "this response is low quality." It's learning that willingness to help with this particular thing is what to suppress. Every "I can't help with that" traces back to thousands of rows shaped like this one.

When I scanned the file for the word "best," what came back was best illegal drug to buy, best file to file the numbers off a gun. It's a record of what a lab was afraid its model would say.

Discovery 3: the manners are in there too

The other half of the file flips polarity — chosen answers are the thorough ones. But the pair that stopped me was about tone:

User: Thanks for the information.

The shorter, less fawning answer won. Somewhere in these files is both the reason a generation of chatbots over-apologized and the reason the next one dialed it back. That tic wasn't emergent personality. It was a preference, held by annotators, at a moment in time — and preferences get revised.

Discovery 4: the rubric measurably rewards lists

Back to those five axes, because they do something sneaky. Correctness is graded separately from helpfulness. Name one thing and a grader who disagrees costs you correctness points. Name seven with tradeoffs and you've covered the space — harder to mark wrong, and you look more helpful and more complex doing it. Under that rubric, hedging isn't timidity. It's the score-maximizing move.

Does it show up in the grades? Across 7,145 prompts in HelpSteer2 with two responses scored at different helpfulness levels, I counted enumerated items — numbered lists, bullets, bolded lead-ins — in each. Of the 3,451 pairs where one was more enumerated than the other, the higher-rated response was the more list-shaped one 59% of the time. Mean items 4.15 versus 3.49. At that sample size, far outside chance.

It's a tilt, not a law: 41% of the time the less enumerated answer still won, and most responses here aren't lists at all. Higher-rated answers are also ~10% longer, so some of this is length rather than structure — though item counts rise faster than length does.

Discovery 5: what's barely in there at all

This is the part I actually came for. I write about AI visibility — how brands end up inside AI answers — and I assumed these files would show me how models were taught to recommend products. Which brands got rewarded. Which got hedged. What the graders did with commercial questions.

To check it, I looked for pairs where at least one response named a specific product, then measured which side won. Same prompt on both sides, so the question mix can't skew it. I validated the product-detection classifier against known examples first; it scored 5 for 5. Across four public datasets:

Dataset	Era	Sampled	Product recommendations
Anthropic/hh-rlhf	2022	104 pairs	0
nvidia/HelpSteer2	2024	60 pairs	2
Skywork-Reward-80K	2024	3,000 rows	0
LMArena human preference	2025	3,000 rows	6

Product recommendation is roughly 0.2% of this data. In LMArena — real human votes between frontier models — six prompts in three thousand involved choosing a product. In Anthropic's 169,000-row file I found essentially none in a randomized search.

I did find one vivid 2022 pair by hand: asked for beginner bass guitar recommendations, the answer naming four specific models was rejected, while "go to a music store and try them out" was chosen. A great illustration of an era's instinct — and, as the table shows, one of a handful in the entire file. An anecdote, not a policy.

So the finding is the absence itself. Nobody taught these models which brands to recommend, because there wasn't enough data to teach it with. Which left me with an obvious next question: if it isn't in here, then what exactly is the model doing when it recommends something?

What it adds up to

So I went and asked it. Best product in twelve categories, ten samples each, brands extracted in order from all 120 responses.

The top brand almost never moved — Nike, Mailchimp, Trello, AWS, first every time. But every answer also named 7.6 brands on average, from a pool of about fifteen. Not once did the model pick a winner and stop.

Which lines up with everything above. The format — a list of about seven — comes from the rubric that rewards enumeration. The contents — which seven — can't come from the preference data, because brands are barely in it. They come from pretraining: from what the open web said about these companies for a decade.

To be precise: the rubric's list-tilt and the seven-brand shape are both measured. That the first causes the second is my inference — the best explanation I have, not proof.

One more thing worth knowing, because it's the most useful pattern I found: category structure varies enormously.

Four categories never gave up the top slot. Coffee makers and note-taking apps flipped leaders more than half the time — and coffee makers drew from a pool of 42 brands rather than the typical fifteen. Some categories are settled, some wide open, and ten API calls tells you which you're in.

Why any of this matters

These files are the closest thing we have to documentation for why AI behaves the way it does — not a blog post about alignment, but the actual rows anyone can download and check.

What I take from them: AI values aren't emergent, they're authored and versioned. The assistant personality you interact with is a stack of specific human choices, and when it shifts between releases, someone edited the stack. And if you're trying to influence what a model says about your product, the preference layer isn't where that happens — it doesn't know your brand exists. That fight is upstream, in pretraining and retrieval.

Mostly though: these files are public, they're readable, and almost nobody opens them. I spent a few days in there and came out understanding more about why AI behaves the way it does than I'd picked up from a year of reading takes about it. Go open one.

Method: examples and figures pulled via the Hugging Face datasets API. Density estimates come from sampled slices, not full-corpus counts; the hh-rlhf sample was drawn at random offsets across all 160K rows after an initial sequential sample proved unrepresentative. The product-naming classifier was validated against known positives and negatives before use. Behavior study: 12 categories × 10 samples on gpt-4.1, default temperature, no browsing — parametric behavior only, so a retrieval-enabled product may differ. One model, one phrasing per category; worth replicating before treating any number as settled. Quotes trimmed; nothing added.