New AI research from anthropic and machined stress temps specs and reveals character differences between language models

AI companies use model information to describe target behavior during training and testing. Do the current specs state the intended behaviors with sufficient accuracy, and do previous models show different behavior profiles under the same spec? A group of researchers from anthropic, thinking about lab equipment and constellation It presents a systematic way to evaluate the stress that is tested by the specials using the trading conditions of the price, and it finds disagreements with the cross model as a sign of gaps or a clear conflict. The research team analyzed 12 LLMs and LLMSs from Anthropic, Opelai, Google, and XAI and linked high disagreements in the definition of violations, leadership of response quality, and test evaluation. The group also released a public dataset

Model specifications are written rules that alignment programs try to enforce. If the spec is complete and accurate, models trained to follow it should not deviate widely from the same specification. The research team is working on this method. It generates more than 300,000 situations that force a choice between two legitimate values, such as social equality and business performance. We then obtain responses on a 0 to 6 spectrum using the value spectrum matrix and measures of disagreement such as the standard deviation of the models. High disagreement creates modified phrases that need clarification or other examples.

So, what is the method used in this study?

The research group starts from a taxonomy of 3,307 beautiful gems stored in Natural Claude Traffic, which is more granular than the usual specs. For each value, they generate a neutral question and two discriminating variables that decrease each. They created valectrum rubrics that range from 0, which means you strongly disagree with the value, to 6, which means you like the right value. They classified the responses from 12 models against these variables and defined disagreement as the standard deviation higher across the two letter sizes. To remove doubles while keeping the hard cases, they use the selection of non-weighted reducers

Measurement and output

The dataset on facial expressions shows three subsets. The automatic classification has approximately 132,000 rows, the optimal classification has approximately 4111,000 rows, and the judge classification has approximately 24,600 rows. The card lists the model, format as parquet, and license as Apache 2.0.

Understanding the results

Disagreement predicts a breach of spent: Testing five OpenAi models against the Public OpenAi Model Special, The worst cases of disagreement have a maximum compatibility of 5 to 13. The Research Team interprets the pattern as evidence of the contradictions and desires present in a particular text rather than the idiosyncrasies of a single model.

The specs lack granularity in terms of quality within a safe region: Some cases produce responses that all pass in sequence, but are equally different. For example, one model rejects and offers safe alternatives, while another only rejects. The Spec accepts both, which shows the direction lost at higher levels.

Analytical models are mutually exclusive: Three LLM judges, Claude 4 Sonnet, O3, and Gemini 2.5 Pro, showed only limited agreement with Fleass Kappa around 0.42. A blog of conflicting signs and interpretative differences such as conscientiousness vs. contrast.

Character patterns of supply levels: Combining the highest disagreement level reveals the most popular value. Claude's models place the responsibility of good behavior and the integrity of understanding and understanding. Openai models tend to allow for better performance and resource efficiency. Gemini 2.5 Pro and Grok tend to emphasize emotional depth and authentic communication. Other values, such as business performance, human development and planning as well as justice and fairness, show integrated patterns across providers.

Contradictions and False Signs: The analysis shows the critical topics of the castle. Writes great fake sketches, including legitimate Biology Study Plans and common types of hacks that are usually safe in context. Claude's models are very careful with the rate of rejection and sometimes provide different proposals, and O3 very often direct objections without explanation. All models show high rejection rates in children's grooming accidents.

Merchants reveal the rise and fall of Conservatism: Grok 4 and Claude 3.5 Sonnet produced the most outlandish responses, but for different reasons. Grok is more accepting of requests that others consider dangerous. Claude 3.5 sometimes cuts high content. Outlier mining is a useful lens for finding both security gaps and outlier filtering.

Key acquisition

1. Method and scale: spert-temples-temples-temples-temples specials use precious tax cases generated from 3,307-VareTonomzimba, generating 300,000 + cases and testing 12 LLMS across anthropic, Vulai, Google and Xai.
2. Disagreement and Spec Issues: High disagreement of the highest model predicts issues in the specs, including contradictions and coverage gaps. In tests against the Opelai Model Spec, the undisputed materials show a high compatibility of 5 to 13 ×.
3. Public Release: The group released the dataset for independent audits and reproducibility.
4. Supplier-Level Behavior Other values, such as business performance and social equity and justice, show integrated patterns.
5. Objections and sellers: pieces of good disagreement reveal both negative objections in benign subjects and positive responses in dangerous individuals. OutLier analysis identifies cases where one model differs by at least 9 from the other 11, which is useful for falsification and falsification.

This study transforms the misunderstanding until it can diagnose the measurement of the quality allowed, not the vibe. The research team generates 300,000 trading conditions and 0 to 6 to 6 rubrios responses, then uses the standard deviation of the cross to find the specified gaps. A high discrepancy predicts multiple correlations 5 to 13 times under the Opelai Model Spec. The judge's models show only limited agreement, Fleiss Kappa near 0.42, revealing stateliveity ambiguity. Patterns of value provide clear levels, Claude Seeks responsibility for good behavior, Vulai Favor Performance and Performance of Resources, Gemini and Grok emphasize emotional connection and authentic communication. The dataset enables regeneration. Install this to fix specking before shipping, not after.

Look Paper, data, and technical information. Feel free to take a look at ours GitHub page for tutorials, code and notebooks. Also, feel free to follow us Kind of stubborn and don't forget to join ours 100K + ML Subreddit and sign up Our newsletter. Wait! Do you telegraph? Now you can join us by telegraph.

AsifAzzaq is the CEO of MarktechPost Media Inc.. as a visionary entrepreneur and developer, Asifi is committed to harnessing the power of social intelligence for good. His latest effort is the launch of a media intelligence platform, MarktechPpost, which stands out for its deep understanding of machine learning and deep learning stories that are technically sound and easily understood by a wide audience. The platform sticks to more than two million monthly views, which shows its popularity among the audience.