AI evaluators

Classified budget estimates indicate the National Security Agency is directing billions of dollars toward testing artificial intelligence models, according to a report drawing on leaked or declassified figures. The scale of the spending suggests US intelligence agencies are moving aggressively to evaluate and adopt AI capabilities. Observers are weighing the national security implications and questions of oversight for such secretive AI procurement.

A new survey indicates that 78% of U.S. homebuyers now use artificial intelligence to help with the home-buying process, from researching listings to evaluating offers. The figure highlights how quickly AI tools have entered real estate, raising questions about the evolving role of traditional agents and how buyers make one of life's biggest financial decisions.

A new open-source project called Graphene has been launched, offering a data analysis toolkit designed to work with AI coding agents. The toolkit was shared as a launch announcement on Hacker News under its Show HN format, drawing early engagement from the developer community. Details beyond the GitHub repository are limited so far, with users likely evaluating its features and usefulness.

A developer has published a write-up after spending a month coding with GLM 5.3 Flash, a model from Chinese AI lab Zhipu. The post is drawing attention among developers weighing cheaper, faster models for everyday programming work, and it is being discussed on Hacker News, where readers are sharing their own experiences with budget coding models.

Anthropic says a popular Chinese open-weight AI model shows 'Mythos-class' hacking capabilities, according to its frontier red-teaming report. The company's findings also detail weak safeguards on open-weight AI systems, raising concerns that such models could be misused for cyberattacks. The report is prompting debate over how open model releases should be evaluated and restricted.

OpenAI has released a repository containing mathematical manuscripts along with supporting proof artifacts produced by its models. The collection gives researchers and mathematicians material to examine how AI systems formulate and verify formal proofs. The release is drawing attention from the AI and mathematics communities, who are assessing the quality and significance of the generated work.

A new report highlights the role of AI evaluators, the specialists tasked with assessing whether advanced AI systems are safe before deployment. Their work is framed as a critical safeguard against catastrophic AI risks, yet it is largely voluntary and under-resourced. Commenters are debating whether this handful of assessors can realistically hold back powerful technology developed by some of the world's richest companies.

Security firm Manifold Security published research asking whether AI models factor morality into their judgments about malicious code. The finding: when asked to assess whether code is malware, language models appear to bring moral reasoning into their analysis rather than relying purely on technical criteria. The report is circulating among developers and security researchers interested in how AI tools evaluate potentially harmful software.

Companies are pouring money into artificial intelligence, but measuring the return on that investment is proving elusive across the industry. Many firms struggle to link AI spending to concrete business outcomes, leaving executives uncertain whether the technology is delivering value. The issue is prompting debate over how AI projects should be evaluated and whether current spending levels are justified.

Banana Pi has introduced the BPI-CM5 Pro, a compute module built on Rockchip's RK3576 chip with a 6 TOPS NPU, 8-32GB of RAM and 8-128GB of eMMC storage. The maker positions it as a board for Edge AI projects and an alternative to the Raspberry Pi Compute Module 4, drawing interest from hobbyists and robotics developers.

OpenAI has made 722 math manuscripts publicly available through a repository on GitHub, opening up a collection of mathematical texts to researchers and the public. The release is being discussed among developers and AI enthusiasts, who see it as a potentially useful resource for training and evaluating mathematical reasoning in AI models.

Andrej Karpathy, the AI researcher and OpenAI co-founder, has shared practical advice on how to interpret and evaluate the outputs of AI language models more clearly. His guidance, circulated widely on X, covers ways users can check whether model responses are accurate rather than taking them at face value. Readers are discussing the tips as interest grows in how everyday users can judge the reliability of AI-generated answers.

United Nations University is working to create a community of independent AI evaluators, aiming to strengthen global oversight of artificial intelligence systems. The initiative points to growing concern that AI development is outpacing reliable, impartial assessment of its risks and impacts. The UN University's role suggests an effort to coordinate expertise across countries and disciplines for credible, independent evaluation capacity.

A new evaluation from Fig reports that frontier AI models, including Astra and Opus 5.5, perform unevenly — 'jagged' — across agentic tasks, doing well on some web browsing challenges while struggling on robotics tasks. The findings suggest that even top models lack consistent reliability, with strong results in one domain not predicting competence in another.

TechCrunch has announced the panel of judges for the Startup Battlefield 200 at TechCrunch Disrupt 2026. The judges will evaluate the competing startups and decide which company takes the top prize at the conference. The lineup leans heavily toward technology and AI-focused investors and operators, drawing attention from the startup community ahead of the event.

A developer has built races between AI coding agents to test which one performs best, challenging the frequent claims that any single agent is "obviously the best". The finding, shared widely in software and open-source communities, was that the fastest agent was not the winner, prompting discussion about how AI coding tools should actually be evaluated.

OpenAI has released hundreds of new mathematical results, adding to a wave of AI-generated output that has already unsettled researchers in the field. The announcement, covered by Scientific American, is drawing attention because of the sheer volume of results and ongoing debate about how AI-generated mathematics should be evaluated and verified.

OpenAI has released findings based on 377 math problems, an announcement reported by The New York Times as further roiling the artificial intelligence field. The release has drawn attention within the AI research community, where debate continues over model capabilities, evaluation methods and the pace of progress on mathematical reasoning benchmarks.

A developer is working on real-time video compositing and trying to bring processing time per frame under 50 milliseconds, a threshold needed for smooth real-time performance. They asked an AI assistant to evaluate performance and are now working on a strip-parallel compositor that splits the frame across threads, sacrificing four threads in the process. The work is being done while also livestreaming and gaming, and the developer has shared the experiment publicly.

Cisco's Foundation AI team published its September update, spotlighting the VLoc Bench and new work on cyber-capability safety. The release covers how the team benchmarks and evaluates AI models for security-relevant capabilities, part of a broader push to make sure advanced AI systems do not amplify cyber threats. Readers in the AI security community are discussing what the benchmarks mean for safety testing.

Bitcoin.com News evaluated America.gov, a federal portal using AI from Google and SpaceX, on Bitcoin policy questions. The portal answered 9 of 12 questions correctly, performing less accurately than ChatGPT and Claude in the same test. The comparison highlights how leading commercial AI chatbots stack up on cryptocurrency policy knowledge, with a government-backed tool trailing behind.

AI Ready Roanoke is evaluating how new technology, particularly artificial intelligence, could be used across the Roanoke Valley in Virginia. The initiative, covered by the Roanoke Times, is examining practical applications for local businesses, institutions, and residents as AI adoption spreads. Details on specific projects, partners, and timelines have not yet been reported.

Data engineering company Innodata has opened a new lab focused on robot training data, drawing attention from investors assessing what the move means for the firm's growth strategy. The company, known for AI data services, is positioning itself in the emerging market for data used to train robotics systems. Market watchers are weighing whether the lab can translate into revenue and how it might affect the stock's performance going forward.

The White House has created a task force to assess the risks of artificial intelligence, according to the Wall Street Journal. The group will evaluate potential dangers from rapidly advancing AI technology, part of a broader effort by the US administration to develop policy on AI safety and regulation.

The Department of Veterans Affairs held a workshop in Boston examining how artificial intelligence could improve care for veterans. The event tested potential applications of AI within VA healthcare services. It reflects the agency's broader effort to evaluate whether AI tools can support clinical decisions, administration, and patient outcomes for former service members.

The American Hospital Association has published an overview of how nurses are using artificial intelligence in their work, presenting the available data in chart form. The roundup highlights adoption patterns and practical applications of AI tools in nursing practice, a topic of growing interest as hospitals weigh efficiency gains against concerns over patient safety and job roles.

Brands are adjusting how they market products as artificial intelligence agents increasingly shop on behalf of consumers. Instead of pitching to human shoppers with emotional advertising, companies are optimizing product data and descriptions so AI assistants can find, evaluate and recommend their goods, a shift with significant implications for e-commerce and digital marketing.

A debate is under way in the scientific community over whether artificial intelligence should be used in the peer review of research papers. Supporters see potential to speed up reviews and ease reviewer shortages, while critics worry about bias, confidentiality of unpublished manuscripts, and the risk of undermining trust in scholarly publishing.

A new paper on arXiv proposes a way to fix the credit assignment problem in GRPO, a reinforcement learning method used for training language models, without evaluating every step of a response. GRPO currently assigns the same reward to all tokens in a completion, making it hard to identify which parts of an output earned the reward. The proposed approach aims to improve this at lower computational cost.

OpenAI announced a partnership with contract management company Ironclad to train and evaluate AI agents on complex contracting workflows. The collaboration aims to advance computer-use AI for professional work, using Ironclad's real-world legal and contracting tasks as a testing ground. It signals a push toward deploying AI agents in specialized business environments beyond general browsing.

A Fast Company article argues that companies adopting AI need clear measurement systems—scoreboards—to track whether their AI investments are actually delivering results. The piece taps into a wider business debate about how firms should evaluate AI tools beyond hype, using metrics and benchmarks to judge performance and justify spending.

Venture capital investors are reportedly exploring what AI cannot explain when evaluating startups, even as AI tools increasingly analyse pitch decks and company data. The discussion centres on whether human judgment still matters in investment decisions once machines can process the numbers. Details of specific firms or funds involved remain limited, and reaction from the wider venture community is not yet clear.

A growing number of employers are permitting job candidates to use AI coding agents during technical interviews. The shift changes what is being assessed: rather than writing code unaided, applicants must show they can direct and work with AI tools effectively. Commenters note this mirrors real-world engineering work, where AI assistance is now standard, and raises questions about how to evaluate core programming ability.

A widely shared essay argues that in the era of AI agents, the harness—the scaffolding of tools, prompts, evaluation and workflow code wrapped around a model—is where a company's actual value sits, not the underlying model itself. As models become commoditised and interchangeable, the author contends the harness is the durable product, and effectively the company's true identity.

A new write-up ranks AI language models by their normative score, tested across twelve different paradigms and repeated twice — once under neutral conditions and once with human priming. The comparison aims to show how priming changes model behavior and judgments, offering a benchmark-style look at consistency across evaluation setups.

Kapa.ai has published a benchmark for evaluating how well retrieval systems let AI agents work with messy, real-world company knowledge bases. The release is drawing attention among developers and AI practitioners, who are discussing how enterprise search and agent performance should be measured outside clean, curated datasets, where documentation is inconsistent, outdated or scattered across tools.

New York law firm Pryor Cashman has published a policyholder's guide to insurance coverage for artificial intelligence, explaining how existing policies may or may not respond to AI-related risks and losses. The guidance walks businesses through evaluating whether their current coverage extends to AI tools, what exclusions to watch for, and how to negotiate better protection. It reflects growing demand from companies using AI who are unsure whether their insurers will cover AI-related liability.