AI benchmarks

A new open-source project called Strata claims it can run the Qwen 3.8 Flash Next 125-billion-parameter model on a single consumer RTX 4090 graphics card at roughly 100 tokens per second. If the benchmarks hold up, it would let hobbyists and small teams run a frontier-scale language model locally without expensive data-center hardware. The project has drawn attention on developer forums, where users are questioning the memory techniques behind the speed claims and awaiting independent replication.

Anthropic's Opus 5.5 model, running as autonomous research agents, has reportedly identified two candidate materials for room-temperature magnetic semiconductors, a long-sought class of materials that could enable spintronic devices without cryogenic cooling. The work, described by AI benchmark firm Vals AI, is drawing attention for showing AI agents contributing genuine materials-science discoveries rather than incremental analysis.

A new benchmark called Jevman has been launched that evaluates AI decision-making models by having them play Pac-Man. The project, presented by Opper AI, uses the classic arcade game as a testing ground for how well language models plan, weigh risks and make sequential decisions. It is drawing attention from developers and AI researchers discussing whether game-based benchmarks meaningfully measure model reasoning ability.

AIMS Lab at Stanford has made available a textbook on AI Measurement Science, a field focused on rigorously evaluating and measuring the performance of artificial intelligence systems. The free online textbook is drawing attention among technologists discussing how AI capabilities should be quantified, benchmarked and validated as systems grow more complex and their evaluation methods come under scrutiny.

New research from Epoch AI finds that frontier AI models fell short of reproducing a human algorithmic innovation, highlighting gaps between benchmark performance and genuine inventive capability. The study examines how well current models can independently rediscover improvements humans devised, and commentators are debating what the results say about AI's real research abilities.

Robotera says its VPP2 world action model has taken the top spot on the RoboDojo leaderboard, scoring 58.5% zero-shot on real ALOHA robot arms. The result, reported by Pandaily, suggests the model can transfer directly to physical hardware without task-specific fine-tuning, a key test for general-purpose robot control.

The Associated Press Stylebook added a dedicated artificial intelligence chapter in 2023 and expanded it earlier this year. The guidance goes beyond terminology, urging journalists to critically evaluate AI tools and their limitations. Commenters are highlighting how the AP's advice is ethically grounded, framing it as a benchmark for responsible AI use in newsrooms.