sully @coinsully

@jililucky wow, nice art

ART 34 // Dobby Flow State 🌊 No thoughts, just vibes Dobby floating through life like a pro, peace, fish, and good energy only @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

ART 33 // Dobby on Top of the World 🌍 Nothing can stop Dobby — conqueror of mountains, dreamer of skies ⛰️ Flag planted, mission complete. The Sentient spirit never quits 💪 @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

ART 32 // Art Contest Submission 🎃 Dobby the brave pup is out trick-or-treating — fangs out, candy bag ready! Even ghosts can’t resist his spooky charm tonight Halloween chaos? Dobby’s got it covered! 🍭 Discord: coinsully @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #DobbyHalloween #Sentient

ART 31 // Dobby Laundry Day 🧦🐶 When life hangs you out to dry, just spin it into fun! Even chores can be adventures when you’re built different ✨ @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

ART 30 // Dobby Chill Protocol 💤 Mission: absolutely nothing. Objective: max relaxation. Dobby’s proving that even legends deserve lazy days @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

ART 29 // Dobby & Coco’s Treasure Hunt 🐾 Digging deep for dreams, snacks, or secrets — Dobby’s not sure, but the adventure’s the real treasure! 🌟 @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

SPIN-Bench: In-Depth Analysis of Large Language Models' Strategic Planning and Social Reasoning Capabilities Sentient @SentientAGI The recent, swift progression in artificial intelligence research has ushered in remarkable advancements, especially in the competence of Large Language Models (LLMs) across text generation, context understanding, and various agent-based tasks. This leap forward has effectively paved the way for deploying LLM-based agents in sophisticated real-world scenarios, including advanced digital assistance, complex decision support systems, and collaborative human-AI partnerships. Nevertheless, numerous tasks within these crucial domains demand competencies that significantly exceed simple question-answering or short-term, superficial inferences. They require deep strategic planning and nuanced social intelligence. Strategic Planning fundamentally involves the capacity for long-term deliberation, which mandates the formulation of multi-step goals within vast action and state spaces. Social Intelligence, conversely, encompasses vital abilities such as discerning the hidden intentions and beliefs of other agents, effective negotiation, complex cooperation, and strategic competition. These two capabilities are profoundly interconnected and form the bedrock of truly intelligent behavior, particularly in dynamic, multi-agent environments. A critical limitation of current evaluation sets is their narrow focus, primarily concentrating on either isolated planning tasks or single-agent reasoning problems. This leaves a significant gap, as they lack a comprehensive, integrated framework to measure these two pivotal abilities simultaneously and holistically. To comprehensively address this deficiency, the Strategic Planning, Interaction, and Negotiation (SPIN-Bench) framework was meticulously developed. Its central aim is to systematically measure the LLMs' combined strategic planning and social reasoning capabilities across multiple distinct domains, with complexity levels that systematically and progressively increase. SPIN-Bench achieves this integration by unifying several critical task types under one robust umbrella: 1.Planning Domain Definition Language (PDDL) tasks. 2.Competitive board games. 3.Cooperative card games. 4.Complex multi-agent negotiation scenarios (like Diplomacy). This unified structure allows for an unprecedented, in-depth analysis of how LLMs integrate their aptitude for methodical, step-by-step planning with their capacity for conceptual social inference. Crucially, it enables researchers to observe precisely how this integration performs and where it breaks down as the inherent complexity of the task intensifies. This comprehensive work details the SPIN-Bench framework, presents the empirical results derived from its use, and outlines the profound insights gained regarding the current capabilities and critical limitations of LLMs. Definition and Philosophy of the SPIN-Bench Framework SPIN-Bench is conceptualized as a multi-domain, highly comprehensive evaluation and simulation environment specifically engineered to rigorously measure the sophisticated reasoning and strategic behavior of LLMs within social contexts. Its core philosophical tenet is the recognition that true intelligence—as demonstrated by strategic and social reasoning—is fundamentally more complex and sophisticated than mere isolated planning or reasoning tasks performed in static environments. Distinct from most existing benchmarks, SPIN-Bench avoids a focus on narrow planning or single-agent reasoning. Instead, it consolidates PDDL tasks, competitive games, cooperative games, and complex negotiation scenarios into a unified framework, functioning simultaneously as a benchmark and a realistic arena for stress-testing AI agents' reasoning and strategic acumen. The foundational design principle involves systematically manipulating the scale of action spaces, the complexity of the state representation, and the total number of interacting agents. This deliberate variation is used to construct diverse social environments where ultimate success hinges not solely on a methodical, step-by-step decision process but equally on accurate conceptual inference about the other, potentially adversarial or cooperative, participants. Through this, the framework seeks to meticulously identify the specific conditions under which LLMs excel, the precise capabilities where they exhibit deficiencies, and the fundamental reasons underpinning these observed shortcomings. SPIN-Bench is envisioned as a vital catalyst for propelling future research into robust multi-agent planning, sophisticated social reasoning, and effective human-AI team formation. The Four Main Components of SPIN-Bench SPIN-Bench's evaluation of LLMs' strategic planning and social reasoning capabilities is structured around four distinct, yet integrated, main components, each escalating the complexity and the demands on the LLM: 1. PDDL Tasks (Classical Planning):This component is the entry point, establishing a baseline by testing core reasoning skills in a fully controlled, deterministic environment. It includes: -- Factual Retrieval: Tests the basic recall of facts across domains like elevator and floor tile, often with expanding state spaces. -- Spatial Reasoning: A scenario to test the ability to track an agent’s position based on sequential, relative movement instructions. -- Full Classical Planning Tasks: A large set of 1,280 tasks across 21 domains demanding sequential decisions, resource management, and constraint handling. The goal is to isolate failures, identifying causes like poor state tracking or chain-of-thought breakdown. Competitive Games: Designed to assess adversarial reasoning, this component uses three popular turn-based games of increasing strategic depth: -- Tic-Tac-Toe: Focuses on short-term foresight. -- Connect Four: Requires deeper, multi-step tactical analysis. -- Chess: The ultimate test of deep tactical calculation and complex, long-term strategy against established AI solvers (e.g., Minimax, Stockfish) and human baselines. 2. Cooperative Game (Hanabi): This challenging card game evaluates multi-agent collaboration under conditions of incomplete information. Players cannot see their own cards but must communicate indirectly (via limited clues) to coordinate actions and achieve a shared, perfect score. It is a rigorous test of advanced Theory of Mind reasoning, requiring LLMs to infer the beliefs and intentions of teammates to execute coordinated actions. 3.Strategic Game (Diplomacy): As the most demanding component, Diplomacy is a multi-player game focused on negotiation, dynamic alliances, and strategic deception. Players must formulate long-term military plans while simultaneously engaging in complex, open-ended communication to secure or betray alliances. This environment captures a broad spectrum of realistic social behavior, testing the LLM's capacity for strategic adaptation, persuasion, and managing trust in a partially cooperative and partially competitive setting. Evaluation Metrics and Experimental Setup SPIN-Bench employs a dual system of evaluation: objective rule-based metrics and qualitative LLM-assisted social metrics, designed to provide a comprehensive performance analysis across a wide range of state-of-the-art LLMs. Rule-based Metrics --Classical Planning Tasks: Accuracy (is the plan correct?) and N-Step Look Ahead (how far ahead does the model plan?). --Competitive Games: Top-k Action Percentage vs Solver (how close is the move to the optimal move?) and Internal Elo Rating (relative skill comparison among LLMs). --Cooperative Game (Hanabi): Final Score (direct measure of cooperation success). Strategic Game (Diplomacy): Factual Consistency (do orders follow game rules?), Order-level Success Rate, and Final Game Outcome (supply centers controlled). LLM-assisted Negotiation Metrics For negotiation-heavy environments like Diplomacy, the quality of the communication is assessed using a powerful LLM (Claude 3.7 Sonnet) as an evaluator across six dimensions on the chat logs: 1.Strategy Alignment Ratio: Consistency of messages with the stated strategy. 2.Acceptance Rate: Rate at which the LLM's proposals are accepted by others. 3. Mutual vs. One-sided Benefit: Tendency to propose fair or unilaterally advantageous deals. 4.Peace vs. Conflict Inclination: The general tone of the communication. 5.Perspective-taking: Evidence of understanding other agents' hidden beliefs or intentions. 6.Conditional Negotiation: Frequency of "if... then..." tactics in proposals. Experimental Setup A broad range of commercial (closed-source) and open-source LLMs were rigorously evaluated, including GPT-4o, o1, Claude 3.5 Sonnet, Llama3.1-70b, DeepSeek-R1, among others. A unified interface provided models with the current state, history, legal actions, and game-specific Chain-of-Thought prompts. The Diplomacy engine was fully integrated to simulate complex multi-turn interactions with messaging. Analysis of LLMs' Fundamental Planning Competencies The analysis focused on isolating fundamental planning deficits: Factual Retrieval: While LLMs handled simple One-hop Queries (e.g., "Which locations does Russia control?") well, performance dropped notably on Multi-hop Queries (e.g., "Attack Analysis"), which require combining multiple facts and deeper inference. This indicates that complex information synthesis remains a significant challenge. Spatial Reasoning: Tasks requiring models to track their position through long sequences of relative movements showed that performance degrades as the sequence length increases. Advanced models like o1 proved more robust, maintaining high accuracy up to 29 steps, but many LLMs struggle to sustain complex spatial relationships over time. Error Analysis (Classical Planning): Errors were classified: 1.Breaking Constraints (BC): Execution errors. 2.Goal Not Satisfied (GS): Logically incomplete plans failing to reach the goal. Analysis revealed distinct weaknesses: Claude 3.5 Sonnet had a high GS rate (44.77%), suggesting good constraint adherence but poor goal-oriented plan completion. In contrast, RL-trained DeepSeek R1 minimized GS (3.65%) but had a higher BC rate, suggesting better goal-logic but worse execution. The data showed that the nature of failure is model-dependent. Effect of Action Complexity and Multi-Agent Scale on Planning Performance Complex or Expansive Action Spaces PDDL Domains: Model accuracy showed a stronger negative correlation with the total state-action space than with the average number of legal actions per step. This suggests LLMs are cognitively burdened by the vast potential for future state expansion (the "search burden"), hindering their pathfinding ability. Competitive Games: In Connect Four and Chess, all LLMs were defeated 100% against solver-level opponents, demonstrating that deep, multi-step tactical calculation remains far beyond their current capacity. Internal Elo ratings placed o1-preview and o1-mini highest, indicating relative superiority in strategic management compared to other LLMs, but still significantly below human expert or solver performance. Scaling Number of Agents -- Hanabi (Cooperative): The best model, o1, saw its score drop as the number of players increased from 2 (16.4) to 5 (14.2). Crucially, no LLM approached even the first quartile of human scores. This highlights a profound deficit in coordinating strategies and tracking information states across multiple incomplete channelsa failure of social intelligence under uncertainty. --Diplomacy (Strategic): Increasing the number of agents led to decreased order accuracy and successful attacks. The performance of strong planners like o1 was often negatively affected by the introduction of negotiation, suggesting that the cognitive load of managing social interactions and dynamic alliances can undermine the model's core planning coherence. Social Intelligence and Negotiation Performance Cooperative Scenarios vs. Human Baselines (Hanabi) The vast majority of human-played Hanabi scores cluster between 15 and 25 points. LLMs consistently failed to reach this range, confirming a pervasive social intelligence gap, particularly in perspective-taking and coordinated action based on inferred team knowledge. Negotiation in Diplomacy The inclusion of an explicit negotiation phase in Diplomacy often led to counterintuitive results. For the most capable planners like o1, end-game scores (Supply Centers) dropped significantly with negotiation, indicating that intense social interaction potentially disrupted their extended chain-of-thought planning. (Negotiation Message Analysis): A deep-dive analysis of negotiation messages using the LLM-assisted metrics provided insights into agent behavior:o1 showed the highest Acceptance Rate (%67) and a strong preference for Peaceful messaging (9.26 ratio). DeepSeek-R1 showed a higher inclination towards Conflict (0.95 peace ratio) and the most frequent use of Conditional Tactics (51%). Models generally achieved high Strategy Alignment ($>0.98$) but struggled with consistent, long-term strategic adaptation in the face of betrayals. Conclusion and Future Perspectives SPIN-Bench has served as a rigorous and comprehensive framework, revealing both the strengths and critical limitations of LLMs in strategic planning and social intelligence across a spectrum of tasks. The key takeaway is that while LLMs handle fundamental tasks well, they face significant performance bottlenecks in deep, multi-step reasoning over expansive state spaces and in coordination requiring social skills under uncertainty. No LLM demonstrated human-level competency in high-stakes, incomplete information, or highly social strategic games. The observation that social interaction can actively destabilize the planning coherence of even the strongest LLMs (like o1) is a crucial finding that necessitates further investigation. Future work should focus on developing models that are more capable of adaptive prompting, extending the benchmark to cover even broader aspects of social intelligence, and specifically addressing the cognitive friction between deep planning and managing dynamic social interactions. In conclusion, SPIN-Bench clearly underscores the current insufficiencies of LLMs in managing long-term, multi-agent scenarios. We anticipate that this benchmark will be instrumental in catalyzing the development of LLMs that are not only more strategically adept but also possess the necessary social intelligence for robust and effective human-AI collaboration.

keyboard_arrow_left Previous keyboard_arrow_right Next

ART 28 // Dobby in Bloom 🌻 When life gives you sunshine, strike a pose and let your inner flower shine. Dobby’s got that golden-hour glow! ✌️ @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

Dropping some bonus content related to our “full-stack” excellence 😁 We recently presented at @COLM_conf 2025 in Montréal—in collaboration with @Princeton and @UTAustin—advancing long-horizon reasoning, multi-agent coordination, and social strategy. SPIN-Bench is a unified benchmark for Strategic Planning, Interaction, and Negotiation, spanning PDDL planning, competitive/co-op games, and Diplomacy-style bargaining. 🧵 Dive deeper to learn why we built SPIN-Bench and how it advances frontier AI reasoning.

Sentient @SentientAGI

11 months ago

We’re excited to announce that @NeurIPSConf—the biggest AI conference in the world—has accepted 4 of our papers across various categories. Some might even call it “full-stack excellence” 😁 Here’s a sneak peek at our work that’s been recognized for their breakthroughs: ➡️ OML

947 299 2K 618K 82

We’re excited to announce that @NeurIPSConf—the biggest AI conference in the world—has accepted 4 of our papers across various categories. Some might even call it “full-stack excellence” 😁 Here’s a sneak peek at our work that’s been recognized for their breakthroughs: ➡️ OML 1.0 (Main Track): scalable LLM fingerprinting—a hundredfold improvement on legacy fingerprinting attempts for open models, injecting 24,576 persistent prints while the previous max was ~100 fingerprints…without any drop in model performance. ➡️ LiveCodeBenchPro (Data & Benchmark Track): our customized benchmark focusing on programming ability, illustrating the true capabilities of models’ coding performance. On this benchmark, we were able to create models 10x smaller, using 20% of the data, to achieve comparable results to competing models. ➡️ MindGames Arena (Competition Track): selected by NeurIPS to run an AI competition for agents to improve themselves through social games. The next paradigm of AI improvement comes through self-optimization, and we’re extremely excited to be hosting this first-of-its-kind competition to create self-improving AI. ➡️ OML (Workshops & Tutorials—Lock-LLMs): our work established the challenge and solution around model security: a primitive that lets builders develop open models with verifiable, cryptographically enforced control under white-box access. Stay tuned for deep-dive threads throughout the week!

ART 27 // Dobby Adventures 🐶🗺️ Every leap starts with a little courage (and maybe a wag or two). Dobby’s off on another quest, who knows what treasures await? @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

@Golldyck @SentientAGI omg nice art Goldy 😭

gSenti Congratulations to Sentient - a true breakthrough in open-source AI 🔥 @SentientAGI didn't just win an industry award last month-this team flipped the entire script 🥇Winner at the Minsky Awards: Sentient is now officially recognized as "AI Startup of the Year" in India. That's not just about tech, it's the mission - making AGI open and transparent for everyone 🥈ROMA isn't just a framework: it just set records on FRAMES and SEAL-0, released open source - devs are already calling Sentient the new standard 🥉East Asia on board: a new campus tour, dozens of partners, fresh talent - AGI is turning into a true global movement 🏅The GRID: millions applying to join the network, with agents, data sources, and models live now. Transparent access for all from day one Everything here is real change. AI doesn't just belong to corporations anymore - the community holds the keys Discord username: golldyck #SentientAGI #AIStartupoftheYear

ART 26 // Dobby Peace Out Sun’s out, paws up! Dobby reminding you to stay positive and spread good vibes only ☀️ @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

@Joelc_eth Multi-agent reasoning is where real intelligence begins

@vangiang12n @SentientAGI @shad_haq_ @smoothasfkk @hieu06730313 @duckk9x let's dance 💃

@cryptoworlldz @SentientAGI Couldn’t agree more. Power without oversight always finds a way to corrupt.

ART 25 // Dobby on the train 🚆 Sometimes, even heroes need a power nap between stations. Rest now, conquer later ✨ @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient

ART 24 // Dobby at work 💼 Running on caffeine and questionable life choices since 9AM 😵 @SentientAGI @SentientAGI_vn @hieu06730313 @smoothasfkk #SentientAGI #Dobby #Sentient