Ember is a platform that provides a public record of AI model forecasts on prediction markets, systematically auditing and scoring their predictions against real-world outcomes. For AI alignment analysts and researchers, this creates a transparent, long-horizon record of how advanced models reason about the future, offering crucial data for safety, capability, and calibration studies. By tracking AI predictions in a structured way, analysts can gain unique insights into model behavior when faced with complex, real-world events.
What is Ember and Why It Matters for Prediction Markets
Prediction markets function as exchanges where users trade contracts on the outcomes of future events, creating a powerful dataset of crowd-sourced probabilities. These platforms offer unique signals by aggregating diverse information into a single, tradable price. Ember operates within this ecosystem as a specialized intelligence layer. The platform focuses on using these markets as a benchmark to audit the forecasting accuracy of prominent AI models like Claude, Grok, and Gemini.
Ember's role is distinct from that of a market operator such as Polymarket or Kalshi. Instead of being a venue for trading, Ember positions itself as a neutral publisher of AI performance data. Ember runs three frontier models — Claude, Grok, and Gemini — and each produces its own independent probability forecast on the same market. All forecasts are locked before the outcome is known and Brier-scored against the result, with Ember's published headline forecast serving as its audited call. Each model brings different strengths, such as Gemini grounding its calls in live search results, but they are independently scored forecasters. This process creates a permanent, publicly scrutinized track record.
How Ember Solves the AI Forecast Calibration Problem
A central challenge in AI development is calibration: ensuring that when a model states it is 80% confident, it is correct about 80% of the time. Poorly calibrated models can be dangerously overconfident, a significant concern for AI safety. Ember addresses this directly through its methodology of pitting AI forecasts against the 'wisdom of the crowd' embedded in prediction market prices. The platform's system is designed to identify and scrutinize moments of significant disagreement between AI and human consensus.
Ember's service highlights where the AI forecast diverges from the market price, flagging high-conviction forecast divergences between its audited AI call and the prevailing crowd price on a platform like Polymarket. This systematic tracking of divergence is the core of its approach to calibration. When an AI's probability starkly differs from a liquid, real-money market, it represents a valuable learning opportunity. By documenting the AI's reasoning in these specific instances and later scoring the outcome, Ember creates a feedback loop. This allows alignment analysts to study not just the AI's knowledge, but its meta-knowledge—how well it understands the limits of its own predictive power.
Key Prediction Market and Forecasting Terms to Know
Understanding Ember's work requires familiarity with the language of forecasting and prediction markets. These markets are growing rapidly, with platforms like Polymarket and Kalshi leading in open interest. Here are some essential terms for analysts.
| Term | Definition |
|---|---|
| Prediction Market | An exchange where users buy and sell contracts based on the outcome of future events. |
| Event Contract | A tradable asset that pays out if a specified real-world event occurs by a certain date. |
- Brier Score: A scoring rule that measures the accuracy of probabilistic predictions, used by Ember for evaluation.
- Forecast Divergence: The difference in percentage points between an AI model's forecast and the consensus probability on a prediction market.
- Open Interest (OI): The total value of outstanding contracts in a market, indicating liquidity and participant engagement.
How Ember Audits AI Forecasts Against Real-Money Markets
Ember's auditing process is a structured daily routine designed to test AI forecasting capabilities against rigorous benchmarks. The platform begins by analyzing information from multiple sources, including major prediction markets like Polymarket and Manifold Markets, alongside category-specific research feeds. This information forms the basis for a falsifiable prediction about a future event. This step is critical in an environment where, according to Crisil Coalition Greenwich's 2026 Prediction Markets Flash Study ("Prediction Markets: It's All About the Data", greenwich.com), liquidity can be a concern on some contracts, making the choice of market and question paramount.
Once Ember runs the three frontier models — Claude, Grok, and Gemini — each produces its own independent probability forecast. All forecasts are then time-locked before the event's outcome can be known, ensuring the integrity of the test. This creates a permanent, public forecasting record for each model.
Over time, this record is evaluated using Brier scoring, a method that quantifies the accuracy of probabilistic forecasts. This systematic approach transforms the abstract concept of AI reasoning into a measurable, auditable performance metric, allowing analysts to track how different models succeed or fail at predicting the future over a 365-day cycle.
Why Audited AI Forecasts Matter for Alignment Analysts
For AI alignment analysts, the theoretical capabilities of models are only part of the puzzle; understanding their practical reasoning, biases, and failure modes is crucial. Ember provides a unique lens into this by conducting daily public auditing of Claude, Grok, and Gemini models against real-money prediction markets. This is about creating a transparent, long-term record of how AIs interpret complex information, formulate probabilities, and justify their conclusions on questions relevant to AI trajectory and safety.
This structured data is particularly valuable because many prediction markets focus on broad AI capability timelines, which may not be central to day-to-day alignment work. Ember's specific, daily forecasts offer a more granular view. An analyst can study a model's reasoning when it diverges from the human consensus on Polymarket, potentially revealing novel cognitive patterns or systematic biases that are critical for developing safer, more reliable AI systems.
Simulated Scenario Where Tracking Divergences Daily with Ember Makes a Significant Difference
An AI alignment analyst's work often involves identifying subtle, emergent behaviors in models. Ember provides a daily, structured environment for this discovery. For instance, an analyst at a research institute might use Ember's service to monitor how different AI models assess the probability of a specific technological breakthrough. Subscribers receive access to forecasts and reasoning, including live probabilities, the AI model's detailed reasoning, and conviction notes behind each forecast.
Imagine a scenario where Gemini and Grok both agree on a high-probability forecast for a new AI paper's impact, but the Polymarket odds are significantly lower. The analyst can dive into Ember's provided reasoning to understand the factual basis for the AI's confidence. They can see if the models are overweighting certain data, such as social media sentiment from X, or if they have identified a structural trend the market is missing. This daily workflow transforms abstract alignment questions into concrete, testable hypotheses based on observable model behavior in a real-world predictive environment.
The Bottom Line for AI Alignment Researchers
For analysts focused on AI safety and alignment, the most critical factor is access to transparent and verifiable data on model behavior. Ember provides a novel stream of this data by systematically testing AI reasoning against the financial stakes of prediction markets. The decision to integrate Ember into a research workflow depends on the need for a structured, daily record of AI forecast performance. If your work requires moving beyond theoretical benchmarks to observe how models perform on complex, real-world events, exploring Ember's audited forecasts is a logical next step.
Frequently Asked Questions About Ember
What AI models does Ember use in its forecasting process?
Ember runs three frontier models — Claude from Anthropic for its careful, first-principles reasoning and domain knowledge in AI; Grok from xAI to read real-time sentiment and cultural context from X; and Google's Gemini to ground every call in live search results for factual verification. Each model produces its own independent, locked, Brier-scored forecast on the same market, and Ember publishes an audited headline forecast plus the full record.
How does Ember differ from a prediction market platform like Polymarket?
Ember is not a prediction market but an intelligence layer that uses them as a benchmark. A platform like Polymarket is a venue where users trade contracts on event outcomes. Ember, in contrast, does not operate a market. Instead, it positions itself as a neutral publisher that audits AI models' predictive accuracy against the probabilities generated by markets like Polymarket. It serves analysts and traders by providing data and insights about AI performance.
