Definition

Calibrated Probability

A probability estimate that accurately reflects the true likelihood of an outcome over many predictions: if you say something has a 70% chance, it should materialize roughly 70% of the time. Calibration is a distinct measure from raw accuracy - a model can win most of its bets while being badly miscalibrated (e.g., stating 95% confidence on events that only happen 60% of the time). LightningRod measured its forecasting model on Polymarket using calibration as a primary metric, not just profit or win rate. Ben Turtle uses Polymarket bets as a forcing function for calibration training: putting money down on a prediction and seeing the outcome is the fastest way to build an accurate internal probability estimator.

From the episode “AI That Predicts the Future”

Hear it on the podcast1 episode

AI That Predicts the Future

Learn the language of AI-native founders

Every week: the new terms, tools, and workflows founders are using, straight from the podcast.

Related terms in AI Fundamentals

Bio-AI hybridSpotitEarly's term for a platform where a biological sensor sits at the front of the pipeline and machine learning handles interpretation and verification.Brand Knowledge GraphEmberos's proprietary mapping of how a brand's entities, products, narratives, and digital assets relate to each other across the internet, enabling predictive visibility modeling.BYO-LLM (Bring Your Own LLM)An architecture pattern where enterprise customers plug their own AI vendor API key into a product so data flows through their existing legal agreements rather than a vendor's.CAI (Collective AI)Collective Health's AI suite built with Google Cloud, spanning a member concierge, an agent copilot, and an enrollment tool.Capability BoundaryThe outer limit of what a person or team can produce at a given level of staffing and skill. AI extends this boundary — enabling smaller teams to produce work previously requiring larger ones. Section operates with 32 people doing the work of 45. For founders, this changes the build vs. hire calculation: always audit your AI-extended capability before adding headcount.Capability overhangThe gap between what frontier models can already do and what they're actually being used for.ChatbotA program that holds a conversation with people through text or voice.Chronological Prediction TrainingLightningRod AI's core training method: given all data up to timestamp A in a corpus, predict what will happen at timestamp B. The model is trained iteratively across an entire dataset of chronologically ordered real-world information. The 'label' is simply what actually happened next - no human annotation required. This approach forces the AI to learn causal relationships rather than surface patterns: to get good at predicting timestamp B, the model must understand what factors in timestamp A's data actually drive outcomes. Ben Turtle calls this the mechanism humans use to build world models - we move through time, predict what comes next, feel surprise when we're wrong, and update our beliefs.

Browse all 1,456 terms