Stability Check
Updated: Mar 20, 2026
What This Chart Shows
This chart monitors whether the underlying causal relationships in your marketing data are stable over time or shifting. It uses two complementary metrics: Drift Score (how much relationship strengths are changing) and Edge Overlap (whether the same relationships exist across time periods). When these metrics show sudden changes, it indicates a "regime shift"—a fundamental change in how your marketing activities influence outcomes. This helps you understand whether your attribution results are based on stable patterns or evolving dynamics that may require attention.
Key Questions This Chart Helps Answer
- Are my marketing causal relationships stable over time, or are they shifting?
- When did significant structural changes occur in my marketing data?
- Are there regime shifts that might explain sudden changes in attribution results?
- Is the model capturing consistent patterns, or is the underlying data structure evolving?
Axes, Metrics, and Units
Element | Description |
|---|---|
| X-axis (both panels) | Window Transition: Sequential time periods compared pairwise (e.g., W1→W2, W2→W3). |
| Y-axis (Top Panel) | Drift Score: Average absolute change in edge weights between consecutive windows. Higher = more change. |
| Blue line + markers (Top Panel) | Drift Score over time. Peaks indicate periods of significant weight changes. |
| Dashed red line (Top Panel) | Threshold: Robust MAD-based cutoff. Drift above this line suggests a regime shift. |
| Star markers (Top Panel) | Regime Shifts: Detected change points where drift exceeds the threshold. |
| Y-axis (Bottom Panel) | Edge Overlap: Percentage of edges that exist in both consecutive windows (0-100%). Higher = more structural stability. |
| Teal filled area (Bottom Panel) | Edge Overlap: Shows structural consistency. Dips indicate edges appearing or disappearing. |
Key Metrics Explained:
Metric | Description |
|---|---|
| Drift Score | Average absolute change in edge weights across aligned edges. Measures how much relationship strengths are changing. |
| Edge Overlap | Intersection over union of edge sets between consecutive windows. Measures whether the same relationships exist. |
| Regime Shift | A detected change point where drift exceeds a robust statistical threshold, indicating a structural break. |
Control Options Reference
Control | Meaning |
|---|---|
| Animation Frame | (If available) Select a specific window transition to highlight. |
How to Interpret the Results
Panel 1 - Drift Score:
- Stable periods: Low, flat drift scores indicate that causal relationship strengths are consistent.
- Spikes: Sharp increases mean relationship strengths changed substantially between those time periods.
- Above threshold: Drift above the dashed line triggers a "regime shift" marker (star), indicating a statistically significant structural change.
- Multiple spikes: Repeated spikes suggest ongoing instability; a single spike may indicate a one-time event.
Panel 2 - Edge Overlap:
- High overlap (near 100%): The same causal relationships exist in both periods—structural stability.
- Low overlap: Edges are appearing or disappearing—the set of active relationships is changing.
- Sudden drops: A sharp drop in overlap indicates that many relationships changed (new ones appeared or old ones became inactive).
Combined interpretation:
- High drift + low overlap = Major structural change (relationships and their strengths both shifted)
- High drift + high overlap = Same relationships, but strengths changed significantly
- Low drift + low overlap = Different relationships, but new ones are similar in strength to old ones
- Low drift + high overlap = Stable system
Practical Applications for Marketers
Application | How to use this chart |
|---|---|
| Attribution validation | Check if attribution results are based on stable patterns or shifting dynamics. Unstable periods may require caution. |
| Campaign timing context | Align regime shifts with known events (new campaigns, market changes, policy updates) to understand causes. |
| Model retraining decisions | Regime shifts may indicate the need to retrain or segment the model for different time periods. |
| Anomaly investigation | Use detected change points as starting points for investigating unexpected performance changes. |
| Confidence calibration | Stable periods give higher confidence in attribution; unstable periods suggest more uncertainty. |
Common Mistakes & Misinterpretations
Mistake | Why it is a problem | How to avoid |
|---|---|---|
| Assuming all spikes are bad | Some drift is normal (seasonality, campaign changes). Not every spike indicates a problem. | Focus on spikes above the threshold and correlate with known events. |
| Ignoring edge overlap | Drift alone doesn't tell the full story. Low overlap with low drift still means structural change. | Always consider both panels together. |
| Over-reacting to single spikes | A one-time spike may be a data artifact or temporary event, not a fundamental shift. | Look for patterns; single spikes may not require action. |
| Expecting perfect stability | Marketing systems are dynamic; some change is expected and healthy. | Use this chart to detect meaningful shifts, not to demand zero change. |
| Not investigating regime shifts | Detected change points are signals, not conclusions. Ignoring them means missing important context. | Use regime shifts as prompts to investigate causes. |
Caveats & Considerations
- Minimum windows: Requires at least 3 time windows for meaningful drift analysis. Fewer windows will show an error message.
- Window granularity: Drift is measured between consecutive windows. The window size (e.g., weekly, monthly) affects sensitivity—smaller windows may show more noise, larger windows may miss short-term shifts.
- Threshold is adaptive: The MAD-based threshold adjusts to the data. In highly stable data, even small drift may exceed the threshold; in volatile data, only large spikes will flag.
- GPU memory: Large graphs (many edges × many windows) may exceed GPU memory. If this occurs, the chart displays an out-of-memory message.
- Correlation with events: Regime shifts are detected statistically, not causally. You must investigate to determine if a shift is due to data issues, market changes, or other factors.
- Not all shifts are actionable: Some regime shifts reflect real market dynamics that don't require model changes—just awareness.