Search & Recommendation

Ideally 55-60 characters

Shan Dou

May 13, 2025

Intervention Harvesting for Position Bias Estimation

Intuitive Explanation

Intervention harvesting is a clever approach to estimate position bias in search rankings without running expensive controlled experiments. Here's the intuition:

In any mature search system (like Google, Amazon, etc.), the ranking algorithms have evolved over time through:

Multiple versions of production models

Various A/B tests

Algorithm updates and improvements

This means that the same item (document, product, etc.) has likely appeared in different positions at different times in your historical data. This creates a natural "randomization" of positions that we can leverage.

Key Insight

If the same item appeared in position 1 sometimes and position 5 other times, we can compare how users interacted with it in these different positions. The difference in click-through rates is largely attributable to position bias rather than relevance (since it's the same item).

How It Works Computationally

Data Collection: Gather historical logs containing:

Items displayed in search results

Their positions

Whether users clicked on them

Which ranking model/version was active

Identify "Natural Randomization": Find items that have appeared in multiple positions across different ranking models or time periods.

Statistical Modeling: For each item i, we can model:

P(click | item i, position j) = relevance(i) × position_bias(j)

Where:

relevance(i) is the inherent quality of item i

position_bias(j) is the bias factor for position j

Estimation Process:

For pairs of observations where the same item appeared in different positions

Set up equations relating click probabilities to position bias factors

Solve these equations (often using maximum likelihood estimation)

Implementation Approach:


For each item i that appears in multiple positions:
  For each position pair (j, k) where item i appeared:
    Compute click-through rate at position j: CTR_j
    Compute click-through rate at position k: CTR_k
    Estimate position_bias(j)/position_bias(k) ≈ CTR_j/CTR_k

Normalize these ratios to get absolute bias values

Advantages Over Traditional Methods

Cost-effective: No need for randomized interventions that might hurt user experience

Uses existing data: Leverages historical logs you already have

More representative: Based on real user behavior over time, not artificial experiments

Continuous updating: Can be updated as new data arrives

This approach essentially "recycles" the natural variation in your ranking system's history to extract valuable position bias insights without disrupting the current user experience.

Here's a more detailed implementation of the intervention harvesting approach for position bias estimation:


import numpy as np
from collections import defaultdict

class PositionBiasEstimator:
    def __init__(self):
        self.item_positions = defaultdict(list)
        self.clicks = defaultdict(list)
    
    def add_data(self, item_id, position, clicked):
        """Add observation to dataset"""
        self.item_positions[item_id].append(position)
        self.clicks[item_id].append(int(clicked))
    
    def compute_ctr_ratios(self):
        """Compute CTR ratios for items appearing in multiple positions"""
        position_ratios = defaultdict(list)
        
        for item_id in self.item_positions:
            positions = self.item_positions[item_id]
            clicks = self.clicks[item_id]
            
            # Group clicks by position
            pos_clicks = defaultdict(list)
            for pos, click in zip(positions, clicks):
                pos_clicks[pos].append(click)
            
            # Compute CTR for each position
            pos_ctrs = {}
            for pos, click_list in pos_clicks.items():
                ctr = sum(click_list) / len(click_list)
                pos_ctrs[pos] = ctr
            
            # Compare CTRs between position pairs
            for pos1 in pos_ctrs:
                for pos2 in pos_ctrs:
                    if pos1 < pos2:
                        ratio = pos_ctrs[pos1] / pos_ctrs[pos2]
                        position_ratios[(pos1, pos2)].append(ratio)
        
        return position_ratios
    
    def estimate_position_bias(self):
        """Estimate absolute position bias values"""
        ratios = self.compute_ctr_ratios()
        
        # Average ratios for each position pair
        avg_ratios = {
            pos_pair: np.mean(ratio_list)
            for pos_pair, ratio_list in ratios.items()
        }
        
        # Use position 1 as reference (bias = 1.0)
        position_bias = {1: 1.0}
        
        # Propagate bias values using ratios
        for (pos1, pos2), ratio in avg_ratios.items():
            if pos1 in position_bias:
                position_bias[pos2] = position_bias[pos1] / ratio
            elif pos2 in position_bias:
                position_bias[pos1] = position_bias[pos2] * ratio
        
        return position_bias

# Example usage
estimator = PositionBiasEstimator()

# Add historical click data
estimator.add_data("item1", 1, True)
estimator.add_data("item1", 3, False)
estimator.add_data("item2", 2, True)
estimator.add_data("item2", 4, False)
# ... add more data ...

# Estimate position bias
bias = estimator.estimate_position_bias()
print("Position bias factors:", bias)

This implementation follows the approach outlined in the document, where we:

Collect click data for items appearing in different positions

Compare CTRs between position pairs

Use maximum likelihood estimation to solve for position bias values

Normalize the results to get absolute bias factors

How to build a website in 10 minutes

Shan Dou