Weighting, Scaling, and Objective Functions in Parameter Estimation

08 Feb, 2024

Parameter Estimation: When to Scale Data and Why Architecture Matters

"Do you always need data transformers when doing parameter estimation?"

Good question. I've been down this rabbit hole before, and the answer reveals some deeper insights about objective function design and optimization signal quality.

The Core Insight: Same Score, Different Mechanics

Here's the key thing: the score (RMSE, R², etc.) looks the same across different model types, but the underlying mechanics require completely different implementations.

You're trying to fit parameters by minimizing an objective function. The optimizer sees the same interface - feed in parameters, get back a loss value.

But most of the time, you're working with one of two setups:

• Surrogate Models(neural nets, random forests, etc.) - trained on historical data
• Simulators / Forward Models (Monte Carlo, differential equations, etc.) - run forward simulations

Each needs different data handling approaches with custom concrete classes for each backend:

# Same interface, different mechanics
def optimize_params():
    if using_surrogate:
        obj_func = SurrogateObjective(model, scaler, targets)
    else:
        obj_func = SimulatorObjective(simulator, targets, bounds)
    
    # Optimizer doesn't care about the difference
    result = minimize(obj_func, initial_params)
    return result

Why Custom Concrete Classes Matter

The objective function quantifies your optimization goal and returns a loss value. The score is consistent, but the mechanics differ completely:

Surrogate models: Call a trained model (neural network, random forest) to predict outcomes Simulators: Call a physics-based simulator that solves differential equations or runs Monte Carlo

class ObjectiveFunction:
    def __call__(self, params):
        raise NotImplementedError

class SurrogateObjective(ObjectiveFunction):
    def __call__(self, params):
        # Mechanics: predict using trained model
        X_scaled = self.scaler.transform(params.reshape(1, -1))
        y_pred = self.model.predict(X_scaled)
        return mse(y_pred, self.targets)

class SimulatorObjective(ObjectiveFunction):
    def __call__(self, params):
        # Mechanics: run physics simulation
        y_sim = self.simulator.run(params)
        weights = 1.0 / (self.bounds['upper'] - self.bounds['lower'])
        return np.sum(weights * (y_sim - self.targets)**2)

This strategy pattern lets you swap backends cleanly while maintaining the same optimization interface.

Data Transformers: When and Why

Surrogate Models: Always Scale

Neural networks are finicky about input scales. Without proper scaling:

• Gradients explode or vanish
• Training becomes wildly unstable
• Convergence takes forever (if it happens at all)

# Required for stable training and prediction
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

Simulators: Skip Input Scaling, Focus on Output Weighting

Here's where it diverges. Simulators don't train on data - they solve equations. Therefore, No gradient flow issues to worry about.

But you still need to handle output scaling, and this is where the real optimization insight comes in. Say you're optimizing a pineapple farm operation with a simulator that outputs:

• Annual production (tons): 100-2000
• Water usage (gallons): 50,000-500,000
• Labor hours: 1,000-10,000
• Fertilizer cost ($): 5,000-50,000

Without proper weighting, your optimizer will obsess over water usage (biggest numbers) and ignore everything else. But here's the key insight: weighting the Y values (outputs) in your loss function dramatically improves the optimization signal.

def objective(params):
    results = simulator.run(params)
    targets = np.array([target_production, target_water, target_labor, target_cost])
    
    # Dynamic weighting based on user bounds
    ranges = upper_bounds - lower_bounds
    weights = 1.0 / ranges
    
    # Weighted loss gives much better optimization signal
    weighted_errors = weights * (results - targets)**2
    return np.sum(weighted_errors)

The weighting transforms a noisy optimization landscape into one with "equal signal". Each output now contributes meaningfully to parameter updates.

Implementation Flexibility

You can still use data transformer classes for simulators if you want consistency across your codebase. Just swap the scaler:

# For surrogates: normalize around training data
surrogate_scaler = StandardScaler()

# For simulators: normalize around user bounds  
simulator_scaler = MinMaxScaler()

The key requirement: users must set reasonable bounds in their configuration for dynamic weighting to work properly.

The Bottom Line

The pattern is: surrogates need input scaling for stability, simulators need output weighting for optimization signal quality. Both solve fundamental problems, but for different reasons.

Key takeaway: While the score looks the same, the mechanics require custom concrete classes. Use strategy patterns to swap backends cleanly, and focus on the right scaling approach for each model type.

Same math, different implementation.

I'd be curious to hear about other edge cases where the scaling strategy matters.