Reflections on the Training Pipeline Architecture

16 Jul, 2025

In reviewing the training pipeline's implementation, a distinct hybrid pattern has emerged.

At its core is a Domain-Driven Design (DDD) approach, but with a strong functional programming influence.
This is all contained within a more conventional object-oriented shell, which feels like a pragmatic choice for integration purposes.

The DDD Foundation: Modeling the Domain

The design leans heavily on DDD concepts. I've found that using Value Objects (VOResult_ModelTest, VODataset) for immutable data transfer and Entities (ENTTrainingJob) for objects with a distinct lifecycle brings a lot of clarity. The SurrogateTrainerPort acts as a classic Domain Service, orchestrating the core logic. This adherence to a Ubiquitous Language (surrogate, training, metrics) keeps the code aligned with the problem space.

# Value Objects - Immutable data containers
VOResult_ModelTest(
    test_metrics=VOLog_ModelMetrics(...),
    feature_importance={...},
    prediction_summary={...}
)

# Entity - Has identity and lifecycle
ENTTrainingJob(
    uid=uuid4(),
    status="COMPLETE",
    results=VOResult_ModelTest(...)
)

Inner Core: Functional Purity

Internally, the pattern follows a "Functional Core, Imperative Shell" philosophy. The core logic—data transformation, metric calculation—is implemented as a collection of functions that operate on immutable data. This is evident in how _compute_model_metrics and _predict don't modify state but rather produce new data. It promotes composability and makes dependencies explicit. The shell then handles the imperative aspects like I/O and stateful interactions (e.g., model training itself, logging).

# Functional composition pattern
def train(...) -> Tuple[ModelT, ENTTrainingJob]:
    # Pure data transformations
    test_metrics = self._compute_model_metrics(model, test_x, test_y, training_data)
    predictions = self._predict(model, test_x)
    feature_importance = self._compute_model_feature_importance(...)
    
    # Immutable result construction
    return model, ENTTrainingJob(results=VOResult_ModelTest(...))

Workflow as a Command/Template

The train method itself has taken the shape of a Command pattern. It encapsulates the entire training workflow into a single, callable unit. The behavior is parameterized through configuration objects (VOMetadata_General, VOConfig_Training), and it aggregates all outputs into a single, structured result tuple. This isolates the execution logic cleanly.

# Command pattern structure
def train(
    metadata: VOMetadata_General,      # Command parameters
    training_data: VODataset,          # Command input
    training_config: VOConfig_Training, # Command configuration
    # ...
) -> Tuple[ModelT, ENTTrainingJob]:     # Command result

Reflections on the Design Philosophy

Looking back, a few core philosophies seem to have guided the design:

Data-Centricity: The architecture is driven by its data structures. This creates clear contracts, though it introduces some verbosity compared to more streamlined functional approaches.
Declarative Configuration: Behavior is defined through configuration objects. This makes the system configurable without code changes, but the config objects themselves can become complex.
Separation of Concerns: There's a clean split between computation (pure functions), orchestration (the service layer), and persistence (handled elsewhere).
Explicit State Management: State changes are explicit and tracked within the ENTTrainingJob entity. This helps with auditing and debugging, but adds complexity over a purely stateless design.

Considering the Alternatives

It's useful to think about why this hybrid emerged instead of a purer pattern.

vs. Pure Functional Programming: A pure FP approach might have used a pipeline of composed functions. The current design is more explicit, breaking down the steps within the service method. This feels more readable for debugging.
```
# Pure FP might look like:
train_result = pipe(
    training_data,
    split_data,
    train_model,
    evaluate_model
)
```
vs. Traditional Object-Oriented Design: A classic OOP pattern would likely involve a stateful TrainingSession object. The chosen stateless service approach feels cleaner, avoiding side effects from method calls modifying internal object state.
```
# Pure OOP might look like:
class TrainingSession:
    def execute(self):
        self.model = self._train()
        self.results = self._evaluate()
```
vs. Event-Driven Architecture: An event-driven model would be asynchronous, with events like TrainingStarted or ModelTrained. The current synchronous workflow is simpler for this use case, where the entire process is a single unit of work.
```
# Event-driven might look like:
training_started_event = TrainingStarted(job_id, config)
model_trained_event = ModelTrained(job_id, model)
```

Observed Strengths

The current approach has several noticeable benefits:

Testability: The functional core is straightforward to unit test.
Debuggability: With immutable data and explicit data flow, tracing a problem is simpler.
Composability: The components can be recombined for different workflows.
Type Safety: The use of typed Value Objects helps catch errors early.
Domain Clarity: The DDD influence keeps the code's concepts close to the business domain.

Acknowledged Trade-offs

The pattern isn't without its costs:

Verbosity: Defining explicit Value Objects and Entities requires more code than other approaches.
Memory Usage: Immutability can lead to increased memory allocations. This is something to monitor.
Learning Curve: The combination of DDD, functional, and OO paradigms requires a developer to be familiar with all three.
Serialization: The rich domain objects require careful handling for serialization and persistence.

Final Thoughts

For now, This appears to be a pragmatic hybrid pattern. It uses DDD for clear domain modeling, functional programming for predictable computation, and the Command pattern for orchestrating the workflow.