Multimodal Models

In addition to deep_lvpm.model.StructuralModel, the package includes alternative multi-view and multimodal representation-learning models in deep_lvpm.multi_model:

  • deep_lvpm.multi_model.CLIP

  • deep_lvpm.multi_model.DGCCA

  • deep_lvpm.multi_model.VICReg

  • deep_lvpm.multi_model.LeJEPA

All four classes subclass torch.nn.Module and follow the same high-level pattern as StructuralModel:

  • provide one torch.nn.Module measurement model per data view

  • provide one regularizer entry per view

  • choose ndims for the shared embedding width

  • call compile with PyTorch optimizer objects

  • train with fit and evaluate/predict with evaluate and predict

Example

import torch
from deep_lvpm.multi_model import CLIP

model = CLIP(
    model_list=view_models,
    regularizer_list=[None for _ in view_models],
    ndims=512,
)

optimizers = [
    torch.optim.Adam(view_model.parameters(), lr=1e-4)
    for view_model in model.model_list
]
model.compile(optimizers)
history = model.fit(train_data, epochs=10)
metrics = model.evaluate(test_data)
embeddings = model.predict(test_data)

The main difference between the classes is the loss function and training objective, not the user-facing training loop.

Metrics

CLIP reports clip_loss.

VICReg reports total_loss, cross_metric, mse_loss, and redundancy.

DGCCA reports total_loss, cross_metric, gcca_loss, and redundancy.

LeJEPA reports total_loss, cross_metric, pred_loss, sigreg_loss, and redundancy.