MAMMA is a novel markerless motion-capture pipeline that recovers SMPL-X parameters from multi-view videos for two-person interaction. Our approach consist of MammaNet, a novel transformer-based dense landmark estimator that predicts 2D surface landmarks, uncertainty, visibility, and contact probabilities. Our network learns per-landmark embeddings which help it handle complex and out-of-distribution poses. To train this network, we introduce MammaSyn, a large-scale synthetic dataset featuring complex multi-person interactions and ground-truth dense landmarks. To match the predicted dense landmarks across views, we compute the symmetric epipolar distance between them and find the optimal matches. Finally, MAMMA uses the predicted and matched 2D landmarks to optimise the SMPL-X paramters. MAMMA leverages the predicted uncertainty and contact information, allowing it to avoid pose priors and complex loss terms.

Our novel dense 2D landmark estimator, MammaNet, extracts image features using ViT-Base and includes an additional CNN to process the mask. It decodes 512 surface landmarks using a Transformer decoder.

Unlike other methods, we learn 512 embeddings (landmark queries), one for each landmark. The network predicts pixel coordinates $\mu_i$, uncertainty $\sigma_i$, visibility $p_i$, and our novel contact probabilities (person-person $pc_i$ and floor contact $fl_i$).

Our learned landmark queries allow MammaNet to generalize to extreme, out-of-distribution poses (left). Additionally, the network produces accurate results even with only partial mask coverage (right).

We introduce MammaSyn, a large-scale synthetic dataset with 2.5M crops, extending BEDLAM with a multi-view setup and higher resolution rendering. It is designed to cover extreme poses, high-fidelity hands, and complex multi-person interactions. Our MammaNet model is trained on MammaSyn and BEDLAM data.

MammaSyn consists of three sub-datasets with motions from different sources:

• MammaSyn-S (Single): BEDLAM and MOYO.
• MammaSyn-I (Interactions): Hi4D, Harmony4D, and Inter-X + our captured interactions.
• MammaSyn-H (Hands): SignAvatars and Interhand2.6M.

Accurate subject identification is essential within each camera view and across multiple views. We use SAM2 segmentation propagation to track persons across the sequence. All detected people across all views are compared pairwise to establish cross-view correspondences.

Given two sets of landmarks $\mathbf{x}_{a},\mathbf{x}_{b}$ from two views, their geometric affinity is defined based on the symmetric epipolar distance $D_g$:

$$ D_g = \frac{1}{2FN} \sum_{i=1}^{FN} \Big( d(\mathbf{x}^{i}_{b},\,\mathbf{F}_{ba}\mathbf{x}^{i}_{a}) + d(\mathbf{x}^{i}_{a},\,\mathbf{F}_{ab}\mathbf{x}^{i}_{b}) \Big) $$

where $\mathbf{F}_{ba}$ is the fundamental matrix between two views and $d(\mathbf{x},\mathbf{l})$ denotes the point–line distance. Optimal one-to-one matches are obtained by solving a Hungarian assignment, and high-affinity matches are linked to form a cycle-consistent correspondence graph.

We fit the SMPL-X body model to our predicted landmarks using a multi-stage optimization process. Unlike other methods, we don't rely on regression initialization; our dense landmarks provide sufficient geometric constraints.

1. Reprojection Error
First, we solve for global translation and rotation by minimizing the reprojection error between our predicted landmarks and the projected body model vertices:

$$ E_{\text{ldmks}} = \sum \rho \left( \frac{\lVert \boldsymbol{\mu} - \Pi(\mathbf{V}, \mathbf{Q}) \rVert}{\sigma} \right) p $$

where $\boldsymbol{\mu}$ are the predicted 2D landmarks, $\mathbf{V}$ are the body vertices, $\mathbf{Q}$ are camera parameters, and $\sigma$ represents uncertainty. We use a robust Geman-McClure function $\rho$ to handle outliers.

2. Pose and Shape
Next, we optimize the full body pose and shape, adding regularization terms to ensure realistic human deformations. We also refine our uncertainty estimates based on the reprojection error, down-weighting predictions that don't align well with the 3D model.

3. Contact Optimization
Finally, we explicitly model person-person and person-ground contact. We use an energy term that prevents interpenetration while encouraging contact where predicted:

$$ E_{\text{cont}} = E_{\text{repulsion}} + E_{\text{attraction}} $$

This ensures that interactions like hugging or dancing are reconstructed with physically plausible contact.

To obtain a robust signal, we average the contact probabilities predicted from each view. The visualization below shows this averaged contact prediction mapped onto the body surface.

For two-person interaction, we evaluate on Harmony4D and CHI3D. MAMMA is also capable to handle single-person motion capture, which is a subset of the problem. For single-person evaluation, we use RICH and MOYO. We also introduce two high-quality datasets obtained in our capture lab: MammaEval-(S)ingles and MammaEval-(D)ance. All datasets were captured with a marker-based system (Vicon) synchronized with a multiview RGB system.

We evaluate accuracy using 37 "held-out" markers, extra markers placed on subjects but not used by the MoSh++ marker-based fitting. We compare the position of these actual markers against virtual markers on the SMPL-X meshes recovered by MoSh++ and MAMMA. Our method achieves comparable accuracy, with less than 1mm difference in error.

We capture a new dataset using MAMMA covering various activities and interactions. You can find it here: MAMMA Dataset.

Beyond studio camera systems, MAMMA can also recover human motion from just four iPhones across both indoor and unconstrained (“in-the-wild”) environments.