Given six or more pairs of corresponding points on two calibrated images, existing schemes for estimating the essential matrix (EsM) use some manifold representation to tackle the non-convex constraints of the problem. To the best of our knowledge, no attempts were made to use the more straightforward approach of integrating the EsM constraint functions directly into the optimization using Adaptive Penalty Formulations (APFs). One possible reason is that the constraints characterizing the EsM are nonlinearly dependent and their number exceeds the number of free parameters in the optimization variable.

This paper presents an iterative optimization scheme based on penalty methods that integrates the EsM constraints into the optimization without the use of manifold-based techniques and differential geometry tools. The scheme can be used with algebraic, geometric, and/or robust cost functions. Experimental validations using synthetic and real data show that the proposed scheme outperforms manifold-based algorithms with either global or local parametrizations.