MaiZaic: a robust end-to-end pipeline for mosaicking freely flown aerial video of agricultural fields

ABSTRACT Unmanned aerial vehicles (UAVs) are increasingly used for high throughput phenotyping. In principle, freely flown vehicles would permit real-time flexibility in identifying and scouting regions of interest. Mosaicking multiple images provides a high resolution global image and consumer-grade UAVs offer low cost, ease of flying, and excellent RGB cameras. The vehicles’ inaccurate telemetry complicates estimating the homographies between pairs of frames, the standard mosaicking approach. Moreover, errors accumulate during computation, distorting later portions of the mosaic. Finally, crop fields are particularly challenging to mosaic because their planting is so regular and the plants are so similar, eliminating distinctive features that could guide mosaicking. We propose MaiZaic, an end-to-end pipeline that dynamically samples video frames using optical flow, automates camera and gimbal calibration, estimates homographies with an unsupervised convolutional neural network, detects shots among frames, and generates mini-mosaics. Together, these techniques significantly reduce errors in the output mosaics. Our deep learning model is trained on a comprehensive video dataset comprising different flight trajectories, maize lines, growth stages, and augmented illumination data. MaiZaic is more accurate and faster than ASIFT and more robust than our earlier CorNet and CorNetv2. We demonstrate MaiZaic’s effectiveness in generating accurate mosaics of imagery captured by freely-flown UAVs and explore its generalizability. Core ideas MaiZaic is an end-to-end pipeline to mosaic freely flown agricultural imagery captured with consumer-grade UAVs. MaiZaic introduces novel algorithms that efficiently choose video frames, calibrate, and mosaic the imagery. Our unsupervised deep homography estimator, CorNetv3, is 14 times faster and 8.59% more accurare than ASIFT. MaiZaic generalizes well and mosaicks maize at different growth stages, objects, trajectories, cameras, and pilots. The mini-mosaicking algorithm improves mosaic accuracy by interrupting error accumulation. Competing Interest Statement The authors have declared no competing interest. Footnotes - APE - average presentage error - ASIFT - affine scale-invariant feature transform - fps - frames per second - LF-NET - local feature network - RGB - red green blue color channels - RMSE - root mean squared error - SURF - speeded up robust features - UAV - unmanned aerial vehicle - UMCD - UAV mosaicking and change detection dataset - VGG8 - visual geometry group (deep neural network of eight layers).