Latte Art CV Pipeline

Computer Vision for Scoring Latte Art at Artly

Artly's robotic barista pours latte art, and this pipeline scores it. An overhead fisheye camera captures each drink. The pipeline removes the lens distortion, detects and segments the cup, warps the latte art onto a canonical circle, and scores it on art, aesthetics, bubbles, contrast, fill, and symmetry.

The robotic barista pouring latte art.
Raw fisheye inputs from overhead camera
Raw fisheye images from the overhead camera.
Defisheye and crop example 1
Defisheye + crop (sample 1).
Defisheye and crop example 2
Defisheye + crop (sample 2).
Defisheye and crop example 3
Defisheye + crop (sample 3).
Defisheye and crop example 4
Defisheye + crop (sample 4).
Cup detection heatmap
Cup detection.
Cup detection refinement
Post-processing.
Cup detection outputs
Cup segmentation.
Transformed latte art to canonical circle
Perspective warp to circle.
Segmented latte art mask
Scoring visualization example 1
Scoring (sample 1).
Scoring visualization example 2
Scoring (sample 2).
Scoring visualization example 3
Scoring (sample 3).
Artly's robotic barista
Artly's robotic barista.
Artly Demo
Customers at Muji Hudson Yards.

← Back to work