Image segmentation
The process of partitioning an image into individual pixels or regions based on semantic categories, effectively creating a pixel-perfect mask for every distinct object present.
What it is
A machine learning model assigns a category label to every pixel in a source image, outputting a bitmask rather than a single classification tag. These models typically use an encoder-decoder architecture where the network first captures spatial features and then reconstructs a pixel-level map of the same resolution as the input. Inference costs are significantly higher than classification because the model must compute a high-resolution tensor output, often resulting in larger payloads and increased GPU memory requirements compared to simple label tagging.
Why it matters
You need this when the position, boundary, or geometry of an object matters for your downstream logic, such as blurring a user's background or calculating the exact area of a physical object. If you only need to know that an image contains a car, use standard classification to save on latency and cost. Ignoring the difference between tagging and segmentation leads to deploying models that are orders of magnitude slower than necessary for the required feature set.
In practice
You will typically interact with this via APIs or libraries that return a separate mask layer or a JSON array of coordinates representing the polygon of each detected object. Watch for the output resolution parameter in your inference request, as requesting a mask at the full input dimensions will drastically increase your latency and memory overhead per request.
The tradeoff
The main tradeoff is precision versus latency, as segmenting at high resolution provides accurate boundaries but drastically increases the time per request and the computational resources required.