CSC 249/449 Computer Vision: Counting Colored Objects Assignment

The goal of this assignment is to count the number of objects of different types in an image, where the objects of different types have different characteristic colors. In this case, the objects are apples, oranges, and bananas, which (supposedly) have different colors (red, orange, and yellow) There are three test images on which you are to run your program in ~nelson/pics/fruit: mixed_fruit1.tiff , mixed_fruit2.tiff , and mixed_fruit3.tiff . There is also a very tough image fruit_tray.tiff , on which you can test the limits of your algorithm. In this picture, the goal is a little ambiguous, as there are some pieces of fruit of which only a small fragment is visible. By my count there are 10 - 11 oranges half or more visible plus 3 - 5 fragments, 8 apples plus 2 fragments, and 5 bananas plus one distinct fragment. Don't use the gifs here for testing - they are dithered pictures with 8 bits of color information, which will make your job even harder. There are also pictures of apples, oranges and bananas on a black background taken in daylight and fluorescent light, which might assist you in designing color models for the different categories.

The ipp utility library has routines input_image() and output_image() for loading and writing color iff and tiff images into an rgb core format, and the ipp graphics library routine display_image() will display an in-core (rgb) color image in x-windows. Note that the routines input_color_image(), output_color_image(), and display_color_image() refern to an obsolete color_image structure that has been superseded by augmenting the standard image structure with multi-band capability. See the utility Readme file for more information on the image format. Other than that, there is little software in the ipp libraries specifically for color images, so you will have to grow your own, or find some other routines.

The program xshow in my bin/(Solaris | PCLinux) will display a color iff or tiff image on an xterm and xv will also allow you to display color tiff images. the program ktv_get_color_image allows you to take a color image from the ktv digitizer (should you want to make additional image on your own), and the program bt_get_image with the -color option will get a color image from the BT848 digitizers on the Linux boxes. The bt_showlive program can also be used to take pictures by hitting the right mouse button inside the live image. This has the advantage that you can see what you are taking a picture of. Resolution and color mode are whatever bt_showlive is given.

There is a simple segmentation routine (con_comps) in the feature library that will group pixels that all have the same value, and might be of use to you if you manage to produce color labels on a pixel by pixel basis. There is also a gray-level region segmentation routine (grow_regions) that attemps a segmentation of a gray-level image. It won't do you much good directly, but you might find it useful as a model for a color segmentation scheme if you decide to take that route. Source for these routines is in ~nelson/programs/src/ipp/feature/lib/region.

You are permitted to use other software for low-level processing of color images (if you can find it), and for performing linear algebra and statistical operations (if you want them). You are NOT allowed to use a previously written color segmentation program, or higher level recognition software. This is what you are supposed to do yourself. (If you rewrite my gray level segmenter, you will have to understand it first, which is likely non-trivial, so that's ok).

Hand in the test images, separate images showing the fruit you found, and also a printed list giving the identity, location, and approximate size of all the objects you found. Also prepare a detailed writeup describing your algorithm (not the code), the difficulties you encoutered and how you resolved them, and evaluating the performance of your algorithm (how well did it do, what were the weaknesses, etc.).

Hints:

• There are a least two possible approaches. You can attempt to label individual pixels as apple, orange, or banana, and use a simple grouping operation to obtain regions with the same label. Alternatively, you could try a more-sophisticated segmentation, and then try to identify whole regions. In either case, you will probably have to filter out small spurious regions, and a merge operation could conceivably help as well.
  
  
• You can model the object classes either in a 3D color space, or in a 2D chromaticity space (without intensity). A difficulty with the 3D case is that the training data you have probably does not span all the different intensities in the test images (particularly for the apples) so you have to make sure this is accounted for somehow. A problem with the 2D case is that it does not allow you to use the darkness of the apples as evidence (and the chromaticity may not be very good for them). Past experience seem to indicate that an hsv space is easier to get good results in than the raw rgb space, though I expect either could be used with an appropriate color model.
  
  
• All the pictures, but particularly the training images have (whitish) highlights in them. You should probably try to eliminate these when constructing your color models (doing it by hand is fair, though time consuming). Otherwise the white regions will tend to make ellipsoidal (and other) representations of the color range too large to be useful.
  
  

Back to vision course main page