CSC 400 Team Project: Parallel Jigsaw Puzzle Solving




Team:

The goal of this project is to create a system to assemble jigsaw puzzles, taking advantage of opprtunities for parallel processing to speed up the process. Specifically, you will be provided with a number of images, each containing several pieces of the puzzle photographed against black velvet. Example. The object is to extract the pieces from the images, and figure out how they fit together. The final output will be a single image showing the assembled scene (or as much of it as you program can produce). You may certainly employ non-imagelike intermediate representations to help in the solution of the problem, but you should produce an image that allows your solution to be evaluated visually. Note that at best, your system will only solve the puzzle down to a (4-way) rotation in the image plane, since I do not expect you to be able to determine which way is up (though you are welcome to try).

Currently, for development data, I have taken imagery for two 100-piece puzzles. They are in ~nelson/pics/puzzles/duck_bunny and ~nelson/pics/puzzles/fish. Each directory contains 34 images, each containing 6 (or 4 in a few cases) puzzle pieces on black velvet. There are two images of each set, numbered consecutively, taken at different exposures. The odd-numbered images are brighter than the even numbered ones. The fish puzzle contains a lot of pieces that are of a generally underwater-blue shade, which could make matching more difficult.

As a culmination, there will be, on a class day near the end of the semsester, hold a demonstration challenge, where the system must attempt to solve previously unseen puzzle (of the same general difficulty as the development data) in reasonable time. The team will also need to give a presentation on their effort (on a class after the demonstration)

As usual, the team is to create a single, coherent, written report on their approach to the project, describing in detail what approaches were used, what did not work, any comparative studies done, what considerations went into the design, where the processing time was spent, how the system would be expected to scale, etc. etc.



There are a number of aspects that need to be addressed. The first is the image-processing/vision aspect. The puzzle pieces need to be extracted from the images. I have tried to take images that will make this as easy as possible. The images are all shot with a very narrow field of view, from directly overhead, and at the same scale and exposure. Thus all the colors and boundaries should match within the limits of the camera and image accuracy. Because of the black background, simple thresholding will (almost) allow the puzzle pixels to be separated from the background. Most of the minor problems here can be fixed up by some simple local filtering (e.g. a 3x3 median filter). A connected components routine can be used to associate pixels belonging to individual pieces.

There are a few complexities you will need to deal with. First, the dynamic range of the camera is not all that might be hoped. If white areas are not saturated, some dark areas look pretty black. Just in case this causes problems, I have taken each image twice at different exposures. Odd numbered images (01, 03, ...) are taken at a bright exposure, and following even numbered images (02, 04, ...) are darker exposures of the same scene. You may make use of these (or not) as you wish.

Second, our building shakes. This means that the pictures of the same scene may be mis-aligned by a couple of pixels. It also means that you may sometimes encounter "interlace artifacts" in the images themselves. To understand this, you need to know that standard video cameras take odd and even lines 1/60 of a second apart and merge them into one image. If the camera moves slightly between fields, a "fingering" or "staircasing" effect shows up along object boundaries. This might have an effect on some matching processes you might use. The effect can be ameliorated in various ways. Median filtering (or other smoothing) helps to some extent. Following this by a dilation and an erosion of the thresholded binary image would probably produce pretty clean regions for segmentation. If you want to get more elaborate the image could be separated into odd and even fields, and a shift correction computed and applied. Or you might even work with a 1/2 resolution image obtained by subsampling.

Third, you will need to rotate the pieces as well as translate them in order to fit them together. There is code in my libaries for rotating images, but there are pixel-level accuracy issues at the boundaries. Also, although the aspect ratio of the pixels is approximately 1 to 1, I do not guarantee it is exactly 1 to 1 (i.e. that the pixels are exactly square.) You may want to carefully measure the real aspect ratio in the Canon cameras from which the images were taken. (They were taken with one of the ceiling cams in the software lab) You will have to take some pics of either an exactly square object, or a rotated object. Knowing this precisely might be the crucial difference in whether you can do shape matching or not.

I have written library code for all the operations mentioned above, including connected components, and many more, so you probably do not need to write or find image processing code. You just need to figure out how to link and use the ipp libraries. There is a program "image_calc" in ~nelson/bin/PCLinux that will let you experiment with all sorts of image processing operations on the images, to see what gives good extraction. You can look at the source in ~nelson/programs/src/integrated/bin to see how to link and use the routines once you have figured out what needs to be done.



The second aspect is the representation and matching problem. You will need to represent the pieces in a way that permits matching operations to be performed efficiently on the color and/or pattern and/or shape. It might be advantageous to identify border pieces. It might also be advantageous to obtain a boundary description broken at "corners" (where 4 pieces come together) for each piece. These points seem like they would be identifiable on the pieces we have, and a boundary segment represented in normalized form could permit quick match checks (e.g. on length) and fast shape-matching - since the endpoints are known, no further rotation would need to be done.


A third inportant aspect involves parallel processing and efficient algorithms for the overall assembly. Since the goal is to solve the puzzle as rapidly as possible, any use of parallel resources, and any heuristics to reduce the number of matches that need to be considered would owrk towards that goal. Human-like strategies of working on groups of pieces of similar color might be helpful. Of course some such strategies might not interact well with division over a moderate (16+) number of processors, so there are tradeoffs to be considered. There is also the issue of how much time is consumed in basic image processing versus matching. Image processing, especially of multiple images, is rather trivially parallelizable. If most of the work is in the production of the piece representations, then parallelizing the matching might be a moot point.


Finally there is the issue of presentation of the processing. It would be nice to have a live display illustrating the progress of the system as it progresses. Some graphical output showing sections as they are constructed, would be interesting. How to do so effectively when many machines are working on the problem at once is an interesting problem


Getting started

As a step towards handling the image-processing aspect of the project, you should you should become familiar with these machine vision resources. Once you have done this, you could check out the program "image_calc" in ~nelson/programs/src/integrated/bin. this allows you to experiment with the routines in my ipp image processing library, which probably contains most of the image-processing capability that you will need. The source code will show you how the routines are used, and the make file will show you how to link the libraries.

Back to CSC 400 main page