Drawing visual query images : use, users and usability of query by drawing interfaces for content based image retrieval systems
MetadataShow full item record
Query by Drawing (QBD) is an approach to Content Based Image Retrieval (CBIR) systems where users express their image needs by drawing an image representative of the images they wish to retrieve. CBIR is based on comparison of the query image and images in an image collection. This approach to image retrieval has been an active field of research for over a decade, but despite this, few end-user applications are available. An often quoted reason for this is that CBIR systems are capable of retrieving images based on low-level image structures such as colours, textures and shapes, while users are primarily interested in the semantic content of the image. The role of the user in image retrieval systems is a relatively unexplored area, and little empirical data has been collected on the expectations, needs and behaviour of these users. Literature in the field suggests that image retrieval based on low-level image structures is not very important for users, and consequently current CBIR systems may not be very useful for end-users. The main motivation behind this research project has been to collect and analyze empirical data on the use and users of QBD CBIR systems. Four major goals were defined for the project: * Understand how users behave when using QBD CBIR systems * Understand how users experience using QBD CBIR systems * Determine if QBD CBIR systems can be a useful tool for end users despite the current challenges related to these systems * Identify potential improvements that can be made to QBD CBIR Systems 30 respondents were asked to perform a set of image retrieval tasks in two different QBD CBIR systems. The respondents represented two different groups of users. The first group represented “non-professional” users, and consisted of 17 information science students. The second group represented “professional” users, and consisted of 14 respondents with a background in visual arts, visual design and industrial design. The two QBD CBIR systems represented two different approaches to the QBD CBIR process. They were selected as representative systems based on an analysis of 59 past and current CBIR systems. The respondents performed a total of 414 queries. The queries and the query sessions were analyzed using three different approaches: * A protocol analysis of the QBD query process based on observation and interface videos * A grounded-theory based approach based on questionnaires, structured interviews, the interface videos and observation * An analysis of the query images drawn by the respondents based on a custom framework created for QBD query images The evaluation indicated that the respondents preferred to keep the query drawings as simple as possible. They wanted to quickly sketch the query images using freehand drawing, and to limit the amount of details to the level they felt that they needed in order to express their image requests. They often created these drawings as visual keywords, i.e. very simple representations of the objects they wanted to retrieve images of. The “non-professional” respondents found the drawing process difficult and challenging. They were frustrated that they were not able to draw the objects in a realistic manner, and felt that they would not be able to fully benefit from the QBD CBIR approach because of this. These respondents also felt that the time required creating QBD CBIR queries was a major obstacle, particularly when compared to creating text based queries. The “professional” respondents were positive towards the QBD CBIR process, and did not experience similar problems related to the drawing process, but they were not willing to spend time drawing realistic query images. The “professional” respondents believed that they would use QBD CBIR systems on a regular basis if such systems were available and could be used on large scale image collections. They described several realistic scenarios where they would have benefited from using QBD CBIR over normal text based retrieval systems. The “non-professional” users were not so sure that they would use these systems for anything other than entertainment. Based on the feedback from the respondents and the evaluation of the QBD CBIR process, a set of prioritized improvements to QBD CBIR systems have been identified. A four-step process for leveraging QBD CBIR systems from research prototypes to full-scale systems that can be of real benefit for real-world users is suggested. These results indicate that the role of QBD CBIR systems may have been understated in literature. Even with the current challenges facing these systems, the feedback from the respondents in this study indicates that, given some changes, users may find QBD CBIR systems a very useful tool, particularly when combined with text based queries.