Content Based Retrieval in Multimedia Databases Gianluigi Ciocca, Raimondo Schettini DISCo - Dipartimento di Informatica Sistemistica e Comunicazione - May 2003 Content-Based Retrieval Text-Based Retrieval Automatic annotation of image content is not achieved yet. Manual annotation is an arduous task. There are unlimited ways of annotating a picture. Some visual contents are difficult to describe.
How to describe visual content? Features definition Color distribution Degree of difficulty Texture pattern Shape Objects Partial semantic Real content How to describe visual content? Features representation Color Color Histogram Color Coherence Vectors Correlograms Spatial Chromatic Histogram... Texture Wavelets Grey-Levels Distribution Gabor Filters Tamura descriptors... Shape Deformable Template Invariant Moments Wavelets Description of boundaries... Other Discrete Cosine Transform Regions Layout/Relationship...
Content-based similarity retrieval The similarity problem The term similarity has different meaning for different people. Even the same person uses different similarity measures in different situations. Blue Tint Content-based similarity retrieval Goals A retrieval system should be able to use low level and high level descriptions of visual contents automatically extracted. cover a wide range of users requirements. adaptively select the retrieval strategy that is most suited for the users needs.
The Quicklook 2 system Overview Web-Demo at http://quicklook.itc.cnr.it The Quicklook 2 system Queries available Pictorial queries Provide an example of ready-made image Sketch an image with the available tool Browse the database to find one or more relevant images with which to begin Textual queries Filter the database with textual keywords Search images by textual annotation similarity Search images by text similarity Advanced queries Combination of one or more of the above queries Relevance feedback
W h Relevance Feedback Human-in-the-Loop Query Reformulation Features Weights Relevant Objects Positive Examples Negative Examples Shape Color Agreement between positive examples Agreement between positive and negative examples Query + * W h Weight Estimation W h W + * h - W h pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie pianta (" pianta con con lunghi steli sinu pianta con lunghi lunghi steli stelisinuosi pianta e larghe con lunghi foglie steli sinuosi ("albero e della larghe foglie ("albero della vita"); sul collo, ampia albero fascia della vita"); con motivo sul a collo, ampia treccia; fascia tipologia con ("albero della vita"); motivo a treccia; tipologia ("albero pianta con della lunghi steli vita"); sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia ("albero della vita"); Pictorial Textual SIMILARITY EVALUATION Ranked Results pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame. osi e larghe foglie ("albe pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame. ro della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame.v pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); Database Relevance Feedback Example
Results Presentation Virtual Museum + = & The result can be presented in a navigable 3D-like virtual environment Different environments Different frames to choose from Image s canvas sizeable Image are shown in linked rooms Quicklook n Other issues Long-Term Learning Database Organization Indexing Structures Meta-data Exploitation Hidden annotations Integration of other media Video Data 3D Objects Data
Database Organization Clustering Can be used to: Improve retrieval efficiency Provide an overview of database content Hidden Annotation Segmentation based on semantic content THESAURUS DEFINITION... Snow Water Ground Sky Vegetation MATCHING /DECISION RULE Vegetation, + Water, Sky
Hidden Annotation High level categorization IMAGE CLASSIFICATION... Close-up Text Graphic Outdoor Indoor Video Indexing Segmentation: shots detection Cuts / Fades Detection Key-Frames
Video Indexing Abstraction: summarization Structure Levels... Key Frame Levels... 3D Objects Challenges 3D Pre-processing Features to use in indexing 3D objects 2D Features 3D Features Invariants Interface to use to submit a query 2D Projections View 3D Examples Matching 2D and 3D data
Multimedia Databases Applications Art gallery, museum. Photographic archives (journalism). Textile databases. Trademarks databases. E-commerce catalogues. Web-searches. Medical image databases. Facial databases (security, law enforcement). Engineering design databases. Geographic databases. Satellite image databases.
QuickLook 2 Scheme