Schaer Roger, Müller Henning

Abstract

This paper discusses the need for retrieval systems and interfaces in today's rapidly growing medical imaging datasets. It then presents Shambala, a web-based front-end for an image retrieval system called ParaDISE (Parallel Distributed Image Search Engine). The Shambala application aims to provide a user-friendly interface for both novice and expert users and takes advantage of modern web technologies to make the user experience interactive and engaging. The interface can be extended to other retrieval systems and is in the process of being integrated with alternative human-computer interaction methods (such as motion sensors or speech recognition) to make it suitable for use in a sterile environment.

Introduction

A vast amount of imaging data is produced daily in hospitals [1]. These images can contain valuable information for aiding medical professionals in diagnosis, treatment planning and research. Therefore, there is a need to enable retrieval from these enormous image collections to better utilise available knowledge.

In past years, several efforts have been made in order to provide textual retrieval as well as CBIR (Content-Based Image Retrieval) systems and search interfaces to users in the medical domain, such as the Khresmoi project [2], a multilingual, multimodal search system for biomedical information and documents. Other examples include GoldMiner [3], which provides filtering based on imaging modality, RadMiner [4], a search tool aimed towards radiologists for text and semantic information, OpenI [5], which allows search based on text and on image examples, as well as Yottalook [6], which supports searching across multiple types of sources.

This paper presents a novel, web-based search interface called Shambala [7], which aims to be easy to use, interactive and modern.

The remainder of the paper is organised as follows. The
methods section describes the technologies used in the development of the Shambala interface, as well as the underlying retrieval system, called ParaDISE (Parallel Distributed Image Search Engine), described in [8]. The
results section presents the features of the Shambala interface and the last section concludes the paper and lists some future development ideas.

Methods

The core concept and the various technologies used in the development of the interface are presented in this section, as well as a description of the retrieval system (ParaDISE) that is utilised by Shambala.

Concept and technologies

Shambala started out as a simple sandbox interface for easy testing of the ParaDISE system, but it evolved over time to provide more features and ease of use. It is built on the following principles:

Easy – The application should be straight-forward to use and not overwhelm the user with many options and parameters. It should also be easy to deploy and not require the installation of new software such as an application server, a database management system, PHP (PHP: Hypertext Preprocessor), etc.

Interactive – The application should provide the user with features such as drag and drop to make searching more engaging and similar to desktop interfaces. It should also react quickly to user input and refresh search results continuously and transparently as the user refines his query.

Modern – The application should integrate modern web technologies, like the HTML5 (HyperText Markup Language 5) File & History APIs (Application Programming Interfaces), as well as HTML5's local storage feature.

With these principles in mind, Shambala was developed as a client-side-only application, based entirely on HTML5 and Javascript. All interactions with the ParaDISE system (which can be hosted on a completely different server, as it is totally independent) use AJAX (Asynchronous JavaScript And XML) calls to the Global ParaDISE web service, shown in figure 2.

ParaDISE backend

The ParaDISE system, used by the Shambala interface, allows indexing and retrieving images using visual features as well as text. The design of the system was based on three main aspects:

Modularity – The system was developed as a series of independent components, increasing the flexibility of the system and making it easier to distribute.

Expandability –Expandability deals with the need to be able to add new features, representations, distances etc. This aspect is obtained by using object-oriented programming and having a plugin organisation of the component libraries. The system should also be easy to integrate with other components or user interfaces (the use of the standard HTTP [HyperText Transfer Protocol] transmission protocol makes this easy). The use of JSON (JavaScript Object Notation), a standardised and light-weight data transfer format allows for interoperability with other retrieval systems.

Scalability –ParaDISE also aims to be a scalable system, both for offline operations (such as indexing a set of images) as well as online operations (searching for images, for example). On the offline part, scalability is made possible by the use of Hadoop [9], a distributed computing framework developed by the Apache Software Foundation. Its use and benefits are discussed in [10]. For online optimisations, parallel computing and efficient indexing techniques for fast similarity search are used to cope with an ever-increasing amount of images.

The architecture of the ParaDISE backend is component-based. The main components of the backend are described below and shown in figure 1:

Extractor – performs the extraction of local visual features from the image.

Descriptor –creates a global representation of the image.

Storer – accesses visual indices (read/write operations).

Fusor –combines multiple result lists from various queries based on a given fusion rule. It can also be used for combining ParaDISE with other systems (advanced text search with semantic aspects, for example).

These four core components are integrated into two composite components (Indexer and Seeker) to perform the fundamental retrieval operations; indexing and similarity search (fig. 1).

ParaDISE web service layer

The service layer of the ParaDISE system, built on top of the backend described in the previous section, is made up of multiple web services that use a REST-style (REpresentational State Transfer) architecture. It also provides additional features such as the text search.

fullscreen

Standard HTTP GET and POST requests are used to communicate with the web services, which simplifies interaction between the system and various client applications (web-based or desktop applications that can be written in any language capable of making HTTP requests).

The various web services are listed and described below:

Caption web service – Responsible for search by keyword. Uses the Lucene [11] search engine library.

Visual web service – Uses the visual information of the images in order to retrieve similar matches using CBIR techniques or index a new set of images. This service is backed by ParaDISE's Seeker and Indexer components.

Compound figure web service – Used to retrieve links between compound figures (multiple images in a single block) and their children (obtained by separating the compound figure into its subparts), as well as getting the coordinates of each child within its parent image (see the Result Zone section for more details and an example).

Fusion web service – Responsible for combining results from different sources (text search, image search) into a single list (based on a given fusion rule). This service is backed by ParaDISE's Fusor component.

Global web service – Provides a façade for client applications, which hides some of the complexity of calling the individual web services in succession by providing general methods such as "search images", which can use text, image examples, or both.

The structure of the web services is shown in figure 2

Datasets

Currently, there are two datasets that can be accessed through the Shambala interface.

fullscreen

ImageCLEFmed 2012 dataset – ImageCLEF [12] is a part of the Cross-Language Evaluation Forum (CLEF), an information retrieval competition. The medical task of ImageCLEF in 2012 used a dataset of over 300,000 images taken from 75,000 articles of the biomedical open access literature.

PubMedCentral dataset –This dataset consists of the Open Access Subset provided by the PubMedCentral [13] digital repository, which contains over 1.5 million images from the biomedical literature. The status of PubMedCentral was taken in 2013, as the resource constantly increases its size.

Results

This section details the features provided by Shambala and contains screenshots of the application to illustrate its use.

Interface overview

An overview of the Shambala interface is shown below in figure 3.

The interface is split up into 5 main zones:

1. Relevant zone – The user can add relevant textual terms and image examples here.

2. Irrelevant zone –The user can filter out textual terms from the results and provide irrelevant image examples here.

3. Results zone –The search results are displayed here.

4. Image sample zone – This pull-up menu allows the user to get random sample images or upload his images that can be added to the relevant or irrelevant zones.

5. Search option zone –This pull-down menu allows the user to customise the search behaviour.

These zones and the features they provide are described in more detail in the next subsections.

Relevant & irrelevant zones

On the left-hand and right-hand sides are containers for textual terms and image examples. The user can type terms he is interested in on the left and terms he wants to exclude on the right. The queries can be simple words or more complex queries that use Lucene's query parser syntax [14], supporting wildcard searches, various Boolean operators, etc.

fullscreen

Additionally, the user can drag images from the results zone or from the image samples pull-up menu to the relevant or irrelevant container. By adding multiple images, he can refine his query to get more relevant results (using Rocchio relevance feedback [15]).

Result zone

The result zone shows a grid of the returned search results, but also provides several interactions for the user. The zone is completely dynamic and will layout the images in a way that optimises the number of shown results given the available vertical space. Additionally, the number of columns changes depending on the size of the browser window in order to accommodate devices with smaller screen resolutions (fig. 4).

Furthermore, the user can click on each result to see a zoomed-in view, the image caption and imaging modality (if detected automatically), along with several basic image editing tools that allow adjusting the image contrast/brightness (useful for radiology). The contrast and brightness can also be adjusted directly by dragging inside the image horizontally or vertically, as in radiology viewing stations. The detailed result view is shown in figure 5.

Finally, the interface treats compound figures, which are images containing multiple subfigures of the same or varying modality (mixed MRI and graphs, for example) and represent a large portion of images in the biomedical literature [16]. These compound figures are separated into their subcomponents by ParaDISE, using a technique described in [17]. All the information concerning the subfigure coordinates within the original parent image are stored and can be accessed, allowing the user to view a separated figure in its original context, as well as navigating between the multiple subfigures of a compound figure (fig. 7). Subfigure results are highlighted in the result list with a visual cue (fig. 6).

Image sample zone

The lower zone can be pulled up by clicking on the red bar at the bottom of the screen (shown in figure 8). Two tabs are then shown: One allows the user to upload image examples from his computer and the other provides random image examples from the ImageCLEF2012 dataset.

fullscreen

fullscreen

fullscreen

fullscreen

The upload tab contains a drop zone for images, which can be dragged directly from the computer into the browser. Figure 9 shows the zone filled with a few uploaded images that can then be added to the relevant/irrelevant image containers.

Search option zone

The upper option zone (fig. 10) can be pulled down by clicking on the red "Search options" tab. A few options are then presented to the user:

fullscreen

fullscreen

Index – The user can switch between two available datasets described in the methods section.

Modalities –The user can choose to filter the results by modality, either keeping all modalities, diagnostic images only, radiology images only, or filter the results automatically based on the modalities of given relevant image examples.

Use image captions –The user can set whether the captions of relevant/irrelevant image examples are also taken into account for the text search or not.

Use semantic terms –Moreover, the user can choose to keep each caption as-is or to replace it with semantic terms extracted from the caption by an external web service.

Conclusions and future work

Shambala is a modern and easy-to-use web interface for searching images. It takes advantage of trends in web development, particularly HTML5 and its APIs. It currently functions in tandem with the ParaDISE retrieval system, but can easily be adapted to other systems that offer web services communicating with the JSON data exchange format. It has a minimalistic yet feature-rich user interface that helps engage users in discovering and interacting with the system. Plans for the future include more focused user tests to determine if the interface can help in reducing the time needed to find relevant information about a given pathology, as well as tightly integrating the interface with alternative human-computer interaction tools such as motion sensors (ex: Leap Motion™ [18], described in [19]) or speech recognition, in order to facilitate its use in a sterile clinical setting, such as an operating room.

fullscreen

Acknowledgments

This work was supported by the EU 7th Framework Programme in the context of the Khresmoi project (FP7 grant 257528).

Correspondence

Correspondence:

Roger Schaer

Techno-Pôle 3

CH-3960 Sierre

roger.schaer[at]hevs.ch

References

1 Akgül C, Rubin D, Napel S, Beaulieu C, Greenspan H, Acar B. Content-based image retrieval in radiology: current status and future directions. Digital Imaging. 2011;208‒222.

2 Hanbury A, Boyer C, Gschwandtner M, Müller H. KHRESMOI: towards a multi-lingual search and access system for biomedical information. Luxembourg, 2011. Med-e-Tel, pp. 412–16.

3 http://goldminer.arrs.org

4 http://www.averbis.de/public/download/RadMiner/_brochure/_en.pdf

5 http://openi.nlm.nih.gov/

6 http://yottalook.com

7 http://shambala.khresmoi.eu

8 Schaer R, Markonis D, Müller H. Architecture and applications of the Parallel Distributed Image Search Engine (ParaDISE). FoRESEE, Stuttgart, Germany, submitted.

9 http://hadoop.apache.org/

10 Markonis D, Schaer R, Eggel I, Müller H, Depeursinge A. Using MapReduce for large-scale medical image analysis. HISB, La Jolla, California, 2012.

11 http://lucene.apache.org/

12 http://www.imageclef.org/

13 http://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/

14 http://lucene.apache.org/core/2_9_4/queryparsersyntax.html

15 Rocchio JJ. Relevance feedback in information retrieval. In: The SMART Retrieval System: Experiments in Automatic Document Processing, Prentice-Hall Inc., 1971. pp. 313‒23.

16 Müller H, Garcìa Seco de Herrera A, Kalpathy-Cramer J, Demner Fushman S, Antani S, Eggel I. Overview of the ImageCLEF 2012 Medical Image Retrieval and Classification Tasks. Rome, Italy, 2012.

17 Chhatkuli A, Markonis D, Foncubierta-Rodríguez A, Meriaudeau F, Müller H. Separating Compound Figures in Journal Articles to allow for Subfigure Classification. In: SPIE Medical Imaging, Orlando, FL, USA, 2013.

18 https://www.leapmotion.com/

19 Widmer A, Schaer R, Markonis D, Müller H. Gesture interaction for content-based medical image retrieval. In: ICMR, Glasgow, UK, 2014.

Copyright

Published under the copyright license
“Attribution – Non-Commercial – NoDerivatives 4.0”.
No commercial reuse without permission.
See: emh.ch/en/emh/rights-and-licences/