Sound and Vision

Preserving 1.7 million objects and counting

The Netherlands Institute for Sound and Vision (Nederlands Instituut voor Beeld en Geluid) is the Dutch national audiovisual archives. The organization collects, preserves and provides access to the country's audiovisual heritage – including all of the radio and television broadcasts produced by Dutch public channels – for as many users as possible: media professionals, education and the general public.

Sound and Vision has one of the largest audiovisual archives in Europe. The collection contains more than 750,000 hours of television, radio, music and film from 1898 until today. The institute has 1.7 million objects in its search engine – with new data added every day – accessed by media professionals through an Extranet service, and by the public through an online portal.

Taking control of search

Until 2005, the only way to access the archives was to make a request through the institute's librarians, and they would find the material through a card index. Later, the card index was digitized and made accessible for the first time through the Verity search engine.

"Everyone was very happy because no one had ever been able to search the archives before," recalls Bouke Huurnink, Senior Researcher at Sound and Vision. "But very soon the number of objects in the archive increased. The search was always slow. Average time for a search was five seconds. We upgraded all our hardware and got a modest performance increase, but it was still slow."

The search engine crashed regularly, for example, if a user ran a query that would return a large number of documents. Users would wait two minutes for results, and then get an error message. This situation caused significant user dissatisfaction.

Indexing also took a long time and often crashed. The institute maintains two indexes for load balancing, but they could not synchronize between them properly. One index would differ slightly from the other, causing results from searches to be very unpredictable.

On top of all these problems, Verity was a proprietary software. It was very difficult and costly for the institute to make any changes to the search engine.

"We wanted control over the search engine," Huurnink explains. "We could see users going to other portals because our search was not good enough. Search is very important for our organization. Our job is to maintain the material and make it findable."

Enhancing the user experience

Sound and Vision adopted Elasticsearch as its main data store and search engine for both the extranet and Internet portal, and performance has noticeably improved.

"We replaced the search engine with Elasticsearch, and it is faster and more stable," Huurnink confirms. "Users are very happy because results come up immediately, and queries do not fail."

Elasticsearch also solved the problems of the previous search engine, including stability and syncing the indexes.

"We support users with guaranteed uptime," Huurnink continues. "The reliability of Elasticsearch is a key advantage."

"Syncing is so much easier now with Elasticsearch," he adds. "We don't have to think about it."

"User experience is the most important thing for us, and Elasticsearch has helped improve it greatly. Our users had wishes that we could not actually achieve before, because the search engine was outdated, but we are now able to fulfill those requests with Elasticsearch."

The improvements enabled by Elasticsearch can be simple but very effective from the user perspective. For example, with Elasticsearch, users can search on multiple dates, so they are able to find television repeats as well as initial broadcasts. The previous system did not allow filtering on multiple dates, but now Elasticsearch makes this feature possible – and this is just one example.

Gaining respect across the industry

The benefits of Elasticsearch go way beyond faster response time and improved reliability. Because the Netherlands Institute for Sound and Vision is a public institution, it is expected to meet industry standards and public expectations. In a political sense, the Institute's reputation is critical, and Elasticsearch has helped strengthen industry and public opinion of the organization.

"Because we are a public institute, research and innovation is very important – it is part of the reason for our existence," Huurnink points out. "We are very involved in different projects around the country to develop innovative interfaces. Elasticsearch provides a huge advantage for us."

Sound and Vision provides access to a variety of organizations across the Netherlands, allowing them to interface with the archives. Prior to Elasticsearch, the institute could only provide dumps of the catalog. Now, with Elasticsearch, the institute offers users development flexibility to integrate with the system.

"Elasticsearch has helped open up our infrastructure, and we are able to easily develop new search indexes as necessary to be able to exchange data with industry users and research partners," Huurnink notes.

"Our institution is one of the cultural centers of the Netherlands," concludes Huurnink. "We want to be seen as innovative, as the leading edge, as part of the open source community. Elasticsearch helps maintain this reputation. That is a very positive advantage for us."