Binary Service

Metadaten zur Dokumentation

Inhalt

Vorhandene Dokumentation - bitte beim Erstellen neuer Dokumentationsteile zum Binary Service einarbeiten: https://wiki.deutsche-digitale-bibliothek.de/x/iYth

WORK IN PROGRESS

Introduction

Based on the concept discussed in early 2018 the Binary Service is meant to handle all kinds of binaries provided during a data-ingest process as it occurs in the DDB. Binaries are submitted to this service by passing a URL to the service from where the binary can be retrieved. After retrieval the Binaries Service will store the binaries and provide them to the user on demand. A special focus lies on the handling of images. These are not only retrieved and provided on demand, they are also scaled to 4 predefined resolution which can be accessed using a IIIF image url syntax. 

Components

The Binary Service is made up of two components: The first component is responsible for retrieving the binaries from the provided URL, scaling images and saving the data to the database. The second component is the delivery component which is targeted at the user and delivery data stored in the database.

Technology

The Binary Service stores its data in a Cassandra Database. Cassandra is a cluster-based distributed database capable of safely storing large amounts of data.

The ingest component is a standalone Java application utilizing FIZPro and Apache Spark to process data on the cluster. 

The delivery component is a simple Java servlet. 

Implementation Details

Ingest component

Sources: https://dev.fiz-karlsruhe.de/stash/projects/DDB/repos/ddb-administration-binaries-service

The ingest component as a standalone Java application meant to run on a cluster node.

It offers an API endpoint to submit a URL together with a context saved along with the URL. The endpoint is non-blocking and immediately returns a reference which can later be used to access the binary and all of its derivatives. Once submitted the URL is saved in the database together with the context and the generated reference. It is also enqueued for processing. As soon as there are enough resources on the cluster, the URL will be processed, meaning the following: 

• Has this URL been processed before: if not, the download file. 
• If yes, the check Last-modified header and E-Tag provided 
• If one of them differs from previous checks, download file. 
• Update "last checked" date
• If a file was downloaded, save if to the database
• If downloaded file was an image, scale to predefined resolutions and save to database.

There are different ways of handling errors during this process, especially if the server from which the binary is supposed to be downloaded responds with an error, the URL may be re-enqueued to try the download again later. 

A further API endpoint provides access to the status of the binaries assigned to a given context. The status information is minimal, consisting of only the number of queued, finished and failed binaries. 

A third endpoint allows deleting all references assigned to a given context. 

Java Package description 

NOTE: Initially both components were in one project, therefore the package structure should be reworked, therefore it will not be documented here.

( DDBBS-61 - Getting issue details... STATUS )

Delivery component

Sources: https://dev.fiz-karlsruhe.de/stash/projects/DDB/repos/ddb-administration-binaries-server

The delivery component is a standard Java Servlet using the JAX-RS implementation Jersey to provide access to the stored binaries and their derivatives. 

Java Package description 

de.fiz.binariesservice.server:

Contains the application main classes and the two classes responsible for serving content for /binary and /image. 

de.fiz.binariesservice.models:

Contains JAX-B classes for xml/json mappings. 

de.fiz.binariesservice.utils:

Utility classes (should be moved to other packages with less generic names)

Cassandra table structure

API Documentation

Side project: Local Binary Provider

Deployments