This tutorial is about compound documents and how they are supported by yuuvis® Ultimate.
Compound documents are in principle combinations of the binary coding of several individual documents, which can be translated back into the individual documents by means of the intervals (ranges) in which the binary content of individual documents can be found. In this tutorial, a short Java application will be created that will form a simple example compound document and demonstrate the import of these documents.
Compound documents, like all documents in redline cloud, consist of content and the associated metadata. The content of a compound document consists of the binary content of the documents contained in the compound document, which we will call subdocuments for the sake of simplicity. To ease the retrieval of the individual subdocuments, an additional set of metadata for each subdocument is imported, each with reference to the specific intervals (ranges) of byte indices denoting the location of the content of the subdocument in its respective compound document. In order to learn how to construct a compound document, we must therefore take a look at both the structure of the binary content and the structure of the metadata.
To compose the content of the compound document, we first need the binary content of each subdocument. FileUtils.readFileToByteArray
(File file) allows you to convert the contents of a file into a ByteArray (transforming it into binary code) that can then be written into our compound document file, or rather its FileOutputStream representation. During that process it's important to save the intervals (ranges) denoting the position of the written ByteArray(s) within the compound document file for each subdocuments' content. To do this, set an auxiliary variable offset to 0 at the beginning of the compound document creation process. For each subdocument added to the compound document, increase offset by the length of the subdocuments' content ByteArray, saving a tuple of the previous and new offset value as the range for the subdocument. That tuple will then be written into the ContentStream object of the subdocuments' metadata.
In the following Java code, the content of a compound document is assembled from three simple text files.
//read ByteArrays from text files
byte[] document1BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test.txt"));
byte[] document2BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test1.txt"));
byte[] document3BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test2.txt"));
//generate file for compound document content
File compoundFile = File.createTempFile("compound", ".bin");
OutputStream bos = new BufferedOutputStream(new FileOutputStream(compoundFile));
//partial document = Teildokument
String[] ranges = new String[3]; //Byte ranges of the partial documents in the compound document
String[] partialNames = new String[3]; //Names of the partial documents
//write partial document bytestreams into binary compound file
long offset = 0;
long document1BAlength = document1BA.length;
String range1 = offset + "-" + (offset + document1BAlength - 1);
bos.write(document1BA);
ranges[0] = range1;
partialNames[0] = "test.txt";
offset += document1BAlength;
long document2BAlength = document2BA.length;
String range2 = offset + "-" + (offset + document2BAlength - 1);
bos.write(document2BA);
ranges[1] = range2;
partialNames[1] = "test1.txt";
offset += document2BAlength;
long document3BAlength = document3BA.length;
String range3 = (offset) + "-" + (offset + document3BAlength - 1);
bos.write(document3BA);
ranges[2] = range3;
partialNames[2] = "test2.txt";
IOUtils.closeQuietly(bos);
Once the creation of the compound documents' content is complete, the corresponding metadata for the compound document and its subdocuments has to be generated for the import. The metadata of a compound document itself is no different from the metadata of any regular document - it consists of properties and a contentStreams object pointing to the binary file. The subdocuments, if they are to be imported together with the compound document, copy the contentStreams object of the compound document and add an attribute "range" to it, in which they enter the byte-digit interval recorded during content creation.
{
"objects": [
{
"contentStreams": [
{
"fileName": "./compound.bin",
"mimeType": "application/octet-stream",
"cid": "cid_63apple"
}
],
"properties": {
"enaio:objectTypeId": {
"value": "document"
},
"name": {
"value": "testCompound"
}
}
},
{
"contentStreams": [
{
"fileName": "./compound.bin",
"range": "0-1244",
"mimeType": "text/plain",
"cid": "cid_63apple"
}
],
"properties": {
"enaio:objectTypeId": {
"value": "document"
},
"name": {
"value": "test.txt"
}
}
},
{
"contentStreams": [
{
"fileName": "./compound.bin",
"range": "1245-2516",
"mimeType": "text/plain",
"cid": "cid_63apple"
}
],
"properties": {
"enaio:objectTypeId": {
"value": "document"
},
"name": {
"value": "test1.txt"
}
}
}
]
}
If subdocuments are to be imported later, the contentStreams object in the subdocument’s metadata comprises the contentStreamId and repositoryId from yuuvis® Ultimate response of the import of the compound document, the "mimeType" attribute befitting that subdocument and the same "range" attribute as with the concurrent import.
{
"objects": [
{
"contentStreams": [
{
"contentStreamId": "8FF6DBAE-1969-11E9-83A4-DFA1C5E44BD0",
"repositoryId": "repo242",
"range": "2517-3811",
"mimeType": "text/plain"
}
],
"properties": {
"enaio:objectTypeId": {
"value": "document"
},
"name": {
"value": "test2.txt"
}
}
}
]
}
Importing a compound document with yuuvis® Ultimate works in the same way as regular imports via POST of a multipart body with metadata and content to the endpoint .../objects.
RequestBody compoundImportRequestBody = new MultipartBody.Builder()
.setType(MultipartBody.FORM)
.addFormDataPart("data", "metaData.json", RequestBody.create(JSON, compoundImportJsonString))
.addFormDataPart("cid_63apple", "compound.bin", RequestBody.create(OCTETSTREAM, compoundFile))
.build();
Request compoundImportRequest = new Request.Builder()
.header("Ocp-Apim-Subscription-Key", key)
.url(baseUrl + "objects")
.post(compoundImportRequestBody)
.build();
Response compoundImportResponse = client.newCall(compoundImportRequest).execute();
Importing very large compound documents (5000+ subdocuments) can overload the Core API and cause some of the import operations to fail. Therefore, it is recommended to stage the import of such compound documents in several episodes: First, import the compound document itself as a single document and the contentStreamID and repositoryId are extracted from the response. From there, import the metadata of all subdocuments subsequently in a series of batch imports that limit the amount of subdocuments per import to less than 5000 to avoid overloading.
When subsequently importing subdocuments, please note that the metadata must be sent in a multipart body when importing the subdocuments, even if this multipart body consists of only one part.
RequestBody postPartialDocumentImportRequestBody = new MultipartBody.Builder()
.setType(MultipartBody.FORM)
.addFormDataPart("data", "metaData.json", RequestBody.create(JSON, postPartialDocumentImportJsonString))
.build();
Request postPartialDocumentImportRequest = new Request.Builder()
.header("Ocp-Apim-Subscription-Key", key)
.url(baseUrl + "objects")
.post(postPartialDocumentImportRequestBody)
.build();
Response postPartialDocumentImportResponse = client.newCall(postPartialDocumentImportRequest).execute();
Compound documents consist of one content document and several subdocuments that reference the content document through a content stream and one or more content ranges. When deleting these documents, it is generally avoided to remove any underlying content that may underlie other subdocuments. The subdocuments can be deleted as required, but the content remains accessible (unlike normal documents).