Package com.ibm.fhir.bulkdata.common

Class BulkDataUtils

java.lang.Object

com.ibm.fhir.bulkdata.common.BulkDataUtils

public class BulkDataUtils
extends Object

Utility functions for IBM COS.

Field Summary

Fields

Modifier and Type	Field	Description
static int	IMPORT_RETRY_TIMES

Constructor Summary

Constructors

Constructor	Description
BulkDataUtils()

Method Summary

Modifier and Type	Method	Description
static void	cleanupTransientUserData(ImportTransientUserData transientUserData, boolean isAbort)
static void	finishMultiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks)
static long	getCosFileSize(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName)
static jakarta.json.JsonArray	getDataSourcesFromJobInput(String dataSourcesInfo)
static Map<Class<? extends Resource>,List<Map<String,List<String>>>>	getSearchParametersFromTypeFilters(String typeFilters)	converts the type filter into a series of search parameters which are used to filter the bulk data export results.
static com.ibm.cloud.objectstorage.services.s3.model.PartETag	multiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, InputStream dataStream, int partSize, int partNum)	Use the passed cosClient to upload part of a multi-part object
static int	readFhirResourceFromHttps(String dataUrl, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static int	readFhirResourceFromLocalFile(String filePath, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static int	readFhirResourceFromObjectStore(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static String	startPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName)
static void	updateSummary(String fhirResourceType, ExportTransientUserData chunkData)	Update the chunkData with the stats from the newly finished upload.
static List<OperationOutcome.Issue>	validateInput(Resource resource)	Validate the input resource and throw if there are validation errors

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

IMPORT_RETRY_TIMES

public static final int IMPORT_RETRY_TIMES

See Also:

Constant Field Values

Constructor Detail

BulkDataUtils

public BulkDataUtils()

Method Detail

startPartUpload

public static String startPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                     String bucketName,
                                     String itemName)
                              throws Exception

Throws:

Exception

multiPartUpload

public static com.ibm.cloud.objectstorage.services.s3.model.PartETag multiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                                                                     String bucketName,
                                                                                     String itemName,
                                                                                     String uploadID,
                                                                                     InputStream dataStream,
                                                                                     int partSize,
                                                                                     int partNum)
                                                                              throws Exception

Use the passed cosClient to upload part of a multi-part object

Parameters:

cosClient -

bucketName -

itemName -

uploadID -

dataStream -

partSize -

partNum -

Returns:

Throws:

Exception

finishMultiPartUpload

public static void finishMultiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                         String bucketName,
                                         String itemName,
                                         String uploadID,
                                         List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks)
                                  throws Exception

Throws:

Exception

cleanupTransientUserData

public static void cleanupTransientUserData(ImportTransientUserData transientUserData,
                                            boolean isAbort)
                                     throws Exception

Throws:

Exception

readFhirResourceFromObjectStore

public static int readFhirResourceFromObjectStore(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                                  String bucketName,
                                                  String itemName,
                                                  int numOfLinesToSkip,
                                                  List<Resource> fhirResources,
                                                  ImportTransientUserData transientUserData)
                                           throws Exception

Parameters:

cosClient - - COS/S3 client.

bucketName - - COS/S3 bucket name to read from.

itemName - - COS/S3 object name to read from.

numOfLinesToSkip - - number of lines to skip before read.

fhirResources - - List holds the FHIR resources.

transientUserData - - transient user data for the chunk.

Returns:

- number of parsing failures.

Throws:

Exception

getCosFileSize

public static long getCosFileSize(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                  String bucketName,
                                  String itemName)
                           throws Exception

Throws:

Exception

readFhirResourceFromLocalFile

public static int readFhirResourceFromLocalFile(String filePath,
                                                int numOfLinesToSkip,
                                                List<Resource> fhirResources,
                                                ImportTransientUserData transientUserData)
                                         throws Exception

Parameters:

filePath - - file path to the ndjson file.

numOfLinesToSkip - - number of lines to skip before read.

fhirResources - - List holds the FHIR resources.

transientUserData - - transient user data for the chunk.

Returns:

- number of parsing failures.

Throws:

Exception

readFhirResourceFromHttps

public static int readFhirResourceFromHttps(String dataUrl,
                                            int numOfLinesToSkip,
                                            List<Resource> fhirResources,
                                            ImportTransientUserData transientUserData)
                                     throws Exception

Parameters:

dataUrl - - URL to the ndjson file.

numOfLinesToSkip - - number of lines to skip before read.

fhirResources - - List holds the FHIR resources.

transientUserData - - transient user data for the chunk.

Returns:

- number of parsing failures.

Throws:

Exception

validateInput

public static List<OperationOutcome.Issue> validateInput(Resource resource)
                                                  throws FHIRValidationException,
                                                         FHIROperationException

Validate the input resource and throw if there are validation errors

Parameters:

resource -

Throws:

FHIRValidationException

FHIROperationException

getSearchParametersFromTypeFilters

public static Map<Class<? extends Resource>,List<Map<String,List<String>>>> getSearchParametersFromTypeFilters(String typeFilters)
                                                                                                                    throws UnsupportedEncodingException,
                                                                                                                           URISyntaxException

converts the type filter into a series of search parameters which are used to filter the bulk data export results.

Parameters:

typeFilters -

Returns:

Throws:

UnsupportedEncodingException

URISyntaxException

getDataSourcesFromJobInput

public static jakarta.json.JsonArray getDataSourcesFromJobInput(String dataSourcesInfo)

updateSummary

public static void updateSummary(String fhirResourceType,
                                 ExportTransientUserData chunkData)

Update the chunkData with the stats from the newly finished upload. The data is added to the summary from the chunkData's currentUploadResourceNum.

Parameters:

fhirResourceType -

chunkData -