Package com.ibm.fhir.bulkdata.common
Class BulkDataUtils
- java.lang.Object
-
- com.ibm.fhir.bulkdata.common.BulkDataUtils
-
public class BulkDataUtils extends Object
Utility functions for IBM COS.
-
-
Field Summary
Fields Modifier and Type Field Description static int
IMPORT_RETRY_TIMES
-
Constructor Summary
Constructors Constructor Description BulkDataUtils()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static void
cleanupTransientUserData(ImportTransientUserData transientUserData, boolean isAbort)
static void
finishMultiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks)
static long
getCosFileSize(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName)
static jakarta.json.JsonArray
getDataSourcesFromJobInput(String dataSourcesInfo)
static Map<Class<? extends Resource>,List<Map<String,List<String>>>>
getSearchParametersFromTypeFilters(String typeFilters)
converts the type filter into a series of search parameters which are used to filter the bulk data export results.static com.ibm.cloud.objectstorage.services.s3.model.PartETag
multiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, InputStream dataStream, int partSize, int partNum)
Use the passed cosClient to upload part of a multi-part objectstatic int
readFhirResourceFromHttps(String dataUrl, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static int
readFhirResourceFromLocalFile(String filePath, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static int
readFhirResourceFromObjectStore(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData)
static String
startPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName)
static void
updateSummary(String fhirResourceType, ExportTransientUserData chunkData)
Update the chunkData with the stats from the newly finished upload.static List<OperationOutcome.Issue>
validateInput(Resource resource)
Validate the input resource and throw if there are validation errors
-
-
-
Field Detail
-
IMPORT_RETRY_TIMES
public static final int IMPORT_RETRY_TIMES
- See Also:
- Constant Field Values
-
-
Method Detail
-
startPartUpload
public static String startPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName) throws Exception
- Throws:
Exception
-
multiPartUpload
public static com.ibm.cloud.objectstorage.services.s3.model.PartETag multiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, InputStream dataStream, int partSize, int partNum) throws Exception
Use the passed cosClient to upload part of a multi-part object- Parameters:
cosClient
-bucketName
-itemName
-uploadID
-dataStream
-partSize
-partNum
-- Returns:
- Throws:
Exception
-
finishMultiPartUpload
public static void finishMultiPartUpload(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, String uploadID, List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks) throws Exception
- Throws:
Exception
-
cleanupTransientUserData
public static void cleanupTransientUserData(ImportTransientUserData transientUserData, boolean isAbort) throws Exception
- Throws:
Exception
-
readFhirResourceFromObjectStore
public static int readFhirResourceFromObjectStore(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData) throws Exception
- Parameters:
cosClient
- - COS/S3 client.bucketName
- - COS/S3 bucket name to read from.itemName
- - COS/S3 object name to read from.numOfLinesToSkip
- - number of lines to skip before read.fhirResources
- - List holds the FHIR resources.transientUserData
- - transient user data for the chunk.- Returns:
- - number of parsing failures.
- Throws:
Exception
-
getCosFileSize
public static long getCosFileSize(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, String bucketName, String itemName) throws Exception
- Throws:
Exception
-
readFhirResourceFromLocalFile
public static int readFhirResourceFromLocalFile(String filePath, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData) throws Exception
- Parameters:
filePath
- - file path to the ndjson file.numOfLinesToSkip
- - number of lines to skip before read.fhirResources
- - List holds the FHIR resources.transientUserData
- - transient user data for the chunk.- Returns:
- - number of parsing failures.
- Throws:
Exception
-
readFhirResourceFromHttps
public static int readFhirResourceFromHttps(String dataUrl, int numOfLinesToSkip, List<Resource> fhirResources, ImportTransientUserData transientUserData) throws Exception
- Parameters:
dataUrl
- - URL to the ndjson file.numOfLinesToSkip
- - number of lines to skip before read.fhirResources
- - List holds the FHIR resources.transientUserData
- - transient user data for the chunk.- Returns:
- - number of parsing failures.
- Throws:
Exception
-
validateInput
public static List<OperationOutcome.Issue> validateInput(Resource resource) throws FHIRValidationException, FHIROperationException
Validate the input resource and throw if there are validation errors- Parameters:
resource
-- Throws:
FHIRValidationException
FHIROperationException
-
getSearchParametersFromTypeFilters
public static Map<Class<? extends Resource>,List<Map<String,List<String>>>> getSearchParametersFromTypeFilters(String typeFilters) throws UnsupportedEncodingException, URISyntaxException
converts the type filter into a series of search parameters which are used to filter the bulk data export results.- Parameters:
typeFilters
-- Returns:
- Throws:
UnsupportedEncodingException
URISyntaxException
-
getDataSourcesFromJobInput
public static jakarta.json.JsonArray getDataSourcesFromJobInput(String dataSourcesInfo)
-
updateSummary
public static void updateSummary(String fhirResourceType, ExportTransientUserData chunkData)
Update the chunkData with the stats from the newly finished upload. The data is added to the summary from the chunkData's currentUploadResourceNum.- Parameters:
fhirResourceType
-chunkData
-
-
-