Package org.linuxforhealth.fhir.bulkdata.common

Class BulkDataUtils

  • public class BulkDataUtils
extends java.lang.Object
    Utility functions for IBM COS.

    • Constructor Summary

      Constructors 
      Constructor Description
      BulkDataUtils()  

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static void cleanupTransientUserData​(ImportTransientUserData transientUserData, boolean isAbort)  
      static void finishMultiPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, java.lang.String bucketName, java.lang.String itemName, java.lang.String uploadID, java.util.List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks)  
      static long getCosFileSize​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, java.lang.String bucketName, java.lang.String itemName)  
      static jakarta.json.JsonArray getDataSourcesFromJobInput​(java.lang.String dataSourcesInfo)  
      static java.util.Map<java.lang.Class<? extends Resource>,​java.util.List<java.util.Map<java.lang.String,​java.util.List<java.lang.String>>>> getSearchParametersFromTypeFilters​(java.lang.String typeFilters)
      converts the type filter into a series of search parameters which are used to filter the bulk data export results.
      static com.ibm.cloud.objectstorage.services.s3.model.PartETag multiPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, java.lang.String bucketName, java.lang.String itemName, java.lang.String uploadID, java.io.InputStream dataStream, int partSize, int partNum)
      Use the passed cosClient to upload part of a multi-part object
      static int readFhirResourceFromHttps​(java.lang.String dataUrl, int numOfLinesToSkip, java.util.List<Resource> fhirResources, ImportTransientUserData transientUserData)  
      static int readFhirResourceFromLocalFile​(java.lang.String filePath, int numOfLinesToSkip, java.util.List<Resource> fhirResources, ImportTransientUserData transientUserData)  
      static int readFhirResourceFromObjectStore​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, java.lang.String bucketName, java.lang.String itemName, int numOfLinesToSkip, java.util.List<Resource> fhirResources, ImportTransientUserData transientUserData)  
      static java.lang.String startPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient, java.lang.String bucketName, java.lang.String itemName)  
      static void updateSummary​(java.lang.String fhirResourceType, ExportTransientUserData chunkData)
      Update the chunkData with the stats from the newly finished upload.
      static java.util.List<OperationOutcome.Issue> validateInput​(Resource resource)
      Validate the input resource and throw if there are validation errors

      • Methods inherited from class java.lang.Object

        clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • BulkDataUtils

        public BulkDataUtils()

    • Method Detail

      • startPartUpload

        public static java.lang.String startPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                               java.lang.String bucketName,
                                               java.lang.String itemName)
                                        throws java.lang.Exception
        Throws:
        java.lang.Exception

      • multiPartUpload

        public static com.ibm.cloud.objectstorage.services.s3.model.PartETag multiPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                                                                     java.lang.String bucketName,
                                                                                     java.lang.String itemName,
                                                                                     java.lang.String uploadID,
                                                                                     java.io.InputStream dataStream,
                                                                                     int partSize,
                                                                                     int partNum)
                                                                              throws java.lang.Exception
        Use the passed cosClient to upload part of a multi-part object
        Parameters:
        cosClient -
        bucketName -
        itemName -
        uploadID -
        dataStream -
        partSize -
        partNum -
        Returns:
        Throws:
        java.lang.Exception

      • finishMultiPartUpload

        public static void finishMultiPartUpload​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                         java.lang.String bucketName,
                                         java.lang.String itemName,
                                         java.lang.String uploadID,
                                         java.util.List<com.ibm.cloud.objectstorage.services.s3.model.PartETag> dataPacks)
                                  throws java.lang.Exception
        Throws:
        java.lang.Exception

      • cleanupTransientUserData

        public static void cleanupTransientUserData​(ImportTransientUserData transientUserData,
                                            boolean isAbort)
                                     throws java.lang.Exception
        Throws:
        java.lang.Exception

      • readFhirResourceFromObjectStore

        public static int readFhirResourceFromObjectStore​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                                  java.lang.String bucketName,
                                                  java.lang.String itemName,
                                                  int numOfLinesToSkip,
                                                  java.util.List<Resource> fhirResources,
                                                  ImportTransientUserData transientUserData)
                                           throws java.lang.Exception
        Parameters:
        cosClient - - COS/S3 client.
        bucketName - - COS/S3 bucket name to read from.
        itemName - - COS/S3 object name to read from.
        numOfLinesToSkip - - number of lines to skip before read.
        fhirResources - - List holds the FHIR resources.
        transientUserData - - transient user data for the chunk.
        Returns:
        - number of parsing failures.
        Throws:
        java.lang.Exception

      • getCosFileSize

        public static long getCosFileSize​(com.ibm.cloud.objectstorage.services.s3.AmazonS3 cosClient,
                                  java.lang.String bucketName,
                                  java.lang.String itemName)
                           throws java.lang.Exception
        Throws:
        java.lang.Exception

      • readFhirResourceFromLocalFile

        public static int readFhirResourceFromLocalFile​(java.lang.String filePath,
                                                int numOfLinesToSkip,
                                                java.util.List<Resource> fhirResources,
                                                ImportTransientUserData transientUserData)
                                         throws java.lang.Exception
        Parameters:
        filePath - - file path to the ndjson file.
        numOfLinesToSkip - - number of lines to skip before read.
        fhirResources - - List holds the FHIR resources.
        transientUserData - - transient user data for the chunk.
        Returns:
        - number of parsing failures.
        Throws:
        java.lang.Exception

      • readFhirResourceFromHttps

        public static int readFhirResourceFromHttps​(java.lang.String dataUrl,
                                            int numOfLinesToSkip,
                                            java.util.List<Resource> fhirResources,
                                            ImportTransientUserData transientUserData)
                                     throws java.lang.Exception
        Parameters:
        dataUrl - - URL to the ndjson file.
        numOfLinesToSkip - - number of lines to skip before read.
        fhirResources - - List holds the FHIR resources.
        transientUserData - - transient user data for the chunk.
        Returns:
        - number of parsing failures.
        Throws:
        java.lang.Exception

      • getSearchParametersFromTypeFilters

        public static java.util.Map<java.lang.Class<? extends Resource>,​java.util.List<java.util.Map<java.lang.String,​java.util.List<java.lang.String>>>> getSearchParametersFromTypeFilters​(java.lang.String typeFilters)
                                                                                                                                                                                          throws java.io.UnsupportedEncodingException,
                                                                                                                                                                                                 java.net.URISyntaxException
        converts the type filter into a series of search parameters which are used to filter the bulk data export results.
        Parameters:
        typeFilters -
        Returns:
        Throws:
        java.io.UnsupportedEncodingException
        java.net.URISyntaxException

      • getDataSourcesFromJobInput

        public static jakarta.json.JsonArray getDataSourcesFromJobInput​(java.lang.String dataSourcesInfo)

      • updateSummary

        public static void updateSummary​(java.lang.String fhirResourceType,
                                 ExportTransientUserData chunkData)
        Update the chunkData with the stats from the newly finished upload. The data is added to the summary from the chunkData's currentUploadResourceNum.
        Parameters:
        fhirResourceType -
        chunkData -