Package com.ibm.fhir.jbatch.bulkdata.export.common

Class SparkParquetWriter

java.lang.Object

com.ibm.fhir.jbatch.bulkdata.export.common.SparkParquetWriter

All Implemented Interfaces:

AutoCloseable

public class SparkParquetWriter
extends Object
implements AutoCloseable

Constructor Summary

Constructors

Constructor	Description
SparkParquetWriter()	Create a SparkParquetWriter that can only write to file URIs
SparkParquetWriter(boolean useIAM, String cosEndpoint, String apiKeyOrAccessKey, String serviceInstanceIdOrSecretKey)	Create a SparkParquetWriter that can write to either file or cos URIs

Method Summary

Modifier and Type	Method	Description
void	close()
void	writeParquet(List<Resource> resources, String outDirName)	Write a list of resources to a parquet file under a single logical file that is actually a directory.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkParquetWriter

public SparkParquetWriter()

Create a SparkParquetWriter that can only write to file URIs

SparkParquetWriter

public SparkParquetWriter(boolean useIAM,
                          String cosEndpoint,
                          String apiKeyOrAccessKey,
                          String serviceInstanceIdOrSecretKey)

Create a SparkParquetWriter that can write to either file or cos URIs

Parameters:

isCOS - whether to use IBM Cloud Identity and Access Management; if false we use "HMAC" auth

cosEndpoint - the S3 endpoint to connect to; include the scheme, typically "https://")

apiKeyOrAccessKey - a valid ServiceCredential ApiKey (for IAM) or AccessKey (for HMAC)

serviceInstanceIdOrSecretKey - a valid ServiceInstanceId (for IAM) or SecretKey (for HMAC)

Method Detail

writeParquet

public void writeParquet(List<Resource> resources,
                         String outDirName)
                  throws FHIRGeneratorException

Write a list of resources to a parquet file under a single logical file that is actually a directory.

Parameters:

resources - the list of resources to write

outDirName - the target directory, using either a file URI or a cos URI like "cos://bucket.service/object-key"

Throws:

FHIRGeneratorException

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception