Class COSReader


  • public class COSReader
    extends Object
    Grabs work from the bucket database and dispatches items to the thread pool.
    • Constructor Detail

      • COSReader

        public COSReader​(ExecutorService commonPool,
                         FileType fileType,
                         COSClient client,
                         Consumer<ResourceEntry> resourceHandler,
                         int maxInflight,
                         DataAccess da,
                         boolean incremental,
                         int recycleSeconds,
                         boolean incrementalExact,
                         double bundleCostFactor,
                         Collection<BucketPath> bucketPaths)
        Public constructor
        Parameters:
        commonPool - thread pool shared by the readers and request handler
        fileType - the file type this read is responsible for processing
        client -
        resourceHandler -
        poolSize -
        da -
        incremental -
        recycleSeconds -
    • Method Detail

      • signalStop

        public void signalStop()
        Tell the main thread of this active object that it should start shutting down
      • waitForStop

        public void waitForStop()
        Tell the main loop thread to stop
      • init

        public void init()
        start the allocation thread
      • mainAllocationLoop

        public void mainAllocationLoop()
        The main loop of this active object
      • markJobDone

        protected void markJobDone​(BucketLoaderJob job)
        Callback when the last record in the job completes
        Parameters:
        job -
      • processNDJSON

        public void processNDJSON​(BucketLoaderJob job,
                                  BufferedReader br)
        Read the resources from the given reader
        Parameters:
        is -
      • process

        protected boolean process​(BucketLoaderJob job,
                                  Resource resource,
                                  int lineNumber,
                                  String line)
        Process the resource parsed from the input stream
        Parameters:
        details - of the job being processed
        resource - parsed from the resource_bundle being processed
        lineNumber - the line number of this resource in the source
        line - the original resource string, for logging if we need to