Class BaseFileReader

  • Direct Known Subclasses:
    COSReader, LocalFileReader

    public abstract class BaseFileReader
    extends java.lang.Object
    Grabs work from the bucket database and dispatches items to the thread pool.
    • Constructor Summary

      Constructors 
      Constructor Description
      BaseFileReader​(java.util.concurrent.ExecutorService commonPool, FileType fileType, java.util.function.Consumer<ResourceEntry> resourceHandler, int maxInflight, DataAccess da, boolean incremental, int recycleSeconds, boolean incrementalExact, double bundleCostFactor, java.util.Collection<BucketPath> bucketPaths)
      Public constructor
    • Constructor Detail

      • BaseFileReader

        public BaseFileReader​(java.util.concurrent.ExecutorService commonPool,
                              FileType fileType,
                              java.util.function.Consumer<ResourceEntry> resourceHandler,
                              int maxInflight,
                              DataAccess da,
                              boolean incremental,
                              int recycleSeconds,
                              boolean incrementalExact,
                              double bundleCostFactor,
                              java.util.Collection<BucketPath> bucketPaths)
        Public constructor
        Parameters:
        commonPool - thread pool shared by the readers and request handler
        resourceHandler -
        poolSize -
        da -
        incremental -
        recycleSeconds -
        incrementalExact -
        bundleCostFactor -
        bucketPaths -
    • Method Detail

      • signalStop

        public void signalStop()
        Tell the main thread of this active object that it should start shutting down
      • waitForStop

        public void waitForStop()
        Tell the main loop thread to stop
      • init

        public void init()
        start the allocation thread
      • mainAllocationLoop

        public void mainAllocationLoop()
        The main loop of this active object
      • markJobDone

        protected void markJobDone​(BucketLoaderJob job)
        Callback when the last record in the job completes
        Parameters:
        job -
      • processThr

        protected abstract void processThr​(BucketLoaderJob job)
        Process this job in a thread-pool thread
        Parameters:
        job -
      • processJSON

        protected void processJSON​(BucketLoaderJob job,
                                   java.io.Reader reader)
        Process a JSON stream (as opposed to an NDJSON stream)
        Parameters:
        job -
        reader -
      • processNDJSON

        protected void processNDJSON​(BucketLoaderJob job,
                                     java.io.BufferedReader br)
        Read the resources from the given reader
        Parameters:
        is -