ch.hevs.io.hadoop

Class MultipleLineTextInputFormat

    • Field Summary

      Fields 
      Modifier and Type Field and Description
      private org.apache.hadoop.io.compress.CompressionCodecFactory compressionCodecs 
      • Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

        BYTES_READ, COUNTER_GROUP, INPUT_DIR, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE
    • Method Summary

      Methods 
      Modifier and Type Method and Description
      org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) 
      protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path file)
      Force files to be unsplittable in order to avoid to split files in the middle of the N lines.
      • Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

        addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, listStatus, makeSplit, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • compressionCodecs

        private org.apache.hadoop.io.compress.CompressionCodecFactory compressionCodecs
    • Constructor Detail

      • MultipleLineTextInputFormat

        public MultipleLineTextInputFormat()
    • Method Detail

      • isSplitable

        protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context,
                          org.apache.hadoop.fs.Path file)
        Force files to be unsplittable in order to avoid to split files in the middle of the N lines.
        Overrides:
        isSplitable in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>
      • createRecordReader

        public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                                               org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                                                 throws java.io.IOException
        Overrides:
        createRecordReader in class NLineInputFormatFixed
        Throws:
        java.io.IOException