Package org.apache.nutch.segment

Class SegmentReader

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.segment.SegmentReader

All Implemented Interfaces:

Configurable, Tool

public class SegmentReader
extends Configured
implements Tool

Dump the content of a segment.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	SegmentReader.InputCompatMapper
static class	SegmentReader.InputCompatReducer
static class	SegmentReader.SegmentReaderStats
static class	SegmentReader.TextOutputFormat	Implements a text output format

Constructor Summary

Constructors

Constructor	Description
SegmentReader()

Method Summary

Modifier and Type	Method	Description
void	dump(Path segment, Path output)
void	get(Path segment, Text key, Writer writer, Map<String,List<Writable>> results)
static Charset	getCharset(Metadata parseMeta)	Try to get HTML encoding from parse metadata.
void	getStats(Path segment, SegmentReader.SegmentReaderStats stats)
void	list(List<Path> dirs, Writer writer)
static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

SegmentReader

public SegmentReader()

Method Detail

dump

public void dump(Path segment,
                 Path output)
          throws IOException,
                 InterruptedException,
                 ClassNotFoundException

Throws:

IOException

InterruptedException

ClassNotFoundException

get

public void get(Path segment,
                Text key,
                Writer writer,
                Map<String,List<Writable>> results)
         throws Exception

Throws:

Exception

getCharset

public static Charset getCharset(Metadata parseMeta)

Try to get HTML encoding from parse metadata. Try Nutch.CHAR_ENCODING_FOR_CONVERSION, then HttpHeaders.CONTENT_ENCODING then fallback StandardCharsets.UTF_8

Parameters:

parseMeta - a populated Metadata

Returns:

Charset

list

public void list(List<Path> dirs,
                 Writer writer)
          throws Exception

Throws:

Exception

getStats

public void getStats(Path segment,
                     SegmentReader.SegmentReaderStats stats)
              throws Exception

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception