Package org.apache.nutch.segment

Class SegmentChecker

java.lang.Object

org.apache.nutch.segment.SegmentChecker

public class SegmentChecker
extends Object

Checks whether a segment is valid, or has a certain status (generated, fetched, parsed), or can be used safely for a certain processing step (e.g., indexing).

Constructor Summary

Constructors

Constructor	Description
SegmentChecker()

Method Summary

Modifier and Type	Method	Description
static boolean	checkSegmentDir(Path segmentPath, FileSystem fs)	Check the segment to see if it is valid based on the sub directories.
static boolean	isIndexable(Path segmentPath, FileSystem fs)	Check if the segment is indexable.
static boolean	isParsed(Path segment, FileSystem fs)	Check the segment to see if it is has been parsed before.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SegmentChecker

public SegmentChecker()

Method Detail

isIndexable

public static boolean isIndexable(Path segmentPath,
                                  FileSystem fs)
                           throws IOException

Check if the segment is indexable. May add new check methods here.

Parameters:

segmentPath - path to an individual segment on disk

fs - the FileSystem that the segment resides on

Returns:

true if the checks pass and the segment can be indexed, false otherwise

Throws:

IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem

checkSegmentDir

public static boolean checkSegmentDir(Path segmentPath,
                                      FileSystem fs)
                               throws IOException

Check the segment to see if it is valid based on the sub directories.

Parameters:

segmentPath - path to an individual segment on disk

fs - the FileSystem that the segment resides on

Returns:

true if the checks pass false otherwise

Throws:

IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem

isParsed

public static boolean isParsed(Path segment,
                               FileSystem fs)
                        throws IOException

Check the segment to see if it is has been parsed before.

Parameters:

segment - path to an individual segment on disk

fs - the FileSystem that the segment resides on

Returns:

true if the checks pass and the segment has been parsed, false otherwise

Throws:

IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem