Package org.apache.nutch.protocol.file

Class File

java.lang.Object

org.apache.nutch.protocol.file.File

All Implemented Interfaces:

Configurable, Pluggable, Protocol

public class File
extends Object
implements Protocol

This class is a protocol plugin used for file: scheme. It creates FileResponse object and gets the content of the url from it. Configurable parameters are file.content.limit and file.crawl.parent in nutch-default.xml defined under "file properties" section.

Author:

John Xing

Field Summary

Fields

Modifier and Type	Field	Description
protected static org.slf4j.Logger	LOG

Fields inherited from interface org.apache.nutch.protocol.Protocol

X_POINT_ID

Constructor Summary

Constructors

Constructor	Description
File()

Method Summary

Modifier and Type	Method	Description
Configuration	getConf()	Get the Configuration object
ProtocolOutput	getProtocolOutput(Text url, CrawlDatum datum)	Creates a FileResponse object corresponding to the url and return a ProtocolOutput object as per the content received
crawlercommons.robots.BaseRobotRules	getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)	No robots parsing is done for file protocol.
static void	main(String[] args)	Quick way for running this class.
void	setConf(Configuration conf)	Set the Configuration object
void	setMaxContentLength(int maxContentLength)	Set the length after at which content is truncated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

File

public File()

Method Detail

setConf

public void setConf(Configuration conf)

Set the Configuration object

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Get the Configuration object

Specified by:

getConf in interface Configurable

setMaxContentLength

public void setMaxContentLength(int maxContentLength)

Set the length after at which content is truncated.

Parameters:

maxContentLength - max content in bytes

getProtocolOutput

public ProtocolOutput getProtocolOutput(Text url,
                                        CrawlDatum datum)

Creates a FileResponse object corresponding to the url and return a ProtocolOutput object as per the content received

Specified by:

getProtocolOutput in interface Protocol

Parameters:

url - Text containing the url

datum - The CrawlDatum object corresponding to the url

Returns:

ProtocolOutput object for the content of the file indicated by url

main

public static void main(String[] args)
                 throws Exception

Quick way for running this class. Useful for debugging.

Parameters:

args - run with no args to print help

Throws:

Exception - if there is a fatal error running this class with the given input

getRobotRules

public crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                          CrawlDatum datum,
                                                          List<Content> robotsTxtContent)

No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.

Specified by:

getRobotRules in interface Protocol

Parameters:

url - URL to check

datum - page datum

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robot rules (specific for this URL or default), never null