Package org.apache.nutch.protocol.ftp

Class Ftp

java.lang.Object

org.apache.nutch.protocol.ftp.Ftp

All Implemented Interfaces:

Configurable, Pluggable, Protocol

public class Ftp
extends Object
implements Protocol

This class is a protocol plugin used for ftp: scheme. It creates FtpResponse object and gets the content of the url from it. Configurable parameters are ftp.username, ftp.password, ftp.content.limit, ftp.timeout, ftp.server.timeout, ftp.password, ftp.keep.connection and ftp.follow.talk . For details see "FTP properties" section in nutch-default.xml.

Field Summary

Fields

Modifier and Type	Field	Description
protected static org.slf4j.Logger	LOG

Fields inherited from interface org.apache.nutch.protocol.Protocol

X_POINT_ID

Constructor Summary

Constructors

Constructor	Description
Ftp()

Method Summary

Modifier and Type	Method	Description
protected void	finalize()
int	getBufferSize()
Configuration	getConf()	Get the Configuration object
ProtocolOutput	getProtocolOutput(Text url, CrawlDatum datum)	Creates a FtpResponse object corresponding to the url and returns a ProtocolOutput object as per the content received
crawlercommons.robots.BaseRobotRules	getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)	Get the robots rules for a given url
static void	main(String[] args)	For debugging.
void	setConf(Configuration conf)	Set the Configuration object
void	setFollowTalk(boolean followTalk)	Set followTalk i.e.
void	setKeepConnection(boolean keepConnection)	Whether to keep ftp connection.
void	setMaxContentLength(int length)	Set the length after at which content is truncated.
void	setTimeout(int to)	Set the timeout.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

Ftp

public Ftp()

Method Detail

setTimeout

public void setTimeout(int to)

Set the timeout.

Parameters:

to - a maximum timeout in milliseconds

setMaxContentLength

public void setMaxContentLength(int length)

Set the length after at which content is truncated.

Parameters:

length - max content length in bytes

setFollowTalk

public void setFollowTalk(boolean followTalk)

Set followTalk i.e. to log dialogue between our client and remote server. Useful for debugging.

Parameters:

followTalk - if true will follow, false by default

setKeepConnection

public void setKeepConnection(boolean keepConnection)

Whether to keep ftp connection. Useful if crawling same host again and again. When set to true, it avoids connection, login and dir list parser setup for subsequent URLs. If it is set to true, however, you must make sure (roughly): (1) ftp.timeout is less than ftp.server.timeout (2) ftp.timeout is larger than (fetcher.threads.fetch * fetcher.server.delay) Otherwise there will be too many "delete client because idled too long" messages in thread logs.

Parameters:

keepConnection - if true we will keep the connection, false by default

getProtocolOutput

public ProtocolOutput getProtocolOutput(Text url,
                                        CrawlDatum datum)

Creates a FtpResponse object corresponding to the url and returns a ProtocolOutput object as per the content received

Specified by:

getProtocolOutput in interface Protocol

Parameters:

url - Text containing the ftp url

datum - The CrawlDatum object corresponding to the url

Returns:

ProtocolOutput object for the url

finalize

protected void finalize()

Overrides:

finalize in class Object

main

public static void main(String[] args)
                 throws Exception

For debugging.

Parameters:

args - run with no args for help

Throws:

Exception - if there is an error running this program

setConf

public void setConf(Configuration conf)

Set the Configuration object

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Get the Configuration object

Specified by:

getConf in interface Configurable

getRobotRules

public crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                          CrawlDatum datum,
                                                          List<Content> robotsTxtContent)

Get the robots rules for a given url

Specified by:

getRobotRules in interface Protocol

Parameters:

url - URL to check

datum - page datum

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robot rules (specific for this URL or default), never null

getBufferSize

public int getBufferSize()