Package org.apache.nutch.protocol.ftp

Class FtpRobotRulesParser

java.lang.Object

org.apache.nutch.protocol.RobotRulesParser

org.apache.nutch.protocol.ftp.FtpRobotRulesParser

All Implemented Interfaces:

Configurable, Tool

public class FtpRobotRulesParser
extends RobotRulesParser

This class is used for parsing robots for urls belonging to FTP protocol. It extends the generic RobotRulesParser class and contains Ftp protocol specific implementation for obtaining the robots file.

Field Summary

Fields inherited from class org.apache.nutch.protocol.RobotRulesParser

agentNames, allowList, CACHE, conf, DEFER_VISIT_RULES, EMPTY_RULES, FORBID_ALL_RULES, maxNumRedirects

Constructor Summary

Constructors

Constructor	Description
FtpRobotRulesParser(Configuration conf)

Method Summary

Modifier and Type	Method	Description
crawlercommons.robots.BaseRobotRules	getRobotRulesSet(Protocol ftp, URL url, List<Content> robotsTxtContent)	The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the URL passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.

Methods inherited from class org.apache.nutch.protocol.RobotRulesParser

getConf, getRobotRulesSet, isAllowListed, main, parseRules, parseRules, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FtpRobotRulesParser

public FtpRobotRulesParser(Configuration conf)

Method Detail

getRobotRulesSet

public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol ftp,
                                                             URL url,
                                                             List<Content> robotsTxtContent)

The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the URL passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.

Specified by:

getRobotRulesSet in class RobotRulesParser

Parameters:

ftp - The Protocol object

url - URL

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robotRules A BaseRobotRules object for the rules