Package org.apache.nutch.protocol.http.api

Class HttpRobotRulesParser

java.lang.Object

org.apache.nutch.protocol.RobotRulesParser

org.apache.nutch.protocol.http.api.HttpRobotRulesParser

All Implemented Interfaces:

Configurable, Tool

public class HttpRobotRulesParser
extends RobotRulesParser

This class is used for parsing robots for urls belonging to HTTP protocol. It extends the generic RobotRulesParser class and contains Http protocol specific implementation for obtaining the robots file.

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	allowForbidden
protected boolean	deferVisits503

Fields inherited from class org.apache.nutch.protocol.RobotRulesParser

agentNames, allowList, CACHE, conf, DEFER_VISIT_RULES, EMPTY_RULES, FORBID_ALL_RULES, maxNumRedirects

Constructor Summary

Constructors

Constructor	Description
HttpRobotRulesParser(Configuration conf)

Method Summary

Modifier and Type	Method	Description
protected void	addRobotsContent(List<Content> robotsTxtContent, URL robotsUrl, Response robotsResponse)	Append Content of robots.txt to robotsTxtContent
protected static String	getCacheKey(URL url)	Compose unique key to store and access robot rules in cache for given URL
crawlercommons.robots.BaseRobotRules	getRobotRulesSet(Protocol http, URL url, List<Content> robotsTxtContent)	Get the rules from robots.txt which applies for the given url.
void	setConf(Configuration conf)	Set the Configuration object

Methods inherited from class org.apache.nutch.protocol.RobotRulesParser

getConf, getRobotRulesSet, isAllowListed, main, parseRules, parseRules, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

allowForbidden

protected boolean allowForbidden

deferVisits503

protected boolean deferVisits503

Constructor Detail

HttpRobotRulesParser

public HttpRobotRulesParser(Configuration conf)

Method Detail

setConf

public void setConf(Configuration conf)

Description copied from class: RobotRulesParser

Set the Configuration object

Specified by:

setConf in interface Configurable

Overrides:

setConf in class RobotRulesParser

getCacheKey

protected static String getCacheKey(URL url)

Compose unique key to store and access robot rules in cache for given URL

Parameters:

url - to generate a unique key for

Returns:

the cached unique key

getRobotRulesSet

public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol http,
                                                             URL url,
                                                             List<Content> robotsTxtContent)

Get the rules from robots.txt which applies for the given url. Robot rules are cached for a unique combination of host, protocol, and port. If no rules are found in the cache, a HTTP request is send to fetch {{protocol://host:port/robots.txt}}. The robots.txt is then parsed and the rules are cached to avoid re-fetching and re-parsing it again.

Following RFC 9309, section 2.3.1.2. Redirects, up to five consecutive HTTP redirects are followed when fetching the robots.txt file. The max. number of redirects followed is configurable by the property http.robots.redirect.max.

Specified by:

getRobotRulesSet in class RobotRulesParser

Parameters:

http - The Protocol object

url - URL

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robotRules A BaseRobotRules object for the rules

addRobotsContent

protected void addRobotsContent(List<Content> robotsTxtContent,
                                URL robotsUrl,
                                Response robotsResponse)

Append Content of robots.txt to robotsTxtContent

Parameters:

robotsTxtContent - container to store robots.txt response content

robotsUrl - robots.txt URL

robotsResponse - response object to be stored