Package org.apache.nutch.util

Class SitemapProcessor

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.util.SitemapProcessor

All Implemented Interfaces:

Configurable, Tool

public class SitemapProcessor
extends Configured
implements Tool

Performs sitemap processing by fetching sitemap links, parsing the content and merging the URLs from sitemaps (with the metadata) into the CrawlDb.

There are two use cases supported in Nutch's sitemap processing:

1. Sitemaps are considered as "remote seed lists". Crawl administrators can prepare a list of sitemap links and inject and fetch only the pages listed in the sitemaps. This suits well for targeted crawl of specific hosts.
2. For an open web crawl, it is not possible to track each host and get the sitemap links manually. Nutch automatically detects the sitemaps for all hosts seen in the crawls and present in the HostDb and injects the URLs from the sitemaps into the CrawlDb.

See Also:

SitemapFeature

Field Summary

Fields

Modifier and Type	Field	Description
static String	CURRENT_NAME
static String	LOCK_NAME
static SimpleDateFormat	sdf
static String	SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT
static String	SITEMAP_OVERWRITE_EXISTING
static String	SITEMAP_REDIR_MAX
static String	SITEMAP_SIZE_MAX
static String	SITEMAP_STRICT_PARSING
static String	SITEMAP_URL_FILTERING
static String	SITEMAP_URL_NORMALIZING

Constructor Summary

Constructors

Constructor	Description
SitemapProcessor()

Method Summary

Modifier and Type	Method	Description
static void	main(String[] args)
int	run(String[] args)
void	sitemap(Path crawldb, Path hostdb, Path sitemapUrlDir, boolean strict, boolean filter, boolean normalize, int threads)
static void	usage()

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

sdf

public static final SimpleDateFormat sdf

CURRENT_NAME

public static final String CURRENT_NAME

See Also:

Constant Field Values

LOCK_NAME

public static final String LOCK_NAME

See Also:

Constant Field Values

SITEMAP_STRICT_PARSING

public static final String SITEMAP_STRICT_PARSING

See Also:

Constant Field Values

SITEMAP_URL_FILTERING

public static final String SITEMAP_URL_FILTERING

See Also:

Constant Field Values

SITEMAP_URL_NORMALIZING

public static final String SITEMAP_URL_NORMALIZING

See Also:

Constant Field Values

SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT

public static final String SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT

See Also:

Constant Field Values

SITEMAP_OVERWRITE_EXISTING

public static final String SITEMAP_OVERWRITE_EXISTING

See Also:

Constant Field Values

SITEMAP_REDIR_MAX

public static final String SITEMAP_REDIR_MAX

See Also:

Constant Field Values

SITEMAP_SIZE_MAX

public static final String SITEMAP_SIZE_MAX

See Also:

Constant Field Values

Constructor Detail

SitemapProcessor

public SitemapProcessor()

Method Detail

sitemap

public void sitemap(Path crawldb,
                    Path hostdb,
                    Path sitemapUrlDir,
                    boolean strict,
                    boolean filter,
                    boolean normalize,
                    int threads)
             throws Exception

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

usage

public static void usage()

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface Tool

Throws:

Exception