Uses of Class
org.apache.nutch.indexer.NutchDocument

Packages that use NutchDocument

Package	Description
org.apache.nutch.analysis.lang	Text document language identifier.
org.apache.nutch.exchange	Control code for exchange component, which acts in indexing job and decides to which index writer a document should be routed, based on plugins behavior.
org.apache.nutch.exchange.jexl	Plugin of Exchange component based on JEXL expressions.
org.apache.nutch.indexer	Index content, configure and run indexing and cleaning jobs to add, update, and delete documents from an index.
org.apache.nutch.indexer.anchor	An indexing plugin for inbound anchor text.
org.apache.nutch.indexer.arbitrary	Indexing filter to add document arbitrary data to the index from the output of a user-specified class.
org.apache.nutch.indexer.basic	A basic indexing plugin, adds basic fields: url, host, title, content, etc.
org.apache.nutch.indexer.feed	Indexing filter to index meta data from RSS feeds.
org.apache.nutch.indexer.filter
org.apache.nutch.indexer.geoip	This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.
org.apache.nutch.indexer.jexl	This plugin implements a dynamic indexing filter which uses JEXL expressions to allow filtering based on the page's metadata
org.apache.nutch.indexer.links
org.apache.nutch.indexer.metadata	Indexing filter to add document metadata to the index.
org.apache.nutch.indexer.more	A more indexing plugin, adds "more" index fields:last modified date, MIME type, content length.
org.apache.nutch.indexer.replace	Indexing filter to allow pattern replacements on metadata.
org.apache.nutch.indexer.staticfield	A simple plugin called at indexing that adds fields with static data.
org.apache.nutch.indexer.subcollection	Indexing filter to assign documents to subcollections.
org.apache.nutch.indexer.tld	Top Level Domain Indexing plugin.
org.apache.nutch.indexer.urlmeta	URL Meta Tag Indexing Plugin
org.apache.nutch.indexwriter.cloudsearch
org.apache.nutch.indexwriter.csv	Index writer plugin to write a plain CSV file.
org.apache.nutch.indexwriter.dummy	Index writer plugin for debugging, writes pairs of <action, url> to a text file, action is one of "add", "update", or "delete".
org.apache.nutch.indexwriter.elastic	Index writer plugin for Elasticsearch.
org.apache.nutch.indexwriter.kafka	Index writer plugin to produce JSON messages to Kafka.
org.apache.nutch.indexwriter.opensearch1x	Index writer plugin for OpenSearch.
org.apache.nutch.indexwriter.rabbit
org.apache.nutch.indexwriter.solr	Index writer plugin for Apache Solr.
org.apache.nutch.microformats.reltag	A microformats Rel-Tag Parser/Indexer/Querier plugin.
org.apache.nutch.scoring	The ScoringFilter interface.
org.apache.nutch.scoring.depth	Scoring filter to stop crawling at a configurable depth (number of "hops" from seed URLs).
org.apache.nutch.scoring.link	Scoring filter used in conjunction with WebGraph.
org.apache.nutch.scoring.opic	Scoring filter implementing a variant of the Online Page Importance Computation (OPIC) algorithm.
org.apache.nutch.scoring.tld	Top Level Domain Scoring plugin.
org.apache.nutch.tools	Miscellaneous tools.
org.creativecommons.nutch	Sample plugins that parse and index Creative Commons metadata.

Uses of NutchDocument in org.apache.nutch.analysis.lang

Methods in org.apache.nutch.analysis.lang that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	LanguageIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.analysis.lang with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	LanguageIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.exchange

Methods in org.apache.nutch.exchange with parameters of type NutchDocument

Modifier and Type	Method	Description
String[]	Exchanges.indexWriters(NutchDocument nutchDocument)	Returns all the indexers where the document must be sent to.
boolean	Exchange.match(NutchDocument doc)	Determines if the document must go to the related index writers.

Uses of NutchDocument in org.apache.nutch.exchange.jexl

Methods in org.apache.nutch.exchange.jexl with parameters of type NutchDocument

Modifier and Type	Method	Description
boolean	JexlExchange.match(NutchDocument doc)	Determines if the document must go to the related index writers.

Uses of NutchDocument in org.apache.nutch.indexer

Fields in org.apache.nutch.indexer declared as NutchDocument

Modifier and Type	Field	Description
NutchDocument	NutchIndexAction.doc

Methods in org.apache.nutch.indexer that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	NutchDocument.clone()
NutchDocument	IndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Adds fields or otherwise modifies the document that will be indexed for a parse.
NutchDocument	IndexingFilters.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Run all defined filters.

Methods in org.apache.nutch.indexer with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	IndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Adds fields or otherwise modifies the document that will be indexed for a parse.
NutchDocument	IndexingFilters.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Run all defined filters.
void	IndexWriter.update(NutchDocument doc)
void	IndexWriters.update(NutchDocument doc)
void	IndexWriter.write(NutchDocument doc)
void	IndexWriters.write(NutchDocument doc)

Constructors in org.apache.nutch.indexer with parameters of type NutchDocument

Constructor	Description
NutchIndexAction(NutchDocument doc, byte action)

Uses of NutchDocument in org.apache.nutch.indexer.anchor

Methods in org.apache.nutch.indexer.anchor that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	AnchorIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The AnchorIndexingFilter filter object which supports boolean configuration settings for the deduplication of anchors.

Methods in org.apache.nutch.indexer.anchor with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	AnchorIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The AnchorIndexingFilter filter object which supports boolean configuration settings for the deduplication of anchors.

Uses of NutchDocument in org.apache.nutch.indexer.arbitrary

Methods in org.apache.nutch.indexer.arbitrary that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	ArbitraryIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The ArbitraryIndexingFilter filter object uses reflection to instantiate the configured class and invoke the configured method.

Methods in org.apache.nutch.indexer.arbitrary with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	ArbitraryIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The ArbitraryIndexingFilter filter object uses reflection to instantiate the configured class and invoke the configured method.

Uses of NutchDocument in org.apache.nutch.indexer.basic

Methods in org.apache.nutch.indexer.basic that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	BasicIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The BasicIndexingFilter filter object which supports few configuration settings for adding basic searchable fields.

Methods in org.apache.nutch.indexer.basic with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	BasicIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The BasicIndexingFilter filter object which supports few configuration settings for adding basic searchable fields.

Uses of NutchDocument in org.apache.nutch.indexer.feed

Methods in org.apache.nutch.indexer.feed that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	FeedIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Extracts out the relevant fields: FEED_AUTHOR FEED_TAGS FEED_PUBLISHED FEED_UPDATED FEED And sends them to the Indexer for indexing within the Nutch index.

Methods in org.apache.nutch.indexer.feed with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	FeedIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Extracts out the relevant fields: FEED_AUTHOR FEED_TAGS FEED_PUBLISHED FEED_UPDATED FEED And sends them to the Indexer for indexing within the Nutch index.

Uses of NutchDocument in org.apache.nutch.indexer.filter

Methods in org.apache.nutch.indexer.filter that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	MimeTypeIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.filter with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	MimeTypeIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.geoip

Methods in org.apache.nutch.indexer.geoip that return NutchDocument

Modifier and Type	Method	Description
static NutchDocument	GeoIPDocumentCreator.createDocFromCityDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromCityService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromConnectionDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromCountryService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromDomainDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromInsightsService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromIspDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
NutchDocument	GeoIPIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.geoip with parameters of type NutchDocument

Modifier and Type	Method	Description
static void	GeoIPDocumentCreator.addIfNotNull(NutchDocument doc, String name, Object value)	Add field to document but only if value isn't null
static NutchDocument	GeoIPDocumentCreator.createDocFromCityDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromCityService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromConnectionDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromCountryService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromDomainDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
static NutchDocument	GeoIPDocumentCreator.createDocFromInsightsService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client)
static NutchDocument	GeoIPDocumentCreator.createDocFromIspDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader)
NutchDocument	GeoIPIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.jexl

Methods in org.apache.nutch.indexer.jexl that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	JexlIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.jexl with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	JexlIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.links

Methods in org.apache.nutch.indexer.links that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	LinksIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.links with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	LinksIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.metadata

Methods in org.apache.nutch.indexer.metadata that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	MetadataIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.metadata with parameters of type NutchDocument

Modifier and Type	Method	Description
protected void	MetadataIndexer.add(NutchDocument doc, String key, String value)
NutchDocument	MetadataIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.more

Methods in org.apache.nutch.indexer.more that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	MoreIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.more with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	MoreIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.replace

Methods in org.apache.nutch.indexer.replace that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	ReplaceIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.replace with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	ReplaceIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.staticfield

Methods in org.apache.nutch.indexer.staticfield that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	StaticFieldIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The StaticFieldIndexer filter object which adds fields as per configuration setting.

Methods in org.apache.nutch.indexer.staticfield with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	StaticFieldIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	The StaticFieldIndexer filter object which adds fields as per configuration setting.

Uses of NutchDocument in org.apache.nutch.indexer.subcollection

Methods in org.apache.nutch.indexer.subcollection that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	SubcollectionIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.subcollection with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	SubcollectionIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.tld

Methods in org.apache.nutch.indexer.tld that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	TLDIndexingFilter.filter(NutchDocument doc, Parse parse, Text urlText, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.indexer.tld with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	TLDIndexingFilter.filter(NutchDocument doc, Parse parse, Text urlText, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.indexer.urlmeta

Methods in org.apache.nutch.indexer.urlmeta that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	URLMetaIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the CrawlDatum object.

Methods in org.apache.nutch.indexer.urlmeta with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	URLMetaIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the CrawlDatum object.

Uses of NutchDocument in org.apache.nutch.indexwriter.cloudsearch

Methods in org.apache.nutch.indexwriter.cloudsearch with parameters of type NutchDocument

Modifier and Type	Method	Description
void	CloudSearchIndexWriter.update(NutchDocument doc)
void	CloudSearchIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.csv

Methods in org.apache.nutch.indexwriter.csv with parameters of type NutchDocument

Modifier and Type	Method	Description
void	CSVIndexWriter.update(NutchDocument doc)
void	CSVIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.dummy

Methods in org.apache.nutch.indexwriter.dummy with parameters of type NutchDocument

Modifier and Type	Method	Description
void	DummyIndexWriter.update(NutchDocument doc)
void	DummyIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.elastic

Methods in org.apache.nutch.indexwriter.elastic with parameters of type NutchDocument

Modifier and Type	Method	Description
void	ElasticIndexWriter.update(NutchDocument doc)
void	ElasticIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.kafka

Methods in org.apache.nutch.indexwriter.kafka with parameters of type NutchDocument

Modifier and Type	Method	Description
void	KafkaIndexWriter.update(NutchDocument doc)
void	KafkaIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.opensearch1x

Methods in org.apache.nutch.indexwriter.opensearch1x with parameters of type NutchDocument

Modifier and Type	Method	Description
void	OpenSearch1xIndexWriter.update(NutchDocument doc)
void	OpenSearch1xIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.rabbit

Methods in org.apache.nutch.indexwriter.rabbit with parameters of type NutchDocument

Modifier and Type	Method	Description
void	RabbitIndexWriter.update(NutchDocument doc)
void	RabbitIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.indexwriter.solr

Methods in org.apache.nutch.indexwriter.solr with parameters of type NutchDocument

Modifier and Type	Method	Description
void	SolrIndexWriter.update(NutchDocument doc)
void	SolrIndexWriter.write(NutchDocument doc)

Uses of NutchDocument in org.apache.nutch.microformats.reltag

Methods in org.apache.nutch.microformats.reltag that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	RelTagIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.apache.nutch.microformats.reltag with parameters of type NutchDocument

Modifier and Type	Method	Description
NutchDocument	RelTagIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Uses of NutchDocument in org.apache.nutch.scoring

Methods in org.apache.nutch.scoring with parameters of type NutchDocument

Modifier and Type	Method	Description
float	AbstractScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)
float	ScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)	This method calculates a indexed document score/boost.
float	ScoringFilters.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)

Uses of NutchDocument in org.apache.nutch.scoring.depth

Methods in org.apache.nutch.scoring.depth with parameters of type NutchDocument

Modifier and Type	Method	Description
float	DepthScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)

Uses of NutchDocument in org.apache.nutch.scoring.link

Methods in org.apache.nutch.scoring.link with parameters of type NutchDocument

Modifier and Type	Method	Description
float	LinkAnalysisScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)

Uses of NutchDocument in org.apache.nutch.scoring.opic

Methods in org.apache.nutch.scoring.opic with parameters of type NutchDocument

Modifier and Type	Method	Description
float	OPICScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)	Dampen the boost value by scorePower.

Uses of NutchDocument in org.apache.nutch.scoring.tld

Methods in org.apache.nutch.scoring.tld with parameters of type NutchDocument

Modifier and Type	Method	Description
float	TLDScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)

Uses of NutchDocument in org.apache.nutch.tools

Methods in org.apache.nutch.tools with parameters of type NutchDocument

Modifier and Type	Method	Description
static org.archive.io.warc.WARCRecordInfo	WARCUtils.docToMetadata(NutchDocument doc)

Uses of NutchDocument in org.creativecommons.nutch

Methods in org.creativecommons.nutch that return NutchDocument

Modifier and Type	Method	Description
NutchDocument	CCIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)

Methods in org.creativecommons.nutch with parameters of type NutchDocument

Modifier and Type	Method	Description
void	CCIndexingFilter.addUrlFeatures(NutchDocument doc, String urlString)	Add the features represented by a license URL.
NutchDocument	CCIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)