Package org.apache.nutch.parse.html

Class DOMContentUtils

java.lang.Object

org.apache.nutch.parse.html.DOMContentUtils

public class DOMContentUtils
extends Object

A collection of methods for extracting content from DOM trees. This class holds a few utility methods for pulling content out of DOM nodes, such as getOutlinks, getText, etc.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DOMContentUtils.LinkParams

Constructor Summary

Constructors

Constructor	Description
DOMContentUtils(Configuration conf)

Method Summary

Modifier and Type	Method	Description
String	getBase(Node node)	If Node contains a BASE tag then it's HREF is returned.
void	getOutlinks(URL base, ArrayList<Outlink> outlinks, Node node)	This method finds all anchors below the supplied DOM node, and creates appropriate Outlink records for each (relative to the supplied base URL), and adds them to the outlinks ArrayList.
void	getText(StringBuffer sb, Node node)	This is a convinience method, equivalent to getText(sb, node, false).
boolean	getText(StringBuffer sb, Node node, boolean abortOnNestedAnchors)	This method takes a StringBuffer and a DOM Node, and will append all the content text found beneath the DOM node to the StringBuffer.
boolean	getTitle(StringBuffer sb, Node node)	This method takes a StringBuffer and a DOM Node, and will append the content text found beneath the first title node to the StringBuffer.
void	setConf(Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DOMContentUtils

public DOMContentUtils(Configuration conf)

Method Detail

setConf

public void setConf(Configuration conf)

getText

public boolean getText(StringBuffer sb,
                       Node node,
                       boolean abortOnNestedAnchors)

This method takes a StringBuffer and a DOM Node, and will append all the content text found beneath the DOM node to the StringBuffer.

If abortOnNestedAnchors is true, DOM traversal will be aborted and the StringBuffer will not contain any text encountered after a nested anchor is found.

Parameters:

sb - a StringBuffer used to store content text found beneath the DOM node... if any exists

node - a DOM Node to check for content text

abortOnNestedAnchors - true to abort if nested anchors are encountered, false otherwise

Returns:

true if nested anchors were found

getText

public void getText(StringBuffer sb,
                    Node node)

This is a convinience method, equivalent to getText(sb, node, false).

Parameters:

sb - a StringBuffer used to store content text found beneath the DOM node... if any exists

node - a DOM Node to check for content text

getTitle

public boolean getTitle(StringBuffer sb,
                        Node node)

This method takes a StringBuffer and a DOM Node, and will append the content text found beneath the first title node to the StringBuffer.

Parameters:

sb - a StringBuffer used to store content text found beneath the DOM node... if any exists

node - a DOM Node to check for content text

Returns:

true if a title node was found, false otherwise

getBase

public String getBase(Node node)

If Node contains a BASE tag then it's HREF is returned.

Parameters:

node - a DOM Node to check for a BASE tag

Returns:

HREF if one exists

getOutlinks

public void getOutlinks(URL base,
                        ArrayList<Outlink> outlinks,
                        Node node)

This method finds all anchors below the supplied DOM node, and creates appropriate Outlink records for each (relative to the supplied base URL), and adds them to the outlinks ArrayList.

Links without inner structure (tags, text, etc) are discarded, as are links which contain only single nested links and empty text nodes (this is a common DOM-fixup artifact, at least with nekohtml).

Parameters:

base - the canonical URL

outlinks - the ArrayList of Outlink's associated with the base URL

node - a Node under which to discover anchors