Package org.apache.nutch.tools

Class AbstractCommonCrawlFormat

java.lang.Object

org.apache.nutch.tools.AbstractCommonCrawlFormat

All Implemented Interfaces:

Closeable, AutoCloseable, CommonCrawlFormat

Direct Known Subclasses:

CommonCrawlFormatJackson, CommonCrawlFormatJettinson, CommonCrawlFormatSimple, CommonCrawlFormatWARC

public abstract class AbstractCommonCrawlFormat
extends Object
implements CommonCrawlFormat

Abstract class that implements { @see org.apache.nutch.tools.CommonCrawlFormat } interface.

Field Summary

Fields

Modifier and Type	Field	Description
protected Configuration	conf
protected Content	content
protected List<String>	inLinks
protected boolean	jsonArray
protected String	keyPrefix
protected static org.slf4j.Logger	LOG
protected Metadata	metadata
protected boolean	reverseKey
protected String	reverseKeyValue
protected boolean	simpleDateFormat
protected String	url

Constructor Summary

Constructors

Constructor	Description
AbstractCommonCrawlFormat(String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config)

Method Summary

Modifier and Type	Method	Description
void	close()	Optional method that could be implemented if the actual format needs some close procedure.
protected abstract void	closeArray(String key, boolean nested, boolean newline)
protected abstract void	closeObject(String key)
protected abstract String	generateJson()
protected String	getImported()
List<String>	getInLinks()	gets set of inlinks
String	getJsonData()	Get a string representation of the JSON structure of the URL content.
String	getJsonData(String url, Content content, Metadata metadata)	Returns a string representation of the JSON structure of the URL content.
String	getJsonData(String url, Content content, Metadata metadata, ParseData parseData)	Returns a string representation of the JSON structure of the URL content.
protected String	getKey()
protected String	getMethod()
protected String	getRequestAccept()
protected String	getRequestAcceptEncoding()
protected String	getRequestAcceptLanguage()
protected String	getRequestContactEmail()
protected String	getRequestContactName()
protected String	getRequestHostAddress()
protected String	getRequestHostName()
protected String	getRequestRobots()
protected String	getRequestSoftware()
protected String	getRequestUserAgent()
protected String	getResponseAddress()
protected String	getResponseContent()
protected String	getResponseContentEncoding()
protected String	getResponseContentType()
protected String	getResponseDate()
protected String	getResponseHostName()
protected String	getResponseServer()
protected String	getResponseStatus()
protected String	getTimestamp()
protected String	getUrl()
void	setInLinks(List<String> inLinks)	sets inlinks of this document
protected abstract void	startArray(String key, boolean nested, boolean newline)
protected abstract void	startObject(String key)
protected abstract void	writeArrayValue(String value)
protected abstract void	writeKeyNull(String key)
protected abstract void	writeKeyValue(String key, String value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

url

protected String url

content

protected Content content

metadata

protected Metadata metadata

conf

protected Configuration conf

keyPrefix

protected String keyPrefix

simpleDateFormat

protected boolean simpleDateFormat

jsonArray

protected boolean jsonArray

reverseKey

protected boolean reverseKey

reverseKeyValue

protected String reverseKeyValue

inLinks

protected List<String> inLinks

Constructor Detail

AbstractCommonCrawlFormat

public AbstractCommonCrawlFormat(String url,
                                 Content content,
                                 Metadata metadata,
                                 Configuration nutchConf,
                                 CommonCrawlConfig config)
                          throws IOException

Throws:

IOException

Method Detail

getJsonData

public String getJsonData(String url,
                          Content content,
                          Metadata metadata)
                   throws IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content. Takes into consideration both the Content and Metadata

Specified by:

getJsonData in interface CommonCrawlFormat

Parameters:

url - the canonical url

content - url Content

metadata - url Metadata

Returns:

the JSON URL content string

Throws:

IOException - if there is a fatal I/O error obtaining JSON data

getJsonData

public String getJsonData(String url,
                          Content content,
                          Metadata metadata,
                          ParseData parseData)
                   throws IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content. Takes into consideration the Content, Metadata and ParseData.

Specified by:

getJsonData in interface CommonCrawlFormat

Parameters:

url - the canonical url

content - url Content

metadata - url Metadata

parseData - url ParseData

Returns:

the JSON URL content string

Throws:

IOException - if there is a fatal I/O error obtaining JSON data

getJsonData

public String getJsonData()
                   throws IOException

Description copied from interface: CommonCrawlFormat

Get a string representation of the JSON structure of the URL content.

Specified by:

getJsonData in interface CommonCrawlFormat

Returns:

the JSON URL content string

Throws:

IOException - if there is a fatal I/O error obtaining JSON data

writeKeyValue

protected abstract void writeKeyValue(String key,
                                      String value)
                               throws IOException

Throws:

IOException

writeKeyNull

protected abstract void writeKeyNull(String key)
                              throws IOException

Throws:

IOException

startArray

protected abstract void startArray(String key,
                                   boolean nested,
                                   boolean newline)
                            throws IOException

Throws:

IOException

closeArray

protected abstract void closeArray(String key,
                                   boolean nested,
                                   boolean newline)
                            throws IOException

Throws:

IOException

writeArrayValue

protected abstract void writeArrayValue(String value)
                                 throws IOException

Throws:

IOException

startObject

protected abstract void startObject(String key)
                             throws IOException

Throws:

IOException

closeObject

protected abstract void closeObject(String key)
                             throws IOException

Throws:

IOException

generateJson

protected abstract String generateJson()
                                throws IOException

Throws:

IOException

getUrl

protected String getUrl()

getTimestamp

protected String getTimestamp()

getMethod

protected String getMethod()

getRequestHostName

protected String getRequestHostName()

getRequestHostAddress

protected String getRequestHostAddress()

getRequestSoftware

protected String getRequestSoftware()

getRequestRobots

protected String getRequestRobots()

getRequestContactName

protected String getRequestContactName()

getRequestContactEmail

protected String getRequestContactEmail()

getRequestAccept

protected String getRequestAccept()

getRequestAcceptEncoding

protected String getRequestAcceptEncoding()

getRequestAcceptLanguage

protected String getRequestAcceptLanguage()

getRequestUserAgent

protected String getRequestUserAgent()

getResponseStatus

protected String getResponseStatus()

getResponseHostName

protected String getResponseHostName()

getResponseAddress

protected String getResponseAddress()

getResponseContentEncoding

protected String getResponseContentEncoding()

getResponseContentType

protected String getResponseContentType()

getInLinks

public List<String> getInLinks()

Description copied from interface: CommonCrawlFormat

gets set of inlinks

Specified by:

getInLinks in interface CommonCrawlFormat

Returns:

gets inlinks of this document

setInLinks

public void setInLinks(List<String> inLinks)

Description copied from interface: CommonCrawlFormat

sets inlinks of this document

Specified by:

setInLinks in interface CommonCrawlFormat

Parameters:

inLinks - list of inlinks

getResponseDate

protected String getResponseDate()

getResponseServer

protected String getResponseServer()

getResponseContent

protected String getResponseContent()

getKey

protected String getKey()

getImported

protected String getImported()

close

public void close()

Description copied from interface: CommonCrawlFormat

Optional method that could be implemented if the actual format needs some close procedure.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface CommonCrawlFormat