Package org.apache.nutch.tools

Class WARCUtils

java.lang.Object

org.apache.nutch.tools.WARCUtils

public class WARCUtils
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static String	COLONSP
static String	CONFORMS_TO
static String	CRLF
static String	FORMAT
static org.archive.uid.UUIDGenerator	generator
static String	HOSTNAME
static String	HTTP_HEADER_FROM
static String	HTTP_HEADER_USER_AGENT
static String	IP
static String	OPERATOR
protected static Pattern	PROBLEMATIC_HEADERS
static String	ROBOTS
static String	SOFTWARE
protected static String	X_HIDE_HEADER

Constructor Summary

Constructors

Constructor	Description
WARCUtils()

Method Summary

Modifier and Type	Method	Description
static org.archive.io.warc.WARCRecordInfo	docToMetadata(NutchDocument doc)
static String	fixHttpHeaders(String headers, int contentLength)	Modify verbatim HTTP response headers: fix, remove or replace headers Content-Length, Content-Encoding and Transfer-Encoding which may confuse WARC readers.
static String	getAgentString(String name, String version, String description, String URL, String email)
static String	getHostname(Configuration conf)
static String	getIPAddress(Configuration conf)
static org.archive.util.anvl.ANVLRecord	getWARCInfoContent(Configuration conf)
static byte[]	toByteArray(org.archive.format.http.HttpHeaders headers)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SOFTWARE

public static final String SOFTWARE

See Also:

Constant Field Values

HTTP_HEADER_FROM

public static final String HTTP_HEADER_FROM

See Also:

Constant Field Values

HTTP_HEADER_USER_AGENT

public static final String HTTP_HEADER_USER_AGENT

See Also:

Constant Field Values

HOSTNAME

public static final String HOSTNAME

See Also:

Constant Field Values

ROBOTS

public static final String ROBOTS

See Also:

Constant Field Values

OPERATOR

public static final String OPERATOR

See Also:

Constant Field Values

FORMAT

public static final String FORMAT

See Also:

Constant Field Values

CONFORMS_TO

public static final String CONFORMS_TO

See Also:

Constant Field Values

IP

public static final String IP

See Also:

Constant Field Values

generator

public static final org.archive.uid.UUIDGenerator generator

CRLF

public static final String CRLF

See Also:

Constant Field Values

COLONSP

public static final String COLONSP

See Also:

Constant Field Values

PROBLEMATIC_HEADERS

protected static final Pattern PROBLEMATIC_HEADERS

X_HIDE_HEADER

protected static final String X_HIDE_HEADER

See Also:

Constant Field Values

Constructor Detail

WARCUtils

public WARCUtils()

Method Detail

getWARCInfoContent

public static final org.archive.util.anvl.ANVLRecord getWARCInfoContent(Configuration conf)

getHostname

public static final String getHostname(Configuration conf)
                                throws UnknownHostException

Throws:

UnknownHostException

getIPAddress

public static final String getIPAddress(Configuration conf)
                                 throws UnknownHostException

Throws:

UnknownHostException

toByteArray

public static final byte[] toByteArray(org.archive.format.http.HttpHeaders headers)
                                throws IOException

Throws:

IOException

getAgentString

public static final String getAgentString(String name,
                                          String version,
                                          String description,
                                          String URL,
                                          String email)

docToMetadata

public static final org.archive.io.warc.WARCRecordInfo docToMetadata(NutchDocument doc)
                                                              throws UnsupportedEncodingException

Throws:

UnsupportedEncodingException

fixHttpHeaders

public static final String fixHttpHeaders(String headers,
                                          int contentLength)

Modify verbatim HTTP response headers: fix, remove or replace headers Content-Length, Content-Encoding and Transfer-Encoding which may confuse WARC readers. Ensure that returned header end with a single empty line (\r\n\r\n).

Parameters:

headers - HTTP 1.1 or 1.0 response header string, CR-LF-separated lines, first line is status line

contentLength - Effective uncompressed and unchunked length of content

Returns:

safe HTTP response header