Package org.apache.nutch.collection

Class Subcollection

  • All Implemented Interfaces:
    Configurable, URLFilter, Pluggable
    public class Subcollection
extends Configured
implements URLFilter
    SubCollection represents a subset of index, you can define url patterns that will indicate that particular page (url) is part of SubCollection.

    • Constructor Detail

      • Subcollection

        public Subcollection​(String id,
                     String name,
                     Configuration conf)
        public Constructor
        Parameters:
        id - Id of SubCollection
        name - Name of SubCollection
        conf - A populated Configuration

      • Subcollection

        public Subcollection​(String id,
                     String name,
                     String key,
                     Configuration conf)
        public Constructor
        Parameters:
        id - Id of SubCollection
        name - Name of SubCollection
        key - SubCollection key
        conf - A populated Configuration

    • Method Detail

      • getName

        public String getName()
        Returns:
        Returns the name

      • getKey

        public String getKey()
        Returns:
        Returns the key

      • getId

        public String getId()
        Returns:
        Returns the id

      • getWhiteList

        public List<String> getWhiteList()
        Returns whitelist
        Returns:
        Whitelist entries

      • getWhiteListString

        public String getWhiteListString()
        Returns whitelist String
        Returns:
        Whitelist String

      • getBlackListString

        public String getBlackListString()
        Returns blacklist String
        Returns:
        Blacklist String

      • setWhiteList

        public void setWhiteList​(ArrayList<String> whiteList)
        Parameters:
        whiteList - The whiteList to set.

      • filter

        public String filter​(String urlString)
        Simple "indexOf" currentFilter for matching patterns. 
          rules for evaluation are as follows:
  1. if pattern matches in blacklist then url is rejected
  2. if pattern matches in whitelist then url is allowed
  3. url is rejected
        Specified by:
        filter in interface URLFilter
        Parameters:
        urlString - the URL string the filter is applied on
        Returns:
        the original URL string if the URL is accepted by the filter or null in case the URL is rejected
        See Also:
        URLFilter.filter(java.lang.String)

      • initialize

        public void initialize​(Element collection)
        Initialize Subcollection from dom element
        Parameters:
        collection - A DOM Element for use in creating the Subcollection

      • parseList

        protected void parseList​(List<String> list,
                         String text)
        Create a list of patterns from a chunk of text, patterns are separated with a newline
        Parameters:
        list - An initialized List to insert String patterns.
        text - A chunkl fo text (hopefully) containing patterns.

      • setBlackList

        public void setBlackList​(String list)
        Set contents of blacklist from String
        Parameters:
        list - the blacklist contents

      • setWhiteList

        public void setWhiteList​(String list)
        Set contents of whitelist from String
        Parameters:
        list - the whitelist contents