com.norconex.collector.http.robot
Class RobotsTxt
java.lang.Object
com.norconex.collector.http.robot.RobotsTxt
- All Implemented Interfaces:
- Serializable
public class RobotsTxt
- extends Object
- implements Serializable
- See Also:
- Serialized Form
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
UNSPECIFIED_CRAWL_DELAY
public static final float UNSPECIFIED_CRAWL_DELAY
- See Also:
- Constant Field Values
RobotsTxt
public RobotsTxt(IURLFilter[] filters)
RobotsTxt
public RobotsTxt(IURLFilter[] filters,
float crawlDelay)
getFilters
public IURLFilter[] getFilters()
getCrawlDelay
public float getCrawlDelay()
Copyright © 2009-2013 Norconex Inc.. All Rights Reserved.