kendra/aws.sdk.kotlin.services.kendra.model/WebCrawlerConfiguration/Builder

Builder

Properties

var authenticationConfiguration: AuthenticationConfiguration?

Configuration information required to connect to websites using authentication.

The 'depth' or number of levels from the seed level to crawl. For example, the seed URL page is depth 1 and any hyperlinks on this page that are also crawled are depth 2.

maxContentSizePerPageInMegaBytes

var maxContentSizePerPageInMegaBytes: Float?

The maximum size (in MB) of a web page or attachment to crawl.

maxLinksPerPage

var maxLinksPerPage: Int?

The maximum number of URLs on a web page to include when crawling a website. This number is per web page.

maxUrlsPerMinuteCrawlRate

var maxUrlsPerMinuteCrawlRate: Int?

The maximum number of URLs crawled per website host per minute.

proxyConfiguration

var proxyConfiguration: ProxyConfiguration?

Configuration information required to connect to your internal websites via a web proxy.

urlExclusionPatterns

var urlExclusionPatterns: List<String>?

A list of regular expression patterns to exclude certain URLs to crawl. URLs that match the patterns are excluded from the index. URLs that don't match the patterns are included in the index. If a URL matches both an inclusion and exclusion pattern, the exclusion pattern takes precedence and the URL file isn't included in the index.

urlInclusionPatterns

var urlInclusionPatterns: List<String>?

A list of regular expression patterns to include certain URLs to crawl. URLs that match the patterns are included in the index. URLs that don't match the patterns are excluded from the index. If a URL matches both an inclusion and exclusion pattern, the exclusion pattern takes precedence and the URL file isn't included in the index.

urls

var urls: Urls?

Specifies the seed or starting point URLs of the websites or the sitemap URLs of the websites you want to crawl.

Functions

authenticationConfiguration

fun authenticationConfiguration(block: AuthenticationConfiguration.Builder.() -> Unit)

construct an aws.sdk.kotlin.services.kendra.model.AuthenticationConfiguration inside the given block

proxyConfiguration

fun proxyConfiguration(block: ProxyConfiguration.Builder.() -> Unit)

construct an aws.sdk.kotlin.services.kendra.model.ProxyConfiguration inside the given block

urls

fun urls(block: Urls.Builder.() -> Unit)

construct an aws.sdk.kotlin.services.kendra.model.Urls inside the given block