Similar pages can be excluded from crawling by specifying URLs to be judged as similar pages using regular expressions in the "Rules for determining similar pages" section of the scan settings.
Only one page out of the URLs matching the regular expression is retrieved, and all other pages are excluded.
For specific configuration examples, click the following buttons on the screen.
In addition to the above, the following cases are also available,
We will describe them below.
Example:
- If the query part contains page parameters
https://example.com/list.php?p=1
https://example.com/list.php?p=2
https://example.com/list.php?p=3
- Regular expression to specify
https://example.com/list.php\?p=[0-9]+
-Specifying Conditions by HTTP Method
In addition to URLs, you can also set exclusion rules by specifying the HTTP method (e.g., GET, POST).
１．On the settings screen, click > "Enter by text list".
２．In the input box, enter the details in the format [HTTP Method],[URL] (comma-separated).
Input Format & Example
|
* The URLs of the target sites can be output from "Report Download > Crawl Report > Crawl Results List".