Yes, some sites or functions may not be crawled, or may be difficult to crawl.
Below is a summary of typical examples where crawling is possible, and where it is unsupported for automatic crawling or manual crawling respectively.
Supported cases for crawling (Typical examples)
-
Adoption of any of the following authentication methods:
- Basic authentication
- Digest authentication
- Form authentication (entering an ID and password into a web form)
- Accessible via the internet
- Accessible using the latest version of Google Chrome without any special configuration
- Use of Japanese or English input forms (*Note 1)
- Functions where the same operation can be executed repeatedly
- Pages requiring email receipt and verification
- Sites allowing concurrent duplicate logins from multiple browsers
- File upload and file download functions
*Note 1: When using the Generative AI option, 14 languages including Japanese and English are supported.
Unsupported cases for both automatic crawling and manual crawling (Typical examples)
- Sites requiring special environment configurations upon access (e.g., specific browser plugins required)
- Sites that prohibit concurrent multiple logins and reject subsequent login attempts
- Sites requiring Multi-Factor Authentication (MFA) (*Note 2)(*Note 3)
- Sites that only function on browsers other than Chrome
- Sites requiring authentication methods other than Basic authentication or Form authentication
- Sites that cannot be accessed from the internet (e.g., intranet sites)
- Sites with authentication requiring human cognition, such as CAPTCHA
- Sites requiring WebSocket communication (HTTP communication areas other than WebSocket
- communication are supported)
- Sites not compatible with TLS 1.2 or higher
- Web sites whose screen structures are not output as HTML elements (e.g., Flutter Canvas rendering)
- Sites where the URL becomes invalid once accessed (e.g., promotional campaign sites)
- Sites requiring drawing or manipulating lines and shapes on images
- Sites exceeding a data size of 8 MB per page
- Functions requiring a specified screen transition sequence for each access (e.g., Registration followed by Deletion)
- Sites requiring the use of camera or microphone functions
*Note 2: It may be possible to crawl post-login pages by using the Custom Header function to attach authenticated header information.
*Note 3: If TOTP authentication is used, it can be handled via manual crawling (TOTP secret key registration feature).
Unsupported cases for automatic crawling only, but supported for manual crawling (Typical examples)
- Systems requiring the input of specific values (e.g., distributor codes, amounts) (*Note 4)
- Sites requiring the download of terms and conditions in PDF format (*Note 4)
- Cases where different values must be entered each time during a new registration process (*Note 4)
- Cases where the maximum registrable limit, such as stock quantity, is fixed and the limit must be managed (*Note 4)
- Functions requiring the input of complex combinations of checkboxes or radio buttons (*Note 4)
- Cases where file uploads have restrictions on size or extension (*Note 5)
*Note 4: Crawling may be possible by using the Form Input Value Change and Re-crawl function.
*Note 5: If a file upload function is detected during automatic crawling, the system attempts to upload an image file. If there are restrictions on upload conditions (such as size or extension), manual crawling must be used.