Crawlera performs HTTPS requests using CONNECT method, transparently tunnelling packets over default HTTPS port 443 to and from the destination server in a secure session.
To fetch HTTPS web pages through Crawlera you need to download and install the certificate file for Crawlera Certificate Authority or disable SSL certificate verification in your HTTP client.
The Crawlera Certificate Authority can be downloaded here: crawlera-ca.crt.
Installing the certificate file on Windows 10
- Press the Win key + R hotkey and input mmc in Run to open Microsoft Management Console window.
- Click File and select Add/Remove Snap-ins.
- In the opened window select Certificates and press the Add > button.
- In Certificates Snap-in window select Computer account > Local Account, and press the Finish button to close the window.
- Press the OK button in the Add or Remove Snap-in window.
- Back in the Microsoft Management Console window, select Certificates under Console Root and right-click Trusted Root Certification Authorities.
- From the context menu select All Tasks > Import to open the Certificate Import Wizard window from which you can add Crawlera's certificate.
More details here.
Installing the certificate file on OS X
- Open Keychain Access window (Launchpad > Other > Keychain Access).
- Select System tab under Keychains, drag and drop the downloaded certificate file (or select File > Import Items... and navigate to the file).
- Enter the administrator password to modify the keychain.
- Double-click the Crawlera CA certificate entry, expand Trust, next to When using this certificate: select Always Trust.
- Close the window and enter the administrator password again to update the settings.
Installing the certificate file on Ubuntu
- Copy the downloaded certificate file to /usr/local/share/ca-certificates/ directory to make it implicitly trusted:
sudo cp crawlera-ca.crt /usr/local/share/ca-certificates/crawlera-ca.crt
Update stored Certificate Authority files:
Installing the certificate file into Mozilla Firefox browser
- Open Preferences, visit Privacy & Security tab, scroll down to Certificates section, click View Certificates... button to open Certificate Manager.
- Under Authorities tab click Import... button, navigate to the certificate file.
- In the opened window (You have been asked to trust a new Certificate Authority (CA)) check the first option Trust this CA to identify websites and click the OK button to finish importing the certificate.
- Click the OK button to save settings and exit Certificate Manager.
Installing the certificate file into Google Chrome browser
- Click the triple-dot icon in the top right corner and choose Settings.
- Scroll to the bottom of the page and click Advanced.
- In Privacy and security section choose Manage certificates.
- Next steps will depend on the operating system. In case of Mac OS, the previous action will open the Keychain Access -- see Installing the certificate file on OS X section above. In case of Windows, the Certificates application should appear -- select Trusted Root Certification Authorities tab, click Import... button, navigate to the certificate file, verify the import was successful and the installed certificate is displayed under Trusted Root Certification Authorities tab, close the Certificates window.
The minimal example to check if the CA is working for you is to try:
curl -x proxy.crawlera.com:8010 -U <API key>: --cacert crawlera-ca.crt https://httpbin.org/ip