Start a new topic

Using Crawlera with C# update

Our friend Chris suggested this new approach for using Crawlera with C#:



Hi, quick suggestion.

The code provided here is quite outdated using HttpWebRequest from .NET 1.1 introduced back in 2003. More modern C# code is more easily understood and can be more performant using async methods and the utility provided by a class such as HttpClient.

I have extended HttpClient to implement a Crawlera client, which you may find helpful to list, as in my experience few people need to use HttpWebRequest these days, and with extending HttpClient it is much more concise and easier to understand what is going on.

I also noticed that the link to the C# examples is broken from your examples page, and can only be found by going via examples in other languages.

public async Task Main(string[] args)
{
string apiKey = "SOME_API_KEY";
using (var client = new CrawleraClient(apiKey))
{
HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");
}
}

public class CrawleraClient : HttpClient
{
public string ApiKey { get; set; }
public static WebProxy Proxy = new WebProxy("http://proxy.crawlera.com:8010");

public CrawleraClient(string apiKey)
: base(new HttpClientHandler() { Proxy = Proxy, PreAuthenticate = true })
{
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
ApiKey = apiKey + ":";
DefaultRequestHeaders.ProxyAuthorization = new AuthenticationHeaderValue("Basic", Base64Encode(ApiKey));
}

public static string Base64Encode(string apiKey)
{
var plainTextBytes = Encoding.UTF8.GetBytes(apiKey);
return Convert.ToBase64String(plainTextBytes);
}
}

Thanks

Chris


1 person likes this idea

Chris,

I'm trying to test your script:


 

using System;

using System.IO;

using System.Net;

using System.Threading.Tasks;


namespace ProxyRequest

{

class MainClass

{

public async Task Main(string[] args)

{

string apiKey = "<my Api Key here>";

using (var client = new CrawleraClient(apiKey))

{

HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");

}

}


public class CrawleraClient : HttpClient

{

public string ApiKey { getset; }

public static WebProxy Proxy = new WebProxy("http://proxy.crawlera.com:8010");


public CrawleraClient(string apiKey)

base(new HttpClientHandler() { Proxy = ProxyPreAuthenticate = true })

{

ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;

ApiKey = apiKey + ":";

DefaultRequestHeaders.ProxyAuthorization = new AuthenticationHeaderValue("Basic"Base64Encode(ApiKey));

}


public static string Base64Encode(string apiKey)

{

var plainTextBytes = Encoding.UTF8.GetBytes(apiKey);

return Convert.ToBase64String(plainTextBytes);

}

}

}

}  


But I'm having this error:


crawlera_c2.cs(19,35): error CS0246: The type or namespace name `HttpClient' could not be found. Are you missing an assembly reference?


I've also tried adding: 


using System.Net.Http;


But still the error, even more issues appear.


Do you have any suggestion?

Thanks,


Best regards,


Pablo Vaz

Scrapinghub Support Engineer

<Chris response>


Hi Pablo 

 

 

No problem, it may be because the client is async and your main method is not async.

Here’s a full example which can be used in a Visual Studio console application project. It will write the response code to the console.

An assembly reference needs to be added for System.Net.Http. I think nothing else is needed more than the default references.

 

 

using System;

using System.Net;

using System.Net.Http;

using System.Net.Http.Headers;

using System.Text;

 

 

namespace CrawleraClientExample

{

 public class Program

  {

 static void Main(string[] args)

  {

 string apiKey = "<my Api Key here>";

 using (var client = new CrawleraClient(apiKey))

  {

 var response = client.GetAsync("https://httpbin.org/ip").Result;

 

 

 Console.WriteLine($"Request finished with status code: {response.StatusCode}");

 Console.ReadLine();

   }

  }

  }

 

 

 public class CrawleraClient : HttpClient

  {

 public string ApiKey { getset; }

 public static WebProxy Proxy = new WebProxy("http://proxy.crawlera.com:8010");

 

 

 public CrawleraClient(string apiKey)

  : base(new HttpClientHandler() { Proxy = Proxy, PreAuthenticate = true })

  {

 ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;

  ApiKey = apiKey + ":";

  DefaultRequestHeaders.ProxyAuthorization = new AuthenticationHeaderValue("Basic", Base64Encode(ApiKey));

  }

 

 

 public static string Base64Encode(string apiKey)

  {

 var plainTextBytes = Encoding.UTF8.GetBytes(apiKey);

 return Convert.ToBase64String(plainTextBytes);

  }

  }

}

<Chris response>


Hi

 

 

You need to make sure you have referenced System.Net.Http.dll correctly, then it will work; the reference is missing.

This may not work if you are using a very old version of .NET framework – but if you are running .NET framework 4.5 or above it should be fine.

 

 

Chris

 Please help, I can't able to get any results using this code and its keep erroring

 

image

Login to post a comment