使用 HttpWebRequest 登录 Https 网站后的 Htmlagilitypack？

Posted 2023-02-19

【中文标题】使用 HttpWebRequest 登录 Https 网站后的 Htmlagilitypack？

【英文标题】：Htmlagilitypack after login to Https Website with HttpWebRequest?

【发布时间】：2016-09-19 23:40:33

【问题描述】：

我想解析一些像pluralsight这样的html网站，例如（https://app.pluralsight.com/id？），那么我怎样才能首先以编程方式登录网站（不使用webbrowser控件）然后调用另一个url（例如：Pluralsight）并获得响应并使用 Htmlagility 包进行解析。

但是我已经写了登录代码，但是不知道下一步。

public class Login

    private CookieContainer Cookies = new CookieContainer();



public void SiteLogin(string username, string password)

    Uri site = new Uri("https://app.pluralsight.com/id?");

    HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(site);
    wr.Method = "Post";
    wr.ContentType = "application/x-www-form-urlencoded";
    wr.Referer = "https://app.pluralsight.com/id?";
    wr.CookieContainer = Cookies;
    var parameters = new Dictionary<string, string>
    "realm", "vzw",
    "goto","",
    "gotoOnFail","",
    "gx_charset", "UTF-8",
    "rememberUserNameCheckBoxExists","Y",
    "IDToken1", username,
    "IDToken2", password
;

    string input = string.Empty;
    using (var requestStream = wr.GetRequestStream())
    using (var writer = new StreamWriter(requestStream, Encoding.UTF8))
        writer.Write(ParamsToFormEncoded(parameters));

    using (var response = (HttpWebResponse)wr.GetResponse())
    

        if (response.StatusCode == HttpStatusCode.OK)
        
            //but I do not know the next step.
        
    


private string ParamsToFormEncoded(Dictionary<string, string> parameters)

    return string.Join("&", parameters.Select(kvp => Uri.EscapeDataString(kvp.Key).Replace("%20", "+")
    + "=" + Uri.EscapeDataString(kvp.Value).Replace("20%", "+")
    ).ToArray());

【参考方案1】：

您必须通过HttpWebResponse.GetResponseStream 获取响应流，然后通过HtmlDocument 的Load 方法加载文档。

var doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(response.GetResponseStream());
//further processing...

【讨论】：

但我想处理另一个发送请求，另一个 url（例如：app.pluralsight.com/library/courses/…）并获得响应并使用 Htmlagility 包进行解析。