How to decode an encoded HttpWebResponse?
hive-169321·@dotnetguru·
0.000 HBDHow to decode an encoded HttpWebResponse?
I have this piece of code to fetch a Page HTML from an URL, however the response content looks encoded. <center>  </center> Code: HttpWebRequest xhr = (HttpWebRequest) WebRequest.Create(new Uri("https://www.youtube.com/watch?v=somevideoid")); xhr.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; //xhr.CookieContainer = request.Account.CookieContainer; xhr.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"; xhr.Headers["Accept-Encoding"] = "gzip, deflate, br"; xhr.Headers["Accept-Language"] = "en-US,en;q=0.5"; xhr.Headers["Upgrade-Insecure-Requests"] = "1"; xhr.KeepAlive = true; xhr.UserAgent = "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"; xhr.Host = "www.youtube.com"; xhr.Referer = "https://www.youtube.com/watch?v=6aCpYxzRkf4"; var response = xhr.GetResponse(); string html; using (StreamReader reader = new StreamReader(response.GetResponseStream())) { html = reader.ReadToEnd(); } These are the response headers: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN Strict-Transport-Security: max-age=31536000 Content-Encoding: br Transfer-Encoding: chunked Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35" Cache-Control: no-cache Content-Type: text/html; charset=utf-8 Date: Sat, 24 Nov 2018 11:30:38 GMT Expires: Tue, 27 Apr 1971 19:44:06 EST P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=it for more info." Set-Cookie: PREF=f1=50000000&al=it; path=/; domain=.youtube.com; expires=Thu, 25-Jul-2019 23:23:38 GMT Server: YouTube Frontend Proxy The response generated by the server is in **br** encoding and I need to decode it. support for **br** encoding is not included in default system compression packages and you will have to install the Brotli.net nuget package. I added this in my code to cover the 3 main encoding types **gzip**, **br** and **defalte** HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse(); Stream responseStream = response.GetResponseStream(); if (response.ContentEncoding.ToLower().Contains("gzip")) responseStream = new GZipStream(responseStream, CompressionMode.Decompress); else if (response.ContentEncoding.ToLower().Contains("deflate")) responseStream = new DeflateStream(responseStream, CompressionMode.Decompress); else if (response.ContentEncoding.ToLower().Contains("br")) responseStream = new BrotliStream(responseStream, CompressionMode.Decompress);