🌎 : Web Traffic Debugging

Back in the good old day you required software to do this, and the one I used was called Fiddler which essentially would create a proxy server on your device, send all the data though that as a man in the middle attack and tell you what was going on.

Warning: Fiddler will create a "man in the middle" attack to be able to monitor traffic inside the HTTPS session, please remember that this is also what threat actors do as well, if you are in a "domain" remember as well that a "subordinate" certificate deployed by your sysadmin can FORCE your browser to trust a certificate that is not the valid certificate.

Information: The padlock in your browser does NOT mean your traffic is secure and you are talking to the "original" host, for that you need to check the HTTPS thumbprint, the padlock simply means you have a secure session, not a secure session with the original host server!

If you use Fiddler then you will need to trust the root certificate and for end to end testing that is more helpful to figure out the root cause, also remember that Fiddler can capture non-web traffic whereas a browser can only capture traffic for that session, this means Fiddler is not dead, its just got a different use case, there are now 2 version for this software Fiddler Classic which is a local application with local logging, and Fiddler Everywhere which capture traffic that can be viewed "in the cloud" and "cross platforms" 

Fiddler also allows you to re-write and intercept headers and data as well, which the development tools does not allow you to do either, so based on your requirement you may still require Fiddler.

However these days all you require is a modern web browser and the F12 key, for developer options and it will do it right there from the browser, all hail megaton and the progress of technology, this guide will cover both.

Traffic : Browser Debugging

Lets start with the browser debugging as that one is easier, as its simply a press of F12, so when you press F12 you will end up on the "elements" page, this shows you the elements of the website you are currently looking at as you see below:



You need to select the network option:


This will then look like this, the first red box is the stop and start capture options, then the one called "preserve log" will ensure when you navigate though the website you will not loose the history, very helpful for many websites so ensure this is ticked....

Network : The Setup

Then you have the timeline on the top in green, and beneath that you have the "traffic" or "endpoints" you are visiting......



This is where the data can be seen and what is going on, however there are some items you need to know, first lets get some data to look at, so select the www.google.co.uk as shown below:


The default view is the response, which will be in HTML and that will be the Google search engine in the raw HTML to confirm this click the preview button and then you will see the HTML without images as they are in a different request in debugging:



Developer Tools : Headers

However the most useful view is "headers" this will show you how the website as been loaded and look like this, here you can see the status code as well, a very helpful diagnostic tool in troubleshooting website issues


Raw Header Data

So for Google this is what we get, this is the raw data and it will be expanded later on in this guide......

    1. Request URL:
      https://www.google.co.uk/
    2. Request Method:
      GET
    3. Status Code:
      200
    4. Remote Address:
      10.245.161.117:8080
    5. Referrer Policy:
      origin
  1. Response Headers
    1. accept-ch:
      Sec-CH-UA-Platform
    2. accept-ch:
      Sec-CH-UA-Platform-Version
    3. accept-ch:
      Sec-CH-UA-Full-Version
    4. accept-ch:
      Sec-CH-UA-Arch
    5. accept-ch:
      Sec-CH-UA-Model
    6. accept-ch:
      Sec-CH-UA-Bitness
    7. accept-ch:
      Sec-CH-UA-Full-Version-List
    8. accept-ch:
      Sec-CH-UA-WoW64
    9. alt-svc:
      h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
    10. cache-control:
      private, max-age=0
    11. content-encoding:
      br
    12. content-length:
      46064
    13. content-security-policy-report-only:
      object-src 'none';base-uri 'self';script-src 'nonce-UGsol6pD_Ax73GVPxVBqsw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    14. content-type:
      text/html; charset=UTF-8
    15. cross-origin-opener-policy:
      same-origin-allow-popups; report-to="gws"
    16. date:
      Tue, 09 May 2023 12:10:40 GMT
    17. expires:
      -1
    18. origin-trial:
      Ap+qNlnLzJDKSmEHjzM5ilaa908GuehlLqGb6ezME5lkhelj20qVzfv06zPmQ3LodoeujZuphAolrnhnPA8w4AIAAABfeyJvcmlnaW4iOiJodHRwczovL3d3dy5nb29nbGUuY29tOjQ0MyIsImZlYXR1cmUiOiJQZXJtaXNzaW9uc1BvbGljeVVubG9hZCIsImV4cGlyeSI6MTY4NTY2Mzk5OX0=
    19. origin-trial:
      AvudrjMZqL7335p1KLV2lHo1kxdMeIN0dUI15d0CPz9dovVLCcXk8OAqjho1DX4s6NbHbA/AGobuGvcZv0drGgQAAAB9eyJvcmlnaW4iOiJodHRwczovL3d3dy5nb29nbGUuY29tOjQ0MyIsImZlYXR1cmUiOiJCYWNrRm9yd2FyZENhY2hlTm90UmVzdG9yZWRSZWFzb25zIiwiZXhwaXJ5IjoxNjkxNTM5MTk5LCJpc1N1YmRvbWFpbiI6dHJ1ZX0=
    20. permissions-policy:
      unload=()
    21. report-to:
      {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
    22. server:
      gws
    23. strict-transport-security:
      max-age=31536000
    24. x-frame-options:
      SAMEORIGIN
    25. x-xss-protection:
      0
  2. Request Headers
    1. :authority:
      www.google.co.uk
    2. :method:
      GET
    3. :path:
      /
    4. :scheme:
      https
    5. accept:
      text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
    6. accept-encoding:
      gzip, deflate, br
    7. accept-language:
      en-GB,en-US;q=0.9,en;q=0.8,ca;q=0.7
    8. cache-control:
      max-age=0
    9. cookie:
      SEARCH_SAMESITE=CgQI9JcB; OGPC=19022622-1:19022972-1:; OGP=-19022972:; OTZ=7004585_52_56_123900_52_436380; SID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdktsjNKk83zTznNOUu7ZzwA.; __Secure-1PSID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdDrrSans4TMM5Hi-zyvFzuA.; __Secure-3PSID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdBmzKviTrGmnYeegu7ezdWw.; HSID=AkuUZrePyeGUtzaCb; SSID=Ab3hDFsafZBthW5P7; APISID=fVS6o6TQQE-GuB_V/AI53r6_kpR5PrdpLH; SAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; __Secure-1PAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; __Secure-3PAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; NID=511=eFnU-UiOiBBI31B_iENWXdPLxzczt-5RC2xsKZMQO-EYXSwkJRjdHKeSPFvmyOjjWqZefa1_UhcT1hUF1glSneu9XCiqqzDsItfiP_XPXLVzyWGFiHYAN9y8I-sGJn1quexilP3PkNL4bNRhtfi_CMVkZ9JywiW-5DDfJZ4oCvcN52PkH1YdR3QGM0-xFqOcED2oM6LKmHbVfgivhOe7eBTAsSyQ7U_YPA; 1P_JAR=2023-5-5-12; AEC=AUEFqZekk-KScCYBtwXuXRKhQzAf-63NrmwqxOEYSsknnNFUKDMF9EwCM5M; __Secure-ENID=12.SE=obg5pS6C7PAiXR2atpDfwLR9Yh5V0FFvNHYM-8zSViBZb9kpg_mYwEMMw9Ef_3U5DhsUlpHkWvXVICGW9YJMp6qy3-iB_-vSQVx_7ZsAlXfBgjfi1NvkIw3mv5XRTsb6VAw8qIq56a-RheBlguLBHr4gRgML85XLSOcoR4KmGn9hdF8AUqgDZNdtoXGEFYXgfRwzJ69UpY2w3bydcnkHaw--OztMINMKvef2i_Qgm8aQnI2EdGzgZ-V1qOSdlBWvuUypPzOUMGChtGETaUqMbivkutF0Fk06Se3SVCy_roO_HOHRs68yyzDr4VXmXtspjOhhex4
    10. sec-ch-ua:
      "Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"
    11. sec-ch-ua-arch:
      "x86"
    12. sec-ch-ua-bitness:
      "64"
    13. sec-ch-ua-full-version:
      "112.0.5615.138"
    14. sec-ch-ua-full-version-list:
      "Chromium";v="112.0.5615.138", "Google Chrome";v="112.0.5615.138", "Not:A-Brand";v="99.0.0.0"
    15. sec-ch-ua-mobile:
      ?0
    16. sec-ch-ua-model:
      ""
    17. sec-ch-ua-platform:
      "Windows"
    18. sec-ch-ua-platform-version:
      "3.0.0"
    19. sec-ch-ua-wow64:
      ?0
    20. sec-fetch-dest:
      document
    21. sec-fetch-mode:
      navigate
    22. sec-fetch-site:
      same-origin
    23. sec-fetch-user:
      ?1
    24. upgrade-insecure-requests:
      1
    25. user-agent:
      Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36
    26. x-client-data:
      CIS2yQEIpbbJAQjBtskBCKmdygEI0ObKAQiWocsBCIurzAEI7Z7NAQiFoM0BCL2izQEI0aLNAQifpM0BCNamzQEI3KbNAQiQqs0BCKSqzQEIiKvNAQi4q80BCM+uzQEY053NAQ==
      Decoded:
      message ClientVariations { // Active client experiment variation IDs. repeated int32 variation_id = [3300100, 3300133, 3300161, 3313321, 3322704, 3330198, 3347851, 3362669, 3362821, 3363133, 3363153, 3363359, 3363670, 3363676, 3364112, 3364132, 3364232, 3364280, 3364687]; // Active client experiment variation IDs that trigger server-side behavior. repeated int32 trigger_variati

So what exactly does this tell us, well when the website works it tells you how the website works and what your browser sends and receives, so lets focus on the handy headers and dtop some that are not helpful right now......

Helpful Headers : Analyse

This is all the headers that teach you the basics of how the browser and the server work lets go though some now, for the basics......

    1. Request URL:
      https://www.google.co.uk/ This is the URL you are visiting, pretty simple but it good in the main endpoint view to have this as you will get lots of these requests in that view, more a reference.
    2. Request Method:
      GET
        This is they type of request you are asking for, here you can see we have a GET which means its "getting" content, for all         the options please see the section below called "Request Method"
    1. Status Code:
      200 This is the HTTP status code, which again is below in the section called "HTTP Codes" in this instance a 200 is OK or more commonly called Success
    2. Remote Address
      131.888.161.117:3129 This is where the request as come from, here it can come from an impossible address from port 8080 which usually indicates a proxy server
    3. Referrer Policy:
      origin This tells you where the request came from, this is set to "origin" as your have visited Google directly and not come from another website to get to it, if you would like a full list, try this
  1. Response Headers
    1. content-security-policy-report-only:
      object-src 'none';base-uri 'self';script-src 'nonce-UGsol6pD_Ax73GVPxVBqsw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    2. cross-origin-opener-policy:
      same-origin-allow-popups; report-to="gws"
    3. server:
      gws
    4. strict-transport-security:
      max-age=31536000
    5. x-frame-options:
      SAMEORIGIN
    6. x-xss-protection:
      0
  2. Request Headers
    1. :scheme:
      https This says you are using HTTPS but this is the server confirming this as the request headers come from the remote server, so this confirms HTTPS
    2. accept-encoding:
      gzip, deflate, br This is the type of encoding or compression algorithm being used, here you can see we are using gzip compression, with the zlib deflate algorithm and finally we are using Brotli compression as well, usually this does not need to be factored in unless you are actively compressing via a proxy or gateway..
    3. cache-control:
      max-age=0 This is how long the data remains "fresh" as its 0 that means it expired immediately so if you refresh the page you get some new "fresh" data.
    4. cookie:
      SEARCH_SAMESITE=CgQI9JcB; OGPC=19022622-1:19022972-1:; OGP=-19022972:; OTZ=7004585_52_56_123900_52_436380; SID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdktsjNKk83zTznNOUu7ZzwA.; __Secure-1PSID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdDrrSans4TMM5Hi-zyvFzuA.; __Secure-3PSID=WAhKVpRoxtJfl_Ei5XF8BppUURYnrdIneGuE2rcVG-2fZKRdBmzKviTrGmnYeegu7ezdWw.; HSID=AkuUZrePyeGUtzaCb; SSID=Ab3hDFsafZBthW5P7; APISID=fVS6o6TQQE-GuB_V/AI53r6_kpR5PrdpLH; SAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; __Secure-1PAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; __Secure-3PAPISID=Y6BvTAlBitZ_CALE/AkoR1rd4POl4jvGyA; NID=511=eFnU-UiOiBBI31B_iENWXdPLxzczt-5RC2xsKZMQO-EYXSwkJRjdHKeSPFvmyOjjWqZefa1_UhcT1hUF1glSneu9XCiqqzDsItfiP_XPXLVzyWGFiHYAN9y8I-sGJn1quexilP3PkNL4bNRhtfi_CMVkZ9JywiW-5DDfJZ4oCvcN52PkH1YdR3QGM0-xFqOcED2oM6LKmHbVfgivhOe7eBTAsSyQ7U_YPA; 1P_JAR=2023-5-5-12; AEC=AUEFqZekk-KScCYBtwXuXRKhQzAf-63NrmwqxOEYSsknnNFUKDMF9EwCM5M; __Secure-ENID=12.SE=obg5pS6C7PAiXR2atpDfwLR9Yh5V0FFvNHYM-8zSViBZb9kpg_mYwEMMw9Ef_3U5DhsUlpHkWvXVICGW9YJMp6qy3-iB_-vSQVx_7ZsAlXfBgjfi1NvkIw3mv5XRTsb6VAw8qIq56a-RheBlguLBHr4gRgML85XLSOcoR4KmGn9hdF8AUqgDZNdtoXGEFYXgfRwzJ69UpY2w3bydcnkHaw--OztMINMKvef2i_Qgm8aQnI2EdGzgZ-V1qOSdlBWvuUypPzOUMGChtGETaUqMbivkutF0Fk06Se3SVCy_roO_HOHRs68yyzDr4VXmXtspjOhhex4 This is your cookie data, which is handy for cookie injection or troubleshooting caching issues, not always helpful but worth mentioning it for awareness.
    5. user-agent:
      Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 This the browser signature, here you can see this is Chrome v112.0 which is correct, and its coming from Windows on a x64 platform - this is a very helpful indicator to some issues as well, for example if someone tells you they are using Edge and you see this, you know technically they are wrong.

Request Method Lookup (Common Ones)

GET
The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.

HEAD
The HEAD method asks for a response identical to a GET request, but without the response body.

POST
The POST method submits an entity to the specified resource, often causing a change in state or side effects on the server.

CONNECT
The CONNECT method establishes a tunnel to the server identified by the target resource.

OPTIONS
The OPTIONS method describes the communication options for the target resource.

HTTP Codes (not all errors)

For a full list please use this has reference, here but the ones you need to know for general troubleshooting are:

200 OK
301 Moved Permanently
302 Found
304 Not Modified
307 Temporary Redirect
308 Permanent Redirect
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
405 Method Not Allowed
500 Internal Server Error
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout

That outlines the basics of how to use the endpoint section, however, remember what is outlined above, is one request out of many, and so using Google as an example is not particularly helpful for debugging, because Google is very rarely not available.

I understanding the fundamentals of how stuff works when it works is the key to understanding why it’s not working when it doesn’t work, usually you will be delving into debugging when the website doesnt work

It is important to note that there are two sides to debugging website traffic, you have the local side, which in this instance is the client for example Chrome as a Browser, then you have the server side which could also be called the remote side, and for many websites, not loading it’s a front end problem, however, depending on your fault, it’s not always possible to get the remote side diagnostic data like in the case of Google, however, that does not mean you cannot troubleshoot the problem partially, and then, if required work with the third-party vendor to a resolution.

Full session trace - Google Trace

First of, let’s get an example of a full section trace, to this point we have been looking at a single web request, so let’s take the full session for Google, I am choosing Google because it’s a pretty simple website that is not going to flood our session traced with data, so it will end up being pretty simple to go through, obviously the more complicated the website the more sessions you’re going to need to analyze for potential problems, but it’s always best to start simple.

<more to come>

Previous Post Next Post

Ω†Ω…ΩˆΨ°Ψ¬ Ψ§Ω„Ψ§ΨͺΨ΅Ψ§Ω„