Announcement

Collapse
No announcement yet.

Network Diagnostics - Xarev's Guide to Traceroute

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [TUTORIAL] Network Diagnostics - Xarev's Guide to Traceroute

    note: I don't know what sub-forum this belongs in. It is mostly aimed at people playing the game who I believe mostly frequent General Discussion so I stuck it here.

    EDIT: If enough people ask me to I will also do a guide on cable modem diagnostics and perhaps home networking performance if there is enough interest. This helps to ensure everything at your location is functioning properly and can also aid greatly in diagnosing last mile connection problems.

    Introduction
    One of the most difficult problems with online multiplayer fps is network performance. Whenever you get a freeze, stutter, lag, rubber banding, sluggishness, or just general inaccurate feeling game play it can be really frustrating and most people immediately blame the server or net code. While this sometimes may be the case there is no real way to take an educated guess at what the problem may be unless you take a look under the hood.

    Luckily there are some awesome and free network tools that allow you to easily pinpoint problems. Unfortunately very few people including many hardcore fps players have any clue about this. Poor network performance is a frequent problem so whenever you have trouble the first thing you should do is test your connection to the server you're playing on and see what is happening. That is what I am going to show you how to do.

    I will try and keep this as brief as possible but it does require a little explanation and learning to make the acquired information useful. It will not be 100% technically accurate for the sake of brevity. Feel free to skim whatever you can if you want to try and quickly get the gist of the material. If you have trouble read through it again more carefully and further research linked terms to try and understand at least a little about what you're trying to find out.

    Internet Communication
    Computers communicating through the internet do so with a public IP address. They are all unique and routers use them to know which way to send your data through the mesh of the internet. At each point a router accepts a packet (chunk of data) and decides where to send it next. Each one of these points is called a "hop" or "node". These hops are your computer, your home router, your isp's routers, massive internet backbone routers, and high quality services closely connected to these information superhighways like game servers, data centers, etc. Your Dsl or Cable modem is NOT a router; though they are often combined with a router and switch as one device.

    Ping
    Ping is the tool traceroute is built on. It sends an echo request to the target and if the target is configured to respond it will send an echo reply back. Measure the amount of time this takes and you've got a great metric to measure network performance. If the reply doesn't come back at all when it should the target is likely dropping lots of packets including this little ping so you know where your UT packets are being lost.
    • Windows – Open Command Prompt and type ping 192.223.29.170
    • Linux – Open a Terminal and type ping 192.223.29.170 Hit Ctrl+C to stop it.


    Traceroute
    We can see exactly how this happens using a network tool called traceroute.

    Windows and Mac
    • PingPlotter It has a free 30 day trial and afterwards you can choose to use the free basic version which does everything you need.



    Linux
    • Mtr - Linode Guide Mtr in a terminal works well and this is a pretty good guide to show you how to install and use it. If anyone knows of a free and awesome GUI traceroute program for Linux please let me know. This is also really useful if you are a server administrator and SSH into your box to reverse traceroute to players to analyze their route if they're having problems.


    Trace!
    • PingPlotter - Fire it up and type into Target Name: 192.223.29.170 Hit the big green arrow and it should look something like this.


     
    Spoiler


    • Mtr – At a terminal type mtr 192.223.29.170 In this mode it will ping all the hops once a second indefinitely. Hit q to quit.



     
    Spoiler


    Your results will be mostly completely different except for the last two hops; however they will likely follow a similar format from the PP image.

    Hops
    1 If you're at home this will usually be your router. If you plug your computer directly into modem the first hop might be all the way to your isp or possibly a network device if you're in an apartment complex.
    2 Because I'm on a cable isp this is the CMTS 30 miles away.
    3 I think this is Comcast's DNS server (turns names like cnn.com into an IP address like 157.166.226.26)
    4-5 More Comcast routers moving it to the link that leaves Utah.
    6-7 Denver Comcast backbone routers
    8 Comcast hands it to Qwest
    9 Major internet backbone, doesn't reply with echo request so we have no information
    10 Also doesn't provide host name to give us a hint what it is (though I'll show you how to get information using just the IP address).
    11 Edge router of NFO server network.
    12 BloodBath Linux server.

    Statistics Configure
    • PingPlotter – Default settings don't show everything. In Menu goto View->Customize View and check everything. Also goto Edit->Options and under -Packet Loss- Packet Loss Scale at % enter 100, and under -Graph Scale- Scale to: 100 (this one is really handy and you should change it to a number relative to your target like if you're just monitoring your connection to your isp, set it to 40 and if you are getting big spikes that is probably terrible.)
    • Mtr – mtr <target> Continuous output in Terminal. Switch -r This does a “report” which pings each hop 10 times and delivers a summary of the results. Switch --no-dns cleans it up so you only see IP addresses.

    Columns
    • Err: This is usually going to just count the number of packets that failed to make it to their target and back (packet loss). If your computer is insane and has something terribly wrong with it you might get errors if it can't even send a packet out (unplug your ethernet cable in back for a second and watch).
    • PL%: Percentage of packets loss. (number of packets lost) / (total number of packets sent). Read the large bolded section “Packet Loss” down below.
    • IP: Internet Protocol address. The unique public internet address that identifies the node.
    • Name: Host name of node, there are tons of useful hints in these.
    • Avg, Min, Max, Cur: Average, Minimum, Maximum, Current - These are statistics based on the "Focus Time" setting in pingplotter. Usually you want to set it to whatever amount of time your graph is displaying.
    • In Mtr you have Host, Loss%, Sent, Last, Average, Best, Worst, Standard Deviation and the “focus time” is probably always the total sent.



    All of those ms values in those columns tell a highly accurate story of what is likely going on with your Unreal Tournament packets going back and forth between you and your target. A millisecond is 1/1000th of a second. 1000 ms is 1 second. So now to explain the significance of ms values for each hop.

    Hop
    1. “home”, 0-1ms - If this is a device in your home or right outside it should be a very low number, 0 or 1, and it should never go above that. If it does something is probably wrong and you should investigate. If you connect your computer directly into your modem (and is not also a router) this first hop could be all the way to your ISP in which the next bullet point would apply to you. Note: If you're connecting through wireless stop it, get an ethernet cable of whatever length necessary and hook up directly with no latency, packet loss, signal problems, bandwidth problems, or interference.

    2 “last mile”, 9ms - A decent time to get from my router and travel 35 miles to Comcast. This is the last mile hop that has been a constant technological problem for decades. If your isp sucks this will range from the low up to 2000ms and / or you will get packet loss often. If this happens too often you should complain to your ISP after first testing your home equipment to make sure all is well. If your connection to your ISP is awesome this will fluctuate only a tiny bit (like 2-5ms or none at all) and you'll never get pl here. This is a major source of trouble for many people and should be your initial focus. If your connection is bad from the start you're already screwed and nothing else is going to completely overcome this.

    3-5 “ISP internal” - Your ISP's internal network should be pretty good and you shouldn't often see a lot of variation, latency between hops or pl here though it is not unusual to have a little occasionally. Portions of this route will likely change depending on your target. You might have chronic problems here with servers to your East but have no trouble at all with ones to your West, etc. This is sometimes the source of “I only have problems when I connect to X server, therefore X server is the problem”; which may not be the case. This same issue may also happen in “Cloud” routes noted next.

    6 “Cloud”: Your situation may be be quite different but once the packet leaves your ISP's home network where you live it is generally out of their hands and if you get a bad route to your target at this point there often isn't much you can do about it except find a different server or hope it is better later. The route and performance can be quite variable. As routers are attacked and countermeasures are taken, traffic fluctuates, you might get a different experience from day to day or even hour to hour.

    12. Target: Generally you'll make it to the game server within 6-15 hops. If it is more than that it is probably a bridge too far. More hops is more prone to problems, less is usually better though I've seen excellent 15 hop routes with no pl and a decent ping work just fine but it is somewhat rare. You usually want ping under 65 for decent game play and under 40 is definitely optimum. Between 65-80 you will usually not have the best experience unless your route is really good (no pl and very small jitter end to end). Higher end players typically demand greater performance with higher consistency and tighter tolerances as a tiny stutter or inaccuracy can have a huge impact at a critical moment of a game. More casual players can generally get away with playing with much more forgiving specs though it may not be as much fun when it gets too bad.

    Packet Loss
    Note: Use command netstats in console of Unreal Tournament to toggle a handy little GUI widget that shows pl, ping, jitter, and traffic volume in and out while you are playing. I try to always have this up and cross reference what it is doing with other tools.

    Determining packet loss is not as simple as just reading the number in the column. Pl is counted as happening when traceroute does not receive an echo reply from the target. That could happen for many reasons such as: the router is configured to not respond, the router is configured to use “ICMP rate limiting” which means it will only reply to a certain number of echo requests per second and so sometimes you'll be the “odd man out” and won't get a reply. Evidence of this is usually a hop that consistently reports 1-5% pl over time and slowly fluctuates up and down a little but otherwise appears normal (no or far less reported pl after this hop).

    So how do you tell what is “real” pl? Well if a certain hop is dropping packets and not replying to echo requests then it should also not be forwarding packets to the next hop. So that means all (or at least most, sometimes a packet or two will sneak through) hops passed the bad hop will report pl as well because they never received the echo request because it was already dropped. It looks like this. Notice the pl at 9 and beyond is greater than 8%. Notice the pl at 11-13 is 36%, 54%,54% but then drops back down to 9% for 14-16. That is probably strong rate limiting and there may not be any pl in those hops. But hop 9 or 8 to 9 there is definitely a problem as the pl continues on all the way to the end.

     
    Spoiler


    Pl usually happens in little concentrated bursts in which 1-5, 20, 40, 60, or even 100% of packets are lost in a period of 5ms, or 100ms, or if its incredibly bad 500ms of total loss. It doesn't take much pl to make the game extremely unplayable. Sustained pl > 2% is generally considered “bad”. If you have a second monitor in which you can have traceroute running while you play you can observe what it looks like while you're playing and quickly get a sense of what indicators correlate with game play experience. You'll likely find it doesn't take much to devastate it, especially if it originates in your first hop to ISP.

    Pictured: Massive first hop (last mile) pl to isp. Needless to say this is far beyond unplayable.

     
    Spoiler


    Packet loss from your target. Generally the game server will either reply 100% or not at all. If you are a server admin and your target is a client's home router or computer you should normally get 100% pl because it is a default security policy to not respond to pings that didn't originate from their home computer. You can usually reach the hop before that and that is usually enough to test their ISP route up until the last hop to their physical location. Though in that case it would likely exclude the critical “last mile” portion.

    Server IP - So how do you get the server's IP address? This can be kind of tricky because you only communicate with the game server using UDP. UDP is a connectionless protocol so using a tool to list TCP connections isn't going to show you where the bulk of your traffic is headed. If somebody knows an easier way to do this please let me know, however the easiest way I have found is to do packet sniffing. This is kind of ridiculous but what else can you do? So to do this you need some sniffing software.

    Windows, Mac, Linux
    • Wireshark Free and relatively easy to use.
    • If you already have another sniffer installed and know how to use it, absolutely use that, no need for anything else.


    This is really fast and simple but all we have to do is first select an interface (your network connection).

     
    Spoiler


    Highlighted in the green box, as soon as you double click that it will begin capturing. You can go ahead and do that then in Menu goto Capture->Stop. We're not quite ready to capture yet. Open UT, click the gear (options)->System Settings->Tab(General)->Resolution pick something as small as possible. Change Display Mode to Windowed, click OK. Now connect to your target server you want the IP from and join a game, either as spectator or player, start a game if you need to. You don't actually have to do anything in the game, just as long as its loaded and you are sitting in a map you're good. Now in Wireshark Menu goto Capture->Start. We only need a second of traffic so start it and stop it fairly quickly while you are still sitting in the game. You can close out UT now if you'd like but may want to wait a second to make sure you got what you're looking for. Make sure you are at the end of the log if you have a bunch of data as this will show what we did at the end right before you stopped capturing. You should notice an enormous amount of traffic going to a particular target (Destination column).

    Notice Source column shows 192.168.1.11 (my IP address on my local network) and Destination column shows 173.199.66.15.choopa.net (target game server). That is a packet being sent out. Notice the reverse of this shows the destination and source column records are switched. That is a packet being received.

     
    Spoiler


    Now if you scroll to the right and look at the other columns this will give us some more hints we're looking at the right traffic. If you don't have anything else going on with your computer, just UT and Wireshark were running, its unlikely you will have much else. Also game traffic is extremely heavy so it will be the vast bulk of what you see. But just to verify you will notice Protocol column shows UDP, Length (entire packet) is only 48-60, and then Len (just the actual UT data) shows a small 6-15 bytes. The packet sizes can vary wildly depending on server tick rate and if the server is doing something funny like some of the EPIC servers send less frequent massive packets of between 300-500 bytes containing many frames of data.

     
    Spoiler


    Ok so the IP address 173.199.66.15 (just the numerical part) is all we're interested here. You can go ahead and enter that into Ping Plotter or Mtr and traceroute to your target. Or you can take the IP address and try to find some GeoLocation data, which I'll show you how to do next.

    GeoLocation
    This is a really easy section. All you need is the IP address and you can plug it in to various geo-location services to see what data it can provide you. It isn't always accurate and sometimes can be very misleading, especially if it involves somebody who wants to mask their identity or is a sensitive service. So here is my favorite website for this:

    • IP Location My favorite so far. Notice it has your IP already entered in and tells you where it thinks you're located. Does it find the location of your ISP? Is it wrong? Scroll down and you'll notice it gives you results from several different databases. Are the results the same? Different? There is also a ton of FAQ's on this site that teaches you all kinds of interesting stuff if you want to know more about all this.



    So try 173.199.66.15 and see what it gives us. Dallas, Texas. ISP is Gameservers.com. It also shows Choopa, LLC which could be a registry, or upstream ISP for GameServers.com, who knows. But this information is quite revealing and tells us what we need to know.

    So now lets try 104.197.84.5, this is an IP I got from Epic Central. You'll find it says Mountain View, Cali. Hmm that doesn't make sense. The ISP is Google Inc. If you traceroute to this address you'll see the route may hint that it goes through Ashburn, Virginia, which certainly is not close to California. What is happening here is the game server is provided through Google compute services and they use VPN technology to create a tunnel through the interwebs containing encrypted, compressed, and maybe some other way tortured game data to and from locations unknown. Look at the hostnames of the routes, the ms time it takes to go from one hop to another and consider that it all may be lying to you. My best guess is I think this server was probably in Ohio but who knows.

    Conclusion
    Hopefully you were able to get something useful out of this and can now test network performance between you and various servers. If you have trouble with any parts or have special problems I'll try to respond when I can and maybe update the guide to be more clear in certain areas.
    Attached Files
    Last edited by Xarev; 06-11-2016, 06:42 PM. Reason: fixed title
    Xarev's Guide to Traceroute
    Add me on Steam for duels

    #2
    Hey Xarev. Great write-up. My question would be more so, Once you do find out why a connection to a certain hub isn't as great as it could be (whether as the admin of the hub, or the client), what can you really do with the information? You can't really fix it can you?
    ON-GOING PROJECTS: DM-BloodCovenant, DM-Campgrounds (Absolute)
    CONTACT & TWITCH: Absolute Discord
    , Twitch

    Comment


      #3
      This is extremely informative and very well written. Nicely done, Xarev.

      Comment


        #4
        Originally posted by Trinatek View Post
        Hey Xarev. Great write-up. My question would be more so, Once you do find out why a connection to a certain hub isn't as great as it could be (whether as the admin of the hub, or the client), what can you really do with the information? You can't really fix it can you?
        If you get a change to your routing it's a great help. How do you do that? I don't know. In UT3 I had a server in Dallas through NFO. Some nights my ping would be 20 higher than normal and it would be terrible to play so I'd email NFO about it and send them some diagnostics that they needed, they'd email internap, and internap would change my routing in Dallas to use different hops. My ping would drop by 20 back to a favorable range. After enough of these instances, NFO was able to get internap to permanently change my routing to the Dallas server. As the owner of the server I was in direct contact with them so I could get them to do this, but I don't know how a person playing on someone else's server could improve their connection

        Thanks for the post, Xarev!

        Comment


          #5
          Originally posted by Trinatek View Post
          Hey Xarev. Great write-up. My question would be more so, Once you do find out why a connection to a certain hub isn't as great as it could be (whether as the admin of the hub, or the client), what can you really do with the information? You can't really fix it can you?
          Depends on who controls the router. If it's your local isp you can complain to them and if they are good they will do everything they can to fix it. If it's a hop very close to your local isp (connecting to another state usually) you might be able to complain to your isp and if it is bad enough they might complain to them or change the route. If it's a large backbone router you are mostly at the mercy of them doing the best they can and this will usually involve mitigating DDOS attacks or congestion. If it's close to the server do like Legionz said and the complaints will be handled through the grapevine.

          It's all largely dependent on if the people you contact care or not and their circumstances. Sometimes they care but there is nothing they can do (local isp infrastructure is terrible but is not cost effective to fix right). A lot of the time they don't care or think you're an idiot who just needs to power cycle your modem. Effective complaining is half the battle and sometimes you have to go through a lengthy, frustrating process to finally get the right message to the right person before anything gets done.

          On the other hand sometimes a local tech will come out and replace a bad filter up on the pole and it works great again. Good luck!
          Xarev's Guide to Traceroute
          Add me on Steam for duels

          Comment


            #6
            Good guide. One thing I would add, the in-game ping might not only be related to the actual ISP (internet connection) but als oto the local LAN or Wi-Fi/WLAN connection. If you're in a household sharing the internet with a router, connections are queued and prioritized (to speed up the process for everyone). This can eventually result into lags/stutters (async data, warping players etc.). The same happens with WiFi/WLAN. It's really important to get a decent connection to your local modem/router. If you need to bridge a long distance, you should consider using cable or repeaters. A bad connection to your internet gateway device is crucial to a good internet connection, such ping could reach a high value easily and will also fluctuate. Additionally, some cheap wireless adapters may overheat, in such cases the connection can get up to seconds. Using good hardware is the rule #1 .

            Also, some people use bandwidth as a reason for bad ping/latency. Higher bandwidth doesn't make anyone connect faster to something. It is just the amount of data wich can be send at once. A package still has to travel the same distance, the same route.
            ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
            ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

            Comment


              #7
              Originally posted by RattleSN4K3 View Post
              Good guide. One thing I would add, the in-game ping might not only be related to the actual ISP (internet connection) but als oto the local LAN or Wi-Fi/WLAN connection. If you're in a household sharing the internet with a router, connections are queued and prioritized (to speed up the process for everyone). This can eventually result into lags/stutters (async data, warping players etc.). The same happens with WiFi/WLAN. It's really important to get a decent connection to your local modem/router. If you need to bridge a long distance, you should consider using cable or repeaters. A bad connection to your internet gateway device is crucial to a good internet connection, such ping could reach a high value easily and will also fluctuate. Additionally, some cheap wireless adapters may overheat, in such cases the connection can get up to seconds. Using good hardware is the rule #1 .

              Also, some people use bandwidth as a reason for bad ping/latency. Higher bandwidth doesn't make anyone connect faster to something. It is just the amount of data wich can be send at once. A package still has to travel the same distance, the same route.
              Absolutely, I've struggled at home trying to balance QoS for the family and my gaming. I'm building a home Linux box that will be my router, file server, firewall, etc. because this Netgear piece of **** just isn't going to cut it. Sure you can get an amazing router for $200 but at that price might as well just turn an old computer into the ultimate router.

              I added an edit to OP earlier as I may in the future do at least a cable guide as I unfortunately have extensive experience dealing with those problems and also a home networking guide because there is a great deal of importance as you've said there as well.
              Last edited by Xarev; 06-08-2016, 07:41 PM.
              Xarev's Guide to Traceroute
              Add me on Steam for duels

              Comment


                #8
                Internet Network Performance is nice to know and have a better understanding of, but what about Network Performance on the PC/System?
                X58 i7 970 4GHz HT off, GTX 1070, 144Hz 1ms, 1080p in-game res.

                Comment


                  #9
                  Originally posted by 213 View Post
                  Internet Network Performance is nice to know and have a better understanding of, but what about Network Performance on the PC/System?
                  Another guide for another day.

                  Note: Could a mod fix the title please. I put [TUTORIAL] in then forgot to take it back out after I selected tutorial option.
                  Xarev's Guide to Traceroute
                  Add me on Steam for duels

                  Comment


                    #10
                    This honestly should be stickied somewhere in my opinion.

                    Comment


                      #11
                      Originally posted by Xarev View Post
                      Note: Could a mod fix the title please. I put [TUTORIAL] in then forgot to take it back out after I selected tutorial option.
                      Advanced edit mode of the first post (Post#1: "Edit post" -> "Go advanced").
                      ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
                      ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

                      Comment


                        #12
                        Originally posted by KmKz View Post
                        This honestly should be stickied somewhere in my opinion.
                        Thanks, I think it could be useful in the long run.

                        Originally posted by RattleSN4K3 View Post
                        Advanced edit mode of the first post (Post#1: "Edit post" -> "Go advanced").
                        Thanks!
                        Xarev's Guide to Traceroute
                        Add me on Steam for duels

                        Comment


                          #13
                          On the Epic Belgian servers, I have been experiencing a 1% Packet Loss In and a max deviation of ping of 100-150. I don't seem to every experience packet loss out and it's only started happening recently. The game is playable, but far from smooth recently.

                          Is this likely to be because of my connection or the server?
                          Current Main Issues: Tri-Rox (Remove), Scoreboards lack player stats,Team Balance.

                          My Pre-Alpha Highlights 2016 to early 2017 https://www.youtube.com/watch?v=IKlKQ349o8A | http://plays.tv/u/Smurgl

                          Comment


                            #14
                            Originally posted by Smurgl View Post
                            On the Epic Belgian servers, I have been experiencing a 1% Packet Loss In and a max deviation of ping of 100-150. I don't seem to every experience packet loss out and it's only started happening recently. The game is playable, but far from smooth recently.

                            Is this likely to be because of my connection or the server?
                            I'm assuming you're talking about netstats. If you see a flickering 1% packet loss that is not optimum but shouldn't be too terrible. If it is sustained that's a bit worse and might cause a lot of small glitches. Your max deviation is quite high and if a lot of packets have high latency I would suspect that is your main problem. If packets arrive too late they are no different than packet loss.

                            This is all just a guess though, the point of the guide is for people to traceroute their path to the server, analyze the results, and find out exactly where the problem is.
                            Xarev's Guide to Traceroute
                            Add me on Steam for duels

                            Comment


                              #15
                              Bumping this thread. Having issues lately with outgoing PL to the Atlanta Absolute server.

                              EDIT: Installed Wireshark, and ran it while booting up the game, no servers came up in Hubs.
                              Last edited by 213; 04-07-2019, 12:31 PM.
                              X58 i7 970 4GHz HT off, GTX 1070, 144Hz 1ms, 1080p in-game res.

                              Comment

                              Working...
                              X