We software developers are good at debugging code related issues but when it comes to issues that require fighting with infrastructure or network then we find ourselves in a difficult position. We can solve these issues if we know the right tool to use. I faced a similar position this week. I am starting a new series where every week or two I will write about a new tool that can help us debug these kind of issues.
This week I was debugging an issue where few requests to the destination server were timing out. These types of issues typically fall under networking errors and require you to use a network diagnostic tool. Most developers start diagnosing the issue using ping and traceroute tool. Both these tools are useful but you have to run them both together to debug the issue. Recently, I discovered MTR which combines ping and traceroute tools in a single tool. I found that most developers that I work with are not unaware of this tool so I decided to document it for future me and others.
mtr stands for My traceRoute. It is useful when you need to figure out number of hops to the destination server or latency at each hop. It also help you see packet loss at each hop so that you can narrow down the place where you might be facing issue. MTR collects information regarding the state, connection, and latency of the intermediate hosts. Thus giving a complete overview of the connection between two hosts on the network.
Installing mtr
To install mtr on Mac you can use brew package manager.
brew install mtr
Once you have installed mtr, you need to create two symbolic links
sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet
If you get either mtr: Failure to start mtr-packet: Invalid argument or mtr: Failure to start mtr-packet: Invalid argument error message that means you have not run above two commands.
For Linux
Ubuntu users
apt update && apt upgrade apt install mtr-tiny
Centos
yum update yum install mtr
Windows
Windows users can download from here.
Once you have done that you can run the mtr tool. We cover that in next section.
Using mtr
We use mtr to find number of hops to destination server and if there is any latency or packet loss at any hop.
To use mtr, you can run the following command.
sudo mtr google.com
The above command will run mtr will continuously in an interactive manner. The output is shown below
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.1 0.0% 10 3.7 3.2 1.2 10.6 3.1
2. abts-north-static-236.220.160.122.airtelbroadband.in 0.0% 10 7.2 10.9 5.6 36.0 9.7
125.18.20.101
3. 72.14.205.93 0.0% 10 7.4 9.9 6.1 19.7 4.6
4. 108.170.251.113 0.0% 10 9.4 10.3 8.3 21.1 3.8
5. 108.170.229.1 0.0% 9 9.2 9.1 8.1 11.1 1.1
6. del03s01-in-f14.1e100.net 0.0% 9 7.2 8.0 7.0 10.1 1.2
Please scroll to the right to see complete output.
The first column shows the IP or DNS of the server receiving the request. As you can see there are 6 hops before request is served by a Google server del03s01-in-f14.1e100.net. If you see 0% packet loss under the Loss% column throughout all the hops of the MTR trace, this indicates that there likely is not a networking issue between your computer and destination host. In the output shown above, we can see that there is no packet loss. This is good thing!
The table also shows the latency in milliseconds at each hop in terms of last, average, best, worst, and standard deviation. Last is the latency of the last packet sent, Avg is average latency of all packets, while Best and Wrst display the best (shortest) and worst (longest) round trip time for a packet to this host.
To exit out from this, you can type q
If you don’t want to see domain names under host and only want to see IP address then you can use -n option.
sudo mtr -n google.com
1. 192.168.1.1 0.0% 9 1.7 1.8 1.3 2.4 0.3
2. 122.160.220.236 0.0% 8 5.5 7.0 5.5 10.4 1.5
125.18.20.101
3. 72.14.205.93 0.0% 8 6.1 9.3 6.1 18.8 4.3
4. 108.170.251.113 0.0% 8 8.8 9.7 8.7 11.1 1.0
5. 108.170.229.1 0.0% 8 15.6 10.0 8.2 15.6 2.4
6. 172.217.31.14 0.0% 8 7.3 9.1 6.7 16.7 3.3
Please scroll to the right to see complete output.
To exit type q.
The above mentioned way to use mtr is not user friendly. Most of the times we want to see results in a user friendly report format. For that, mtr supports report format that you can use as mentioned below.
sudo mtr -r -c 30 google.com
The c option flag sets how many packets are sent and recorded in the report. When not used, the default will generally be 10, but for faster intervals you may want to set it to 30 or 100. The report can take longer to finish when doing this.
The output is shown below
Start: 2019-06-01T23:23:22+0530 HOST: XE-TMac-Shekhar-Gulati.loca Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 30 1.4 2.1 1.3 9.0 2.0 2.|-- abts-north-static-236.220 0.0% 30 8.8 6.9 5.5 11.0 1.3 3.|-- 182.79.217.89 0.0% 30 7.4 14.6 6.5 39.8 10.8 4.|-- 72.14.243.0 0.0% 30 10.3 10.1 8.8 19.7 2.0 5.|-- 74.125.244.194 0.0% 30 11.0 10.9 9.6 16.2 1.5 6.|-- 216.239.42.48 0.0% 30 33.5 33.5 32.0 40.4 1.8 7.|-- 209.85.245.151 0.0% 30 33.1 33.9 33.0 38.7 1.2 8.|-- 108.170.248.177 0.0% 30 33.5 35.1 33.0 46.1 3.2 9.|-- 209.85.242.111 0.0% 30 31.9 33.2 31.4 43.0 2.5 10.|-- bom05s15-in-f14.1e100.net 0.0% 30 32.7 33.5 32.6 36.2 1.0
The command will take 30 seconds to run as it will send request every 1 second for 30 times.
As you can see in the above output, route for this request was different from the previous one. The destination server that served our request has host name bom05s15-in-f14.1e100.net which in previous run was del03s01-in-f14.1e100.net.
1e100.net is a Google-owned domain name used to identify the servers in their network.
There are many other options in mtr that you can see under the man page of mtr.
Things to know
In the previous section you learnt how to use mtr. There are few things that you know when you are trying to debug issues.
- If you see a packet loss at any hop in the mtr report that may indicate a problem with that particular router. Before you are sure that it is an issue please check the subsequent hop. If that hop shows 0.0% then the issue could be ICMP rate limiting by the service providers. ICMP messages are sent during ping. These messages are given low priority so service provides can ignore them. So, this might give illusion of packet loss when there is no loss.
- You should also try to run mtr is reverse direction as well i.e. from destination to host. This will help you understand return path.
Shekhar Gulati 