Today I’d like to share my experience with a TFTP DOS attack on phone systems.

One Friday afternoon around 2pm, we received a support ticket stating that our customer has problems with 2 phone systems.  Each were rebooting randomly every 5-10 minutes.  It was interesting to note that these 2 systems were located in different states; one in Minnesota, and the other in New Jersey.

I logged into both of them and found that both systems were rebooting automatically because of system overload.  These systems are Linux based so I immediately ran the “top” command to find what is going on. To my surprise I found that the system load – number of processes waiting for resources – was around 30 to 40.  So the system watchdog is seeing problems with services running very slow and reboots the system.

First, I disabled the watchdog to stop automatic systems’ reboots and started to troubleshoot. Running “top” again I found that no process is using CPU or memory more than it should. But system load was again in the 40s.

At this point, another customer calls and reports the same problem. Now from Fargo, ND.

This looked like a DOS attack , but on what service?  I couldn’t run tcpdump, it just never starts  – load is too high.

Long story short ( -actually about 15 min ), after looking at many possible angles, I looked at TFTP service logs.  Keep in mind that the systems were running very, very sluggish at this point and running “less /var/log/tftp.log” took about 3 min to execute.

And there it was: thousands of requests for phone configuration, using Polycom’s MAC address range, the one starting with 0004f2 . Thousands of “get 0004f2xxxxxx.cfg”.

This attack was neither smart nor cunning. Just sending around 1000 new tftp file download requests per second, going through what looked like all MAC addresses in the Polycom range.

Looked up IP addresses of attacking devices in GEO IP database: Dusseldorf , Germany. Some hosting company as usual: no abuse department, no emergency numbers, support contacts hidden behind a customer login page.  And now it’s around 5 PM.

I needed to react. Killed TFTP service. Made no difference.

Set up a rule in IPtables ( Linux’s built-in firewall ) to block incoming TFTP traffic:  “iptables -I INPUT  -p UDP–dport 69 -j DROP”.  End of story?

Nope.

Incoming TFTP “get” requests just keep coming and the same rate. How do I know? Running “ifconfig” and looking at the incoming packet count.  Seems the connection tracker table in Linux kernel is being overloaded, or network driver just can’t keep up with the number of new UDP requests. Go figure, at this point I don’t have time or means to stop it and return the system to working state.

Remember, the end goal is to stop the attack, but I can’t just unplug these systems from the Internet; they are using a SIP provider to send and receive calls. I need to restore their service.

The first two of the affected systems are using Comcast Business  coax Internet connections. Calling Comcast revealed an interesting fact: they CAN block the attack. But, the customer needs to sign in for a special “protection” plan and only then they will do something about it. It’s Friday and already around 6PM. I won’t be able to get anything done on Friday evening, everyone who could agree to Comcast “protection” contract has gone home already.

Talking with the Internet provider of the customer in Fargo – system number three – was as disappointing as talking to Comcast: they could possibly do something , but it would take a long time.

So, I decided to just to wait. After all, TFTP server is disabled and blocked on all systems, so there is no reason for this attack to continue for much longer.

And then, around midnight, it just stopped.

I understand the intent behind this attack:

1. Polycom is the number one SIP phone manufacturer in the world ( or was one, not sure at the moment )
2. Get a configuration file for a Polycom phone configured on the system
3. Setup international dialing through a phone system and sell it trough your own service
4. Profit

What is puzzling me about this attack was the very high rate with which new requests for configuration files were sent. TFTP servers are rate limited anyway, at least on the phone systems we deploy. So, by overwhelming the systems with new requests, attackers removed any possibility to get a configuration file, even if they sent a request with a matching MAC address in the name.

And here is the lesson we learned: time to disable TFTP services on all phone systems we support; there are other ways to provision phones. Goodbye TFTP, you have served us well for 37 years ( TFTP protocol was first defined in 1980) , but it’s time for you to go.  HTTPS only from now on.