20 replies to this topic

[capnjosh]

(Update: we just disabled the BGP announce to Telia in our data centers, so the connectivity issues should be solved now)

 

 

I found why Hawken has been having troubles today, specifically in Iceland and Frankfurt (edit: Ashburn as well).  Here's where WinMTR can be so very helpful.  In the below tables of data, look at the "Loss %" columns.

 

This image shows all the routers that packets between the Iceland servers have to go through in order to get to the Storm service machines:

 

Here's a WinMTR from the Frankfurt machines:

 

 

(edit: adding Ashburn server WinMTR)

Here's ashburn:

 

 

 

Notice anything consistent?  Yup, those routers with "telia" in the name seem to be having troubles.

 

 

Now, here's a WinMTR from the Amsterdam server, which is not having any troubles:

 

 

Notics how the packets get through fine with no huge "losses"?  That's because the datacenter in Amsterdam does not send traffic through Telia equipment.  That is, they do not "peer" with Telia.  However, both the Iceland datacenter and the Frankfurt datacenter do "peer" with Telia.

 

I'll try to update this post with whatever Telia ultimately posts on what happened on their end.

 

What do we do for right now?  Stop gameservers on Frankfurt and Iceland, and move them to Amsterdam.

[Kopra]

TeliaSonera is considered to be the least reliable operator in Finland if that tells anything.

[DerMax]

No surprise here � Telia's been absolute garbage for as long as I can remember. You can easily google gamers' struggle with Telia's huge packet loss. I myself can attest to that, as my ISP sometimes routes me to EU servers through Sweden, which means one of the nodes will inevitably be Telia.net.

[Dnyarri]

Does this explain why I in the United States still struggle to even get to the login screen and them an still unable to login? It makes perfect sense if I either need to send or receive data through those Telia servers, but I am confused as to why I would be given my location.

 

Thank you for looking into the issues though :)

[capnjosh]

In Singapore, Internap peers with Telia and AWS does not.  The Singapore server people send complaints about is the Internap one.  It seems Telia has an opportunity to become everyone's favorite peering partner if they went through and made their routes faster :)

[capnjosh]

We just disabled the BGP announce to Telia in our data centers, so the connectivity issues should be solved now.

[capnjosh]

Telia is having transatlantic backbone congestion issues on a couple links and a problem in LA that they are investigating at this time.  Kind of a rough week for them (likely the reason for what seemed to be an escalation in the number of reports on "server problems" and login issues over the past week).

[karnak]

Good find capnjosh and thanks for letting us know.

[Dnyarri]

I still can't get past syncing account when I try to login. :(

[capnjosh]

Does this explain why I in the United States still struggle to even get to the login screen and them an still unable to login? It makes perfect sense if I either need to send or receive data through those Telia servers, but I am confused as to why I would be given my location.

 

Thank you for looking into the issues though :)

 

It depends on how your ISP has set up its "peers".  If your ISP has a deal with Telia, they may be relaying your traffic through telia, at least sometimes.  It depends on your location purely because your ISP may have differeing peering arrangements depending on who else has routers in the same datacenter.

 

As to exactly why you sometimes can't log in, you would have to run a WinMTR or a tracert (from cmd window) pointed at the server IP in order to really know.  I'd love to look through a WinMTR and and pcap file if you'd be willing to do that for us.  Submit a support ticket if you're interested.

[capnjosh]

I still can't get past syncing account when I try to login. :(

 

Dnyarri, I bet if you ran a WinMTR pointed at v2.services.live.hawken.meteor-ent.com we'd see where your packets are getting lost.

 

WinMTR: http://winmtr.net/download-winmtr/

 

Either submit a ticket with the results or send me a PM.  I'm curious to see what it shows.

[Merl61]

Love seeing Josh post more regularly. Keep it up:) 

[BL4CK_W1D0W]

agreed... :) Good work CpnJosh. Both, log in and account sync issues fixed. Everything works pretty well again. Thx-a-lot.

 

Regards, BL4CK_W1D0W.

[Amidatelion]

This is what I was talking about. Good work.

[talon70]

Thanks for the saturday werken to fix the saturday herken!

[twp2015]

We just disabled the BGP announce to Telia in our data centers, so the connectivity issues should be solved now.

 

Can you point me to where you found out about this problem at Telia? Hoping Telia posted it somewhere or some official record. We've been trying to identify the source of an impact elsewhere and all our data pointed to telia as well but I cant find "proof" that they had a problem or the times of impact.  Thanks!

[n3onfx]

I log in around once a week to see if Hawken is playable again for me but nope, in fact the ping is now in the 100 range up from 32-48 before Reloaded and the hitreg/stuttering is atrocious. Playing mostly Dirty Bomb now and the stable 32 ping without any lag-related effect is refreshing, In fact Hawken is the only online game that gives me this kind of performance, playing on US servers from EU before Reloaded gave me better performance than playing on EU right now.

 

I'm still up to try and find out why, can we have some server addresses to use WinMTR on so we can send back reports?

[TheButtSatisfier]

Love the diligence and transparency. Keep it up!

[StubbornPuppet]

I'd like to add a bit - to a point that these screenshots help illustrate.

 

When there are connectivity or lag issues in an online video game, most peoples instinct is to blame, or at least yell at, the game developers.  "Your net-code sucks."  "Your servers suck." "blah - blah..."  When, in fact, most lag, sync problems and connectivity issues actually occur during the multiple hand-off's of data along the way between the server and the players.  Telia just happens to be an example that is consistently popping up right now.  So, Telia is a big offender... but this sort of crap happens intermittently all over the place.

 

This is why I used to be able to get a lag free, well synced experience back in 1999 on Alien vs. Predator... over a DIAL-UP connection, but I can't play any games today without some sort of problems cropping up all the time.  The internet was 'simpler'.

Edited by StubbornPuppet, 18 September 2015 - 08:26 AM.

[TheButtSatisfier]

stuff

 

I'm sorry, but no. You're wrong. Straight up. What you are telling me contradicts so much complaining I've read on these forums, and if you were right it would just flip-turn my world upside down. I need a single scapegoat for all my Hawken problems, and if you take that away from me then I'll be forced to think critically. Next I'll be all understanding and patient and fuzzy bunny, and if I don't complain obnoxiously and publicly (because wtf would I submit a support ticket for a support issue when I can fan the flames of ignorance find like-minded individuals here) then I won't get the attention I deserve.

Edited by TheButtSatisfier, 18 September 2015 - 12:07 PM.

[StubbornPuppet]

I'm sorry, but no. You're wrong. Straight up. What you are telling me contradicts so much complaining I've read on these forums, and if you were right it would just flip-turn my world upside down. I need a single scapegoat for all my Hawken problems, and if you take that away from me then I'll be forced to think critically. Next I'll be all understanding and patient and fuzzy bunny, and if I don't complain obnoxiously and publicly (because wtf would I submit a support ticket for a support issue when I can fan the flames of ignorance find like-minded individuals here) then I won't get the attention I deserve.