Jump to content

Photo

Weekly Update - 2015-08-13 - Server Performance

- - - - -

  • Please log in to reply
29 replies to this topic

#1
LadyTiggs

LadyTiggs

    Super Troll

  • Members
  • PipPipPip
  • 545 posts

CapnJosh has posted his weekly update:

 

I figured I'd describe the hosting situation and what we've seen in regards to reported "server performance problems":

 

1. I have personally obtained dedicated physical servers in all locations.  They are generally running Xeon E3-1270 processors.  We cannot really get much faster CPUs.

 

2. All gameserver hosts are on at least 1Gb internet uplinks.

 

3. We have had gameserver hosts in 13 different datacenters and we've used no fewer than 6 independent hosting companies.  We have tried both C-States enabled and disabled, various power configurations, hyperthreading enabled and disabled, turbo enabled and disabled, spread spectrum enabled and disabled.

 

4. Several times now when we have received Wireshark capture files, we have seen very high levels of "Duplicate ACKs" and "retransmission retries", both of which most often are indicative of packet loss.

 

5. CPU utilization on the gameserver hosts is relatively low

 

6. We have built out reporting that shows all client-reported ping times by server.  The ping times are as expected, given geographical proximity.  There are exceptional cases too, where a single user is showing either high pings or wild ping ranges even when they are geographically right next to many users who report low, steady pings.

 

7. The survey results came back showing that only 4% of respondents indicated they had very bad network problems, with another 11% or so indicating slightly higher levels of performance issues.  The remaining 89% indicated no performance issues or they "sometimes" experienced them.

 

8. Gameserver telemetry has not been indicating high tick processing times (indicating server CPU overload)

 

9. Gameserver management systems are exactly as they were before we ever worked on Hawken

 

Now, there have been cases where gameserver hosts have been overloaded, or we have received splash damage from DDoS activity, or some routing peer has had network problems, or I have screwed up some server configuration, or the back-end data layer has had problems.

 

However, from all I have seen, the problem is very likely *not* the hosting, and that's based on the above data.

 

Remaining things to pursue may be: 

1. optimizations to replication data size

2. adding further network monitoring features to the client and server (to comprehensively monitor all routes for all players, e.g. WinMTR and UDP delivery rate software that can run in the background on all clients all the time or as desired)

3. adding client-side network configuration reporting features (to be able to see what configs may or may not predict network performance results)

4. investigate what else could have changed over the past year

5. More WinMTR routes from client machines while they are experiencing what looks like server performance issues (these will show if it's due to packet loss and at which router packets are being lost)

 

Regarding the high number of gameserver instances, we will be reducing those, but we want to gather data from all the gameserver hosts during the same period of time so we can compare "apples to apples".

 

The dev team hiring process is still ongoing.  I know for a fact there is frustration we have not staffed up and already released the first patches - hey, I wish we could have already reached that point, so I feel your pain.  It's moving and it's definitely happening though.  Once again (as we all feel like is stated too often), it's a matter of "when" not "if".

 

-capnjosh


  • Guns_N_Rozer, Elite_is_salty, Miscellaneous and 12 others like this

~~Do not meddle in the affairs of dragons for thou art crunchy and go well with ketchup!!~~

 


#2
DerMax

DerMax

    Advanced Member

  • Members
  • PipPipPip
  • 908 posts

Whenever I lag out, the problem is 99% of the time on my side (=ISP derping). I'm pretty sure it's the same for some of the players who claim to be having connectivity issues.


  • Guns_N_Rozer and Odinous like this

#3
crockrocket

crockrocket

    Advanced Member

  • Members
  • PipPipPip
  • 1989 posts

Yeah, while I was noticing many of the common server issues for a while, I haven't had any major problems in about a month now, except on my end.


                                                                    JgQjgkx.png

 

Salvage: An Idea to Stop Leavers

Player Retention & Howken

 

[14:31] <Crafty> I know that in my balls
[14:32] <Crafty> hawken is unlike anything Ive played

 

 


#4
eth0

eth0

    Advanced Member

  • Members
  • PipPipPip
  • 608 posts

Really appreciate the update guys, keep up the great work!


  • comic_sans likes this

Pubstomping is a whole different strategic discussion, however, and usually just becomes an exploration of the ethics of dumpstering randos.

UYoy63i.png

Why mech game make when you no mech game have you don't want to make? 


#5
JackVandal

JackVandal

    Advanced Member

  • Members
  • PipPipPip
  • 501 posts

Keep up the good work! Cant wait to see what else you guys roll out.


"but the dead horse has been beaten so many times it's practically a pulpy mess in the barn by now."

-M1lkshake


#6
Elite_is_salty

Elite_is_salty

    Advanced Member

  • Members
  • PipPipPip
  • 1144 posts

3 days ago I'd get for example 170 in dallas. Yesterday and the day before it I'm getting 195.

It's very confusing man!

 

I even went and bought new ethernet cable and such to make sure it's not on my end. Yesterday I also went to the ISP's office department in the state and I asked if there was any bandwidth throttling.

 

I've tried many things.

 

 

One can say that I'm just too far to get a good ping, and I agree, but why is it getting worse?

 

For a good good while I was getting a solid 150 in ashburn. Now I don't even dream of a 170.

 

 

Maybe certain settings offer maximum performance, so I'll make sure to actively report any changes if ever my ping gets lower. Other players should do too. That should give josh an idea of what might be the optimal settings or provider.


  • DerMax, Guns_N_Rozer and DM30 like this

The PC community is the red headed step child Reloaded never wanted but got saddled with when they married the PC community's mother.

---------------------------------------------------------------

Shoutout to mah real Africans out there.


#7
DM30

DM30

    Advanced Member

  • Members
  • PipPipPip
  • 180 posts
LA still sucks for me, but the other NA hosts have been pretty steady for the past couple of weeks. I don't envy you guys trying to get this worked out, but I wish you the best of luck and your efforts are appreciated.

Hawken gameplay

 

TPG Playlists -- Season 2 | Season 3


#8
StubbornPuppet

StubbornPuppet

    Advanced Member

  • Members
  • PipPipPip
  • 1508 posts

Well, I've stated before that I am certain my ISP is throttling my gaming traffic - whether I'm on a PSN, XBL or PC gaming connection, I can clearly watch my bandwidth and ping start to go down.  Same things happen when I log into to Netflix.  This doesn't happen when I'm doing anything else internet related, and it doesn't matter whether I'm actually playing a match or just logged into the game sitting at the main screen.  I can even go and upload a gigantic video file to cloud storage and/or download a file of the same size at the same time... but the moment I log into one of the above services, my upload, download and ping all drop by about 50%.

 

The ISP denies it.  It is illegal for them to do it.  But nobody is really out there working to catch them at it or has the resources to do so effectively.  Even with fines, the ISP's save more money doing it than they would have to pay for the ones where someone gets pissed off enough and has the time to press charges and prove it.


To be serious for a moment this is just a joke

 


#9
Nept

Nept

    Advanced Member

  • Members
  • PipPipPip
  • 939 posts

Within the past 2-3 days, my Dallas ping has increased from ~70 to ~130-140.  That's across all Dallas servers, private and public.  Could be completely unrelated to server-side changes, ofc, but was wondering whether anything was altered in the last little bit.


  • Elite_is_salty likes this

#10
Badtings

Badtings

    Advanced Member

  • Members
  • PipPipPip
  • 466 posts

thx for the update!


8CyRDl7.gifGYM1W9I.png


#11
Draigun

Draigun

    Advanced Member

  • Members
  • PipPipPip
  • 218 posts
 

 

[...]

 

Most games require a broadband connection that follows industry standard speeds. The global standard is around the 25 mbps range. Obviously, if you have lower bandwidth, or not enough bandwidth to even handle the netcode of a game, you will experience performance problems.

 

You could have a gigabit connection for both up and downstream channels, utilizing a certified DOCSIS 3.1 modem, having the router receive Internet through a triple-shielded CAT6a cable. Even so, the client may be utilizing insufficient specifications that could reduce signal quality tremendously. SNR problems do happen in certain households, perhaps the SNR is outside the bounds of what is considered acceptable by that ISP. Maybe cross-interference with certain electronics nearby occasionally reduce signal strength, causing perceived timeouts/dropouts across any appliance utilizing the Internet.

 

There are hundreds of more in-depth, technical problems that exists throughout the world of ISPs than what I've mentioned. Any number of them could be causing such problems, but yet it is extremely difficult to pinpoint exactly what it could be, under today's standards.


Edited by Draigun, 13 August 2015 - 12:54 PM.

TpsOr7F.png


#12
Elite_is_salty

Elite_is_salty

    Advanced Member

  • Members
  • PipPipPip
  • 1144 posts

Within the past 2-3 days, my Dallas ping has increased ...

 

That has also happened to me.


The PC community is the red headed step child Reloaded never wanted but got saddled with when they married the PC community's mother.

---------------------------------------------------------------

Shoutout to mah real Africans out there.


#13
PoopSlinger

PoopSlinger

    Advanced Member

  • Members
  • PipPipPip
  • 588 posts

Good on getting Chicago servers.  Playing on them with other midwesterners has shown me that my pings may be attributed to ISP.  A guy living 100 miles further away from Chicago was getting more 10-20 ping less and it was more stable than mine.

 

In closing

-Keep on trucking

-fuzzy bunny Charter


  • eth0 likes this

khn3gAi.jpg?1CitkI9t.jpgGkp2fB7.jpg

Come on Crafty, you have been officially called out on your lies. Your online reputation is at stake here, this is just like an old school street race running for pink slips. Its run what you brung and hope its enough. Put up or shut the fuzzy bunny up.


#14
Call_Me_Ishmael

Call_Me_Ishmael

    Advanced Member

  • Members
  • PipPipPip
  • 1114 posts

Whenever I lag out, the problem is 99% of the time on my side (=ISP derping). I'm pretty sure it's the same for some of the players who claim to be having connectivity issues.

 

 

It doesn't matter what exit point I use, or if I use a VPN (mine or my employer's).  My service provider is Verizon, and it is FIOS (FTTH), 150Mbps.  I'll send Josh a wireshark log next time I have 'server issues'.

 

I had about two weeks of really smooth gameplay where it all felt pretty-much like "Old Hawken" (about a year ago).  I started getting issues again two days ago.


  • DerMax likes this

Did I say Call Me Ishmael?

 

You should call me Luna.


#15
hoghead

hoghead

    Advanced Member

  • Members
  • PipPipPip
  • 348 posts

232.jpg

Thanks for the update. Were still here. :smile:


  • eth0, comic_sans, 1uster and 3 others like this

#16
LarryLaffer

LarryLaffer

    Advanced Member

  • Members
  • PipPipPip
  • 71 posts

Whenever I lag out, the problem is 99% of the time on my side (=ISP derping). I'm pretty sure it's the same for some of the players who claim to be having connectivity issues.

Is there a reliable way to find that it's ISP's fault? 'Cause I don't have such problems (unstable ping, packet losses) in any other online game I play (HotS, BF3/4, QL) but Hawken. I may get occasional lags rarely, but ping is much more stable in other games. Yesterday I had 76ms in Hawken (yay! that's normal ping) at 9pm local time (at this time our network is quite loaded actually), but after midnight ping rose up to 100ms. So I just want to be sure that problem is/isn't on my side.



#17
n3onfx

n3onfx

    Advanced Member

  • Members
  • PipPipPip
  • 511 posts

Is there a reliable way to find that it's ISP's fault? 'Cause I don't have such problems (unstable ping, packet losses) in any other online game I play (HotS, BF3/4, QL) but Hawken. I may get occasional lags rarely, but ping is much more stable in other games. Yesterday I had 76ms in Hawken (yay! that's normal ping) at 9pm local time (at this time our network is quite loaded actually), but after midnight ping rose up to 100ms. So I just want to be sure that problem is/isn't on my side.

 

 

You can install WinMTP, find some public server IPs in FrankFurt and Amsterdam (there's a bunch of public Battlefield and other game servers IPs you can find in google) and let WinMTP run with the IP.

 

If you notice same-ish pings than in Hawken and packet loss rates above 5-10% then there is a problem on your side or between you and the server. It could be your connection, it could be your ISP throttling bandwidth for online games (although that's very unlikely to happen in most EU countries) or it could be wonky routes to the server (your access to the server is being rerouted through long and unnecessary routes due to faulty routing, this kind of problem usually resolves itself though).

 

I had massive problems for 2 months only in Hawken, WinMTP gave 0% packet loss, minimal jitter and low pings to Frankfurt and Amsterdam but Hawken was borderline unplayable with everything indicating big packet loss (damage done to others being delayed, shooting and dodging having random delay, commands like dodging, shooting or boosting not registering several times a round, hit reg failing about half the time, stuttering on every dodge and so on). Other games were fine, my connection was stable (true FTTH with an ethernet cable) to distant servers. My isp does not throttle online gaming. I tried everything I could find on my end.

 

The problems suddenly stopped 2 days ago, my ping has dropped by half and all the precedent issues have disappeared. BUT my connection has not changed, I tinkered with nothing on my end, the traceroutes are the same as before and WinMTP gives me the same results. I'm a bit scared to launch Hawken again and see the old problems are back, but I'm pretty sure the problems I described had to be something about the servers themselves. If nothing on my end or between me and the servers has changed between issues -> no issues, what am I missing?


Edited by neon, 14 August 2015 - 03:17 AM.

  • LarryLaffer and LoC_TR like this

t

t

DWEH3ZP.png   CRITICAL  RqKpxHn.png    ASSIST   VDNrFxD.png

t

t


#18
jjm1

jjm1

    Advanced Member

  • Members
  • PipPipPip
  • 60 posts

No doubt a lot of lag can be attributed to crappy routing. I live in Perth, Australia and have noticed that different servers in Singapore have entirely different routes.

 

I'm not an expert, so I don't know if its up to your hosts to make accurate routes with Telstra or if I can try to contact them and spend days on the phone with a script bot to see if they are interested at all.

 

These are my experiments with two hawken servers and valve.net Singapore for comparison.

 

 

Perth > HAWKEN Tokyo
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    0 |  100 |  100 |   17 |   24 |   72 |   19 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   72 |   20 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   28 |   82 |   25 |
|tengige0-7-0-0.pthw-core01.perth.net.reach.com -    0 |  100 |  100 |   20 |   26 |   72 |   22 |
|i-0-0-1-0.istt-core02.bx.telstraglobal.net -    0 |  100 |  100 |   67 |   73 |  117 |   70 |
|   i-0-0-0-1.istt04.bi.telstraglobal.net -    0 |  100 |  100 |   67 |   71 |  114 |   68 |
| ix-23-0.tcore1.SVQ-Singapore.as6453.net -    4 |   88 |   85 |  144 |  154 |  205 |  145 |
| if-20-2.tcore2.SVW-Singapore.as6453.net -    8 |   76 |   70 |  186 |  192 |  235 |  186 |
|  if-2-2.tcore1.SVW-Singapore.as6453.net -    0 |  100 |  100 |  186 |  193 |  238 |  187 |
|      if-6-2.tcore2.TV2-Tokyo.as6453.net -    3 |   92 |   90 |  185 |  191 |  235 |  186 |
|      if-2-2.tcore1.TV2-Tokyo.as6453.net -    0 |  100 |  100 |  185 |  192 |  235 |  186 |
|                           180.87.180.66 -    0 |  100 |  100 |  159 |  166 |  204 |  159 |
|                               27.0.0.80 -    0 |  100 |  100 |  162 |  169 |  210 |  163 |
|                               27.0.0.70 -    3 |   92 |   90 |  162 |  171 |  212 |  163 |
|ec2-52-69-216-25.ap-northeast-1.compute.amazonaws.com -    2 |   96 |   95 |  161 |  169 |  211 |  163 |
|________________________________________________|______|______|______|______|______|______|
   
   
   
Perth > HAWKEN Singapore
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    2 |   96 |   95 |   18 |   25 |   69 |   18 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   74 |   19 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   27 |   74 |   23 |
|bundle-ether7.fli-core1.adelaide.telstra.net -    2 |   96 |   95 |   50 |   60 |  104 |   55 |
|bundle-ether9.win-core10.melbourne.telstra.net -    0 |  100 |  100 |   61 |   68 |  119 |   63 |
|bundle-ether12.ken-core10.sydney.telstra.net -    2 |   96 |   95 |   73 |   83 |  133 |   73 |
|bundle-ether1.pad-gw11.sydney.telstra.net -    0 |  100 |  100 |   73 |   83 |  131 |   77 |
|bundle-ether1.sydp-core04.sydney.reach.com -    0 |  100 |  100 |   74 |   83 |  131 |   77 |
|i-0-1-0-45.sydp-core03.bi.telstraglobal.net -    3 |   92 |   90 |   74 |   79 |  130 |   76 |
|i-0-6-0-3.1wlt-core01.bx.telstraglobal.net -    2 |   96 |   95 |  221 |  230 |  284 |  223 |
|   i-0-4-0-6.eqla01.bi.telstraglobal.net -    2 |   96 |   95 |  220 |  227 |  268 |  221 |
| pacnet-peer.eqla01.pr.telstraglobal.net -    2 |   96 |   95 |  221 |  229 |  287 |  223 |
|       te0-1-2-0.wr1.sin0.asianetcom.net -    2 |   96 |   95 |  226 |  233 |  284 |  228 |
|         gi1-0-0.gw5.sin1.asianetcom.net -    0 |  100 |  100 |  225 |  235 |  287 |  225 |
|                 INH-0009.asianetcom.net -    2 |   96 |   95 |  225 |  231 |  278 |  226 |
|       mpr1.te4-1.bbnet2.sin003.pnap.net -    0 |  100 |  100 |  224 |  237 |  422 |  225 |
|                           202.58.11.250 -    2 |   96 |   95 |  224 |  244 |  409 |  234 |
|                            72.5.160.114 -    2 |   96 |   95 |  224 |  232 |  287 |  228 |
|________________________________________________|______|______|______|______|______|______|
   
   
 
Perth > valve.net Singapore
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    0 |  100 |  100 |   17 |   24 |   77 |   18 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   76 |   18 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   25 |   75 |   20 |
|tengige0-8-0-5-1.pie-core1.perth.telstra.net -    0 |  100 |  100 |   19 |   26 |   75 |   20 |
|gigabitethernet0-7-0-0.pthp-core01.perth.net.reach.com -    0 |  100 |  100 |   18 |   25 |   72 |   19 |
|i-0-5-4-0.skdi-core01.bx.telstraglobal.net -    0 |  100 |  100 |   68 |   75 |  122 |   72 |
|   i-0-0-0-4.istt04.bi.telstraglobal.net -    2 |   96 |   95 |   68 |   74 |  117 |   69 |
| pacnet-peer.istt03.pr.telstraglobal.net -    2 |   96 |   95 |   67 |   72 |  117 |   67 |
|                              61.8.59.82 -    2 |   96 |   95 |   68 |   72 |  114 |   71 |
|                         sgp-1.valve.net -    0 |  100 |  100 |   69 |   75 |  115 |   70 |
|________________________________________________|______|______|______|______|______|______|


#19
eth0

eth0

    Advanced Member

  • Members
  • PipPipPip
  • 608 posts

Kansascity felt smooth as butter last night ~40-50 ping. Can't wait to try Chicago!

EDIT: Getting 20-30 in Chicago and my Dallas experience is similar to Kansascity. So far all are better than my current Ashburn experience, ~50-70.


Edited by eth0, 17 August 2015 - 08:33 AM.

Pubstomping is a whole different strategic discussion, however, and usually just becomes an exploration of the ethics of dumpstering randos.

UYoy63i.png

Why mech game make when you no mech game have you don't want to make? 


#20
JackVandal

JackVandal

    Advanced Member

  • Members
  • PipPipPip
  • 501 posts

232.jpg

Thanks for the update. Were still here. :smile:

We are watching, alllllways watching


  • hoghead likes this

"but the dead horse has been beaten so many times it's practically a pulpy mess in the barn by now."

-M1lkshake


#21
hoghead

hoghead

    Advanced Member

  • Members
  • PipPipPip
  • 348 posts

 

No doubt a lot of lag can be attributed to crappy routing. I live in Perth, Australia and have noticed that different servers in Singapore have entirely different routes.

 

I'm not an expert, so I don't know if its up to your hosts to make accurate routes with Telstra or if I can try to contact them and spend days on the phone with a script bot to see if they are interested at all.

 

These are my experiments with two hawken servers and valve.net Singapore for comparison.

 

 

Perth > HAWKEN Tokyo
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    0 |  100 |  100 |   17 |   24 |   72 |   19 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   72 |   20 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   28 |   82 |   25 |
|tengige0-7-0-0.pthw-core01.perth.net.reach.com -    0 |  100 |  100 |   20 |   26 |   72 |   22 |
|i-0-0-1-0.istt-core02.bx.telstraglobal.net -    0 |  100 |  100 |   67 |   73 |  117 |   70 |
|   i-0-0-0-1.istt04.bi.telstraglobal.net -    0 |  100 |  100 |   67 |   71 |  114 |   68 |
| ix-23-0.tcore1.SVQ-Singapore.as6453.net -    4 |   88 |   85 |  144 |  154 |  205 |  145 |
| if-20-2.tcore2.SVW-Singapore.as6453.net -    8 |   76 |   70 |  186 |  192 |  235 |  186 |
|  if-2-2.tcore1.SVW-Singapore.as6453.net -    0 |  100 |  100 |  186 |  193 |  238 |  187 |
|      if-6-2.tcore2.TV2-Tokyo.as6453.net -    3 |   92 |   90 |  185 |  191 |  235 |  186 |
|      if-2-2.tcore1.TV2-Tokyo.as6453.net -    0 |  100 |  100 |  185 |  192 |  235 |  186 |
|                           180.87.180.66 -    0 |  100 |  100 |  159 |  166 |  204 |  159 |
|                               27.0.0.80 -    0 |  100 |  100 |  162 |  169 |  210 |  163 |
|                               27.0.0.70 -    3 |   92 |   90 |  162 |  171 |  212 |  163 |
|ec2-52-69-216-25.ap-northeast-1.compute.amazonaws.com -    2 |   96 |   95 |  161 |  169 |  211 |  163 |
|________________________________________________|______|______|______|______|______|______|
   
   
   
Perth > HAWKEN Singapore
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    2 |   96 |   95 |   18 |   25 |   69 |   18 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   74 |   19 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   27 |   74 |   23 |
|bundle-ether7.fli-core1.adelaide.telstra.net -    2 |   96 |   95 |   50 |   60 |  104 |   55 |
|bundle-ether9.win-core10.melbourne.telstra.net -    0 |  100 |  100 |   61 |   68 |  119 |   63 |
|bundle-ether12.ken-core10.sydney.telstra.net -    2 |   96 |   95 |   73 |   83 |  133 |   73 |
|bundle-ether1.pad-gw11.sydney.telstra.net -    0 |  100 |  100 |   73 |   83 |  131 |   77 |
|bundle-ether1.sydp-core04.sydney.reach.com -    0 |  100 |  100 |   74 |   83 |  131 |   77 |
|i-0-1-0-45.sydp-core03.bi.telstraglobal.net -    3 |   92 |   90 |   74 |   79 |  130 |   76 |
|i-0-6-0-3.1wlt-core01.bx.telstraglobal.net -    2 |   96 |   95 |  221 |  230 |  284 |  223 |
|   i-0-4-0-6.eqla01.bi.telstraglobal.net -    2 |   96 |   95 |  220 |  227 |  268 |  221 |
| pacnet-peer.eqla01.pr.telstraglobal.net -    2 |   96 |   95 |  221 |  229 |  287 |  223 |
|       te0-1-2-0.wr1.sin0.asianetcom.net -    2 |   96 |   95 |  226 |  233 |  284 |  228 |
|         gi1-0-0.gw5.sin1.asianetcom.net -    0 |  100 |  100 |  225 |  235 |  287 |  225 |
|                 INH-0009.asianetcom.net -    2 |   96 |   95 |  225 |  231 |  278 |  226 |
|       mpr1.te4-1.bbnet2.sin003.pnap.net -    0 |  100 |  100 |  224 |  237 |  422 |  225 |
|                           202.58.11.250 -    2 |   96 |   95 |  224 |  244 |  409 |  234 |
|                            72.5.160.114 -    2 |   96 |   95 |  224 |  232 |  287 |  228 |
|________________________________________________|______|______|______|______|______|______|
   
   
 
Perth > valve.net Singapore
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              xx.xx.xx.xx -    0 |  100 |  100 |    0 |    0 |    1 |    0 |
|                           xx.xx.xx.xx -    0 |  100 |  100 |   17 |   24 |   77 |   18 |
|                         xx.xx.xx.xx -    0 |  100 |  100 |   18 |   24 |   76 |   18 |
|bundle-ether7.wel-core3.perth.telstra.net -    0 |  100 |  100 |   19 |   25 |   75 |   20 |
|tengige0-8-0-5-1.pie-core1.perth.telstra.net -    0 |  100 |  100 |   19 |   26 |   75 |   20 |
|gigabitethernet0-7-0-0.pthp-core01.perth.net.reach.com -    0 |  100 |  100 |   18 |   25 |   72 |   19 |
|i-0-5-4-0.skdi-core01.bx.telstraglobal.net -    0 |  100 |  100 |   68 |   75 |  122 |   72 |
|   i-0-0-0-4.istt04.bi.telstraglobal.net -    2 |   96 |   95 |   68 |   74 |  117 |   69 |
| pacnet-peer.istt03.pr.telstraglobal.net -    2 |   96 |   95 |   67 |   72 |  117 |   67 |
|                              61.8.59.82 -    2 |   96 |   95 |   68 |   72 |  114 |   71 |
|                         sgp-1.valve.net -    0 |  100 |  100 |   69 |   75 |  115 |   70 |
|________________________________________________|______|______|______|______|______|______|

 

mint%20blackboard.jpgAhh....... I see where your coming from. :unsure: Are you like a scientist or something? [God I'm dumb.]  :ermm: Excuse me while I go mow the grass.


Edited by hoghead, 14 August 2015 - 09:38 AM.

  • 1uster likes this

#22
jjm1

jjm1

    Advanced Member

  • Members
  • PipPipPip
  • 60 posts

mint%20blackboard.jpgAhh....... I see where your coming from. :unsure: Are you like a scientist or something? [God I'm dumb.]  :ermm: Excuse me while I go mow the grass.

Its pretty hardcore isn't it  :cool:

 

Anyone who's ever done a tracert in the windows command prompt will know what it is though.


  • nepacaka likes this

#23
hestoned

hestoned

    Advanced Member

  • Members
  • PipPipPip
  • 252 posts

glad to hear it sounds like you guys are getting close optimizing the servers


  • comic_sans likes this

#24
Draco3

Draco3

    Advanced Member

  • Members
  • PipPipPip
  • 120 posts

To add to this (forwarded this on already to the capn).  I, like jjm1 am in Perth and this is my route through my service provider (AMNET).  What's interesting is how I get to Singapore2 (seems to be NSW, then Canberra before I hit the Asia routers. Seems to be a big bottleneck at 202.58.11.250 (which is Internap Network Services (Singapore) Co. Ltd.).  

 

Host                                                                     % - Loss - Sent - Recv - Best - Avg - Wrst - Last
my.router -----------------------------                        0 1037 1037 0 1 5 3
Lo0.amnet.l1-45sgt-se800-01.wa.amnet.net.au  1 1034 1033 12 15 186 18
vl70.ten7-3.cr01.wa.amnet.net.au                       0 1037 1037 12 23 231 52
te3-2.br02.wa.amcom.net.au                              1 1030 1028 12 22 242 15
ten-0-0-0-823.bdr01.per02.wa.VOCUS.net.au    0 1037 1037 12 15 198 15
ten-0-2-0-1.cor01.per02.wa.VOCUS.net.au        1 1034 1033 220 223 449 222
ten-0-2-0-7.cor03.syd03.nsw.VOCUS.net.au     0 1037 1037 220 224 466 222
ten-0-1-0-3.cor01.syd04.nsw.VOCUS.net.au     0 1037 1037 221 224 457 224
ten-0-0-0-2.cor01.sjc01.ca.VOCUS.net              0 1037 1037 220 223 462 222
bundle-100.cor02.sjc01.ca.VOCUS.net              1 1034 1033 220 224 465 224
ten-0-2-0.bdr01.sjc02.ca.VOCUS.net                 0 1037 1037 220 223 457 222
0-3.sw4.sjc1.asianetcom.net                              0 1037 1037 231 234 477 233
te0-3-0-2.wr1.sin0.asianetcom.net                    1 1034 1033 391 395 589 393
te0-0-4-0.wr2.sin0.asianetcom.net                    1 1034 1033 393 398 606 399
gi6-0-0.gw2.sin1.asianetcom.net                      1 1034 1033 397 400 593 402
gi14-0-0.gw5.sin1.asianetcom.net                    0 1037 1037 397 400 614 403
INH-0009.asianetcom.net                                 0 1037 1037 227 230 466 228
mpr1.te3-1.bbnet1.sin003.pnap.net                  0 1037 1037 227 233 471 230
202.58.11.250                                                   1 1034 1033 227 240 1088 243
72.5.160.114                                                     1 1034 1033 0 230 354 231

Edited by Draco3, 17 August 2015 - 04:18 AM.


#25
StubbornPuppet

StubbornPuppet

    Advanced Member

  • Members
  • PipPipPip
  • 1508 posts

^Ha ha, "Internap".  Is that where the traffic goes to sleep along it's way?


To be serious for a moment this is just a joke

 


#26
jjm1

jjm1

    Advanced Member

  • Members
  • PipPipPip
  • 60 posts

 

To add to this (forwarded this on already to the capn).  I, like jjm1 am in Perth and this is my route through my service provider (AMNET).  What's interesting is how I get to Singapore2 (seems to be NSW, then Canberra before I hit the Asia routers. Seems to be a big bottleneck at 202.58.11.250 (which is Internap Network Services (Singapore) Co. Ltd.).  

 

Host                                                                     % - Loss - Sent - Recv - Best - Avg - Wrst - Last
my.router -----------------------------                        0 1037 1037 0 1 5 3
Lo0.amnet.l1-45sgt-se800-01.wa.amnet.net.au  1 1034 1033 12 15 186 18
vl70.ten7-3.cr01.wa.amnet.net.au                       0 1037 1037 12 23 231 52
te3-2.br02.wa.amcom.net.au                              1 1030 1028 12 22 242 15
ten-0-0-0-823.bdr01.per02.wa.VOCUS.net.au    0 1037 1037 12 15 198 15
ten-0-2-0-1.cor01.per02.wa.VOCUS.net.au        1 1034 1033 220 223 449 222
ten-0-2-0-7.cor03.syd03.nsw.VOCUS.net.au     0 1037 1037 220 224 466 222
ten-0-1-0-3.cor01.syd04.nsw.VOCUS.net.au     0 1037 1037 221 224 457 224
ten-0-0-0-2.cor01.sjc01.ca.VOCUS.net              0 1037 1037 220 223 462 222
bundle-100.cor02.sjc01.ca.VOCUS.net              1 1034 1033 220 224 465 224
ten-0-2-0.bdr01.sjc02.ca.VOCUS.net                 0 1037 1037 220 223 457 222
0-3.sw4.sjc1.asianetcom.net                              0 1037 1037 231 234 477 233
te0-3-0-2.wr1.sin0.asianetcom.net                    1 1034 1033 391 395 589 393
te0-0-4-0.wr2.sin0.asianetcom.net                    1 1034 1033 393 398 606 399
gi6-0-0.gw2.sin1.asianetcom.net                      1 1034 1033 397 400 593 402
gi14-0-0.gw5.sin1.asianetcom.net                    0 1037 1037 397 400 614 403
INH-0009.asianetcom.net                                 0 1037 1037 227 230 466 228
mpr1.te3-1.bbnet1.sin003.pnap.net                  0 1037 1037 227 233 471 230
202.58.11.250                                                   1 1034 1033 227 240 1088 243
72.5.160.114                                                     1 1034 1033 0 230 354 231

 

 

I cant work this one out.
 
'ten-0-2-0-1.cor01.per02.wa.VOCUS.net.au' is in Canberra? Otherwise it jumps from 15 to 223 before leaving Perth, which is impossibad. If our ISPs are diverting away from the SEA-ME-WE 3 line to Singapore then we should be getting ~75ms via the one scenic pleb route that goes Perth>Adelaide>Melbourne>Sydney, after that it goes international and takes around 120ms Sydney>Singapore. I don't know why anything (or anybody) would go to Canberra.
 
I Just pinged 75ms to 202.58.11.250 and 233ms to 72.5.160.114 BTW. weird.
 
--------
edit:
 
Just did some research: that route uses MPLS ICMP Tunneling, so that's why the pings looked wrong.

Edited by jjm1, 17 August 2015 - 11:56 PM.


#27
Draco3

Draco3

    Advanced Member

  • Members
  • PipPipPip
  • 120 posts
Likely going to Canberra to snoop the traffic.

#28
TangledMantis

TangledMantis

    Advanced Member

  • Members
  • PipPipPip
  • 236 posts

At work, ive been dealing with a lot of high latency and aggrivating jitter regarding some hosted services we use (citrix traffic) 

 

After a lot of testing and blaming, I found that the issue has to do with ISP peering. I also found out that Comcast in particular is notorious for having crappy peering. They 'meet the requirements' and thats all. Ping times are all over the place and after running some traces and other tests, I found that the smoking gun appears after comcast OR centurylink pass their traffic off to Time Warner and vis versa. (we have cable and fiber with both providers at work.) Based on some SIP issues I had with Level 3, I dont think their infrastructure plays nice with others much better than the rest. 

 

I think Reloaded is doing the best they can. The Internet Exchange Points are clogged and working at 100%. Depending on the time and demand, they can jam up and slow everything down. For busts or web browsing its not noticable, but when you need low latency and low jitter, these points can be a nightmare.


  • StubbornPuppet likes this

#29
Draco3

Draco3

    Advanced Member

  • Members
  • PipPipPip
  • 120 posts

It's likely I missed something a little while back, but when the new Singapore2 servers came online was a new provider used?  I am making a complete assumption here that traffic was following a similar path to the S1 servers as they do with the S2 servers.  While S1 and S2 co-existed in the Asia lobby for a while, connections to S1 were under 100ms whereas connections to S2 were above 200ms (this is what I am seeing from Perth).  Of course, no WinMTR was performed at the time of the S1/S2 co-existance but either the provider changed or there are additional hops introduced with a relocation to an alternate data center.   



#30
StubbornPuppet

StubbornPuppet

    Advanced Member

  • Members
  • PipPipPip
  • 1508 posts

At work, ive been dealing with a lot of high latency and aggrivating jitter regarding some hosted services we use (citrix traffic) 

 

After a lot of testing and blaming, I found that the issue has to do with ISP peering. I also found out that Comcast in particular is notorious for having crappy peering. They 'meet the requirements' and thats all. Ping times are all over the place and after running some traces and other tests, I found that the smoking gun appears after comcast OR centurylink pass their traffic off to Time Warner and vis versa. (we have cable and fiber with both providers at work.) Based on some SIP issues I had with Level 3, I dont think their infrastructure plays nice with others much better than the rest. 

 

I think Reloaded is doing the best they can. The Internet Exchange Points are clogged and working at 100%. Depending on the time and demand, they can jam up and slow everything down. For busts or web browsing its not noticable, but when you need low latency and low jitter, these points can be a nightmare.

 

Yup, all of that and more.  We've been dealing with the same thing for a couple of years where I work... and it's getting progressively worse.  As the internet is becoming more and more busy on a daily basis, ISP's really, really need to step up all efforts to add more trunks, more data centers... more everything.

 

Unfortunately, all of these are 'for-profit' businesses.  They do the minimum they can in order to ride as big of a profit margin as they can - as few pipes, as few upgrades, as few IEP's, as few trunks and as little investment as possible to allow traffic to 'flow'.   But it's becoming like LA rush-hour traffic all over the world and everyone is stuck in it... especially the bigger the car (data packets) you have to drive.

 

Another factor is maintenance.  The size of the network maintenance staff hasn't grown to keep up with demands.  So, when a service center has a problem... or a line gets accidentally cut (which happens multiple times a day, believe it or not), it takes longer to get it repaired than it used to even a year ago.


Edited by StubbornPuppet, 20 August 2015 - 06:35 AM.

  • TangledMantis likes this

To be serious for a moment this is just a joke

 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users