74 replies to this topic

[capnjosh]

Since there will be more and more tournaments in Hawken coming up, it seemed appropriate to release some data on how match outcomes by team and map.  For the bookies amongst us, you may be able to adjust your payouts and odds with this data ;)

 

This is a dashboard I've made that refreshes with new data over time, so we can see what happens when updates are released.

 

Does the Bazaar (aka "Sahara") map really put one team at a big disadvantage?

 

Here's the overall for the past few months, and yes these numbers are pretty stable over time:

 

 

That's interesting and all, but what does it look like when you break it all down by avg MMR range (darker colors indicate more matches).  :

 

 

I'm told this sort of data has never been released before.  Well, I suppose we'll find out how productive it really is ;)

 

 

-capnjosh

[Hyginos]

Very interesting...

 

WTF is going on at 1200 MMR?

Edited by Hyginos, 27 May 2015 - 02:09 PM.

[brackets]

Hard to consider the significance when you don't know which spawn is team 0 and which spawn is team 1.

[Shadeness]

What game mode is this data from? All of them?

[coldform]

This statement was removed due to conclusions jumping.

Edited by {TDM} coldform, 28 May 2015 - 07:29 AM.

[capnjosh]

Hard to consider the significance when you don't know which spawn is team 0 and which spawn is team 1.

 

It helps address the top-level question: "Is there truly a problem and how bad is it?"  After that you can drill down more.

[americanbrit14]

Very interesting...

 

WTF is going on at 1200 MMR?

Its all dem smerfs yo.

[crockrocket]

The heavily skewed maps at 1200 flip winning side for 1300 and 1400.... Is there any chance mm has something to do with this? 

 

 

Also, other than Sahara what do the code names correspond to?

Edited by (TDM)crockrocket, 27 May 2015 - 02:16 PM.

[capnjosh]

What game mode is this data from? All of them?

 

Yes, it's all the game modes.  When I was pulling in game mode data the visuals started to get pretty dense.  Here's by game mode and map.

 

 

 

But really, you'd want to see it additionally broken down by MMR range:

 

[capnjosh]

Very interesting...

 

WTF is going on at 1200 MMR?

 

New users are losing 3 out of 4 matches.  That's what seems to be going on ;)

[americanbrit14]

The heavily skewed maps at 1200 flip winning side for 1300 and 1400.... Is there any chance mm has something to do with this? 

 

 

Also, other than Sahara what do the code names correspond to?

umm, lets see, i know Valkirie is front line. alleys i'm guessing is prosk, titan and andromeda... no clue

 

they have to be uptown and origin, but which is which?

Edited by americanbrit14, 27 May 2015 - 02:21 PM.

[brackets]

I was surprised to see that Facility was the least balanced map by a large amount. I thought it was one of the better balanced maps -- and it's pretty damn symmetrical

[capnjosh]

The heavily skewed maps at 1200 flip winning side for 1300 and 1400.... Is there any chance mm has something to do with this? 

 

The heavily skewed win/loss stuff at the low MMRs seems to be due to the winning team having one or more players with outlier numbers of kills (and with the "correct" MMR".  There is no way the MatchMaker can tell which 1250MMR player is actually really good.  It really just implies we need to get some more aggressive mid-match balance code that acts based on "real-world" performance within the match.  That seems to be where the opportunities lie.

[capnjosh]

umm, lets see, i know Valkirie is front line. alleys i'm guessing is prosk, titan and andromeda... no clue

 

they have to be uptown and origin, but which is which?

 

Welcome to the tip of the iceberg of when analytics becomes hard - mapping and remapping names, codes... then imagine when they change meanings over time, and then logging formatting changes as well ;)

 

It's no excuse though... I wanted to get *something* out there to chew on.  We can then pick what seems to be an outlier and then zoom in on that.

[crockrocket]

Can we get some context as to what number is what game mode, and 1 or 0 for prosk or sentium?

 

 

 

The heavily skewed win/loss stuff at the low MMRs seems to be due to the winning team having one or more players with outlier numbers of kills (and with the "correct" MMR".  There is no way the MatchMaker can tell which 1250MMR player is actually really good.  It really just implies we need to get some more aggressive mid-match balance code that acts based on "real-world" performance within the match.  That seems to be where the opportunities lie.

 

In theory shouldn't the win percentage still a lot closer? Why does the anomaly keep ending up on the same team?

Edited by (TDM)crockrocket, 27 May 2015 - 02:29 PM.

[Dr_Freeze001]

umm, lets see, i know Valkirie is front line. alleys i'm guessing is prosk, titan and andromeda... no clue

 

they have to be uptown and origin, but which is which?

Titan is Origin

Andromeda is Prosk

Alleys is Uptown

Edited by (KDR) Dr_Freeze001, 27 May 2015 - 02:28 PM.

[hestoned]

Andromeda = prosk
Titan = origin
alleys = uptown
Valkirie = frontline

[capnjosh]

 

In theory shouldn't the win percentage still a lot closer? Why does the anomaly keep ending up on the same team?

 

That's a good point.  It may take a deep dive into exactly how the game server assigns which team is 1 and which is 0.  And, could that indicate something about how it attempts to auto-balance teams during the match?  For that matter, could it indicate some predictability with how it assigns teams?  Sounds like we have some very specific questions to answer there.

[crockrocket]

That's a good point.  It may take a deep dive into exactly how the game server assigns which team is 1 and which is 0.  And, could that indicate something about how it attempts to auto-balance teams during the match?  For that matter, could it indicate some predictability with how it assigns teams?  Sounds like we have some very specific questions to answer there.

That's exactly what I was wondering :)

 

 

EDIT: Wait so you're saying that 1 or 0 doesn't indicate a specific team spawn? Like team 1 could be assigned either team as far as what we see on our end?

Edited by (TDM)crockrocket, 27 May 2015 - 02:36 PM.

[brackets]

Mode 11 is where we see the massive gap between team-0 and team-1 on lower average MMR games -- and in fact only maps which have this mode have such a massive gap in low-mmr team performance. Is mode 11 bots-tdm? It has the same maps I believe; then the problem is simply that low-MMR bots are too strong.

[Amidatelion]

Can we get some context as to what number is what game mode, and 1 or 0 for prosk or sentium?

 

 

 

 

In theory shouldn't the win percentage still a lot closer? Why does the anomaly keep ending up on the same team?

 

Quoting for which team = which number info plz

 

EDIT: on close examination, here are the game mode types

 

0 - SG

2 - TDM

4 - MA

7 - Bot Destruction

11 - BotsTDM

Edited by Amidatelion, 27 May 2015 - 02:49 PM.

[Amidatelion]

Mode 11 is where we see the massive gap between team-0 and team-1 on lower average MMR games -- and in fact only maps which have this mode have such a massive gap in low-mmr team performance. Is mode 11 bots-tdm? It has the same maps I believe; then the problem is simply that low-MMR bots are too strong.

 

That's botstdm. Look at the map rotation.

 

Ditto for 7, its bot destruction, hence the 100% win rate.

[capnjosh]

That's exactly what I was wondering :)

 

 

EDIT: Wait so you're saying that 1 or 0 doesn't indicate a specific team spawn? Like team 1 could be assigned either team as far as what we see on our end?

 

Here's what I found in an old analytics DB.  I'll be adding this lookup table for use in future runs:

 

Id Name

0 R_Siege

1 R_Deathmatch

2 R_TeamDeathmatch

3 UNUSED

4 R_MissileAssault

5 R_HordeMode

6 (UNUSED)

7 R_Coop

8 R_Ctf

9 any

10 HawkenTR

11 R_BotsTDM

[brackets]

That's botstdm. Look at the map rotation.
 
Ditto for 7, its bot destruction, hence the 100% win rate.

So confirmed -- the peculiar behaviour at 1200 MMR is due to OP bots; bots-tdm is most prevalent with low MMR players, and 1300+ MMR players have developed the skills necessary to defeat them (even as they get harder as your MMR increases).

Edited by brackets, 27 May 2015 - 02:42 PM.

[crockrocket]

4 siege 0 MA? My guess from looking at the rotation. And 2 is tdm

 

Ninja'd

 

 

Hmm #5 and #8. HawkenTR? 

Edited by (TDM)crockrocket, 27 May 2015 - 02:43 PM.

[capnjosh]

Removing things like Coop and BotsTDM, here's what comes out (and adding friendly names in):

 

[brackets]

I think you want to filter coop bot destruction and bots TDM out of the data, because they skew the results especially at low MMRs.

[edit] Hey you did this already!

Edited by brackets, 27 May 2015 - 02:48 PM.

[SS396]

That's a good point.  It may take a deep dive into exactly how the game server assigns which team is 1 and which is 0.  And, could that indicate something about how it attempts to auto-balance teams during the match?  For that matter, could it indicate some predictability with how it assigns teams?  Sounds like we have some very specific questions to answer there.

So...

 

if a veteran were smurfing, the balancer shouldn't place them on the same team every match, unless the balancer uses some sort of algorithm based on callsign.  I would expect that it would be completely random as to which team was assigned.

 

However, I thave a feeling I think I know whats going on, PARTIES.  If two veterans smurf, and join a party, I'll bet the balancer always places that party on the same team, and then tries to figure out what to do with the rest of the players.  Its why at the start of a match when theres a party in the lobby, you see the scoreboard freak out and move players around numerous times, until it finds something valid.

 

The question is, do you have data from the time when parties did not exist to compare the two scenarios?

[capnjosh]

 

EDIT: Wait so you're saying that 1 or 0 doesn't indicate a specific team spawn? Like team 1 could be assigned either team as far as what we see on our end?

 

It's something to confirm of course, but I was thinking more along the lines of "does the game server consistently place the highest MMR player on one team, so does that indicate some predictability in the outcome).

[?FTD? eXeon]

8 R_Ctf

Tell me more please

[crockrocket]

It's something to confirm of course, but I was thinking more along the lines of "does the game server consistently place the highest MMR player on one team, so does that indicate some predictability in the outcome).

 

I meant, do these particular 1's and 0's have a correspondence to which spawn. For example, in Bazaar, can team 1 (after mm creates team 1) be assigned to either high or low spawn, or is 1 always high spawn. If the 1/0 corresponds to a specific spawn, it could be possible the issue lies with map balance rather than mm.

 

I see what your saying though, and if that is the case I guess it doesn't matter if the teams are tied to spawn or not.

[Amidatelion]

So...

 

if a veteran were smurfing, the balancer shouldn't place them on the same team every match, unless the balancer uses some sort of algorithm based on callsign.  I would expect that it would be completely random as to which team was assigned.

 

However, I thave a feeling I think I know whats going on, PARTIES.  If two veterans smurf, and join a party, I'll bet the balancer always places that party on the same team, and then tries to figure out what to do with the rest of the players.  Its why at the start of a match when theres a party in the lobby, you see the scoreboard freak out and move players around numerous times, until it finds something valid.

 

The question is, do you have data from the time when parties did not exist to compare the two scenarios?

 

I actually thought this myself a while back and kept an eye out for people I knew were partied. The matchmaker did seem to prefer placing a given party on Sentium, but it wasn't 100%. Over the course of 7 games, it went to Sentium 6 times. Not sure what happened on the off one and clearly more data is required.

[crockrocket]

  Its why at the start of a match when theres a party in the lobby, you see the scoreboard freak out and move players around numerous times, until it finds something valid.

That little pre-match freak out may also be indicative of where it tends to place the highest player, even when not partied. 

[SS396]

Tell me more please

capture the flag....  internal map code name rogue.  first team to score 3 points wins, 20 minute long match.  I don't know how far it was in production but it was fairly close from the last time I heard about it.

[capnjosh]

So...

 

if a veteran were smurfing, the balancer shouldn't place them on the same team every match, unless the balancer uses some sort of algorithm based on callsign.  I would expect that it would be completely random as to which team was assigned.

 

However, I thave a feeling I think I know whats going on, PARTIES.  If two veterans smurf, and join a party, I'll bet the balancer always places that party on the same team, and then tries to figure out what to do with the rest of the players.  Its why at the start of a match when theres a party in the lobby, you see the scoreboard freak out and move players around numerous times, until it finds something valid.

 

The question is, do you have data from the time when parties did not exist to compare the two scenarios?

 

 I rechecked the logs to confirm - there's nothing that tags users as being in a common party.  It appears we'll have to add that.

[crockrocket]

We could probably do some tests on the ts. Maybe throw a 2600+ in with is in a sub 2k lobby for a full map rotation or two and see if there's a trend. Same with parties.

Edited by (TDM)crockrocket, 27 May 2015 - 02:59 PM.

[capnjosh]

I meant, do these particular 1's and 0's have a correspondence to which spawn. For example, in Bazaar, can team 1 (after mm creates team 1) be assigned to either high or low spawn, or is 1 always high spawn. If the 1/0 corresponds to a specific spawn, it could be possible the issue lies with map balance rather than mm.

 

I see what your saying though, and if that is the case I guess it doesn't matter if the teams are tied to spawn or not.

 

I expect the 1s and 0s are consistent team numbers that can predict spawn locations - that's how I would have done it; however, if they happened to start out with a random number generator or something when initializing the map.... It's on the "to be cofirmed list".  Yay for uncertainties.

[LoC_TR]

I'm surprised at how lopsided some of the uniform maps like wreckage and origin on missile assault at higher mmrs. These are average mmrs of the match correct? Something tells me these get a bit skewed because it's attempting to balance an odd amount of players more often. Is this data pulled since you made the mmr threshold changes or from the past year or so? Does this also include private servers?

[Amidatelion]

I'm surprised at how lopsided some of the uniform maps like wreckage and origin on missile assault at higher mmrs. These are average mmrs of the match correct? Something tells me these get a bit skewed because it's attempting to balance an odd amount of players more often. Is this data pulled since you made the mmr threshold changes or from the past year or so? Does this also include private servers?

 

This has to go a ways back - Bazaar Siege is on there.

[SS396]

That little pre-match freak out may also be indicative of where it tends to place the highest player, even when not partied. 

I don't think so, mostly because when there are no parties present the scoreboard doesn't freak out multiple times moving people around.

 

 

This has to go a ways back - Bazaar Siege is on there.

Not exactly, some private servers have enabled that map for siege, like Charcoals.  I've played it a few times lately.