Jump to content

Photo

MatchMaking results after the 800MMR range week

- - - - -

  • Please log in to reply
31 replies to this topic

#1
capnjosh

capnjosh

    Advanced Member

  • Administrators
  • 262 posts

Some people have suggested I may have not been as "interacty" of late.  Oh, for shame!  Well, yeah, you're right.  I've scheduled more forum/interaction time; it make sense after all!  Besides, I like you guys, so I need to make extra sure I communicate that better.

 

Anyways, on to the datas!

 

We set the acceptable MMR to 800 plus or minus.  Predictably, this would have the effect of matching much higher and much lower MMR players together.  Well, here's what the data shows:

 

This is the avg MMR spread of the winning vs losing team  Yeah, the winning team went up a little more than did the losing team:

NVwxREE.png

 

 

This is another view of nearly the same data, showing the avg MMR spread and the Standard Deviation of that spread.  As an overall view, it definitely shows that high-MMR players indeed were matched right along side lower-MMR players:
Va8mHwU.png

 

 

This shows the Standard Deviation of the Kill Spread and Max MMR player on each team.  That jump up in August was from the 16-slot TDM servers.  Notice both jumped up during the 800MMR time period, with the winning team showing slightly larger increases than the losing team:

F2TfWDe.png

 

 

Now what did all this do for retention rates?  The top orange line is "1-Day Retention", meaning the % of first time logins that came back again.  The other lines, in order, are 3-day retention rates, 7-day, 15-day, and 30-day.  I've not included the actual numbers, since this sort of data is something that's usually kept pretty confidential, so feel very very speshal!!! ;)  It's interesting, the 800MMR range certainly didn't destroy anything, though generally you'd want a longer test period :

XlBraU7.png

 

 

 

We've taken another look at the MatchMaker code, and we made a few changes that may improve the matchmaking experience:

 

1. Focus more on matching players to the highest-suitable-MMR match for them, rather than throwing that out the window in favor of putting players in a match to address the case where there is a one-player difference.

 

The expected effect is that there will be fewer cases where a high-MMR player is put into a match that is *actually* pretty well-balanced, but because one team has an extra player, the matchmaker prioritizes filling that slot at the expense of overall MMR-based matchmaking.  A hoped-for knock-on effect is that fewer people feel as much of the frustration that leads to leaving a match early, which further exacerbates the situation.

 

 


  • phed, dorobo, Dr_Freeze001 and 11 others like this

#2
cbrxx

cbrxx

    Advanced Member

  • Members
  • PipPipPip
  • 67 posts

I'd need someone smarter than me to explain those graphs in person. 


  • nepacaka and System64 like this

#3
phed

phed

    Advanced Member

  • Members
  • PipPipPip
  • 311 posts

we like you too cap!


  • System64 likes this

#4
capnjosh

capnjosh

    Advanced Member

  • Administrators
  • 262 posts

Ashfire said I need to do a better job of annotating them.  He's right.  For future ones I'll work on efficiently doing that (it can be so very time-consuming).

 

In this case, we really want that last chart - the Retention Rate one - to move up.  That means more new players stick around longer.  The top charts we generally expect to be ones where we want the lines to get closer and move down (indicating teams with players of "more equal" skill level and matches demonstrating closer levels of competition).


  • MomOw and System64 like this

#5
DieselCat

DieselCat

    Advanced Member

  • Members
  • PipPipPip
  • 711 posts

Do players forming parties have any kind of  effect on these numbers ?   I'm assuming that partied groups are still a viable function,  which can throw the matchmaking out of whack.

 

*+


  • StubbornPuppet likes this

Just Relax....and take life one game at a time....

Don't run to your death....walk

 

th_Duckman.jpg   th_82c0a97c-98de-4aac-be47-05e5e099be80.

 

*+

 


#6
6ixxer

6ixxer

    Advanced Member

  • Members
  • PipPipPip
  • 1158 posts

the change of priority to not just fill the slot, but aim for the match skill will hopefully mean noobs feel less stomped and stay longer.

 

I noticed the higher MMR split recently, I took some SRD screenshots and saw a match with mid 1600 average had a max of 1900 and then things changed. a 2250 MMR entered and bumped the average to mid 1800. they had a significant impact and meant their team won convincingly which didn't bother me as I saw the ringin and the next match was fair after autobalance.



#7
MomOw

MomOw

    Advanced Member

  • Members
  • PipPipPip
  • 1039 posts

@captnjosh : thx for sharing all that data ! Are you ploting graphs depending on date for us to see the trend ?

I guess that you have tools to analyse all these data :-) ... we do have this at work http://www.minitab.com/en-us/

I mean that the pertinent graoh should be :

- win ratio depending on std MMR dev

- retention depending on win ratio


Edited by MomOw, 08 September 2015 - 09:13 PM.

IRZUTYo.png

gXO9Nfd.pngmXasTsY.pngft4VqcE.png

 

KDR Member | Streamer | Priority Target II

Spoiler

#8
Rainbow_Sheep

Rainbow_Sheep

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Besides, I like you guys


tumblr_lfi0d14ciE1qaxft4.png
  • dorobo, Miscellaneous, LaurenEmily and 1 other like this

Spoiler

#9
dorobo

dorobo

    Advanced Member

  • Members
  • PipPipPip
  • 990 posts

Not sure how hard that would be to implement but what if we had like a slider for mmr range ingame? Everyone could decide that judging by their needs and abilities. This would probably break many things and messup all these pretty graphs  :ninja:

 

work-work-work-work.gif?w=682&h=282



#10
DieselCat

DieselCat

    Advanced Member

  • Members
  • PipPipPip
  • 711 posts

Could always go back to that bracket system from closed beta where one could choose a grouped mmr with a set +/-  such as 1200-1500, 1600-1800, 1900-2100 etc. or whatever it was as a temporary fix till something better can be devised (i.e. more players)

 

I played a lot today and was entered into matches with a mmr spread avg. of 1600-2250. except for some really lopsided games it seemed to work not great, but (fairly well)

 

*+ 


  • dorobo likes this

Just Relax....and take life one game at a time....

Don't run to your death....walk

 

th_Duckman.jpg   th_82c0a97c-98de-4aac-be47-05e5e099be80.

 

*+

 


#11
crockrocket

crockrocket

    Advanced Member

  • Members
  • PipPipPip
  • 1989 posts

Could always go back to that bracket system from closed beta where one could choose a grouped mmr with a set +/-  such as 1200-1500, 1600-1800, 1900-2100 etc. or whatever it was as a temporary fix till something better can be devised (i.e. more players)

 

I played a lot today and was entered into matches with a mmr spread avg. of 1600-2250. except for some really lopsided games it seemed to work not great, but (fairly well)

 

*+ 

 

I kinda like this. 1900-2100 is what I actually want in a pub server right now; maybe this system would help the right matches form in general. The next category in this should be 2100+, and likewise the low end should be x-. I would like to see this system tried without locking, or perhaps limited to a section above or below you, though that might fck over the outliers. 


                                                                    JgQjgkx.png

 

Salvage: An Idea to Stop Leavers

Player Retention & Howken

 

[14:31] <Crafty> I know that in my balls
[14:32] <Crafty> hawken is unlike anything Ive played

 

 


#12
StubbornPuppet

StubbornPuppet

    Advanced Member

  • Members
  • PipPipPip
  • 1508 posts

Like Diesel Cat was saying, "Parties really mess up the balancer."

I'm not opposed to being able to "party up" - that's a good thing.

What should never happen is the balancer keeping all players in a party on the same team.

 

Taking out that ooonnnneee little oversight on the part of Adhesive would make a huge difference.


  • _incitatus, DieselCat, 1uster and 1 other like this

To be serious for a moment this is just a joke

 


#13
_incitatus

_incitatus

    Advanced Member

  • Members
  • PipPipPip
  • 994 posts

Like Diesel Cat was saying, "Parties really mess up the balancer."
I'm not opposed to being able to "party up" - that's a good thing.
What should never happen is the balancer keeping all players in a party on the same team.

Taking out that ooonnnneee little oversight on the part of Adhesive would make a huge difference.


Especially when you have 2500+ players partied up together in pubs. Hard to balance teams when you can't split the two highest players in the lobby to different teams.
  • DieselCat and StubbornPuppet like this

#14
DieselCat

DieselCat

    Advanced Member

  • Members
  • PipPipPip
  • 711 posts

Johnny says to is buds Joey and Clarence...."Hey Bros...lets do a party group and roll in Hawken.....Yeah...right-on man, we're gonna rekt this fuzzy bunny !"

 

But don't cry about it if you and your grab-ass friends get split up on different teams from time to time. You party goers will survive and not add to fuzzy bunnyng up MM.

 

Sorry...but this fuzzy bunny just messes up my blood pressure.  :wallbash:

 

*+


Just Relax....and take life one game at a time....

Don't run to your death....walk

 

th_Duckman.jpg   th_82c0a97c-98de-4aac-be47-05e5e099be80.

 

*+

 


#15
Call_Me_Ishmael

Call_Me_Ishmael

    Advanced Member

  • Members
  • PipPipPip
  • 1114 posts

I VERY RARELY ever party up.  Maybe 10 times outside of (ages ago) TAW practice.

 

I agree it wrecks games and is largely why I don't do it.  I think it's easy to spot in some cases, and not very easy to detect in others.  It does motivate me to try harder.


Edited by Call_Me_Ishmael, 09 September 2015 - 03:28 PM.

Did I say Call Me Ishmael?

 

You should call me Luna.


#16
IareDave

IareDave

    Advanced Member

  • Members
  • PipPipPip
  • 359 posts
The matchmaking still refuses to place me in a match closest to my mmr and instead places me with sub 1200 mmr (as I stated in a thread I made a few days ago).
  • Veritasnous and kaiserschmarrn_ like this

#17
JeffMagnum

JeffMagnum

    Advanced Member

  • Members
  • PipPipPip
  • 781 posts

The matchmaking still refuses to place me in a match closest to my mmr and instead places me with sub 1200 mmr (as I stated in a thread I made a few days ago).

 

working as intended

git gud


  • 1uster, Veritasnous and kaiserschmarrn_ like this

d1eZeG3.png


#18
Pleasure_Mortar

Pleasure_Mortar

    Advanced Member

  • Members
  • PipPipPip
  • 361 posts

Just out of curiosity I re-installed Hawken last night 

First game I joined was TDM 5-32 and 7 minutes left. MMR range 1450-2600.

I jumped in (and immediately out) of a couple of similar games. 

 

The games I decided to stay had usually a few (sometimes just one) good players and a couple of really bad ones.

One out of 5 encounters with an enemy resulted in a good challenging fight and the rest was just stomping.

 

I'm not sure if the 800MMR range has been canceled again, but I didn't have any fun at all.

Uninstalled the game again after a couple of hours



#19
Hyginos

Hyginos

    Advanced Member

  • Members
  • PipPipPip
  • 1337 posts

Does the matchmaker ask why it can't find a suitable game? My theory is that after searching for a while the MM marks a player as an outlier but doesn't differentiate between high and low outliers. That would explain the 2000 mmr spread DM lobbies.


  • _incitatus likes this

MFW Howken

 

My post count is neat.


#20
6ixxer

6ixxer

    Advanced Member

  • Members
  • PipPipPip
  • 1158 posts
You're hard to please if you installed, just to test and then uninstalled.
...or really hard up for disk space.

The Capn has tested 300-500-800 and always asks for feedback so he learns if it is a good experience.
If you uninstall over not liking something that is being tested then how do we know you aren't just too picky and not worth listening to anyway.

I found your feedback somewhat ambiguous and overly self important when i believe objective and helpful would have been the desired response.
  • DieselCat and Dawn_of_Ash like this

#21
Pleasure_Mortar

Pleasure_Mortar

    Advanced Member

  • Members
  • PipPipPip
  • 361 posts

You're hard to please if you installed, just to test and then uninstalled.
...or really hard up for disk space.

The Capn has tested 300-500-800 and always asks for feedback so he learns if it is a good experience.
If you uninstall over not liking something that is being tested then how do we know you aren't just too picky and not worth listening to anyway.

I found your feedback somewhat ambiguous and overly self important when i believe objective and helpful would have been the desired response.

 

I'm kinda hard up or diskspace but that's not the reason I uninstalled the game. 

The reason is when I have Hawken installed I often tend to just jump in for a quick game or two, then get frustrated by the imbalanced games but instead of quitting immediately I give it another try and maybe another try until I spend too much time in the game without really enjoying it. So just avoid getting lured in again and again I uninstalled the game. 

 

When I provide feedback I keep it just to how the experience was for me. Usually that's what ppl designing userexperiences are looking for.

It's not my job to figure out how to change things and I'm certainly not going to start to do lists for Reloaded or tell them what to do.

 


  • dorobo, BimboRotto and DeeRax like this

#22
HorseHeadProphet

HorseHeadProphet

    Advanced Member

  • Members
  • PipPipPip
  • 167 posts

800 spread had the servers wide open. It was wild. Jumping into servers with 1400 players had me like:

 

Spoiler

 

But the stompings really were too severe. If the problem of one team blowing out the other was bad before the 800 spread, it was ten times worse now. People were ragequitting, hackusing, crying OP.

 

The reality was that they were simply the level of player that will try and attack a Brawler head-on in a Reaper, or they present a stationary target, or they'd attack when outnumbered, or allow me to make completely ridiculous escapes, time and time again. Those players need more time to develop their skills and level up, not get mashed out of being interested in the game altogether. Timing the 10K reward weekend with a spread like that, made that weekend like an all-you-can-eat buffet of beef for mid to high level players, and an agonizing few days of cruel abuse for the newer/lower mmr players.

 

 

 That kind of online experience just couldn't be allowed to continue, due to these, IMHO, completely predictable results.

 

And today we're back to:

 

Spoiler

 

I mean the only real options the game has, as I see it, are to set the range somewhere in the middle. Or overhaul the matchmaking/mmr system entirely. The size of the player base leaves only so many options.

 

 


The Hawken forum's a forum, but it's like the only one you get yelled at for using.


#23
Sylhiri

Sylhiri

    Advanced Member

  • Members
  • PipPipPip
  • 403 posts

Those players need more time to develop their skills and level up, not get mashed out of being interested in the game altogether. Timing the 10K reward weekend with a spread like that, made that weekend like an all-you-can-eat buffet of beef for mid to high level players, and an agonizing few days of cruel abuse for the newer/lower mmr players.

 

You'd think they would take this time to learn from high MMR players. They are doing them a favor by teaching them through experience, low MMR players should be thankful. Geez



#24
Hyginos

Hyginos

    Advanced Member

  • Members
  • PipPipPip
  • 1337 posts

If you're having trouble finding a game, hop on one of the non-passworded private servers and sit there for a bit. If enough people make a habit of that we may be able to self correct to some degree.


MFW Howken

 

My post count is neat.


#25
DieselCat

DieselCat

    Advanced Member

  • Members
  • PipPipPip
  • 711 posts

You'd think they would take this time to learn from high MMR players. They are doing them a favor by teaching them through experience, low MMR players should be thankful. Geez

 

Though maybe new players should or just be aware of what you're saying, I don't think they're immediately looking at it from that perspective when first starting. Unless that player is really experience from other FPS and can adapt very quickly.

 

*+ 


Just Relax....and take life one game at a time....

Don't run to your death....walk

 

th_Duckman.jpg   th_82c0a97c-98de-4aac-be47-05e5e099be80.

 

*+

 


#26
Dedhed

Dedhed

    Advanced Member

  • Members
  • PipPipPip
  • 196 posts

Are there changes to the auto balancer being made as well as match maker? Or is that even necessary? Also, how do you see the mmr range for other players in a match?


"One day I will leave this world and dream myself to reality" -- Chief Crazy Horse

 

"Love is the Law, Love under Will" -- Aleister Crowley

 

"This is like talking to breakfast cereal" -- Otherland


#27
pheint

pheint

    Newbie

  • Members
  • Pip
  • 2 posts

So I got back into Hawken this past week after a year hiatus, and fuzzy bunny did I have some good times and more recently not so good times.  It seemed in every match there was someone who completely destroyed, myself included.  And while it was entertaining when the pop. was higher, since the weekend excitement has dwindled so has the mm quantity=/=quality.  For a new / returning person wanting to try out new stuff it didn't work at all.  After my matches I'd check profiles and most often it was with people that have less total XP than I do against 1-2 people with 10x more XP.  Granted experience does  not equal skill but it does offer some level of an advantage.  I played with and against Analysis today and while it was good fun and he was a nice guy, the matches were completely onesided, leading to many 5-6v2s not something someone jumps into a game looking forward to.  

 

Would figuring xp amounts into the mm process improve the overall experience?  Perhaps.


  • DieselCat likes this
"Because while the truncheon may be used in lieu of conversation, words will always retain their power.
Words offer the means to meaning, and for those who listen, the enunciation of truth."

#28
DieselCat

DieselCat

    Advanced Member

  • Members
  • PipPipPip
  • 711 posts

So I got back into Hawken this past week after a year hiatus, and fuzzy bunny did I have some good times and more recently not so good times.  It seemed in every match there was someone who completely destroyed, myself included.  And while it was entertaining when the pop. was higher, since the weekend excitement has dwindled so has the mm quantity=/=quality.  For a new / returning person wanting to try out new stuff it didn't work at all.  After my matches I'd check profiles and most often it was with people that have less total XP than I do against 1-2 people with 10x more XP.  Granted experience does  not equal skill but it does offer some level of an advantage.  I played with and against Analysis today and while it was good fun and he was a nice guy, the matches were completely onesided, leading to many 5-6v2s not something someone jumps into a game looking forward to.  

 

Would figuring xp amounts into the mm process improve the overall experience?  Perhaps.

 

You make some good points here and I believe I was there with you in some of those match today.

 

This afternoon I was playing for a while and there weren't many people on but there were some and a small selection of matches to choose from. What I saw was a form of musical chairs between different servers with the same groups of players leaving then re entering later once that server went to a different map.

 

I understand the map selection can get boring and some like to find familiar faces, or a player is having a difficult time with lag or high ping issues, I get that.... but at present,  that's all we have.  

What I think some fail to understand, or maybe some do but just don't care, is that people coming and going in a server screws up the matchmaking abilities of the MM to try and do it's job.

 

If people can stay for consecutive matches in a given server, it's more likely to end up being better balanced between teams, (unless it's a group formed up in a party) than if players are constantly jumping in and out of games.

 

Stay if one can, more times than not,  it will improve the team balance for better matches.

 

*+ 


  • pheint likes this

Just Relax....and take life one game at a time....

Don't run to your death....walk

 

th_Duckman.jpg   th_82c0a97c-98de-4aac-be47-05e5e099be80.

 

*+

 


#29
crockrocket

crockrocket

    Advanced Member

  • Members
  • PipPipPip
  • 1989 posts

I'm assuming you forgot to mention that after said week the range is not 800 anymore. Care to enlighten us as to what it is currently?

 

 

EDIT: Unless someone joined right after I checked herokuapp, it is currently less than 350. Account at ~1460 could not manually join 1800 mmr server.


Edited by (TDM)crockrocket, 11 September 2015 - 02:33 AM.

                                                                    JgQjgkx.png

 

Salvage: An Idea to Stop Leavers

Player Retention & Howken

 

[14:31] <Crafty> I know that in my balls
[14:32] <Crafty> hawken is unlike anything Ive played

 

 


#30
pheint

pheint

    Newbie

  • Members
  • Pip
  • 2 posts

You make some good points here and I believe I was there with you in some of those match today.

 

This afternoon I was playing for a while and there weren't many people on but there were some and a small selection of matches to choose from. What I saw was a form of musical chairs between different servers with the same groups of players leaving then re entering later once that server went to a different map.

 

I understand the map selection can get boring and some like to find familiar faces, or a player is having a difficult time with lag or high ping issues, I get that.... but at present,  that's all we have.  

What I think some fail to understand, or maybe some do but just don't care, is that people coming and going in a server screws up the matchmaking abilities of the MM to try and do it's job.

 

If people can stay for consecutive matches in a given server, it's more likely to end up being better balanced between teams, (unless it's a group formed up in a party) than if players are constantly jumping in and out of games.

 

Stay if one can, more times than not,  it will improve the team balance for better matches.

 

*+ 

 

We did play a few together yesterday ya I remember.  Nabbed you on my friends list from over the weekend too. :)  

 

Anyways, yes I agree that the longer the MM has to work with a set group of players the more equal it will balance those teams.  I noticed you trying to encourage people to stay instead of dropping out so the MM can do it's job; I'll try and do the same as I am not anywhere near the top as far as MMR so maybe I can impel some of the less experienced players to stick it out another round too.

 

What Josh mentioned they are working on at the end of his post is encouraging.  There was more than a few games that I dropped into that were fairly low-scoring even after 3-4 minutes, for which I was clearly not intended but was just to fill that one player disparity and utterly ruined the game for the other team.


  • DieselCat likes this
"Because while the truncheon may be used in lieu of conversation, words will always retain their power.
Words offer the means to meaning, and for those who listen, the enunciation of truth."

#31
Dedhed

Dedhed

    Advanced Member

  • Members
  • PipPipPip
  • 196 posts

Found the answer to my question regarding mmr in servers: http://hawken.herokuapp.com/

 

Thanks to Shadeness for making this


"One day I will leave this world and dream myself to reality" -- Chief Crazy Horse

 

"Love is the Law, Love under Will" -- Aleister Crowley

 

"This is like talking to breakfast cereal" -- Otherland


#32
StubbornPuppet

StubbornPuppet

    Advanced Member

  • Members
  • PipPipPip
  • 1508 posts

Still, the problem with tinkering with the matchmaking system is that there aren't enough players for any reasonable system that accomodates a happy level of skill segregation to work.  That's why I'd rather they not bother with it too much right now - if they build a system that works now, it probably won't work well when/if Hawken gets a bunch of new players.


To be serious for a moment this is just a joke

 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users