110 replies to this topic

[Saturnine]

Someone mentioned it in the data collection thread I believe, but it'd make things a lot easier for you, and for respondents if this sort of research was compiled into a google survey. Just create the survey, send the link out, and get respondents with easily compilable data. You'll be able to get a wider subset of the population, and there'll be smaller barriers in the way of completing and interpreting it. 

 

A few people have said that 40/700 is a good sample size, but the problem with that is is you're comparing the number of simultaneous players at a given point to the amount of respondents - the active player base is measured in the thousands. 

 

Anyway, if you were to make a google survey of these and related questions, you could link it on here, on reddit, on the facebook page, through twitter, on the steam forum, on a steam announcement, people could link it in game if we had a nice shortened link for it, etc etc. I feel like we could get a much wider sample that could better represent players as a whole, rather than the forum users.

 

Either way, thanks a ton for this work you've done! It was a lot of effort to go through, and this information is certainly very useful and interesting to read through. :)

[LRod]

Ishmael, thanks for the reply. I agree on the sample size being relevant but disagree on the sample distribution.

Jeff, I understand the intention of the survey, and also appreciate the thoughtful approach.

Spanner, I agree completely with points 1 & 2, and think this is great preliminary data when viewed through the lens of analyzing a specific subset of the population.

And 'yes', I LOVE data and good discussion on the analysis. Bravo all around!

[Mergaz]

As the average MMR of respondents is above the population mean, just do a search with the same number of players on each tier of MMR (8-10 players to each rating range). Thus, even given the small sample size of 40-50 players could have a more faithful representation since most players use the forum have higher MMR and therefore more interest in topics for discussion, but are a small % of total pop.

That's why she lost credibility because clearly only the players of smaller portion (high MMR) of the population actively opined.

Edited by Mergaz, 28 March 2015 - 07:56 AM.

[Mergaz]

Sorry, double post

Edited by Mergaz, 28 March 2015 - 07:47 AM.

[OdinTheWise]

 

but yes, you should have used google survey

Edited by OdinTheWise, 28 March 2015 - 08:40 AM.

[(Unknown)1590d2c747fabd]

I am out of LIKEs for the day so virtual high fives all around:

 

[AsianJoyKiller]

There's an intangible truth that some people are ignoring. And some people are going to get upset at this and make claims about how it's just "elitism", but it's observable and verifiable.

The lower you go in MMR, the less a player understands about the game and how certain elements affect balance.

The lower you go in MMR, the less a player notices about the naunces of balance.

So while it's the true average MMR might be in the 1500-1600 range, the fact is, a large majority of those players also have a significantly inferior understanding of the game.

I'm not saying that responses from lower MMRs should be ignored, but when analyzing data and what it means, you have to consider the fact that many of people in that range don't fully understand what they are talking about, and the implications that certain balance decisions could have.

[Amidatelion]

...but it's observable and verifiable.

Evidence: "Hellfires are OP."

[Sylhiri]

Scanner in its current form is almost universally hated.

 

Almost everyone seems to hate Bunker equally.

 

 

I had a tear of joy roll down my cheek when I read these.

[Hijinks_The_Turtle]

At least we all agree on Bunker and Scanner.  Good thing the community has it together on things we really hate.

[eth0]

Numbers are cool and a good read. Thanks Jeff!

[shosca]

There's an intangible truth that some people are ignoring. And some people are going to get upset at this and make claims about how it's just "elitism", but it's observable and verifiable.

The lower you go in MMR, the less a player understands about the game and how certain elements affect balance.

The lower you go in MMR, the less a player notices about the naunces of balance.

So while it's the true average MMR might be in the 1500-1600 range, the fact is, a large majority of those players also have a significantly inferior understanding of the game.

I'm not saying that responses from lower MMRs should be ignored, but when analyzing data and what it means, you have to consider the fact that many of people in that range don't fully understand what they are talking about, and the implications that certain balance decisions could have.

 

mmr distribution (based on data from jan) Mean is 1466.189 and std dev is 138.8235:

Edited by shosca, 28 March 2015 - 01:25 PM.

[Call_Me_Ishmael]

Ishmael, thanks for the reply. I agree on the sample size being relevant but disagree on the sample distribution.

 

A few people have said that 40/700 is a good sample size, but the problem with that is is you're comparing the number of simultaneous players at a given point to the amount of respondents - the active player base is measured in the thousands. 

 

Gents, it has to do with the size of the proportion (i.e. number of options), the confidence level, and the amount of type-II skew you expect.  But... heteroskedastic data will throw this off. It really doesn't depend on the total population (the 700 in S9's quote above) - it's saying the sample you collect has to be x-big to be significant at that confidence level.

 

I agree with S9 - making a Google survey will help a lot.

[Houruck]

I am going to read this thoroughly tomorrow (it is 22:46 here right now).

Why did you split UK and EU into different groups?

 

Edit: I saw your previous thread but never had the chance to post. :(

Edited by Houruck, 28 March 2015 - 01:46 PM.

[deidarall]

Dang it missed the thread to post on, but this is great and interesting reading. 

 

Also, note that the views on air mechanics are still this divided after this amount of time to what I actually consider a surprising amount considering the time, not to mention I would be interested to find out what builds these people started on, preassending or post steam ect. That is a massive data point to consider. As it makes sense for the current community to fixate on things being the same or moving in a direction that is consistent with keeping for example air movement. 

 

Remember hawken lost more and more players due to the nature of design changes in each of it's patches. The players who hate the AC, stuff and the air movement are likely gone. 

 

Cause it has been 10 months. 

 

That is a massive issue I have with how the data is showed here, we have no idea what the general history of the people are voting is build wise. 

Edited by deidarall, 28 March 2015 - 01:52 PM.

[JeffMagnum]

I am going to read this thoroughly tomorrow (it is 22:46 here right now).

Why did you split UK and EU into different groups?

 

Originally I was going to see the effect that ping had on opinions as I assumed that a lot of people would be reporting US-E pings regardless of where they lived, but since almost everyone reported what they got on local servers, I couldn't get much from it. 

 

The main reason I didn't make a survey using Google Forms is because it would've been trivially easy for one person to throw all the data off by completing it multiple times since the sample size was so small. Making people post their responses here was probably more inconvenient for everyone involved, but at least I knew I wasn't wasting my time or misinforming everyone. 

[AsianJoyKiller]

Evidence: "Hellfires are OP."

Exactly. Or how the Scout is "OP". Or how it's "impossible" to hit AC users. Those are obvious examples, but it does affect how people view the game in more subtle ways.

Edited by AsianJoyKiller, 28 March 2015 - 02:22 PM.

[talon70]

Most interesting stats. Even being a small sample its got really really good info. Great job!

 

Thanks for the hard work!!!!

[crockrocket]

Today... Today OP was a pretty cool guy.

No one is going to spend weeks and weeks of their precious time collecting the data and analyzing it.

Unless we had a script...

Here are the statistics from Hyginos' histogram, I think they are relevant here.

http://i.imgur.com/mgfmnuW.png

Edited by crockrocket, 28 March 2015 - 04:16 PM.

[Call_Me_Ishmael]

Okay folks, data crunching time.

 

First, I spent about 5 hours rearranging Jeff's data to a form Excel can do analysis on.  Jeff, I beg of you, please use Google Survey for the follow up.

 

Second, the naysayers about having enough data points are correct - not everyone answered each question, and some points are not very statistically-significant because of the sample size.  We should do this again, and get more respondents.  Jeff, I beg of you, please use Google Survey for the follow up...

 

Now I need to explain some things about data analysis. 

1. I can't crunch non-numeric data in Excel (Excel just can't do it.  MatLab or MiniTab might, but I have Excel).  So, I had to create a key for each multiple-choice.  In general, I put the tightest restriction in the choice selections as 1, no-change (say, to air-speed) as 3, and the most dev effort/biggest/most liberal change to the game as 5 or 6.  Further, I had to carefully replace/place 0 where there was no response from a user, and 1 in some cases where no response was because of the way Jeff created the question (i.e. 'There should be a leaderboard' maps to 2, and 'There should not be a leaderboard' maps to 1).
   
   This means I have to be VERY careful with the output, to make sure I sort the input data on non-0 values, then delete the 0-value rows from the data set.
    
2. Excel 2007 could not do multiple regression on more than 16 columns of input to the output variable (limited to 16 x's to 1 y).  I will use Excel 2013 if I ever put my Windows drive back into my other laptop (it's currently Linux).
    
3. Because of the large number of questions, and the inversion of the dependent variables, and 1) and 2) above, it'll take a few days to answer all the questions (such as 'is the desire to hold-steady/slightly increase ground speed heavily dependent upon/correlated with MMR, and is that statistically significant of the larger population as a whole?' BTW, 'yes'.)

So... first some data key/tables:

 

 

 

Some summary stats on the users/ping/region

 

and a histogram of the MMR of the responses, representative of the forum respondents:

 

Next, I did the big-daddy regression (well, to the point I could, that 16-column limitation).  I was looking for multicollinearity or an inverse cause-effect (High MMR is driven by the desire to have shorter TTK, things like that).  I didn't find inverse cause/effect, but did find the expected multicollinearity.  But...

 

That p-value for Ping says something to me, that it COULD be significant with a simpler model.  Note the R-square is high, and p-value is vanishingly-small.  This says that the data is significant, there is a significant correlation between MMR and ping.

 

Every MS increase in ping essentially results in a 22.8 increase in MMR.  Not what you'd expect, huh? 

 

Okay, the last bit (and what will take the longest to do for 23 independent variables/survey questions), is in the form of 'is the median ground speed answer from the community significant, and is the answer controlled by higher-MMR players?'.  This is thorny - because Reloaded has to know/decide if they want to cater to us here reading this (we're pretty-high MMR, see that histogram above), or the general 'casual player'.

 

So, tonight's answer: 

 

YES, this is significant, and yes, it's slightly related to MMR.  Every 1000 points you go up in MMR, you're more likely to get an increase in the desire for increased ground speeds. Here's the distribution - what the regression is saying is that it's the higher-MMR players who want to #increasethespeeds:

 

Enough for tonight.  I can do some of the other 23 in the coming days...

 

 

 

 

[Call_Me_Ishmael]

Just read Jeff's post on Google Survey...

 

I guess if we do a follow up I can devote a few hours grooming data again.

[OdinTheWise]

i think Ahab has found his whale. i have a feeling that Ahab has history with data analysis. but i think a follow study would be helpful. 

[Call_Me_Ishmael]

i think Ahab has found his whale. i have a feeling that Ahab has history with data analysis. but i think a follow study would be helpful. 

 

Heh. I'm a guy who runs business (engineering, planning, business units) by data.  You can't manage what you can't measure...

 

When I finally learned multiple regression, I was like, 'Where the HELL have you been all my life?'

[OdinTheWise]

Heh. I'm a guy who runs business (engineering, planning, business units) by data.  You can't manage what you can't measure...

 

When I finally learned multiple regression, I was like, 'Where the HELL have you been all my life?'

this makes perfect sense 

[Luaq]

In my opinion it is theoretically possible to add such surveys in the game itself. When Laila asked to immediately after entering the game "Help improve the game, honestly answer a few questions."

That's what I've been trying to say in other threads. For mass data, it is useful, now to you treat the information, that is the dev's decision, to make the difference between who really plays, who doesn't and who is just starting off.

My only worry in the results  is about the "raising weapon delay"
I'm REALLY surprised, I should say shocked.

People (the ones who actually voted) really want it out? Isn't it all part of the game ambiance and immersion of being in a heavy mech? I'm gonna go a little bit off topic here since it wasn't a subject in the charts but when the devs added a Visor (HP and boost bar) instead of leaving this information where it was suited to be, on the dashboard like any man driven machine, really got my "immersion of being in a mech" down. I kindda feel a weapon delay would also add to that deception. But I guess the numbers have it right for some reason.

All that said though, THANK YOU op for all this! Even it is not quite representative, it is still very interesting. at least to see that a portion of people who are active in the forums made their point.

Edited by Luaq, 28 March 2015 - 08:43 PM.

[M4st0d0n]

There's an intangible truth that some people are ignoring. And some people are going to get upset at this and make claims about how it's just "elitism", but it's observable and verifiable.

The lower you go in MMR, the less a player understands about the game and how certain elements affect balance.

The lower you go in MMR, the less a player notices about the naunces of balance.

So while it's the true average MMR might be in the 1500-1600 range, the fact is, a large majority of those players also have a significantly inferior understanding of the game.

I'm not saying that responses from lower MMRs should be ignored, but when analyzing data and what it means, you have to consider the fact that many of people in that range don't fully understand what they are talking about, and the implications that certain balance decisions could have.

 

I'm all for balancing the game with the insight of pro players, but some people need a little reality check. The comp scene is very small and I couldnt think of anyone making a living with Hawken. Wich is the definition of "pro". So let's just keep it at the betatesting stage and listen to everyone, from the beginner to the statpadder who played 1000 hours of one mech and didnt bother to buy the others.

[FlamingBeaker]

My only worry in the results  is about the "raising weapon delay"
I'm REALLY surprised, I should say shocked.

People (the ones who actually voted) really want it out? Isn't it all part of the game ambiance and immersion of being in a heavy mech? I'm gonna go a little bit off topic here since it wasn't a subject in the charts but when the devs added a Visor (HP and boost bar) instead of leaving this information where it was suited to be, on the dashboard like any man driven machine, really got my "immersion of being in a mech" down. I kindda feel a weapon delay would also add to that deception. But I guess the numbers have it right for some reason.
 

 

See this thread:

 

https://community.pl...boosting/page-2

 

More specificly, this part from PCP_MD:

 

<quote>

 

To illustrate what i mean, i'll compare the scout and the zerker. The scout has higher boost and walk speed, while the zerker is faster in the air. The zerker lost the shorthop attack, but that was the glitchfix, not the boost delay increase. The scout on the otherhand lost the ability to quickly reposition closer to an enemy to maintain effective range. This is extremely important for shotgun type weapons because they are worthless at long range and even bad at medium range. After the patch, with the added boost delay, getting in range for a shot with a shotgun meant giving the opponent a free shot at you and giving them far more time to react. Also, when chasing, the extra boost delay gave the opponent a chance to get out of effective range if you simply boosted and fired. It became more useful to boost, jump, then fire, when chasing, because it allowed for more forward velocity than waiting for the boost delay to end, then walking. I mained scout before the patch and would often forward-boost, shoot, forward-boost, shoot (not leaving the ground). After the patch, attempting to maintain effective range meant giving the opponent a free shot at me, every time i wanted to reposition quickly. The clunk is real. Got zerker later on, and despite the huge amount of time i spent maining scout, i was better as a zerker, simply because i was airbourne and didn't have to deal with a delay every time i wanted to fire. Zerkers can fly faster than any walking speed and fire without stopping, AND they have longer range than a scout. Zerkers lost a glitch, scouts lost their whole playstyle.

 

<unquote>

 

 

I never got why the delay was an issue, until this post by PCP_MD made it clear by example.

[Dancing]

I personally would like a survey like this sent out to all current players in game, with a reward of MC. Games are kind of funny because they need to strike the balance between pandering to the elite and the crowd. If this game ends up becoming a Quake-clone with jet boosters, why would I continue to play it?

[Mergaz]

I personally would like a survey like this sent out to all current players in game, with a reward of MC. Games are kind of funny because they need to strike the balance between pandering to the elite and the crowd. If this game ends up becoming a Quake-clone with jet boosters, why would I continue to play it?

This poll of most players with high average MMR representing less than 5% of the playerbase. It is because of this and other that this game ended stagnant and old DEVs simply not released new patches due low financial return. Neither bothered to do a search with the same number of players per lane of MMR ...

Edited by Mergaz, 29 March 2015 - 07:47 AM.

[AsianJoyKiller]

I'm all for balancing the game with the insight of pro players, but some people need a little reality check. The comp scene is very small and I couldnt think of anyone making a living with Hawken. Wich is the definition of "pro". So let's just keep it at the betatesting stage and listen to everyone, from the beginner to the statpadder who played 1000 hours of one mech and didnt bother to buy the others.

You can't just balance based on how many players the game has. Doing so is completely idiotic and bad game design. That is an extremely great way to kill a game and lose players.

Also, Mergaz, you need to stop making up lies and spoutng them as truth. It's not like you're the only one here to talk to the devs. There's a lot of players who were around more than you, are more experienced than you, and knew what was going on with ADH better than you. You can't keep making up lies about why ADH failed, because we know for a fact that it's not true and we will call you out on making up fuzzy bunny.

Or, I suppose you can keep lying about it, but every time you do you just will make yourself look stupider and less credible.

[LoneSyndal]

This poll of most players with high average MMR representing less than 5% of the playerbase. It is because of this and other that this game ended stagnant and old DEVs simply not released new patches due low financial return. Neither bothered to do a search with the same number of players per lane of MMR ...

I haven't posted here or even bothered to look here for ages and can safely say, what playerbase?

 

Simply put, the game didn't sit well with many players old or new. Devs simply not having funds for an unfinished game isn't our fault either. My friends who started the game couldnt last more than a day mostly because... there's surprisingly nothing interesting to keep them there in long term.

[Mergaz]

I haven't posted here or even bothered to look here for ages and can safely say, what playerbase?

 

Simply put, the game didn't sit well with many players old or new. Devs simply not having funds for an unfinished game isn't our fault either. My friends who started the game couldnt last more than a day mostly because... there's surprisingly nothing interesting to keep them there in long term.

 DEVs already have science on the players retention problems and a new team is ahead of the game in place of ADH. Let's wait to see if they melhorarm the metagame and add more content to prevent the evasion of players, especially the new pilots who are so necessary for this game sustain life.

We can only hope that the new DEVs are not accommodated and find that the game is good as it is now.

Edited by Mergaz, 29 March 2015 - 03:22 PM.

[AsianJoyKiller]

We can only hope that the new DEVs are not accommodated and find that the game is good as it is now.

You realize they already said they want to keep the game similar to how it is now?

[Mergaz]

You realize they already said they want to keep the game similar to how it is now?

For Real? Hughes said the same thing when they asked him if the steam patch was the final version of Hawken to launch ... I see the failure repeating itself again then ...

[OdinTheWise]

For Real? Hughes said the same thing when they asked him if the steam patch was the final version of Hawken to launch ... I see the failure repeating itself again then ...

i think you are well off base, what you think happened at ADH and acctually happened may be two different things

[Superkamikazee]

There's an intangible truth that some people are ignoring. And some people are going to get upset at this and make claims about how it's just "elitism", but it's observable and verifiable.

The lower you go in MMR, the less a player understands about the game and how certain elements affect balance.

The lower you go in MMR, the less a player notices about the naunces of balance.

So while it's the true average MMR might be in the 1500-1600 range, the fact is, a large majority of those players also have a significantly inferior understanding of the game.

I'm not saying that responses from lower MMRs should be ignored, but when analyzing data and what it means, you have to consider the fact that many of people in that range don't fully understand what they are talking about, and the implications that certain balance decisions could have.

 

Devs need to determine the "right" meta and gameplay mechanics.

 

Game "feel" (speed, responsiveness, immersion) low MMR / new player opinion is imperative to gather. 

 

Game play mechanic balance is determined by top tier high MMR players.

[AsianJoyKiller]

Devs need to determine the "right" meta and gameplay mechanics.

 

Game "feel" (speed, responsiveness, immersion) low MMR / new player opinion is imperative to gather. 

 

Game play mechanic balance is determined by top tier high MMR players.

If only those things were so easy to seperate. Unfortunately it is not so, and thus, the controversy over what is "right" between higher and lower MMR players.

[WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW]

There's an intangible truth that some people are ignoring. And some people are going to get upset at this and make claims about how it's just "elitism", but it's observable and verifiable.

The lower you go in MMR, the less a player understands about the game and how certain elements affect balance.
The lower you go in MMR, the less a player notices about the naunces of balance.

So while it's the true average MMR might be in the 1500-1600 range, the fact is, a large majority of those players also have a significantly inferior understanding of the game.

I'm not saying that responses from lower MMRs should be ignored, but when analyzing data and what it means, you have to consider the fact that many of people in that range don't fully understand what they are talking about, and the implications that certain balance decisions could have.

 

The phrase "intangible truth" prompts this post.

 

Those two assertions you present are false. While using MMR to correlate game skill is appropriate for matchmaking, it does not reflect game understanding or balance nuances or game elements' interactions with balance. The process of applying theory to execution and extrapolating execution into theory is easily confounded by numerous factors such as equipment, time spent playing, player attributes (e.g. able-bodied, ability to learn, ability to communicate), and so forth. Additionally, different games will lend themselves to different accessibility and potential impact wrt strategy and execution.

 

If players in a low MMR range do not typically fully understand what they are talking about, it should be relatively easy to deconstruct their arguments. I hope no one falls into the dishonest practice of belittling an argument through MMR (which is ultimately irrelevant).

Edited by WWWWWWWWWWWWWWWWWW, 29 March 2015 - 06:14 PM.

[M4st0d0n]

You can't just balance based on how many players the game has. Doing so is completely idiotic and bad game design. That is an extremely great way to kill a game and lose players.
 

I frankly dont understand what you are talking about... Was just pointing out that according less value to one's user feedback because he harvested less MMR is, if not a mistake, at least a gross ad hominem.

[Nightfirebolt]

Yeah. While the amount of work JeffMagnum put into compiling all this data is impressive, I don't believe it accurately reflects the entire playerbase. 42 people isn't the greatest sample size, especially since the game normally has 400+ people in-game at all times.

 

Something that might've helped his poll is if he had left it up for more than three days. I didn't get a chance to weigh in, and I bet a lot of other people didn't, either.

Edited by Nightfirebolt, 29 March 2015 - 07:38 PM.