Darkened Software

Was going through googles webmasters tools and noticed they have added a new page in their lab section called Site Performance.  Clicking on it I got some bad news about my website.

“On average, pages in your site take 3.7 seconds to load (updated on Oct 29, 2009). This is slower than 60% of sites.”

Wow that is not so good.  So I installed google’s Page Speed addition to the Firebug pluggin for Firefox.  This is a great tool by google as it shows a great ordered list of worst offenders.  Then it goes beyond that and gives you estimates of how much faster things could be if you gzip compressed your code, minimized you DNS look-ups, adjusted your image cache dates and a bunch of other techniques.

My offenders were not quite what I expected.

• 248k of Java download
• 7 DNS look-ups
• 243k of image transfers

Tracking it all down I found out Google Maps was the source of my problems. Kind of ironic; I need new google tools to figure out that google was the screwing me in the first place.

It loads in all of Java and not just a stripped down lib, then it contacts another site to convert your coords to what it needs, contacts itself (google) to look up your map and then send you a image of where you are.

Hmm, those little maps on the bottom of your page are cute but not worth that, gone.

The only thing I wish I could do at the Performance Site is request it to be re-ran now that I am done optimizing.  In general though very cool tools and I hope it gets out of the lab and becomes part of the webmasters tool suite soon.

Seems like the average console game company’s gameplay testing process starts off with an email like this…

We’re having a playtest this Saturday afternoon, from 1:00 – 4:00. If any of you have gamer friends that would like to participate, have them fill out this NDA + survey and send it back to me.

People showing up are sat down in from of a dev box with the latest copy of the game and promised free pizza if they finish the game and fill out all the questionnaire.  Sometimes if the company is really on it, the testers play is recorded so it can be double checked and reviewed later by all the developers.  This testing is usually < 100 players and is typically only done only 3-4 times during the project at milestones like first playable, alpha, beta.   From this very small sampling of data feedback is given to the development team and at many publishers it is even used to help estimate pre-order numbers and determine a games slice of the marketing budget.

Its not that they do not get any useful data out of this at all, but nothing pulled out of this could be called qualitative and if someone tried to pass this off for testing in any other industry, they would likely be fired.

It is not the developers or publishers fault that there is little to no real focus testing, they do not really have a choice.  Console manufacturers only allow pre-release game disks to only run on special $10k development boxes that the public is not allowed to have.  So even though there are millions of xBox360′s out there, your testing pool is still only as many development boxes your studio plus publisher can allocate to it.

Even if your publisher has a ton of money to put in hardware, it is such a issue to coordinate getting that many people on site it can still only be done once in awhile.  The results of the game not being tested that often is so many changes are getting tested all at once that it is hard to determine or isolate what change is effecting the feedback.  There is also problems that we can not verify what live testers say about their playing habits or any other information they give us.

What has this done is placed undue burden on the game designers as they have to be able to come up with new and exciting game designs and get it exactly right as they will get very little chances for any real significant unbiased feedback before they ship.  With the punishment for unpopular levels being a pink slip is it any wonder that designers tend to go with tried and true designs that they are sure will work.  They might be more willing to take risks and move games foward if they had a quick feedback loop to tell them if they are heading in the right direction.

We need the kind of testing online PC games can pull off:

• Easy distribution of game .
• Connection to running games so info can be put in database.
• Stats on user to ensure they fit our testing needs.
• Online way to fill out survey and give feedback

We could have this kind of testing because the console manufacturers already have everything we need, we just do not have access to it.  They just need to let the developers get access to the play stats on xbox Live so we can pick the right testers for our games.  Then they need to allow us to release xBox arcade type pre-releases of the games so users can play test them for us.

It could work like this:

• You create you’re xBox Live account and you select if you would be interested in getting access to games early for gameplay testing.  It warns them that their stats could then be searched by developers.
• If you say yes you get put in a database and your playing stats of games is recorded so developers can decide if your good for the type of testing they need.
• Developer can then send you messages that you have been approved for testing and requesting you download their game.
• You Sign an online NDA and then start the game.
• Once they quit the game it takes them to a survey page and they give feedback.

Suddenly with one database programmer and a community manager you could have access to and do coordinated testing with millions of users.

• With the xbox Arcade type download system tests could get out to testers very quickly.
• With access to that many testers and stats on them we could do things like control groups and very fine grain testing we have not been able to do before.
• Given its all online your results are instant and developers can put in code to record all kinds of details you do not get from surveys or just video recorded sessions.  You could even save off a full replay of the session to get reviewed and analyzed later.

Potential issues:

• Restrictions to ensure it is not abused for marketing purposes.
• What to do during console transitions and your making a launch title as there are no consoles out yet.  Maybe that just becomes another hardship of being a launch title.
• Developers will not longer be able to control screenshots and movies during the testing phase.  They can make them sign NDA’s but given that is it not done at their building things will end up online.

The potential gains in game quality and developer cost savings are huge as currently we often do not get feedback until it is way to late or too expensive to do anything about it.  I do not know if it would be as easy to implement as I think, but it can not hurt if everyone mentions it to their account rep.  Maybe someone at Microsoft or Sony will look into it.

At least once every year or two I think it is important to take an interview even though you might not looking be looking at moving.  It ensure your interviewing skills are fresh and that you have not lost touch with reality and became irrelevant outside the current company.

If you can not get at least one offer letter a year for more than you are currently making then it means either your vastly over payed or more than likely your skills have stagnated and you need to kick it up a notch.  Even beyond just keeping your public speaking skills in shape there are other reasons to stay in the interview process.

If you can not get at least one offer letter a year for more than you are currently making then it means either you’re vastly over paid, or more than likely your skills have stagnated and you need to kick it up a notch.

Even beyond just keeping your public speaking skills in shape there are other reasons to stay in the interview process.

Reasons to get interviewed:

1. Its good to see how a lot of other places work, so it is often an great learning experience if you do not mind asking a lot of questions.
2. Its good to see what techniques others are using in the interview process to verify peoples skills and see if personalities will fit in.
3. Some times they will ask really hard questions that will make you think, it is a chance to reflect and improve yourself.
4. There is a lot of feedback in the interview process

• If they phone interview you, it means you resume is still relevant.
• If they bring you in for an interview that means you sound intelligent on the phone.
• If they do not walk you to the door right away that means the lowest people at the company think you have skills.
• If you make it to the end of the interview that means even people outside your little world think you can communicate and you have skills the company needs more of.
• If they send you a offer letter that means you really do still have a lot of relevant skills and your personality does not aggravate people in < 1 hour.

Reasons to interview others:

1. Again hear how other places work and borrow good idea’s
2. Meet interesting people, I have kept in contact with people I have not hired just because they were doing interesting things that made me think.
3. See the different mind games that go on in interviews, they present themselves in the best light, and you have to figure out what they might be covering up.  You basically have < 1 hour to figure out what makes them tick, verify their skills and sell them that this is the best place to work.  It is a hard job that only gets easier with practice.
4. Being on the other interviewing side of the process will give you a better feeling for what is going on the next time you have to be the interviewee.

So take a day out of your year to get  interviewed somewhere, it is better to keep these skills sharp while you still have a job vs getting laid off and then blowing the first X number of interviews because you are out of practice.

Reject conventional interviewing wisdom:

One thing I have messed up in the past and I  think many people still mess up today is they believe interviewing with more companies always increases their chances of getting a job.  That is true in a low skill environment when their needs can be meet with the next person that walks through that door and can hold a shovel.  There is no reason for them to waste time interviewing many to find the best person that can hold a shovel; anyone will do.  In that situation only by being in a lot of places will you have a good chance of being the next resume/person they see after they decide they need someone.

But in a skilled industry they do not want the next person, they want the best person for their money and only by being their #1 pick will you end up with a job.  You need to convince 8 – 10 people that you can not only do the job but you will take work off their plates, not be a drag on the team, can grow with the company, will not leave before the game ships and will be useful on the next project.  Plus if your not the last person they interview your impression has to be so good it survive more recent memories about others they have talked to.

In this environment it does you ABSOLUTELY  ZERO good to interview at a lot of places and always be their # 2 or #3 pick.  You are still just as unemployed as the next person that got drunk and did not even show up to the interview…

So do not waste your time mass spamming companies and then coming to these interviews without a full plan to make them believe you are perfect for that job.  If you have no idea what the company really needs given their history and specific details on how you would fill it; you are unlikely to ever be their top pick for the current job.  If you have not through about what they would likely need in the future given where they are given their past course then it would be harder to convince them of your future worth to the company.

Seriously your company specific cover letter and resume alone should have all ready have convinced them you can do this job.  The interview itself should just be for them to convince themselves that their first impression was right, you play well with their team and you should be their #1 candidate.  If your spending your time in the interview figuring out what they do and what they need.  You are now fighting an uphill battle and by the time you figure it out you will not have enough time to convince them of much.  Spend more time researching each spot you apply to and less time doing mass interviews, you will find that you get better offers a lot faster.