Darkened Software


Why game development Middleware pitches are wastes of time

by on Jan.31, 2011, under Development, Industry, Lessons, Programming

I am a really big fan of Middleware as there are many things I never want to be an expert on and just want to pay money for them to work.  I really wish that during the process of trying to obtaining said software I did not have to sit through the dreaded “Middleware sales pitch”.

Why do we have to sit through them at all:

Unlike other industries most game middleware companies will not just let you download their SDK, see if it works for you and then let you pay them money if you so desire.  They insist on coming and giving you a presentation before they even allow you have access to a time limited trial version of their SDK.  This makes very little sense, if I am coming to you to download your SDK then it is very likely I do not need to be sold on it.  What I need is time to integrate it and see if it solves my problems, you forcing me to meet with you is actually slowing down your eventual goal of selling me anything.  The only good thing you could do is bring with you so much information about best way to integrate it and other best practices that saves me time in my investigation.  But sadly these sales pitches do not contain this type of information…

Even worse sometimes we are forced to sit through them even when we are not interested in the middleware being presented.  For example someone above you in the org chart decides they like middleware X and unless you want it to be forced on you and your team you need to go to the presentation and ask the right questions to get it shut down.  These are the worst meetings possible as you have to piss off the presenter by asking “bad” questions and also make your boss feel like you are effectively dumping on his idea. 

How can we make them better:

Its unlikely we can make such meetings go away but maybe we can convince middleware providers of what they really should be providing in these meetings to make them a little less painful or more useful. 

A way too standard middleware presentation:

  • Most middleware presentations start of with a great little store of how middleware company x came to being.  Some guy in his mother basement had a mission to solve problem X and after years of blood and tears he has grown solution Y into the coolest company ever. 
  • Then they tell you just enough about how they solve the problem to sound impressive but not really give you enough information to see if it will work for your case.
  • Next they show you a bunch of really bad demo’s because they are not game developers and are too cheap to hire real game developers to build them examples that might prove anything at all.
  • They tell you about a bunch of companies that are using it but have not shipped any games on it yet.
  • They answer any questions about features with the standard “That is in the next version coming out this summer”.
  • They answer any tech questions with “It is easy”, “Send us an email, We will get back to you” or “Can’t answer that as it depends on the level”

But does this do us any good, they are telling a story of thier accomplishments and how wonderful everything is which makes them feel good but does not really help us in our struggle.  They need to remember that in order to get our money they need to tell us a story that makes us believe everything in the furture will much better than it is now.  The only way they are going to do that is by understanding our problems and convincing us they can and are dedicated to helping us solve them.

What are our problems:

  • Time ( the complete lack of it )
    • We don’t have time to write all the software we would like to
    • We don’t have time to find and hire all the people to build all the software we would like to
    • Most of us have very unmoveable milestones that we are under contract to hit.
    • Most of us have teams of people that are waiting for software to be done so they can be productive
    • When we have issues or questions we can not wait on answers we need them a long time ago.
    • Games are highly iterative and anything that slows down the pipeline is scary ( crashing / reliability / slow builds ).
  • Resources( aquire &  managing )
    • People with specific skill sets are hard to find.
    • People in general are hard to manage.
    • Large amount of hardware and software resourcs is a huge maintance cost
  • Money ( the complete lack of it )
    • There is not a lot of profit margin in most developer contracts so the team can not afford to be wrong very much.
    • Developers do not just get 20 million to do a game up front, we get a little bit each milestone which is usually just enough to each the next milestone.
    • Also our requirements change quite a bit over the project so we do not like to spend a lot of money up front as we can not get it back when things change.

So if they were designing a presentation that would tell us how they can solve our problems it seems like they would start the presentation off with a few questions:

  • Is there any experiences in your games you feel need improvments?
  • Is there any parts of your game development process that you would like to improve?

Then instead of giving us a shot gun overview of everything the have ever built they could focus in on solving our problems and concerns.  Now that they have our goals in mind how can they make us believe they can solve our problems.  The last part of the their problem is how to take us down the path between our problem and believing they can solve it in a way that is worth us paying them money. 

Most presentations really fail in this area as they might start off by showing us a very limited demo that only proves that their system will work in a most limited case.  Then they jump back to the very start of the process and show you how to set up the tools, build the assets and get into their simple test level.   The problem is from the very start I am not convinced it will work in my case so I am not going to give you my full attention to your demonstration. 

The improved presentation:

  • Show me that produce X can solve the problem at the scale and complexity I will be faced with.   This might mean going out and getting a game developer to build real world levels for you to demo with and not just showing us your internal test levels.   
  • Next show the real run time costs, be able to bring up memory, cpu stats.  I am not going to listen to the rest of your details if I do not believe that what your showing me could ever co-exist with my game.  Absolutly nothing turns a developer off faster than hearing “I do not know” to our critical runtime questions. 
  • Working backward show us the pipeline that gets the goods into the game.  If we do not know that it will not turn our finely tuned pipeline into the bottle neck of production we are not going to listen to anything else you might say.  Remember Time is our #1 development problem
    • How long does it take to run and how much will it add to our iteration time?
    • Can it be distributed?
    • On average how long does it take to integrate into a tool path?  Provide both examples and references.
  • Next part is the content generation iteself, again this must be shown on a real world examples.  I need to be able to determine total time for my game and how this might change the makeup of my development team.  If I can not figure out how long it would take me to process our assets then I am not going to risk switching my system over to something that is suddenly going to require massive outsourcing or hiring.  Remember Resources  is our #2 development problem
  • Support, turn around time, escalation procedures, avalibility ( I work nights & holiday, are you guys there too? ), ability to help debug, bug patching, source if needed, how often releases, how far back in releases do you support, how quickly do you upgrade to new console libs? 
  • Total Cost and payment schedule, what is the price for everything between now and when we are scheduled to ship.  There is way to much hidden costs in these presentations and we are always hearing things like $40k per platform.  But then 2 weeks later we send an email and get informed that email support is another 10k per year and god forbid if you ever need to get someone out to help you do last min debugging during ship.  Also can we pay over time so that is it does not completely restrict our budget?  Lastly Money is our #3 development problem

 These changes to a more customer concerns focused presentation would move middleware meetings off my list of least favorite development activities and might even make them finally useful.

Leave a Comment :, , more...

Game industry, there are server farms for hire!

by on Nov.22, 2010, under Industry, Programming

Seems in ~30 years we have come full circle on the transition from big university run servers to the desktop computer and back again.  With very accessible systems like Windows Azure and Amazon Elastic as soon as your personal computer or meager company network is not enough for the task you can submitting jobs to the big server farms and go on to other work while you wait for the results.  It is just like the 70’s except much faster, easier and not done on punch cards.

An interesting time in development when you think about it, anyone with a credit card and a little programmer time can now get server farm scale work done almost instantly without the lead time and overhead issues.  Since your renting time from a massive server farms even the size of your problem is really no longer an issue in so as much as your task is able to be subdivided and distributed.

In the game industry we are all tired of completely wasting  > 12 hours on a lighting builds that ends up failing right before a milestone because they are run on hobbled together network of users computers that run so slow because they are also being used to generate content.  Every company wishes they could run a 128 unit server farm like Bungie were artists just submit the job and then go back to doing other stuff.  But not a lot of publishers will cover the overhead for a little developer to build out a server farm for their game.

Well the playing field has just been leveled, get one slightly parallel threading aware programmer to port your lighting and other slow pre-processing systems to Azure and the whole export level process can get done in minutes for just a few dollars.

Think about it the possibilities are endless.  There are so many things I have in the past resisted from putting into the level pipeline because I did not want to make the already insane 12 hour build time even bigger.  But now those concerns effectively go away and one should put the collision optimization, lighting, audio pre-processing, texture optimization, pathfinding generation and everything else we currently know about today in there.   Offload it from your content generators computers and give them their time back to be creating pixels and levels for the game.

I can not wait to see what we will have in games when companies start throwing some real calculations at their levels as part of the iteration process.

Leave a Comment :, , , more...

Book Review: Pragmatic thinking and learning by Andy Hunt

by on Aug.27, 2010, under Programming


This book should be within the top top 3 in your list of books to read this year.  Andy Hunt managed to follow up his amazing first book ” The Pragmatic Programmer” with sequel that might even be more profound.

It starts off with a basic breakdown of how the different parts of the brain work and the interactions between them.  Nothing revolutionary and all things you could find in other books like “How the mind works”.  But then he starts to tie it back to our daily programming tasks and show how each of our different skill sets taxes the different parts and some common things we do that make it harder on ourselves.  A great example was music, most programmer listen to something while coding and debugging.  But voice process is handled by the left side of the brain which needs to be thinking about the code your writing.  Music without words is fine as only the right side of the brain is needed to process it.

He found some research that goes against the theory that your born with the max # of brains cells and you just slowly loose them over time.  For years they thought his because mice in labs never seem to gain new ones.  But then someone finally wondered if environment played a factor and put the mice in a more natural and interesting environment instead of just a cage.  They started growing new brain cells right away and rewiring how their mind worked.  Something to think about at your current job, are you stuck in a cubical cage running the same maze every night and not growing your brain?

He also shows how programmers really need to not neglect the right side of their brain as it is instrumental for tasks like debugging which we think are left brain but really not.  We have all had that moment were on the drive home we think of the solution to a problem that we could not get all day.  Its not that we did not know the solution all along it’s just the right side of our brain was blocked by the left all day and could not give us the answer.   Learning how to get answers from the right side quickly instead of at the pace of 1 per night is very important and he does a great job of explaining how to help this process.

That is just the tip of the iceberg of all the info in “Programmatic thinking and learning”, I could go on for pages about all the things I learned.  Its a great book that ever programmer should read twice.

1 Comment :, , , more...

Interviewing with two game console dev support teams

by on Jul.27, 2010, under Graphics, Programming

Some time ago I interviewed at Sony based on the recommendations of a friend who had joined their R&D department and was really loving his new job.  Their R&D department at the time only need a hard core graphics specialist but they had some other positions that they wanted to interview me for.  So I took a day and went to see what things were all about at Sony.  Most of their jobs were doing some sort of lead tech support for other developers so I expected I would get grilled on:

  • Tech & coding skills
  • Customer support
  • Documentation
  • Time management skills

After an entire day of interviews I had not had any questions on anything that was not tech & coding.  In fact many of the questions were so esoteric that I do not believe they were really looking for a programmer but a PS3 manual with a better personality.

eg. Interviewer:   How many instructions would this take on the cell processor

float temp = (bool) 0;

Me:  Not sure, why would you ever cast a bool to a float?

Interviewer:  You wouldn’t, but if you did how many instructions would it be?

Me:  Not sure, on the PS2 they had an Zero register so it would have been 1 instruction but since your asking I am sure that is no longer the case.

Interviewer:  See he does not know.

This in my mind clears up a lot of stuff, if they do not value customer support, documentation or coding samples to even ask about it in a interview then it is no wonder they are so badly know for the poorest developer support in the industry.

Contrast that to the interviews at Microsoft I recently went through for similar positions.  The first half of the day was all about tech and coding but they asked relevant questions about algorithms, error checking, comments and architecture.  Second part of the day was all about dealing with customers, creating white papers, writing good sample code and other tasks.  Last part of the interview was about what makes great games and user experiences, tech trends and how it will shape the games of the future and last how to communicate and let people know how to take advantage of it.

Perfectly clear why in one generation Microsoft has taken over the console market, they understand what is real dev support is and know that supporting the dev teams has to be the one of the core’s pillars in getting the best games.

Leave a Comment :, , , , more...

Googles new site performance lab is pretty good

by on Dec.28, 2009, under Lessons, Programming

Was going through googles webmasters tools and noticed they have added a new page in their lab section called Site Performance.  Clicking on it I got some bad news about my website.

“On average, pages in your site take 3.7 seconds to load (updated on Oct 29, 2009). This is slower than 60% of sites.”

Wow that is not so good.  So I installed google’s Page Speed addition to the Firebug pluggin for Firefox.  This is a great tool by google as it shows a great ordered list of worst offenders.  Then it goes beyond that and gives you estimates of how much faster things could be if you gzip compressed your code, minimized you DNS look-ups, adjusted your image cache dates and a bunch of other techniques.

My offenders were not quite what I expected.

  • 248k of Java download
  • 7 DNS look-ups
  • 243k of image transfers

Tracking it all down I found out Google Maps was the source of my problems. Kind of ironic; I need new google tools to figure out that google was the screwing me in the first place.

It loads in all of Java and not just a stripped down lib, then it contacts another site to convert your coords to what it needs, contacts itself (google) to look up your map and then send you a image of where you are.

Hmm, those little maps on the bottom of your page are cute but not worth that, gone.

The only thing I wish I could do at the Performance Site is request it to be re-ran now that I am done optimizing.  In general though very cool tools and I hope it gets out of the lab and becomes part of the webmasters tool suite soon.

Leave a Comment :, , , , , more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...