David Bain had the following possible motive to compel him to commit the murders. 

A combination of perceived betrayal (from his parents) and jealousy (of his siblings), and added to that a sense of loss of control.

Betrayal

There are indications that David had an argument with both of his parents about him moving out of the family home to make his own way in the world. He related to his then girlfriend that moving out of home was "out of the question".  David could have interpreted this desire to throw him out of the family home as betrayal.  In addition, there are indications that his mother, Margaret was shifting her attentions increasingly from David towards Stephen and that she was admonishing him for his treatment of his siblings.

Jealousy

David had reason to be jealous of the attention that his parents increasingly gave to his siblings, especially Arawa, who had recently been Head of School, and Stephen, who was now turning into a young adult and starting to achieve.  In this respect the betrayal and jealousy are intertwined.  David had failed his first year at University and had spent a year on the dole and so had not much to show for himself, whereas some of his siblings were high achievers.  The jealousy he felt could have been considerable.

Control

David was controlling and manipulative.  As his siblings grew older he felt that he was losing control over them.  When his parents were in conflict with each other there was a power vacuum in the family that he could fill.  Once it became apparent that they were healing their differences and going ahead with plans to build the new house, then his position of control weakened.  Laniet had already left home and he daren't think what might happen to his position in the family once all of them had gone.

Evidence

This motive is articulated by David himself in the booklet published by Joe Karam in 2001 called Innocent: Seven Critical Flaws in the Wrongful Conviction of David Bain.  The following is a quote from David:

"I find this state of limbo depressing and often hard to live with [referring to being in prison], but it is eminently easier than being constantly crushed by shattered dreams, destroyed plans, broken promises and betrayals, by all I once held dear."

Interpretation

The first part of the sentence clearly refers to his imprisonment (this being written in 2001 when he was still incarcerated), however the second part clearly refers to when he was not in prison.  It most likely refers to the environment he felt that he was in when he was with his family.  We know that they were a close knit family and therefore "by all I once held dear" must refer to his family.  Who else would this refer to?  Tellingly, he says "once held dear" which infers that he no longer holds them with any affection.  This is supported by observations of lack of emotion following the murders and the lack of any reports of David expressing any feelings of loss from having all his biological family killed or anger at his father for supposedly doing it.  The complete passage in the Innocent booklet from which the quote is drawn is also absent of any reference to his family or any feelings of loss or anger in respect of what happened.

Further to the above, he says that his current "state of limbo" is "eminently easier than" the previous [family] environment, which infers that he does not regret the loss of his family.  Being sent to prison merely frustrated his plans.

Pyschologists would refer to the quoted passage as a Freudian Slip.