I've been thinking about untrustworthy characters. You know the ones: Han Solo, Captain Jack Sparrow, Dustfinger from Inkheart. The guys the hero shouldn't trust, but somehow do anyway.
It occurred to me that one thing these guys all have in common is that they're highly motivated. Take Han Solo in the first Star Wars movie (Or Episode IV if you count the prequel trilogy). If he doesn't get lots of money quick, every bounty hunter in the galaxy will be after him. Sure, he took care of Greedo no problem, but eventually superior numbers will get him. To save his skin, he needs cash. Rescuing princesses pro bono publica goes against his story goal. (So does returning to save the Rebels from the death star instead of rushing back to Tatooine to pay off Jubba. In fact, Han blows his story goal so badly that Empire Strikes Back ends with him in the clutches of a ruthless bounty hunter.)
Captain Jack Sparrow in Curse of the Black Pearl is equally motivated. He wants his ship back, and he'll do whatever it takes to get it. Sure he doesn't want to see Will and Elizabeth hurt -he's not heartless- but that doesn't keep him from using them as bait.
As for Dustfinger, he's pretty driven. Not only does he betray our heroes early on, but when he learns that if he reaches his goal he'll die, the discovery doesn't actually change his mind, it just makes him a bit depressed.
I think what makes these guys come across as unpredictable is that they don't have the same goal as the rest of the heroes. We know they're highly motivated, what we don't know is how far they'll go to get what they want. Will they betray the hero to get what they're after? Maybe they like the hero. Maybe they make an effort to save him -but still betray his cause, then wonder what he's so grumpy about. Or maybe they decide their personal goal isn't so important after all and put all their energy into making sure the heroes win.
If they decide to throw over their own goal out of loyalty to the heroes, it shouldn't be an easy choice. After all, they've wanted whatever they want since before they met the heroes. Letting it go now should be hard -hard enough that your audience doesn't know until the last moment which direction they'll go. Maybe (as in Star Wars) that choice ripples through the rest of the story, causing future problems for the unpredictable character and the heroes. Your heroes might have to decide if they're loyal back -or if their cause is more important. If they decide it is, does their friend resent being hung out to dry? And how does that affect his loyalty? Or the other character's perception of his loyalty? Will they still trust him if they know they turned traitor on him? Should they?
There are lots of possibilities as soon as you have a character with a different goal from your hero. The guy doesn't even have to look shifty. There may be lots of other things that make those wonderful, lovable rogue characters, but different goals is one that's definitely worth playing with.