Yeah, Sirius is wary of adopting Harry. For one thing, he doesn't want to be seen as trying to "replace" James as his father. He also, I think, doesn't want Remus to feel left out because clearly only one of them even CAN adopt Harry.

There's also the fact that if he adopts Harry, his entailed inheritance will pass on to Harry instead of Draco, as it would have. And Harry would also carry certain titles, as the Heir of the House of Black, that Sirius doesn't really care to pass on. :)