Difficult.

He was difficult to be around but I enjoyed spending time with him. It meant I was not the most offensive person in the room.