Count your calories.
http://www.fitday.com/ - Can get a fair estimation using that. How tall are you? You may think you don't eat much but some foods can suprise you. Sometimes I have trouble breaking 2k calories in a day and a couple days ago I had a meal that was about 900 by itself.
How tall are you and and how much cardio do you do? Working your butt if is good, but only if you're doing it right. Need high intensity cardio. Otherwise it may be difficult just to burn the calories you eat in a day, let alone burn MORE (which is the only way to lose weight).
Oh.. and I'm trying to gain weight.. so on the off chance theres a way to switch genetics... ahhh... oh well. Sucks, but lots of cardio is the only real way.