Castle Mamba MAX 5700 1/10 Motor prombles?

[RCHabit]

Hello,

Has any body had any problems with the mamaba max 1/10 system with the 5700 motor? I have

I have a stampede
I installed the mamba max system,

1st battery used a 8.4 batt in it.. Worked great

2nd battery I used a 9.6v batt.. worked great

3rd battery I used was 11.1 5kmah max amp lipo.. worked great for about 10min and then locked the brushless up..?

Sent it back to castle and they gave me a new one, @ N\C (great service too)

I installed the new one, I used the pattern again with the battery's 8.4 then 9.6 then 11.1, when i used the 11.1 for about 15minutes the truck stopped.. The motor is getting power but does not have enough to move, it just bucks? The new brushlees motor is only about 1 hour old?

Do I have to get a fan for the motor or heat sink? I'm I doing something wrong? Will using the brake to much heat up the brushless?
I'm new to brushless

Thanks
Any help or suggestions is appreciated

Has any body else had the same problem?