I appriciate the thoughts but I did all those checks myself and told about them in my first post
Now it is clear that the problem is in the ESC. The motor ran fine with another ESC and my RX8 ESC didn't work fine with other motor either!
I even updated the ESC software with Hotwire but really nothing went better. I still can't change any mode value on the ESC! When I try to change the value with INC button it kicks out of that Mode and doesn't change the value... Seems like a software problem to me!
I will be sending it back to Tekin for warranty repair. And I want to thank Randy for some really fast advices I got from him through e-mail on Friday night