denso contacts

It could also be the starter. It's hard to tell. There are a pair of copper contacts in the starter that wear out after a while resulting in intermittent contacts. The repair is rather inexpensive but requires removing and disassembling the starter. Check out this site for more info. http://nationsautoelectric.com/densoparts.html

Whoever does the job must hold the contacts in place with a C-clamp and a piece of wood or using a bench press, so that the contacts don't move while the screws are being fastened. Otherwise, it may fail again after a short while.
http://www.toyotanation.com/forum/showthread.php?t=222481

This sounds like the very common Denso starter solenoid contacts problem. Do a search for "starter contacts" on this forum and read up on the problem. This has been discussed repeatedly on this forum. When I had this problem, I decided to remove the starter myself and have the whole unit professionally rebuilt by an auto electrical rebuilder in my area. That rebuilder came highly recommended by a professional tech. I felt much better about having him rebuild my OEM starter, than picking up a rebuilt unit in an auto parts store, or just doing the contacts myself. I intended to keep this car for a long time.

my car is presently doing the same thing, its the starter solenoid, unfortunately its hard to pick up just the solenoid from an autoparts store, I've called everywhere. Its unessary to replace the whole starter, you just need to replace the solenoid. (its attached to the starter, 2 bolts disconnect it from the motor.) Rock auto has just the solenoid for 100$, a starter is about 175-200$ (auto parts store price)