Although I stress that you shouldn’t focus entirely on your physical being, I also advice against ignoring it. You should keep your body strong. A good way to build a manly physique (and improve testosterone levels) is to lift weights.
You don’t need to become a roided-up bodybuilder (like in the documentary Bigger, Stronger, Faster). In fact, most women would find that unattractive. But it is good to be reasonably strong and to appear “fit”.
How? Don’t make it complicated. Get a weight bench, and a barbell, and some dumbbells. A complete set will be easy to find (and inexpensive) on Craigslist. Learn the basics (from the web or a book, if you don’t have a knowledgable workout partner). Work out on a regular basis.
Even if you don’t work out intensely and expertly, but just lift what you can as part of your normal routine, you will benefit. The important thing is that you do it with regularity.
Lift weights. It will help you be a better man.