What type of off-roading are you looking for? More expedition (camping, dirt roads, maybe some light trails with regular highway miles)? Trail riding? Daily driver, or will you still drive a car to/from work? There are different needs for each area, and the recommendations would differ (ie. 2.5L is okay for trail duty but if you were hauling a weeks worth of camping gear, 4L all the way). What you consider "crazy mods" or "just tires" might not match up with what others here have in mind.
The best piece of advice I can give is put as much as you can into the initial purchase of a good base. If the frame/body is shot, it takes a lot of work to clean up. Then get out, drive the wee out of it, and find out what works, and what doesn't, and move on from there. Chris/2Greys is a good example of that, and the well thought out direction he's taken is a result of seeing what worked and what didn't for him.
Good luck!