Profitable custom instructions provide higher performance for a given reconfigurable area. Hence, choosing profitable custom instructions that are also area-time efficient is essential if design constraints must be met by field-programmable-gate-array (FPGA)-based reconfigurable processors. In this paper, we propose a framework for FPGA-based reconfigurable processors in order to rapidly identify a reduced set of profitable custom instructions without the need for actual hardware synthesis. The proposed framework is capable of estimating the area utilization and latencies of custom instructions on lookup-table-based commercial FPGAs. Simulations based on 15 applications from benchmark suites show that the proposed method provides, on average, an area reduction of over 29% for loss of mere 1.3% in compute performance. Our evaluations also confirm that the proposed framework is superior to an existing area-optimization approach that relies on exploiting the regularity of custom instruction data paths. In particular, an average area-time product gain of over 59% was achieved by deploying a reduced set of custom instructions obtained using the proposed framework.