We study a directed polymer model in a random environment on infinite binary trees. The model is characterized by a phase transition depending on the inverse temperature. We concentrate on the asymptotics of the partition function in the near-critical regime, where the inverse temperature is a small perturbation away from the critical one with the perturbation converging to zero as the system size grows large. Depending on the speed of convergence we observe very different asymptotic behavior. If the perturbation is small then we are inside the critical window and observe the same decay of the partition function as at the critical temperature. If the perturbation is slightly larger the near critical scaling leads to a new range of asymptotic behaviors, which at the extremes match up with the already known rates for the sub- and super-critical regimes. We use our results to identify the size of the fluctuations of the typical energies under the critical Gibbs measure.