Accurate short-term sea level forecasting is essential for early detection of extreme sea level events such as storm surges in order to ensure public safety and reduce the impact on coastal economies. Sea level is mainly influenced by astronomic and atmospheric factors. We propose HIDRA, a novel residual approach to sea level forecasting -- by estimating and subtracting the astronomic influence from the tidal signal using standard physics-based approaches, we disentangle the two influences and build a network to fully focus on the more complex atmospheric-based part of sea level fluctuations. HIDRA introduces a trainable atmospheric spatial encoder and feature fusion of atmospheric and tidal features into an end-to-end network, which enables discriminative feature construction for the task of sea level prediction. Evaluation on two sea level forecasting tasks (Koper and Acqua Alta) demonstrates a great generalization capability of HIDRA. In comparison with the state-of-the-art numerical NEMO model, HIDRA achieves 38% lower RMSE in general and 41% lower RMSE on storm surge events, while having vastly lower computational complexity -- HIDRA achieves more than half a million times lower CPU times, producing predictions in a fraction of a second and thus significantly reducing the energy footprint of sea level prediction.