Forests play a crucial role in the global carbon (C) cycle by storing and sequestering a substantial amount of C in the terrestrial biosphere. Due to temporal dynamics in climate and vegetation activity, there are significant regional variations in carbon dioxide (CO2) fluxes between the biosphere and atmosphere in forests that are affecting the global C cycle. Current forest CO2 flux dynamics are controlled by instantaneous climate, soil, and vegetation conditions, which carry legacy effects from disturbances and extreme climate events. Our level of understanding from the legacies of these processes on net CO2 fluxes is still limited due to their complexities and their long-term effects. Here, we combined remote sensing, climate, and eddy-covariance flux data to study net ecosystem CO2 exchange (NEE) at 185 forest sites globally. Instead of commonly used non-dynamic statistical methods, we employed a type of recurrent neural network (RNN), called Long Short-Term Memory network (LSTM) that captures information from the vegetation and climate's temporal dynamics. The resulting data-driven model integrates interannual and seasonal variations of climate and vegetation by using Landsat and climate data at each site. The presented LSTM algorithm was able to effectively describe the overall seasonal variability (Nash-Sutcliffe efficiency, NSE = 0.66) and across-site (NSE = 0.42) variations in NEE, while it had less success in predicting specific seasonal and interannual anomalies (NSE = 0.07). This analysis demonstrated that an LSTM approach with embedded climate and vegetation memory effects outperformed a non-dynamic statistical model (i.e. Random Forest) for estimating NEE. Additionally, it is shown that the vegetation mean seasonal cycle embeds most of the information content to realistically explain the spatial and seasonal variations in NEE. These findings show the relevance of capturing memory effects from both climate and vegetation in quantifying spatio-temporal variations in forest NEE.