Six niobium rich glasses presenting different sodium contents were synthetized. On the basis of complementary infrared and Raman analysis, the influence of the sodium content and its role on the structure of the glasses prior to and following poling was examined. Correlative second harmonic generation (SHG)/Raman microscopy on the poled glasses cross section has shown a co-localization of the SHG signal and the structural variations. It also evidenced similar structural rearrangements whether sodium is removed through poling (230–300 °C) or through the alkali content defined by the starting glass compositions (melted at 1300–1500 °C). The effect of the sodium content on the optical properties, prior to and after thermal poling, is also discussed. It was found that refractive index variations induced by poling (ranging between 10−3 and 3 × 10−2) are mainly the result of a density decrease in the poled region rather than compositional and structural changes. The electro-optic origin of the poling-induced second order nonlinear response is confirmed by the Maker Fringe SHG analysis, and the evolution of χ(2) (2–2.5 pm/V) with the sodium content is discussed based on a selection of parameters influencing either the third order susceptibility [χ(3)] or the internal electric field strength of the poled layer.