As the energy consumed in manufacturing and operating information and computing technologies (ICT) accounts for an increasing percentage of worldwide carbon emissions, there is a growing urgency to address the carbon footprint generated by the computing infrastructure. Although the world has witnessed a drastic increase in computing performance for large neural network models with the use of more transistors and miniaturized dimensions, the advanced fabrication of these high-end integrated circuits incurs more energy and environmental costs than use-phase costs, making it a critical sustainability challenge. Emerging technology such as silicon photonics is being proposed for both communication and computation in neural network accelerators due to better performance-per-watt and higher bandwidth density compared to traditional electronics. In this paper, we conduct a comprehensive carbon emission analysis on hardware accelerators for neural network applications considering both traditional electrical and emerging silicon photonics technology. We propose a carbon-emission-based sustainability model that considers photonic devices and interconnects. The model accounts for both the carbon emissions due to manufacturing as well as the lifetime operation of the devices. We compare three designs - electrical, photonics, and hybrid architectures - that explore the trade-offs between computational performance and environmental costs for different neural network models. With our proposed sustainability-aware design, we demonstrate that utilizing silicon photonics improves computational performance along with superior environmental sustainability.