Ferroelectrics (FE) have been the materials of interest for non-volatile memory applications for many decades due to their hysteretic charge-voltage characteristics. However, recently, the possibilities of integrating an FE in the gate stack of a transistor (forming a ferroelectric transistor or FEFET) have opened new avenues for harnessing the distinct features of the FE in computation more effectively. The FEFETs not only enhance the design of non-volatile memories, but also lead to the unique possibilities of introducing non-volatility in close proximity with the compute elements. In this paper, we comparatively analyze several device and circuit design aspects of FEFETs and FE capacitors from the perspective of designing non-volatile memory and logic. We discuss the effect of integrating an FE in a transistor structure on the remnant polarization and coercive voltage and show the importance of FE thickness optimization to design a non-volatile transistor. We also present the circuit design possibilities with non-volatile FEFETs. First, the design of memories with separate read-write paths is discussed. Compared to FE capacitor based memories, FEFETs achieve enormous distinguishability and near read disturb free operation albeit with 2.5X higher cell area and 3.6X higher write energy at iso-write time. Second, we describe the opportunities that non-volatility combined with the three terminal architecture of FEFETs presents in the design of low power non-volatile flip-flops. Compared to FE capacitor based flip-flops, FEFET based design yields upto 50% lower energy and up to 40% lower delay for data back-up, along with 30% lower area.