It is suggested that wearing a prescription goggles for swimming. As long as you are handling your contact lens hygiene and storage with proper care before and after swimming, you may still wear your contact lens in day time, otherwise glasses is suggested for day time wearing.
To avoid infection, It is highly suggested to rinse your eyes thoroughly with saline solution before putting back on your contact lens after swimming.