Over the last few years, several research works have been performed to monitor fish in the underwater environment aimed for marine research, understanding ocean geography, and primarily for sustainable fisheries. Automating fish identification is very helpful, considering the time and cost of the manual process. However, it can be challenging to differentiate fish from the seabed and fish types from each other due to environmental challenges like low illumination, complex background, high variation in luminosity, free movement of fish, and high diversity of fish species. In this paper, we propose YOLO-Fish, a deep learning based fish detection model. We have proposed two models, YOLO-Fish-1 and YOLO-Fish-2. YOLO-Fish-1 enhances YOLOv3 by fixing the issue of upsampling step sizes of to reduce the misdetection of tiny fish. YOLO-Fish-2 further improves the model by adding Spatial Pyramid Pooling to the first model to add the capability to detect fish appearance in those dynamic environments. To test the models, we introduce two datasets: DeepFish and OzFish. The DeepFish dataset contains around 15k bounding box annotations across 4505 images, where images belong to 20 different fish habitats. The OzFish is another dataset comprised of about 43k bounding box annotations of wide varieties of fish across around 1800 images. YOLO-Fish1 and YOLO-Fish2 achieved average precision of 76.56% and 75.70%, respectively for fish detection in unconstrained real-world marine environments, which is significantly better than YOLOv3. Both of these models are lightweight compared to recent versions of YOLO like YOLOv4, yet the performances are very similar.