We perform a combined potential vorticity analysis and energy analysis to analyze instabilities and mesoscale motion in a continuously stratified, terrain-following coordinate ocean model. Care is exercised to ensure that the potential vorticity is conserved along isopycnals in the absence of sources and sinks. We also derive expressions for the energy components and the associated conversion terms suitable for a terrain-following coordinate ocean model. To demonstrate the usefulness of the new combined analysis tool for detection of instabilities, we analyze results from a numerical model experiment in which the ocean model is contained in a long, rectangular channel closed at one end. The model is forced by a uniform along-channel wind towards the closed end. We analyze the model response in two different sub-domains. The first is far from both the open boundary and the bight and resembles an open channel with upwelling at one side and downwelling at the other. The second is near the closed end, and represents a dynamically new, and as it turns out, different area. The analysis for the upwelling side of the open channel domain gives results that compare favorably with previous studies and shows that an upwelling front is indeed unstable. The analysis of the results from the downwelling side corroborates earlier suggestions that a downwelling front is stable. The analysis of the bight area reveals that it is more or less similar to the downwelling area of the open channel domain due to propagation of Kelvin waves from the downwelling boundary to the upwelling boundary via the closed end. In summary the application of the combined analysis to the results from the numerical experiment provides an excellent example of how well a potential vorticity and energy analysis complement each other.