The instrument coordinate system's Y axis is whatever is most appropriate to the users of the instrument in question (usually either a slit or the instrument detector's Y axis). The transformation from instrument coordinates to pointing origin coordinates consists of an optional Y flip followed by a rotation (see See f/25, Guider, NIRC and Chopper Orientations , for an example).
Luckily, instrument slits seem usually to be vertical on instrument detector displays, meaning that the instrument coordinate system origin can usually be placed at the bottom left-hand corner of the display (this is the case for LRIS and NIRC, maybe also for HIRES and LWS?), which is nicely consistent with the TV coordinate system.