The existing, sophisticated AR gadgets in the market today are mostly exorbitantly priced. This limits their usage for the upcoming academic research institutes and also their reach to the mass market in general. Among the most popular and frugal head mounts, Google Cardboard (GC) and Wearality are video-see-through devices that can provide immersible AR and VR experiences with a smartphone. Stereo-rendering of camera feed and overlaid information on smartphone helps us experience AR with GC. These frugal devices have limited user-input capability, allowing user interactions with GC such as head tilting, magnetic trigger and conductive lever. Our paper proposes a reliable and intuitive gesture based interaction technique for these frugal devices. The hand gesture recognition employs the Gaussian Mixture Models (GMM)based on human skin pixels and tracks segmented foreground using optical flow to detect hand swipe direction for triggering a relevant event. Real-time performance is achieved by implementing the hand gesture recognition module on a smartphone and thus reducing the latency. We augment real-time hand gestures as new GC’s interface with its evaluation done in terms of subjective metrics and with the available user interactions in GC.