Publications

2026

1. 

   Sketch3R: Rapid and Realistic 3D VR Sketch Creation to Shape Retrieval

   Mritunjoy Halder, Shivam Ashok Shukla, Lokender Tiwari, and 2 more authors

   In , 2026

   Abs DOI

   Large 3D shape repositories are rapidly expanding, driven by advances in generative modeling, making efficient shape retrieval increasingly important for authoring tools. While text queries capture high-level semantics, they often fail to convey precise geometric details. 3D sketches provide a more expressive means of representing shape geometry, and recent AR/VR developments have made sketch-based retrieval practical. However, existing 3D sketch datasets face three major limitations: (1) reliance on quad meshes or voxel hulls, which often fail on complex or non-manifold shapes; (2) use of fixed-size point clouds that discard stroke connectivity and limit geometric fidelity; and (3) dependence on expensive curve-based or multi-view rendering pipelines, which hinder large-scale data generation. Limited point cloud representations also fail to capture sketch connectivity and topology when used to train retrieval models. To address these challenges, we propose Sketch3R, a scalable framework that converts arbitrary 3D meshes into human-like VR sketches using a graph-based representation that preserves stroke connectivity and adapts to sketch complexity. Leveraging this representation, Sketch3R employs a lightweight graph-attention Siamese network for efficient and accurate sketch-to-shape retrieval. Experiments demonstrate that our method outperforms prior approaches in both accuracy and speed, while robustly handling 3D shapes across diverse topologies.

2. 

   UltraNormal: Scaling Diffusion-based Surface Normal Estimators to Ultra-High-Resolution

   Satyam Bhardwaj, Lokender Tiwari, Shivam Ashok Shukla, and 3 more authors

   Computer Graphics Forum , 2026

   Abs DOI

   Most of the current diffusion-based surface normal estimators were trained at low resolutions (e.g., mathematical equation), significantly below the resolution of modern cameras. Applying such models to high-resolution images causes two issues: (1) downsampling removes high-frequency geometric details, and (2) direct high-resolution inference loses global context. To address these challenges, we propose UltraNormal, which augments existing pretrained diffusion-based surface normal estimators by using overlapping sliding patches with test-time guidance to inject global context. UltraNormal initializes each patch with the latent of the corresponding low-resolution estimate enhanced with high-frequency details from the local estimate and resumes the reverse diffusion process from an intermediate time step. This approach maintains geometric accuracy of the global prediction while adding high-frequency details without retraining the diffusion model. To the best of our knowledge, this is the first test-time method to scale low-resolution diffusion-based surface normal estimators to 4K resolution without any retraining. We build UltraNormal upon three different base models and evaluate on both real-world data and a 4K photorealistic synthetic dataset. We perform extensive ablations and demonstrate improved accuracy, sharpness, and surface reconstruction quality compared to the base model.

2025

1. 

   GeMR: Multi-modal Interactive 3D Scene Composition in XR

   Raghav Mittal, Lokender Tiwari, Shivam Ashok Shukla, and 2 more authors

   In , 2025

   Awarded Abs DOI

   Best Oral Paper Award

   This paper introduces a novel mixed-reality (MR) 3D authoring platform designed to enable real-time, interactive composition of 3D scenes in augmented and mixed reality environments. The platform proposed and integrates two key innovations: a three-tier 3D asset generation workflow and a 3D sketch-to-shape retrieval algorithm. The three-tier architecture addresses the challenge of time-consuming 3D asset creation by allowing users to engage with the generation process at intermediate stages, enabling rapid validation and refinement of user inputs. This approach not only enhances interactivity but also accelerates the realization of user imagined assets. Complementing this is our 3D sketch-to-shape retrieval system, which leverages an intuitive, robust sketching module to query a library of generated or artist-created 3D assets. This mechanism facilitates precise retrieval of objects matching both the category and rough shape of a user drawn 3D sketch of the assets. The platform supports multi-modal, intuitive user inputs such as speech, text, and freehand 3D sketches, making it accessible to both novice and expert users for creating, retrieving, modifying, and composing 3D assets and textures. By combining generative models with intuitive interaction, the platform serves as a flexible testbed for 3D asset development and a tool to democratize 3D scene authoring. We show effectiveness of GeMR on various use-cases, including scenario composition for human-robot collaboration. We conducted a user study to evaluate whether our system effectively democratizes the 3D scene composition. We compared the performance of 3D modeling experts and novice users on a standardized scene creation task. Both novice and expert users reported equally good usability ratings and low perceived workload. Our findings highlight that an integration of generative models and multimodal intuitive inputs represents a significant leap in making the 3D scene authoring more accessible.

2. 

   FAV3R: Fast and Accurate 3D VR Sketch to 3D Shape Retrieval

   Mritunjoy Halder, Shivam Ashok Shukla, Lokender Tiwari, and 2 more authors

   In , 2025

   Abs DOI

   With the increasing demand for 3D content in AR/VR applications, 3D shape libraries are expanding rapidly. As a result, the ability to retrieve 3D shapes from these libraries has become an essential component. It is well established that text-based queries are insufficient for accurately describing geometric shapes, and 2D sketches lack the necessary detail. In contrast, 3D sketches are more effective at conveying geometric properties. In this paper, we introduce a new learning-based approach for retrieving 3D shapes from 3D VR sketches. Since there is a lack of large-scale datasets containing paired 3D VR sketches and 3D shapes, our second contribution is a novel data generation technique that automatically converts 3D shapes into human-like 3D VR sketches. We conduct a comprehensive evaluation of our data generation and retrieval methods against state-of-the-art approaches, demonstrating that our method is both efficient and achieves superior retrieval accuracy.

3. 

   InGenCo : Integrated In-Place 3D Scenario Generation and Collaboration

   Raghav Mittal, Lokender Tiwari, Satyam Bhardwaj, and 2 more authors

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2025

   Abs DOI

   Human-robot collaboration is essential in AI-powered Industry 5.0 and personal spaces, aiming to boost human creativity and efficiency. Traditional methods involve creating virtual scenarios in VR to simulate interactions, but they often rely on pre-defined 3D assets or procedural generation, which are time-consuming, expensive, and limited by restricted design options and the need for 3D expertise. Many systems are also desktop-based and lack intuitive interfaces, limiting interactive scenario creation and in-place testing. This paper presents an integrated XR-based system that combines GenAI based 3D scenario generation and human-robot collaboration in real-time. It allows users to create and modify virtual environments on-the-fly using voice and gesture inputs, and test collaborations directly in mixed reality, accelerating decision-making. The system, powered by Generative AI and XR, enables even non-experts to generate 3D assets, assemble them, configure robots, and perform collaborative tasks. It features an intuitive interface and multiple configuration options. Through experiments and ablations, the system’s effectiveness is validated, showing its potential for interactive 3D scenario creation and in-place collaboration in XR, with applications in robotics, gaming and other fields.

2024

1. 

   A transmission model based deep neural network for image dehazing

   Tannistha Pal, Mritunjoy Halder, and Sattwik Barua

   Multimedia Tools and Applications, 2024

   Abs DOI

   In recent years, one of the main contributors to traffic fatalities and poor vision in poor weather conditions has been rising. In order to lower accident rates, improving visibility is essential for driver assistance systems, image acquisition, and surveillance systems. Thus, as a result, this research introduces a novel Deep Neural Network (DNN) based on a scattering model for defogging an image that has been affected with fog. The proposed model consists of two components: the fog dilution model and the diluted fog removal model. In addition, we have incorporated a deep learning-based transmission estimation module. The fog dilution model is designed to mitigate the presence of fog, while the diluted fog removal model aims to completely eliminate the remaining fog. To enhance the accuracy of transmission estimation, we employ a green channel prior-based approach. This approach effectively reduces the distortion caused in the sky in the resulting defogged image. By combining these components, our model offers a comprehensive solution for fog removal, with improved image quality and reduced sky artifacts. In addition to addressing the shortcomings of current vision enhancement techniques as mentioned in the paper, the proposed method also satisfies human perceptual needs. Experimental results have demonstrated that the proposed model delivers superior visibility enhancement results for both non-reference and full-reference metrics following a qualitative comparison with eight state-of-the-art defogging techniques. The experimental results have proven that the proposed method achieves superior performance in terms of qualitative evaluation on non-reference metric (i.e., in terms of e = 0.50, = 0.15, r = 1.49) and reference metric (i.e. in terms of MSE= 397.16, PSNR = 23.38, NCC = 0.99, MD = 24.39, NAE = 0.13) compared with eight state-of-the-art dehazing methods. Furthermore, based on the average computational time achieved by the proposed method (0.17 s using HSTS dataset), it can be highly suitable for real-time applications. The suggested approach could therefore be employed as a viable route for improving vision in surveillance and acquisition systems while lowering the risk to users’ safety.

2. 

   Anomalous activity detection for mobile surveillance robots

   Snehasis Banerjee, Balamuralidhar Purushothaman, and Mritunjoy Halder

   May 2024

   US Patent App. 18/473,595

   Abs

   Technical challenge in unusual human activity detection task is to rightly identify only unexpected or unusual movements from constant regular movements present in a scene, with most techniques built on understanding that camera is static. However, ego view camera of mobile surveillance robot is in motion as robot navigates. Embodiments herein provide a method and system for anomalous activity detection for mobile surveillance robots by mimicking ‘Konio-Parvocellular-Magno’ cells of the human brain into a NN model, which are responsible for detecting slow, normal, and swift changes in perceived scenes. To detect anomalous activity, the static or normal movements of scene captured by ego view camera are identified as redundant information and only RoI is forwarded for further processing using the Optical flow and SSIM techniques. The NN model mimicking KPM is trained only on the RoI to detect normal or anomalous activity.

2023

1. 

   Anomalous activity detection from ego view camera of surveillance robots

   Mritunjoy Halder, Snehasis Banerjee, and Balamuralidhar Purushothaman

   In 2023 International Joint Conference on Neural Networks (IJCNN), May 2023

   Abs DOI

   Can a surveillance robot autonomously detect anomalous activity from its ego view camera perception? This is a challenging task as it requires identifying what is normal and what is an abnormal pattern - given the variations of possible anomalies and abnormalities. This paper presents an architecture and method based on a spatio-temporal convolution neural network to detect and classify anomalies. This work is inspired by the ‘Konio-Magno-Parvocellular’ cells of the human brain, which is claimed to aid humans in organizing changes in perceived scenes. The model is trained and tested on a benchmark video dataset of human activity. We have obtained 91% testing accuracy on this dataset. Experiments in simulation as well as deployment on a real robot shows that the proposed methodology can identify anomalous activities effectively. We have also listed down the observations from practical deployment of the model.

2. Multi-feature based hazy image classification for vision enhancement

   Tannistha Pal, Mritunjoy Halder, and Sattwik Barua

   Procedia Computer Science, May 2023

   Abs DOI

   Deteriorating contrast, low glow, restricted dynamic range, poor resolution details, non-bright natural landscape colours, and reduced saturation of an image are subjected to various degrees of influence and deterioration in hazy weather conditions. Dehazing haze degraded images becomes challenging if they are not classified as hazy or clear, given that image dehazing techniques can only be used with hazy images. The competence to differentiate between hazy and clear images can not be left to human perception; hence a robust model is needed that classifies the input image into hazy and clear. Thus, we propose a nine unique features-based image classification framework based on K-Nearest Neighbour (KNN), which can accurately classify hazy and clear images. Experimental results demonstrate that the proposed method can efficiently classify the hazy and clear images, with an accuracy of 92%, a precision of 0.90, recall of 0.96, and F1 score of 0.93 for a benchmark dataset, which has both theoretical and practical implications.

3. A deep learning model to detect foggy images for vision enhancement

   Tannistha Pal, Mritunjoy Halder, and Sattwik Barua

   The Imaging Science Journal, May 2023

   Abs DOI

   Fog limits meteorological visibility, posing a significant danger to road safety. Poor visibility is considered as a significant contributor to road accidents in foggy weather conditions. Regardless, image defogging techniques can only work with foggy images. In a real-time system, however, defogging foggy images becomes difficult if they are not identified as foggy or clear. Because we cannot rely on human vision to distinguish between foggy and clear pictures, we need a robust model that classifies the input image as foggy or clear based on some features. This paper proposes a robust Deep Learning (DL) model based on Convolutional Neural Network (CNN) for classifying the input as foggy and clear. The proposed Deep Neural Network (DNN) architecture is efficient and precise enough to classify images as foggy or clear, with a training time complexity of O(n2) and a prediction time complexity of O(n). The experimental results reveal promising results in both qualitative and quantitative assessments. The model has an accuracy of 94.8%, precision of 91.8%, recall of 75.8% and F1 score of 80.3% when evaluated on the SOTS dataset, indicating that it might be utilized to mitigate the safety risk in vision enhancement systems.

4. Dehazing and vision enhancement: challenges and future scope

   Sattwik Barua, Tannistha Pal, and Mritunjoy Halder

   In IET Intelligent Multimedia Processing and Computer Vision, May 2023

   Abs DOI

   Poor visibility of outdoor images has been drastically increased. Applications using computer vision, including surveillance systems, intelligent transportation systems, are not able to function properly due to limited visibility. Numerous image dehazing methods have been introduced as a solution to this problem, and they are crucial in enhancing the functionality of several computer vision systems. The dehazing approaches are intriguing to researchers as a consequence. In order to demonstrate that dehazing techniques could be successfully used in actual practice, this study conducts an extensive examination of the state-of-the-art dehazing approaches. In contrast, it motivates scholars to apply some of these methods for removing haze from hazy images. In this chapter, we discuss several robust mathematical models along with some neural network-based approaches and their implementations in various aspects. Finally, we address several concerns about difficulties and potential future applications of dehazing approaches. Due to poor visibility conditions, the visibility of outdoor images is drastically decreased. Applications using computer vision, including surveillance systems, intelligent transportation systems, etc., are not able to function properly due to limited visibility. Numerous image dehazing methods have been introduced as a solution to this issue, and they are crucial in enhancing the functionality of several computer vision systems. The dehazing approaches are intriguing to researchers as a consequence. In order to demonstrate that dehazing techniques could be successfully used in actual practice, this study conducts an extensive examination of the state-of-the-art dehazing approaches. In contrast, it motivates scholars to apply some of these methods for removing haze from hazy images. We keep an eye on the robust mathematical models along with some neural network-based approaches and their implementations in various aspects. Finally, we address several concerns about difficulties and potential future applications of dehazing approaches.

5. Detecting Emotional Sentiment in Cartoons

   May 2023

   Revelation23

   Abs

   Social media platforms are widely used by individuals and organizations to express emotions, opinions, and ideas. These platforms generate vast amounts of data, which can be analyzed to gain insights into user behavior, preferences, and sentiment. Accurately classifying the sentiment of social media posts can provide valuable insights for businesses, individuals, and organizations to make informed decisions. To accomplish this task, a customized private cartoon dataset (original images) of social media posts has been provided, which contains labels for each post’s emotion category, such as happy, angry, sad, or neutral. The task is to build and fine-tune a machine-learning model that accurately classifies social media posts into their corresponding emotion categories, using synthetic images.

2022

1. 

   A Framework for Sex Identification, Accent and Emotion Recognition from Speech Samples

   Sattwik Barua, Mritunjoy Halder, and Mohit Kumar

   In 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT), May 2022

   Abs DOI

   The purpose of understanding and interpreting the world lies in the cognitive system of a living body. The difficulties that humans face in acknowledging cognitive senses and abilities are only because of the singularity of our views on human cognition. If animals have feelings, there might be a case to be made that robots do have. Non-expert users interacting with robots may mislead the functionality of the robot. If someone is present in front of a robot and the robot can determine their sex, accent of English and emotion simply from their speech data, it helps in development of an embedded cognitive model. Though having limitations, studies in neuroscience have shown that animal models have significantly contributed to the understanding of the mechanistic and functional aspects of cognitive activities such as speaking. As a result, the usage of robots in a range of industries requires the capacity to make understandable and expressive speech. Thus, we present three models for identifying sex, accent, and emotion. To attain better outcomes, we have proposed a Deep Neural Network (DNN) and adjusted it as needed. To evaluate how well the model can perform, we have employed the attention layer, bidirectional Long Short Term Memory (LSTM) and Dropout concepts and judged our trainable datapoints to be very large. The experimental results show our superior findings, in terms of accuracy obtained in identification of sex and accent: 94.62%, 97.37% consecutively and recognition of emotion: 99.84%, which suggest that our study might be applied in the future as a noteworthy solution for the aforementioned problems.