Introduction
This project leverages Tangible User Interfaces (TUI) to create an interactive platform where children's drawings are transformed into dynamic stories using AI. By capturing hand-drawn sketches through a camera and processing them with pre-trained image recognition models, the system generates meaningful narratives, enhancing creativity, collaboration, and storytelling skills.
Tangible User Interface (TUI)
A Tangible User Interface (TUI) is an interaction model that enables users to engage with digital information through physical objects. By seamlessly integrating the physical and digital worlds, TUIs provide a more intuitive and natural way to interact with technology, enhancing user experience and accessibility.
Benefits of TUIs
- Intuitive Interaction: TUIs leverage users' natural ability to manipulate physical objects, making digital interactions more accessible and user-friendly.
- Enhanced Engagement: The physical nature of TUIs fosters immersive experiences, increasing user involvement and encouraging exploration.
- Collaborative Opportunities: TUIs facilitate shared interactions, allowing multiple users to engage with digital content through collaborative physical interfaces.
- Accessibility: TUIs can be designed to accommodate users with disabilities, offering alternative methods for interacting with digital systems.
Project Aim
To develop an interactive platform that enables children to bring their imagination to life by drawing objects, which are then processed using AI to generate a cohesive and creative story.
Proposed Solution
The system captures children's drawings through a camera, identifies them using a pre-trained image recognition model, and integrates them into an AI-generated story. It supports:
- Up to three children drawing one object each, or a single child drawing up to three objects.
- Real-time detection and recognition of drawings.
- AI-powered storytelling based on identified elements.
Target Audience
- Primary school children (ages 5–12): Encourages creative thinking and storytelling.
- Teachers and facilitators: Provides an engaging classroom activity.
- Parents: Enhances parent-child bonding through interactive storytelling.
Benefits for Children
- Stimulates creativity and imagination.
- Encourages teamwork and collaboration.
- Develops storytelling skills in an interactive manner.
Tools
- Physical Tools: Drawing utensils, paper, digital drawing pads.
- Digital Tools: TUI interface, camera, AI-driven image recognition, and the storytelling platform.
Technologies Used
- Development Environment: Visual Studio Code for front-end and back-end development.
- Camera Integration: Captures children's drawings in real time.
- Image Recognition: Pre-trained “Doodle Net” library for object detection.
- Story Generation: Gemini API for dynamic and contextually relevant storytelling.
Resource Links
Usability Testing
- Model 1: Directly drawing on a digital screen.
- Model 2: Drawing on paper or an iPad, captured via a camera.
Learnings from the Project
Understanding Tangible User Interfaces (TUI):
- Gained a deeper understanding of how physical interactions can enhance digital experiences.
- Explored the role of TUIs in education and their impact on child-centered learning.
Future Scope:
- Expanding the AI model to recognize more complex drawings.
- Enhancing customization options to allow children to personalize their stories further.
- Exploring integration with interactive storytelling apps for wider accessibility.
