Integrating Human Perception: The Evolution of Multimodal LLMs

As AI advances rapidly, specialized models now interpret the world through multiple senses, much like humans do. These multimodal large language models (LLMs) revolutionize business interactions by combining the strengths of language models with the power to understand images, sounds, and videos, extending beyond text-based interactions.

OpenAI’s GPT-4V system card emphasizes the importance of incorporating additional modalities, such as image inputs, into LLMs as a crucial step in AI development. This multimodal shift is reflected in new models like DeepMind's Flamingo and Microsoft's KOSMOS-1, and chatbots such as ChatGPT and Gemini.

Utilizing multimodal LLMs can boost productivity and innovation across industries, from healthcare and technology to finance and automotive. Powered by vast datasets and innovative designs like transformers, these AI systems bring us closer to natural humanlike interactions. However, it’s crucial to consider ethical implications while using these technologies.

Multimodal LLMs simultaneously process and generate various data types, such as text, images, and audio. Using advanced techniques like deep learning, neural networks, and transformers, these models can comprehend text sentiment, recognize objects in images, and interpret audio cues, integrating this information to form a holistic understanding. 

Here are the different types of interactions multimodal LLMs can manage:

• Cross-modal interactions: The input and output consist of different modalities. For instance, providing text to the model generates a corresponding image (text-to-image), or describing an image in text format (image-to-text).
• Multimodal input processing: These models interpret inputs from different data sources. For example, analyzing an image and reading the text within it to provide a summary or detect patterns.
• Multimodal output generation: These models produce outputs across different modalities. For example, creating both a text description and a relevant image or video sequence.

Multimodal LLMs also enhance accessibility by quickly analyzing different data types, including tables, charts, and graphs, allowing businesses to make faster, better-informed decisions. 

Comparison with unimodal learning

Multimodal LLMs emerged to overcome the intrinsic limitations of unimodal LLMs, which process only a single data type and often fail to reflect the complexity of human perception and interaction. Unimodal systems struggle with complex tasks that involve multiple data types, like summarizing patient data in healthcare, analyzing visual and textual data in e-commerce, or integrating visuals with text in robotics.

Incorporating multimodal data significantly enhances AI's understanding and boosts model performance. For example, in healthcare, it enables faster and more accurate diagnostics by integrating imaging data, text reports, and lab results, enhancing medical productivity and patient outcomes.

Advantages of multimodal LLMs

Multimodal LLMs offer various benefits, including:

• Improved decision-making: Combining text and visuals supports sophisticated decisions, increasing an organization's agility in tackling complex issues.
• Extensive understanding: A complete understanding of diverse data types improves market analysis and offers customer behavior insights.
• Enhanced user experience: Customized interactions considering both text and visuals engage customers more effectively, leading to higher satisfaction.
• Operational efficiency: Streamlined processes reduce the time and effort needed for tasks, enabling better resource allocation.

Additionally, multimodal LLMs enhance downstream tasks, such as generating image captions, answering visual questions, and performing natural language inference. Their ability to integrate non-textual context makes them resilient to noisy or incomplete data, unlike traditional models.

Beyond text, multimodal LLMs handle various inputs, including sensory data, making computer-user interactions more versatile. They can also blend data types to generate detailed illustrations or infographics.

At a high level, a multimodal LLM consists of the following components:

1. Modality encoders: Each modality (text, image, audio, video) has its own dedicated encoder to convert raw data into numerical representations or embeddings. For example: 
   a. Text encoders: BERT, GPT
   b. Visual encoders: Vision Transformers (ViT), CLIP
   c. Audio encoders: HuBERT, Whisper
2. Input projector: It aligns features from various modalities into a common space using linear projection or more complex mechanisms like cross-attention. This step ensures that the LLM can seamlessly process and relate information from various modalities.
3. LLM backbone: The core of the model, the LLM backbone, processes the aligned features. It incorporates advanced capabilities such as zero-shot learning, few-shot learning, and chain-of-thought reasoning that allows the model to handle various tasks with minimal training data. Examples of LLM backbones include GPT, LLaMA, and Flan-T5.
4. Output projector: It converts the encoded data from the LLM backbone into the desired modality for the target task. It can project outputs into different modalities, similar to how modality encoders handle various input types. The output projector often uses techniques such as Tiny Transformers or Multi-Layer Perceptrons (MLPs) to perform these conversions efficiently.
5. Modality generator: It takes the features processed by the output projector, refines them through a denoising process, and generates outputs in the required modalities, like images, audio, or video.
   
   Depending on the target task, this component may use advanced techniques like Latent Diffusion Models (LDMs) for image synthesis, Zeroscope for video generation, or AudioLDM2 for audio production. It often leverages pre-trained models like VAEs (Variational Autoencoders) or UNet architectures to ensure high-quality output.

Original image source

Here are some key methodologies used for building multimodal LLMs, enhancing the model's effectiveness and efficiency:

• Unified representation learning: Data from various modalities is integrated into a unified representation, ensuring that each modality retains its unique characteristics while coordinating with others. Techniques like contrastive learning are used to find a shared feature space where text, audio, and images can be compared directly.
• Cross-modal attention mechanisms: Implementing cross-modal attention mechanisms allows the model to focus on relevant information from one modality based on the context provided by another. For example, when answering questions about an image, the model uses textual descriptions to guide its attention to the relevant parts of the image. 
• Generative pretraining for multimodality: Pretraining the LLM on a diverse set of multimodal data allows the model to gain a broad understanding, which can then be fine-tuned for specific applications like generating captions for images or converting text to images, enhancing its versatility and utility.
• Efficient data processing pipelines: Handling diverse datasets for multimodal LLM training requires efficient data processing pipelines optimized for loading, preprocessing, and batching mixed data types. This optimization is vital for managing the high computational demands associated with training multimodal models.

Implementing multimodal LLMs presents several challenges and ethical considerations, including:

• Data representation: Different modalities have distinct data formats and characteristics. Creating a unified representation to capture the richness of each modality is challenging.
• Data bias: These models may inherit biases from training data, potentially leading to stereotypes or unfair judgments.
• Complex integration: Integrating text and images, or other data types, requires intricate modeling and contextual understanding.
• Understanding nuances: Training models to grasp abstract concepts like humor or sarcasm is difficult, and mastering these subtleties of human communication remains a challenge.
• Computational requirements: The vast computational resources necessary to train these models are expensive and energy-intensive, potentially limiting their access and application.

Let's look at some successful multimodal AI examples demonstrating the power and potential of multimodality:

• Meta’s ImageBind
  Developed and open-sourced by Meta AI, ImageBind is a groundbreaking model that creates a unified representation of six different data types: text, image/video, audio, 3D depth, thermal imaging, and motion (through IMU sensors). It gives machines a comprehensive understanding that links a photo's visual elements with its potential sound, shape, temperature, and movement. This model has already outperformed models trained on a single data type and could transform content recognition, creation, and moderation.
• OpenAI’s GPT4-V
  GPT4-V, or “GPT-4 Turbo with Vision,” is at the forefront of commercial AI applications, capable of processing images as input. This model stands out for generating image captions, providing detailed analysis of real-world images, and interpreting documents with visuals and text. With limits of 128,000 input tokens and 4,096 output tokens, GPT4-V enables developers to build intricate and intelligent applications with visual interpretation capabilities.
• Google’s Med-Palm 2
  Designed by Google's health research teams, Med-Palm 2 showcases excellence in medical knowledge, even scoring at an expert level on simulated U.S. Medical Licensing Exams (with 85%+ accuracy). It's one of the first multimodal LLMs that can interpret medical images, like X-rays, alongside text, which could revolutionize the way medical professionals diagnose and study diseases.
• Microsoft’s Kosmos-1
  Microsoft's Kosmos-1 stands out with its zero-shot and few-shot learning abilities. This multimodal LLM (MLLM) strives to align perception with language understanding, enabling it to visualize and describe scenarios accurately. It was trained on a diverse range of web data, including text, images, and their descriptions.
• Salesforce’s BLIP-2
  Salesforce has revealed BLIP-2, an advanced open-source visual-language model focusing on compute-efficient vision-language pretraining (VLP). It uses existing pretrained models to bridge the gap between visual and textual understanding in LLMs in a scalable and cost-effective way. The innovative Querying Transformer (Q-Former) helps integrate visual understanding into models that were previously text-only.

Advancements in multimodal training are setting the stage for more sophisticated AI applications, including:

• Dual-CNN (Convolutional Neural Network) framework for enhanced multimodal neural data integration
  A recent study introduced a dual-CNN framework, which efficiently integrates neural signals from various modalities, such as EEG and EOG, without relying on extensive prior knowledge. This innovation enhances the accuracy of complex tasks like sleep scoring by incorporating temporal correlations between brain states.
  
  Such frameworks are likely to play a crucial role in future research, particularly in fields like neuroscience and diagnostics, where the fusion of multimodal data can lead to more precise and reliable outcomes.
• Revolutionized construction inspections 
  A recently proposed "AutoRepo" framework presents a groundbreaking approach to automating construction inspection reporting using multimodal LLMs. This framework integrates three main components: a critical scene data acquisition module, a core LLM module, and a report generation module.
  
  The framework was validated in a real-world case study, demonstrating significant improvements in inspection efficiency, accuracy, and resource management compared to traditional manual methods. AutoRepo marks a significant leap forward in the field of construction technology, paving the way for safer and more efficient project management.
• On-device scene text recognition
  "Lumos", an innovative framework developed by Meta Reality Labs, integrates an on-device scene text recognition (STR) component with cloud-based, multimodal LLMs to deliver high-quality, low-latency text understanding from images.
  
  The framework addresses key challenges such as latency, computational efficiency, and the complexity of recognizing text in diverse, real-world scenarios, ensuring efficient text recognition and improved question-answering accuracy by 28%. Frameworks like Lumos hold immense potential to advance the capabilities of multimodal LLMs in tasks requiring scene text recognition.
• Advanced geospatial intelligence
  Emerging research, such as the study on the geographic and geospatial capabilities of multimodal LLMs, highlights the growing interest in exploring how these models can be applied to new domains. The ability of models like GPT-4V to analyze and interpret geographic data opens up potential applications in navigation, environmental monitoring, and disaster response.
  
  However, current limitations, including challenges with precise localization and complex data interpretation, highlight the need for further research to enhance these capabilities. As multimodal LLMs continue to evolve, they are expected to play a pivotal role in advancing geospatial intelligence and broadening the scope of AI-driven applications.
• Enhanced autonomous driving
  In a new study, researchers proposed a LimSim++ framework that represents a significant step forward in autonomous driving research, integrating multimodal LLMs into complex, real-world simulations. The framework simulates long-term, dynamic environments, providing a critical infrastructure for continuous learning and model refinement.
  
  As this research area advances, the focus will likely shift towards enhancing the scalability and efficiency of these models, enabling them to meet the intricate demands of autonomous driving with greater precision. 

Generative AI (genAI) is rapidly growing, projected to reach US$4.31 trillion by 2030. AI visionary Mustafa Suleyman suggests we're entering an "interactive phase," where conversational interfaces will become the norm, enhancing human capabilities and transforming our interaction with technology from clicks to natural conversation. However, ethical considerations like inherent biases and privacy issues must be addressed. Combining groundbreaking AI with moral sensitivity is crucial to ensuring these technologies benefit society.

In summary, while multimodal LLMs represent a new era of AI sophistication and contextual understanding, it's crucial to acknowledge their limitations and societal impacts alongside their capabilities. The future of multimodal LLMs offers immense potential for innovation, but it demands a commitment to responsible development and ethical practices.

At Turing, we’ve helped top foundation LLM companies, such as OpenAI, Google, Meta, Anthropic, and more, optimize the way they approach LLM model evaluation, factuality and data analysis, multimodal reasoning, LLM training, and more.