Nicholas Moratelli

I'm a second-year PhD Student in Artificial Intelligence Engineering at AImageLab research group under the supervision of Prof. Rita Cucchiara.

Email  /  CV  /  Bio  /  Scholar  /  Linkedin  /  GitHub  /  HuggingFace

Research

My research interests include Computer Vision, Natural Language Processing, and Generative AI. The following is a list of the research projects I have developed.

LLaVA-MORE enhances the well-known LLaVA architecture by integrating for the first time the use of LLaMA 3.1 as the language model.

In this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues.

In this paper, we propose a new training paradigm termed Direct CLIP-Based Optimization. Our approach jointly learns and optimizes a reward model that is distilled from a learnable captioning evaluator with high human correlation. This is done by solving a weighted classification problem directly inside the captioner. At the same time, DiCO prevents divergence from the original model, ensuring that fluency is maintained.

In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements.

In this paper, we propose PAC-Score++, a learnable metric that leverages the CLIP model, pre-trained on both web-collected and cleaned data, and regularized through additional pairs of generated visual and textual positive samples. Exploiting this stronger and curated pre-training, we show that integrating PAC-Score++ into a captioner fine-tuning stage results in semantically richer captions with fewer repetitions and hallucinations.

This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation.

In this paper, we propose Self-Cap, a captioning approach that relies on a learnable reward model based on self-generated negatives that can discriminate captions based on their consistency with the image. Specifically, our discriminator is a fine-tuned contrastive image-text model trained to promote caption correctness while avoiding the aberrations that typically happen when training with a CLIP-based reward.

The effectiveness and expressive power of prompts are limited by the fact that they can only lie at the input of the architecture. In this paper, we critically question the usage of learnable prompts, and instead leverage the concept of "implicit memory" to directly capture low- and high-level relationships within the attention mechanism at any layer of the architecture, thereby establishing an alternative to prompts in FSL.

This article tackles the task of generating fine-grained and accurate natural language descriptions of fashion items, a recently-proposed and under-explored challenge that is still far from being completely solved. We developed a Transformer model capable of reading and retrieving items from external memory through cross-attention operations, regulating the flow of information from the external memory using a novel fully-attentive gate.

Participation to National and European Projects

Academic Service