VLM Engineer

Technology Innovation Institute (TII) is a publicly funded research institute, based in Abu Dhabi, United Arab Emirates. It is home to a diverse community of leading scientists, engineers, mathematicians, and researchers from across the globe, transforming problems and roadblocks into pioneering research and technology prototypes that help move society ahead.

Artificial Intelligence and Digital Research Centre

As part of TII’s Artificial Intelligence Research Center, the Extreme-Scale Language Model team is developing and implementing innovative deep learning technologies with a broad range of applications, from Natural Language Processing to Perception and Vision. Our team has developed the Falcon models and is now planning to continue our journey into cutting-edge applied research in the topic of large language models.

Key Responsibilities:

• Vision Model Ablation Studies: Conduct comprehensive ablation studies on vision models to assess the impact of various components and configurations. Collaborate with researchers to analyze and report on the effectiveness of different model architectures and settings.
• Data Ablation Research: Partner with team members to perform data ablation studies, identifying optimal data types and structures for training vision-language models. Also, analyze the impact of different data inputs on model performance, particularly focusing on vision-language alignment.
• Model Evaluation: Develop and implement robust evaluation protocols for vision language models. Assess model performance across diverse benchmarks and real-world scenarios.
• Model Training and Optimization: Engage in model training, with an emphasis on integrating LLMs with vision models like CLIP. Technical

Skills Required:

• Expertise in machine learning, particularly in vision-language models and LLMs.
• Strong understanding of model architectures like CLIP and their application in vision language tasks.
• Proficiency in distributed training techniques and multi-GPU optimization.
• Experience with deep learning frameworks (e.g., PyTorch).
• Strong analytical skills for conducting ablation studies and evaluating model performance.
• Familiarity with dataset curation and processing for vision and language tasks.

Qualifications:

• PhD degree in deep learning.
• Proven track record of research and development in vision-language models. •
• Publication record in top-tier conferences is highly desirable

At TII, we help society to overcome its biggest hurdles through a rigorous approach to scientific discovery and inquiry, using state-of-the-art facilities and collaboration with leading international

institutions. Our rigorous discovery and inquiry-based approach helps to forge new and disruptive breakthroughs in AI, advanced materials, autonomous robotics, cryptography, digital security, directed energy, quantum computing and secure systems.