Deep Learning for Natural Language Processing

Neural natural language processing (NLP) underpins some of the most important technologies of the information age. It is found in tools for web search, advertising, emails, customer service, translation, and virtual agents, among many other applications. Most recently, large language models (LLMs) like the ones powering ChatGPT have been shown to have surprisingly varied knowledge and abilities far beyond the tasks they were trained for, and this has opened new and potentially very important application possibilities for NLP. This micro-credential will introduce students to the neural network architectures that power modern NLP including LLMs like GPT. Students will learn how such networks function and will be given the opportunity to train NLP systems using popular open-source neural NLP toolkits and libraries.

The micro-credential will progress through three main learning blocks.

• The first block will impart theoretical understanding of the principal neural network architectures used for NLP, including feed-forward, recurrent and transformer network architectures, graph-based neural networks, and large-scale pretrained language models. Students will be introduced to the mathematical foundations of the relevant machine learning models and their associated optimisation algorithms.
• In the second learning block, students will gain practical understanding and skills in solving a number of NLP tasks by applying end-to-end neural architectures, fine-tuning existing neural language models on specific problems, and other approaches, covering a range of applications including analysing latent dimensions in text, transcribing speech to text, translating between languages, and answering questions. Students will learn about challenges, risks, and opportunities arising from the applications of deep learning techniques to such tasks.
• The third learning block will cover recent applications of neural networks including LLMs to multimodal and multilingual tasks that were largely infeasible before the emergence of modern neural network architectures.

On successful completion of this micro-credential the learner will be able to:

• Reflect on and assess the theoretical underpinnings and practical applications of a range of different neural models used to solve NLP tasks, and how to select and apply optimisation algorithms for them
• Design, test and implement neural attention mechanisms and sequence embedding models, and combine these modular components to build state of the art NLP systems.
• Critically assess the range of available commonly used toolkits, libraries, reusable trained models and datasets in neural NLP, understand their possible uses, and assess their limitations
• Critically assess and choose appropriate neural architectures for different NLP tasks, taking into account computational requirements, and adapting techniques from different subfields, languages and domains
• Design, test and implement common neural network models for NLP tasks including those first introduced in the Foundations of NLP module (CA6010).
• Critically assess and apply in practice reusable word and higher-level representations in neural NLP, and the difference between non-contextualised word vectors (word2vec, GloVe, etc.), and contextualised word vectors (ELMo, BERT, etc.), and the methods used to produce them.
• Reflect on the challenges posed by pre-trained neural language models, including issues of bias and factual correctness in generated text
• Reflect on and apply in practice knowledge about the possibilities opened up by modern neural architectures in enabling learning across languages and modalities
• Reflect on and apply in practice learning relating to working and communicating effectively in a team to design and implement solutions for new domains or unfamiliar contexts, justifying the proposed design and development strategy.

A Primary Honours degree, Level 8 in Electronic/Electrical/Computer Engineering, Applied Physics, Computer Sciences or other Cognate/Engineering Disciplines. Applications are also invited from diverse educational and/or employment backgrounds, with applications evaluated on a case-by-case basis.

And also to indicate the required documentation:

• Please provide Academic Transcripts for final year of study where appropriate (English translation)
• All applicants must submit a copy of their passport.

If applicable, evidence of competence in the English language as per DCU entry requirements.