From pipelines to prompts

NLP in the LLM era

Traditional natural language processing (NLP) relies on task-specific pipelines (for sentiment analysis, entity recognition, topic modelling, etc.) and labour-intensive labeled datasets. Today, a single pre-trained large language model (LLM) can handle all tasks through plain-language prompts.

But is a generative model always the right tool for the job?

In this webinar, I will compare the results, data constraints, and compute requirements of running the same task on the same corpus through three eras of NLP: lexical search, text embeddings, and prompting a small LLM. We will see that embeddings remain a practical and efficient choice for semantic search. Small LLMs capable of running on a laptop give the best results if prompted properly and fine-tuned, but at a steep computational price and higher query response times.

Slides (Click and wait: this reveal.js presentation is heavy and takes some time to load.)

Slides content for easier browsing.