RLHF: Alignment: Key components of ChatGPT

Chat-GPT has been recently quite a popular AI model from openAI, with other AI firm’s announcing their own versions of similar projects such Google bard, facebook-llama based chatbots, and many more.

GPT stands for a generative pre-training, using a form of training employed for the large corpus of text to create language models, as it uses a large neural network with billions of parameters they are also known as large language models (LLM). There exist a range of LLM, after the initial development since 2018, these were initially designed such that they can be employed for downstream task, such, dialog, chat, translations, summarization, sentiment analysis etc. Since they are trained to learn core abstractions, they can generative new text. The modern language model-based frameworks have also been employed to generate in vision context, though diffusion-based model is more popular these days, employed by firms like mid-journey, google, stability ai and more. Here we focus our discussion on Chat-GPT, but the concepts are applicable boardly, and would provide some innovative application e.g. MRI imaging, or other biological applications, where is possible obtain generative models on specific datasets.

The other key recipe of the Chat-GPT is the reinforcement Learning from Human Feedback (RLHF), where in the generative model (large language model in this case) is training with datasets containing human feedback to improve performance. For example, training datasets consist of prompts which and response which are…