Re-understanding Prompt: Starting from the Principles of GPT Models
What is a Prompt? How to Understand a Prompt?
A System Prompt is certainly a type of Prompt, and an extremely important part of it.
In a narrow sense, a Prompt can be equated with a System Prompt, but in a broader sense, a Prompt is not limited to the System Prompt alone.
This needs to be discussed from the principle of the GPT model. We know that the GPT model uses [previous content as input] to predict [subsequent content as output]. How can we simply and specifically understand this?
Take out your mobile phone and use your input method to type. As shown in the image below, we know that current input methods have intelligent input functions. When you type (spring sleep), the input method predicts (unaware of dawn); when you type(spring sleep, unaware of dawn), the input method predicts the subsequent content as (everywhere, the cry of birds is heard). Simply put, the GPT model does something similar to an input method: it “strives to continue text in a statistically reasonable way.” (In reality, it is a complex process for large models, and researchers are still exploring their in-depth principles; for us, this simple understanding suffices.)
Understanding this principle is crucial because, on a deeper level, it means: as long as we enable the machine to predict the next word accurately enough, it can accomplish many complex tasks! What is the upper limit? No one knows until practice tells us. This gives people unlimited imagination — the realization of Artificial General Intelligence (AGI). Although many interesting applications have emerged since ChatGPT became popular, what truly shocked me was the moment I realized this conclusion when first experiencing GPT-4. GPT-3.5 didn’t surprise me much, but watching GPT-4 type out the results I had in mind word by word on the screen filled me with anticipation for the future!
With this principle understood, it becomes easier to grasp what a Prompt is for the GPT model. In the input method example above, (spring sleep) in the first image is our input, and (unaware of dawn) is the model’s output, so is a Prompt. In the second image, (spring sleep, unaware of dawn) is the model’s input (with being the model’s previous output), and (everywhere, the cry of birds is heard) is the model’s output, so is a Prompt.
In multi-turn conversations, the model uses not only the System Prompt as input but also the user’s inputs and the model’s previous outputs to predict subsequent text.
Therefore, to make the most of large models, both users and prompt engineers should recognize that “all content used by the model to predict output results is a Prompt.” That is, not only the System Prompt is a Prompt, but the content we input in subsequent conversations is also a Prompt, and even the model’s previous outputs serve as Prompts for its subsequent outputs.