2

I believe that it is recommended to have a tiny bit of temperature with GPT 3 even for noncreative tasks like 0.2 or something (I am not entirely sure why).

Last I checked, and if I remember correctly, the examples from openai on their GitHub page use 0 temperature.

Is there any benefit in choosing a non-zero temperature for the chatgpt API when the query does not request a creative task? If so, are there some categories or examples?

[EDIT: to make the answer less subjective, perhaps I could ask what are the benefits of increasing the temperature in the chatGPT API]

userrandrand
  • 121
  • 5

2 Answers2

2

Like with many things, it depends on what you're trying to accomplish.

The two major parameters you can tweak are:

  • temperature - Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.
  • top_p - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered

In terms of some reference values, the forum post on Mastering Temperature and Top_p in ChatGPT API provides the following examples:

Use Case Temp Top_p Description
Code Generation 0.2 0.1 Generates code that adheres to established patterns and conventions. Output is more deterministic and focused. Useful for generating syntactically correct code.
Creative Writing 0.7 0.8 Generates creative and diverse text for storytelling. Output is more exploratory and less constrained by patterns.
Chatbot Responses 0.5 0.5 Generates conversational responses that balance coherence and diversity. Output is more natural and engaging.
Code Comment Generation 0.3 0.2 Generates code comments that are more likely to be concise and relevant. Output is more deterministic and adheres to conventions.
Data Analysis Scripting 0.2 0.1 Generates data analysis scripts that are more likely to be correct and efficient. Output is more deterministic and focused.
Exploratory Code Writing 0.6 0.7 Generates code that explores alternative solutions and creative approaches. Output is less constrained by established patterns.

Note: It is generally recommended that you set either temp or top_p, but not both. So for the table above, the values can be seen as roughly equivalent ways to achieve the desired outcome.

Another reason you might include a higher temperature is if the user indicates the first response is wrong and you want to attempt again, but get a different response. Although re-prompting with the new information can also work as well.

KyleMit
  • 121
  • 5
  • Thanks. I suppose the temperature=0.2 rather than 0 for coding is for alternative solutions when trying again (like the regenerate answer button in the web interface). I am not sure however if the table means T=0.2 OR top_p=0.1 or it means T=0.2 AND top_p=0.1. In the API documentation for `top_p` they mention `We generally recommend altering this or temperature but not both.`, it's mentioned here : https://platform.openai.com/docs/api-reference/chat/create – userrandrand Jun 01 '23 at 18:55
  • Yeah, should have clarified - I'll update the answer. The table means you should set one OR the other, but not both – KyleMit Jun 01 '23 at 19:07
0

Temperature = 0 means deterministic. The same input gives the same output every time.

If you are fine with fixed outputs, go for temperature = 0, since it's easier to debug.

Rexcirus
  • 1,131
  • 7
  • 19
  • Hi, thanks. I know that T=0 means deterministic. I am more interested in the advantages of adding temperature with the chatgpt 3.5 model. I believe it is recommended to add a little bit of temperature to the gpt 3 model to avoid it repeating itself in some cases. ChatGPT is a less raw model than GPT 3 and is more knowledgeable of what people like and dislike (such as answers with a lot of repetitions). Hence, I am wondering if there is still a reason to add temperature to the ChatGPT model. – userrandrand Mar 29 '23 at 08:58
  • 1
    The only reason I see is that it's weird to have the same exact response every time, in normal conversations. Imagine replying always "Hello there" to my "Hello". Usually some days you would say "Hi" or "Hello" to reply instead. ChatGPT repeating itself seems more a problem which could be handled with repeat penalty. – Rexcirus Mar 29 '23 at 15:24
  • That's a good point for a conversation indeed. I suppose if these gpt models are ever trained in chess and are equipped with some tree search algorithms or something it could also be good to not have a predictable game also. Those are some benefits of nondeterministic answers. I am still interested in how the type of answers change when the temperature is increased. I might run some statistics to see how answers change. I am not yet sure what to measure other than correctness. Maybe I could also message word frequency and text length or tendency to digress. – userrandrand Mar 30 '23 at 04:44
  • `temperature = 0` does not necessarily imply that the output will be deterministic. See the OpenAI docs. – nbro Apr 13 '23 at 20:35
  • 2
    Can you share a link to the relevant documentation page? – Rexcirus Apr 13 '23 at 21:58
  • @nbro I think Rexcirus was refering to your comment but did not mention your username – userrandrand Jun 01 '23 at 18:41