I have learned a bit recently about prompt strategies. For example, there was a paper about how just by saying “Let’s think step by step” can increase answer quality by like 40%. I have also come to appreciate that models like GPT4 sometimes actually do better with short prompts than long ones. One paper I think found that LLMs have “recency bias”, so if you give sample text, then an instruction, it does better than instruction, then sample text, because in that case, it pays less attention to the instruction.
I have struggled a lot with basically zero or few shot prompting GPT-4 to give me highly concrete information and/or a specifically actionable plan or set of steps.
To give an example, it very often gives you very vague, general advice like,
“If you’re looking for a job, trying looking around on online job websites, or contacting a local employment agency”.
If I give it more specific information, and try really quite hard to get it to tell me something way more specific, at best it might add in some very common sites and places, like,
“First, think of what jobs you might like, based on your skills. Then, search for those keywords in a job listing site, like Monster.com or Indeed. Also, consider contacting the local municipal job center, [City Job Center’s name, address, phone number.”
It has been quite hard for me to try to get GPT-4 to be way, way more like a hardcore data-crawling machine, so to speak. It would be really nice to know if there was a special trick that has been discovered - just like the surprising efficacy of 5 words, like “Let’s think step by step” - where you basically tell it that you only want specifics, and you don’t want just like, the top three - ideally, you want it to figure out every single known job website or app on the internet, every single known job center in your county, every single employment agency and recruitment firm too, all of their names, links to their webpages, etc. Given that some GPT-4 systems are able to search the web, the requested task could make it clear that the model is free to use any information it already possesses internally; search amply and procedurally on the web to find more information that it needs; but furthermore, if it does not know, that is fine, but in that case, it should provide further, actionable steps for the human to take, like a specific place they could ask, or specific google keywords they recommend searching for.
Similar to information, I find it difficult to get GPT-4 to make a set of instructions that totally eliminates as much open-endedness or choice as possible - in which every single conceivable way of breaking down a task into tiny actions is present. Instead of saying, “make accounts on glassdoor and LinkedIn.com. Register with your email. Fill out your profile with relevant information”, I want to understand how to get it to say something like, “Ok, your name is ___. What’s your email address? And main skills? Got it. Ok, let’s start with LinkedIn because _____ (intelligent justification, even statistically backed, for why it has a high success rate). Based on this data analysis I found / made, it turns out there’s very high demand for this very specific job title right now, on LinkedIn: ___. And I can easily imitate some common resumes of people in those fields. So, here is the text of your resume: ___. Download and save that as a Word document. Now click this link here: _____, and click “apply” - that job is nearby you and it’s probabilistically likely you may get it. Submit it. Next, check your email once every 3 hours, because ____”.
The question here is not so much wanting a true AGI / AutoGPT, but just trying to know powerful simple keywords, prompts, commands, etc., that just help the model understand the difference between specific and vague. The word “specific” has not helped me at all, unfortunately, unless I go through some iterative prompting / fine-tuning, but it’s not convenient enough for daily use.
Any research-backed findings on this?
Thanks.
