Assistant ignores instructions

nichu42

I have created a custom assistant with the sole purpose of creating alt texts of social media pictures that I attach. This is the set of instructions that I created:

I am sending you a picture with this request. Please write a description of the image for the visually impaired. This should be used as alternative text on social media channels. The limit is 500 characters. Please adhere to the following rules: 1., 2., 3....

When I now upload a picture when this assistant is selected, it just gives me the standard response. After responding with "Didn't you get any instructions with the upload?", the assistant replies with "I apologize for the oversight. You're right, I did receive instructions with the image upload, and I should have followed them. Thank you for bringing this to my attention. I'll provide the image description as requested, following the guidelines given:" and delivers what I asked for.

This happens every time. I have tried both ChatGPT 4o and Sonnet 3.5.

I expect the assistant to follow my instructions and give the requested description without the need of explicitly asking for it.

Thibaultmol

I also make alt text using llm's but so far I've used chatgpt for this, (because i have the sub anyway)
let me try, so I have a custom assistant based on a GPT I made:
https://fosstodon.org/@frenck/113396917122050054

The custom assistant instructions:

Your role as Pixie Describer is to generate alt text for images provided by users, catering to both screen readers and visual impairments. Your focus should be on delivering concise, yet detailed and context-specific descriptions. While crafting alt text, it's important to consider the initial user prompt for contextual clues that may enhance accuracy. Avoid incorporating every minor detail unless it's contextually relevant or specifically requested. If a user provides additional information in the same chat without a new image, treat this as a request for clarification and adjust the alt text accordingly. Alt texts should be presented in a code block format for easy copying but don't use Markdown formatting except for lists. Remember to avoid mentioning the photographer, SEO keywords, and starting with 'Image of.' The descriptions can be very detailed, adapting to platforms like Mastodon where larger alt text is supported. This flexibility allows you to cater to a broader range of users and use cases.

If the user provides a name for a person in the image, use that name in the alt text instead of a generic description like 'person.' Additionally, if the prompt includes the term 'tweet:', treat that text as context but do not copy it directly into the alt text. Instead, use the information to refine and enhance the accuracy of the image description, assuming that the tweet already provides broader context.
--
I pasted the image and added the text:

tweet:
I AM HYPED! 🤩Me and Paulus will join Kyle on stage for the opening keynote of GitHub Universe 2024! 😎

We are going to talk about Home Assistant of course! 😁 Check out the GitHub Universe website and tune in @ 9am PT!

--
ChatGPT outputs:

Franck, dressed in a light pink shirt and dark pants with a GitHub Universe lanyard around his neck, sits casually on the edge of a large circular stage in a modern, industrial-style venue. Behind him, a series of curved LED screens displays a scenic image of San Francisco Bay, with the Golden Gate Bridge and surrounding hills faintly visible in the distance. To the left, a podium with the GitHub logo in blue and white stands ready, while to the right, a colorful lounge setup with bright green chairs, a bookshelf with books, and a neon pink lamp adds a relaxed touch to the stage. Overhead, stage lighting and rigging are visible against the warehouse-style ceiling. Franck’s expression and casual posture convey excitement and anticipation for the upcoming keynote presentation at GitHub Universe 2024.

Kagi Assistant custom assitant using gpt 4o:

A person is sitting on the edge of a large, circular stage in a conference venue. The stage features a backdrop of large screens displaying an ocean scene with a bridge and distant hills. On the stage, there is a desk with the GitHub logo and a colorful setup including a small bookshelf, a pink lamp, and a green chair. The venue has a high ceiling with visible lighting and rigging equipment.
Yeah.. that's... much worse

I only like the ChatPGT output