Today I played a bit with Goodfire AI's Ember tool, which is their mechanistic interpretability tool. Ember is a nice UI around their SDK. You can inspect which concepts are activated when a model outputted a specific token, and search through all the concepts known by the model. It lets you intervene on a model's behavior by searching through its known concept vectors and boosting or de-boosting selected concepts during generation.

This is cool. Instead of prompting a model to change its behavior, you can calibrate the behavior with more granularity, like when TARS had its humor setting reduced by 25% in Interstellar.

They also have a higher-level AutoSteer method. Auto Steer is pretty neat: it simplifies the rather than having to manually search for concepts relating to a desired behavior, you can simply tell it the desired behavior and it will automatically identify which concepts to boost in order to get the desired behavior. For example, if I want it to be a "very pushy sales person", it automatically selects a few features as well as their relative weighting:

I started the conversation with tell me about Meta's Ray ban glasses and got a standard sales pitch, as expected.

This example is pretty straightforward, but what if you want to do something a little bit less obvious? For example, what if your desired behavior is talk like mark twain on his death bed? I checked, and the model does not have any features specific to Mark Twain, much less on his death bed . Decomposing this into the correct blend of concepts and their coefficients would take some guesswork.

Auto Steer proposed a few settings for the prompt which were reasonable. It didn't magically capture the essence of a dying Mark Twain. There's probably a few things going on: , and Ember limits the steering vectors to three, which are not enough concepts to capture "Mark Twain on his deathbed". Also, the base model (Llama 3.3 70B) is post-trained to make it aware that it is a language model, so I would need to add negative steering vectors to suppress it talking about its "systems".

I wonder how they achieved this. Perhaps given the prompt very pushy sales person, they generate a bunch of texts exemplifying a very pushy sales person, then a bunch that are unrelated to that concept, and identify which features were more active relative to the negative control. I notice that Auto Steer does not seem to suggest negative prompts.

One thing I find interesting is that adding a bunch of negative steering vectors with a weak magnitude of -0.1 it did not seem to have an effect, but if I increased the magnitude of the negative steering vectors to -0.3 it will eventually break the model. For the negative concepts, I chose food and Japanese-related concepts, which should be pretty orthogonal to my question about Ray ban glasses. Nevertheless, the strong negative steering vectors broke the model, making it repeat itself over and over until it reached the token limit.

I wonder how they combine the concepts to create the steering vector. The naive way to do it is with a weighted average of the component steering vectors in Euclidean space, but that doesn't give the right answer when you're dealing with vectors on a unit sphere. I go into that topic in my post on weighted spherical averages. but I don't know how they would incorporate the negative steering vectors.

Limitations

You also have to be careful with the strength of the steering vectors. If you make the magnitudes of the steering vectors too strong, the model will get carried away by them and ignore your prompt. If you make them even stronger, the model goes crazy and outputs incoherent text.

Conclusion

This is super cool. As far as I'm aware, Goodfire is the only company which is creating a platform for model interpretability and steering. I think it's good that they are doing this as a product, because it focuses them to create techniques which are scalable and broadly applicable to a variety of model architectures and modalities. This is the industrialization of mechanistic iterpretability and steering, which I think is one of the most important problems to be solved right now. I am also glad that they are a public benefit corporation, which is a relief because it makes them more aligned. And they have the dream team of interpretability researchers. I wish the Goodfire team the best of luck!