Globally setting TokenSelectionBiases (logit_bias) when invoking a function

[Charles Chen (CharlieDigital)]

Use Case

We have a set of plugins and functions which we would like to globally and dynamically inject a set of TokenSelectionBiases to ban or reduce the occurrence of specific tokens.

Analysis

My first attempt was via the KernelArguments constructor. However, the observation here is that if I pass in OpenAIPromptExecutionSettings to the KernelArguments constructor, it wipes out the configuration that was loaded when the prompt is imported via ImportPluginFromPromptDirectory.

In other words, the execution_settings loaded via importing the plugin are replaced entirely.

My second attempt was via the FunctionInvoking event delegate before invoking:

var semanticFunction = kernel.Plugins.GetFunction(pluginName, functionName);

kernel.FunctionInvoking += (obj, args) =>
{
    foreach (var key in args.Arguments.ExecutionSettings?.Keys ?? Array.Empty<string>())
    {
        Log.Here().Debug("Updating settings for: {ExecutionSetting}", key);

        (args.Arguments.ExecutionSettings![key] as OpenAIPromptExecutionSettings)
            !.TokenSelectionBiases = ... // INJECT THE BIASES HERE
    }
};

var result = await semanticFunction.InvokeAsync(kernel, variables, cancellation);

However, the execution_settings are nowhere to be found and in fact, this makes sense because args.Arguments.ExecutionSettings doesn't contain the execution_settings in the config.json.

Question

Given that, is there a mechanism to globally inject a dynamically created TokenSelectionBias (logit_bias) for all executions?

Where are the execution_settings from the config.json stored?

Can they be modified programmatically after loading?

[Mark Wallace (markwallace-microsoft)]

I am cleaning up old unanswered discussions