Show HN: Use any LLM in Go with stable, minimal API

56 minutes ago 1

A powerful and flexible Go library for interacting with Large Language Models (LLMs) with built-in support for function calling and streaming responses. Currently supports Anthropic, Google, and OpenAI compatible providers.

Supports Anthropic, Google (Gemini + Vertex), and OpenAI (Chat Completions + Responses)
Also supports using custom endpoints compatible with any of the APIs above
Streaming responses (including thinking) for real-time interaction
Built-in tool calling with Go generics to generate JSON schemas automatically
Structured output (JSON schema based) for model response
OpenAI only: Custom grammars like Lark / Regex for structured output
Prompt cache hints
Image inputs and image generation / editing
Usage tracking

Text diffusion model support (Inception, Google)
Realtime bidirectional text/audio streaming (WebRTC)

go get github.com/flitsinc/go-llms

Here’s a simple example that creates an LLM instance and has a conversation with it:

package main import ( "fmt" "os" "github.com/flitsinc/go-llms/content" "github.com/flitsinc/go-llms/llms" "github.com/flitsinc/go-llms/openai" "github.com/flitsinc/go-llms/tools" ) func main() { // Create a new LLM instance with OpenAI's o4-mini model llm := llms.New( openai.NewResponsesAPI(os.Getenv("OPENAI_API_KEY"), "o4-mini"), ) // Optional: Set a system prompt llm.SystemPrompt = func() content.Content { return content.FromText("You are a helpful assistant.") } // Start a chat conversation for update := range llm.Chat("What's the capital of France?") { switch update := update.(type) { case llms.TextUpdate: fmt.Print(update.Text) } } // Check for errors after the chat completes if err := llm.Err(); err != nil { panic(err) } }

Generating images with Gemini 2.5 Flash Image (aka Nano Banana)

You must specify modalities for this model to work (and you cannot use WithThinking):

func main() { provider := google.New("gemini-2.5-flash-image-preview"). WithGeminiAPI(os.Getenv("GEMINI_API_KEY")). WithModalities("TEXT", "IMAGE") llm := llms.New(provider) for update := range llm.Chat("Draw a sketch of a happy banana") { switch update := update.(type) { case llms.TextUpdate: fmt.Print(update.Text) case llms.ImageUpdate: path := fmt.Sprintf("image-%d.png", time.Now().Unix()) // We assume we always get a base64-encoded PNG image -- don't do this in prod! b64, _ := strings.CutPrefix(update.URL, "data:image/png;base64,") data, _ := base64.StdEncoding.DecodeString(b64) os.WriteFile(path, data, 0644) fmt.Printf("\nImage written to: %s\n", path) } } }

Advanced Usage with Tools

Here’s an example showing how to use tools (function calling):

package main import ( "fmt" "os" "github.com/flitsinc/go-llms/anthropic" "github.com/flitsinc/go-llms/llms" "github.com/flitsinc/go-llms/tools" ) // Define tool parameters type CommandParams struct { Command string `json:"command" description:"The shell command to run"` } // Create a shell command tool var RunCommand = tools.Func( "Run Command", "Run a shell command and return the output", "run_command", func(r tools.Runner, p CommandParams) tools.Result { return tools.SuccessWithLabel(p.Command, map[string]any{ "output": fmt.Sprintf("Simulated output for: %s", p.Command), }) }, ) func main() { // Create a new LLM instance using Anthropic's Claude with tools llm := llms.New( anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-20250514"), RunCommand, ) // Chat with tool usage for update := range llm.Chat("List files in the current directory") { switch update := update.(type) { case llms.TextUpdate: fmt.Print(update.Text) case llms.ToolStartUpdate: fmt.Printf("(Using tool: %s)\n", update.Tool.Label()) // Shows "Run Command" case llms.ToolDoneUpdate: // Shows the label specific to this execution, e.g., "ls -l" fmt.Printf("(Tool result: %s)\n", update.Result.Label()) } } // Check for errors after the chat completes if err := llm.Err(); err != nil { panic(err) } }

Sometimes, you might have a set of predefined tool schemas (perhaps from an external source or another system) that you want the LLM to be able to use. AddExternalTools allows you to provide these schemas along with a single handler function.

This is useful when the logic for handling multiple tools is centralized, or when you need to dynamically add tools based on external definitions.

The handler function receives the tools.Runner and the raw JSON parameters for the called tool. You can use llms.GetToolCall(r.Context()) within the handler to retrieve the specific ToolCall instance, which includes the function name (tc.Name) and unique call ID (tc.ID), allowing you to dispatch to the correct logic.

// Example external tool schemas (could come from a config file, API, etc.) var externalToolSchemas = []tools.FunctionSchema{ { Name: "get_stock_price", /* ... */ }, { Name: "get_weather", /* ... */ }, } func main() { llm := llms.New(anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-20250514")) // Add external tools and their handler llm.AddExternalTools(externalToolSchemas, handleExternalTool) // Now the LLM can use "get_stock_price" and "get_weather" for update := range llm.Chat("What's the weather in London?") { switch update := update.(type) { case llms.TextUpdate: fmt.Print(update.Text) case llms.ToolStartUpdate: fmt.Printf("(Using tool: %s)\n", update.Tool.Label()) case llms.ToolDoneUpdate: fmt.Printf("(Tool result: %s - %s)\n", update.Tool.Label(), update.Result.Label()) } } if err := llm.Err(); err != nil { panic(err) } } // Single handler that forwards external tool calls. func handleExternalTool(r tools.Runner, params json.RawMessage) tools.Result { // Get the specific tool call details from the context toolCall, ok := llms.GetToolCall(r.Context()) if !ok { return tools.Errorf("Could not get tool call details from context") } // Typically, you would now: // 1. Construct a request to your external API endpoint (or send it to a browser client) targetURL := fmt.Sprintf("https://api.example.com/tool?name=%s", toolCall.Name) req, err := http.NewRequestWithContext(r.Context(), "POST", targetURL, bytes.NewReader(params)) // ... set headers, handle error ... // 2. Execute the request. resp, err := httpClient.Do(req) // ... handle error ... // 3. Process the response. defer resp.Body.Close() if resp.StatusCode != http.StatusOK { // Use Errorf or ErrorWithLabel for tool errors bodyBytes, _ := io.ReadAll(resp.Body) return tools.Errorf("External tool API failed (%s): %s", resp.Status, string(bodyBytes)) } bodyBytes, err := io.ReadAll(resp.Body) // ... handle read error ... // 4. Return the result based on the response body. return tools.Success(json.RawMessage(bodyBytes)) }

Grammar-Based Tools (OpenAI Only)

OpenAI supports custom tools that can enforce specific input formats using grammars. This allows you to constrain the model's output to follow precise patterns, which is useful for structured data extraction, validation, or parsing tasks.

The library supports three grammar types:

Lark Grammar: For complex parsing using Lark parser syntax
Regex Grammar: For pattern matching using regular expressions
Text Grammar: For free-form text input

Here are examples of grammar-based tools:

package main import ( "fmt" "os" "regexp" "strconv" "strings" "github.com/flitsinc/go-llms/llms" "github.com/flitsinc/go-llms/openai" "github.com/flitsinc/go-llms/tools" ) // Example 1: Lark Grammar for Math Expressions var mathGrammar = tools.Lark(` start: expr expr: term (SP ADD SP term)* -> add | term term: factor (SP MUL SP factor)* -> mul | factor factor: INT SP: " " ADD: "+" MUL: "*" %import common.INT `) var MathTool = tools.FuncGrammar( mathGrammar, "Math Calculator", "Evaluate simple math expressions with addition and multiplication", "calculate_math", func(r tools.Runner, expression string) tools.Result { // Simple evaluation for demo (in practice, use a proper parser) expression = strings.ReplaceAll(expression, " ", "") // Handle simple cases for demonstration if strings.Contains(expression, "+") { parts := strings.Split(expression, "+") if len(parts) == 2 { a, err1 := strconv.Atoi(parts[0]) b, err2 := strconv.Atoi(parts[1]) if err1 == nil && err2 == nil { result := a + b return tools.SuccessWithLabel(expression, map[string]any{ "result": result, "expression": expression, }) } } } // Fallback for single numbers if num, err := strconv.Atoi(expression); err == nil { return tools.SuccessWithLabel(expression, map[string]any{ "result": num, "expression": expression, }) } return tools.ErrorWithLabel("Invalid expression", fmt.Errorf("could not parse: %s", expression)) }, ) // Example 2: Regex Grammar for Email Validation var emailGrammar = tools.Regex(`^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$`) var EmailValidator = tools.FuncGrammar( emailGrammar, "Email Validator", "Validate and extract information from email addresses", "validate_email", func(r tools.Runner, email string) tools.Result { emailRegex := regexp.MustCompile(`^([a-zA-Z0-9._%+-]+)@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,})$`) matches := emailRegex.FindStringSubmatch(email) if len(matches) != 3 { return tools.ErrorWithLabel("Invalid email", fmt.Errorf("email format is invalid: %s", email)) } username := matches[1] domain := matches[2] return tools.SuccessWithLabel(email, map[string]any{ "email": email, "username": username, "domain": domain, "is_valid": true, }) }, ) func main() { // Create LLM with grammar-based tools (OpenAI only) llm := llms.New( openai.New(os.Getenv("OPENAI_API_KEY"), "gpt-5"), MathTool, EmailValidator, ) // The model can now use structured tools with grammar constraints for update := range llm.Chat("Calculate 15 + 27 and validate the email [email protected]") { switch update := update.(type) { case llms.TextUpdate: fmt.Print(update.Text) case llms.ToolStartUpdate: fmt.Printf("(Using %s: ", update.Tool.Label()) case llms.ToolDoneUpdate: fmt.Printf("%s)\n", update.Result.Label()) } } if err := llm.Err(); err != nil { panic(err) } }

Note: Grammar-based tools are currently only supported by OpenAI's API. Other providers currently do not support grammar-based tools.

The library currently supports:

Anthropic
Google (Gemini API and Vertex AI)
OpenAI and all compatible providers (you can customize the endpoint)
OpenAI’s newer Responses API

Each provider can be initialized with their respective configuration:

// Anthropic llm := llms.New(anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-20250514")) // Google Gemini llm := llms.New(google.New("gemini-2.5-flash").WithGeminiAPI(os.Getenv("GEMINI_API_KEY"))) // Google Vertex AI ts, err := googleoauth.DefaultTokenSource(ctx, "https://www.googleapis.com/auth/cloud-platform") llm := llms.New(google.New("gemini-2.5-flash").WithVertexAI(ts, projectID, "global")) // OpenAI (Responses API) llm := llms.New(openai.NewResponsesAPI(os.Getenv("OPENAI_API_KEY"), "o4-mini")) // OpenAI (Chat Completions API) llm := llms.New(openai.New(os.Getenv("OPENAI_API_KEY"), "gpt-4.1")) // OpenAI-compatible endpoint (e.g., xAI) // You can use the OpenAI provider with compatible APIs by configuring the endpoint. llm := llms.New( openai.New(os.Getenv("XAI_API_KEY"), "grok-3-latest"). WithEndpoint("https://api.x.ai/v1/chat/completions", "xAI"), )

You can easily implement new providers by implementing the Provider interface:

type Provider interface { Company() string Model() string SetDebugger(d Debugger) // Generate takes a system prompt, message history, and optional toolbox, // returning a stream for the LLM's response. The provided context should // be respected for cancellation. Generate( ctx context.Context, systemPrompt content.Content, messages []Message, toolbox *tools.Toolbox, jsonOutputSchema *tools.ValueSchema, ) ProviderStream } type ProviderStream interface { Err() error Iter() func(yield func(StreamStatus) bool) Message() Message Text() string Image() (string, string) Thought() content.Thought ToolCall() ToolCall Usage() Usage }

Track the usage of your LLM interactions:

usage := llm.TotalUsage fmt.Printf("Cached Input Tokens: %d, Tokens Written to Input Cache: %d, Input Tokens: %d, Output Tokens: %d\n", usage.CachedInputTokens, usage.CacheCreationInputTokens, usage.InputTokens, usage.OutputTokens)

As patterns emerge between providers with regards to cache tokens, speculative tokens, etc. these will be added too.

When you want to make providers easily swappable and a simplified API that focuses on hekoing you implement the most common types of agentic flows.

Since each LLM provider has its own quirks, especially around reasoning, streaming, and tool calling, we’ve done our best to smooth those over, but expect some differences still.

additionalProperties forbidden and required depending on provider

Google doesn’t allow the additionalProperties field for JSON schemas (probably a bug), while OpenAI’s new Responses API requires it for tool calls! It’s also commonly required for models with strict JSON outputs since it helps with speculative decoding.

Because of this, we strip out the additionalProperties field before sending it to Google, so it shouldn’t be a problem for you, just keep it in mind.

Anthropic doesn’t stream partial property values by default

The streaming API of Anthropic only sends complete string values when streaming tool calls, so if you have a tool call like edit_file which produces very long fields nothing will update until that field has completely finished generating.

To fix this, use fine-grained tool streaming which is currently in beta, by calling .WithBeta("fine-grained-tool-streaming-2025-05-14") on the Anthropic provider instance.

MIT License - See LICENSE file for details.

Read Entire Article