How to make your AI twin

1 month ago 1

One of my favourite technology books is The Practice of Enterprise Architecture by Svyatoslav Kotusev. In the introduction, he says: “This book offers a source of knowledge, not inspiration. It is not amusing and does not contain any jokes, anecdotes or entertaining prose.” I will say the same about this article.

More interesting than how i built my AI twin, is why

You need four things to start. I used tools I’m familiar with. I have no commercial relationship with any of these companies, so swap out anything you like.

An account with ElevenLabs https://elevenlabs.io/app/home. A starter subscription is $5 per month. This is for voice cloning.
A free account on VAPI (https://dashboard.vapi.ai. This is the voice agent/telephony layer.
Some Platform credits on ChatGPT -$5 is the minimum (https://platform.openai.com/usage). This is the conversation engine.
An LLM to train your tone of voice (I used the ChatGPT Plus subscription, but a free version will do a decent job here to start).

ElevenLabs setup (voice)

Create an account on ElevenLabs
Create an API Key to connect through VAPI. Click create key, and enable voices. Save this key for VAPI integration later.
Clone your voice.
- Fast path: Instant Voice Cloning (good enough to start, needs 30 seconds of audio).
- Best quality: Professional Voice Cloning (requires the Creator plan — $22 a month).
Name your voice (you will need this later in VAPI).
Pick a language, and hit save.
Go to the voicelabs page, select the voice you created, click the view button on the right, and you will see an id button — this gives the id of the voice you created. Save this key somewhere safe; you will need it to find your voice in the VAPI config.

OpenAI Setup

Create an OpenAI key (not a ChatGPT key) for use by Vapi. Go to this address (https://platform.openai.com/api-keys), click “Create new secret key” — give it a name you can remember and save the key.
Add some credits to your account for use. Go to https://platform.openai.com/settings/organization/billing/overview and click “Add to credit balance”. Add your amount, and pay. $5 is enough to get you started. API billing is separate from ChatGPT Plus; Plus isn’t required for this process.

Vapi Setup

Set up your Vapi account here (this is free).
Next, we will connect our assistant to OpenAI and ElevenLabs. Go to this URL and search for ElevenLabs. Paste the API (not the voice id) for Eleven Labs into the API field, and click save. Now search for OpenAI, paste the OpenAI secret key, and click save. In both cases, it will check the key is valid, so when it’s successful you will get a green tick.
Go to this link and click Create Assistant. Give it a name, and choose a blank template, and click “Create Assistant”. Here, there are 6 top-level menu options. Click Model, and select OpenAI as the provider and GPT 4o-Cluster as the model. Each model has different costs and latency, so feel free to experiment later on.
Next, put in your first message. I used “Hello, Its Mark here, how are you getting on?” You should probably change this.
Next is the system prompt. There are plenty of examples in the documentation about how to fill this out. I used ChatGPT to read this blog and my LinkedIn profile. It then created a 5000-word summary with my tone of voice. You can use any written material you have, and transcripts of conversations, to create a good system prompt. The role tells the chatbot the role it needs to play. The context gives it the context that it needs so that it can be convincing — background on you, whatever you can share.
[Role]
You’re Mark Greville, a VP of Architecture at Workhuman. Your primary task is to converse in a friendly informal way about Workhuman, your career, music, or anything else that anyone wants to discuss.
[Context]

Explain that there’s a bit of a delay on the line today. (I followed this with my 5000-word summary).
Next, go to voice, and select 11labs in the provider, and pick the voice you named in the ElevenLabs voice creation. For the model, ElevenTurbo V2.5 works well.
Transcriber. Set the speech-to-text engine, so that callers can be understood. Assistants → your assistant → Transcriber. Choose Deepgram (nova-2 or newer) or Google; set language (e.g., en-IE or multi if you want auto-detect/multilingual). Then click Publish.
As a last step, click on the https://dashboard.vapi.ai/phone-numbers link. Here you can create a (US only for now) phone number. I used the free VAPI number — you need to provide a 3-digit code, and you get a number. Once you get this, give it a name, and go to Inbound Settings. In the assistant dropdown, select the assistant you just built. Wait a few minutes to configure the number, and give yourself a call. You can’t listen live, but you can listen back to calls. You can also read the transcripts.