A quick test of ChatTTS.
Project repo: https://github.com/2noise/ChatTTS
Utilities
A conversation class that helps to fix speakers. Invoke the talk_all() to convert a list of (name, message) tuples to audio conversation.
class Conversation:
def __init__(self, chattts_model):
self.speakers = {}
self.chattts_model = chattts_model
def talk_one(self, name, message):
if name not in self.speakers:
self.speakers[name] = self.chattts_model.sample_random_speaker()
print(f'sample new speaker: {name}')
else:
print('existing speaker')
spk_emb = self.speakers[name]
params_infer_code = {'spk_emb': spk_emb}
wavs = chat.infer([message], params_infer_code=params_infer_code)
return wavs[0]
def talk_all(self, chats):
wavs = []
for name, message in chats:
print(name, message)
wav = self.talk_one(name, message)
wavs.append(wav)
return wavs
Output audio files to WAV and MP3 given a file prefix:
from scipy.io import wavfile
from pydub import AudioSegment
def output_audio(wavs, output_prefix):
output_file = f'{output_prefix}.wav'
audio_data = np.concatenate(wavs, axis=1)
sample_rate = 24_000
wavfile.write(output_file, sample_rate, audio_data[0])
audio = AudioSegment.from_wav(output_file)
audio.export(f'{output_prefix}.mp3', format='mp3')
Parse HuluNote
I use https://www.hulunote.com to log messages in wechat groups that I manage. It supports to export messages into JSON format. Some tweak is needed before passing to ChatTTS.
Functions to parse HuluNote JSON:
import re
def parse_chat_log(chat_log):
pattern = r"^(.*?):: (.*?)$"
matches = re.findall(pattern, chat_log, re.MULTILINE)
parsed_messages = []
for match in matches:
name = match[0]
message = match[1]
parsed_messages.append((name, message))
return parsed_messages
def refine_chat_text(line):
replace_texts = {
'[偷笑]': '[laugh]'
}
for k, v in replace_texts.items():
line = line.replace(k, v)
return line
Application
Parse HuluNote:
fn = '5-31-2024,导出葫芦笔记数据.json'
import json
raw = json.loads(open(fn).read())
len(raw)
for day in raw:
if day['title'] == '2024-05-27':
chatlog = day['children']
break
chats = []
for c in chatlog:
if ('fromusername' in c['string']) and ('tousername' in c['string']):
continue
chats.extend(parse_chat_log(c['string']))
Init model:
import torch
torch._dynamo.config.cache_size_limit = 64
torch._dynamo.config.suppress_errors = True
torch.set_float32_matmul_precision('high')
import ChatTTS
chat = ChatTTS.Chat()
chat.load_models()
Put them all together:
conv = Conversation(chat)
wavs = conv.talk_all(chats)
output_audio(wavs, 'complaints-group-consistent-speakers')
Example and Glitches
Here's one example I generated via WeChat group chat log.
There are some glitches in my initial test:
- Punctuations are not well handled. The model tries to pronounce
"(", "[", "...".
- Numbers are not well handled. Audio corresponding to texts like "100", "20%" are all garbled.
- When one piece of texts are too long, the latter portion becomes pure noise somehow. One may consider to break the text into shorter sentences.
