The Great Text to Speech Rabbit Hole

This isn't related to the Minecraft city generation project some of you have been following, but I thought this would be a good place to talk about something I've been working on for a while. If you own an apple device, you might have played with some of the text to speech(TTS) they include. Apple computers have had TTS for, literally, decades, but the most recent versions of voices are pretty good. The absolute best voice, quality wise, is one called Alex, introduced in OS X 10.5.
[1]From a practical point of view, geology and soil science are more complicated than astrophysics.
The reason I say Alex is the best is because the voice simulates taking a breath. It seems trivial, but that is a great auditory cue to listeners about how the speaker is organizing what they are going to say. If you are face to face with someone, there are even subtle facial cues during that breath that clues the listener in on what is going to be said next.

So, Alex breathes. But he, and the other voices are not exactly warm feeling. I've been putting together some weekly hour long 'story hours' and am modeling the narration roughly off of Zero Hour and the "Tokeo Rose" esthetic. Here's a clip

The voice I am using is "Moria," which was added to the possible voice choices with OS X 10.7. In the clip above, I'm adding effects to distort the voice and try and make it sound like it's coming over sketchy, old equipment, but it's still clearly an inhuman voice. A friend commented on how annoying he found it, so I started thinking about ways around that.

The easy solution would be for me to just do the voice, but I hear my voice enough. If computer voices were not good enough, maybe there was an intermediate. TTS voices prior to those introduced in OS X 10.7 allow one to change the pitch, duration, volume and other attributes for each phoneme (or at least the OS X version of phonemes).

Now, for some reason, when I think of testing audio, I think of a scene from the movie "Ten Things I Hate About You" in which Heath Ledger sings "You're just to good to be true" by Frankie Valli. Seriously, I test mics at the start of conference calls by singing a few lines.

Like me, the Alex voice is not much of a singer. We both mean well, but sort of suck. Take a listen:



In theory, it's possible to get Alex to sing, but to do so by hand would not be fun, and I was looking for an automated way to get the TTS voices to sound a bit more human. Burried in the OS X Developer tools is an application called Repeat After Me (RAM).

The way it is supposed to work is, you enter the text string you want the computer to say in the "Text" box. Click on "To Phonemes" to convert the string to OS X phonemes (Anything more than 100 phonemes will make later steps fail). Then, clicking on "Build Graph" generates a frequency graph of what that voice's speech rules indicate are reasonable guesses about how the phrase will be spoken.

RAM is not the most transparent application, and the documentation is not great, but each of the little dots in the "Tune" window are draggable. You can also add more drag points by holding Shift while clicking on the pitch line. Option+dragging a point will let you move that indvididual point. The real magic happens when you import a sound file. In this case, it's Heath Ledger singing.

I grabbed the audio of Health Ledger singing the song and pumped it through the Sound eXchange program (SoX). First, to increase the overall volume of the track, and second to automatically split the track into separate ones, based on pauses



That file was imported into RAM. Clicking the "Impose Durations, "Extract Pitch," and "Impose Pitch" buttons Makes RAM tray and make your typed text match the durraction of the audio file, and uses the pitch profile of the audio file as the basis for the pitch of the typed text. Lastly, you click on "Tune", and the end product is a series of text that the pre OS X 10.7 voices can use as the basis of how to speak.

According to RAM, to get the voice Alex to sing the phrase "You're just too good to be true" like Heath Ledger, you need the following:

[[inpt TUNE]]
~
y {D 50; P 265.0:0 162.0:41}
2AO {D 170; P 183.0:0 200.0:44}
r {D 20; P 239.0:0 245.0:55}
_
J {D 110; P 245.0:0 220.0:6 220.0:33 220.0:67}
1UX {D 120; P 227.0:0 234.0:35 237.0:85}
s {D 120; P 218.0:0 218.0:29}
t {D 20; P 218.0:0 218.0:13 218.0:63}
_
t {D 70; P 218.0:0 218.0:6 218.0:19 218.0:25 218.0:38 218.0:69}
1UW {D 220; P 232.0:0 229.0:4 227.0:37 227.0:44 229.0:74 229.0:81}
_
g {D 100; P 225.0:0 225.0:8 225.0:42}
1UH {D 150; P 237.0:0 237.0:4 237.0:11 242.0:19 245.0:26 259.0:70 259.0:85}
d {D 140; P 259.0:0 259.0:20 259.0:40}
~
t {D 120; P 259.0:0 259.0:67}
AX {D 160; P 227.0:0 227.0:5 229.0:30 227.0:75}
_
b {D 60; P 199.0:0 199.0:8 199.0:38 190.0:69}
1IY {D 170; P 195.0:0 195.0:7 198.0:37 198.0:44 200.0:52 300.0:56 400.0:85}
_
t {D 210; P 399.0:0 398.0:5 396.0:21 394.0:53}
r {D 30; P 222.0:0 222.0:5 222.0:20 222.0:35 222.0:45 222.0:75}
1UW {D 550; P 222.0:0 222.0:2 222.0:29 234.0:100}
. {D 460}
[[inpt TEXT]]

Which sounds like:



This is far from a quick process, and there's no similar command line tools that I am aware of. Enter AppleScript, System Events, GUI Scripting, and Python.

RAM is on a scriptable application in that it is not made to allow AppleScript to easily control it. You can control applications using System Events and GUI scripting though. The downside of doing it this way is that the computer is following a script of actions in a very braindead way, and it's impossible to use the computer for anything else, since the script is mimicking user input going to the application. So, it is doable, but feels like a Rube Goldberg machine.

I wrote a Python backend that takes arguments on the name of a file with dialog to be spoken (in this case, one line of text for each audio file made by SoX), an output file name, and the computer voice to use. The Python script does most of the housekeeping and then passes info to a seperate applescript that automates all of the steps I outlined above (plus saving the resulting RAM file in case you want to edit it later). For each short audio file, the Python script get back the tunes phoneme text, and joins them together making a single string of text representing the song. It sends the string to the system and saves the resulting audio to a file, effectively providing an automated workflow for taking recorded audio and a transcript and producing a computer generated audio recording that captures some of the human reader's nuances.

Now, RAM doesn't always do a good job in getting the durations or imposing pitch. Take for example this:

It's clear that RAM failed to get the durations correct, and the tuning of the frequencies could be better at the start. A quick manual change results in:

When you just let the scripts run without any interference, a wholly automatically generated file sounds like:



But if you reopen the saved RAM files and manually adust the durations a bit, you get something closer to the original. Below are a few samples of the whole process sung by the Victoria (introduced in OS X 10.0), Vicki (introduced in OS X 10.4), and Alex (introduced in OS X 10.6). It is worth noting that even older voices sound passable. Minor hand tweeking of durations and pitch results in a robotic Heath Ledger, presented with his original version for comparision.
Victoria (OS X 10.0):

Vicki (OS X 10.4):

Alex (OS X 10.6):

Heath Ledger:

Timesink part 13: Not dead

Been a while since I posted anything, so I thought I would say everything is still going.

Minecraft has gone through a number of updates, and as a consequence I decided to roll a new world. As this is version 2 of the idea, I realized the number of cities I had created was far too many. 30+ abandoned cities might be great on a server with a hundred or more players, but the server I admin has under 20 players, and there are rarely more than five people on at a time. For that reason, I have lowered the total number of cities to 19.

Along the way, I have revised and extended my various scripts. Right now I have two basic ones: cityPlacer.py and mapExplorer.py

The cityPlacer is pretty straight forward: it uses method I described previously to teleport an admin around the world, leave them in a spot to allow cities to generate around them, and then moves them to their next spot. The mapExplorer script is pretty cool, though. It, too, teleports the player around, but it does so only to expose a map the player holds.

In this way I can have maps for each city without having to manually fly around. An interesting consequence of this is if the rate of teleporting is high enough, the maps will not fully render. This creates a really delightful ancient feel to the maps.

Being able to automate the exploration of the city centers is pretty sweet.

In upcoming posts, I'll share the new look of the teleportation gates.

Timesink part 12: Something H found

After writing some basic text introducing the player to H and their notes, and ending by saying that H was going through the gates, I needed to find someplace interesting for H to set up shop.

This is complicated slightly because the server is "live" in that there are players on it. They have figured out how to use the gates in a simple fashion and go on trips, which meant anyplace I found had to be somewhat isolated. I chose a location that is a few gate trips away from the spawn gate in the thought that the further away, the less chance of being visited. I also scouted the area looking for evidence that people had been visiting. It was a nice area: hilly, but with lots of water, which meant the architectural style is patterned after the Maya, but it also has sort of stilt houses built on water. I found a temple not too far from the gate with a house, which meant H could have a home near what he found in the temple.

 So, what did H find? A sort of map room showing a global view of where cities are in the world. The interior of the temple was hollow, which gave me lots of floor space, so I started by making a series of concentric circles. From the top of the temple I took a screenshot and overlaid it with a spiral plot of city locations (discussed inpost 10).

Concentric circles with the city-spiral map overlay

It is hard to tell in the above image, but the actual city locations are small blue circles. In each of these locations I placed a redstone lamp and started wiring each spiral arm up. The end result, as seen from the top, is pretty darned good.

Map as seen from above.

The map isn't really meant to be seen from straight above, though. It's meant to be viewed from a special area where the player steps on pressure plates. Each plate lights up a different spiral arm and plays a different musical note. Hopefully the players will start to figure out the musical connection to the gate addresses. If not, H will lay it all out in one of his notes.

Since the map is really meant to be seen in action, how about a video? 

Maybe more notes from H next time.

Timesink: Redstone interlude

I've been working on an abandoned building that H finds which has a map of the cities on the floor in the form of redstone lamps.

Since I'm still working on it, I'll share this:



More soon.

Timesink part 11: "H"

When I started planning all this, I knew I wanted there to be a series of notes left for the player. There's a delightful bukkit plugin called Courier which let's one use maps as notes. It's really very neat. It might be outdated when writable books are introduced in the next Minecraft release, but the maps are great for now.

The notes are left by a character the player never sees, and the notes are only signed "H". "H" is someone that awoke in the strange world of minecraft, and started keeping a journal. It is meant to act as both story facilitator and tutorial. God, Minecraft needs some sort of tutorial. Hey Mojang: I have an idea on how to provide a tutorial that seasoned played would approve of. Drop me a line, huh?

The more astute reader that happens to play Minecraft might be able to guess who "H" is, in the end. But let's not drop that spoiler just yet. Suffice it to say that things don't go well for "H", and they don't make it home again. They're trapped forever. (For those that have figured it out: wouldn't a tutorial as told by H be a great tie in?)

Ok. Here are some drafts of the notes written by "H". Maybe too many? The idea will be to place them in various places "H" will have stayed. So, once the player is on the trail of "H", they can follow and find more notes. In time, these notes should help the player learn how to correctly operate the gate network, build their own gates from scratch, and eventually explain what happened to these cities and, what became of "H".

If you are reading this note, you have somehow found your way to this...place. I have been here for a few days and still don't know where I am or even how I arrived here. I went to bed and when I awoke, I was standing in the center of a stone ring. Perhaps the same happened to you.

Luckily I was able to find this nearby village. In the time I have been here the locals have not uttered a single word. I wonder if they can talk.... I have tried to get information about this place using gestures, but they either don't understand, or don't care.

Or maybe they they know the answers themselves.
-H
===================
I've come to the conclusion that the natives did not construct the stone circles. I don't think they built this large building--I think of it as "the storehouse" because it has so many chests--either. The architecture is too different from the other buildings here. The same goes for the market square with the colourful stalls and small fountain. But if they didn't build the stone rings and the other, obviously different structures, who did? It's almost like this place was built as a place to trade with the natives. But where are the builders, and can they help me get home. 

I'm settling into my bedroom, and least I am comfortable.
-H
===================
What is this horrible place! When I arrived, there was a new moon in the sky. Over the past several days I have noticed the villages become extremely agitated at night, rushing into their homes and refusing to emerge until the morning. I have heard some things in the night, but didn't think anything of them. 

But tonight is the new moon and the might is full of horrors! There are things that can not be in the dark. Half rotten beings assault the doors in the village. I saw one zombie succeed in entering a home and the inhabitance.... Skeletons--SKELETONS--walk the streets, firing arrows at anyone unwise enough to go out. Green...things silently lurk in the dark and giant spiders roam the landscape. 

Through the town giant iron men patrol the streets, battling the night creatures. These robots, these golems of iron attack everything up the green creatures. 

I have fled the comforts and warmth of the guest rooms and moved into this dank cellar. 

Tomorrow I will do what I can to hide this place. And I must have weapons.
-H
=================
The new moon has passed and the diversity of creatures in the night is decreasing. Maybe with the full moon the creatures will be gone. 

I have fashioned a simple pickaxe and have begun looking for iron. Coal is plentiful, so I hope I will be able to build a simple furnace for smelting iron. With metals comes machined tools.
-H
================
If you had told me I would be able to work diamond into tools, I would have doubted you. But here.... What is this place?
-H
===============
The creatures are back, but I feel safe in my subterranean home. I'm still very frightened by the creatures in the night, and dread to think about the villagers, but I also know that the strange iron golem will protect them. 

I have made a curious discovery. There are strange red stones found deep un the earth. I initially tried to mine them using my iron tools, but they disintegrated. My bizarre diamond pickaxe was able to extract them however. The ore, when struck, glows with a light of its own. My initial fear was that they were radioactive, but I do not believe so, now. Whatever these red stones are, they can be used as an inextinguishable source of light. One can also draw on surfaces, like chalk, and the red stone dust will convey power certain distances. 

This gives me ideas on how to better hide my home.
-H
================
I have turned my attentions to the stone circle, trying to understand its purpose. It's clear that the upright ring is composed mainly of obsidian, with a single block of lapis lazuli in the base's center. The glowing block is yet another mystery, as is the sign. Domanvisce. Is that the name of this place? Maybe some warning? When I first arrived I did not notice the water bucket that hovers. I can see it, yet can not touch it. Another mystery. This place is maddening. 

There is a button, and when I press it, an apparatus lights up, but that is all. Could the ring be a mechanism of some sort?
-H
================
While exploring, I came across a statue of a sphinx to the southeast of the village. I am embarrassed I had not seen it before. The black sphinx rests upon a huge sandstone base. I am slightly embarrassed to say I began tunneling into the base. I can't really say what for. I was driven by the stories about the great treasures of Egypt. 

I immediately broke through into a large chamber. There were more of the strange hovering items, and a chest full of redstone, signs, and obsidian. I was this a storehouse? I claimed the redstone but left the rest behind and bricked up the opening. I do not know if the villages would dare loot from the statue, and I feared they might become angry if they saw the damage I had caused.
-H
================
I have discovered an entrance to the mechanism of the ring! I believe it is a machine. There are a series of red stone wires drawn onto the floor, leading to and from a number of devices, eventually ending at a second sign, repeating the same word: domanvisce. It's my hope that I'll be able to repair the device, but I need to mine for more of the redstone; I used all I had found for torches.
-H
================
An astounding discovery! While mining, I came across a small vein of lapis lazuli. After I had extracted it, I was cleaning one of the shards, accidentally hit it against a stone, and suddenly found myself standing at the stone ring. I thought I had gone mad (madder?) but upon making the motions like I was going set down a block, or make a redstone wire, I was again transported to the stone circle. No matter where I have gone, the lapis lazuli fragment will bring me back to the ring. Quite a handy thing to remember. I swear I will never set out without at least one fragment.
-H
================
Repairing the ring's mechanism was simple in the end. There was a single wire which had broken. Now the strange current flows from the button and produces musical notes from the mechanisms. Thus far, I have only activated the machine from the interior. It grew too late in the day for me to stay out and try the mechanism. Tomorrow I will see what the ring does.
-H
================
Thank god I always carry a piece of lapis lazuli with me, now. If I hadn't I fear I would be lost! Well, more lost than I am just being in this land. 

I activated the gate and it was filled with beautiful, clear water, but it did not cascade down. It remained within the confines of the ring. Another wonder. I stepped forward and touched the water and suddenly found myself in another place, at another gate! All about the outside, objects floated: earth, a water bucket, a torch, a feather, plant, egg, and a piece of redstone. Another sign hung, visible from the gate--Derclocdoman--and surrounding me were numerous buildings, all in a style unlike what I had grown accustomed. Unlike in Domanvisce, I could find no villagers.  

Hoping the gate mechanism worked, I pressed the button, the gate filled with water, and again I was transported. But not back to where I had come from; to a new land, with new structures. Each new place had a structure of obsidian, but few could be called rings. Each took me to a new place--Derclocanmith, Derclocoplandi, Dercloaer, Derclocovisce, Derclocotine--and then back to Derclocodoman. There didn't seem to be any way to return to the place I started, Domanvisce. Then I remembered my lapis shard and returned...if not home, than the place I have come to regard as safe. 

My mind is too awhirl to try and understand how these gates work. I feel certain there must be a way to return to Domanvisce using the gates.
-H
=================
I have taken numerous travels through the gates, and have explored the 'Derclocs'. I have seen massive castles. Strange homes on stilts, floating on water. Ancient temples. But no people. In some places I have seen iron golem patrolling, but what happened to the people they protected? 

As much as I fear doing so, I should travel though the gates for an extended time. If I am going to understand the gates and how to use them, I have to actually use them. 

If you have found these notes, then seek me out through the gates. I will leave redstone torches as a sign.
-H
=================

Timesink part 10: A working gate network

I have been spending far too long thinking about the gates and entire gate system. I finally just sat down and made myself start implementing things. So let's look at some sketches:

Once I drew arms onto the plot, everything clicked as to how I could set up the network. I have something like 40 more gate to place and setup. Let me just pimp the bukkit plugin Gates. I mentioned it before, but it really is worth another (short) plug. If you run a bukkit server and are looking for a gate-based teleportation plugin, you really need to check it out. The biggest downside with it is the inability to design gates of custom shapes like you can with some of the other gate-based teleportation plugins. But the use of a chest to enter a gate address really makes this one shine for me.

One can get around the custom shape limitations a bit with some creative use of the WorldEdit plugin tools. Once the gate has been created and correctly activated, one can change the type of blocks or even remove them using WorldEdit tools and the gate will still work. On the server I'm running, lapis lazuli has special teleportation-related abilities, so the center point of the gate is a lapis lazuli block. I also remove portions of the outer ring so each arm of the spiral has a specific gate shape.

You may note that each gate has distinctive markings on the ground in front of the gate as well as items to the sides leading up to the gate. These have to do with which arm the gate is in, and also encodes information about the address of the gate you are at. Hopefully other things found on the server will make it clear how the addresses work.

With each gate, you can either enter a destination address, or just hit the button, 

which will take you to that gates pre-set autodial location. Under each gate site are a series of music blocks and redstone circuits, so before the gate activates, a series of music notes play. 

The sequence of the notes, again, conveys information. Specifically, it can tell the user how far out on a spiral arm they are, as well as gives hints about relating alchemical-based items (earth, fire, water, etc) to numbers.

To wrap everything up, here are some shots of the various "cities" that can be found around six of the gates. In some images, iron golems can be seen patrolling the areas around gates. Remember: the placement of all of the buildings was done automatically (as described previously). The placement of the gates and the size of the gate areas were done manually and are meant to (again) give information to the player about the gate networks and where other cities are.

Once I finish placing the gates (I try and do at least 6/night), I will start writing the notes as left by the unseen "H". Next time, who "H" is, what he learned, and what became of him.

Time sink: Interlude

I have not abandoned things, but testing the server setup is going slower than I would like. For the time being, enjoy something from the interwebs.