I'm working on a project that makes use of Day's Biographical Dictionary of the the History of Technology as a source. What I wanted to do required the book's information to be a database format, though. How do you go from "dark data" to something a computer can use? First, hope there is an ebook.
In this case, there is. I was able to get a PDF of the text, but it was still a book mean to be read and understood by humans. After thinking about it, I realized the format of the book was quite excellent and would allow for automated processing.
Each entry began with the name of an inventor, and ended with the initials of the editor (or editors) that worked on that section. In between those START and STOP markers were well defined details on things like birth (day, month, year, location), death (day, month, year), etc. There was also a page at the beginning of the book that listed all the editors and their abbreviations used at the end of each entry. Enter python, but practically any language could have parsed this. Without going into a lot of detail:
Read the pdf's editor index and stored it as a list and saved it to a file.
Read in the book's index of names and stored the names as a list and saved it as a file.
Read in the body of the pdf and stored it as text.
Using the editor list and the name list, I split the main body of the text into a giant list where each entry began with the name of someone in the index and ended with the initials of one of the editors.
Of course it wasn't that easy. For example, I discovered numerous instances where a name was spelled on way in the text, and another way in the index. After numerous rounds of cleanup to account for differences in names or oddities in formatting, I ended up with data I could write to a CSV for other processing.
Tonight, out of curiosity, I sat down and looked at the distribution of birth years for all inventors in my database born in or after 1690.
I am sure there are errors in the database since I have not throughly gone over it and checked entries, but this first pass showed N=1065 individuals, with a mean birth year of 1830 (median of 1834). I was somewhat surprised at how the numbers fell off as you get into the 20th century, but I have a thought as to why that might be. The book is a biographical take on the history of technology. For that reason, it is necessarily biased toward individuals, not for the work of teams of people. For example, who invented the atomic bomb? Yes, Leo Szilard famously came up with the idea and patented it, but he certainly didn't build one in his back shed. A team built The Gadget.
Don't misunderstand me to be on the "the age of the lone inventor is dead" wagon. I'm not weighing in on that. I'm simply trying to explain what I'm seeing and guessing at the bias in this one source.
UPDATED: 15 March, 2015. Maybe I should start another page....
Have you read the blog entries at Some Kind of Wonderful Noise? It chronicles how an adult (an ADULT!), born and raised in the U.S.A. Canada (EDIT: Canada is still not an excuse) somehow slipped through the cracks and never saw movies like Star Wars, Alien, Indiana Jones, etc. He live blogs his reactions and thoughts while watching these cultural touchstones. I highly recommend starting from his very first entries and reading them in order, because you can watch his working cultural vocabulary build, and he begins to make comments you'd expect from seasoned MSTies. It's like watching an AI bootstrap it's cultural subroutine.
The reason I mention Some Kind of Wonderful Noise is because my wife and I have been intentionally showing our sons various (mainly scifi) movies to introduce them to ones that we deem important to having a working cultural vocabulary. What does it mean when someone says "light is green? Trap is clean." Well, now my sons can use it in a sentence to explain the state of something. Shaka. When the walls fell.
Numerous people have asked for a list of the movies we've shown, so here it is. I'll update this list as more movies get shown. Please note that not everything on the list is what we would consider important. Some are just fun.
-----Time Travel Theme-----
Back to the Future 1,2,3
The Time Machine (1960)
Bill and Ted's Excellent Adventure
Time Bandits
Groundhog Day
The Time Machine (2002)
Flight of the Navigator
Planet of the Apes
Run Lola Run
My science project
The girl who leapt through time (2006 cartoon)
Stargate
Close Encounters of the 3rd Kind
The Explorers
Teen Wolf
Forbidden planet
Big Trouble in Little China
The Blob (short: Yip Yip's discover a telephone)
Men in black (short: Kermit the Frog. News Flash on Pinocchio)
The Goonies
Superman (1978)
Superman 2
Iron Giant
The Frighteners
Rewatch Star Wars (ep 4,5,6 and then 1,2,3)
Beatlejuice
Escape to Witch Mountain
Return to Witch Mountain
Cat from Outer Space
The Absentminded Professor
Who Framed Roger Rabbit
Gremlins
-----Stop motion theme-----
Jason and the Argonauts
The 7th Voyage of Sinbad
Golden Voyage of Sinbad
Mighty Joe Young (1949)
Clash of the Titans
The Beast from 20,000 Fathoms
E.T.
Wargames
Labyrinth
Short Circuit
Teenage Mutant Ninja Turtles (1990)
5th Element
Batman (1989)
War of the Worlds (1953)
War of the Worlds (2005)
Independence Day
Spiderman (2002)
Spiderman 2 (2004)
Peter Pan (2003)
Spiderman 3 (2007)
Hook
Weird Science
Frankenstein (1931)
Creature from the Black Lagoon (1954)
The Wolf Man (1941)
Mothra reboot (1996)
Mothra (1961)
Popeye
Land Before Time
Weird Science
-Short: Minuscule v1e1 "Le coccinelle"
The Adventures of Baron Munchausen
-Short: Minuscule v1e2 "Catapulte"
-----Avenger's Theme-----
Iron Man
Iron Man 2
-Short: Minuscule v1e3 "Bouse de lá!"
Thor (2011)
-Short: Minuscule v1e4 "Deux chenilles"
Captain America: The First Avenger (2011)
The Avengers (2012)
Iron Man 3 (2013)
Thor 2: The Dark World (2013)
Captain America 2: Winter Soldier (2014)
Guardians of the Galaxy (2014)
The Mask (1994)
The Last Unicorn (2012)
Justice League: War (2014)
Justice League: Throne of Atlantis (2015)
-----Star Wars Theme-----
Star Wars, A New Hope
Star Wars, Empire Strikes Back
Star Wars, Return of the Jedi
Star Wars, Attack of the Clones
Star Wars, The Clone Wars (2003)
Star Wars, Revenge of the Sith
Escape from Alcatraz (1979)
Wall-E (2008)
The Rocketeer (1991)
This isn't related to the Minecraft city generation project some of you have been following, but I thought this would be a good place to talk about something I've been working on for a while. If you own an apple device, you might have played with some of the text to speech(TTS) they include. Apple computers have had TTS for, literally, decades, but the most recent versions of voices are pretty good. The absolute best voice, quality wise, is one called Alex, introduced in OS X 10.5. [1]From a practical point of view, geology and soil science are more complicated than astrophysics.
The reason I say Alex is the best is because the voice simulates taking a breath. It seems trivial, but that is a great auditory cue to listeners about how the speaker is organizing what they are going to say. If you are face to face with someone, there are even subtle facial cues during that breath that clues the listener in on what is going to be said next.
So, Alex breathes. But he, and the other voices are not exactly warm feeling. I've been putting together some weekly hour long 'story hours' and am modeling the narration roughly off of Zero Hour and the "Tokeo Rose" esthetic. Here's a clip
The voice I am using is "Moria," which was added to the possible voice choices with OS X 10.7. In the clip above, I'm adding effects to distort the voice and try and make it sound like it's coming over sketchy, old equipment, but it's still clearly an inhuman voice. A friend commented on how annoying he found it, so I started thinking about ways around that.
The easy solution would be for me to just do the voice, but I hear my voice enough. If computer voices were not good enough, maybe there was an intermediate. TTS voices prior to those introduced in OS X 10.7 allow one to change the pitch, duration, volume and other attributes for each phoneme (or at least the OS X version of phonemes).
Now, for some reason, when I think of testing audio, I think of a scene from the movie "Ten Things I Hate About You" in which Heath Ledger sings "You're just to good to be true" by Frankie Valli. Seriously, I test mics at the start of conference calls by singing a few lines.
Like me, the Alex voice is not much of a singer. We both mean well, but sort of suck. Take a listen:
In theory, it's possible to get Alex to sing, but to do so by hand would not be fun, and I was looking for an automated way to get the TTS voices to sound a bit more human. Burried in the OS X Developer tools is an application called Repeat After Me (RAM).
The way it is supposed to work is, you enter the text string you want the computer to say in the "Text" box. Click on "To Phonemes" to convert the string to OS X phonemes (Anything more than 100 phonemes will make later steps fail). Then, clicking on "Build Graph" generates a frequency graph of what that voice's speech rules indicate are reasonable guesses about how the phrase will be spoken.
RAM is not the most transparent application, and the documentation is not great, but each of the little dots in the "Tune" window are draggable. You can also add more drag points by holding Shift while clicking on the pitch line. Option+dragging a point will let you move that indvididual point. The real magic happens when you import a sound file. In this case, it's Heath Ledger singing.
I grabbed the audio of Health Ledger singing the song and pumped it through the Sound eXchange program (SoX). First, to increase the overall volume of the track, and second to automatically split the track into separate ones, based on pauses
That file was imported into RAM. Clicking the "Impose Durations, "Extract Pitch," and "Impose Pitch" buttons Makes RAM tray and make your typed text match the durraction of the audio file, and uses the pitch profile of the audio file as the basis for the pitch of the typed text. Lastly, you click on "Tune", and the end product is a series of text that the pre OS X 10.7 voices can use as the basis of how to speak.
According to RAM, to get the voice Alex to sing the phrase "You're just too good to be true" like Heath Ledger, you need the following:
[[inpt TUNE]]
~
y {D 50; P 265.0:0 162.0:41}
2AO {D 170; P 183.0:0 200.0:44}
r {D 20; P 239.0:0 245.0:55}
_
J {D 110; P 245.0:0 220.0:6 220.0:33 220.0:67}
1UX {D 120; P 227.0:0 234.0:35 237.0:85}
s {D 120; P 218.0:0 218.0:29}
t {D 20; P 218.0:0 218.0:13 218.0:63}
_
t {D 70; P 218.0:0 218.0:6 218.0:19 218.0:25 218.0:38 218.0:69}
1UW {D 220; P 232.0:0 229.0:4 227.0:37 227.0:44 229.0:74 229.0:81}
_
g {D 100; P 225.0:0 225.0:8 225.0:42}
1UH {D 150; P 237.0:0 237.0:4 237.0:11 242.0:19 245.0:26 259.0:70 259.0:85}
d {D 140; P 259.0:0 259.0:20 259.0:40}
~
t {D 120; P 259.0:0 259.0:67}
AX {D 160; P 227.0:0 227.0:5 229.0:30 227.0:75}
_
b {D 60; P 199.0:0 199.0:8 199.0:38 190.0:69}
1IY {D 170; P 195.0:0 195.0:7 198.0:37 198.0:44 200.0:52 300.0:56 400.0:85}
_
t {D 210; P 399.0:0 398.0:5 396.0:21 394.0:53}
r {D 30; P 222.0:0 222.0:5 222.0:20 222.0:35 222.0:45 222.0:75}
1UW {D 550; P 222.0:0 222.0:2 222.0:29 234.0:100}
. {D 460}
[[inpt TEXT]]
Which sounds like:
This is far from a quick process, and there's no similar command line tools that I am aware of. Enter AppleScript, System Events, GUI Scripting, and Python.
RAM is on a scriptable application in that it is not made to allow AppleScript to easily control it. You can control applications using System Events and GUI scripting though. The downside of doing it this way is that the computer is following a script of actions in a very braindead way, and it's impossible to use the computer for anything else, since the script is mimicking user input going to the application. So, it is doable, but feels like a Rube Goldberg machine.
I wrote a Python backend that takes arguments on the name of a file with dialog to be spoken (in this case, one line of text for each audio file made by SoX), an output file name, and the computer voice to use. The Python script does most of the housekeeping and then passes info to a seperate applescript that automates all of the steps I outlined above (plus saving the resulting RAM file in case you want to edit it later). For each short audio file, the Python script get back the tunes phoneme text, and joins them together making a single string of text representing the song. It sends the string to the system and saves the resulting audio to a file, effectively providing an automated workflow for taking recorded audio and a transcript and producing a computer generated audio recording that captures some of the human reader's nuances.
Now, RAM doesn't always do a good job in getting the durations or imposing pitch. Take for example this:
It's clear that RAM failed to get the durations correct, and the tuning of the frequencies could be better at the start. A quick manual change results in:
When you just let the scripts run without any interference, a wholly automatically generated file sounds like:
But if you reopen the saved RAM files and manually adust the durations a bit, you get something closer to the original. Below are a few samples of the whole process sung by the Victoria (introduced in OS X 10.0), Vicki (introduced in OS X 10.4), and Alex (introduced in OS X 10.6). It is worth noting that even older voices sound passable. Minor hand tweeking of durations and pitch results in a robotic Heath Ledger, presented with his original version for comparision.
Victoria (OS X 10.0):
Been a while since I posted anything, so I thought I would say everything is still going.
Minecraft has gone through a number of updates, and as a consequence I decided to roll a new world. As this is version 2 of the idea, I realized the number of cities I had created was far too many. 30+ abandoned cities might be great on a server with a hundred or more players, but the server I admin has under 20 players, and there are rarely more than five people on at a time. For that reason, I have lowered the total number of cities to 19.
Along the way, I have revised and extended my various scripts. Right now I have two basic ones:
cityPlacer.py and mapExplorer.py
The cityPlacer is pretty straight forward: it uses method I described previously to teleport an admin around the world, leave them in a spot to allow cities to generate around them, and then moves them to their next spot.
The mapExplorer script is pretty cool, though. It, too, teleports the player around, but it does so only to expose a map the player holds.
In this way I can have maps for each city without having to manually fly around.
An interesting consequence of this is if the rate of teleporting is high enough, the maps will not fully render. This creates a really delightful ancient feel to the maps.
Being able to automate the exploration of the city centers is pretty sweet.
In upcoming posts, I'll share the new look of the teleportation gates.
After writing some basic text introducing the player to H and their notes, and ending by saying that H was going through the gates, I needed to find someplace interesting for H to set up shop.
This is complicated slightly because the server is "live" in that there are players on it. They have figured out how to use the gates in a simple fashion and go on trips, which meant anyplace I found had to be somewhat isolated.
I chose a location that is a few gate trips away from the spawn gate in the thought that the further away, the less chance of being visited. I also scouted the area looking for evidence that people had been visiting.
It was a nice area: hilly, but with lots of water, which meant the architectural style is patterned after the Maya, but it also has sort of stilt houses built on water. I found a temple not too far from the gate with a house, which meant H could have a home near what he found in the temple.
So, what did H find? A sort of map room showing a global view of where cities are in the world. The interior of the temple was hollow, which gave me lots of floor space, so I started by making a series of concentric circles. From the top of the temple I took a screenshot and overlaid it with a spiral plot of city locations (discussed inpost 10).
Concentric circles with the city-spiral map overlay
It is hard to tell in the above image, but the actual city locations are small blue circles. In each of these locations I placed a redstone lamp and started wiring each spiral arm up. The end result, as seen from the top, is pretty darned good.
Map as seen from above.
The map isn't really meant to be seen from straight above, though. It's meant to be viewed from a special area where the player steps on pressure plates. Each plate lights up a different spiral arm and plays a different musical note. Hopefully the players will start to figure out the musical connection to the gate addresses. If not, H will lay it all out in one of his notes.
Since the map is really meant to be seen in action, how about a video?
When I started planning all this, I knew I wanted there to be a series of notes left for the player. There's a delightful bukkit plugin called Courier which let's one use maps as notes. It's really very neat. It might be outdated when writable books are introduced in the next Minecraft release, but the maps are great for now.
The notes are left by a character the player never sees, and the notes are only signed "H". "H" is someone that awoke in the strange world of minecraft, and started keeping a journal. It is meant to act as both story facilitator and tutorial. God, Minecraft needs some sort of tutorial. Hey Mojang: I have an idea on how to provide a tutorial that seasoned played would approve of. Drop me a line, huh?
The more astute reader that happens to play Minecraft might be able to guess who "H" is, in the end. But let's not drop that spoiler just yet. Suffice it to say that things don't go well for "H", and they don't make it home again. They're trapped forever. (For those that have figured it out: wouldn't a tutorial as told by H be a great tie in?)
Ok. Here are some drafts of the notes written by "H". Maybe too many? The idea will be to place them in various places "H" will have stayed. So, once the player is on the trail of "H", they can follow and find more notes. In time, these notes should help the player learn how to correctly operate the gate network, build their own gates from scratch, and eventually explain what happened to these cities and, what became of "H".
If you are reading this note, you have somehow found your way to this...place. I have been here for a few days and still don't know where I am or even how I arrived here. I went to bed and when I awoke, I was standing in the center of a stone ring. Perhaps the same happened to you.
Luckily I was able to find this nearby village. In the time I have been here the locals have not uttered a single word. I wonder if they can talk.... I have tried to get information about this place using gestures, but they either don't understand, or don't care.
Or maybe they they know the answers themselves.
-H
===================
I've come to the conclusion that the natives did not construct the stone circles. I don't think they built this large building--I think of it as "the storehouse" because it has so many chests--either. The architecture is too different from the other buildings here. The same goes for the market square with the colourful stalls and small fountain. But if they didn't build the stone rings and the other, obviously different structures, who did? It's almost like this place was built as a place to trade with the natives. But where are the builders, and can they help me get home.
I'm settling into my bedroom, and least I am comfortable.
-H
===================
What is this horrible place! When I arrived, there was a new moon in the sky. Over the past several days I have noticed the villages become extremely agitated at night, rushing into their homes and refusing to emerge until the morning. I have heard some things in the night, but didn't think anything of them.
But tonight is the new moon and the might is full of horrors! There are things that can not be in the dark. Half rotten beings assault the doors in the village. I saw one zombie succeed in entering a home and the inhabitance.... Skeletons--SKELETONS--walk the streets, firing arrows at anyone unwise enough to go out. Green...things silently lurk in the dark and giant spiders roam the landscape.
Through the town giant iron men patrol the streets, battling the night creatures. These robots, these golems of iron attack everything up the green creatures.
I have fled the comforts and warmth of the guest rooms and moved into this dank cellar.
Tomorrow I will do what I can to hide this place. And I must have weapons.
-H
=================
The new moon has passed and the diversity of creatures in the night is decreasing. Maybe with the full moon the creatures will be gone.
I have fashioned a simple pickaxe and have begun looking for iron. Coal is plentiful, so I hope I will be able to build a simple furnace for smelting iron. With metals comes machined tools.
-H
================
If you had told me I would be able to work diamond into tools, I would have doubted you. But here.... What is this place?
-H
===============
The creatures are back, but I feel safe in my subterranean home. I'm still very frightened by the creatures in the night, and dread to think about the villagers, but I also know that the strange iron golem will protect them.
I have made a curious discovery. There are strange red stones found deep un the earth. I initially tried to mine them using my iron tools, but they disintegrated. My bizarre diamond pickaxe was able to extract them however. The ore, when struck, glows with a light of its own. My initial fear was that they were radioactive, but I do not believe so, now. Whatever these red stones are, they can be used as an inextinguishable source of light. One can also draw on surfaces, like chalk, and the red stone dust will convey power certain distances.
This gives me ideas on how to better hide my home.
-H
================
I have turned my attentions to the stone circle, trying to understand its purpose. It's clear that the upright ring is composed mainly of obsidian, with a single block of lapis lazuli in the base's center. The glowing block is yet another mystery, as is the sign. Domanvisce. Is that the name of this place? Maybe some warning? When I first arrived I did not notice the water bucket that hovers. I can see it, yet can not touch it. Another mystery. This place is maddening.
There is a button, and when I press it, an apparatus lights up, but that is all. Could the ring be a mechanism of some sort?
-H
================
While exploring, I came across a statue of a sphinx to the southeast of the village. I am embarrassed I had not seen it before. The black sphinx rests upon a huge sandstone base. I am slightly embarrassed to say I began tunneling into the base. I can't really say what for. I was driven by the stories about the great treasures of Egypt.
I immediately broke through into a large chamber. There were more of the strange hovering items, and a chest full of redstone, signs, and obsidian. I was this a storehouse? I claimed the redstone but left the rest behind and bricked up the opening. I do not know if the villages would dare loot from the statue, and I feared they might become angry if they saw the damage I had caused.
-H
================
I have discovered an entrance to the mechanism of the ring! I believe it is a machine. There are a series of red stone wires drawn onto the floor, leading to and from a number of devices, eventually ending at a second sign, repeating the same word: domanvisce. It's my hope that I'll be able to repair the device, but I need to mine for more of the redstone; I used all I had found for torches.
-H
================
An astounding discovery! While mining, I came across a small vein of lapis lazuli. After I had extracted it, I was cleaning one of the shards, accidentally hit it against a stone, and suddenly found myself standing at the stone ring. I thought I had gone mad (madder?) but upon making the motions like I was going set down a block, or make a redstone wire, I was again transported to the stone circle. No matter where I have gone, the lapis lazuli fragment will bring me back to the ring. Quite a handy thing to remember. I swear I will never set out without at least one fragment.
-H
================
Repairing the ring's mechanism was simple in the end. There was a single wire which had broken. Now the strange current flows from the button and produces musical notes from the mechanisms. Thus far, I have only activated the machine from the interior. It grew too late in the day for me to stay out and try the mechanism. Tomorrow I will see what the ring does.
-H
================
Thank god I always carry a piece of lapis lazuli with me, now. If I hadn't I fear I would be lost! Well, more lost than I am just being in this land.
I activated the gate and it was filled with beautiful, clear water, but it did not cascade down. It remained within the confines of the ring. Another wonder. I stepped forward and touched the water and suddenly found myself in another place, at another gate! All about the outside, objects floated: earth, a water bucket, a torch, a feather, plant, egg, and a piece of redstone. Another sign hung, visible from the gate--Derclocdoman--and surrounding me were numerous buildings, all in a style unlike what I had grown accustomed. Unlike in Domanvisce, I could find no villagers.
Hoping the gate mechanism worked, I pressed the button, the gate filled with water, and again I was transported. But not back to where I had come from; to a new land, with new structures. Each new place had a structure of obsidian, but few could be called rings. Each took me to a new place--Derclocanmith, Derclocoplandi, Dercloaer, Derclocovisce, Derclocotine--and then back to Derclocodoman. There didn't seem to be any way to return to the place I started, Domanvisce. Then I remembered my lapis shard and returned...if not home, than the place I have come to regard as safe.
My mind is too awhirl to try and understand how these gates work. I feel certain there must be a way to return to Domanvisce using the gates.
-H
=================
I have taken numerous travels through the gates, and have explored the 'Derclocs'. I have seen massive castles. Strange homes on stilts, floating on water. Ancient temples. But no people. In some places I have seen iron golem patrolling, but what happened to the people they protected?
As much as I fear doing so, I should travel though the gates for an extended time. If I am going to understand the gates and how to use them, I have to actually use them.
If you have found these notes, then seek me out through the gates. I will leave redstone torches as a sign.
-H
=================
