Blog Assignment 3: Dialog Evaluation
We'll be using Rick Alterman's Blogging tool for this assignment. Stay tuned for details on how to use it.
Part 1: Read and blog: Due Friday December 9th
- Read/skim the papers below
- Select one to read carefully (or use these as a jumping off point to find a diferent paper on the topic)
- Write a short summary and commentary on the paper and post it on the blog
- Commment on other posts
Part 2: Come up with your own ideas: Due BY 1 PM on December 12th
- Come up with some ideas about evaluating dialog systems (can be based on the reading or other encounters with evaluation practices)
- Consider factors such as how broad it can/should be, what work needs to be done in advance (e.g. annotation), does it encourage research in a useful direction, etc.
- Create 1-2 slides summarizing your ideas and email them to me BY 1 PM on Monday December 12th. (if you don't have powerpoint, just email me the bullets and I'll put them in a slide. I'm going to combine them all into one deck. Put your name on the footer of the slide(s).
- We will go around the room and you will each talk through your slide(s) and we'll use that as a jumping off point for coming up with one or more concrete plans for evaluating dialog systems.
Papers
- Subject-based Evaluation Measures for Interactive Spoken Language Systems, Price, Hirschman, Shriberg, Wade, 1992
- PARADISE: A Framework for Evaluating Spoken Dialogue Agents, Walker, Litman, Kamm, Albella, 1997
- Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems, Walker, Passonneau, Boland, 2001
- Galaxy-II as an Architecture for Spoken Dialogue Evaluation, Polifroni & Seneff, 2000
- Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems, Schatzmann, Georgila, Young, 2005
Blog Assignment 2: Development environment review
Due September 26. Submit via Latte.
Updated due date: 1 review due by Monday 9/25 and the second due Wednesday 9/27.
The goal is for each toolset to have at least 2 reviewers. Do the review independently (even if somoeone else has already posted a review of that tool. Find something new to say). If you have a strong opinion of what you want to review, email me my COB today, September 19th. After that, check here to see what's open and email me your choices.
Submit reviews for two of the development environments available through the AVIOS student contest. Consider elements such as the following:
- What part of the problem does the toolset solve?
- Can you build a full speech application?
- On what platforms?
- How do you specify the dialog/control structure?
- How do you define grammars, prompts?
- What are its stated advantages?
- Do you see any disadvantages?
- What questions are unanswered?
- NOTE: You are not expected to fully test the toolset, just evaluate what they present, including any demos
| Development tool | Reviewer 1 |
|---|---|
| Angel | John, Sophie |
| AT&T | Pan, Te, Olga, Nick |
| Cepstral | |
| Android | Russel, Benny, Theo, Esther |
| Olympus | Pan, Ricky |
| Openstream | Theo, Mike |
| Nuance | |
| Voxeo | John, Esther, Sophie, Olga, |
| Wami | Michael, Ricky, Mike, Nick |
| iSpeech | Michael, Te, Russel, Benney |
Blog Assignment 1: Speech application review
Due September 12. Submit via Latte.
- Select a speech application and try it out
- Identify what worked and what didn't, how easy it was to use, how useful it was.
- Write a short review describing the application (functionality, platform) and how well it works (usefulness, limitations) and post it to the class blog under "applications.
- Read the reviews from other classmates and comment on how it compares to the app your reviewed, whether you've used anything similar, or ask for more information about the app or its performance.