Annotation Results

  • Map Task Dialog d204
  • Verbmobil Dialog r148c
  • TRIPS Dialog 971202.1238
                    VERBMOBIL    TRIPS        MAPTASK
                    K  PA  PE    K  PA  PE    K  PA  PE
      answer      .37 .78 .64  .59 .91 .78  .53 .86 .70
      info-req    .58 .85 .65  .62 .88 .68  .62 .90 .75
      IOL         .35 .66 .48  .59 .8  .51    -   -   -
      IOS         .42 .68 .44  .27 .75 .66    -   -   -
      statemnt    .34 .67 .5   .32 .7  .56  .55 .75 .46
      agree       .46 .71 .47  .52 .77 .52  .21 .72 .65
      under       .15 .80 .76  .4  .77 .62  .29 .72 .61
      info-level  .49 .87 .74  .55 .87 .71  .41 .89 .82
      OFF         .26 .94 .92    -   -   -    -   -   -
      
    Abbreviations used: IOL for Influence-on-listener, IOS for Influence-on-speaker, and OFF for Other-forward-function.

    Discussion

    email mcore@cs.rochester.edu to add your comments.

    mcore's comments:

    Influence on Speaker

  • Kappas: Verbmobil > Trips
  • Disagreements about action-directives lead to disagreements about commits. When the action is joint, an utterance will be labeled as commit if it is an action-directive.
  • Similarly disagreements on open-option caused trouble with labeling offers
      TRIPS 
        42 u: delta bridge is out
      * 43 s: the highlighted actions may need to be revised
      
      VERBMOBIL
        25 b: are you available just before noon
      * 26 a: we can meet at noon
      
        28 b: on campus or off
        29 A: your choice
      * 30 B: I say if I have got enough money to go to Acapulco
               I have got enough money to go to one of those silly places on Craig street
      
  • Disagreements about what is an accept (of a directive) lead to disagreements on what is a commit

      VERBMOBIL
       20 B: I think the first day that is really good for me
       21 B:   is the eighteenth that is a Tuesday
      *22 A: okay
      
      TRIPS
       48 u: send truck two 
       49     along the coast instead
      *50 s: done
      

    Agreement

  • Kappas: Trips > Verbmobil > Maptask
  • Is a response to a speech recognition error a hold?
      TRIPS
      21 U: what if we went along coast instead
      22 S: I'm sorry but that input didn't make sense to me.
      
      60 U: WHERE ARE THE PEOPLE
      61 U: LEFT [LEFT mistakenly inserted]
      62 S: I don't understand your reference to the people left
      
      45 U: SEND_A [A mistakenly inserted] TRUCK TWO
      46 U: ALONG THE COAST INSTEAD
      47 S: Sorry, I don't know how to do that.
      
  • Some people label answers as accepts/rejects
      TRIPS
      
      23 u: what if we went along the coast instead
      24 s: that option would take 8 hours and 52 minutes
      
      VERBMOBIL
      25 b: are you available just before noon
      26 a: we can meet at noon
      
      MAPTASK
      11 G: Do you have any landmarks just below
      12 F: uh no
      
  • Other people labeled implicit accepts (accepting by not rejecting and instead making a request or statement).
  • Reject v. Reject-part
      VERBMOBIL
      3 a: how 'bout
      4 a: how 'bout next week
      5 a: what day are good for you
      6 a: what days are good for you
      7 b: actually next week I am on vacation
      
  • What is a hold?
      MAPTASK
       18 F: and an iron bridge
      *19 G: an iron bridge
      
      VERBMOBIL
       10 a: how 'bout Monday
       11 b: Monday the tenth
       12 a: uh-huh 
      

    Influence on listener

  • Kappas: Trips > Verbmobil
  • Statement v. directive (some utterances are both but people couldn't agree on which ones)
      verbmobil
       25 b: are you available just before noon
      *26 a: we can meet at noon
      
       28 b: on campus or off?
       29 a: your choice
      *30 b: I say if I have got enought money to go to ...
      
      trips
      8  s: I'm afraid I didn't understand you
      
      42 u: delta bridge is out
      43 s: the highlighted actions may need to be revised
      
  • Action-directive v. Info-request (again some utterances are both but which ones)
      TRIPS
      21 u: what if we went along coast instead
      
      63 u: where are the people
      
      Verbmobil
      26 a: we can meet at noon
      
      29 a: your choice
      
  • The difference between open-option and action-directive was difficult for people to see.
      VERBMOBIL
      2 a: we should meet again
      3    how 'bout
      4    how 'bout next week
      
      9 a: I guess we will have to meet the week after that
      
      34 a: let us just wander up Craig
      35    and pick one we like that day
      

    Statement

  • Kappas: Maptask > Verbmobil > Trips
  • See Influence on listener
  • Surface Form of Accept can confuse
      verbmobil
       21 b: is the eighteenth that is a Tuesday
      *22 a: okay
       23 a: want to have lunch
      *24 b: that sounds pretty good
      
      trips
      36 u: go on to abyss
      37 s: your wish is my command
      
  • Answers usually are asserts, what is an answer?
      verbmobil
      
      28 b: on campus or off
      29 a: your choice
      
      31 b: how about Great Scott
      32 a: sounds great except they ...
      
    Answer

  • Kappas: Trips > Maptask > Verbmobil
  • See Info-request
  • Non-yes/no, wh answers are confusing
      TRIPS
      5     U: SHOW_ME A MAP OF PACIFICA 
      6     S: Ok
      
      Verbmobil
      7 a: what days are good for you
      8 b: actually next week I am on vacation
      
      28 b: on campus or off
      29 a: your choice
      
      31 b: how about Great Scott 
      32 a: sounds great except they have been out of business for a while 
      
      MapTask
      
      41 g: is 
      42    the hills?
      43 f: that's where my hills are
      
  • How far does an answer go?
      verbmobil
        10 a: how 'bout monday
        11 b: monday the tenth
        12 a: aha
      * 13 b: well unfortunately my vacation runs ...
      * 14     I was planning on being on a beach....
        15 a: well
        16    when are you getting back
      * 17 b: I get back on the 15th rest up on the 16th
      * 18     which is a Sunday and ...
      * 19     but I have a seminar all day
      * 20     I think the first dat that is really good for me
      * 21     is the 18th that is a Tuesday
      
    In TRIPS, responding to questions with "I did not understand you" also confused people.

    Info-level:

  • What is Task-management?
      TRIPS
      
      2: s: I'm ready to start
      
      24: s: that option would take 8 hours and 52 minutes
      25: u: forget it
      
      38: u: and take the people there to Delta
      39  s: alright
      40  u: that looks good
      
      73  s: simulation succeeded in 94% of 250 runs..
      
      MAPTASK
      
      6: G: just below that do you have a well
         F: No
         G: Oh
         g: great
      
  • If something was Other-forward-function it often was tagged Communication-management versus Task.
  • Tagging something an acknowledgment versus an accept meant a difference of Communication-management versus Task.
      Maptask
      22 F: In the middle of the map
      23 F: right at the top
      24 g: okay
      
  • What is an acknowledgment? (versus no understanding tag)
      Maptask
      
      5 f: up by sandy shore
      6 g: by sandy shore
      
      20 f: in the middle of the map
      21 f: right at the top
      22 g: okay
      23 g: I don't have the iron bridge
      

    Understanding:

  • Kappas: Trips > Maptask > Verbmobil
  • Accept v. only acknowledge
      TRIPS
      5: u: show me a map of pacifica
      6: s: okay
      
      13 u: use a truck get the people from Calypso
      14 u: to delta
      15 s: ok
      
      Verbmobil
      20 b: I think the 1st day that is really good for me
      21 b:  is the 18th that is a Tuesday
      22 a: okay
      
      Maptask
      59 f: Do you have that?
      60 g: I have a dead tree at the fork
      61 f: yeah
      
  • Acknowledge v. no understanding
      verbmobil
        7 b: actually next week I am on vacation
      * 8 a: gosh
        9 a: I guess we will have to meet the week after that
      
        14 b: I was planning on being on a beach in Acapulco about that point
      * 15 a: well
        16 a: when are you getting back
      
      maptask
      7 g: just below that do you have a well
      8 f: no
      9 g: oh
      
    Info-requests:

  • Kappas: TRIPS = Maptask > Verbmobil
  • Speech recognition issues
      TRIPS
      7 u: clear (should be where are) the people
      8 s: I'm afraid I didn't understand you
      9 u: where are the people
      
      16 u: it how long will a take
      17 s: I'm afraid I didn't understand you
      18 u: how long will that take
      
  • Info-request v. Action-directive (it could be both but people disagreed)
      Verbmobil
      10 a: how about Monday
      
      3 a: how 'bout
      4 a: how 'bout next week
      5 a: what day are good for you
      6 a: what days are good for you
      
      24 a: want to have lunch
      

    Last change: 5 May 98 by mcore

    Click here to send in comments or questions.

    ^^
    Back to the Dialog Annotation Page