• Home
  • VR project application plan

VR project application plan

http://acl.ldc.upenn.edu/W/W06/W06-3510.pdfSimon, Staffan


Compositionality for multidimensional semantics

Design a framework combining statistical and logical aspects of meaning. Compositionaltity is well-developed for the latter but not for the former.

+ philosophical examples
-> not just intersection

solutions: transposition / normalisation (Gärdenfors), or subclassifiers

Steels Fluid CG? Have they done learning of classifiers at the same time as having a compositional language

Will interact with learning; compositionality makes learning more complicated.

Implementing Semantic Coordination in ISU framework

Implement the theory of semantic coordination developed in the earlier project (Semantic Coordination in Dialogue, RJXXX), and its extensions in the present project. Use Maharani (...) as platform.
[Denna implementation kommer att göras med hjälp av den open source-plattform som CLT och Talkamatic gemensamt kommer att släppa under våren 2012.]

Machine learning för multidimensional (incl perceptual) semantics.

Different statistical models will be implemented and evaluated to investigate the generality of the idea of perceptual meanings as classifiers of perceptual input.

  • generalise to any classifier (conditional random fields, unsupervised Bayesian learning, ...)
  • problem is more complex when compositional meanings are involved
  • argue that feedback (incl corrective) can help speed up learning
  • learning is faster for symbolic meaning, e.g. ontological, compositional (but need probabilistic model a la Steedman to allow for initial errors)
  • weighted TTR for probabilistic inference
  • clustering/lazy learning initially, moving to generalisation learning when sufficient structure has been learned
  • dialogue helps in reducing learning space, e.g. parallel syntax for learning compositional & ontological meaning
  • similarity measures over TTR structures; "similar objects have similar similarity metrics" (Tenenbaum); get help from ontological knowledge; irrelevant parameters already filtered out when learning concepts
  • use ontological info to learn kind of classifier; interaction between ontological meaning and perceptual meaning; dialogue can help learning of perceptual meaning directly or indirectly via ontological meaning (ontological category provides suitable dimensions of classification)

- overall question: how much does "the dialog factor" reduce the need for large amounts of data

ewan klein instruction based learning; decomposing actions into known actions

Parsing into  TTR
  * CCG parser (Clark & Curran) for open-domain
  * GF
  * Steve Pulman; partly statistical, output feature structures, pronoun resolution
  * Robin's OZ implementation

start from CCG grammar, collect corpus, add semantic rules to CCG grammar for sentences in corpus
for new input convered by grammar but not by semantic rules, need to learn semantics

need examples of interactions!

examples from papers
left or right game, extended
3d space with points of clouds (SLAM) - Kinect?

distinguish where we are going vs. what we will do in the project

existing classifiers off the shelf? with APIs
SIFT features

talk about pictures? parts of bicycle, relations beween parts...
Companions project, talk about pictures, who's on the picture
"this is my holiday in spain" -> wikipedia: spain -> madrid -> "Did you go to madrid?" "No, but Barcelona"; use geotags etc (metadata)

Lapata: generate contextually appropriate captions for pictures

- share resources w other users; repository (SkyNet...)

Avoid "machine child" which needs to be taught everything; we aim to do something useful instead

APPLICATION & EVALUATION (not in this project)

Connecting to robots (or something else)


or connect to the internet?

Mirella Lapata (Edinburgh) learning from pictures


1. simulation
2. camera, other sensors (Kinect?)
3. robot with sensors, moves around

1. CCG grammar
2. CCG + wordnet
... + picture databases etc
3. + internet (speculative)

Exploiting digitial knowledge

System should be allowed to use any info it has access to (wikipedia, etc) to learn new words, e.g. to generate clarifications or guess; check if a definition found on the web is acceptable to the user.

Domain theories from text (Maria Liakata);
sell -> own (enthymemes); ILP generates enthymemes?



Use external knowledge?

Simplistic robots & sensors, makes dialogue more interesting?

Game engine instead? or driving simulator?


Budget kommer att vara en heltid, troligen fördelad på 2 disputerade forskare med 50% var, varav en är jag själv.

lönekostnader 37.800 kr/mån * 12 mån * 3 år * 1,52 LOP = 2.068 kSEK
resor etc c:a 60 ksek
total projektbudget exk OH blir då ungefär 2,1 mSEK (löneuppräkning ej inräknat)



Wyat et al. 2010. Self-understanding and self-extension: a systems and representational approach. http://www.cs.bham.ac.uk/~nah/bibtex/papers/wyattetal2010tamd.pdf An example of machine learning method where model is learned incrementally from observation. Instance based learning: cf. Chapter 8 of Mitchell Machine learning book.

Skočaj et al. 2011. A system for interactive learning in dialogue with a tutor. http://cogx.eu/publications/skocajIROS11/

Stephen Pulman and Maria Liakata 2003. Learning domain theories. RANLP 2003. www.cs.ox.ac.uk/files/235/ranlp03.pdf

Maria Likata and Stephen Pulman. 2004. Learning theories from text.
http://dl.acm.org/ft_gateway.cfm?id=1220382&type=pdf&CFID=68054766&CFTOKEN=16882264 COLING '04 Proceedings of the 20th international conference on Computational Linguistics

James R. Curran, Stephen Clark, and Johan Bos (2007): Linguistically Motivated Large-Scale NLP with C&C and Boxer. Proceedings of the ACL 2007 Demonstrations Session (ACL-07 demo), pp.33-36. http://aclweb.org/anthology-new/P/P07/P07-2009.pdf http://aclweb.org/anthology-new/P/P07/P07-2009.bib

Klaus-Peter Gapp: A Computational Model of the Basic Meanings of Graded Composite Spatial Relations in 3D Space. AGDM 1994: 66- ftp://ftp.cs.uni-sb.de/pub/papers/SFB314/b111.ps.gz

Other papers by Gapp http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/g/Gapp:Klaus=Peter.html

Luc Steels and Joachim De Beule (2006) A (very) Brief Introduction to Fluid Construction Grammar Third International Workshop on Scalable Natural Language Understanding (ScaNaLU 2006) June 8, 2006, following HLT/NAACL, New York City http://acl.ldc.upenn.edu/W/W06/W06-3510.pdf



To the top

Page updated: 2012-02-28 14:55

Send as email
Print page
Show as pdf