Read these two for sure:
Johns, B. T., & Jones, M. N. (to appear). Perceptual inference from global lexical structure. Topics in Cognitive Science.
Riordan, B., & Jones, M. N. (2011). Redundancy in linguistic and perceptual experience: Comparing distributional and feature-based models of semantic representation. Topics in Cognitive Science, 3:2, 303-345.
This one is optional:
Johns, B. T., & Jones, M. N. (2010). Evaluating the random representation assumption of lexical semantics in cognitive models. Psychonomic Bulletin & Review, 17, 662-672.
First off, the Riordan & Jones paper was way over my head. I’m sure I will be the minority opinion when I say that I preferred the Johns & Jones paper but that’s okay with me.
ReplyDeleteI found the parts of this paper I could understand quite interesting. I really admire Barsalou’s attempt at a bottom-up explanation for representations, and I think PSS is a great step in that direction. Johns & Jones seem to appreciate it as well although from a methodological standpoint rather than a theoretical one (as far as I could tell). I was a little confused about their insistence that PSS can’t handle abstract concepts when in Barsalou’s 2008 paper there is a specific section on embodied linguistics (or semantic embodiment). I am intrigued by the Lakoff & Johnson theories, and I think they do a great job of explaining abstractions (Despite what one of the commentators on the metaphor presentation said, semantic embodiment is not only slightly embodied it is actually radically embodied. Embodied philosophers actually very rarely are willing to go as far as Lakoff & Johnson do). Perhaps I missed something though.
However, I do think there is some merit to Johns & Jones goals. Barsalou does admit that in language comprehension there are often instances where multimodal simulation does not occur. In a 2008 paper by Barsalou he posits the LaSS model that I am sympathetic to, but I do not know how well it would fit with Johns & Jones theory. Personally I think that trying to somehow keep language as separate from multimodal processing is unnecessary because I think if we are going to go halfway towards modal representations we might as well go all the way and ground language as well. Once again, however, I may have simply missed something.
I thought the Johns & Jones paper was much more straightforward and less detail-intensive, thus, I also liked it more. I got the gist of Riordan & Jones and think the findings are important (especially for the integrated model proposed in the other paper), but I think I would just get lost and confused if I tried to wrap my head around all the details.
ReplyDeleteThe theory underlying the integration of distributional and perceptual information seems to make sense. But once I got a general idea of how this model works, it seemed obvious to me that this kind of model could be applied to more than just the integration of perceptual features. As long as you have some sort of empirically derived index where features are mapped onto words, you can integrate distributional information with virtually any type of feature. I think this could be an especially interesting way to look at emotional valence (among other things).
In John’s paper “Generating Perceptual Representations from Global Lexical Similarity”, GPR model provides definitions of linguistic, perceptual, and lexical information. In the linguistic information, John explained the vector of word in the document which seemed different from that in LSA. “If a word is present in a given document, that vector element is coded as one; if it is absent, it is coded as zero.” This seems that the word vector is coded like 1, present; 0, absent, not the frequency of words occurred in the document, doesn’t it?
ReplyDeleteIn the GPR model, John proposed that a partial probe is compared to memory (a word with a linguistic vector, but an empty perceptual vector). Here I have a doubt that what measure is best to identify the certain word has perceptual information in our memory or not, and what criteria are used to do this task?
In step 1, the perceptual vector is related to the weighted similarity to lexical entries that have non-zero perceptual vectors. P represents the probe word, which is much easier understood, but T represents the lexical trace for a word, which is difficult for me to understand. Does this word represent the word without perceptual information? And how to find the statistical value of lexical trace for that word? At the beginning, I think lexical trace is the similarity of that word with other words, but it seemed that I was wrong, for in the following the model indicated that S represents the similarity. I would like to know how the equation in step I is obtained.
In simulation 2.1, the affordances in the example “Hang the coat on the ___”, how the afforded word=vacuum clearner, and non-afforded word=cup were selected. Were they chosen intentionally or randomly?
Well... at the risk of losing all credibility, I'm just going to say that I don't know/understand what a vector is. One definition I found online was "a feature vector is an n-dimensional vector of numerical features that represent some object" Really? Where were you in 6th grade when we learned not to define a word by itself. I understand what the second part is saying, but I don't know how you obtain this 'vector,' nor how to interpret it when I see it. I know it's come up in class already quite a few times, but when vectors came up again in these two articles, it was the last straw.
ReplyDeleteSo, on to the readings. In the Riordan and Jones article, they cite that "Maki and Buchanan demonstrated that these measures of semantic similarity were separable, encoding somewhat different types of semantic similarity, and they argued that distributional similarity is a distinct type of semantic similarity related to ‘‘thematic’’ associations between words." Although the Riordan and Jones article disputes this and their results provide support that not all distributional models reflect semantic similarity, if we just assume for a second that Maki and Buchman were right, and the two encode slightly different types of semantic similarity, how do we know which kind is more valuable? I suppose it would depend on what you are using the information for, yes?
I found the results from the Johns and Jones article to be quite impressive. I appreciated that they tried to blend together existing theories rather than thinking that they have to 'choose a side.' I hope that this paper gets a lot of attention in the upcoming years and that more work is done in regards to this subject. I'm curious to see how their model would handle abstract words. Could these be grounded within perceptual information as well? What about the meaning of something like 'post-modernism?' I wonder how their model might handle that.
Both papers were thorough accounts (read: long and full of head-spinning details), but I can definitely see more of Riordan & Jones' view. The idea that semantic categories are both linguistic and embodied is just more palatable to me (as opposed to worrying about only the embodied/grounded aspect). I do like that J&J highlights the importance of the amount of experience that the model has with language greatly improves it's inferences about the perceptual representation of a word (in the TASA vs. Wikipedia copora). I presume that we (people) learn language in much the same way. An issue I have with the J&J paper, however, is that they are inflating the importance of grounded cognition--AND they say that embodied representations of verbs will have to be included in future work. Much of what I've read (which is not exhaustive, but not small either) about embodied cognition theory is centered around action (pull/push a drawer, "smile" by having a pen in your teeth and evaluate comics, etc.). Okay, I'm probably overstating it, but that part still gets on my nerves. But back to my point, it just makes sense that there are both perceptual and linguistic features when you (or a model like LSA) interprets co-occurrence, relatedness, or what have you. If anyone wonders why I rail against the full-on embodiment approach you can read: http://madresearchlab.org/Selected_Publications_files/Louwerse2007.pdf It not only shows the interdependence of grounded and linguistic features, it also illustrates that LSA can even be used for embodied perceptions because language is set up due, in part, to our grounded experiences.
ReplyDeleteAnd this leads me to try to answer Haiying's question about the affordances. In Glenberg and Robertson (2000) they used an afforded object (unusual, but makes sense) such as "After emerging from the stream, Andrew used his shirt to dry his feet" or non-afforded object "... Andrew used his glasses to dry his feet." And lo and behold, people judged the first one to make more sense, and LSA didn't find a difference. [By the way, that whole paper was basically to say how much LSA sucks because it doesn't use grounded affordances. Just an FYI, if you think LSA is at least semi-useful.] But Louwerse (2007, referenced above) found that a cosine matrix (MDS) fed into LSA can have a positive correlation (around .30) with Glenberg and Robertson's sensibility ratings. And as he states, it's not conclusive, but it doesn't rule out the comparability either. But to ACTUALLY answer the question about how they chose the word: If i remember correctly, they just came up with a word that might work, but was unusual (the afforded word), and then one that didn't make sense (the non-afforded word). I believe they also used a related word (one that made 'more' sense, but still wasn't the expected target word. One of the examples from G&R's first experiment was: scenario- Dude was cold... used scarf to cover face (target)... used newspaper to cover face (afforded)... used matchbook to cover face (non-afforded)... used ski-mask to cover face (related). Okay, yeah.
I didn't mean to write so much, and I don't want to sound like a 'Negative Nancy' (I have a sunny disposition, really!). But I think there's room for linguistic and embodied features and we don't have to say one or the other is the definitive one. (That's why I like the R&J paper). :D
I found the integration of Perceptual symbol systems theory (PSS; Barsalou, 1999) and distributional model appealing. While PPS proposes that sensory exposure will consolidate the learning of words, distributional models demonstrated the abundance of language environments. I wonder how this integration will be able explain human’s emotion learning/recognition.
ReplyDeleteIn one of the studies conduced in our laboratory, we found that videos alone provided coders more consistent information in interpreting babies’ affect compared to that provided by both audio and video (and of course, audio alone provided little information about one’s information). Moreover, both video alone and audio/video provided lower agreement between coders in infants’yvocalization types compared to that provided by audio alone. it just occurred to me that we would assume that when provided with both audio and video, people will have better chance getting higher agreement in emotion reading or/and vocal type categorization, but it is not the case.
Same question here with PPS theory, to what extent sensory information assists human’s word recognition/learning, and what aspects of information would help most?
I agree with the sentiment that the Riordan & Jones paper was a little hard to follow, but I do like the idea of a combination of linguistic and embodied features for putting together knowledge and representation.
ReplyDeleteI did have a questions about one aspect of the Riordan & Jones paper (R&J), where they talk about the different contexts that can be used to derive linguistic semantic meaning. When they begin discussing the different distributional models, R&J discuss the representations that can be derived from context word and context region. Is there a reason that both of these contexts would make sense to use? It seems to me that the context word would just be a partial representation or a subset of the representation that would be derived from the context region model. By taking into account the entire region, I would think, a more complete representation of each word can be derived. Using the entire region would also include co-occurrence, so would the co-occurrence model be an incomplete representation? There is probably work comparing the use of these two models, but I wonder if comparing these two models would almost be similar to looking at developmental differences. For example, children often learn grammar rules, but then overly apply them to incorrect situations. So I wonder if the representation derived only from co-occurrence would be similar. The representation would not necessarily be incorrect, but would be lacking the bigger picture of the word's meaning. This is probably not the big of a deal in this line of research, but it was something that struck me as a little confusing.
-Blair
First of all, I complete agree with Yuna's statement/question, "I wonder how this integration will be able explain human’s emotion learning/recognition." I think this is a great point to bring up.
ReplyDeleteIn the Johns and Jones article, they discuss how perceptual information is a probability vector over perceptual features generated by human subjects in the GPR model. I am wondering how these perceptual features they mention (e.g., has fur being for a dog) could parallel some perceptual information that could be tied to a feature of emotion. Even though they do say, "It is important to note that these types of feature norms include much information that is non-perceptual (e.g., taxonomic, situational), and are unable to represent more complex perceptual information such as embodied interaction; nonetheless, they are a useful starting point," I am curious if this includes some emotional features or not (or could it). In other words, if we say sunshine, would that have a more positively valenced "tag"? Even though it would differ from person to person, is it plausible to think it can be included?
I think I would really benefit from an explanation of how it works. I relate to Jackie in that I think the definition may be simply unfamiliar to me.
grrrr. i wrote up my whole thing and then instead of hitting post i quit chrome. so forgive me if this post is useless as i am trying to reconstruct what i wrote instead of do it all again.
ReplyDeleteI appreciate the approach here. In Max's lab we usually take what they call perceptual and linguistic information and use it predict RT or neural activity or whatever on a variety of tasks. Here I think it is super that that linguistic information is used to predict the related perceptual information. I also was glad to see that reverse inferencing is possible. We have found that linguistic information is relatively more important early in processing whereas perceptual information is more important later. So, I would imagine that this reverse inferencing is only used when the word is entirely unfamiliar. I think then I had something written about how when there is impoverished linguistic information (e.g., with a small corpus) the model is less successful. The reverse in this case would also be true (i.e., impoverished perceptual information would make the model less useful) , but perhaps to a lesser degree? Anyhow, I thought the J&J paper was quite well done and I enjoyed it.
R&J was also quite clever and wow did they include a lot of models. They were quite exhaustive considering many of those models are pretty similar to begin with. I liked the use of wordnet for semantic class information. However, it would be super if someone could explain the operationalization of purity and entropy… I understood what they wrote about purity and entropy and what the results were, but I'm not sure why they imply such results. Maybe I will just re-read that part later cause it is pretty vital to understand for this paper. I think Blair's comment on context word and region is an interesting notion.
For me, I found the R&J article more interesting than the J&J article, but that may have to do with the time of day each of those got read. I was very into his idea of testing different types of model classes to look at the quality of their semantic clusterings.I thought, though, that topic models were put at a bit of a disadvantage in this case. Using a pre-set number of topics without really assessing the topic quality for that corpus can yield pretty poor results. Also, don't topics necessarily lack purity because they incorporate every word of the corpus? I know they used a different clustering procedure for the topic models, but it was really unclear to me how that procedure worked. The corpora used weren't fantastic for topic models either, with parts of speech included for the TASA one (I assume it treated each tag as a word), and CHILDES being dialogue between caregivers and children. They're just not optimal environments for topic models, which may have been part of the authors' point (distributional models alone are not enough). I was just a bit puzzled, but perhaps this is due to a lack of clarification in the article (ahh, space constraints).
ReplyDeleteAlso this: "For example, distributional models seem to give more weight to information about actions, functions, and situations, and less to information about direct perception related to objects (e.g., texture, internal properties, etc.)." (R&J) I'd argue that topic models could come up with perception related topics or words in a topic (I have seen categories of color words, and colors in a topic), but it entirely depends on whether the perceptions of objects occur in the corpus. But, perhaps I misinterpret their meaning.
This comment has been removed by the author.
ReplyDeleteI wish the style of the Riordan article was more clear and that the structure was a bit more user-friendly. The article is, in my opinion, completely unforgiving with somebody who doesn’t interiorize each and every phrase to a high level of detail. Impossible to skim through! The Johns article is better from these points of view. I wish I could write like them! However, Riordan’s models are “cleaner”, more appealing and the results are easier to understand.
ReplyDeleteComparing the distributional representation of a word with manually built perceptual representation of the same word seems to me a valuable and much needed endeavor. I found the results very interesting and, once again, convincing towards the usefulness of coocurrence-based semantic representations, be it context-word models or context-region models.
I tend to believe that researchers like Barsalou as well as proponents of the distributional models do not actually contradict each other. They just arrive at the same answers from different directions. And this is mostly due to a wonderful trick of distributional models: the vast excerpts of human language that are available as training corpora have an incredibly powerful noise-cancelling effect and, truly, the “latent” semantics emerge above the logic-bound language. Moreover, I think that besides not contradicting each other, they complement each other, i.e. perceptual representations bring accuracy and perhaps a better mental model, whereas distributional representations bring scalability and a better integration of abstract words.
Oh, by the way (I see that Haiying mentioned this too) what is the lexical "trace" of a word in the Johns article?
ReplyDeleteTHIS IS Shi's POST! (apparently she couldn't post it directly so she asked me to post it from my account)
ReplyDeleteShi says:
Sorry this is so late, I cannot post onto the comment section, so I have to use Cristian’s account to post mine, and the blog has deleted my post three times already so I’m tired of writing the same thing over and over and over again, so this is gonna be especially short:
Jackie: the definition of a vector is “a quantity possessing both magnitude and direction, represented by an arrow the direction of which indicates the direction of the quantity and the length of which is proportional to the magnitude”
I think the articles are intriguing, in that the models argue that semantic similarity can give our perceptual representations. Carl Cai gave a presentation on Chinese LSA space versus English LSA space yesterday, and he showed that, for Chinese, the words are composed of characters, and using characters as single words themselves can give you a LSA space that is way better than English. He also showed that for English, words that are akin to Chinese, such as the word “butterfly” is composed of “butter” and “fly” two words that have very different meanings and are virtually unrelated to “butterfly” have LSA similarities. This shows that when two words form another word, and that word has been used frequently enough, semantic representation for similarity between the two initial words is formed.
Cool stuff!
---Shi
I was going to say something about the definition of vector, too, but kind of forgot. Shi describes a more mathematical, Euclidean type of vector. Quite often though, "vector" gets used in a semi-computer science kind of way, where a vector is simply an array or list of elements (or items). So, the "n dimensionality" has to do with the number of items in your array. The authors of these works talk about vectors like a context vector, where every element in the vector represents a word's "presence" or "absence" in a document (basically, 1 or 0). Typically, an element's placement in the vector is meaningful, like all of the first elements in the context vectors refer to Document 1, the second element refers to Document 2, etc. This way, vectors are comparable. So, a word like "strawberry" could have a context vector of <0, 0, 1, 0>, which says that it is present in Doc 3, but absent from Doc 1, 2, and 4.
ReplyDeleteThe authors also talk about perceptual vectors, where a bunch of humans came up with features for objects (e.g., , ). Then, probabilistic vectors are made, where each feature they created has a vector of its own, and the vector contains the probability that it applies to a set of ordered words. So, we could have a list of words like "dog, cat, potato", and then the feature , and it's vector could be <.90, .85, 0>, so dog and cat probably have the feature , but potatoes don't.
This is probably an incomplete or oversimplified definition, but it's basically something like that. I didn't know if we'd get to this in class, so I thought I'd throw in my two cents (or one and a half or whatever it's worth). Aaand reading over it, it sounds like a garble of words, so maybe I made matters worse.