The reading for this week is:
Read this carefully:
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/dp1.LSAintro.pdf
Read this, skip the math if you want:
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/p995.pdf
Read these if you are interested (optional):
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/xhu1.pdf
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/xhu2.pdf
I am very interested in this week’s topic – LSA, but after reading these articles, some questions were put forward:
ReplyDelete1. LSA represents a variety of human cognitive phenomena like word-categorization. Does word-categorization mean parts of speech?
2. LSA determines the similarity of meaning of words through the words' association. Does similarity merely refer to synonyms, or also antonyms as well or even include word association like if a word “chair” occur, LSA will find its association like “sit”?
3. One of the limitations that Landauer (1998) mentioned “It makes no use of word order, thus of syntactic relations or logic, or morphology.” In this way, LSA only is useful in the aspect of words’ meaning, not the language semantic. For example, when we look at the following two sentences, the semantic meaning is different. This is a table.
Is this a table?
LSA will show they are the same meaning, wont’ it?
4. Landauer (1998) said that LSA induces representations are between unitary expressions of meaning – words and complete meaningful utterances in which they occur – rather than between successive words.
This is what differs from what I thought. For example, LSA calculates the adjacent words successively occurred. And then, LSA provides the patterns of occurrences. Is this correct?
This comment has been removed by the author.
ReplyDeleteFirst of all, bleh! I hated these readings and for obvious reasons. The first one was a total nightmare for me because it was long and dry. The second one was even more dry but at least it was really short.
ReplyDeleteI think the aim of LSA is admirable. I imagine that the association of learned knowledge based upon some kind of flexible statistical probability is the way the mind sifts through knowledge. However, I am always cautious with accounts that treat the mind as some kind of computational engine, which is the impression I got from reading the article. It is entirely possible that I simply didn’t fully understand it.
I am looking forward to a full explanation in class!
I got a bit lost in the details of how LSA works, specifically in the details of SVD and what output (probabilities?) ultimately results from this process. Put another way, how exactly does one use the output from LSA that's been "trained" on a corpus to answer problems on synonym-antonym or subject-matter tests? Despite the fact that my understanding of the process that LSA uses to answer these test problems is poor, I still think it is interesting that LSA's performance mapped on to human performance so well across several different domains.
ReplyDeleteOne bit that I found particularly interesting was from the essay grading result where it is suggested that "the holistic semantic representation of a passage relies primarily on word choice" and little on other factors like syntax. This seems to provide justification for the "bag of words" approach, but could there be cases where this does not hold true (e.g. in texts where word order and syntax are more crucial for understanding)? Could this approach be more problematic with languages other than English?
This work is fascinating. I was awed to read that LSA "makes no use of word order, thus of syntactic relations or logic, or of morphology", yet still "extracts correct reflections of passage and word meaning". That's grand. (Sometimes in conversation I only hear certain words, yet still follow the speaker reasonably well; is this a similar phenomenon or am I just a jerk that gets lucky with my conversational follow-through?) How does LSA do that? Was it explained and I missed it? If LSA can effectively determine meaning regardless of syntax or logic, of what use are they? It almost seems as though syntax and logic serve as buffers between the unlearned and its inherent meaning. LSA has no interest in "learning" only "identifying". As though LSA bypasses these because it doesn't need them to "comprehend" the meaning. (I don't understand what I just wrote, but it feels good.)
ReplyDeleteThe second article did well to highlight some of the holes LSA and other Semantic Space models seem to leave unaddressed. Not to say they need what these deficiencies reveal, but only serve to tighten the results and abilities of these models. In all regards, I am eager to see how the use of these models will evolve, as they are all relatively young methods.
I'm going to skip over my overall dislike of this article, and of course how the details were way over my head and just jump right into specific parts of the article.
ReplyDeleteI liked the comment by Luno "If LSA can effectively determine meaning regardless of syntax or logic, of what use are they?" I think that's an interesting point. My quick take on it would be that LSA doesn't do as good a job as we do at determining meaning, so maybe that difference is due to our use of word order and logic. This reminded me of our earlier reading (week 1) on topic models. Both LSA and topic models do not take word order into account, but I wonder why that is. Why it's unable to do so. Could that be changed? Is anyone trying to do that? Would it even serve a purpose? Could LSA and topic models be combined in use somehow? Answers to some of these questions might have been stated in the reading on topic models or we may have even talked about it during that class, so ignore me if I am merely forgetting. Now that I think of it... what is the main difference between LSA and topic models? Don't they have the same general goal? Or am I just demonstrating how much I don't understand these topics?
Toward the end of the article the authors also state that "No previous theory in linguistics, psychology or artificial intelligence research has ever been able to provide a rigorous computational simulation that takes in the very same data from which humans learn about words and passages..." Really? Can this be? There's nothing else?
On to the second article.
The article states that "Semantics of any level of the language entities can be represented numerically or algebraically (p. 996)." How, exactly? How is meaning turned into numbers? I really do have a deep desire to understand math. X0 = {x1, x2, ..., xN}. I look at that and all I think is, In this font, if I stare too long at the lowercase x's, they stop looking like x's, and eventually they morph and begin to look like infinity signs or interpretive art. The sad thing is that if I looked at that in high school, I would have had a lot better understanding of it. And don't even get me started on page 997.
On page 998- "The representation of LSA is a finite dimensional vector. It can be viewed as infinite vector with finite non-zero entries." Um... what?
So it looks like my main issue is the overwhelming amount of math in this article. I don’t think I have anything else to say about it.
In my next life I would like to be a mathematician, please.
I think LSA is the bee's knees. It's not a panacea, but it is able to look at a lot of aspects that we mere mortals could never find enough time in the day to do. It reminds me of a concept I've read (I think it's from a piece of fiction, but I can't be sure. I paraphrase, of course): If you look at a thousand paintings you can be an expert in art. You don't have to be classically trained, you see enough examples of something and you will eventually see the cohesive patterns, whether you're conscious of it or not. I think we all initially learned our first language like that. So, while I'm usually spouting off about the lack of attention given to context often found in research studies, I think LSA shows patterns that are FROM given contexts (sorry for caps, there's no italics here). I'm not familiar with the intricacies of LSA, but if you use huge pieces of diverse corpora (e.g., encyclopedias, newspapers, textbooks, etc.) as your "co-occurrence pool," those frequencies have to indicate something important, and we (humans) most likely can't notice them all.
ReplyDeleteHowever, I can think of at least one issue that can arise: If you can pick the corpora used (as it seems that you can, but I could be totally not getting the way it works), can't a researcher just pick those bodies of text that are more likely to give the desired co-occurrences? For instance, if a researcher thinks that in the last 5 years, the economy is in a terrible downward spiral (not true). They assume that the word "finance" would more frequently occur with "destitute" as the last few years progressed. They look at tons of newspaper articles, and lo and behold, they found a strong correlation. They could leave out textbooks, bank memoranda, and all kinds of things that could show a more full view of that relationship (or lack thereof). I know this is a crude example, but I think it would be easy to at least accidentally pick the "right" sources to support an idea. This sounds like the opposite of my previous point about looking at 1000 paintings, but I'm talking about (unconsciously) stacking the deck. I would assume that people reviewing the article would notice a lack of diverse corpora, but maybe not. We're not as smart as LSA. ;)
Also, polysemy can be tricky, as I believe Andrew mentioned in class. I think one day it will be possible to account for these issues. We just have to wait until one of us figures out how.
LSA is interesting, for its emphasis on word contexts just like the way we emphasize sociocultural contexts when we talk about how social factors influence language learning.
ReplyDeleteSome questions regarding properties of Latent Semantic Analysis:
-LSA’s mathmatical inference of word-context association can better predict human’ semantic judgment than that predicted by contiguity frequencies/co-occurrence counts/and correlations in usage (Lander, Foltz, & Laham, 1998, p4).
Q: Since texts in LSA reflects human’s semantic knowledge, are there any golden rules for how to select a body of well-represented texts? And how much and to what degree it needs to be in order to approximate human’s knowledge?
Q: bank
1. Harry Potter runs along the bank of River Alpha.
2. Harry Potter runs on the bank.
3. Harry Potter robs the bank.
are they stored like separate word entries, bank2, bank2, and bank3, in LSA?
or are they all belonging to the same word category “bank?” (which is less possible, I assume.)
then what about:
1. Mary robs the bank.
2. Jeff robs the bank.
3. Mary and Jeff rob the bank.
4. Mary robbed the bank.
5. Jeff robbed a bank.
6. Mary and Jeff robbed a bank yesterday.
7. Mary and Jeff robbed the bank yesterday morning.
….
are these banks counted as separate word entries or not?
This week's paper is very helpful in explaining why LSA is a conerstone in understanding discourse processes. I was amazed at all the things that LSA can do; anything I can think of it seemed to have already accomplished by LSA. the Landauer paper also did a really good job in addressing all the limitations that LSA can have, but they almost seemed trivial, as LSA can overcome these limitations with very high correlations.
ReplyDeleteI'm still not entirely understanding how LSA got it's dimension scores. Just to want to make sure: so every word in a text or sentence is assigned either a 1 or 0, positive or negative in a dimensional matrix (I'm still not clear which ones are supposed to be positive or negative, I only understood that if a word appears, it's a 1, if it doesn't, it's a 0, I don't know if this is correct or not), and then the numbers are averaged to get a score such as: -.0066?? I still don't really get it.
Also, I was intrigued that LSA can have such a high accuracy without taking word order into account, could this suggest that human beings also process language in some ways similar? How well does LSA do in other languages? does it show the similar accuracy? even better? There has been a lot of arguments regarding which language(s) is better in terms of resolving ambiguity, can LSA determine that as well?
I'm glad we had to read this, it is a pretty clear explanation compared to anything else I've read about LSA.
ReplyDeleteWith topic models we saw application in other fields (and with Andrew's imdb example). I assume it's alright to similarly plug in other sorts of data as long as they are arranged in a similar hierarchical manner (passage-word). If that is the case, can LSA (or some other algorithm) take another level into account within the same analysis(book-passage-word) and have a 3D matrix? Or would you just have to run something looking at book-passage and passage-word? What about even more than 3 dimensions? Possible? Also what happens when number of words is > number of passages (this was mentioned in one of the Hu et al. additional articles that I accidentally read).
I have a few more technical questions as well:
Why is the preprocessing step necessary? I assume it is to sort of even the playing field for frequent/less frequent words to be able to equally contribute to the meaning of a passage. If that is so, then why not have such a step for other models?
In SVD I do not understand why there is a third matrix with scaling values. Anyone who can explain this? I guess these are the singular values so in signal processing would these values represent the sources? Why are they only on the diagonal?
In other kinds of factor analysis, it seems like it is easy to assign categories to the factors (as we saw with topic models). In the Hu et al. article the explicit features of the dimensions were discussed. Why are they assumed to be latent if they really aren't? When would they be latent versus identifiable?
Like Jackie, I too wish I was super awesome at linear algebra but alas…
I have used LSA in two analyses related to state standards for high school biology and was surprised both times by how effective this bag of words approach was. Using LSA to create a distance matrix between state standards allowed for classification of state standards according to depth of knowledge represented and even caused a grouping based on whether or not a glossary was included in the state standard.
ReplyDeleteI do wonder though, if there would be a way to combine LSA methods with another bag of words approach like LIWC analyses? For example, could you derive semantic similarity between two documents based on the words that appear near each other and the "type" of word that LIWC classifies them as? I think this would be very interesting because it would allow for further classification of documents and why they are similar. Right now you could create a semantic distance matrix between documents using LSA and then run the same documents through LIWC and determine the proportional occurrences of different "types" of words. But would you achieve the same outcome if these two methods were combined? Other forms of text analysis or word categorization could be combined with LSA similarly, LIWC just came to mind most easily because it seems to be a compatible method.
-Blair
First, I want to reiterate what many of you have already noted in that this paper was very helpful to get a clear picture of how multifarious LSA can be.
ReplyDeleteOne thing that I was wondering is about the irrelevance of word order and the use of LSA to score holistic essay quality. I think I understand generally the 5 methods they us (and also note they do caution that further research needs to be done on this), but I would be interested to learn more about these data sets they used. I can grasp how key words etc can yield the types of correlations with expert scores that are mentioned, but I am curious how it must parallel and amazed how well it seemed to do.
Also, it would be very interesting to see what happens over various types and topics of essays (e.g., emotional essay where key concepts and words may be vastly different, or free-writes, or socially charged essays where there are two clearly opposing sides).
Another thing that Shi mentioned that I wondered was about how well LSA is in other languages and how they may compare to English.
I was greatly impressed by the abilities of LSA. I had seen a similar mathematical method before in determining co-occurrence and found it very interesting how it was adapted to language. My biggest 'complaint' is that the long list of its glowing reports actually makes me skeptical; What are instances or application areas in which it fails? Any problems besides those expressed in the paper?
ReplyDeleteThe biggest impression is how it can be expanded in a larger language processing systems. It struck me as a exemplar method of semantic relations. As it encounters new corpra, it develops new examples of semantic co-occurrence. It seemed that it can be matched with an 'inference' program to be used to define words (eg, doctor occurs with patient and physician for which LSA is indifferent but theoretically some other system could interpret).
It struck me that LSA's best quality was also what caused its limitations. The fact that it is context nonspecific is what allows the breadth of its co-occurrence findings but what also causes its inability to infer and need for large amounts of memory, processing, and multiple examples. The article mentioned that in the comparison tests that certain biases or filters were used (stop words, etc). Have the effect of using certain biases been explored? Are there trade-offs that the model gains or loses? I was thinking of David Blei's talk in which he mentioned that if the topic model was to strict in changing topics that it wasn't dynamic enough to adapt to new or changing topics but if it was not strict enough, you didn't get useful or insightful topics.
Overall I think LSA is very impressive and has many applications once they can be matched with other systems. And with everyone's comments so far, it should be an interesting class as long as we're not hanging on semantics :P
I am going to play devil's advocate... kinda, because in part I believe what I am about to say. In class, it seems like there's a fair amount of eyebrow furrowing and arm crossing when the notion of "bag of words" is brought up. Invariably, someone talks about X murdering Y, and how order definitely matters in the context of this sentence, and so topic models and LSA could not capture its semantic essence. This is seen as a drawback. My argument is this: on a document/passage level, it does not matter who killed who. Or, perhaps more importantly, if the entire article is about who killed who, that single sentence does not matter because the rest of the article's text can lead us to the killer. Jackie (and others previously) mentioned that word order seems important, and could this be built into LSA and LDA. I say that it's an accomplishment to get the results that we do WITHOUT word order. In fact, I think I heard someone say once that when word order is used in conjunction with LSA or built into it (can't remember which), results are not generally improved. So, I posit that for understanding the majority of a word's meaning, its context in terms of neighboring words is far more important than the syntax of the sentence. Maybe this is even true of us as humans - does context and other word usage inform of us a speaker's meaning more than the structure and word order of a sentence? Perhaps this is not as devil's advocate-y as I had hoped, but I think it would be nice to have a discussion about context vs syntax and their importance to semantic meaning.
ReplyDeleteAlso, I'm very interested in Rick's notion of the importance of the corpus. It does seem that the selection of the corpus from which meaning is extracted is important - for instance, think of LSA that uses Fox "News" articles vs MSNBC articles. The world would appear to be a very different place in either case, despite that the articles could be selected from exactly the same period of time.
By the way, Rick, I have a nice quote that complements your "paintings" scenario. It's from a Sherlock Holmes story (Red-Headed League): "As a rule, when I have heard some slight indication of the course of events, I am able to guide myself by the thousands of other similar cases which occur to my memory." In short, given enough data (experience/context), one can make inferences about similarities. It's tangentially related, but I've been dying to share it since I found it, so...
That's a great quote, Whitney! I think it's very related. :)
ReplyDeleteThe first article: one of the classical texts of LSA scripture, clear and concise, presents both limitations and solid results. In my opinion, LSA together with LDA (and variants) are the pinnacle of Statistical Natural Language Processing. Or at least among the very few widely used models which are PURELY Statistical. A large number of the best NLP applications used adapted/perfected versions of LSA spaces.
ReplyDeleteLet us not forget, that, in its elegant simplicity, an LSA space is nothing more than a huge word-document matrix. Or, better put, a “refined” word-document matrix, with a number of very smart mathematical transformations that are proven to keep the maximum amount of relevant information and giving away information regarded as not-as-important and noise, that tries to transform the simple coocurrence properties into high-dimensional semantic properties. I have problems with many assumptions of LSA, not necessarily the fact that it does not take into consideration word order, syntax, etc (like some of my colleagues already mentioned), but first and foremost a more sensible one in my opinion: what are the criteria for selecting a document. Sentence, paragraph, page, article, chapter? How about titles, news lines, stanzas? But the bottom line is that LSA WORKS.
Second article (Hu et al.): Interesting (and much needed) numerical approach to LSA spaces, namely proposing a metric of similarity between vector representations. Now, it can be argued that it is unclear whether what the metric measures is language (style + content), corpora, or LSA applications (like choosing number of dimensions, preprocessing, weighting, etc), however, I believe this is one of its strengths and that it is flexible enough to measure any of the attributes mentioned above.
I am sorry, but I cannot miss the opportunity to reiterate the question that I posed before Blei’s presentation and then explained on the respective discussion board (this time with the newly acquired vernacular): could one build an artificial corpus that given as an input to the LSA and LDA algorithms respectively would yield results with ~1 combinatorial, permutational and quantitative similarities? If so, what properties must be respected?
---Cristian
P.S. to Benjamin: I respect and understand your subjective reaction to the articles and LSA. And I believe that “cautious” is a perfectly reasonable word. However, I will tell you why I don’t share your opinion when I read the phrase: “I am always cautious with accounts that treat the mind as some kind of computational engine, which is the impression I got from reading the article”.
First of all, I believe you must agree that at least in SOME occasions it is right to study people’s responses with the assumption that their “mind” (I will use your term although I don’t know very well what it means but I hope we refer to the same thing) as a computational engine. For instance, people use correct logical inference and the tools of math and science every day in math class, science class and also (even though much less obvious) in normal verbal communication. So, at least sometimes, people WANT to use their brain as a computational engine and when you ask somebody what is 14*45 they employ similar brain processes as when asked what is 56-145 or to compute the acceleration of a body. These processes are also pretty similar across eras, brain ages, latitude and longitude.
(continued!)
(...)
ReplyDeleteSecond of all, nowhere in the article is there mentioned that it attempts to cover all or most of the aspects of the mind, (or brain activity, whichever you prefer). And I believe you don’t want to imply that any approach that is not holistic, that does not try to model the entirety of the human mind is secondary and inherently inferior. I do not believe that you would argue against mathematicians that are concerned with concepts and operations so abstract and far from our immediate representation of reality, that they have a small chance of being useful in a real-world application.
In the case of mathematicians, let them count and compare their infinities, because even though you might believe they are useless, at least they are right (mathematically). And in the case of LSA, judge it by the results, not the appeal of the theory behind it, because even if you think it is not right (philosophically), it sure is useful.
And I hope you don’t equate following my advice to turning away from philosophy and towards a more applications/results oriented science, like psychology or sociology. That’s as far as possible from my intentions. But maybe you would be more successful by being less dismissive of non-attractive and very narrow and specific theories that do not appeal to you. I think your motivation should stem from reading exactly these theories and their results, because they would allow you to try to adjust your complex “mind” models to the empirical data, not the other way around.
P.P.S. to David: I share your curiosity about a task-dependent measure of similarity and/or efficiency between different languages when it comes to LSA spaces.
to Jackie: it was a delight to read such a personal, honest, witty, funny, insightful and brave comment. I wish all forum posts that I will ever read in my life were like that (Including mine :( )