This is the blog for the Cognitive Science Seminar (COMP/PHIL/PSYC 7514/8514) at the University of Memphis, fall 2011.
On this blog you can post your responses to readings, which will also be distributed on the blog.
The first reading (for Sept 7th, responses due by noon Sept 6th) is here:
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/Blei2011.pdf
You will need your email uid and password to access this and other readings.
Don't forget to sign up for a day to present on the doodle poll (url is on the syllabus)
The slides for the first talk (my intro talk) are here
https://umdrive.memphis.edu/aolney/internal/cogsci_readings/introduction.pptx
Optional reading (psych applications of topic models)
http://psiexp.ss.uci.edu/research/papers/Griffiths_Steyvers_Tenenbaum_2007.pdf
Hi! This is Rick. So, we're just posting here, yes? Okay, great!
ReplyDeleteTheoretically, I think this LDA stuff is awesomes! But I do hope to see a better graphical interface for real world usage (as alluded to in future directions). I've seen the graphical usage of topic model fit (Figure 3, p.5) in library catalog searches. I don't really use it. I don't know if it's that the design is less familiar to me, or is not really conducive to how we naturally deal with related data. The better development of the application of this kind of research is mindblowing, however. So someone tell me if this is how we should go about this. :)
I am not well versed in this subject so I found the article dense and unforgiving. Obviously this makes it somewhat difficult for me to comment on the article.
ReplyDeleteMy question is, could these topic models be useful for explaining the associative processes used by the brain during cognition? The function of these models described in the article seems to be purely for data collection and analysis, but, as a supporter of theories of cognitive processes that are based primarily upon association, I am forced to wonder if there could be a broader application for these ideas.
Hopefully this is not a stupid question. However, this is a cognitive science seminar rather than a data collection seminar so I figure that it is at least worth asking.
I just wanted to quickly say that I thought Benjamin's idea about how/if this could be applied to associative processes during cognition was a really interesting point, and I'd like to know if any work is being done in this regard. As far as the article goes, I was disappointed that the figures did not help me to grasp the concept any better, but rather added confusion in some instances. Although I definitely walked away having a grasp on the general concept, I'm still fuzzy on the details, especially how the models work (i.e., the processes that take place to give the output on documents' topics). I found the final section about future avenues to be most interesting and was particularly surprised to hear about the disconnect between how the models are used and how they are evaluated. This seems like a huge oversight, and it makes me wonder about other instances of inefficient evaluations in other areas of study within, and outside of, psychology.
ReplyDeleteI thought it was interesting that LDA was compared to principle component analysis. It is great that we can use these algorithms to spit out these sets of words that represent a latent theme in the data, but have topic models reached a point where they can identify what these themes are and give them meaningful labels? I believe the article touched on this a little bit, but it did not go into much detail. When doing principle component analysis and factor analysis in psychology, obviously it is important to be able to identify what the factors actually represent. I was just curious if topic models are capable of this yet.
ReplyDelete- David
I think incorporating additional data into the LDA models is particularly interesting. Specifically, I was thinking about adding in tags at the word level for the emotional or valence value of each word (or more likely each content-rich word, since if, and, the, etc. do not generally carry any type of valence). This could allow for three topic models: (1) word only, (2) valence only, and (3) word + valence models. A comparison between the three models could give additional information about the topics being discussed. Perhaps even revealing if the author holds positive or negative views towards specific topics. This could also reveal if authors are taking a balanced view on the different sides of a controversial topic, or through their word usage are clearly taking one side despite discussing both topics.
ReplyDeleteThis could also be applied to transcriptions of speeches. Specifically, the variation in prosody, intonation, and other vocal features could be tagged with each individual word and analyzed in much the same way that the emotional value of each word was analyzed when building topic models.
- Blair
There seem to be wide-ranging applications for even the simplest of these models. Notes regarding hand calculations, and their inherent futility in data sets of certain sizes, strengthen the necessity for more advanced versions of topic modelers.
ReplyDeleteSome questions:
1. "Omitting words like “but” and “yet”" – What if the topic centers around conjunctions in grammatical settings? (Author-topic models might satisfy this concern, at least in part.)
1a. What about satire? Parody? Sarcasm?
2. Does LDA assume a hierarchy of topics?
2a. Is it right to assume there is a clearly defined topic in a collection of documents?
3. How/why does the algorithm assume a certain number of topics? Arbitrary? (Not sure this was explained in the text.)
4. "We formally define a topic to be a distribution over a fixed vocabulary." - What is the fixed vocabulary; how is it defined?
5. (These systems seem to be defining a hierarchy based on an implied foundation.) - What are the components of the foundation?
6. "A central research goal of modern probabilistic modeling is to develop efficient methods for approximating it." - Are these essentially filters, or do they build the approximations?
- Jeremy
Like David, I was reminded of MDS/PCA the entire time I was reading this, so I was glad when the it was stated that LDA relies on these same techniques. However, I wonder what are the main differences between LDA and other algorithms that rely on singular value decomposition? It seems like you would be able to get the same type of clustering using other methods? What is the advantage with LDA? When does it work best? When doesn't it work too well? In other words, why use this instead of other models?
ReplyDeleteAlso, footnote 1 tells us that the topics must be generated beforehand & I didn't quite understand this. Figure captions for figures 2 and 3 mention X-number topic models. How are these determined & specified such that in one model we are going to get 100 topics and the other will only have 20? I assume then that this is determined before the algorithm is run as footnote one suggests? With MDS there seems to be a cutoff of where the categories are no longer semantically related. Is the same true here? I guess this is the same question as Jeremy #3.
I also just wanted to mention that I found the idea of hierarchies of topics and author-topic models really interesting/useful.
Topic models can be used to discover the topics covered in a document, as well as the extent to which each topic is covered in a particular document. As Blei often points out, this can make grouping documents by topic quite easy, which could enrich information searches. But really, topic modeling is discovering the groups of hidden variables that contributed to the creation of the document. Largely, these are content/genre dependent - neuroscience articles would greatly overlap with one another in terms of topic distributions, but would not overlap so well with comedy reviews. However, a document is composed of more than just content words. Could LDA and other topic modeling algorithms be used to discover things such as "authorness" (the author chooses the words in the document, after all), time period (word selection is heavily influenced by common vocabulary), and location (as diction changes from region to region)? Obviously, the algorithms aren't optimized for this, but it seems that if one had enough documents, the topics and the words that fall into them could tell us more than simply content.
ReplyDeleteI think to some extent, this has been accomplished by author-topic models. However, those seem to work optimally for documents that have been co-authored (though I have seen some techniques where two author's independent works have been put into the same document and the model distinguished between them, to some degree), but I am more interested in this idea of an authorship "fingerprint" - e.g. can we tell Shakespeare from Marlowe based on word choice in similar topics? Could we use LDA to discover the true author of unknown texts, given it is someone we know to have authored other texts? This idea is also a way to bypass the need to tag documents with meta-data by hand.
Just a thought. It seems like using LDA for new purposes or in conjunction with other tools/algorithms might yield some interesting and helpful solutions.
I like topic and thematic structure, but I am not familiar with the algorithm models.
ReplyDeleteLDA model provides a powerful and useful tool for discovering the hidden and blended topics; however, the difficult part to understand is the notation in 2.1. Could this notation be easily expressed by a graph?
Another difficult part is Figure 5. It showed the benefits of the topic distributed over words in the texts on science. I am wondering whether LDA is still powerful when it is used in other genres like narrative novels.
Hi sorry this is late, but I finally figured out how to post (that i have to create my own blog)
ReplyDeleteregarding the paper, I would like to know the number of "fixed vocabulary" and how they determined that, and what did they do for vocabulary overlapping. Also, I don't know if you rate the vocabulary with respect to keyward searches they can fall into the same distribution.
In reading the paper, I was also a bit confused by how the fixed vocabulary is defined. If the same words had a different meaning, given different contexts (maybe literal or a joke), then how would that have an effect?
ReplyDeleteI think it is really interesting when he talks about having an effective interface. I had not thought of the importance of that until he brings it up in the paper, but I'm curious as someone who has no previous experience, what would be some essentials to make it "effective"?
test
ReplyDeleteI found two ideas very intriguing in this paper: one is the reversed generative process of LDA, and the other is the thematic/statistical actualization of our collect knowledge about words and topics.
ReplyDeleteIn the Optimality Theory of phonology, linguists also use a reversed generative process to choose the best fit of a hidden structure from all possible candidates (candidates of INPUT) on the basis of characteristics of the OUTPUT. The concept of LDA is basically the same, except for some assumptions and hidden rules of LDA do not seem so clear to me. For example, the concept of randomness in generating the words out of a collection of documents (p 3). The assignment/definition/selection/connections of topic structures is never random, isn't it? What is the theoretical reason accounting for this randomness? Also in author's definition of topic, “a distribution over a fixed vocabulary,” the term “a fixed vocabulary” is confusing. If topic is a distribution over a specific set of vocabulary, what is this specific set of vocabulary? A dictionary? And there are more questions, as raised by Jeremy.
Secondly, the idea of adding valence (as mentioned by Blair, great idea!), affect, voice, and even subtexts of the words into the topic model, through which we can probably settle down the authorship disputes of, say, Twelfth Night (thanks Whitney, for reminding me of my Shakespeare class discussion). I think that LDA probably can serve our goals well, for its unique characteristics providing us a shared ground for doing contrast and comparison among abundance of word knowledge within one collection set.
Yuna
Hope this is how I do it. Sorry this is close to the deadline. Topic models seem to be the AI equivalent to schema theory in psychology. As topic models develop it should be interesting to see if they can be used to approximate the human brain in placing things in context. I thought of the difficulties of Watson (Jeopardy CPU) seem to fit the progression on the paper quite well.
ReplyDeleteMost of my questions stem from the meta-data analysis. Incorporating many of the nuaces mentioned by other ppl above; eg, sarcasm, voice, etc. How well does it handle different languages. The "bag-of-words" method I would imagine would not be affected much, but if you try to take into account the many aspects of language, the analysis seems to become more language specific. I also wonder how they could incorporate changes in a particular language over time and how different subject areas change at different rates (eg, science vocabulary vs slang).
They seem to have an immediate application in research in particular with the graphical representations in which a human operator can interface with.
I am posting this again, since it just disappeared a few minutes ago!
ReplyDeleteAll my following comments are based on my understanding of the article "Introduction to Probabilistic Topic Models" by David Blei which I will summarize in the following:
In the larger field of probabilistic modeling, a class that contains methods like ... Topic Models are nothing else than an approach to use the observed data (also called evidence) in order to derive assumed hidden variables by computing the conditional distribution of the given variables given the observed variables.
In the case of Topic Models the observed data are the words occurring in different documents of a corpus. The hidden structure that the method tries to unfold are the topics, which are a distribution over words, corresponding to the natural human notion of a conversational topic. In this article the author insists on LDA (Latent Dirichelet Allocation), which is claimed to be the simplest Topic Model.
Although the method of computation (the actual algorithm) is not described in the article, we are given a black-box description of the input and the output of the method and the general principles that are followed in the actual computation.
Given a fixed number of assumed topics and the corpus itself, the computation of the model parameters determines (not by exhaustive search but by approximation algorithms like sampling algorithms or variational methods) the topic proportions of each document and the topic assignments of each word in each position of each document. The resulting model, therefore, shows for each document the distribution over topics, which is a useful new metric of text similarity, also shows for each token the topic it was assigned to, useful for word-sense-disambiguation, and finally, perhaps the least important but definitely the most attractive, the definition of each topic by its most frequent (expected) words. This is a definitely useful and extremely appealing latent property of a corpus and by this it makes history together with LSA, both being well grounded in cognitive science and proven extremely reliable in applications.
The excellent article of David Blei includes presenting the limitations of LDA and describing some attempts to escape these disadvantages, both in terms of relaxing assumptions and incorporating meta-data.
My other comments would address a few issues:
- I do see the motivation to further read into this topic given the fact that I am very interested about perspectives on pre-processing. None were discussed in the article for obvious reasons.
- I do see the motivation to (and I encourage my fellow colleagues to do the same) to get instructed on the actual algorithm used by LDA, which, from my general recollection, is an iterative approach that I think would be very useful for any researcher in sciences that build statistical models, just like paths-in-graphs algorithms are a must for computer scientists and engineers who often find themselves in the position to adapt one of the general methods to their particular problem.
- I suspect it is not going to be the case in the presentation, but I would be very interested in seeing a few examples of applications of LDA in which LDA is not the central point, but merely a tool.
And finally one question that I cannot even attempt to answer so far:
- Is there a relation between the topics of a corpus (as obtained by LDA) and the latent concepts of that corpus (as obtained by LSA)? If so, how does that relation change if we manipulate the number of topics and the number of LSA dimensions?
--Cristian
Dr. Olney,
ReplyDeleteI would like to try to explain the question that I asked before the presentation.
My question was:
"Could there an artificial corpus be produced such that if given as input to the LDA algorithm and also to the LSA algorithm, the obtained results are identical or very similar?"
The result of the LDA algorithm is a pre-specified number of topics (say 100) which are a distribution over all words in the corpus, which means that for each word we have a number between 0 and 1, which also allows us to chose the first n words (say 20) and consider them "representative" for that topic.
The result of the LSA algorithm is a pre-specified number of dimensions, say 100, which can be interpreted as the cosine similarity to real words. So let's consider, in this case also, the first 20 words in terms of cosine similarity (normalized projection on the dimension vector/axis). Although it could be a stretch, we could say that the 20 words are the most "representative" for that dimension.
What if for each of the 100 pairs of topic vs dimension, each first 20 words would be the same? That is my question.
Do you consider this impossible for theoretical reasons (like orthogonality of some sort)? In case you do consider it possible, could you think of a simple (maybe even canonical) example?
Thank you, I hope I made myself clear this time.
---Cristian
Cristian-
ReplyDeleteThe problem is that the LSA dimensions have no obvious meaning. Carl and Xiangen have looked at this, and sometimes you can find early dimensions that correspond to frequency type metrics. I gave a presentation in the spring where I showed something like this with PCA on documents.
The best way to compare (that I can think of) is to do what I suggested before: for each word, find a collection of near neighbors and compare these neighborhoods for both LSA and LDA. Xiangen has done a lot of work on these types of comparisons across similarity metrics, so he would be a good person to talk to or read his papers.
If you go this route then it's a word/word similarity type of comparison, rather than a topic type comparison. BTW other kinds of matrix factorization, like NMF, can be sparser and more aligned with the topic based comparison you are thinking of. If you consider the differences b/w SVD and NMF, I think it might be clearer.