I appeal the very stupid editors comments on the bootstrap and guess what another methods man agrees with him and so it gets rejected again. So here is a little response.
IT IS ABOUT THE BIOLOGY.
I really don't care about the methods as they are all heuristic approximations with more incorrect assumptions than you can wave a stick at. The trees are right putting bootstraps on them does not make them any more right. I am telling a story about biology not about maths.
Now I can do a story about maths.
Bootstraps were invented by Efron who wrote a nice book about them with Tibshirani that maybe some of the editors and "experts" in phylogenetics might like to read. We all know that bootstrap is a resampling method where you resample with replacement a set of data. We do this in order to construct confidence intervals for complicated functions by simulations rather than an analytical solution which often is too complex or does not exist.
Now there are two key points:
1) If your sample is biased your bootstrap will still be biased and your confidence interval will still be wrong.
2) Resampling creates an identical and independently distributed distribution (you lose all correlation between variables when you resample).
Extending these two points for phylogenetics.
In phylogenetics they carry out convenience samples i.e. this is the set of sequences that someone happens to have collected but the have no idea if they are a representative or good sample of any kind. If I try to get convenience sampling based research published in almost any field (except phylogenetics) I would have my work rejected by most statisticians as wrong. So we suspect that samples are biased and if this is true then the bootstrap is not going to tell us much about this bias. In fact Efron and Tibshirani discuss this very problem on p138 of their book where they say bias estimation is an interesting but very tricky problem.
If you are using a technique where those correlations define your output - like say tree building in phylogenetics. Bootstrapping is a fairly stupid thing to do. Why do I want to create bootstraps which lose the correlated properties of my data? To do it properly you can read chapter 9 of Efron and Tibshirani which says the bootstraps are on the covariate vectors not on the data itself. So as far as I can tell all of the bootstrap implementations in phylogenetics don't do this.
This is a rejection of a paper where I am talking about the interesting bit of biology, not methods. The interesting bit is that not all H5N8 influenza virus is from the same ancestor. This means that viral subtypes spontaneously are recreated multiple times by reassortment of the viral neuraminidases and hemagglutinins. This means when you create a tree of a subtype it might not be homologous as has been found recently in a paper published in Science about Dengue which argues that the whole serotype argument for Dengue does not hold water. You see that is interesting biology but you miss it while you get tied up in your bootstraps.
As time goes on I am even more convinced that I am right and the biology I discovered is definitely important and significant and that the method pedants like the grammar pedants are defending an empty palace. They try to defend beautiful methods that actually have no relationship to reality.
Friday, 2 October 2015
Monday, 21 September 2015
Is my phylogenetic analysis right? Or why some editors are too stupid for words
I had an editor decide he was going to try and school me on statistics. I had used an approximate method to do a phylogenetic analysis which they objected to. Where are the bootstraps they say why use FastTree and not something else? (unspecified I might add). I needed to do a more robust and proper phylogenetic analysis.
Now I have two trees each with about 1500 sequences. These are for two genes in the same organism. I created an approximate tree for both and the two trees AGREED. They show the same seven sub-trees that are the key result of the whole paper. That means that from TWO INDEPENDENT SAMPLES I get the SAME POINT ESTIMATE AGREEING SEVEN TIMES. So for naive me I think that is PROOF BEYOND REASONABLE DOUBT that the two trees are correct. You simply cannot get the SAME wrong answers in two independent datasets 7 times, and if I consider the ordering of all of the other sequences they agree in many hundreds of positions.
Now I could do bootstrap, but this is just a permutation test to check if the algorithm is working properly it tells you ABSOLUTELY NOTHING about whether you have the correct biologically sound tree (check out Page and Holmes Molecular Evolution a Phylogenetic Approach for a discussion of the problems of verifying trees and that if trees from other genes agree they might be right).
Then again how many bootstraps should I do with 1500 sequences? It is a permutation test so lots is a good answer. Looking at a guide from Stata for a dataset of 448 they carried out 4000 bootstraps (they tried 40,000 as well and it gave about the same result). So for 1500 sequences I need at least 15,000 bootstraps and likely even more. So exactly what programme can I run on a fairly normal commercial PC that will run 15,000 bootstraps on 1,500 sequences in a reasonable amount of time? Even when I do that what programme will then be able to assemble and count these 15,000 trees to prepare a bootstrap tree? Even if I did would it tell me anything more than the two independent trees have already told me about my tree being correct?
Yes, the editor is an idiot. Yes, they have very little idea about what they are talking about. Yes they are too stupid for words.
They are so caught up with the technicality of bootstrap and maximum likelihood, confidence intervals and prior probability that they forgot what the ultimate arbiter of a good tree is: Does the biology make sense?
Now I have two trees each with about 1500 sequences. These are for two genes in the same organism. I created an approximate tree for both and the two trees AGREED. They show the same seven sub-trees that are the key result of the whole paper. That means that from TWO INDEPENDENT SAMPLES I get the SAME POINT ESTIMATE AGREEING SEVEN TIMES. So for naive me I think that is PROOF BEYOND REASONABLE DOUBT that the two trees are correct. You simply cannot get the SAME wrong answers in two independent datasets 7 times, and if I consider the ordering of all of the other sequences they agree in many hundreds of positions.
Now I could do bootstrap, but this is just a permutation test to check if the algorithm is working properly it tells you ABSOLUTELY NOTHING about whether you have the correct biologically sound tree (check out Page and Holmes Molecular Evolution a Phylogenetic Approach for a discussion of the problems of verifying trees and that if trees from other genes agree they might be right).
Then again how many bootstraps should I do with 1500 sequences? It is a permutation test so lots is a good answer. Looking at a guide from Stata for a dataset of 448 they carried out 4000 bootstraps (they tried 40,000 as well and it gave about the same result). So for 1500 sequences I need at least 15,000 bootstraps and likely even more. So exactly what programme can I run on a fairly normal commercial PC that will run 15,000 bootstraps on 1,500 sequences in a reasonable amount of time? Even when I do that what programme will then be able to assemble and count these 15,000 trees to prepare a bootstrap tree? Even if I did would it tell me anything more than the two independent trees have already told me about my tree being correct?
Yes, the editor is an idiot. Yes, they have very little idea about what they are talking about. Yes they are too stupid for words.
They are so caught up with the technicality of bootstrap and maximum likelihood, confidence intervals and prior probability that they forgot what the ultimate arbiter of a good tree is: Does the biology make sense?
Monday, 27 April 2015
Quick to Block
I generally find those who are swift to block aren't worth talking to. In that I include James Delingpole, Guido Fawkes and Damian Thompson. They are the fingers in their ears debaters who like their own voice and their own opinions much more than anybody else's.
If you read the book Emotional Vampires you will recognise them clearly. You will also know that they will never recognise this about themselves and that arguing with them is a waste of breath. They are always right, always perfect, they never make a mistake. I have worked around people like this where you walk on egg-shells not to say the wrong thing or do the wrong thing and it was the most miserable experience of my life.
If you read the book Emotional Vampires you will recognise them clearly. You will also know that they will never recognise this about themselves and that arguing with them is a waste of breath. They are always right, always perfect, they never make a mistake. I have worked around people like this where you walk on egg-shells not to say the wrong thing or do the wrong thing and it was the most miserable experience of my life.
It is always disappointing interacting with those you admire.
I have to remind myself that I should not overly admire people as they often prove to be as flawed as everyone else. So I believe strongly in rights for sex workers and I was disappointed to learn that Tina Fey does a lot of anti-sex worker "comedy". Today I was being generous and giving her the benefit of the doubt and so I was thinking aloud, well she may just be delivering the material, she might not be the writer and if you have a show you can just be there to deliver the lines and not think too much about the content.
So anyway Dr Brooke Magnanti had made the allegation and there was a link to the Saturday Night Live routine. I had sent her a tweet saying about maybe she is just a performer. To which she responded strongly. I have seen interactions on Twitter before and I would say she often responds pretty strongly. Often this is with justification and my tweet certainly annoyed her. So I carefully wrote another saying it is not an excuse but implying that nobody is perfect and if she was a writer as well she had no excuses at all.
So Brooke Magnanti's response was:
Along of course with a block. So while I am very disappointed with Tina Fey. However I am also disappointed with Brooke Magnanti and this is a bigger disappointment to me personally because she was the person I admired that I was alluding to in the title.
Now with more time to do my research and get an informed opinion rather than just living of tweets I know that Fey was the chief writer for SNL when they did the French Hooker sketches and that is just one of a long line of offences (http://titsandsass.com/category/tina-fey-hates-sex-workers/).
Stoya's article is great as usual. So Fey is bad and didn't deserve the benefit of the doubt it is a sort of, whatever as I am not that big a fan. I just wanted to be sure she was not being done an injustice, as I have jumped on too many band-waggons on Twitter and I was balance and thoughtful today. I make plenty of mistakes and so sending Magnanti my other tweet was one. Fey is definitely a slut shamer and sex worker hater.
So anyway Dr Brooke Magnanti had made the allegation and there was a link to the Saturday Night Live routine. I had sent her a tweet saying about maybe she is just a performer. To which she responded strongly. I have seen interactions on Twitter before and I would say she often responds pretty strongly. Often this is with justification and my tweet certainly annoyed her. So I carefully wrote another saying it is not an excuse but implying that nobody is perfect and if she was a writer as well she had no excuses at all.
So Brooke Magnanti's response was:
@ardalby Dude. I get you want to argue this but fuck off, she makes jokes about dead bodies of people like me. Begone.
Now with more time to do my research and get an informed opinion rather than just living of tweets I know that Fey was the chief writer for SNL when they did the French Hooker sketches and that is just one of a long line of offences (http://titsandsass.com/category/tina-fey-hates-sex-workers/).
Stoya's article is great as usual. So Fey is bad and didn't deserve the benefit of the doubt it is a sort of, whatever as I am not that big a fan. I just wanted to be sure she was not being done an injustice, as I have jumped on too many band-waggons on Twitter and I was balance and thoughtful today. I make plenty of mistakes and so sending Magnanti my other tweet was one. Fey is definitely a slut shamer and sex worker hater.
Friday, 10 April 2015
False Discovery Rate vs p-values
The problem with p-values are they are just based on an aside made by Fisher who said that any data that was more than 2 standard deviations away from the mean would be unusual. So the cult of the p-value was born.
If you look at Bayesian analysis of a rare disease (rare is less than 1 case in 10,000) where you have a test that is correct 99% of the time for true positives and also has a false positive rate of 1% (p-value) then you will still have a large number of cases where you identify the disease where it isn't actually present.
So for examples in a population of 1,000,000 you may have 100 cases of which your very good test finds 99 and only misses one. But it also finds 9999 cases that are not actually real. You false cases vastly outnumber your real cases. So if you are diagnosed your probability of having the disease is 99/10098, or less than 0.1% and that is with a p-value of 0.01!
The same thing happens if I do multiple tests. If I set a p-value of 0.05 which is quite typical there is a 1/20 chance of seeing a result when it is not there. So if I do 20 tests on average 1 of them will show significance. This is easily corrected by Bonferroni's method amongst others.
In genetic analysis you get complex problems with tens of thousands of variables that are all tested simultaneously. You also have massively under-powered studies because your sample numbers are small and so, sample size << number of variables. So you will always be doing many more tests than are justified by the resulting data and you will almost always be over-fitting the model.
This is why the false discovery rate is so important in genetic analysis but really when you look deeply at this it will still mean that even with careful use of FDR most genetic results from big data experiments will turn out to be wrong.
Here are the professionals talking about false discovery rates.
Selective Inference and False Discovery Rate I
Selective Inference and False Discovery Rate II
Estimating Local False Discovery Rate in Differential Expression
Interpreting p and q values in Genetic Analysis
If you look at Bayesian analysis of a rare disease (rare is less than 1 case in 10,000) where you have a test that is correct 99% of the time for true positives and also has a false positive rate of 1% (p-value) then you will still have a large number of cases where you identify the disease where it isn't actually present.
So for examples in a population of 1,000,000 you may have 100 cases of which your very good test finds 99 and only misses one. But it also finds 9999 cases that are not actually real. You false cases vastly outnumber your real cases. So if you are diagnosed your probability of having the disease is 99/10098, or less than 0.1% and that is with a p-value of 0.01!
The same thing happens if I do multiple tests. If I set a p-value of 0.05 which is quite typical there is a 1/20 chance of seeing a result when it is not there. So if I do 20 tests on average 1 of them will show significance. This is easily corrected by Bonferroni's method amongst others.
In genetic analysis you get complex problems with tens of thousands of variables that are all tested simultaneously. You also have massively under-powered studies because your sample numbers are small and so, sample size << number of variables. So you will always be doing many more tests than are justified by the resulting data and you will almost always be over-fitting the model.
This is why the false discovery rate is so important in genetic analysis but really when you look deeply at this it will still mean that even with careful use of FDR most genetic results from big data experiments will turn out to be wrong.
Here are the professionals talking about false discovery rates.
Selective Inference and False Discovery Rate I
Selective Inference and False Discovery Rate II
Estimating Local False Discovery Rate in Differential Expression
Interpreting p and q values in Genetic Analysis
Friday, 20 March 2015
Science free Science
When I started protein modelling we studied one protein at a time and tried to work out in detail how it functioned and what it did and each lab was a world leader in that protein. Then along came high-throughput and structural genomics.
In some ways this was good as we had missed all the connections but it also means that a lot of specialist expertise was lost and detailed studies that did not have the high-throughput angle were not funded. Losing that expertise was bad, we lost expertise on transcription factors and individual protein family databases disappeared to be replaced by the EBI and NCBI monoliths.
Then once these early big data adopters, bioinformaticians as we called them then found that they were still missing the connections that high-throughput was supposed to give them the new buzz became systems biology. Really this was just dusting off the work of Bertalanffy and others who had tried before but were data poor and so couldn't find any solutions. I was at a meeting about Systems Biology at Exeter and there was one glum group the biological phenotype researchers. They had been doing systems biology all the time and now they were going to be bull-dozed over by the bioinformatics revolution as the big data people with no local expertise demolished their field and took all the funding. The perfect analogy is the local stall (Mon and Pop store for US readers) when Wall-mart arrives. So another field got concreted over by the bioinformatics juggernaut. The real sense of systems biology was lost and it failed because what is needed was a paradigm shift in the way of thinking and big data is rooted in reductionism and computability
Now they want to get big data from health services because despite all of this analysis and all the work of the last 2 decades we still have no idea what we are doing and we haven't cured cancer. So now the bioinformaticians are moving into health-care data. Now they are going to concrete over the public health specialists and epidemiologists as all the big money is going to be directed into these monolithic projects that will yet again fail to find a cure for cancer or understand how living things work.
So I feel sad for the loss of all those disciplines and all those experts and I have to include myself among the bioinformaticians. At least I can say I was never a very pushy or successful one. I admire a lot of the people driving the steam-roller. They are good and honest scientists most of them (there are always exceptions more interested in themselves than science). But I wish they would just stop the Big Data juggernaut for a quiet coffee break and have a think about what we can do not to wipe these fields out, but to learn from them. The real article that said we had all gone mad was Chris Anderson's The End of Theory in Wired, Now I also like Peter Norvig and his work and ideas but in his quotes it goes too far. Big data is good and important and I am glad we live in the Google world where I no longer have to memorise endless tedious facts. I am glad we have machines that can deal with massive amounts of data, using whatever your favourite result finding algorithm is, be it SVMs, neural nets, solitons or whatever else gets you excited.
But a former student of mine who was doing a post-doc at a very prestigious US University after completing a PhD at Cambridge asked one of the leading bioinformaticians (he has many hundred publications) about the biological meaning of the patterns found and about how the data had been collected and what the limitations were and he shrugged his shoulders and said he had no idea. That is what we are missing and that is why Big Data fails, because it has no context and no big picture. If I show you a picture of Madonna from the 1990s Erotica Tour and ask you to say what you see I will say Madonna in her silly pointy bra outfit. Another generation will not recognise her, some might say Lady Gaga, some might say a singer, some might say a woman in underwear. Knowledge out of context does not work and Big Data is leading us to Science free Science. It will give us answers like 42 is the answer to Life the Universe and Everything but we won't know why.
So lets stop concreting over the epidemiologists, the systemists, the experts on a single protein, the public health specialists and lets start listening to them rather than the white noise of high throughput data.
In some ways this was good as we had missed all the connections but it also means that a lot of specialist expertise was lost and detailed studies that did not have the high-throughput angle were not funded. Losing that expertise was bad, we lost expertise on transcription factors and individual protein family databases disappeared to be replaced by the EBI and NCBI monoliths.
Then once these early big data adopters, bioinformaticians as we called them then found that they were still missing the connections that high-throughput was supposed to give them the new buzz became systems biology. Really this was just dusting off the work of Bertalanffy and others who had tried before but were data poor and so couldn't find any solutions. I was at a meeting about Systems Biology at Exeter and there was one glum group the biological phenotype researchers. They had been doing systems biology all the time and now they were going to be bull-dozed over by the bioinformatics revolution as the big data people with no local expertise demolished their field and took all the funding. The perfect analogy is the local stall (Mon and Pop store for US readers) when Wall-mart arrives. So another field got concreted over by the bioinformatics juggernaut. The real sense of systems biology was lost and it failed because what is needed was a paradigm shift in the way of thinking and big data is rooted in reductionism and computability
Now they want to get big data from health services because despite all of this analysis and all the work of the last 2 decades we still have no idea what we are doing and we haven't cured cancer. So now the bioinformaticians are moving into health-care data. Now they are going to concrete over the public health specialists and epidemiologists as all the big money is going to be directed into these monolithic projects that will yet again fail to find a cure for cancer or understand how living things work.
So I feel sad for the loss of all those disciplines and all those experts and I have to include myself among the bioinformaticians. At least I can say I was never a very pushy or successful one. I admire a lot of the people driving the steam-roller. They are good and honest scientists most of them (there are always exceptions more interested in themselves than science). But I wish they would just stop the Big Data juggernaut for a quiet coffee break and have a think about what we can do not to wipe these fields out, but to learn from them. The real article that said we had all gone mad was Chris Anderson's The End of Theory in Wired, Now I also like Peter Norvig and his work and ideas but in his quotes it goes too far. Big data is good and important and I am glad we live in the Google world where I no longer have to memorise endless tedious facts. I am glad we have machines that can deal with massive amounts of data, using whatever your favourite result finding algorithm is, be it SVMs, neural nets, solitons or whatever else gets you excited.
But a former student of mine who was doing a post-doc at a very prestigious US University after completing a PhD at Cambridge asked one of the leading bioinformaticians (he has many hundred publications) about the biological meaning of the patterns found and about how the data had been collected and what the limitations were and he shrugged his shoulders and said he had no idea. That is what we are missing and that is why Big Data fails, because it has no context and no big picture. If I show you a picture of Madonna from the 1990s Erotica Tour and ask you to say what you see I will say Madonna in her silly pointy bra outfit. Another generation will not recognise her, some might say Lady Gaga, some might say a singer, some might say a woman in underwear. Knowledge out of context does not work and Big Data is leading us to Science free Science. It will give us answers like 42 is the answer to Life the Universe and Everything but we won't know why.
So lets stop concreting over the epidemiologists, the systemists, the experts on a single protein, the public health specialists and lets start listening to them rather than the white noise of high throughput data.
Monday, 2 March 2015
Amazon Vine and helpful votes.
I am a reviewer for Amazon Vine and I also review outside the Vine program. Since joining Vine I have seen my helpful review percentage fall and I have dropped out of the Top 1000 reviewers. So I thought I would check if there is a statistical difference between helpful review percentage for vine items and non-Vine items. So I did a simple chi-squared test.
The Chi-square statistic is 17.3343. The P value is 3.1E-05. This result is significant at p < 0.05.
That is a very significant difference. There is quite a lot of bias because I have many more non-Vine reviews that have been there for much longer than the Vine reviews and my most helpful review is a non-Vine review but this does still support what a lot of Vine members say that some people systematically go around clicking on the unhelpful review button for anything Vine reviews.
The Chi-square statistic, P value and statement of significance appear beneath the table. Blue means you're dealing with dependent variables; red, independent.
The Chi-square statistic is 17.3343. The P value is 3.1E-05. This result is significant at p < 0.05.
That is a very significant difference. There is quite a lot of bias because I have many more non-Vine reviews that have been there for much longer than the Vine reviews and my most helpful review is a non-Vine review but this does still support what a lot of Vine members say that some people systematically go around clicking on the unhelpful review button for anything Vine reviews.
Subscribe to:
Posts (Atom)