Bisexuality and Long COVID: Significant Mistakes Were Made Attempting to Replicate a Connection

I honor a blogger’s request to let him know if he gets something significantly wrong and I suggest a lot of people will have much to tell him.

May 9, 2023

A trusted source alerted me that I would find something upsetting in Scott Alexander’s Astral Codex Ten Substack article, “Replication Attempt: Bisexuality and Long COVID.” My friend was correct, as usual.

Amazingly, Scott has an open request that we point out his mistakes in what he writes.

At his Astral Codex Ten website, Scott Alexander has a Mistakes page that states

I don't promise never to make mistakes. But if I get something significantly wrong, I'll try to put it here as an acknowledgement and an aid for anyone trying to assess my credibility later.

I like to point out mistakes before they are made.

A growing number of groups and individuals pay me a modest retainer to pre-review their research or writing plans. I give them advice about what their project is likely to yield. Sometimes I help them revise the plans a bit. Some of my best advice I can offer to some clients is not undertake overly ambitious projects. They would not be able to marshal the expertise and resources that are needed. A satisfied client might say “Ouch! I would rather have this report ahead of time, rather than as a post-mortem on an exciting idea that flew high, but then crashed and burned after a lot of effort.”

Scott does not yet subscribe to my service, so I am working for free. I hope he will consider this my unsolicited report a favor worthy of a tip. He might want to send me some money to PayPal.

I did not write this report primarily for Scott. I intend it as a favor to people suffering from Long Covid and other poorly understood physical health conditions, as well as the bisexuals people whom Scott claims are at risk for COVID. Or anyone who is offended by anyone making important claims about health that are not supported by evidence.

Many will be offended by Scott’s article for different reasons. My report helps them to understand why the article should not be taken seriously: It is not as grounded in science as it is made to look. However, the article does deliver a slap in the face near the end. I am not sure that Scott meant it that way. Let’s be charitable and accept that Scott believed he was reporting science in the most compassionate way possible.

If we find fault with Scott’s arguments for a connection between bisexuality and long COVID, we should honor his open request and tell him clearly why.

However, Scott is more likely to respond to the feedback if he does not feel mobbed. My readers know I have a special sensitivity to anyone being mobbed. If readers heed me, they will not unduly hassle Scott or bring up things that are not relevant. Please model good behavior because Scott has more followers and you might want to reach them as well. Speak your mind as if they are willing to listen.

A clickbait title and two final paragraphs

I was not thrilled with Scott’s clickbait title. “Replication attempt” implies something was done with a finding that was not worth attempting to reproduce. “Bisexuality and Long COVID” is a juicy pairing of hot topics that may attract the wrong kind of crowd looking for confirmation of some prejudices and a nasty fight, rather than enlightenment.

Let’s jump to his final two paragraphs in a brief article that drew a much longer thread of over 400 comments from readers:

…the signal here is strong enough that I’m going to walk my previous statement back and be much less sure that there aren’t a lot of psychosomatic cases (I still think most likely some cases are organic, just because you usually need a few organic cases to seed psychosomatic conditions, and there’s no reason why a serious and novel virus shouldn’t cause occasional post-viral syndrome).
If this were true, we could think of the excess psychosomatic cases of long COVID as a culture-bound mental illness, and consider compassionate treatment for the organic and unavoidable psychosomatic cases - alongside unawareness campaigns aimed at minimizing avoidable psychosomatic ones.

The idea of an “unawareness campaign” is clever, but the rest of that sentence will make some people apoplectic. Scott might be accused of trivializing their profound and enduring distress, their inability to get back to normal, and their worries whether they can ever get back to their old normal. People with long COVID hear the dog whistle sounding in “psychosomatic.” That word seems innocent enough. but it greases the slippery slope from their having puzzling symptoms of some post-viral physical condition to something being “all in their heads.” In common parlance, people with long Covid are getting hysterical about nothing, perhaps to make someone feel sorry for them.

Skipping back to Scott’s opening sentences:

I learned from Pirate Wires that CDC data show bisexuals were about 50% more likely than heterosexuals to report long COVID.

Is this just because more women than men are bisexual, and more women than men get long COVID? Not exactly; in the data they cite, women (regardless of sexuality) have an 18% rate, and bisexuals (regardless of gender) have a 22% rate.

I have trouble believing such a strong association exists between bisexuality and reports of long Covid. I wonder if Pirate Wires really captured a lurking powerful finding that no one else noticed in the CDC data or if they merely cherrypicked a finding that disappears with proper control of confounds.

I won’t explain “confound” here, but I will give some examples. A few decades ago, researchers discovered that drinking coffee in America was associated with dying from lung cancer, whereas tea was the apparent culprit in the United Kingdom and Ireland. Eventually, the researchers noticed that people were likely to smoke when drinking coffee or tea. No effect of hot drinks was found for nonsmokers. Then there is the excess of Lisa Minelli vinyl in their record collections allegedly found to distinguish early victims of HIV/AIDS…Could a swift record burning have saved lives?

At this point, if I were simply wanting to pillory Scott, I might criticize the logic in the question he asked in the passage above or his use of numbers to answer it. But let’s get to the larger issues in the survey of his readership he handcrafted to get these numbers. Scott started from scratch. He went to trouble of writing how own questions and putting together a long online survey for his readers in order to generate lots of numbers. He analyzed some of his data and included numbers in the two important tables in his article. I can give a lot of valuable feedback about this effort. I’ll start with advice that is too late for Scott this time.

“Scott, you got yourself in serious trouble making up questions and playing survey research scientist. Survey research is an art and craft done with an increasingly sophisticated methodology. If you thought you needed to do a survey, you should have used off-the-shelf instruments with validated cutpoints, as well as the standard questions used to assess gender and sexual preference. To analyze your data, you should have hired graduate students who know something about causal inference from noncausal survey observational data.

Health survey research is serious business that should not be undertaken by amateurs

ISR in the Snow By Sheila Babbit Wikipedia

I have great respect for the work that goes into developing a valid survey, recruiting representative sample motivated to complete the survey with minimal missing data, and then analyzing and interpreting the results. Before joining the faculty of the University of Michigan School of Medicine, I spent some time at the University of Michigan Institute of Social Research (ISR) with its world-class Survey Research Center (SRC). The SRC of ISR offers training in sophisticated survey design and analysis. When somebody was writing a detailed budget or work plan for funding of a large survey, I would participate in sessions in which anyone who was enthusiastic about adding some topics to a survey had to suggest an equal number to drop. We would consult experts who would providing cost-analysis of adding questions and the effects on interesting and retaining participants until they completed all questions.

The stint at ISR served me well when I had to get my own grants and assist others in the family practice and psychiatry departments.

Sampling Scott’s micro-brew ands giving him feedback from a survey research perspective

I examined Scott’s hand-crafted ACX survey here and you can too.

Here is a sampling of reasons why its use with a sample of his readers cannot produce interesting and valid results, despite what must have been an enormous effort.

Scott,

1. Your survey is much too long and your sample will be biased by lots of people giving up without leaving you their data or they will follow your advice and only answer the questions they chose. The instructions accompanying the questionnaire suggest that this is OK. You would have been better to have communicated that completing this survey is serious business and they should do the best they can to answer every question.

2. Your questions that are intended to assess mental disorder are odd and they likely provide uninterpretable and misleading data. It is never a good idea to use data from screening question data rather than a structured interview in which the researcher can explain what is meant by questions and probe respondents’ responses to see if they understood what was being asked. This is a big deal. Even with well-validated questions and cutoff points, most endorsements of mental disorder will be false positives and so you will substantially overestimate the prevalence of disorder and distort its correlates.

3. The wording of multiple-choice response options is too creative to work. You are left not knowing why respondents endorsed particular mental disorders. They might endorse major depression because they have been in treatment for years or only because they consulted a primary care physician and were required to complete a depression screening questionnaire. As is all too common that physician told the respondent they were depressed but the physician accepted a false-positive without conducting the necessary follow up interview.

4. I don’t want to go into too much detail, but here is another example of many problems with you making too much of your data concerning mental disorder. I do not know what to make of self-report borderline personality disorder. Some women are told they have BPD by their boyfriends when they get upset. Some depressed women are told by their therapists that they have a chronic condition, BPD, while the women are still depressed. When they recover, many formerly depressed women would not be diagnosed as BPD. Many women are told by their therapists that they have BPD because they became particularly annoying in a particular therapy session. They do not meet formal diagnostic criteria for BPD.

5. Your survey asks an extraordinary number of detailed questions about politics and attitudes toward issues like abortion or climate change. This will exhaust some respondents, but others will get a clear message of what is behind your survey. On that basis, they may change their answers, motivated to either please or defy you, depending how much they agree with your opinions. Please keep in mind that survey questions are not answered in isolation but are influenced by context.

6. I would not take your questions too seriously whether respondents have had COVID. It is not surprising that you report that you “got much lower rates of Long COVID than the CDC, more like 3% than 20%.” Do you think your readers should believe you or CDC or neither?

I am near the end of the time I allotted to probing Scott’s survey, I think it is dead in the water as a way of collecting meaningful data. He could seek a private consultation if he does not think so. I am sure my readers who agree with my analysis could keep probing and discover some unusual things. If you have time to spare, you could honor Scott’s request for feedback

Here are some hints. Why would Scott ask about burping and give survey participants so many options for an answer? Why would Scott ask if participants had an imaginary friend as a child? By what theory does he think that is related to later sexuality or Covid? Why would Scott ask so many questions about medical conditions and symptoms that would require a knowledge of medicine very few of his participants would have?

Overall, Scott asks so many leading questions in his survey and offers so many different multiple choice options that should greatly confuse participants and anyone who seeks to re-analyze the data he gathered. The biggest question is why he chose to cherry pick an alleged association between bisexuality and long COVID out of all the associations he could possibly tease out.

[A chance for survey participants to endorse Neoreactionary as their political affilaition and that they would prefer to have a CEO-like monarchy]

We do not have to know or care how amateurish the survey methodology is to dismiss the numbers in his tables as not telling us anything. Let’s look at a few numbers and see why.

Numbers may not lie, but they can provide untrue answers.

My experience is that most of the time readers do not actually check numbers in tables and instead rely on what authors say about them. Scott actually refers to the numbers in the table in his text. I could readily see that even if he had a superb survey to produce numbers, these would not be very interesting. Here’s why the numbers should be dismissed by anyone serious about relying on evidence

The numbers in key cells are too small. To understand that, we need to appreciate that numbers are not exact but are estimates and have confidence intervals indicating how precision there is to them. Scott does not provide these confidence intervals, although he gives in his text the results of analyses he claims show their statistical significance, and these analyses should have allowed hand calculation of confidence intervals.

This does not matter because we can simply eyeball the numbers and see something is wrong. Scott tells us elsewhere he had 7,341 people complete his survey. That number is huge, but we do not even have to know that to see he is dealing with very small differences. Woman who say that they are bisexual rather than some other sexual orientation may not necessarily be different in other women. It is strange that there are only 38 avowed homosexual women in Scott’s study and only one said they had long COVID. Of 254 women who endorse bisexuality, only 18 said that they had long COVID. Of 514 women who endorse heterosexuality, 17 say they had long COVID. Don’t even bother with the Other category. These numbers are so small, we cannot take them too seriously. We definitely cannot say that we are confident that these numbers would be significant with a larger sample. We must say that there is nothing to say.

Keep in mind these variables come from a cross-sectional survey. I am confident that each of the variables sexuality and long COVID has larger associations with other variables and we could not rule out a confound. Recall what I said about hot drinks and lung cancer, but these data are messier than that.

Scott’s blog have thousands of comments from readers, almost all of whom offer speculative theories about why these results were obtained. Their theories are only opinions. They are being confused by noise when there is no signal. Keep in mind that these readers were a prime source of participants completing the survey. They have strong and often unusual opinions, which may account for the unusual numbers, but they are mere opinions, neither strengthened nor weakened by results of the survey.

Amateurs pull all-nighters and speculate when real survey scientists have gone to bed.

The offense of suggesting long COVID is psychosomatic and bound by culture

I understand why anyone committed to settling important disputes with evidence will take offence at Scott Alexander and his thousands of follows making a fuss about bisexuality as a cause of long Covid. Please note my doubt Scott and his crew want to argue that long Covid causes bisexuality in women, but not men and does not cause homosexuality or heterosexuality in either men or women. but how could they refute that hypothesis?

Scott throws around phrases like “excess psychosomatic cases, ” and “you usually need a few organic cases to seed psychosomatic conditions, and there’s no reason why a serious and novel virus shouldn’t cause occasional post-viral syndrome” and “compassionate treatment for the organic and unavoidable psychosomatic cases - alongside unawareness campaigns aimed at minimizing avoidable psychosomatic ones.”

These phrases sound medical and scientific only to people who are unfamiliar with medicine and science. Worse (if there can be a worse), the words have nothing to do with the survey data Scott took such pains to gather and interpret.

But let me step back into this discussion and offer my expertise about the meaning of “psychosomatic” in science and in pseudoscientific attacks on patients with misunderstood physical health conditions. In science, “psychosomatic” has often been attached to physical health conditions before their diagnosis and biology are properly understood. In uncorrected popular prejudice, the term “psychosomatic” has often been used to deny patients appropriate medical treatment for a medical problem.

Salvatore Minuchin’s family psychosomatics was all the rage in the 1970s and 1980s for blithely blaming families for their members having diabetes, asthma, or anorexia. Most experts who rely on evidence would agree that Barbara Anderson and I provided a persuasive evidence in two papers that Salvatore Minuchin and his colleagues grossly misinterpreted their own data and ignored data from others.

I take the blame for the title of our second paper. I was upset with so many people hurting so many families and children arguing for from data that were not believable that I made sure our title announced we were done.

Coyne JC, Anderson BJ. The “psychosomatic family” reconsidered II: Recalling a defective model and looking ahead. Journal of Marital and Family Therapy. 1989 Apr;15(2):139-48.

The paper was not cited much because after it appeared, because most people were persuaded to stop talking about the “psychosomatic family” and moved on. Dr. Anderson went to some wonderful work showing that if we stopped assuming that poor control diabetes could be improved by collaborating with teens in plans for managing their illness. If the teens themselves were not up to the task, we would educate their parents to get involved in a non-blaming way. Clinics stopped reporting “brittle diabetes.”

I wish exorcizing bad ideas about “psychosomatic” were so easy. The task is hard because ideas get so entrenched when there is no look at quality data.

Scott Alexander can plead that he is being compassionate in offering the opinion that patients with long COVID are a mix of pure physical health conditions and psychosomatic ones. Evidence-based professionals do not make that distinction. Even when they are confused, they recognize that is splitting the mind from the body.

I have completed my task of showing just some of the ways in which he made huge mistakes in generating and interpreting survey data. I invite responsible citizen-scientists from communities of chronically ill patients hurt by misuse of the label “psychosomatic” how they have been stigmatized and denied proper medical tests or even an admission that the inferior care that has been inflicted upon them is based on prejudice.

I really do not have much advise to readers, especially women, who are offended by the opinions that Scott and his many followers voice about sexual preference. I would not know where to begin. This is not a dispute about evidence.

I would welcome a response from Scott and his colleagues that explained why they believe there is a connection between sexual preference and developing long Covid, but I ask for evidence, not opinion or anecdotes.

Sometimes, the best thing to do is just walk away from a futile argument. Jokes will be funny to only your side about what how frustrating it gets when a farmer attempts to teach a pig to dance or sing.

Like Scott Alexander, I encourage readers to point out where I have made serious mistakes. I would especially like if readers would point out misspelling, grammatical errors, or any signs that my dyslexia met up with autofill and slipped past my proofreading. That is embarrassing but happens all the time.

CoyneoftheRealm on Substack