Fever IQ: Pi-community results

10/22/2020

August 31

Dear Pioneers,

3.5 million of you have made history by participating in FeverIQ, making it the first global deployment of confidential computing. Your contributions are the foundation of a private COVID risk calculator that is freely available to everyone, wherever they live. Thanks to you, anyone in the world can now screen for COVID more accurately than just taking their temperature or using a static checklist. You can estimate your risk of testing positive at https://healthcheck.feveriq.com/calculator, and businesses and schools can use the same system for free.

From a technology perspective, you are participating in the world’s first global use of confidential computing to protect people’s privacy, whilst still being able to help scientists and researchers understand a pandemic, proving that privacy-preserving cryptographic techniques are ready for prime-time!

The raw symptom data you contributed never left your phone. The data we have are insights confidentially derived from your data, and are freely available to doctors and scientists, as noted on feveriq.com. If you wish to see or access the derived data, simply email contact@enya.ai, specifying your academic or medical affiliation. Since the files are quite large (>0.5 GB), we will work with you to figure out how to transfer those files to you.

Note that we do not have any identifiable information (since we do not collect that information) and we also do not have any raw symptom data, since that information is protected through secure multiparty computation. Through the power of confidential computing, we are able to determine your Hamming distance to preconfigured symptom constellations without your unprotected symptom data ever leaving your phone. This approach has the benefit of strong privacy protection but also has limitations – for example, it is not possible to train new classifiers on the granular/detailed symptom data, since those data never leave your phone.

Thanks again for making history together! Please let us know if you have any thoughts or feedback; just email us at contact@enya.ai.

Best,

The Enya Team

July 15.

Based on data from about 2.5 Million contributors, we have been able to build a machine learning model that predicts the risk of being SARS positive. This algorithm will be useful globally in places where molecular testing is not available or where the local testing capacities are overwhelmed.

COVID risk estimation is inherently difficult due to the biology of the SARS-CoV-2 virus. The virus has been associated with more than 20 symptoms, which seem to occur in different combinations in different people. More problematically, almost all of those symptoms overlap with those of very common conditions such as allergies, cold, and flu, making COVID hard to reliably diagnose based only on (early/mild) symptoms. The figure shows the performance of 16 different classifiers, including CDC symptom list approaches, a recent algorithm published in Nature Medicine, and the Enya classifier. Again, the real difficulty with COVID is not needing to detect the likely presence of some kind of respiratory illness, which would be much easier, it's correctly discriminating among four widespread alternatives with broadly overlapping symptoms - cold, flu, COVID and allergies.

Each curve shows how well a classifier can distinguish between people with and without COVID. The straight diagonal line represents a “coin-toss” diagnostic test that correctly classifies people as having COVID 50% of the time. The more accurate the classifier, the further away it will be from the straight line. The green line (Enya’s classifier) is best as judged by its area under the curve (AUC) – it does as much as 2.7 fold better than other classifiers. Enya’s COVID risk classifier is based on data from more than two million people all around the world and gets updated every few days. Based on data availability, the model also considers location and other factors that make it more sophisticated than other risk models.With this unique (and continuously growing) real world training set with symptoms from almost 2.5M million people, and test results for more than 30,000 people across many geographies, we are able to provide machine learning tools that help businesses and schools make the best decisions and help their employees and students to get the best care and testing when it is warranted.

June 8.

Based on data from about 700,000 contributors, of which 13,311 volunteered their PCR or serological test results for SARS-CoV-2, we were able to compare the performance of four widely used diagnostic criteria. For example, the CDC criteria for "flu" include fever and body aches, whereas the criteria for COVID include fever and shortness of breath. How well do the various criteria perform in the real world, in terms of their ability to discriminate among a cold, a flu, and COVID?

Just like in our preliminary work, we found that people with positive and negative COVID tests reported symptoms all across the four diagnostic criteria. This means is that there are few – if any – truly definite and unique symptoms of COVID, that allow it to be easily differentiated from the common cold and the flu. Interestingly, we did note a different magnitude of reported symptoms. For example, although people with negative and positive COVID tests both exhibited fever and cough, the patient population with a positive COVID test tended to report more symptoms. The figure shows the total scores (sum of the four subscores) of people with a positive COVID test (orange) and with a negative COVID test (blue). As can be seen, the total scores for the positive COVID population had a longer tail, typically due to the COVID+ population reporting more symptoms, relative to the people with a negative COVID test.

We then used standard machine learning and statistical approaches to parametrize a classifier for COVID, using the four subscores we securely compute for each person based on their self-reported symptoms. Since we are using privacy-preserving computation, we cannot construct new classifiers based on individual symptoms, since those are hidden from us. However, we can train classifiers on the Hamming distances of each individual relative to the four preconfigured symptom constellations. The figure shows the intersection of the ROC functions positive predictive value (PPV) and true positive rate (TPR). In this example, a cutoff value of ~0.32 provides balanced positive and negative label success. The cutoffs can be chosen based on how the classifiers will later be used for early detection or diagnosis.

May 12.

Based on 55,000 high-quality datasets contributed by the Pioneers in just 14 hours, which is absolutely amazing, we were able to investigate the predictive power of the current COVID differential diagnostic criteria. We tested whether the diagnostic criteria for COVID, the Flu, COVID-neuro, or the common cold were strongly associated with a positive COVID-test. There were about 3,000 Pioneers who had been tested (5.5%), and of those, 453 (1%) were COVID-positive. Unfortunately, 21% of the Pioneers with a positive COVID test had zero "traditional" COVID symptoms, but instead perfectly fit the diagnostic criteria for the Flu. Similarly, 15% of the COVID-positive Pioneers only had symptoms of the common cold but did not have any symptoms of either COVID or the Flu. What this suggests is that the use of symptoms (e.g. fever or dry cough) to discriminate among COVID, the common cold, and the Flu, results in many mistakes. As more and more Pioneers help, we will be able to improve the statistical robustness of these early trends, and we can even thing about publishing a scientific paper based on data from our Pioneers. This will be the worlds' first example of how a distributed community can help medicine, all without endangering the privacy of all of our Pioneers. The plot shows how people with a positive COVID test scored in four categories, e.g. Flu or COVID. The "Flu" distribution shows that most people with COVID were thought to have a Flu based on their medical symptoms - but in reality, they had COVID.

April 20.

Based on case reports from doctors, preprints, and analytics of social media feeds, we are adding three possible symptoms to FeverIQ: hallucinations, blue/blistered toes, and allergies. The association of these symptoms with COVID is currently unconfirmed and better data are needed.

April 14.

Based on the first several hundred contributors, we are already seeing interesting trends that fit the known biology of the disease and its symptoms. There are hints that a widely circulated symptom checklist (flu vs. COVID vs. cold) is able to categorize patients into "cold" or "COVID/flu", but may not be as good at further discriminating between COVID and flu. Also, there seem to be people with one or more symptoms of COVID, but without fever. If true - and much more data will be needed - this would raise the possibility that simply measuring a person's temperature may not reliably detect all carriers of 2019-nCoV.