Ground Truth Challenge: The Rules

These rules are also available on Github here.

[version 1.2.1: Updated July 26 2021]

These are the rules for our Ground Truth Challenge, an experiment in "quick and dirty" sensemaking. For context and background on the challenge itself, head here.

We will give a $100 Amazon gift card (or a similar donation to a charity of your choice, or applicable alternative) per documented non-corrected false statement spoken by Bret Weinstein, Heather Heying, Pierre Kory, Robert Malone or Steven Kirsch in the four podcasts listed in the rules below, or for each claim that cannot be adequately cited. Additional rewards available for indicating sections that should need a citation, and also for helping respond to falsification attempts and providing references to important passages. All rewards will be given out up to a total of $10,000.

To participate, please read the rules below.

Please note: These rules may be reviewed throughout the challenge. This is a beta process and we will be closely monitoring throughout to identify improvements or correct flaws. We will make a best effort attempt to communicate any rule changes through the @BetterSkeptics Twitter account.

Summary

• Submissions of types Falsification and Unsupported Claim will be accepted starting July 16 2021 at 11pm UTC and ending July 23, 2021 at 11pm UTC or when the $10,000 total has been reached, whichever happens first. See Submission deadline heading further down for notes on exceptions. Also note that the challenge will likely run over for a few days after that deadline, since every challenge validated in the first round opens up a 48h window for responses. (See section 3 on Responses)
• See "Submission rules" below for how to make your submission and claim rewards of $50-$100 depending on the contribution. Any submissions that don't precisely follow the rules will be ignored.
• Referees will be reviewing submissions on a rolling basis, but will have two more days after the end of submissions, possibly more if required, to process as many entries as possible.
• Everyone can observe the process by opening this Google spreadsheet where all submissions will be entered and reviewed by the referees and volunteers.
• This is a two phase process: Stage 1 is about submitting Falsification candidates and Unsupported Claim candidates, while Stage 2 is about refutations of the Falsification candidates that have been validated, as well as providing effective citations for the validated Unsupported Claim candidates. See next section for more details.
• These rules may be reviewed and updated throughout the challenge. This is a beta process and we will be closely monitoring throughout to identify improvements or correct flaws. We will make a best effort attempt to communicate any rule changes through the @BetterSkeptics account on Twitter under the hashtag #gtcrules.

Two stage process

We designed this challenge as a two-stage process. In a nutshell:

Stage 1

• Participants file submissions.
• We check each submission according to our rules, to make sure they meet basic criteria. Ideally, this step should be automated in the future. Think of it as the spam filter.
• Submissions that pass the filter get submitted to referees for asssessment.
• Once a submission has been reviewed by each referee, it is marked as either "valid" or "invalid" based on the total score awarded.
• Invalid submissions end here. Valid ones go on to Stage 2

Stage 2

• Here, validated submissions get announced on Twitter, and people have 48 hours to submit rebuttals.
• Uncontested submissions get automatically awarded the prize money
• Contested submissions get reviewed again by the referees, now taking into account the new evidence. If the claim is reversed, the response that reversed it gets the prize money. If the claim withstands the challenge, however, it goes on to get the award.

Subject materials

We've chosen and transcribed four recent podcasts featuring Bret Weinstein which discussed COVID-19 vaccines and/or Ivermectin. These are:

• Covid, Ivermectin, and the Crime of the Century. June 1, 2021. Video - Transcript
• How to Save the World in Three Easy Steps. June 11, 2021. Video - Transcript
• Joe Rogan Experience #1671 - Bret Weinstein & Dr. Pierre Kory. June 22, 2021. Video - Transcript
• Bret and Heather 87th DarkHorse Podcast Livestream: We Must Drive this Virus to Extinction. July 12, 2021. Video - Transcript

Submission types

There are four types of submissions possible:

• Falsification: You wish to prove that a statement in the podcasts is demonstrably false. Validated entries will be awarded a $100 Amazon Gift card or equivalent, provided they are not successfully overturned by a response.
• Unsupported claim: You wish to prove that a statement in the podcasts is unsupported by readily available evidence. Validated entries will be awarded a $50 Amazon Gift card or equivalent, provided they are not successfully overturned by a response.
• Response to a falsification: You wish to overturn a verdict by referees validating a falsification entry. Successful falsification responses will be awarded a $100 Amazon Gift card or equivalent.
• Response to an unsupported claim: You wish to overturn a verdict by referees validating an unsupported claim entry. Successful unsupported claim responses will be awarded a $50 Amazon Gift card or equivalent.

Submission rules

In the case of duplicate submissions, we will only consider one of the claims, based on the Twitter timestamp and/or strongest case put forward.

Please read the rules below carefully. Submissions that do not meet all the specifications below will be rejected so make sure to read them in detail and help the referees make the best decision possible. Unless overturned by a successful response, referee decisions are final and will not be reversed for the duration of the process.

1. Falsification submissions (Stage 1)

Here, we are looking for proof of non-corrected false statements. Correction may have been provided in the same transcript or later. Please note, the evidence you bring up must have been available at the time that the podcast was published. Submit a tweet (or a thread of up to 5 tweets) covering the following:

• Your first tweet must start with "@BetterSkeptics #gtcentry Falsification:". Any submissions that do not include this hashtag will be ignored.
• Target quote: You must include the exact part of the transcript (and which podcast it was from) that you believe you can falsify. Please include the whole sentence(s) in question and timestamp if possible. Attach a screenshot to the tweet if necessary.
• Supporting material: Link to resources that support your reasoning. Please keep your resources as narrowly focused on your claim as possible. Make sure to focus on primary materials and factual statements, not "he said she said" which by definition cannot be used to prove factual statements unless those statements are about who said what.
• Reasoning: Your argument that falsifies the target quote. Please keep your reasoning as narrowly focused on your claim as possible, and make it easy for the referees to identify any necessary facts in your supporting materials.

Referees will grade submissions based on the following scale:

• 5/5 challenge demonstrates target quote to be logically impossible
• 4/5 challenge demonstrates target quote to be practically impossible
• 3/5 challenge demonstrates target quote to be false with very high likelihood
• 2/5 challenge demonstrates target quote to have likelihood of being false
• 1/5 challenge demonstrates target quote to have low likelihood of being false
• 0/5 challenge is fallacious, invalid, unsound, unclear, or does not present sufficient evidence.

2. Unsupported claim submissions (Stage 1)

Here, we are looking for proof that a statement key to the central message of the podcast cannot be supported by readily available evidence. Submit a tweet (or a thread of up to 5 tweets) covering the following:

• Your first tweet must start with "@BetterSkeptics #gtcentry Unsupported claim:". Any submissions that do not include this hashtag will be ignored.
• Target quote: The exact part of the transcript (and which podcast it was from) that you believe requires citation. Please include the whole sentence(s) in question and timestamp if possible. Attach a screenshot to the tweet if necessary.
• Reasoning: Explain as clearly as possible to the referees why this particular claim is not easy for the listener to find citation for.

Referees will grade submissions based on the following scale:

• 5/5 citation request demonstrates target quote as essential to be cited
• 4/5 citation request demonstrates target quote as useful to be cited
• 3/5 citation request demonstrates target quote as desirable to be cited
• 2/5 citation request demonstrates target quote as optional to be cited
• 1/5 citation request demonstrates target quote as excessive to be cited
• 0/5 citation request is fallacious, invalid, unsound, unclear, or does not present sufficient evidence.

3. Responses (Stage 2)

All entries that score a total of 9 points or more in the first round move on to the second phase, for rebuttals.

Once this happens, the @BetterSkeptics account will automatically broadcast the submission with the hashtag #gtcvalid, giving 48 hours for people to submit any responses.

To contest a "Falsification" verdict, you must:

• Reply to the tweet from @BetterSkeptics announcing the validation of the initial claim
• Limit your response to a thread of up to 5 tweets
• Provide your reasoning: Explain as clearly as possible to the referees why this falsification should not have been validated. Provide sources if needed.

To contest an "Unsupported claim" verdict, you must:

• Reply to the tweet from @BetterSkeptics announcing the validation of the initial claim
• Limit your response to a thread of up to 5 tweets
• Provide a citation: This is the material that you believe factually backs up the challenged statement.
• Provide your reasoning: If needed, explain as clearly as possible to the referees why this particular cited resource factually backs up the challenged statement. Please prefer as primary sources as possible.

Once the response time is completed, referees will review both the submission tweet that was made on the @BetterSkeptics account and the original submission tweet's replies when assessing responses. They will then rescore the original submission based on the new evidence provided.

If the referees' view of the original submission is altered by the new evidence, each referee will indicate the reply/replies that helped change their minds. Each referee controls 1/3 of the total award for each reversal, so if all 3 referees point to the same reply, it will receive the full award. If they are split, or if a referee indicates multiple responses as helping change their mind, the award will be split accordingly.

Qualifying the successful challenges

Once the project ends, referees will qualify the nature of all the challenges that passed both rounds.

I.e., for each challenge, referees will assess: Does it refute or invalidate a central claim of the podcast? Is it a nitpick? Does it mainly provide a useful clarification, rather than a falsification? And so forth.

One tool we may use here is Paul Graham's disagreement hierarchy.

Submission limit

Submissions are capped at 5 per person. However, this can be increased: for every challenge validated in the first round, a person can submit another 3 submissions. Once someone's submissions are used up, all further posts will be rejected.

For each original submission that gets validated in the first round, 3 new submissions will become available, but the submitter will have to make new ones. This is to ensure that the submitter selects the most potent entries at their disposal, not just the ones they had originally sent beyond the limit.

A note on strategy -- given that the sanity check and referees may reject submissions for reasons that can be upon resubmission, it is unwise to use all your submissions at once. Make sure to prepare your submissions as carefully as possible and in particular ensure they meet the submission completeness and uniqueness criteria.

Submission completeness

Each submission MUST contain all the necessary material to document your claim or contest. Referees are not obliged to do their own research to document a claim, it must be the claim or contest itself that explains its case clearly and concisely. If a claim requires more than 10 minutes from the referee, it may be rejected even if the necessary material may be, in principle, found.

Submission uniqueness

Submissions referring to the same claim and using the same reasoning for Falsification or Unsupported claim will be considered duplicates. This is true also in the case where the claim being made is in different parts of the target material, so long as it is the same claim in essence.

Retweets of already submitted claims will automatically be considered duplicates of the claims submitted. Quote tweets whose content meets the original criteria will be accepted as submissions.

If several challenges are validated against the same claim and succeed past the refutation stage, only the strongest -- or the first one by time of initial submission in the case or identical scores -- will be rewarded.

Submission deadline

All challenges must be submitted before the assigned deadline. However, because referees are unlikely to have reviewed everything before the deadline, there are two exceptions to this rule:

• Responses can be submitted anytime within 48 hours of a claim being validated, including if the claim is validated after the deadline has passed.
• If a submitter has hit their limit and gets entries validated after the project deadline, they get an extra 24 hours -- counting from any new validations -- to submit additional challenges (3 per newly validated claim) they may have. Example: Jane has submitted 5 entries, the max number allowed. They are not reviewed before the project deadline, and one gets validated after the deadline passes. Jane may submit 3 new entries, counting 24h from when her challenge was validated.

Sanity checking

The volunteer members will filter submissions on a rolling basis, based on criteria such as completeness/relevancy/spam. Based on this submissions will either be submitted to the referees or discarded.

Referee review

Each approved submission will go on a referee-specific sheet where each referee can mark their opinion and rating on the claim (see below for scoring scale). All the opinions will be aggregated on a final sheet that adds up all the opinions to mark the claim as validated or not.

Awards

Awards will be given to:

• Unique submissions that pass the first round and are uncontested in the second round
• Unique submissions that pass the first round and sustain the challenges in the second round
• Responses that successfully reverse a submission.

Questions or feedback?

Send us a tweet @BetterSkeptics with hashtag #gtchelp or send us a DM and we'll aim to respond as fast as possible.