Subscribe
Sign in
Trusted News — This plugin analyses the quality of news articles.
Trusted News uses AI to assist newsreaders in evaluating the quality of the online content they read. In its first release, it scores the objectivity for a selected article, testing whether it is written from a neutral perspective as opposed to a personal one.
24
Replies
Best
Alex Gorman
Never before has there been a product timed more perfectly!
(A little dramatic I know but this is the kind of product I’ve been wanting for a while)
Having it based on an objectivity scale, I’m interested about the products ability to work with opinion pieces and interviews? As those types of articles are supposed to have no objectivity.
Really great concept. Might Be what reporting today really needs.
Report
Share
Cool idea! I was wondering how do you decide at which part of the article is the article itself, versus those that aren't part of the article?
I also tried with several articles just now, and most appeared to be ~6x-7x, perhaps the model requires more training before being able to judge better.
Btw I had also tried with a Chinese article, the plugin just stuck there, maybe you might also want to look into handling different languages
Another domain which you can look into references. An article that references to a lot of different places can be deemed more objective, in general, compared to one that did less
@tonyngws good Q. A lot of this is to do with the quality of external text extraction. Right now we go through and include comments as well on the page and headlines, this is a feature to work through. The objectivity model is currently quite linguistic - we want to deploy BERT based models soon to enhance this. We will also add error handling on the languages very soon! Finally on references , this is fantastic - we'll get on that for next release :)
if you wondering how we calculate the objectivity score, here is the answer:
We split the article into sentences, then identify if the sentence is objective or subjective. As a final score, we divide the number of objective sentences by the total number of sentences. For example, an article with 5/10 sentences as objective would have a 50% objectivity score👌.
How to identify the objective sentences 🤔? Objectivity is given when a text is written from a neutral rather than a personal perspective. Phrases like” in my opinion” or,” I think 🤔”, or "I love 💖", or "I like 😍" are used by authors to reflect their individual thoughts, beliefs, and attitudes, these keywords and others are identified as subjective keywords to predict the sentence subjectivity score with the help of library pattern3.en (1) that also includes a value of subjectivity for every word. Therefore we use it with a unique machine learning model to obtain an objectivity score for sentences. Values range from 0 to 1, with a value near to 0 indicating objectivity and a value near to 1 indicating subjectivity. The overall score for subjectivity contained in an article is calculated as the average overall sentences. However, since we want to examine the objectivity and not the subjectivity of a text, the values need to be inverted: Objectivity= 1−Subjectivity. This score is normalized by multiplying it by 100 to attain a consistent score range for all labels.
all this research results are published in a research paper in the attached URL(2).
(1) https://stackabuse.com/python-fo...
(2) https://www.researchgate.net/pub...
@sameh_frihat So, if I was to lay out something totally false as if it was facts, not using "like" and "I think", would it still come out as objective? If so, I think it'd be good to add verbiage mentioning that you do not fact-check, but only check the style of the text, not it's content. Great product otherwise, love to see people try to clean up the cesspool called the internet :P
@sameh_frihat @lgerbeaux this is a great piece of feedback and simple to clarify, we will do that!
@sameh_frihat I appreciate the spirit of this, but doesn't this (paradoxically) promote bias and misinformation by landing square in the "false balance" trap? Objectively presenting lies is a far greater problem than bias towards the truth. If you can figure out how to use ML to fact check, that would be the holy grail.
We launched Trusted News because we felt there was not a great browser extension to be able to analyse the credibility of content being read online, at scale, and quickly.
Most browser tools before this rely on understanding the website, rather than the article itself, and by seeing if the page is on a blacklist that the maker has compiled. This leads to inherent judgement bias on which sites go in the list or not.
Judging articles purely on how they frame things is harder to do, but the right way to go as we judge writers for what they write, not who they are.
We have a number of features to add to this, including:
- Show you which words and phrases led to the objectivity score
- Change the scoring to be more of a Grade Scale e.g. "Very Objective, Neutral, One-Sided, Aggressively Biased"
- Improving the objectivity classification AI using additional training data
- Automatically improving the scores using user feedback
- Showing you a link to a diametrically opposite objectivity article, so you see another viewpoint
For now, keen to see your feedback!
An amazing tool for discerning the objectivity of articles. Does not work with some URLs, but produces results most of the time.
I recently created an application (Fallacy Filter) which analyzes News Articles and assigns Similarity Scores to both the Document as a whole, and each of its individual statements with the help of Natural Language Processing.
It is also currently featured on Product Hunt.
You should add a page in your site where you check every article politician shared/tweeted. Then we can see if they proof check or just post whatever they want
Hello Product Hunt!
in the next 2 weeks trusted news V2 will be launched with the following new features:
- highlight objectivity key phrases, to allow users to understand from where we get the article objectivity score.
- domain objectivity score, for example, the BBC overall objectivity score:75%* and the score will be updated automatically.
- recommend a similar article objectivity score.
- improve objectivity model accuracy.
@jake_goldstein yes absolutely. You can email info@factmata.com and we can sort out a key for you.
@smrutiranjan_swain @sameh_frihat can reply on the F1 scores of our objectivity model on our own test set
@smrutiranjan_swain we are using a regression model, we achieve 9.35 RMSE.
