Today I am presenting to the community a Sentiment Analysis Report I’ve been working on as a passion project. This will be pretty long and wordy. Highly recommend reading through to understand what’s going on but if you just want the sweet and quick-
Real quick before I dig in, this was the first time I worked on a Natural Language Processing (NLP) Project. The entire process- from learning NLP, figuring out best approach, execution and report creation was done in the past 10 days. I plan on iterating on this next month & refining the process a lot!
The report primarily focuses on 2 things- general sentiment and sentiment towards different types of proposals.
The end goal of this analysis was to validate the sentiments echoed by the community. For example- Can we quantify that the community prefers Tech-focused projects more than Social-Justice projects? Are DAOs still seen in a negative light? As you will see in this report, we definitely can validate this.
The project uses Google AI’s “BERT” model. For those interested- a well-explained read about the model.
Since one of the main goals is to determine what type of proposals the community prefers, I decided to focus on comments made in the “Funding Proposals” section on the talk.harmony.one forum. This offers the widest variety of posts to analyze & is the primary home of discourse surrounding proposals.
Some key considerations made regarding the data collected:
- The “zkDAO” sub-section under “Funding Proposals” was not considered as it has little to no community discourse
- Harmony Team comments were removed since the focus of this is the community
- Proposal Author comments have been removed (to prevent bias)
- Only English comments have been considered
- Only comments made in the month of May (includes proposals posted before May)
- Deleted posts/comments have not been included
A total of 958 comments from 183 proposals have been considered.
1) Proposal Classification
All 183 proposals were classified into 4 categories-
- Tech- Games, DeFi, Tech products. Ex- OpenSwap, EvoVerses, 1Wallet
- Marketing- Primary focus on marketing Harmony via various means. Ex- ONEweekly
- Social- Primary focus on social impact. Ex- Vegan Africa DAO, Pivot DAO
- Hybrid- Tech + Social or Marketing + Social. Ex- Most Regional DAOs
This classification makes it easy to differentiate the core focus of the project and check how well community sentiment aligns with it.
2) Sentiment Scoring
The BERT model makes it extremely easy to score sentiment. You simply feed a sentence to the model and it returns a score between 1 to 5. The scores are explained below-
1- Very Negative
2- Slightly Negative
4- Slighly Positive
5- Very Positive
Below are some examples of how different sentences are scored-
Being trained on millions of texts, the model yields very accurate results. There are some limitations though. Comments must be <= 512 characters in length. There are several comments on the forum that go over this range. The workaround here is to truncate the comment to 512 characters. A “head” and “tail” truncation was done to get 256 characters from the start of the comment and 256 from the end. This method is surprisingly extremely accurate. More can be read about this and several other methods in this research paper.
3) Gathering Insights
With all the comments scored on sentiment, the next step is to analyze the results and collect important insights. Before moving onto the results, this is what I focused on here-
- How polarized is the community sentiment?
- Does the community have a preference for certain types of proposals?
- Which proposals have the best score and which have the worst?
- Are DAOs really viewed so negatively?
The screenshots below are from the PDF report above- link here.
1) Sentiment Distribution
The Mean Sentiment Score comes out to be Neutral at 2.93 (~3)
Despite the neutral score, most comments (~61%) are actually Very Positive or Very Negative
This is a strong indication of highly polarized opinions on proposals on the forum. This polarization is forum-wide, so it’s not that every proposal has polarized discussions, rather, most proposals will only have mostly positive or mostly negative comments.
2) Proposal Preference Analysis By Proposal Type
The Proposal Types ranked in terms of Sentiment Score are:
The community has a quantifiably distinct preference for Tech Proposals over Social Proposals. Social Proposals have by far the most negative sentiment comments at ~62%
3) The Best and The Worst Proposals
Links to Proposals mentioned above-
- ONEweekly Video Series Continuation
- OpenSwap’s request to build Harmony’s all-in-$ONE platform
- SonicSwap 2.0 - transition from a simple yield farm to a complex mini-ecosystem within Harmony!
Projects with a proven track record of impactful work score exceptionally well.
Note: Projects with at least 8+ community comments were considered for the above ranking. This is because proposals with very few comments can have their Sentiment Score impacted by even 1 contrarian comment.
4) Pretty Word Clouds
225+ comments were taken from the Top 15 and Bottom 15 ranked proposals (>= 8 comments)
"DAO" is the 9th Most Frequently Used Word in the Bottom 15 proposals but ranks 84th by frequency in the Top 15 proposals, indicating clear community animosity
Why is the Word Cloud shape a butterfly you ask? Idk, it just looked cool haha.
The community has a clear, quantifiable preference for Tech-related undertakings
Ex- OpenSwap, SonicSwap, DeArt Gallery
"Social Impact"-based undertakings are perceived in a distinct, negative light
Ex- Vegan Africa DAO, SpectrumX DAO
A proven record of bringing value to the ecosystem is highly respected & supported
Ex- ONEweekly videos
DAOs are still seen with a negative association
Transparent, open and honest communication goes a long way in swinging community sentiment about proposals. Below is an example of how good communication can turn the tide of sentiment-
Here is a somewhat opposite case where the accusations and doubts on a post never get addressed-
Link to proposal- Vegan Africa DAO - Project X Grant Proposal
This should be an encouraging insight for Team Leads and Devs who post on this forum. Clearly communicating goals, addressing doubts & being welcoming are well appreciated by the community.
That pretty much wraps it up folks! I think this got a lot longer than I expected it to. Spent a good 30+ hours getting all the work done + compiling reports on this. It has been a massive learning experience to say the least. I plan on doing this report monthly and have tons of plans to improve and expand its scope!
Let me know your thoughts on this!
P.S- Shoutout to @ILL_DIE_TRYING! His hard work on this bounty was part of what motivated me to try applying my skills to something like this. Shoutout to @Sbae for encouraging me to post this on the forum!