CrowdPeer

I've talked with Louis Fisher and Huw Williams from CrowdPeer (twitter). They also posted a proposal for the ASAPbio Sprint, as I mentioned in the previous article.

Since our projects have a lot of overlap (as I see it, Unfold Research is a direct superset of the CrowdPeer), we've talked about topics such as:

• reputation and privileges per tags (aka per research area)
• tags and structured data in the review/comments
• embedding semantics into the content
• multiple levels of interactions and expressivity - voting vs reactions
• problem of potentially gaming the system
• h-index vs reputation
• acquisition strategies and content seeding
• incentives - reputation and badges

Some other non project specific topics:

• funding opportunities
• for-profit vs non-profit

My reasoning for privileges was that it is fine simply because the real-world interactions function the same way - somebody does something for you and earns your trust that way, and only after that will that individual be included into a more private social circle and will have more insight and opportunities. Copying that mechanics into one where privileges get unlocked by collecting more points (trust) on the platform is an obvious and straightforward technical implementation of it. We also agreed that the ability to "downvote" creates a big difference in enabling to identify bad content and malicious behavior, and is absolutely necessary in order to achieve success with more complicated dynamics.

While talking about the acquisition strategies, I expressed my concern with the approach of different academic platforms where they simply open the platform to the public and "hope" that people will come. But without real value for users to come to the platform (let's say - very little content on the platform), and no reason to stay (no real incentive nor reason to put something onto the platform and accumulate benefits over time) - platforms can't get users and they die.

Talking about the topic of h-index vs reputation points, we agreed that there indeed is a HUGE difference between the two, and in a very interesting way. h-index is working for an individual - it maximizes the metric of a single author and its work. On the other side, the reputation is actually speaking more about the contributions to the community - the metric depends on the community's perception of the work done by a person. In other words, you can on your own work to raise your h-index, but you cannot be on your own for you to get more reputation points. That shift in the internal workings of the metric makes all the difference, and in my opinion can be solely responsible for incentivizing much, much better sharing of the information, hacking of the results and overall collaboration.

We've also touched on the subject of reputation and incentives, and while we know that platforms can function with a lot of pro-bono contributions by the community (examples of Wikipedia and Stackoverflow first come to mind), I did express my doubt that simple badges could so drastically influence the academic hive mindset (yes, h-index is basically "a badge", but it's the first one, so that gives it a huge advantage and an unfair one, so it's can't be really compared) and have practical consequences (maybe I'm too pessimistic on that). Either way, that's the reason I introduced payments as part of the proposal, not to just solve the problem of incentives, but also to reorganize the academic business model as it's too unfair and uneven.

I also mentioned reactions (not so much inspired by social networks and messaging, but platforms like Github) as a really good way to, at the same time, get rid of unconstructive commentary ("thank you", "this is good/bad"), simplify parsing of the information (it's easier to just quickly glance at the reactions' icons than reading text) and still enable people to express their sentiments (grateful, angry, happy). Louis summarised it beautifully as "multiple levels of interactions". There indeed is a problem of semantics associated with votes - are votes about the quality of work, or its correctness, or do you express gratitude for being helpful or... Reviews are there for more rich and insightful commentary, but most of the interactions need a quick way to be executed, and in my opinion, reactions are a great way for sentiments, while votes can be more concerned with factual parts.

On the topic of for-profit vs non-profit, we also agreed that non-profits can/do lack incentivies to actually solve the problem and continiously make progress, and the situation is worsened by the lack of motivation to compete when two non-profits appear - instead of competing and thus making their product better for the end users in order for them to attract more users than their competition and thus survive longer. Also, we are all saddened by the fact that VC-funded projects get ostracized by the academic community, despite the ability to provide more concrete value for them. The combination of these things makes the pool of opportunities for funding projects in meta-academia really small.

It was a solid talk and exchange of opinions and it's good to find more like-minded people. Good luck to Louis and Huw!