From ced540194bf01a7050de3c8d769035871d56a87d Mon Sep 17 00:00:00 2001 From: wea_ondara Date: Mon, 2 Nov 2020 16:54:14 +0100 Subject: [PATCH] wip --- text/1_intro.tex | 4 +- text/2_relwork.tex | 34 +++--- text/3_method.tex | 6 +- text/4_datasets.tex | 6 +- text/bib.bib | 264 ++++++++++++++++++++++---------------------- todo2 | 2 +- 6 files changed, 158 insertions(+), 158 deletions(-) diff --git a/text/1_intro.tex b/text/1_intro.tex index 9c56e8d..4ec3392 100644 --- a/text/1_intro.tex +++ b/text/1_intro.tex @@ -2,7 +2,7 @@ %grammarly checks -StackExchange is a Q\&A platform and consists of 174 communities \cite{stackexchangetour}. Each community evolves around a specific topic, for instance, StackOverflow focusing on software engeneering, or AskUbuntu focusing on the Ubuntu operating system. This distincts StackExchange from other Q\&A sites such as \emph{Yahoo! Answers} where no such differentiation into topics exists. %TODO ref +StackExchange is a Q\&A platform and consists of 174 communities\footnote{\url{https://stackexchange.com/tour}}. Each community evolves around a specific topic, for instance, StackOverflow focusing on software engeneering, or AskUbuntu focusing on the Ubuntu operating system. This distincts StackExchange from other Q\&A sites such as \emph{Yahoo! Answers} where no such differentiation into topics exists. %TODO ref % stackexchange and how it developed via stackoverflow \cite{mamykina2011design} good description of SO % Design Lessons from the Fastest Q&A Site in the West \cite{mamykina2011design} early investigation of so @@ -13,7 +13,7 @@ StackExchange is a Q\&A platform and consists of 174 communities \cite{stackexch % write new numbers about so %TODO -In August of 2018, the StackExchange team introduced a small change that may have had a huge impact on the platform. They added a new feature to visibly mark questions from new contributors, as part of their effort to make the site more welcoming for new users \cite{post2018come}. Specifically, members who want to answer a question created by a new contributor are shown a notification in the answer box that this question is from a new contributor. The StackExchange team hopes that this little change encourages members to be more friendly and forgiving toward new users. +In August of 2018, the StackExchange team introduced a small change that may have had a huge impact on the platform. They added a new feature to visibly mark questions from new contributors, as part of their effort to make the site more welcoming for new users\footnote{\url{https://meta.stackexchange.com/questions/314287/come-take-a-look-at-our-new-contributor-indicator}}. Specifically, members who want to answer a question created by a new contributor are shown a notification in the answer box that this question is from a new contributor. The StackExchange team hopes that this little change encourages members to be more friendly and forgiving toward new users. % write about the change investigated % stackexchange new contriutor post: https://meta.stackexchange.com/questions/314287/come-take-a-look-at-our-new-contributor-indicator?cb=1 % what did change intend? diff --git a/text/2_relwork.tex b/text/2_relwork.tex index 4b6b5cb..7085ed9 100644 --- a/text/2_relwork.tex +++ b/text/2_relwork.tex @@ -13,14 +13,14 @@ This section is divided into two parts. The first part explains what StackExchan \section{Background} StackExchange\footnote{\url{https://stackexchange.com}} is a community question and answering (CQA) platform where users can ask and answer questions, accept answers as an appropriate solution to the question, and up-/downvote questions and answers. StackExchange uses a community-driven knowledge creation process by allowing everyone who registers to participate in the community. Invested users also get access to moderation tools to help maintain the vast community. All posts on the StackExchange platform are publicly visible, allowing non-users to benefit from the community as well. Posts are also accessible for web search engines so users can find questions and anwsers easily with a simple web search. StackExchange keeps an archive of all questions and answers posted, creating a knowledge archive for future visitors to look into. -Originally, StackExchange started with StackOverflow\footnote{\url{https://stackoverflow.com}} in 2008 \cite{atwood2008stack}. -Since then StackExchange grew into a platform hosting sites for 174 different topics \cite{stackexchangetour}, for instance, programming (StackOverflow), maths (MathOverflow\footnote{\url{https://mathoverflow.net}} and Math StackExchange\footnote{\url{https://math.stackexchange.com}}), and typesetting (TeX/LaTeX\footnote{\url{https://tex.stackexchange.com}}). -Questions on StackExchange are stated in natural English language and consist of a title, a body containing a detailed description of the problem or information need, and tags to categorize the question. After a question is posted the community can submit answers to the question. The author of the question can then accept an appropriate answer which satisfies their question. The accepted answer is then marked as such with a green checkmark and shown on top of all the other answers. Figure \ref{soexamplepost} shows an example of a StackOverflow question. Questions and answers can be up-/downvoted by every user registered on the site. Votes typically reflect the quality and importance of the respective question or answers. Answers with a high voting score raise to the top of the answer list as answers are sorted by the vote score in descending order by default. Voting also influences a user's reputation \cite{stackexchangetour, movshovitz2013analysis}. When a post (question or answers) is voted upon the reputation of the poster changes accordingly. Furthermore, downvoting of answers also decreases the reputation of the user who voted \cite{stackoverflowvotedown}. -Reputation on StackExchange indicates how trustworthy a user is. To gain a high reputation value a user has to invest a lot of time and effort to reach a high reputation value by asking good questions and posting good answers to questions. Reputation also unlocks privileges which may differ slightly from one community to another \cite{stackoverflowprivileges, mathoverflowprivileges}. +Originally, StackExchange started with StackOverflow\footnote{\url{https://stackoverflow.com}} in 2008\footnote{\label{atwood2008stack}\url{https://stackoverflow.blog/2008/08/01/stack-overflow-private-beta-begins/}}. +Since then StackExchange grew into a platform hosting sites for 174 different topics \footnote{\label{stackexchangetour}\url{https://stackexchange.com/tour}}, for instance, programming (StackOverflow), maths (MathOverflow\footnote{\url{https://mathoverflow.net}} and Math StackExchange\footnote{\url{https://math.stackexchange.com}}), and typesetting (TeX/LaTeX\footnote{\url{https://tex.stackexchange.com}}). +Questions on StackExchange are stated in natural English language and consist of a title, a body containing a detailed description of the problem or information need, and tags to categorize the question. After a question is posted the community can submit answers to the question. The author of the question can then accept an appropriate answer which satisfies their question. The accepted answer is then marked as such with a green checkmark and shown on top of all the other answers. Figure \ref{soexamplepost} shows an example of a StackOverflow question. Questions and answers can be up-/downvoted by every user registered on the site. Votes typically reflect the quality and importance of the respective question or answers. Answers with a high voting score raise to the top of the answer list as answers are sorted by the vote score in descending order by default. Voting also influences a user's reputation \cite{movshovitz2013analysis}\footref{stackexchangetour}. When a post (question or answers) is voted upon the reputation of the poster changes accordingly. Furthermore, downvoting of answers also decreases the reputation of the user who voted \footnote{\url{https://stackoverflow.com/help/privileges/vote-down}}. +Reputation on StackExchange indicates how trustworthy a user is. To gain a high reputation value a user has to invest a lot of time and effort to reach a high reputation value by asking good questions and posting good answers to questions. Reputation also unlocks privileges which may differ slightly from one community to another \footnote{\url{https://mathoverflow.com/help/privileges/}}\footnote{\url{https://stackoverflow.com/help/privileges/}}. With privileges, users can, for instance, create new tags if the need for a new tag arises, cast votes on closing or reopening questions if the question is off-topic or a duplicate of another question, or when a question had been closed for no or a wrong reason, or even get access to moderation tools. -StackExchange also employs a badge system to steer the community \cite{stackoverflowbadges}. Some badges can be obtained by performing one-time actions, for instance, reading the tour page which contains necessary details for newly registered users, or by performing certain actions multiple times, for instance, editing and answering the same question within 12 hours. +StackExchange also employs a badge system to steer the community \footnote{\label{stackoverflowbadges}\url{https://stackoverflow.com/help/badges/}}. Some badges can be obtained by performing one-time actions, for instance, reading the tour page which contains necessary details for newly registered users, or by performing certain actions multiple times, for instance, editing and answering the same question within 12 hours. Furthermore, users can comment on every question and answer. Comments could be used for further clarifying an answer or a short discussion on a question or answer. -For each community on StackExchange, a \emph Meta page is offered where members of the respective community can discuss the associated community \cite{stackoverflowmeta, mamykina2011design}. This place is used by site admins to interact with the community. The \emph Meta pages are also used for proposing and voting on new features and reporting bugs. \emph Meta pages run the same software as the normal CQA pages so users on vote the ideas and suggestions in the same way they would do on the actual CQA sites. +For each community on StackExchange, a \emph Meta page is offered where members of the respective community can discuss the associated community \cite{mamykina2011design}\footnote{\url{https://stackoverflow.com/help/whats-meta/}}. This place is used by site admins to interact with the community. The \emph Meta pages are also used for proposing and voting on new features and reporting bugs. \emph Meta pages run the same software as the normal CQA pages so users on vote the ideas and suggestions in the same way they would do on the actual CQA sites. \begin{figure} \includegraphics[scale=0.47]{figures/stackoverflow_example_post} @@ -69,8 +69,8 @@ These platforms allow communication over large distances and facilitate fast and All these communities differ in their design. Wikipedia is a community-driven knowledge repository and consists of a collection of articles. Every user can create an article. Articles are edited collaboratively and continually improved an expanded. Reddit is a platform for social interaction where users create posts and comment on other posts or comments. Quora, StackExchange, and Yahoo! Answers are community questions and answer (CQA) platforms. On Quora and Yahoo! Answers users can ask any question regarding any topics whereas on StackExchange users have to post their questions in the appropriate subcommunity, for instance, StackOverflow for programming related questions or MathOverflow for math related questions. CQA sites are very effective at code review \cite{treude2011programmers}. Code may be understood in the traditional sense of source code in programming related fields but this also translates to other fields, for instance, mathematics where formulas represent code. CQA sites are also very effective at solving conceptual questions. This is due to the fact that people have different areas of knowledge and expertise \cite{robillard1999role} and due to the large user base established CQA sites have, which again increases the variety of users with experise in different fields. Despite the differences in purpose and manifestation of these communities, they are social communities and they have to follow certain laws. -In their book on ''Building successful online communities: Evidence-based social design`` \cite{kraut2012building} \citeauthor{kraut2012building} lie out five equally important criteria online platforms have to fulfill in order to thrive. 1) When starting a community, it has to have a critical mass of users who create content. StackOverflow already had a critical mass of users from the beginning due to the StackOverflow team already being experts in the domain \cite{mamykina2011design} and the private beta \cite{atwood2008stack}. Both aspects ensured a strong community core early on. -2) The platform must attract new users to grow as well as to replace leaving users. Depending on the type of community new users should bring certain skills, for example, programming background in open source software developement, or extended knowledge on certain domains; or qualities, for example, a certain illness in medical communities. New users also bring the challenge of onboarding with them. Most newcomers will not be familiar with all the rules and nuances of the community \cite{yazdanian2019eliciting, hanlon2018stack}. 3) The platform should encourage users to commit to the community. Online communities are often based on voluntary commitment of their users \cite{ipeirotis2014quizz}, hence the platform has to ensure users are willing to stay. Most platforms do not have contracts with their users, so users should see benefits for staying with the community. 4) Contribution by users to the community should be encouraged. Content generation and engagement are the backbone of an online community. 5) The community needs regulation to sustain it. Not every user in a community is interested in the wellbeing of the community. Therefore, every community has to deal with trolls and inappropriate or even destructive behavior. Rules need to be established and enforced to limit and mitigate the damage malicious users cause. +In their book on ''Building successful online communities: Evidence-based social design`` \cite{kraut2012building} \citeauthor{kraut2012building} lie out five equally important criteria online platforms have to fulfill in order to thrive. 1) When starting a community, it has to have a critical mass of users who create content. StackOverflow already had a critical mass of users from the beginning due to the StackOverflow team already being experts in the domain \cite{mamykina2011design} and the private beta\footref{atwood2008stack}. Both aspects ensured a strong community core early on. +2) The platform must attract new users to grow as well as to replace leaving users. Depending on the type of community new users should bring certain skills, for example, programming background in open source software developement, or extended knowledge on certain domains; or qualities, for example, a certain illness in medical communities. New users also bring the challenge of onboarding with them. Most newcomers will not be familiar with all the rules and nuances of the community \cite{yazdanian2019eliciting}\footnote{\label{hanlon2018stack}\url{https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/}}. 3) The platform should encourage users to commit to the community. Online communities are often based on voluntary commitment of their users \cite{ipeirotis2014quizz}, hence the platform has to ensure users are willing to stay. Most platforms do not have contracts with their users, so users should see benefits for staying with the community. 4) Contribution by users to the community should be encouraged. Content generation and engagement are the backbone of an online community. 5) The community needs regulation to sustain it. Not every user in a community is interested in the wellbeing of the community. Therefore, every community has to deal with trolls and inappropriate or even destructive behavior. Rules need to be established and enforced to limit and mitigate the damage malicious users cause. %new structure: % list community knowledge platforms @@ -96,9 +96,9 @@ The onboarding process is a permanent challenge for online communities and diffe %marginalized groups -\citeauthor{slag2015one} investigated why many users on StackOverflow only post once after their registration \cite{slag2015one}. They found that 47\% of all users on StackOverflow posted only once and called them one-day-flies. They suggest that code example quality is lower than that of more involved users, which often leads to answers and comments to first improve the question and code instead of answering the stated question. This likely discourages new users from using the site further. Negative feedback instead of constructive feedback is another cause for discontinuation of usage. The StackOverflow staff also conducted their own research on negative feedback of the community \cite{silge2019welcome}. They investigated the comment sections of questions by recruiting their staff members to rate a set of comments and they found more than 7\% of the reviewed comments are unwelcoming. +\citeauthor{slag2015one} investigated why many users on StackOverflow only post once after their registration \cite{slag2015one}. They found that 47\% of all users on StackOverflow posted only once and called them one-day-flies. They suggest that code example quality is lower than that of more involved users, which often leads to answers and comments to first improve the question and code instead of answering the stated question. This likely discourages new users from using the site further. Negative feedback instead of constructive feedback is another cause for discontinuation of usage. The StackOverflow staff also conducted their own research on negative feedback of the community\footnote{\label{silge2019welcome}\url{https://stackoverflow.blog/2018/07/10/welcome-wagon-classifying-comments-on-stack-overflow/}}. They investigated the comment sections of questions by recruiting their staff members to rate a set of comments and they found more than 7\% of the reviewed comments are unwelcoming. -One-day-flies are not unique to StackOverflow. \citeauthor{steinmacher2015social} investigated the social barriers newcomers face when they submit their first contribution to an open-source software project \cite{steinmacher2015social}. They based their work on empirical data and interviews and identified several social barriers preventing newcomers to place their first contribution to a project. Furthermore, newcomers are often on their own in open source projects. The lack of support and peers to ask for help hinders them. \citeauthor{yazdanian2019eliciting} found that new contributors on Wikipedia face challenges when editing articles. Wikipedia hosts millions of articles \cite{sizeofwikipedia} and new contributors often do not know which articles they could edit and improve. Recommender systems can solve this problem by suggesting articles to edit but they suffer from the cold start problem because they rely on past user activity which is missing for new contributors. \citeauthor{yazdanian2019eliciting} proposed a solution by establishing a framework that automatically creates questionnaires to fill this gap. This also helps matching new contributors with more experienced contributors that could help newcomers when they face a problem. +One-day-flies are not unique to StackOverflow. \citeauthor{steinmacher2015social} investigated the social barriers newcomers face when they submit their first contribution to an open-source software project \cite{steinmacher2015social}. They based their work on empirical data and interviews and identified several social barriers preventing newcomers to place their first contribution to a project. Furthermore, newcomers are often on their own in open source projects. The lack of support and peers to ask for help hinders them. \citeauthor{yazdanian2019eliciting} found that new contributors on Wikipedia face challenges when editing articles. Wikipedia hosts millions of articles\footnote{\url{https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia}} and new contributors often do not know which articles they could edit and improve. Recommender systems can solve this problem by suggesting articles to edit but they suffer from the cold start problem because they rely on past user activity which is missing for new contributors. \citeauthor{yazdanian2019eliciting} proposed a solution by establishing a framework that automatically creates questionnaires to fill this gap. This also helps matching new contributors with more experienced contributors that could help newcomers when they face a problem. \citeauthor{allen2006organizational} showed that the one-time-contributors phenomenon also translates to workplaces and organizations \cite{allen2006organizational}. They found out that socialization with other members of an organization plays an important role in turnover. The better the socialization within the organization the less likely newcomers are to leave. This socialization process has to be actively pursued by the organization. One-day-flies may partially be a result of lurking. Lurking is consuming content generated by a community but not contributing content to it. \citeauthor{nonnecke2006non} investigated lurking behavior on Microsoft Network (MSN) \cite{nonnecke2006non} and found that contrary to previous studies lurking is not necessarily a bad behavior. Lurkers show passive behavior and are more introverted and less optimistic than actively posting members of a community. Previous studies suggested lurking is free riding, a taking-rather-than-giving process. However, the authors found that lurking is important in getting to know a community, how a community works and learning the nuances of social interactions on the platform. This allows for better integration into the community when a person decides to join the community. StackExchange, and especially the StackOverflow community, probably has a large lurking audience. Many programmers do not register on the site and those who do only ask one question and revert to lurking, as suggested by \cite{slag2015one}. @@ -106,11 +106,11 @@ One-day-flies may partially be a result of lurking. Lurking is consuming content % DONE Non-public and public online community participation: Needs, attitudes and behavior \cite{nonnecke2006non} about lurking, many programmers do that probably, not even registering, lurking not a bad behavior but observing, lurkers are more introverted, passive behavior, less optimistic and positive than posters, prviously lurking was thought of free riding, not contributing, taking not giving to comunity, important for getting to know a community, better integration when joining -The StackOverflow team acknowledged the one-time-contributors trend \cite{hanlon2018stack, silge2019welcome} and took efforts to make the site more welcoming to new users \cite{friend2018rolling}. They lied out various reasons: Firstly, they have sent mixed messages whether the site is an expert site or for everyone. Secondly, they gave too little guidance to new users which resulted in poor questions from new users and in the unwelcoming behavior of more integrated users towards the new users. New users do not know all the rules and nuances of communication of the communities. An example is that ''Please`` and ''Thank you`` is not well received on the site as they are deemed unnecessary. Also the quality, clearness and language quality of the questions of new users is lower than more experienced users which leads to unwelcoming or even toxic answers and comments. Moreover, users who gained moderation tool access could close questions with predefined reasons which often are not meaningful enough for the poster of the question \cite{hanlon2013war}. Thirdly, marginalized groups, for instance, women and people of color \cite{hanlon2018stack, stackoversurvey2019, ford2016paradise}, are more likely to drop out of the community due to unwelcoming behavior from other users \cite{hanlon2018stack}. They feel the site is an elitist and hostile place. -The team suggested several steps to mitigate these problems. Some of these steps include appealing to the users to be more welcoming and forgiving towards new users \cite{hanlon2018stack, silge2019welcome, spolsky2012kicking}, other steps are geared towards changes to the platform itself: The \emph{Be nice policy} (code of conduct) was updated with feedback from the community \cite{jaydles2014the}. This includes: new users should not be judged for not knowing all things. Furthermore, the closing reasons were updated to be more meaningful to the poster, and questions that are closed are shown as ''on hold`` instead of ''closed`` for the first 5 days \cite{hanlon2013war}. Moreover, the team investigates how the comment sections can be improved to lessen the unwelcomeness and hostility and keep the civility up. +The StackOverflow team acknowledged the one-time-contributors trend\footref{hanlon2018stack}\footref{silge2019welcome} and took efforts to make the site more welcoming to new users\footnote{\label{friend2018rolling}\url{https://stackoverflow.blog/2018/06/21/rolling-out-the-welcome-wagon-june-update/}}. They lied out various reasons: Firstly, they have sent mixed messages whether the site is an expert site or for everyone. Secondly, they gave too little guidance to new users which resulted in poor questions from new users and in the unwelcoming behavior of more integrated users towards the new users. New users do not know all the rules and nuances of communication of the communities. An example is that ''Please`` and ''Thank you`` is not well received on the site as they are deemed unnecessary. Also the quality, clearness and language quality of the questions of new users is lower than more experienced users which leads to unwelcoming or even toxic answers and comments. Moreover, users who gained moderation tool access could close questions with predefined reasons which often are not meaningful enough for the poster of the question\footnote{\label{hanlon2013war}\url{https://stackoverflow.blog/2013/06/25/the-war-of-the-closes/}}. Thirdly, marginalized groups, for instance, women and people of color \cite{ford2016paradise}\footref{hanlon2018stack}\footnote{\label{stackoversurvey2019}\url{https://insights.stackoverflow.com/survey/2019}}, are more likely to drop out of the community due to unwelcoming behavior from other users\footref{hanlon2018stack}. They feel the site is an elitist and hostile place. +The team suggested several steps to mitigate these problems. Some of these steps include appealing to the users to be more welcoming and forgiving towards new users\footref{hanlon2018stack}\footref{silge2019welcome}\footnote{\url{https://stackoverflow.blog/2012/07/20/kicking-off-the-summer-of-love/}}, other steps are geared towards changes to the platform itself: The \emph{Be nice policy} (code of conduct) was updated with feedback from the community\footnote{\url{https://meta.stackexchange.com/questions/240839/the-new-new-be-nice-policy-code-of-conduct-updated-with-your-feedback}}. This includes: new users should not be judged for not knowing all things. Furthermore, the closing reasons were updated to be more meaningful to the poster, and questions that are closed are shown as ''on hold`` instead of ''closed`` for the first 5 days\footref{hanlon2013war}. Moreover, the team investigates how the comment sections can be improved to lessen the unwelcomeness and hostility and keep the civility up. -The StackOverflow team partnered with \citeauthor{ford2018we} and implemented the Mentorship Research Project \cite{ford2018we, hanlon2017mentorship}. The project lasted one month and aimed to help newcomers improve their first questions before they are posted publicly. The program went as follows: When a user is about to post a question the user is asked whether they want their question to be reviewed by a mentor. If they confirmed they are forward to a help room with a mentor who is an experienced user. The question is then reviewed and the mentor suggests some changes if applicable. These changes may include narrowing the question for more precise answers, adding a code example or adjusting code, or removing of \emph{Please} and \emph{Thank you} from the question. After the review and editing, the question is posted publicly by the user. The authors found that mentored questions are received significantly better by the community than non-mentored questions. The questions also received higher scores and were less likely to be off-topic and poor in quality. Furthermore, newcomers are more comfortable when their question is reviewed by a mentor. -For this project four mentors were hand selected and therefore the project would not scale very well as the number of mentors is very limited but it gave the authors an idea on how to pursue their goal of increasing the welcomingness on StackExchange. The project is followed up by a \emph{Ask a question wizard} to help new users as well as more experienced users improve the structure, quality, and clearness of their questions \cite{friend2018rolling}. +The StackOverflow team partnered with \citeauthor{ford2018we} and implemented the Mentorship Research Project \cite{ford2018we}\footnote{\url{https://meta.stackoverflow.com/questions/357198/mentorship-research-project-results-wrap-up}}. The project lasted one month and aimed to help newcomers improve their first questions before they are posted publicly. The program went as follows: When a user is about to post a question the user is asked whether they want their question to be reviewed by a mentor. If they confirmed they are forward to a help room with a mentor who is an experienced user. The question is then reviewed and the mentor suggests some changes if applicable. These changes may include narrowing the question for more precise answers, adding a code example or adjusting code, or removing of \emph{Please} and \emph{Thank you} from the question. After the review and editing, the question is posted publicly by the user. The authors found that mentored questions are received significantly better by the community than non-mentored questions. The questions also received higher scores and were less likely to be off-topic and poor in quality. Furthermore, newcomers are more comfortable when their question is reviewed by a mentor. +For this project four mentors were hand selected and therefore the project would not scale very well as the number of mentors is very limited but it gave the authors an idea on how to pursue their goal of increasing the welcomingness on StackExchange. The project is followed up by a \emph{Ask a question wizard} to help new users as well as more experienced users improve the structure, quality, and clearness of their questions\footref{friend2018rolling}. % DONE One-day flies on StackOverflow \cite{slag2015one}, 1 contribution during whole registration, only user with 6 month of registration @@ -129,7 +129,7 @@ For this project four mentors were hand selected and therefore the project would %TODO Unwelcomeness is a large problem on StackExchange; not so strong; maybe other sentence -Unwelcomeness is a large problem on StackExchange \cite{hanlon2018stack, friend2018rolling, ford2016paradise}. Although unwelcomeness affects all new users, users from marginalized groups suffer significantly more \cite{hanlon2018stack, vasilescu2014gender}. \citeauthor{ford2016paradise} investigated barriers users face when contributing to StackOverflow. The authors identified 14 barriers in total hindering newcomers to contribute and five barriers were rated significantly more problematic for women than men. On StackOverflow only 5.8\% (2015 \cite{stackoversurvey2015}, 7.9\% 2019 \cite{stackoversurvey2019}) of active users identify as women. \citeauthor{david2008community} found similar results of 5\% women in their work on \emph{Community-based production of open-source software} \cite{david2008community}. These numbers are comparatively small to the number of degrees in Science, Technology, Engineering, and Mathematics (STEM) \cite{clark2005women} where 20\% are achieved by women \cite{hill2010so}. Despite the difference, the percentage of women on StackOverflow has increased in recent years. +Unwelcomeness is a large problem on StackExchange \cite{ford2016paradise}\footref{friend2018rolling}\footref{hanlon2018stack}. Although unwelcomeness affects all new users, users from marginalized groups suffer significantly more \cite{vasilescu2014gender}\footref{hanlon2018stack}. \citeauthor{ford2016paradise} investigated barriers users face when contributing to StackOverflow. The authors identified 14 barriers in total hindering newcomers to contribute and five barriers were rated significantly more problematic for women than men. On StackOverflow only 5.8\% (2015\footnote{\url{https://insights.stackoverflow.com/survey/2015}}, 7.9\% 2019\footref{stackoversurvey2019}) of active users identify as women. \citeauthor{david2008community} found similar results of 5\% women in their work on \emph{Community-based production of open-source software} \cite{david2008community}. These numbers are comparatively small to the number of degrees in Science, Technology, Engineering, and Mathematics (STEM) \cite{clark2005women} where 20\% are achieved by women \cite{hill2010so}. Despite the difference, the percentage of women on StackOverflow has increased in recent years. %discrimitation % DONE Paradise Unplugged: Identifying Barriers for Female Participation on Stack Overflow \cite{ford2016paradise} gender gap, females only 5\%, contribution barriers, found 5 gender specific (women) barriers among 14 barrier in total, barriers also affect groups like industry programmers @@ -163,7 +163,7 @@ Although gaining reputation takes time and effort, users can take certain advant % DONE Design Lessons from the Fastest Q&A Site in the West \cite{mamykina2011design} understanding SO success, 1) productive competition (gamification reputation), 2) founders were already experts on site the created (ensured success early on, founders involved in community not external), 3) meta page for discussion and voting on features (same mechanics as on SO page) -Complementary to the reputation system StackOverflow also employs a badge system \cite{stackoverflowbadges} to stimulate contributions by users \cite{cavusoglu2015can}. The goal of badges is to keep users engaged with the community \cite{li2012quantifying}. Therefore, badges are often used in a gamification setting where users contribute to the community and are rewarded for their behavior if it alignes with the requirements of the badges. Badges are visible in questions and answers as well as the profile page of the user and can be earned by performing certain actions. Badges are often seen as a steering mechanism by researchers \cite{yanovsky2019one, kusmierczyk2018causal, anderson2013steering}. Although users want to achieve badges and are therefore steered to perform certain actions, steering also occurs in the reputation system. However, badges allow a wider variety of goals, for instance, asking and answering questions, voting on questions and answers, or writing higher quality answers. +Complementary to the reputation system StackOverflow also employs a badge system \footref{stackoverflowbadges} to stimulate contributions by users \cite{cavusoglu2015can}. The goal of badges is to keep users engaged with the community \cite{li2012quantifying}. Therefore, badges are often used in a gamification setting where users contribute to the community and are rewarded for their behavior if it alignes with the requirements of the badges. Badges are visible in questions and answers as well as the profile page of the user and can be earned by performing certain actions. Badges are often seen as a steering mechanism by researchers \cite{yanovsky2019one, kusmierczyk2018causal, anderson2013steering}. Although users want to achieve badges and are therefore steered to perform certain actions, steering also occurs in the reputation system. However, badges allow a wider variety of goals, for instance, asking and answering questions, voting on questions and answers, or writing higher quality answers. Badges also work as a motivator for users \cite{anderson2013steering}. Users often put in non-trivial amounts of work and effort to achieve badges and so badges become powerful incentives. However, not all users are equal and therefore do not pursue badges in the same way \cite{yanovsky2019one}. Contrary to \cite{anderson2013steering}, \citeauthor{yanovsky2019one} \cite{yanovsky2019one} found that users do not necessarily increase their activity prior to achieving a badge followed by an immediate decrease in contribution thereafter but users behave differently based on their type of contribution. The authors found users can be categorized into three groups: Firstly, some users are not affected at all by the badge system and still contribute a lot to the community. Secondly, users increase their activity too before gaining a badge and keep their level of contribution afterward. Finally, users increase their activity before achieving a badge and return to their previous level of engagement thereafter. Different badges also create status classes \cite{immorlica2015social}. The harder a badge can be earned by users, the more unique it is within the community and therefore the badge symbolizes some sort of status. Often rare badges are hard to achieve and take significant effort. For some users, depending on their type, this can be a huge motivator. \citeauthor{kusmierczyk2018causal} found first-time badges play an important role in steering users \cite{kusmierczyk2018causal}. The steering effect only takes place if the benefit to the user is greater than the effort the user has to put into to obtain the badge. If the effort is greater the user will likely not pursue the badge and therefore the steering effect will not occur. @@ -180,7 +180,7 @@ Different badges also create status classes \cite{immorlica2015social}. The hard % DONE Steering user behavior with badges \cite{anderson2013steering} # all abount badges, steering users, motivation, user may put in non trivial amounts of work to achieve badges -> powerful incentives, badges used in multiple ways (steer users to ask/answer more questions, voting, etc.) -Quality is often a concern in online communities. Platform moderators and admins want to keep a certain level of quality or even raise it. However, higher-quality posts take more time and effort than lower-quality posts. In the case of CQA platforms, this is an even bigger problem as higher quality answers fight against fast responses. Despite that, StackOverflow also has a problem with low quality and effort questions and subsequent unwelcoming answers and comments \cite{silge2019welcome}. StackOverflow has grown into a large community and larger communities are harder to control. \citeauthor{lin2017better} investigated how growth affects a community. They looked at Reddit communities that were added to the default set of subscribed communities of every new user (defaulting) which lead to a huge influx of new users to these communities as a result. The authors found that contrary to expectations, the quality stays largely the same. The vote score dips shortly after defaulting but quickly recovers or even raises to higher levels than before. The complaints of low-quality content did not increase, and the language used in the community stayed the same. However, the community clustered around fewer posts than before defaulting. +Quality is often a concern in online communities. Platform moderators and admins want to keep a certain level of quality or even raise it. However, higher-quality posts take more time and effort than lower-quality posts. In the case of CQA platforms, this is an even bigger problem as higher quality answers fight against fast responses. Despite that, StackOverflow also has a problem with low quality and effort questions and subsequent unwelcoming answers and comments \footref{silge2019welcome}. StackOverflow has grown into a large community and larger communities are harder to control. \citeauthor{lin2017better} investigated how growth affects a community. They looked at Reddit communities that were added to the default set of subscribed communities of every new user (defaulting) which lead to a huge influx of new users to these communities as a result. The authors found that contrary to expectations, the quality stays largely the same. The vote score dips shortly after defaulting but quickly recovers or even raises to higher levels than before. The complaints of low-quality content did not increase, and the language used in the community stayed the same. However, the community clustered around fewer posts than before defaulting. \citeauthor{tausczik2011predicting} found reputation is linked to the perceived quality of posts in multiple ways \cite{tausczik2011predicting}. They suggest reputation could be used as an indicator of quality. Quality also depends on the type of platform. \cite{lin2017better} showed that expert sites who charge fees, for instance, library reference services, have higher quality answers compared to free sites. Also, the higher the fee the higher the quality of the answers. However, free community sites outperform expert sites in terms of answer density and responsiveness. diff --git a/text/3_method.tex b/text/3_method.tex index 21cfe6c..c438fa5 100644 --- a/text/3_method.tex +++ b/text/3_method.tex @@ -1,10 +1,10 @@ \chapter{Method} -StackExchange introduced a \emph{new contributor} indicator to all communities on $21^{st}$ of August in 2018 at 9 pm UTC \cite{post2018come}. This step is one of many StackExchange took to make the platform and its members more welcoming towards new users. This indicator is shown to potential answerers in the answer text box of a question flagged as from a new contributor as shown in figure \ref{newcontributor}. The indicator is added to a question if the question is the first contribution of the user or if the first contribution (question or answer) of the user was less than 7 days ago \cite{sonic2018what}. The indicator is then shown for 7 days from the creation date of the question. Note that the user can be registered for a long time and then post their first question and it is counted as a question from a new contributor. Also, if a user decides to delete all their existing contributions from the site and then creates a new question this question will have the \emph{new contributor} indicator attached. The sole deciding factor for the indicator is the date and time of the first non-deleted contribution and the 7-day window afterward. +StackExchange introduced a \emph{new contributor} indicator to all communities on $21^{st}$ of August in 2018 at 9 pm UTC\footnote{\label{post2018come}\url{https://meta.stackexchange.com/questions/314287/come-take-a-look-at-our-new-contributor-indicator}}. This step is one of many StackExchange took to make the platform and its members more welcoming towards new users. This indicator is shown to potential answerers in the answer text box of a question flagged as from a new contributor as shown in figure \ref{newcontributor}. The indicator is added to a question if the question is the first contribution of the user or if the first contribution (question or answer) of the user was less than 7 days ago \footnote{\label{sonic2018what}\url{https://meta.stackexchange.com/questions/314472/what-are-the-exact-criteria-for-the-new-contributor-indicator-to-be-shown}}. The indicator is then shown for 7 days from the creation date of the question. Note that the user can be registered for a long time and then post their first question and it is counted as a question from a new contributor. Also, if a user decides to delete all their existing contributions from the site and then creates a new question this question will have the \emph{new contributor} indicator attached. The sole deciding factor for the indicator is the date and time of the first non-deleted contribution and the 7-day window afterward. \begin{figure} \centering\includegraphics[scale=0.47]{figures/new_contributor} - \caption{The answer box a potential answerers sees when viewing a question from a new contributor. \copyright{Tim Post, 2018, \url{https://meta.stackexchange.com/users/50049/tim-post}} in \cite{post2018come}} + \caption{The answer box a potential answerers sees when viewing a question from a new contributor. \copyright{Tim Post, 2018, \url{https://meta.stackexchange.com/users/50049/tim-post}}\footref{post2018come}} \label{newcontributor} \end{figure} @@ -19,7 +19,7 @@ To measure the effectiveness of the change this thesis utilizes Vader, a sentime % sentiment calculation via vaderlib, write whole paragraph and explain, also add ref to paper \cite{hutto2014vader} \section{Data gathering and preprocessing} -StackExchange provides anonymized data dumps of all their communities for researchers to investigate at no cost on archive.org \cite{archivestackexchange}. These data dumps contain users, posts (questions and answers), badges, comments, tags, votes, and a post history containing all versions of posts. Each entry contains the necessary information, for instance, id, creation date, title, body, and how the data is linked together (which user posted a question/answer/comment). However, not all data entries are valid and therefore cannot be used in the analysis, for instance, questions or answers of which the user is unknown, but this only affects a very small amount entries. So before the actual analysis, the data has to be cleaned. Moreover, the answer texts are in HTML format, containing tags that could skew the sentiment values, and they need to be stripped away beforehand. Additionally, answers may contain code sections which also would skew the results and are therefore omitted. +StackExchange provides anonymized data dumps of all their communities for researchers to investigate at no cost on archive.org \footnote{\label{archivestackexchange}\url{https://archive.org/download/stackexchange}}. These data dumps contain users, posts (questions and answers), badges, comments, tags, votes, and a post history containing all versions of posts. Each entry contains the necessary information, for instance, id, creation date, title, body, and how the data is linked together (which user posted a question/answer/comment). However, not all data entries are valid and therefore cannot be used in the analysis, for instance, questions or answers of which the user is unknown, but this only affects a very small amount entries. So before the actual analysis, the data has to be cleaned. Moreover, the answer texts are in HTML format, containing tags that could skew the sentiment values, and they need to be stripped away beforehand. Additionally, answers may contain code sections which also would skew the results and are therefore omitted. % data sets as xml files from archive.org \cite{archivestackexchange} %cleaning data diff --git a/text/4_datasets.tex b/text/4_datasets.tex index 9708cb1..382f143 100644 --- a/text/4_datasets.tex +++ b/text/4_datasets.tex @@ -2,7 +2,7 @@ %TODO maybe more text %general -StackExchange provides complete datasets of its communities for research purposes on archive.org \cite{archivestackexchange}. StackExchange also provides a short guide on how to interpret the provided data, as some data values are strictly numerical and do not convey any meaning without the knowledge of what these values represent. +StackExchange provides complete datasets of its communities for research purposes on archive.org\footnote{\url{https://archive.org/download/stackexchange}}. StackExchange also provides a short guide on how to interpret the provided data, as some data values are strictly numerical and do not convey any meaning without the knowledge of what these values represent. This thesis investigates the largest datasets available and includes the datasets of the following communities: \begin{itemize} \item StackOverflow.com @@ -46,7 +46,7 @@ New users asked 2880039 questions with an average of 1.240 questions per new use \end{figure} \section{math.stackexchange.com} -``Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields.'' \cite{mathstackexchangecom} +``Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields.''\footnote{\url{https://math.stackexchange.com/}} The community has 624671 registered users of which 17074 were active between December 2019 and February 2020. Members asked 1170938 questions in total and gave 1565188 answers with an average answer density of 1.336 answers per question. New users asked 265704 questions with an average of 1.336 questions per new user during their first week after first contribution. @@ -160,7 +160,7 @@ New users asked 47035 questions with an average of 1.126 questions per new user \end{figure} \section{stats.stackexchange.com (Cross Validated)} -``Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.'' \cite{statsstackexchangecom} +``Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.''\footnote{\url{https://stats.stackexchange.com/}} The community has 227032 registered users of which 4485 were active between December 2019 and February 2020. Members asked 151777 questions in total and gave 148046 answers with an average answer density of 0.975 answers per question. New users asked 57636 questions with an average of 1.112 questions per new user during their first week after first contribution. diff --git a/text/bib.bib b/text/bib.bib index 953b63a..ed1563f 100644 --- a/text/bib.bib +++ b/text/bib.bib @@ -143,132 +143,132 @@ booktitle={CHI}, year={2018} } -@online{atwood2008stack, - title={Stack Overflow Private Beta Begins}, - author={Atwood, Jeff}, - url={https://stackoverflow.blog/2008/08/01/stack-overflow-private-beta-begins/}, - date={2008-08-01} -} -@online{hanlon2018stack, - title={Stack Overflow Isn't Very Welcoming: It's Time for That to Change}, - author={Hanlon, Jay}, - url={https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/}, - year={2018} -} -@online{friend2018rolling, - title={Rolling out the Welcome Wagon: June Update}, - author={Friend, Joe}, - url={https://stackoverflow.blog/2018/06/21/rolling-out-the-welcome-wagon-june-update/}, - year={2018} -} -@online{silge2019welcome, - title={Welcome Wagon: Classifying Comments on Stack Overflow}, - author={Silge, Julia and Punyon, Jason}, - url={https://stackoverflow.blog/2018/07/10/welcome-wagon-classifying-comments-on-stack-overflow/}, - year={2019} -} -@online{stackexchangetour, - title={Tour - StackExchange}, - author={}, - url={https://stackexchange.com/tour}, - year={} -} -@online{stackoverflowvotedown, - title={Help Center > Privileges > Vote down}, - author={}, - url={https://stackoverflow.com/help/privileges/vote-down}, - year={} -} -@online{stackoverflowprivileges, - title={Help Center > Privileges}, - author={}, - url={https://stackoverflow.com/help/privileges/}, - year={} -} -@online{mathoverflowprivileges, - title={Help Center > Privileges}, - author={}, - url={https://mathoverflow.com/help/privileges/}, - year={} -} -@online{stackoverflowbadges, - title={Help Center > Badges}, - author={}, - url={https://stackoverflow.com/help/badges/}, - year={} -} -@online{stackoverflowmeta, - title={What is "meta"? How does it work?}, - author={}, - url={https://stackoverflow.com/help/whats-meta/}, - year={} -} -@online{stackoversurvey2015, - title={Developer Survey Results 2015}, - author={}, - url={https://insights.stackoverflow.com/survey/2015}, - year={2015} -} -@online{stackoversurvey2019, - title={Developer Survey Results 2019}, - author={}, - url={https://insights.stackoverflow.com/survey/2019}, - year={2019} -} -@online{hanlon2013war, - title={The War of the Closes}, - author={Hanlon, Jay}, - url={https://stackoverflow.blog/2013/06/25/the-war-of-the-closes/}, - year={2013} -} -@online{hanlon2017mentorship, - title={Mentorship Research Project - Results + Wrap-Up}, - author={Lustig, Kristina}, - url={https://meta.stackoverflow.com/questions/357198/mentorship-research-project-results-wrap-up}, - year={2017} -} -@online{spolsky2012kicking, - title={Kicking off the Summer of Love}, - author={Spolsky, Joel}, - url={https://stackoverflow.blog/2012/07/20/kicking-off-the-summer-of-love/}, - year={2012} -} -@online{sizeofwikipedia, - title={Wikipedia:Size of Wikipedia}, - author={}, - url={https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia}, - year={2020} -} -@online{jaydles2014the, - title={The NEW new “Be Nice” Policy (“Code of Conduct”) — Updated with your feedback [duplicate]}, - author={Jaydles}, - url={https://meta.stackexchange.com/questions/240839/the-new-new-be-nice-policy-code-of-conduct-updated-with-your-feedback}, - year={2014} -} -@online{post2018come, - title={Come Take a Look at our New Contributor Indicator!}, - author={Post, Tim}, - url={https://meta.stackexchange.com/questions/314287/come-take-a-look-at-our-new-contributor-indicator}, - year={2018} -} -@online{sonic2018what, - title={What are the exact criteria for the “new contributor” indicator to be shown?}, - author={Sonic the Anonymous Hedgehog and Nickel, Brian}, - url={https://meta.stackexchange.com/questions/314472/what-are-the-exact-criteria-for-the-new-contributor-indicator-to-be-shown}, - year={2018} -} -@online{statsstackexchangecom, - title={Cross Validated Main page}, - author={}, - url={https://stats.stackexchange.com/}, - year={} -} -@online{mathstackexchangecom, - title={Mathematics Stack Exchange Main page}, - author={}, - url={https://math.stackexchange.com/}, - year={} -} +//@online{atwood2008stack, + // title={Stack Overflow Private Beta Begins}, + // author={Atwood, Jeff}, + // url={https://stackoverflow.blog/2008/08/01/stack-overflow-private-beta-begins/}, + // date={2008-08-01} +//} +//@online{hanlon2018stack, +// title={Stack Overflow Isn't Very Welcoming: It's Time for That to Change}, +// author={Hanlon, Jay}, +// url={https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/}, +// year={2018} +//} +//@online{friend2018rolling, +// title={Rolling out the Welcome Wagon: June Update}, +// author={Friend, Joe}, +// url={https://stackoverflow.blog/2018/06/21/rolling-out-the-welcome-wagon-june-update/}, +// year={2018} +//} +//@online{silge2019welcome, +// title={Welcome Wagon: Classifying Comments on Stack Overflow}, +// author={Silge, Julia and Punyon, Jason}, +// url={https://stackoverflow.blog/2018/07/10/welcome-wagon-classifying-comments-on-stack-overflow/}, +// year={2019} +//} +//@online{stackexchangetour, +// title={Tour - StackExchange}, +// author={}, + // url={https://stackexchange.com/tour}, +// year={} +//} +//@online{stackoverflowvotedown, +// title={Help Center > Privileges > Vote down}, +//// author={}, +// url={https://stackoverflow.com/help/privileges/vote-down}, +// year={} +//} +//@online{stackoverflowprivileges, +// title={Help Center > Privileges}, +// author={}, +// url={https://stackoverflow.com/help/privileges/}, +// year={} +//} +//@online{mathoverflowprivileges, +// title={Help Center > Privileges}, +// author={}, +// url={https://mathoverflow.com/help/privileges/}, +// year={} +//} +//@online{stackoverflowbadges, +// title={Help Center > Badges}, +// author={}, +// url={https://stackoverflow.com/help/badges/}, + // year={} +//} +//@online{stackoverflowmeta, +// title={What is "meta"? How does it work?}, +// author={}, +// url={https://stackoverflow.com/help/whats-meta/}, +// year={} +//} +//@online{stackoversurvey2015, +// title={Developer Survey Results 2015}, +// author={}, +// url={https://insights.stackoverflow.com/survey/2015}, +// year={2015} +//} +//@online{stackoversurvey2019, +// title={Developer Survey Results 2019}, +// author={}, +// url={https://insights.stackoverflow.com/survey/2019}, +// year={2019} +//} +//@online{hanlon2013war, +// title={The War of the Closes}, +// author={Hanlon, Jay}, +// url={https://stackoverflow.blog/2013/06/25/the-war-of-the-closes/}, +// year={2013} +//} +//@online{hanlon2017mentorship, +// title={Mentorship Research Project - Results + Wrap-Up}, +// author={Lustig, Kristina}, +// url={https://meta.stackoverflow.com/questions/357198/mentorship-research-project-results-wrap-up}, +// year={2017} +//} +//@online{spolsky2012kicking, +// title={Kicking off the Summer of Love}, +// author={Spolsky, Joel}, +// url={https://stackoverflow.blog/2012/07/20/kicking-off-the-summer-of-love/}, +// year={2012} +//} +//@online{sizeofwikipedia, +// title={Wikipedia:Size of Wikipedia}, +// author={}, +// url={https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia}, +// year={2020} +//} +//@online{jaydles2014the, +// title={The NEW new “Be Nice” Policy (“Code of Conduct”) — Updated with your feedback [duplicate]}, +// author={Jaydles}, +// url={https://meta.stackexchange.com/questions/240839/the-new-new-be-nice-policy-code-of-conduct-updated-with-your-feedback}, +// year={2014} +//} +//@online{post2018come, +// title={Come Take a Look at our New Contributor Indicator!}, +// author={Post, Tim}, +// url={https://meta.stackexchange.com/questions/314287/come-take-a-look-at-our-new-contributor-indicator}, +// year={2018} +//} +//@online{sonic2018what, +// title={What are the exact criteria for the “new contributor” indicator to be shown?}, +// author={Sonic the Anonymous Hedgehog and Nickel, Brian}, +// url={https://meta.stackexchange.com/questions/314472/what-are-the-exact-criteria-for-the-new-contributor-indicator-to-be-shown}, +// year={2018} +//} +//@online{statsstackexchangecom, +// title={Cross Validated Main page}, +// author={}, +// url={https://stats.stackexchange.com/}, +// year={} +//} +//@online{mathstackexchangecom, +// title={Mathematics Stack Exchange Main page}, +// author={}, +// url={https://math.stackexchange.com/}, +// year={} +//} @inproceedings{yanovsky2019one, title={One Size Does Not Fit All: Badge Behavior in Q\&A Sites}, author={Yanovsky, Stav and Hoernle, Nicholas and Lev, Omer and Gal, Kobi}, @@ -395,12 +395,12 @@ booktitle={Eighth international AAAI conference on weblogs and social media}, year={2014} } -@online{archivestackexchange, - title={Files for stackexchange}, - author={}, - url={https://archive.org/download/stackexchange}}, - year={} -} +//@online{archivestackexchange, +// title={Files for stackexchange}, +// author={}, +// url={https://archive.org/download/stackexchange}, +// year={} +//} @article{bernal2017interrupted, title={Interrupted time series regression for the evaluation of public health interventions: a tutorial}, author={Bernal, James Lopez and Cummins, Steven and Gasparrini, Antonio}, diff --git a/todo2 b/todo2 index 72867c8..842377a 100644 --- a/todo2 +++ b/todo2 @@ -6,7 +6,7 @@ related work ausbauen mehr referencen -50 refs +50+ refs links -> foot notes