Behavioral Big Data and its Challenges for Behavioral Researchers

Galit Shmueli*

Behavioral big data (BBD) captures human and social actions and interactions at a new level of detail. BBD studies and applications are quickly growing in popularity in industry as well as in academic research. “Behavioral” in BBD highlights its focus on human and social behavior, and “big” – the novelty of its scale.

The combination of ‘‘behavioral’’ and ‘‘big’’ creates challenges and opportunities for statisticians, data miners, and other data scientists since the great majority of researchers in these communities are not trained in the behavioral sciences and are therefore unfamiliar with study design, ethical conduct, and research methods for studies with human subjects. It also creates challenges for social and behavioral scientists who have classic statistical modeling training and experience.

Academic researchers are now using BBD in fields that earlier had small behavioral data—such as psychology, management, marketing, information systems, sociology, political science, education—as well as in fields that earlier dealt with big inanimate data—production, biology, and engineering. Based on BBD studies presented at conferences and published in journals, it appears that these two types of researchers are encountering new dilemmas and challenges. My article Research Dilemmas with Behavioral Big Data (Big Data, 2017, Vol 5 Issue 2) examines the new challenges and dilemmas behavioral researchers face in the new BBD era.

Scientific research aims at discovering scientifically valid regularities that generalize from the data sample to a population of interest. To achieve this, behavioral studies include four major elements: the researcher, the human subjects, the research question, and the data. In BBD studies, challenges arise due to the changes in relationships between these four elements. For example, the distance between the researcher and the human subject is typically much larger in BBD studies compared to “small behavioral data” studies. The figure highlights some of the main issues.

It is worth noting that the availability of behavioral big data is bringing closer different disciplines: behavioral sciences, medical and life sciences, data sciences, management, law, and even engineering. The challenges to each of these communities is slightly different (see, for example, my slides on BBD in Healthcare or on BBD and Quality Engineering), which means interdisciplinary collaborations and research are extremely important and useful.

 


Galit Shmueli, Ph.D., is the Tsing Hua Distinguished Professor at the Institute of Service Science, and Director of the Center for Service Innovation & Analytics at the College of Technology Management, National Tsing Hua University, Taiwan. Before joining NTHU, she was the SRITNE Chaired Professor of Data Analytics and Associate Professor of Statistics & Information Systems at the Indian School of Business, and tenured Associate Professor at University of Maryland’s Smith School of Business.

Needed: More Science on Communicating Science

Nora Newcombe

Scientists often feel that, like Rodney Dangerfield, they get no respect.  Policy makers ignore their evidence. The public stereotypes them as nerdy guys in white coats with glasses. Legislators cut their funding sources and single them out for Golden Fleece awards. FABBS exists to try to address these issues: to get the word out about good science and to defend against misperceptions.

But how can we best do this work? A recent report from the American Academy of Arts and Sciences provides some relevant data and begins to delineate what we know, and don’t know, about the public face of science. It’s the first of a projected three reports from their Public Face of Science project, of which I am a member.

Readers may want to begin by taking an on-line quiz. It’s devilishly difficult, and you may be surprised at your score.

Here are some take-home messages. Although public trust in many institutions is deteriorating in the U.S., public trust in science has been relatively high and stable. This good news surprises many scientists, and it’s a welcome relief. However, trust in science varies importantly by demographics. Elderly and less-educated Americans express lower trust. Even more importantly, against a background of general trust, responses vary by topic; for example, substantial proportions of respondents question the safety of vaccines. Furthermore, attitudes on topics vary by age, education, and other factors in poorly understood ways. No set of factors consistently relates to attitudes. These data suggest to me that we are not very clear about what is going on and need more research to gain the understanding we need to do better.

Here is one example where we need research. Scientists often assume that “to know me is to love me.” Thus, they advocate for more science education, both formal education in school and informal education in museums, zoos, and at science festivals. In addition, there has been recent work on better communication with journalists, such as media training for scientists. However, the American Academy report summarizes evidence that simple knowledge of science, at least as assessed by fact-based quizzes, does not always relate positively to using science in decision making, and sometimes even relates negatively. Thus, more science education may not be the royal road to effecting change. One possibility is that what is needed is more than facts, and maybe not even primarily facts. Perhaps the public needs to understand the process of science: its curiosity-driven essence, it’s (ultimately) self-correcting nature, what it counts as evidence and what it doesn’t. That’s a plausible hypothesis, but it needs evaluation.

In short, the behavioral and brain sciences are essential to meeting the challenge of understanding how best to communicate science, and how to better integrate science in informing private and public decision making. How can scientists better communicate science? What is the role of formal and informal science educators, and the media? Can we teach science reasoning skills at earlier levels? One of the most surprising things I learned in participating on the task force is that we don’t know the answers to these questions. I think we have the tools to answer them, however, so I hope that scientists and funders can work together on answering them.

For the full report, and yet more research questions, see the American Academy of Arts and Sciences report.

Changes to the human subjects system: a view from someone formerly on the inside

Philip Rubin*

The past years has been a time of concern and confusion for the behavioral sciences community on matters of federal policy and regulations related to human subjects. In particular, the recent announcement of delays in what appeared to be a pending update to the Common Rule has created a high degree of uncertainty related to the conduct of ongoing research and the submission of proposals for proposed work.

The Federal Policy for the Protection of Human Subjects was originally published in 1991 and consists of a set of related regulations spanning 16 federal departments and agencies. The regulations were last revised on January 15, 2009. Subpart A of the regulations is often referred to as “The Common Rule,” because this portion is shared in common by the participating federal agencies or departments. It is highly unusual for that many agencies to enter into a joint rule making process of this scale. Progress can be very slow. To assist with ongoing communication, the Office of Science and Technology Policy (OSTP) in the Executive Office of the President, created a subcommittee of the National Science and Technology Council (NSTC) in the 1990s called the Human Subjects Research Subcommittee (HSRS) that met regularly for many years to consider possible changes and/or improvements to the regulations. HSRS’s tenure ended during the Obama administration, although members of the group continued to meet informally.

In response to many concerns over the decades since the release of the regulations, the Department of Health and Human Services (HHS) and OSTP announced on July 22, 2011, via an Advance Notice of Proposed Rulemaking (ANPRM), that the federal government was considering regulatory changes focused on issues related to the ethics, safety, and oversight of human research, and intended to strengthen protections for human research subjects. After several years, there were reports that this modernization process had stalled. The complexity of the human subjects enterprise and the regulations, and the often conflicting interests of the many interested stakeholders, cannot be understated. Although considerable progress had been made on a number of issues, internal disagreements at HHS (CDC, FDA, NIH) had apparently brought things to a standstill, particularly around the issues of biospecimens and consent.

It appeared in 2014 that the process started in 2011 of modernizing the human subjects regulations might be abandoned. Concerns about the possible collapse of these efforts were raised in the press, by advocates, and within the government, by HHS, NIH, the Office of Management and Budget (OMB), OSTP, and others. President Obama tasked OSTP with helping to keep the effort going and moving it along as rapidly as possible, with the goal of releasing a final rule before the end of his administration, if feasible. At that time, I was the Principal Assistant Director for Science at OSTP. With the assistance of OSTP’s Assistant Director for Forensic Science, Tania Simoncelli (now at the Chan Zuckerberg Initiative), and with the encouragement of OMB/OIRA (Office of Information and Regulatory Affairs), a short-term interagency group, called The Common Rule Modernization Working Group (CRMWG), was created in February 2014. Membership in the CRMWG included representatives from the Common Rule agencies/departments. It was co-chaired by Dr. Jerry Menikoff, Director of the Office of Human Research Protections (OHRP) at HHS; Dr. Joanne S. Tornow, Deputy Assistant Director of the SBE Directorate at NSF (now Acting Assistant Director for Biological Sciences); and me. This was the first opportunity for interagency discussion beyond HHS addressing possible modernization of the regulations. For a number of reasons, the interagency review and approval process can often be difficult and move very slowly, however progress was made in this working group and the process of drafting a Notice of Proposed Rulemaking (NPRM) was started.

In addition to interagency conversations, individual agency/department review and approval was also required. Again, this can also be a difficult process, particularly if this activity is not a key priority for an agency. The working group also considered advice from a variety of outside stakeholders, including official comments and other reports and documents. For example, the National Academies Division of Behavioral and Social Sciences and Education (DBASSE) and its boards, including the Board on Behavioral, Cognitive and Social Sciences (BBCSS) created an ad hoc committee that produced a key report in 2014. This consensus report, entitled Proposed Revisions to the Common Rule for the Protection of Human Subjects in the Behavioral and Social Sciences “examine[d] how to update human subjects protections regulations so that they effectively respond to current research contexts and methods. With a specific focus on social and behavioral sciences, [it aimed] to address the dramatic alterations in the research landscapes that institutional review boards (IRBs) have come to inhabit during the past 40 years. The report [aimed] to balance respect for the individual persons whose consent to participate makes research possible and respect for the social benefits that productive research communities make possible.” Reports such as this one were taken very seriously by the CRMWG.

I left OSTP in February 2015, and Tania Simoncelli took over the lead for OSTP. Discussion, drafting, and agency review and approval continued throughout 2015, with constant attention from OIRA and OHRP. The NPRM was published in the Federal Register on September 8, 2015. Again, comments were sought from stakeholders “… on proposals to better protect human subjects involved in research, while facilitating valuable research and reducing burden, delay, and ambiguity for investigators.” The NPRM posed a number of questions to stakeholders and generated over 2,100 comments and raised considerable concerns. As a result of these comments, HHS and the other federal agencies made a number of significant changes from the proposed rule.

On January 19, 2017, the last day of the Obama Administration, the “Final Rule” was published in the Federal Register. Key elements included: improvements to the consent process; requirements, in many cases, to use a single institutional review board (IRB) for multi-institutional research studies; changes related to biospecimens and consent; establishment of new exempt categories of research based on the level of risk they pose to participants; and removal of the requirement to conduct continuing review of ongoing research studies in certain instances where such review does little to protect subjects. Although the Trump administration has appeared supportive, its general approach has been to carefully scrutinize regulations developed by the previous administration, particularly those released during its final year. In addition, many concerns have been raised by the research community, the public, advocacy groups, federal agencies, and other stakeholders, about the proposed, revised Common Rule, related to inconsistencies, complexity, and other issues.

An interim final rule (IFR), titled Federal Policy for the Protection of Human Subjects: Delay of the Revisions to the Federal Policy for the Protection of Human Subjects, was released on January 17, 2018, that delays the effective date and general compliance date of the revised Common Rule for six months, with July 19, 2018 as the new effective date and general compliance date. All provisions are covered except the cooperative research (“single IRB”) provision, whose compliance date remains January 20, 2020. July 19, 2018 is the new effective date and general compliance date for the revised Common Rule. According to the IFR, “The federal departments and agencies listed in this document are in the process of developing a proposed rule to further delay implementation of the 2018 Requirements. The limited implementation delay accomplished by this interim final rule both provides additional time to regulated entities for the preparations necessary to implement the 2018 Requirements, and additional time for the departments and agencies listed in this document to seek input from interested stakeholders through a notice and comment rulemaking process that allows for public engagement on the proposal for a further implementation delay.”

So, where do things currently stand? Federal agencies are considering an additional delay until January 21, 2019. Because of the change of administration, it was not clear what would happen with the regulations. There appears to be interest in the agencies and OMB in moving ahead with an eventual rollout, but additional time might be needed to manage this complex process. In general, our community is supportive of the proposed changes that they have seen so far. For some, the possible additional delays are very frustrating. It is important that the agencies be given adequate time for implementation, should things move ahead. Critical to continued progress on these matters is community engagement and dialogue with the federal government. Leadership and vision in the academic community has come from many. I am encouraged to see the degree of engagement on the part of professional societies and other organizations that advocate for, or support, our communities. At the federal level, these issues continue to be of importance. Behind the scenes, many individuals at OHRP and OMB/OIRA have labored tirelessly to consider the broad policy impacts of regulatory change, the complexity of the details and their implementation, and the need to engage and balance the often competing interests of a broad array of stakeholders. Staff at many federal agencies, from NSF to the Department of Defense, have spent countless hours lending their expertise, reviewing documents, and working cooperatively.

For now, I counsel patience. The federal government represents a broad array of interests and stakeholders. Getting two agencies to work together can be extremely difficult. Herding sixteen of them can be extremely challenging, and can result in things moving very slowly. In addition, the implementation process at individual agencies or departments can be complex, including the development of agency-specific guidance and training. This sometimes requires multiple levels of internal scrutiny and approval. Things appear to be moving ahead, but additional delays may occur. As we wait, other steps can be taken or considered. I hope that at the federal level an interagency process be put back in place, if it has not already been done. As future changes happen related to human subjects and, possibly, problems emerge, this will help foster communication within government and make sure that the diversity of opinions and approaches reflected in the many agencies joined in the Common Rule and the communities that they serve are adequately represented. If there are additional delays in this process, use this as an opportunity to indicate your support for those aspects of the proposed regulations that are important to you and your colleagues, and encourage the government to make sure that these changes are eventually enacted. Also needed at some agencies is a better understanding and appreciation of the areas of mind, brain, and behavior and the methodologies used by these communities in basic research. If you have thoughts about these matters, I encourage you to make sure that your voice is heard, individually and/or collectively, through your organization, institution, professional societies, advocacy groups, etc. Stay engaged in the process, but also be patient, if possible. Working together is our best shot for increasing protections for research participants, making critical advances in research and education, and strengthening our nation.

Philip Rubin, Ph.D.
Chief Executive Officer Emeritus
Haskins Laboratories
prubin49@gmail.com

P.S. I am sorry about all the acronyms — welcome to the government!


Philip Rubin, Ph.D. is the Chief Executive Officer emeritus and former Senior Scientist at Haskins Laboratories. He is also an adjunct professor in the Department of Surgery, Otolaryngology, at the Yale University School of Medicine, a Research Affiliate in the Department of Psychology at Yale, and a Fellow at Yale’s Trumbull College. From 2012 through February 2015, Rubin was the Principal Assistant Director for Science in the Office of Science and Technology Policy (OSTP) in the Executive Office of the President of the United States, where he also served as Assistant Director for Social, Behavioral, and Economic Sciences, led the White House neuroscience initiative, and was a senior advisor on national policy. During that period of time he was also a Senior Advisor in the Social, Behavioral and Economic Sciences directorate at the National Science Foundation (NSF). He is the former co-chair of the National Science and Technology Council (NSTC) Committee on Science, with the Directors of the NIH and NSF, and co-chair of the interagency Common Rule Modernization Working Group.

From 2006-2011, Rubin was the Chair of the National Academies Board on Behavioral, Cognitive, and Sensory Sciences. From 2009-2011, he was a member-at-large of the Board of the Federation of Associations in Behavioral & Brain Sciences. From 2000 through 2003, Dr. Rubin served as the Director of the Division of Behavioral and Cognitive Sciences at the NSF. While there he was the NSF ex officio representative to the National Human Research Protection Advisory Committee and the Secretary’s Advisory Committee on Human Research Protections, established to provide advice to the Secretary of Health and Human Services. He was also the co-chair of the interagency NSTC Committee on Science Human Subjects Research Subcommittee under the auspices of OSTP.

Limping Toward the Goal Posts: Basic Science and Clinical Trials at NIH

Paula Skedsvold and Jeremy Wolfe*

In the summer of 2017, researchers in the mind, brain, and behavioral science communities became engaged in a dialogue of sorts with the National Institutes of Health about their plan to reclassify a great deal of basic science research as “clinical trials” and, as a result, subject that research to a number of policies specific to clinical trials. Our conversation with NIH has progressed, but it is not finished.

Progress

To be sure, there has been progress. Through interactions with NIH staff, the case studies that define whether a research project is or is not a clinical trial have been revised on multiple occasions, resulting in an apparent narrowing of the reach of the clinical trial definition to include less basic behavioral and brain science. The latest revision of the case studies can be found here.

Case 18 (in its many parts) has been central to the definition of clinical trials from the vantage point of basic behavioral and brain science. That case is revised in the newest list and it seems that much of our basic science (behavioral and fMRI) has been removed from the clinical trials framework. Cases 24 and 26 have also been of concern to our science community. In the latest revision, these case studies have been removed. It is unclear at this time whether the case studies are undergoing revision and will be posted later.

Beyond the case studies, there is other progress to report. Basic scientists have been concerned about a variety of issues including the implications of the new policies for grant application and review and for training from undergraduate to postdoctoral levels. To get a better understanding about progress made in these areas, FABBS immediate Past-President Jeremy Wolfe posed a series of questions to NIH’s Deputy Director for Extramural Research, Michael Lauer. These questions and NIH’s answers were just published in Nature Human Behavior.

In addition, NIH is signaling that the agency is prepared to troubleshoot new issues related to implementation of the clinical trials policies as they arise. We are interacting with the agency about the best ways to collect and convey this information.

Getting to the Core Issue(s)

As we understand it, NIH’s goals in taking these actions were to address problems identified by the Government Accountability Office in collecting information about the clinical trials it funded. In addition, there were concerns that the results of clinical trials were not being published in a timely manner.

As FABBS and other organizations have conveyed to NIH, there are alternative vehicles for registering studies and reporting findings that are better tailored for basic science. For those basic science studies that are still deemed to be clinical trials, other systems such as the Open Science Framework may be a user-friendly alternative. Since last summer, there has been movement in this direction as well. NIH is currently in discussions with Open Science Framework Director Brian Nosek about the possibility that basic scientists might register and report through OSF rather than clinicaltrials.gov, if they so choose. Doing so would reduce the number of irrelevant fields for basic scientists and, hopefully, make the process easier. According to Nosek, OSF and NIH are working through technical and implementation details so that, if an investigator used OSF, it would be as efficient and effective a process as possible. Since NIH needs a public database to house the information, data elements from OSF would be pulled into clinicaltrials.gov.

A Path Forward

Most basic scientists would like to maintain the historical distinction between basic science and clinical trials. The community is not opposed to the goals of openness and accountability. However, the path to those goals does not require a fundamental change in the meaning of the term “clinical trial.” Although there is progress with NIH, it is unclear where this will land, but we will continue to interact with agency staff to get this right. As long as the policies affect basic brain and behavioral science, we will be at the table trying to make them work for our scientists. We will also continue to share information with organizations such as the Council on Governmental Relations, which has taken a leadership role on this issue for higher education organizations.

FABBS is also engaged in conversations with key staff on Capitol Hill about the continued concerns of the community, and Congressional staff are in communication with NIH. These conversations will also continue. While it is not an easy process for anyone involved, we are hopeful that we will continue to make progress one way or another.


*Paula Skedsvold, JD PhD, is Executive Director of the Federation of Associations in Behavioral and Brain Sciences (FABBS).

 

 

 

 

*Jeremy Wolfe, PhD, is immediate Past-President of FABBS and Professor at Brigham & Women’s Hospital and Harvard Medical School.