Shari Lawrence Pfleeger
In her long career, RISCS advisory board member Shari Lawrence Pfleeger has served as developer and maintainer for real-time, business-critical software systems, president of Systems/Software, Inc., principal scientist at the Contel Technology Center, principal scientist at MITRE Corporation’s Software Engineering Center, and a senior researcher at RAND Corporation. From 2010 to 2016 she served as the research director for Dartmouth College’s Institute for Information Infrastructure Protection, a consortium of 26 US universities and national laboratories doing cyber security research; she is also former editor-in-chief of IEEE Security & Privacy. RISCS director Angela Sasse cites Lawrence Pfleeger’s pioneering multi-disciplinary work as an inspiration for the design of RISCS.
Lawrence Pfleeger set out to explain the difficulties and benefits of approaching problems from multiple perspectives. Her training as a mathematician taught her to break problems up into pieces and organise them to make the sub-problems more tractable. Lawrence Pfleeger particularly praised the the earlier talk by Yasemin Acar because, in presenting her research with Sascha Fahl, Acar included the changes in study design made after the pilot test and explained their findings’ limitations. Failing to include this type of information is a common problem, and it presents difficulties for the final stage of interdisciplinary research: fitting all the pieces together. Lawrence Pfleeger went on to discuss a number of other issues that create problems in that process: invention versus innovation; the different approaches different disciplines take; the lack of a common language; and a group of practical barriers posed by academia and publication in general.
Invention versus innovation
In a November 2017 article at The Atlantic, Derek Thompson distinguished an invention from an innovation. He contended that inventions require scientists and researchers to work in a lab insulated from profit considerations, while innovation attempts to put an invention to commercial use, encouraged by competition and consumer choice. The difference can be illustrated by examining the transistor versus the transistor radio. The former is an invention; the latter is an innovation that required many stages of ideas and discoveries, many paths and dead ends, before becoming a product; even then, the timing and social environment must be right. Video telephones, for example, were demonstrated at the 1964 New York World’s Fair and dismissed for social reasons; an improved version in 1992 met the same fate. Today, everyone has them, and they are widely used, especially by younger people.
The two types of progress require different scientific approaches. Inventions require mostly mathematics and engineering science; they are focused on either entirely new functionality (for example, the video transmitter) or a new way of providing an existing functionality, either as a breakthrough or as a series of incremental inventions (video telephones). For innovations, many disciplines must align in the right way, and people and market-oriented sciences are as important as the invention itself. Most important is understanding people, their needs, and how they will want to accomplish something in the near future.
Different disciplines, different approaches
Every discipline has its own notion of a scientific approach. The traditional scientific method, in which you begin with a question, consider the background, form a hypothesis, experiment to test the hypothesis, collect observations, draw a conclusion, and explain what you learned, is not the way all scientists do science or the way some social sciences work. Much of social science is observation: how are people behaving and why, and how they might be induced to change.
Consequently, experiments may not be the right place to start. There may be no hypothesis. It may not – as in software – be possible to build two options and compare the results. Not every variable may be controllable, as in Acar’s study. Essential variables may not always be evident until a study or a product fails. Measurements may be hard or impossible to take. Regulatory issues – such as data protection law – may interfere with development.
Lawrence Pfleeger’s suggested solution is to adapt the scientific method to the situation at hand as necessary, while ensuring that all parties are agreed that the method is sound enough that the results of the work can be used as a basis for next steps.
In search of a common language
On a project that brought together social scientists and computer scientists, Lawrence Pfleeger encountered many language problems. The clearest example is the world “code”, which meant entirely different things to the two groups. The project had to devise terminology that all could understand and use consistently.
As a result of that experience, she recommends starting a glossary, populating it with the terminology you want to use in discussing the project, and agreeing on common definitions. Even when some processes seem the same they may be very different; anthropologists and psychologists do coding very differently from each other.
Assumptions and principles may also seem the same yet be quite different; for example, there may be profound disagreements about acceptable sampling and sample sizes. Everyone is taught that samples must be randomised, but the population being studied may not have a normal distribution and a particular innovation may therefore need a different sampling method. At RAND, the Department of Justice funded a cyber security survey of US businesses with an intended sample size of 11,000. To ensure the businesses in the sample were sufficiently varied, the researchers used stratified sampling, a two-step process that first identifies the distribution of business types, and then does random sampling within each stratum. In other cases, “snowball sampling” is used to ask those surveyed first to pass the survey on to their friends. There are dozens of types of sampling, each with a specific intention and each requiring different statistical analysis.
In search of common principles
Every discipline has its own set of principles, both explicit and implicit. Lawrence Pfleeger presented some preliminary work by William Butz about making each discipline’s principles explicit. For physics, the principles are usually clearly-stated and based on observation and experiment. For ethics, implicit principles include the respect for individuals as autonomous agents. A more general principle is independent verifiability, incorporating informed consent where applicable. Sociology’s principles or assumptions include the ideas that people behave differently in groups than they do as individuals; that societies are organised into distinct social units that set rules for their members; and that any group of people has characteristics that individual members do not. Finally, statistics holds that it is possible to know something about a whole group by examining only a subset.
A multidisciplinary project needs to consider which of these principles should be acknowledged and used. Then, based on those principles, the same discussion needs to take place about the processes that will be used to understand a problem, conduct research, and apply findings.
In search of appropriate measures of success
Projects need both short-term and long-term measures of success appropriate to the problem they’re trying to solve. Cyber security is full of bad examples, particularly selection bias – examining only successful gamblers, for example, would lead you to the conclusion that gambling is inevitably profitable. Lawrence Pfleeger recommends starting with simple measures.
For academics, publishing outside their own disciplines may pose problems regarding promotions and tenure. How, for example, will the committee know about or be able to evaluate a computer scientist’s publications in a social science journal?
Good measures are needed for social as well as technological impact. At RAND, it was tempting to focus on the easy-to-count number of downloads of a particular report, but if the goal is changing policy only one download by the right person may suffice.
Ethical review – in the US, known as scrutiny by an Institutional Review Board (IRB) – is necessary when people are involved. Earlier, Acar mentioned that in Germany ethical review is not yet considered in computer science research.
There are likely also to be organisational barriers to multidisciplinary work. Lawrence Pfleeger’s recommendations:
– Agree on the allocation of credit and intellectual property ownership in advance. Consortia have been killed by such disagreements.
– Foster mutual respect among team members. In one study Lawrence Pfleeger found the technologists had no respect for social scientists unless they knew how to write computer code.
– Consider how projects fit with doctoral research and its time scales.
– Consider the role of public policy. A particular innovation may require rule changes. The differing privacy laws between the EU and US are a good example of public policy barriers to research design and to application of findings.
Despite these known problems, Lawrence Pfleeger maintains that multi-disciplinary research has many benefits. Problem-solving gains from multiple perspectives, which generate new solutions and new ideas. The kinds of questions that get asked about assumptions help reveal weaknesses and suggest improvements. Applying new approaches can yield great breakthroughs.
As an example, Lawrence Pfleeger cited Google’s change of approach between the 2006 and 2016 versions of its Translate by comparing the results of translating the Jorge Luis Borges quotation, “Uno no es lo que es por lo que escribe, sino por lo que ha leído”. The first version took a statistical and mathematical approach that laid out a language’s definitions and rules, based on analysing billions of already-published web pages of human-translated material. The result: “One is not what is for what he writes, but for what he has read.” The later version incorporated a child psychologist’s view, using neural networks to learn language closer to the way a child does. The result: “You are not what you write, but what you have read.”
As a second example, she cited Computer Security Incident Response teams (CSIRTs). Each team member has individual competencies – specific skills and a desire to learn – but the team has collective competencies, such as the ability to pool resources and collaborative skills.
For all these reasons, Lawrence Pfleeger believes that the benefits of multi-disciplinary collaboration far outweigh the difficulties. Someone working on a project who needs expertise in statistics, for example, will find recruiting an expert from an adjacent department a better approach than trying to learn the subject themselves. “We need to learn to collaborate,” she concluded.
This talk was presented at the February 2018 RISCS Community meeting.