Public Sector Learning Program Evaluation: Fill & Download for Free

The best ways to learn social sector consulting are: (1) be employed by a social sector consulting firm; (2) prepare and submit proposals to a Government agency that has issued an RFP for a social sector project; and (3) if you are successful in the proposal, work on the contract and build your expertise.Social sector consulting might include the following types of services- program evaluation, policy analysis, training and technical assistance, management reporting systems, cost analysis, economic analysis, and financial management. These services might be applied to client agencies at the Federal State or local levels, as well as other non-profit agencies. Social sector consulting might be applied to the following types of programs- healthcare, substance abuse, education, social welfare, housing, urban planning, and criminal justice.My own management consulting career began with a small public sector consulting firm in Pittsburgh, specializing in services to State and local government agencies. I did not specifically seek this segment of the consulting space to work on. However, in my work as a system analysis engineer at General Electric Vo., I had got involved in a healthcare quasi consulting engagement to plan a new extended care facility. While the healthcare field looked interesting,I could just as well have started my consulting career in private sector consulting or military consulting. The opportunity was present with a small public sector consulting firm (which included some of the social services sector) and I pursued it. As it turned out, I had a real passion for management consulting and did well in it.The public sector focus, which had further focus in Federal Government and health, education and environmental agencies, seemed to click. I didn’t know that much about social sector consulting when I started, but at my first consulting firm (CONSAD Research Corp), I got involved in healthcare and substance abuse consulting, along with other fields, and I gradually increased my understanding of the field. I wrote competitive proposals, and some of them won.After 3.5 years with two small public sector consulting firms, I joined the large management consulting firm Booz Allen Hamilton just before my 30th birthday. I became a vice president 3.5 years later. Later in mid career, I started my own consulting firm and ran it for 20 years before retirement. I retained some of the focus in healthcare/social service consulting in my own firm.

The short answer is that I think it will have an enormous impact; in the early days, as used “off the shelf,” but in the longer run econometricians will modify the methods and tailor them so that they meet the needs of social scientists primarily interested in conducting inference about causal effects and estimating the impact of counterfactual policies (that is, things that haven’t been tried yet, or what would have happened if a different policy had been used). Examples of questions economists often study are things like the effects of changing prices, or introducing price discrimination, or changing the minimum wage, or evaluating advertising effectiveness. We want to estimate what would happen in the event of a change, or what would have happened if the change hadn’t taken place.As evidence of the impact already, Guido Imbens and I attracted over 250 economics professors to an NBER session on a Saturday afternoon last summer, where we covered machine learning for economists, and everywhere I present about this topic to economists, I attract large crowds. I think similar things are true for the small set of other economists working in this area. There were hundreds of people in a session on big data at the AEA meetings a few weeks ago.Machine learning is a broad term; I’m going to use it fairly narrowly here. Within machine learning, there are two branches, supervised and unsupervised machine learning. Supervised machine learning typically entails using a set of “features” or “covariates” (x’s) to predict an outcome (y). There are a variety of ML methods, such as LASSO (see Victor Chernozhukov (MIT) and coauthors who have brought this into economics), random forest, regression trees, support vector machines, etc. One common feature of many ML methods is that they use cross-validation to select model complexity; that is, they repeatedly estimate a model on part of the data and then test it on another part, and they find the “complexity penalty term” that fits the data best in terms of mean-squared error of the prediction (the squared difference between the model prediction and the actual outcome). In much of cross-sectional econometrics, the tradition has been that the researcher specifies one model and then checks “robustness” by looking at 2 or 3 alternatives. I believe that regularization and systematic model selection will become a standard part of empirical practice in economics as we more frequently encounter datasets with many covariates, and also as we see the advantages of being systematic about model selection.Sendhil Mullainathan (Harvard) and Jon Kleinberg with a number of coauthors have argued that there is a set of problems where off-the-shelf ML methods for prediction are the key part of important policy and decision problems. They use examples like deciding whether to do a hip replacement operation for an elderly patient; if you can predict based on their individual characteristics that they will die within a year, then you should not do the operation. Many Americans are incarcerated while awaiting trial; if you can predict who will show up for court, you can let more out on bail. ML algorithms are currently in use for this decision in a number of jurisdictions. Goel, Rao and Shroff presented a paper at the AEA meetings a few weeks ago using ML methods to examine stop-and-frisk laws. See also the interesting work using ML prediction methods in the session I discussed on “Predictive Cities”: 2016 ASSA Preliminary Program where we see ML used in the public sector.Despite these fascinating examples, in general ML prediction models are built on a premise that is fundamentally at odds with a lot of social science work on causal inference. The foundation of supervised ML methods is that model selection (cross-validation) is carried out to optimize goodness of fit on a test sample. A model is good if and only if it predicts well. Yet, a cornerstone of introductory econometrics is that prediction is not causal inference, and indeed a classic economic example is that in many economic datasets, price and quantity are positively correlated. Firms set prices higher in high-income cities where consumers buy more; they raise prices in anticipation of times of peak demand. A large body of econometric research seeks to REDUCE the goodness of fit of a model in order to estimate the causal effect of, say, changing prices. If prices and quantities are positively correlated in the data, any model that estimates the true causal effect (quantity goes down if you change price) will not do as good a job fitting the data. The place where the econometric model with a causal estimate would do better is at fitting what happens if the firm actually changes prices at a given point in time—at doing counterfactual predictions when the world changes. Techniques like instrumental variables seek to use only some of the information that is in the data – the “clean” or “exogenous” or “experiment-like” variation in price—sacrificing predictive accuracy in the current environment to learn about a more fundamental relationship that will help make decisions about changing price. This type of model has not received almost any attention in ML.In some of my research, I am exploring the idea that you might take the strengths and innovations of ML methods, but apply them to causal inference. It requires changing the objective function, since the ground truth of the causal parameter is not observed in any test set. Statistical theory plays a bigger role, since we need a model of the unobserved thing we want to estimate (the causal effect) in order to define the target that the algorithms optimize for. I’m also working on developing statistical theory for some of the most widely used and successful estimators, like random forests, and adapting them so that they can be used to predict an individual’s treatment effects as a function of their characteristics. For example, I can tell you for a particular individual, given their characteristics, how they would respond to a price change, using a method adapted from regression trees or random forests. This will come with a confidence interval as well. You can search for my papers on arXiv.org e-Print archive; I also wrote a paper on using ML methods to systematically asses the robustness of causal estimates in the American Economic Review last year. I hope that some of these methods can be applied in practice to evaluate randomized controlled trials, A/B tests in tech firms, etc. in order to discover systematically heterogeneous treatment effects.Unsupervised machine learning tools differ from supervised in that there is no outcome variable (no “y”): these tools can be used to find clusters of similar objects. I have used these tools in my own research to find clusters of news articles on a similar topic. They are commonly used to group images or videos; if you say a computer scientist discovered cats on YouTube, it can mean that they used an unsupervised ML method to find a set of similar videos, and when you watch them, a human can see that all the videos in cluster 1572 are about cats, while all the videos in cluster 423 are about dogs. I see these tools as being very useful as an intermediate step in empirical work, as a data-driven way to find similar articles, reviews, products, user histories, etc.

On paper you have a somewhat similar background as I had when I got into consulting at the age of 26. Let me describe how I made the transition to consulting, and then I’ll provide some suggestions. I had a BSEE from the Univ. of Pennsylvania and a little under 5 years’ experience at Bell Telephone Co. of PA (1.5 years) and General Electric (3.25 years) as an engineer.I was working as a System Analysis Engineer. I never liked engineering, not being technically inclined, although I was analytically oriented, but applied to business more than science or engineering. I developed an interest in operations research and started sending resumes to various small firms practicing OR/management science. That gave me some insight into management consulting.Trying to diversify from the aerospace business, GE got involved in a hospital planning project. I was working on the project as a junior analyst when the project manager resigned. At age 24 I was managing a project to develop a new extended care facility. It involved data collection by a team of registered nurses, who reviewed medical records from a large acute care general hospital. This was closer to consulting and my interest increased.I accelerated sending out resumes and got a job with a small public sector consulting firm, CONSAD Research Corporation in Pittsburgh, PA. In obtaining that position, I had expressed my interest in consulting to two colleagues at GE and a hospital planning firm GE was working with; they contacted my target firm on my behalf.That was the beginning of a 40-year career in management consulting. To build on my experience in public sector consulting, I moved to Wash., DC, where most of the Federal Government clients were. I got a job as a senior analyst with a small consulting firm in Silver Spring, MD, and then joined Booz Allen Hamilton as a consultant just before my 30th birthday. I became a vice president 3.5 years later.I started my own consulting firm in my mid 40s, coincident with going back part time to complete my MBA degree in Finance. Although I had been successful in consulting, I always felt a little lacking without an advanced degree. Since at many of my clients like the National Institutes of Health, the majority of professionals had advanced degrees and many had doctorates, I felt more confident when I obtained the MBA credential.I selected consulting because it: (1) supported many of my interests and strengths, including analysis, writing, organization, presentation, and business; (2) involved helping organizations and people; (3) was flexible to support working for various size firms; and (4) was reasonably prestigious and offered a good income.My consulting practice included management reviews, process improvement, program evaluation, management systems development, and facilitation, mainly for Federal Gov’t agencies in the health and environmental fields. I developed a specialty in performance-based contracting. I ran my firm for 20 years before retiring.If you are really interested in consulting. I suggest the following:1. Assess your potential fit for consulting. In several other posts, I listed the following skills as being characteristic of successful management consultants: (1) strong analytical skills; (2) high energy level, including above average initiative; (3) good communication skills (writing, presentation and interpersonal); (4) assertive and reasonably competitive; (5) flexible and willing to change; and (6) reasonably creative (development of recommendations).2. Get involved in either a part time or full time MBA program. Although I didn’t have an MBA when I entered consulting, it would have been easier if I did. I had started an MBA part time at Bell Telephone and GE, but didn’t continue. I was very assertive and had some fortuitous events. I don’t think you should count on similar events toward your cause. A full time MBA is preferable, if possible.3. Perform an outstanding job in your current position. I don’t know what the scope of your current engineering work is. Try to gain experience in project management, data analysis, writing, oral presentation, client relations, and process improvement.4. Apply to a broad spectrum of consulting firms- large, medium and small. When I did this, I sent resumes out via regular mail. Today, you would use the Internet, looking at LinkedIn and the Web sites of target consulting firms.5. Develop your interest in consulting by reading articles and books about your desired field of consulting. (When I did this, I focused on operations research, which morphed into management consulting.)6. To further develop your interest in consulting, consider joining a relevant professional association. I wouldn’t do this first, but it can be a source of learning more about the consulting profession and developing contacts.7. Obtain a position with a smaller consulting firm. This may be your entre to a top large consulting firm (if that is your goal).