Senior Project Independent Study Time Log: Fill & Download for Free

Decent pay, faster than average job growth 16% in 10 years.Quick Facts: Medical and Clinical Laboratory Technologists and Technicians2015 Median Pay $50,550 per year $24.30 per hourNumber of Jobs, 2014 328,200Job Outlook, 2014-24 16% (Much faster than average)Employment Change, 2014-24 52,100What Medical and Clinical Laboratory Technologists and Technicians DoMedical laboratory technologists (commonly known as medical laboratory scientists) and medical laboratory technicians collect samples and perform tests to analyze body fluids, tissue, and other substances.Laboratory personnel wear protective masks, gloves, and goggles to ensure their safety.Medical laboratory technologists (commonly known as medical laboratory scientists) and medical laboratory technicians collect samples and perform tests to analyze body fluids, tissue, and other substances.DutiesMedical laboratory technologists and technicians typically do the following:Analyze body fluids, such as blood, urine, and tissue samples, and record normal or abnormal findingsStudy blood samples for use in transfusions by identifying the number of cells, the cell morphology or the blood group, blood type, and compatibility with other blood typesOperate sophisticated laboratory equipment, such as microscopes and cell countersUse automated equipment and computerized instruments capable of performing a number of tests at the same timeLog data from medical tests and enter results into a patient’s medical recordDiscuss results and findings of laboratory tests and procedures with physiciansSupervise or train medical laboratory techniciansBoth technicians and technologists perform tests and procedures that physicians and surgeons or other healthcare personnel order. However, technologists perform more complex tests and laboratory procedures than technicians do. For example, technologists may prepare specimens and perform detailed manual tests, whereas technicians perform routine tests that may be more automated. Medical laboratory technicians usually work under the general supervision of medical laboratory technologists or laboratory managers.Technologists in small laboratories perform many types of tests; in large laboratories, they sometimes specialize. The following are examples of types of specialized medical laboratory technologists:Blood bank technologists, or immunohematology technologists, collect blood, classify it by type, and prepare blood and its components for transfusions.Clinical chemistry technologists prepare specimens and analyze the chemical and hormonal contents of body fluids.Cytotechnologists prepare slides of body cells and examine these cells with a microscope for abnormalities that may signal the beginning of a cancerous growth.Immunology technologists examine elements of the human immune system and its response to foreign bodies.Microbiology technologists examine and identify bacteria and other microorganisms.Molecular biology technologists perform complex protein and nucleic acid tests on cell samples.Like technologists, medical laboratory technicians may work in several areas of the laboratory or specialize in one particular area. For example, histotechnicians cut and stain tissue specimens for pathologists, who are doctors who study the cause and development of diseases at a microscopic level.Technologists and technicians often specialize after they have worked in a particular area for a long time or have received advanced education or training in that area.Work EnvironmentAbout half of all medical laboratory technologists and technicians were employed in hospitals in 2014. Others worked in doctors’ offices or diagnostic laboratories.Medical laboratory technologists operate sophisticated laboratory equipment, such as microscopes and cell counters.Medical laboratory technologists held about 164,800 jobs in 2014. The industries that employed the most medical laboratory technologists in 2014 were as follows:Hospitals; state, local, and private 58%Medical and diagnostic laboratories 17%Offices of physicians 8%Colleges, universities, and professional schools; state, local, and private 5%Medical laboratory technicians held about 163,400 jobs in 2014. The industries that employed the most medical laboratory technicians in 2014 were as follows:Hospitals; state, local, and private 44%Medical and diagnostic laboratories 19%Offices of physicians 12%Colleges, universities, and professional schools; state, local, and private 5%Medical laboratory personnel are trained to work with infectious specimens or with materials that are caustic or produce fumes. When they follow proper methods to control infection and sterilize equipment, the risk decreases. They wear protective masks, gloves, and goggles for their safety.Technologists and technicians can be on their feet for long periods, and they may need to lift or turn disabled patients to collect samples.Work SchedulesMost medical laboratory technologists and technicians work full time. Technologists and technicians who work in facilities that operate around the clock, such as hospitals and some independent laboratories, may work evening, weekend, or overnight hours.How to Become a Medical and Clinical Laboratory Technologist or TechnicianMedical laboratory technologists typically need a bachelor’s degree. Technicians usually need an associate’s degree or a postsecondary certificate. Some states require technologists and technicians to be licensed.Medical laboratory technologists typically need a bachelor’s degree.Medical laboratory technologists typically need a bachelor’s degree. Technicians usually need an associate’s degree or a postsecondary certificate. Some states require technologists and technicians to be licensed.EducationAn entry-level job for technologists usually requires a bachelor's degree in medical technology or life sciences.A bachelor’s degree program in medical laboratory technology, also known as a medical laboratory scientist degree, includes courses in chemistry, biology, microbiology, math, and statistics. Coursework emphasizes laboratory skills, including safety procedures and lab management.The courses may be offered through a university or hospital-based program that students attend during their senior year of college. College graduates who major in other sciences and meet a program’s prerequisites, such as having completed required courses in biology and chemistry or maintaining a certain GPA, also may apply to a medical laboratory science program.Medical laboratory technicians often complete an associate’s degree program in clinical laboratory science. A limited number of 1-year certificate programs are available from hospitals, and admission requirements vary. The Armed Forces and vocational or technical schools also may offer certificate programs for medical laboratory technicians. Technician coursework addresses the theoretical and practical aspects of each of the major laboratory disciplines.High school students who are interested in pursuing a career in the medical laboratory sciences should take classes in chemistry, biology, and math.Licenses, Certifications, and RegistrationsSome states require laboratory personnel to be licensed. Requirements vary by state and specialty. For specific requirements, contact state departments of health, state boards of occupational licensing, or visit The American Society for Clinical Laboratory Science.Certification of medical laboratory technologists and technicians is required for licensure in some states. Although certification is not required to enter the occupation in all cases, employers typically prefer to hire certified technologists and technicians.Medical laboratory technologists and technicians can obtain a general certification as a medical laboratory technologist or technician, respectively, or a certification in a specialty, such as cytotechnology or medical biology. Most credentialing institutions require that technologists complete an accredited education program in order to qualify to sit for an exam. For more credentialing information, visit the National Accrediting Agency for Clinical Laboratory Sciences.Important QualitiesAbility to use technology. Medical laboratory technologists and technicians must understand how to operate complex machinery.Detail oriented. Medical laboratory technologists and technicians must follow exact instructions in order to perform tests or procedures correctly.Dexterity. Medical laboratory technologists and technicians need to be skilled with their hands. They work closely with needles and precise laboratory instruments and must handle these tools effectively.Physical stamina. Medical laboratory technologists and technicians may work on their feet for long periods while collecting samples. They may need to lift or turn disabled patients to collect samples for testing.AdvancementAfter additional education, work experience, or certification, technologists and technicians may specialize in one of many areas of laboratory science, such as immunology, histotechnology, or clinical chemistry. Some medical laboratory technicians advance to technologist positions after gaining experience and additional education.PayThe median annual wage for medical and clinical laboratory technologists and technicians was $50,550 in May 2015.Medical and Clinical Laboratory Technologists and TechniciansMedian annual wages, May 2015Medical and clinical laboratory technologists $60,520Medical and clinical laboratory technologists and technicians $50,550Health technologists and technicians $42,190Medical and clinical laboratory technicians $38,970Total, all occupations $36,200Note: All Occupations includes all occupations in the U.S. Economy.Source: U.S. Bureau of Labor Statistics, Occupational Employment StatisticsThe median annual wage for medical and clinical laboratory technologists was $60,520 in May 2015. The median wage is the wage at which half the workers in an occupation earned more than that amount and half earned less. The lowest 10 percent earned less than $41,510, and the highest 10 percent earned more than $84,300.The median annual wage for medical and clinical laboratory technicians was $38,970 in May 2015. The lowest 10 percent earned less than $25,890, and the highest 10 percent earned more than $60,810.In May 2015, the median annual wages for medical laboratory technologists in the top industries in which they worked were as follows:Hospitals; state, local, and private $61,300Medical and diagnostic laboratories 61,010Offices of physicians 57,550Colleges, universities, and professional schools; state, local, and private 54,420In May 2015, the median annual wages for medical laboratory technicians in the top industries in which they worked were as follows:Colleges, universities, and professional schools; state, local, and private $40,290Hospitals; state, local, and private 39,890Offices of physicians 39,320Medical and diagnostic laboratories 37,410Most medical laboratory technologists and technicians work full time. Technologists and technicians who work in facilities that are always open, such as hospitals and some independent laboratories, may work evening, weekend, or overnight hours.Job OutlookEmployment of medical laboratory technologists and technicians is projected to grow 16 percent from 2014 to 2024, much faster than the average for all occupations. An increase in the aging population is expected to lead to a greater need to diagnose medical conditions, such as cancer or type 2 diabetes, through laboratory procedures.Medical and Clinical Laboratory Technologists and TechniciansPercent change in employment, projected 2014-24Medical and clinical laboratory technicians 18%Health technologists and technicians 16%Medical and clinical laboratory technologists and technicians 16%Medical and clinical laboratory technologists 14%Total, all occupations 7%Note: All Occupations includes all occupations in the U.S. Economy.Source: U.S. Bureau of Labor Statistics, Employment Projections programEmployment of medical laboratory technologists is projected to grow 14 percent from 2014 to 2024, much faster the average for all occupations. Employment of medical laboratory technicians is projected to grow 18 percent from 2014 to 2024, much faster than the average for all occupations.An increase in the aging population is expected to lead to a greater need to diagnose medical conditions, such as cancer or type 2 diabetes, through laboratory procedures. Prenatal testing for various types of genetic conditions also is increasingly common. Medical laboratory technologists and technicians will be in demand to use and maintain the equipment needed for diagnosis and treatment.The number of individuals who have access to health insurance is expected to continue to increase because of federal health insurance reform. As a result, demand for the services of laboratory personnel may grow as more patients who were previously uninsured, or found treatment to be cost-prohibitive, seek laboratory tests.Job ProspectsJob prospects will be best for medical and clinical laboratory technologists and technicians who complete an accredited education program and earn professional certification.Employment projections data for medical and clinical laboratory technologists and technicians, 2014-24SOURCE: U.S. Bureau of Labor Statistics, Employment Projections programClinical laboratory technologists and techniciansEmployment, 2014 328,200Projected Employment, 2024 380,300Change, 2014-24 Percent 16%, Numeric 52,100Medical and clinical laboratory technologistsEmployment, 2014 164,800Projected Employment, 2024 187,900Change, 2014-24 Percent 14%, Numeric 23,100Medical and clinical laboratory techniciansEmployment, 2014 163,400Projected Employment, 2024 192,400Change, 2014-24 Percent 18%, Numeric 29,000More Information, Including Links to O*NETLearn more about medical and clinical laboratory technologists and technicians by visiting additional resources, including O*NET, a source on key characteristics of workers and occupations.For more information about medical laboratory technologists and technicians, visitThe American Society for Clinical Laboratory ScienceAmerican Society of CytopathologyFor a list of accredited and approved educational programs for medical laboratory personnel, visitNational Accrediting Agency for Clinical Laboratory SciencesFor information on certification, visitAmerican Association of BioanalystsAmerican Medical TechnologistsAmerican Society for Clinical PathologyO*NETCytogenetic TechnologistsCytotechnologistsHistotechnologists and Histologic TechniciansMedical and Clinical Laboratory TechniciansMedical and Clinical Laboratory Technologists

My goals here are three-fold:To give non-Canadian readers a window into the story that’s going to determine whether Trudeau gets re-elected this year.To give Canadian readers a clearer and more complete sense of what’s happened than is easily found in any other single source.To make a few arguments re: what does and doesn’t matter here, and why. (And in the case of what doesn’t matter, outlining a theory on why some are pretending otherwise.)By necessity, what follows isn’t exactly short. But I’ve done my best to keep it as interesting and concise as possible while still hitting the above goals.For those unfamiliar with my writing, two quick notes: (1) My only interest in the partisan side of politics is deconstructing it. I have no team or tribe. (2) To ensure the most accurate takes possible, I offer rewards for all corrections.Ok, enough housekeeping.[EDIT: My original answer here was written March 1st. I returned on March 13th to make a few minor corrections and to address some common questions / objections. Where it made sense, I added edit snippets throughout the main text. Where I felt additions would be too distracting, I saved new commentary for the end. You can track all changes via the answer’s edit log.]Background ContextHere’s what brought us to today, in six bullet points:SNC-Lavalin is an EPC firm, which is to say that they’re the folks governments turn to when they want to contract out large infrastructure projects. SNC has roughly 50k employees globally, including 9k or so in Canada, with some 700 of those Canadian jobs being in the Montreal area (where SNC is currently headquartered). As recently as last October, SNC was worth about $9bn CAD, which is a fair amount for a Canadian company. If not quite a crown jewel, they were right at the top of the next tier down.SNC engaged in some shady stuff between 2001 and 2011, leading to a mire of lawsuits and investigations. As evidence of their misdeeds mounted, thousands of employees left, the board was made over, and a host of new compliance procedures were put in place to ensure that The Bad Times were behind them.In July 2012, the Harper government (the Conservative majority that preceded Trudeau’s Liberal majority) had Canada’s national contracting office revise their anti-corruption rules, with the net effect being that any vendor found guilty of certain crimes would be “rendered ineligible” for future federal contracts for a period of 10 years (reducible to 5 with good behavior). The Conservatives also made further amendments over the following years to reduce options for leniency, largely (it’s assumed) to position themselves against the Liberals, who had a party history of bedfellowing with shady corporations.For obvious reasons, SNC didn’t care much for this. They began lobbying for Canada to adopt what many other countries call deferred prosecution agreements (DPAs), which are something like plea deals, where a corporation can admit wrongdoing and submit to certain penalties and government-supervised renovations without bearing the full weight of a potentially crippling criminal conviction. (The logic here being that it’s not generally fair or useful to punish an entire enterprise for the sins of a few, especially if those few are no longer there.)The Trudeau government, which came to power in late 2015, ultimately did move forward with creating a DPA regime this past September. Unsurprisingly, SNC was quick to request an invitation for entry, arguing that they’d already done all the sorts of penance and reform that a DPA would require, and that further punishment was basically just value-destruction with no upside.The person responsible to decide on SNC’s request said no, setting off a chain of dramanoes just now reaching their crescendo.Now, before we can unpack the decision itself, we have to make a quick detour into the structure of the Canadian government — which I promise isn’t (quite) as boring as it sounds!Super Fun Learning Time!Trudeau, in addition to being Prime Minister (PM), is a Member of Parliament (MP) representing a riding in Montreal. That’s because in Canada the PM is always head of the executive branch and a sitting member of the legislature, with those two branches of government being heavily intertwined.[EDIT: Mike Hewson pointed out that all ministerial roles, including PM, can legally be filled by Senators and/or credentialed professionals, though this is only applicable in fringe cases where no suitable MP is found for a given role, which almost certainly would never be the case for PM — though this did happen twice in Canada’s early days when the sitting PM died in office.](I’m going to skip over the roles of the Queen, her Governor General, her Privy Council, and the Canadian Senate — mostly because those are all legacy institutions that hold marginal effective power today. If a PM has the backing, or “confidence”, of a majority of individual MPs, the PM effectively is the government. They own nearly all executive powers, and have enormous influence over legislation. Individual MPs have latitude to vote as they will, but those in the PM’s party will generally support the PM on all but rare “vote your conscience” items. The only part of the federal government that a majority-party PM has no real influence over is the judicial branch.)Anyway, there’s this other thing that the Harper government did (again, presumably) to brand themselves in distinction to the scandal-ridden Liberal Party of the early 2000s.In brief:The Conservatives instituted the Office of the Director of Public Prosecutions, which was to be subordinate to (but somehow independent of) the Department of Justice.The Department of Justice is headed by the Attorney General (AG), who is almost always a sitting MP selected for said purpose by the PM. In this new arrangement, the Director of Public Prosecutions (DPP) would be a civilian appointed by the sitting Attorney General.The idea here is that the DPP would be two steps removed from the PM, which would theoretically protect the former from undue political influence from the latter as far as deciding whom to prosecute.(Oh, and by the way, the Attorney General is also the Minister of Justice, which is a totally different hat that the same person always wears. It’s a confusing duality not worth getting into here, outside the basic idea that the same person is to be considered independent or not-so-independent depending on the hat they’re wearing in a given instant, which often leads to all the obvious complexities that one would expect.)Now, if you’re wondering what the division of power is between the AG and DPP on a practical level, there’s a handy guide for that very thing: Relationship between the Attorney General and the Director of Public Prosecutions.In a way, it’s the most “Canadian Politics in a Nutshell” thing you could ever read — almost 4,000 words of high-sounding rhetoric (“independence!”, “justice!”, “accountability!”) that accomplishes basically nothing outside of making the system even more convoluted and bureaucratic.Two reasons I say that:The DPP serves at the pleasure of the AG, who serves at the pleasure of the PM. While this gap between the DPP and PM probably does make it slightly harder for an otherwise good PM to improperly influence an otherwise good DPP, it doesn’t at all solve the problem that a bad PM would appoint a complicit AG, who would hire a complicit DPP.The AG can overrule and/or sideline the DPP at their own discretion, making the whole thing kinda pointless. (The AG would have to publish a memo of sorts explaining why, but that’s about it.)The obvious (if uncharitable) reading here is that Harper wasn’t all that interested in changing anything, so much as he was very interested in the optics of being seen to change things. That this chess move would also make it near impossible for his successor to reverse the changes without massive blowback (despite the changes being largely symbolic) was just an added bonus.[EDIT: For more on Harper’s presumed insincerity, see this combined with this. Also note that the current chair of SNC’s board was a key figure in Harper’s government for three years, as he’d been in several governments prior. He was expecting his calls to be answered, regardless of who was PM at the time. The idea of a Conservative government handling the SNC case any differently on a practical level seems like bad fiction to me, and I’m not sure on which grounds someone could even argue otherwise. I suspect Harper was unsurprised to hear that his trap had worked, if also a bit surprised to learn it was the SNC deferral that sprang it.]This all in mind, let’s get back to the current narrative.A Series of Unfortunate DecisionsBefore the new law even came into effect, the SNC began pushing hard for their chance at a deferral. But Canada’s current DPP, Kathleen Roussel, for reasons still unknown, decided that she was going to tell SNC “no deal”, leading her to issue an internal memo to this effect on September 4th of last year. The contents of said memo aren’t public at this point, having been private to the AG. But the fact of the decision was communicated to the PM’s office, which Trudeau and co. were, well, not entirely satisfied with. This set off a chain of contacts over the next three months or so, which broadly consisted of the PM’s office looking for the AG to either reconsider or allow an outside legal opinion.As to Trudeau’s motivations here, we can make some reasonable guesses:SNC is headquartered in Trudeau’s home city.Trudeau has an election coming up, and Quebec often plays the deciding role. SNC is a big deal to Quebec, and a conviction could cause deep harm to SNC. (At the least, the conviction would play poorly, whatever the ultimate economic consequences.)The Liberal Party has always been corporate-friendly (in ways both good and bad). In particular, they’ve largely been against value-destruction as a general principle. And Trudeau almost certainly sees refusing SNC’s request to be textbook value-destruction.Anyway, whatever his motivations or their relative weighting, Trudeau’s entreaties to Wilson-Raybould to intervene didn’t change her mind. But they did cause significant friction between them, which Trudeau was unable to keep entirely private. This was a gift that Andrew Scheer feasted upon. (Scheer is the current leader of the Conservatives, and Trudeau’s chief competition in this year’s election.)As a further gift to Scheer, Trudeau initiated a cabinet shuffle in January, which included the reassignment of Wilson-Raybould to Veterans Affairs (with additional responsibilities in National Defence). While this was met with some suspicion, the real drama began on February 7th when The Globe and Mail published a rundown of the spreading rumors. This in turn led to someone asking Trudeau a few days later if he and Wilson-Raybould were still on good terms. His answer was to the effect of “well, she’s still here working for me, which should be its own answer” — which, uh, backfired spectacularly in that she resigned a few hours later.[EDIT: The timing of an ethics probe may have also played into her timing. Also, it isn’t clear how aware Trudeau was of how Wilson-Raybould felt until the shuffle. She says it should have been obvious. He says it wasn’t. As of this time, no documentary evidence has come out proving either right or wrong.]Her resignation obviously raised even more questions, which ultimately led to her appearing before the House Justice Committee to address concerns over whether Trudeau and co. had crossed any legal lines in their lobbying.The FalloutYou can read Wilson-Raybould’s opening remarks here.Being as objective as able, I’d summarize them as follows:Trudeau and team lobbied aggressively on SNC’s behalf, and he made it clear that he was displeased with her and Roussel for being inflexible.While she didn’t accuse them of breaking any hard laws, she feels they did cross well into “inappropriate” territory, both in tone and frequency of approach, including after she’d basically said “no means no”.She feels that she was “demoted” because of her stand.All said, she seemed entirely credible. Her notes were thorough and it’s hard to imagine her having lied on any point. Even so, there’s the open question of interpretation, especially as it concerns that last bullet point.While there’s much that’s still unclear, we do know that, as far as immediate causes go, she was reassigned as part of a larger cabinet shuffle triggered by someone else’s resignation. This in mind, Trudeau’s official position has been “had Scott Brison not stepped down, Jody Wilson-Raybould would still be minister of justice and attorney general of Canada.”Now, Scheer is obviously hoping for a smoking gun that will prove this statement false. But we don’t have one yet, and it doesn’t seem all that likely to me that one exists. Could Trudeau have opportunistically used the shuffle to punish her? It’s possible. But it’s also possible that he was simply annoyed at how she handled the whole affair, leading him to decide that he’d prefer her elsewhere. I can see how either scenario could be viewed as objectionable, but I’m less sure that even the former rises to the level of criminal obstruction. Whatever Trudeau’s motivations in shuffling her, his replacement AG has so far left the existing DPP in place, which means nothing involving SNC-Lavalin has actually changed. I suppose you could still argue pettiness, but pettiness isn’t quite a crime.[EDIT: There’s also the possibility that there was no pettiness at all and that the shuffle was exactly and only for the reasons that Trudeau’s former top aide suggested. I found his testimony compelling in its own right, and I’m not sure how to adjudicate between the two accounts outside further evidence. My lean is that it feels a little unlikely for there not to have been some secondary motive, however small. But YMMV. I get into this a bit more in the edits at the end.]Anyway, all those arguments are meaningless to Scheer, mostly because it’s very convenient for him to not consider them.(Note: As I don’t want to give anyone cause to believe that I’m meaningfully biased, I’ll point out before continuing that one of the last two votes I cast was for a Conservative. And I’m fine with throwing more votes in their direction — just as soon as they stop nominating feckless lizardpeople like Andrew Scheer.)A Study in InsincerityWhile I wasn’t much of a Harper fan, my dislike for him was mostly benign. Had he won against Trudeau in 2015, I’d have made a vaguely disappointed clicking noise and then gone back to whatever I was doing. I ultimately voted against him because I was displeased with how he seemed to court the alt-right as it became clear he was going to lose — but his work as PM was largely … fine?Andrew Scheer, however, is a different category of conservative. Andrew Scheer, in a nutshell, is the kind of person you’d get if you isolated all the unhealthy impulses that Harper struggled with and then doused them with growth hormones (and then also stripped most of Harper’s policy/strategy IQ).He made a speech yesterday in response to Wilson-Raybould’s testimony, of which I’ll share just one excerpt:The testimony Canadians have just heard from the former Attorney General Jody Wilson-Raybould tells the story of a Prime Minister who has lost the moral authority to govern. A Prime Minister who allows his partisan political motivations to overrule his duty to uphold the rule of law. A Prime Minister who doesn’t know where the Liberal Party ends and where the Government of Canada begins. And a Prime Minister who has allowed a systemic culture of corruption to take root in his office and those of his most senior cabinet and public service colleagues.Now, much of this is just your run-of-the-mill disingenuous nonsense. But even in that ignoble context, I still find this one line incredible:a Prime Minister who doesn’t know where the Liberal Party ends and where the Government of Canada begins.Scheer seems to be making one of two absurd arguments here: (1) that “the government” is somehow separable from the declared values/views/proposals of those specifically elected to form said government; (2) that a majority government should set the values/views/proposals they were elected on aside so as to privilege the values/views/proposals which voters judged less attractive.I honestly don’t know which one of those ideas is less preposterous, but that Scheer would employ that kind of cutesy phrase despite it not actually meaning anything is one of many reasons I hope he’s never allowed to run anything more important than a blender. (Like, is it better if he just didn’t think through how dumb the sentence was, or if he did and said it anyway? And in the case of the latter, what does it say about the state of things if he judged this a viable tactic?)[EDIT: I don’t know how I forgot this, but by far the funniest thing here is that Harper had literally renamed the Government of Canada as “The Harper Government” in 2011! And guess who worked for Harper at the time? I wonder if he was this passionate about the distinction then?]Anyway, let’s shift from the statement to the motivation behind the statement. Who benefits from the SNC being prosecuted? Scheer! Who benefits from Trudeau being perceived as trying to interfere with this prosecution? Scheer! He wins either way. The only way he doesn’t win is if he’s forced to actually give his own opinion on why SNC does or doesn’t deserve a deferral.Lucky for him, no one is going to force him to do that — which I’d argue is symptomatic of the defining problem in this whole mess.About That DeferralBefore we move on to why Scheer is able to get away with all this, a few final words about SNC.What Scheer would prefer lost in the hubbub is that a deferral is not an exoneration. It’s a decision to choose a scalpel over a sledgehammer. While there may be times when the latter is the more appropriate tool, Scheer hasn’t really made any argument to that end (nor will he on this side of the election). His argument is simply “the DPP said sledgehammer, so Trudeau has to use the sledgehammer, which I don’t otherwise have an opinion on, but hey it does win me a lot of votes! — and PS, if Trudeau argues against the sledgehammer, it’s because he’s a coward/crook who hates Canada!”Trudeau’s response hasn’t been “c’mon, let’s let them off lightly because they’re my yacht buddies and besides I need their votes!” but rather “let’s pursue a form of justice which doesn’t introduce unnecessary collateral damage — and PS, I’d rather keep those votes thank you very much”.All else being even, it isn’t unreasonable for politicians to lean towards decisions that benefit them in the polls. But all else isn’t even here. One of these decisions is better for Canada, and one is worse.Now, is it possible that the DPP and AG actually had good reasons to stick it to SNC? Could be. The problem is that neither explained their reasons, both citing that it would be inappropriate to comment while SNC is pursuing their appeal of said decision in the courts. (This is probably a reasonable position now, though I’m less sure it’s a good reason for not having explained the decision at the time.)[EDIT: Just to be super clear on this point: the DPP’s Section 13 memo was read by the AG and no one else that we know of. The AG’s deputy didn’t read it. No one in the PM’s office read it (though Wilson-Raybould says a copy was forwarded, and then re-sent again after her conversion with Wernick on Dec 18th). SNC was never told why they were rejected. It’s all a mystery. And despite this being the first decision on this law, the AG refused to solicit an external opinion from a former Chief Justice of Canada. The AG had the right to make this set of decisions, but it’s hard to understand the logic here.]While their silence keeps us from perfect clarity, there are two dominant theories for why the DPP and AG decided against SNC’s plea:They agreed on the technical argument that SNC is legally ineligible for deferral on the explicit grounds of national economic interests (which is to say that Canada can’t use that as a factor in its judgment).They found compelling evidence that SNC hasn’t reformed and/or wouldn’t offer good faith cooperation in context of a plea.In the first case, it’s hard to argue against this reading of the law being facially correct. Even so, there are non-trivial counter-arguments: (i) Governments make this kind of self-benefiting decision all the time. The game is that you simply list reasons other than “national economic interest” when filling out the paperwork, regardless of how important said interests actually were. Now, maybe countries shouldn’t do this. But they do. And while I personally see the appeal of “let’s go by the book, even if mostly alone, even if that’s a net tax”, I don’t think people who take the other side are necessarily bad. (ii) If you have enough other reasons outside the national economic interest, it isn’t clear to me that it being helpful to the national economic interest is bad. (iii) Pragmatic flexibility is half the Liberal Party’s platform. Most voters who had an informed opinion here would have expected Trudeau to take precisely the stance he has. As such, you could argue that this is a form of mandate that he’s actually responsible to uphold.In the second case, the counter-argument is simply that over-ruling the PM’s judgment shouldn’t actually be their call, regardless of what the current letter of the law may say. Remember that this whole new structure was a Harper innovation, and arguably just a symbolic one. And while the AG has always theoretically been independent from the PM’s office, there are realist arguments for why this distinction has always been something of a legal fiction.(To be clear, we don’t know which — if either — of these arguments Trudeau is leaning on, largely because we don’t know what Roussel’s thinking was.)Formalist v. Realist(As preface for what follows, I’m not trying to convince anyone that one particular school of thought here is better or more right than the other. My point is that both are valid, in as much as they’re both logically robust frameworks that you could believe and defend without being inherently bad or crazy.)As to what I mean by the realist school, there’s a significant parallel here (in a narrow way) to the current situation in the US with Trump. When his new AG (who at the time wasn’t yet his AG, but who had been an AG before under Bush Sr.) issued an unsolicited memo outlining an argument that Trump was constitutionally OK to reassign and fire his way to outcomes he wanted without this implying actual obstruction of justice, this was widely met with cries of “treason!” — despite (a) that not being what treason means, and (b) it being a coherent and well-argued theory of law.Now, sure, you or I may disagree with this philosophically. And we may be right! But the idea that the chief executive has extraordinary and unilateral powers over nearly all executive affairs isn’t necessarily as dangerous as it may seem at first blush.Over-simplifying this a bit, imagine two competing scenarios:You restrain the executive’s powers with a complicated set of rules which are really hard to enforce with any consistency and which involve endless subjective judgment calls and which can easily be misused by a belligerent opposition to thwart the executive doing its job and which really don’t offer much effective restraint anyway.You restrain the executive’s powers with two simple levers: (a) in the case of gross judgment, you vote the executive out next election; (b) in the case of gross judgment that will cause more harm if left that long, you vote them out by pressuring your reps for impeachment / a no-confidence vote.Now, yes, there are real objections to this second system, which we’ll get to in a second. But just keep this idea of two approaches in mind as we consider a statement from Wilson-Raybould’s remarks yesterday:We either have a system that is based on the rule of law, the independence of prosecutorial functions and respect for those charged to use their discretion and powers in a particular way, or we do not.This is basically an argument for that first system — which, on its face, seems entirely reasonable. The rule of law is obviously good! And independence sure sounds like something we’d want! But step back for a minute. Let’s recall that the AG is a member of the executive (not the judiciary). They serve at the pleasure of the PM. And while we’d want the AG to have their own personal philosophy and set of legal interpretations, they weren’t elected to enact those. (In the US, the AG isn’t even an elected official at all!) And if the PM/President happens to have a different philosophy and set of legal interpretations (which they, in effect, were elected to enact), it isn’t at all obvious why they must lose in the event of a tie.By way of analogy, imagine that I’m a hiring manager working directly under the CEO in a public company. She’s the one hired by the shareholders (via the board), not me. She’s hired me to be an extension of her vision. To the degree that I do this well, all is well. But if she and I disagree on whether to hire a given candidate, my objection of “well, our bylaws say this is my call” isn’t itself all that compelling. I can go to the board and say “hey, she overruled me and our bylaws say she can’t do that”, but for the board to be fully interested they’d want me to also add “and her judgment was dangerously flawed for x reason”, else the board would just say “yes, well, we hired her and not you, so if she thinks your judgment is wrong, you’re not really fulfilling the function you were hired to fulfill, so I’m not sure why you’re still here”. What they care about most is whether the CEO displayed poor judgment relative to the standard they were hired to uphold. Whether or not bylaws were broken along the way is somewhat incidental. (While some board members care a lot about bylaws, that concern is more often about organizational dynamics than any higher theory of justice. If they like the CEO’s vision and you were hired to execute that vision and the CEO no longer feels like you are executing that vision, arguing “but the bylaws!” is probably not going to save your job.)To be ultra clear, this isn’t to say that a CEO or PM or President should be allowed to “get away” with whatever they want. There are many occasions where a PM will want something that’s not actually consistent with the platform on which they were elected (or that’s just generally bad in some moral or ethical sense) — in which case we would want the AG to object, and object strongly. But in cases where the AG loses this argument, we’d also expect them to no longer be AG thereafter, which is itself fully consistent with a healthy system provided that their exit triggers a thorough review.To use an extreme example, imagine that Trudeau tells his AG to tell her DPP to bring a Biblical flood of lawsuits against his next-door neighbor because they objected to his backyard Nickelback concert. In the realist view, it doesn’t necessarily matter if the AG says yes or no, or even whether they have the latitude to make that decision. What matters is that the public is informed so that we can all decide whether this is something we object to or not. And if we do, enough of us will call up our local MP and say “if you don’t stop this guy immediately, we’ll vote in your opponent next year”, which they’ll take as impetus to go vote the PM out. It’s no less an effective check against the PM’s abuse of power than the AG having theoretical independence. It’s just a different mechanism. Sure, there are plenty of people who prefer one mechanism over the other (which is the kind of viewpoint diversity that’s good and healthy!), but it’s hard to argue that this realist view is essentially wrong. Having an independent AG is not as structurally important as having an independent judiciary — provided that the actions of the executive are regularly and efficiently reported back to a voting public interested in holding leaders to account.And it’s exactly with this last bit that things get thorny.A RequiemThe phrase “constitutional crisis” has been bandied about a lot in both Canada and the US in recent days. Yet few of the underlying situations really seem to fit the bill, at least relative to a much deeper constitutional crisis that’s been growing unchecked for decades now.The constitution (both in the US and Canada, and in nearly every developed democracy) depends upon an engaged citizenry willing to demand and do, and a press capable of giving said citizenry the data on which they can fairly decide.The problem is that none of this works if we don’t share common ideas of which bad things are especially bad, or if we don’t all trust that at least a few impartial and talented journalists will always ask the right questions to empower useful explanations of what the executive did so that we can vote and/or pressure our reps accordingly.The challenge as I see it is that Canadians have long been remarkably ill-served by their press.Huge chunks of the country only have a Postmedia outlet as their local paper. (Postmedia being the antithesis of unbiased.)There are less sensational papers, including some large national ones. But none are especially good at what we need them to be good at. (If you can find me an explainer from a major outlet that’s anywhere near as thorough or clear as this one, I’ll send you $25. And I really don’t mean this to my own aggrandizement. I just don’t think one exists.) [EDIT: I’m happy to 10x this offer to $250 CAD just in case $25 isn’t enough of an incentive.]Most papers have dedicated an increasing amount of space to opinion pieces, which are quite good for clicks and quite bad for reader education. (They’re too short, too slanted, and they mostly use the little space they have to tell rather than show.)Most opinion columnists sell a partisan spin, which only serves to divide people into camps that inevitably grow further and further apart, thus making voter coordination across party lines difficult to impossible, all while also reducing common ideas of which things are worth coordinating on.No outlets have shown a willingness or ability to force Trudeau or Scheer to answer hard questions. (Hard to say if they’re too worried about losing access, too self-focused to coordinate, or too distracted to see the civic necessity of getting those answers.)I’ve polled a bunch of my Canadian friends — all bright young people who regularly vote and try to do so intelligently. None could really explain what happened with Trudeau and SNC, nor were they sure where to turn to rectify the gap. Plenty of stories were a search away. But which would give my friends the context required to understand Trudeau’s decisions for what they were?(That’s a non-rhetorical question, by the way. My wallet is open if I’ve misspoken here.)Why This MattersConsider this current case. It’s fine for someone to say “hey, I believe in closely following the letter of the law in all cases” and for them to thus side against Trudeau here (assuming he did in fact inappropriately pressure Wilson-Raybould). Like, I may not personally find this to be the world’s most sophisticated ideology, but the point is that we all tend to think our own judgment best, which is exactly why we do things like vote in free multi-party elections. Lots of people probably believe that an AG should be 100% independent in every way, and these voters should be welcome to form a party around that belief! But that party wouldn’t quite be today’s Liberal Party (or the Conservatives), which is something that’s been poorly explained to voters.The resulting issue is that I don’t think most people will go to the polls this October with optimal clarity in mind. While some are only ever going to vote against the candidate they hate more, many with less tribal feelings are going to be swayed by a faulty assumption that Trudeau committed some especially heinous crime here, where the reality (to the current public evidence) is more that, at worst, he and his AG were approaching from two different angles, with one of them ultimately having the trump-card of being the elected PM.There are all sorts of valid reasons why someone might vote for or against Trudeau. But I think it’s important that those votes are cast in light of what actually happened here and what it actually implies — which Scheer is actively trying to muddy and misrepresent, which the media is largely unwilling to combat, which I think is probably a bad thing?EDITS: ROUND #1Original answer written March 1st. Coming back on the 13th to get around to some needful updates/corrections/addendums.Some I’ve made above; others I’ll list here in no particular order:Trudeau and Butts have been arguing that they do believe in full prosecutorial independence. But I almost wish they wouldn’t. It’s hard to believe that they (or any PM team) totally believes it to be the best possible mechanism. I get that it’s scary to say anything else (imagine the headlines!), but this feels like a good opportunity to maybe start talking about all the stuff I’ve outlined here. (I suppose there’s a world in which they could totally believe in the idea. Butts certainly sounded sincere when he talked about it. But I just can’t get there as it concerns a PM. The sorts of people who win national elections aren’t generally the sorts to take being overruled by an underling all that well. I only leave this door open because of how authentic Butts seemed on the point.)My original piece included this note: “In the interests of precision, the most recent source I could find said that SNC has 3,400 employees in Quebec (vs. just Montreal). But as their headquarters are in Montreal, I’m assuming the bulk are there. I could be wrong.” As an update, this authoritative-seeming Globe & Mail article puts the number in Quebec at 2,500 and the number in Montreal at 700.Lots of commentary out there about just how at risk those 9,000 Canadian jobs were (and about how a federal debarment wouldn’t necessarily influence bidding on provincial/municipal projects). Though I’ve made a few edits to account for these arguments, they all seem peripheral to me for two reasons: (i) if SNC was found guilty, this would almost certainly impact their employee retention and bidding prospects in a general downstream way (we have evidence of recent press impacting them already); (ii) while most displaced workers would find new jobs, there’s no obvious replacement within Canada for SNC in terms of EPC firms. (For more, I thought this take was balanced and thorough — though I did find the final four sentences wildly upsetting.)I found this personal testimony from a current SNC exec (who lives and works in Saskatchewan) worth reading. His main point is that those trying to politicize this as some Liberal gambit to exclusively favor Quebec are overlooking that some 2/3rds of SNC’s Canadian workforce don’t live or work in Quebec. (On a political level, an SNC conviction would definitely hurt Liberals more. But his point that Trudeau is fighting for jobs that are mostly not in Quebec is certainly valid.)It still isn’t clear to me which laws/precedents are shielding the DPP’s SNC-related memo(s) from public review. The court has since ruled against SNC’s appeal, and it really feels like this whole debate would be much simpler if we all knew exactly why Roussel and Wilson-Raybould felt so strongly against SNC being eligible for a deferral. (FWIW, you can read the full text of the deferral-related legislation here.)For those asking, I’m 100% behind a thorough investigation. Let’s get lots of uninhibited testimony, and let’s subpoena relevant emails/texts, etc. Sunlight is indeed the best disinfectant. But let’s also report those findings in a cautious and contextual and non-sensational way.There’s a representative thread here on why the AG's independence is somehow sacrosanct. All such threads/articles I've come across share the same hallmarks: (i) they ignore/discount that individuals serving at the pleasure of the PM can never be truly independent; (ii) they make a weird assumption that independence is a required pre-condition to keeping a PM from interfering with prosecutions in a gross way (when a non-independent prosecutor could just as easily report the PM for gross judgment). I don’t know who is debating that bad judgment is anything other than bad, or who is suggesting we shouldn’t deter/punish it. The question is whether a certain old and imported legal doctrine is the most effective mechanism to ensure an end we all agree is important.The way most journalists use the word “political” in the phrase “political interference” is also weird! Whether interference is “political” has nothing to do with whether it’s good or bad. Take the case of weed-related prosecutions. Most would say that more political interference would have been good (in terms of directing prosecutors to not prosecute any more pot cases while new legislation was framed). The fact that Trudeau and Wilson-Raybould didn’t interfere with those prosecutions is a widespread criticism. That resources were spent fining and jailing people for trivial amounts of pot possession after Trudeau’s majority election on a “legalize pot” platform is, to many, a symptom of a flawed system, not proof of some sacred pillar doing its job. The public will was pretty clear about this (and had been for some time). Following the existing law to the letter led to an outcome that most consider bad. Governments semi-regularly opt to not prosecute existing laws for various political reasons (most of them positive/healthy). This isn’t necessarily a meaningful threat to the rule of law. (This is doubly true when it’s a new law with no prosecutorial precedent either way.)On a related note, I think “rule of law” is one of those phrases where everyone has some idea of what they mean by it, but where few actually have a robust conception that could survive a hard cross-examination. By any conception, sure, inconsistent and partial rulings are generally to be avoided. But the idea that prosecutorial independence (which just shifts who gets to make the decisions which some will find inconsistent and/or partial) has some necessary role in supporting the rule of law is curious. Taking up the example of pot again, look how many Americans were angry at Jeff Sessions for increasing the number of pot prosecutions given that this was contrary to Trump’s platform. Sessions’ decision caused all sorts of confusion/chaos, and it clearly favored one demographic of voters at the expense of another. No one is questioning that he was legally allowed to make said decision (the federal law was clear, and the restrictions placed upon on it by the previous AG were non-binding), but I’m not sure how any could argue that his decision reflects the rule of law working in a positive way either.A few good write-ups about the Shawcross doctrine (the basis in Canadian legal theory for prosecutorial independence) here, here and here. The last link (about the UK implementation) is interesting in that it describes a “Shawcross letter” (i.e., a letter sent by the AG to their ministerial colleagues to solicit their input). This was, in effect, what Wilson-Raybould didn’t do. She made up her mind independently, then dismissed those who expressed contrary views. Had she gathered up all available views to inform her decision (rather than decide based exclusively on her and her DPP’s legal judgment), one imagines that others would have found less cause to ensure their views were being heard/considered. (She had the legal right to make up her mind independently, but that hardly means she was right in doing so.)I’ll have to write a separate answer about Butts’ testimony at some point, but the big things for me were: (i) it gave a pretty good explanation for why Wilson-Raybould was shuffled; (ii) Butts made an extraordinary and easily-falsified set of claims about her interactions with him about the SNC file, which, if true, are enormously problematic for her case.The other thing that came out of Butts’ testimony was that Trudeau really dropped the ball in offering Wilson-Raybould (who is Indigenous) the Indigenous Services portfolio as opposed to the Crown-Indigenous Relations portfolio, with that difference being, roughly, the difference between being in charge of restructuring relations (the latter) and being in charge of administering welfare (the former). She was never going to accept the former, and he and Butts have no real excuse for not foreseeing this.The real crisis here (if we must use the term crisis) seems to be that Trudeau was so out of tune with two of his ministers, and that he was unable to keep them in the fold after they raised their objections (even if the way they raised them was less than ideal).On the subject of raising objections, Scott Welch wrote a great companion answer here about the various opportunities (if not obligations) that Wilson-Raybould had to report any inappropriate or unethical behavior on the part of Trudeau or his staff. She, to our current knowledge, made use of none of them. Prior to her being shuffled, her only vocalized discontent came in the form of telling some people (paraphrasing) “stop lobbying already!”, which is not at all the same thing as raising a formal concern/complaint about misconduct. (And again we have Butts’ testimony that his sole one-on-one conversation with her about SNC came at the end of a friendly two-hour dinner which she initiated, with no other text or email ever being sent to him on the subject. Per his testimony, she was the one who brought it up then, and her after-dinner text said nothing further about it. He claims she never laid out her concerns until during one of their “four or five” “long” and “personal” conversations they had after she was transferred, and that he was dumbfounded when she mentioned her suspicion on why she had been moved.)While she isn’t a disinterested party (and while I think some of her criticisms are overly strong), I thought Sheila Copps’ interview with CBC contained some solid points, namely: (i) that saying “I’ve made up my mind, now go away” is not exactly consistent with the role of an AG/MOJ when your colleagues don’t feel heard, (ii) that it’s kinda weird to quit a cabinet without also quitting caucus, (iii) that the number of meetings which Wilson-Raybould took on this file was not especially high, (iv) that the original decision to not prosecute was split. (As an aside, I really dislike that interviewer. He’s part of the problem.)EDITS: ROUND 2So, the Conservatives have started a “Let Her Speak” campaign pushing for a second (at least) round of testimony from Wilson-Raybould. While I’m broadly supportive of this, it’s worth noting that Trudeau has a rational case for saying “no” that isn’t solely rooted in being afraid of some harmful truth being revealed. From his perspective, Wilson-Raybould is on a mission to take him down. Let’s assume that every word she said in the first testimony was true. There was nothing particularly damning in it from a legal context, but it played very poorly for Trudeau all the same. He’s now facing a decision between: (i) allowing her to speak again and the same thing happening again and it hurting him more in the polls, (ii) not allowing her to speak again, which will hurt him in the polls now, but which may also cap the damage. Were I his advisor, I’d push him toward letting her speak and then prepping a killer set of rebuttals (assuming the facts are indeed on his side). More downside, but more upside. That said, it would seem that he’s being pushed in a different direction. While I’m not so much a fan of this, I wouldn’t infer from it (as many are) that it’s necessarily a sign of guilt.Ok, a bit more about Scheer. I keep getting comments wondering why I dislike him so much. First, I should point out that most Conservatives never cared much for him either until he was their sole hope against Trudeau. He didn’t crack 20% in opinion polls until after Kevin O’Leary dropped out (and that’s among Conservative Party members). He won the nomination on the 13th ballot, having trailed on all 12 ballots prior. The fact that he won (which was contested due to significant inconsistencies) was largely viewed as a frantic rejection of Maxime Bernier (who had led on all of the first 12 ballots), not an endorsement of Scheer. From his first speech post-nomination, he’s relied on red meat, generalities, and strawmen. That’s not what we need from the Leader of the Opposition. We need someone able to carefully and accurately deconstruct the PM’s decisions, showing voters a detailed vision of some plausibly better way. Jack Layton was that guy. Harper, in his own way, was that guy. Scheer is not that guy. He once wrote an op-ed in support of Brexit — aka, arguably the most ill-conceived set of decisions in modern political history. (He was still in support of it as recently as this past November, which is just wild given how things have unfolded. It’s one thing to be theoretically in favor of sovereignty. It’s quite another to be in favor of a nation jumping off a cliff to get it.) Scheer’s campaign chair, Hamish Marshall, is also a former director of Rebel Media, which is morally inexcusable in itself. (Marshall says he had nothing to do with editorial there and that he eventually did resign. But this rundown of all the content they published before he quit is deeply disturbing — not to mention that we have no reason to believe that he quit for moral reasons.)I’d been waiting for a transcript of Nathalie Drouin’s (Wilson-Raybould’s former deputy) testimony before the House Justice Committee (which it seems doesn’t exist anywhere, maybe due to some rule). But snippets were reported by the CBC. A few interesting takeaways: (i) JWR had given Drouin an instruction not to talk about the SNC case by September 17th, which was super early into the process; (ii) JWR forbade Drouin from answering a question from the PCO (civil service) about the potential impact of SNC failing to get a deferral; (iii) Drouin was never told what the evidence against SNC’s case was. This is all very odd.An important open question: where did the original leak to the Globe & Mail come from? And why was a leak made to the press before internal remedies were exhausted (or even attempted)?An interesting tidbit from Butts’ testimony that I missed the first time around: Wilson-Raybould was the second minister that Trudeau attempted to move from a dream portfolio to Indigenous Services using the same logic. The difference is that the first person said yes. (Incidentally, this was the other minister that later quit in solidarity with Wilson-Raybould.) While it was still an unwise move, this does lend credence to the argument that Trudeau was doing it to signal continued support for his Indigenous reconciliation efforts, which is to say it could have been a bad decision made with positive intent. (Logically, though, this same move having been made before might have been a really convenient cover. I don’t know how we’d know for sure unless we’re just taking someone’s word.)Echoing what Copps said about the insubstantial meeting count, Butts estimated the number of meetings on the TMX pipeline deal as being around 100. By comparison, the SNC file ended up drawing 10 meetings and 20 contacts over a little more than three months. While the context is a bit different with an AG vs. other ministerial roles, this number still doesn’t seem outlying.Lots of people have been mentioning how prosecutorial independence is “constitutionally guaranteed” (or other wording to that effect). This is true in a sense, but that sense is mostly misleading. Canada is unlike the US in that we have no single document called The Constitution. What we have is a messy patchwork of acts, statutes, orders, and precedents. Prosecutorial independence is a “constitutional convention”, meaning that it’s an unwritten rule with no binding power over Parliament. In the absence of contrary legislation, conventions are the best practices which all are generally expected to follow. But not only have conventions been broken from time to time with little consequence, the House is free to pass new laws to make written what is unwritten, and the courts (explicitly) have no power to overrule. So if Trudeau were to decide tomorrow “hey, let’s do away with this thing” and if enough MPs were to say “yeah, let’s do that”, then the bill would be passed. (There are more steps, but the gist is that there’s no way to stop a majority-supported bill without sparking an actual constitutional crisis likely to resolve in the House’s favour.) While I can’t imagine that any PM would try this in the current climate, there’s nothing actually legally stopping them.

I gave this talk earlier this week at Hadoop World(http://www.hadoopworld.com/session/linkedin-products-you-may-like/), a conference that is evangelizing Hadoop by way of highlighting how people across the industry are solving big business challenges by leveraging Hadoop. I am posting here the slides with an approximate transcript of my talk.Ever since I studied Machine Learning and Data Mining at Stanford 3 years ago., I have been enamored by the idea that it is now possible to write programs that can sift through TBS of data to recommend useful things.So here I am with my colleague Adil Aijaz, for a talk on some of the lessons we learnt and challenges we faced in building large-scale recommender systemAt LinkedIn we believe in building platform, not verticals. Our talk is divided into 2 parts. In the first part of this talk, I will talk about our motivation for building the recommendation platform, followed by a discussion on how we do recommendations. No analytics platform is complete without Hadoop. So, in he next part of our talk, Adil will talk about leveraging Hadoop for scaling our products.‘Think Platform, Leverage Hadoop’ is our core message.Throughout our talk, we will provide examples that highlight how these two ideas have helped us ‘scale innovation’.With north of 135 million members, we’re making great strides toward our mission of connecting the world’s professionals to make them more productive and successful. For us this not only means helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.With terabytes of data flowing through our systems, generated from member’s profile, their connections and their activity on LinkedIn, we have amassed rich and structured data of one of the most influential, affluent and highly-educated audience on the web.This huge semi-structured data is getting updated in real-time and growing at a tremendous pace, we are all very excited about the data opportunity at LinkedIn.For an average user, there is so much data, there is no way users can leverage all the data on their own.We need to put the right information, to the right user at the right time.With such rich data of members, jobs, groups, news, companies, schools, discussions and events. We do all kinds of recommendations in a relevant and engaging way.We have products like‘Job Recommendation’: here using profile data we suggest top top jobs that our member might be interested in. The idea is to let our users be aware of the possibilities out there for them.‘Talent Match’: When recruiters post jobs, we in real-time suggest top candidates for the job.‘News Recommendation’: Using articles shared per industry, we suggest top news that our users need to keep them updated with the latest happenings.‘Companies You May Want to Follow’: Using a combination of content matching and collaborative filtering, we recommend companies a user might be interested in keeping up-to date with.‘PYMK’: based on common attributes like connections, schools, companies and some activity based features, we suggest people that you may know outside of LinkedIn and may want to connect with at LinkedIn.‘Similar Profiles’: Finally our latest offering which we released a few months ago for recruiters, Similar Profiles. Given a candidate profile, we suggest top Similar candidates for hiring based on overall background, experience and skills.We have recommendation solutions for everyone, for individuals, recruiters and advertisersIn our view, recommendations are ubiquitous and they permeate the whole site.Before we discuss our motivation for building a recommendation platform or a discussion on how we do recommendation or how we leverage Hadoop. Let’s first answer a basic question: Are recommendations really important?To put things in perspective, 50% of total job applications and job views by members are a direct result of recommendations. Interestingly, in the past year and half it has risen from 6% to 50%.This kind of contribution is observed across all our recommendation products and is growing by the day.Let us start with an example of the kind of data we have.For a member, we have positions, education, Summary, Specialty, Experience and skills of the user from the profile itself. Then, from member’s activity we have data about members connections, the groups that the member has joined, the company that the member follows amongst others.Before we can start leveraging data for recommendation, we first need to clean and canonicalize it.Lets take an example for matching member to jobs,In order to accurately match members to jobs, we need to understand that all the ways of listing these titles refer to the same entity ‘Software Engineer’‘Technical Yahoo’ is a Software Engineer at Yahoo‘Member Technical Staff’ is a Software Engineer at Oracle‘Software Development Engineer’ is a Software Engineer at Microsoft‘SDE’ is a Software Engineer at AmazonSolving this problem is itself a research topic broadly referred to as ‘Entity Resolution’.As another example, How many variations do you think, we have for the company name ‘IBM’?When I joined LinkedIn, I was surprised to find that we had close to 8000+ user entered variations of the same company IBM.We apply machine learnt classifiers for the entity resolution using a host of features for company standardization.In summary, data canonicalization is the key to accurate matching and is one of the most challenging aspects of our recommendation platform.Now, we will discuss our motivation behind building a common platform by way of 3 key example trade-offs that we’ve encountered. In LinkedIn ecosystem, one trade-off that we encountered is that of real-time Vs time independent recommendations.Lets look at ‘News Recommendation’, which finds relevant news for our users. Relevant news today might be old news tomorrow. Hence, News recommendation has to have a strong real-time component.On the other hand, we have another product called ‘Similar Profiles’. The motivation here is that if a hiring manager, already know the kind of person that he wants to hire. He could be like person in his team already or like one of his connections on LinkedIn, then using that as the source profile, we suggest top Similar Profiles for hiring. Since, ‘people don’t reinvent themselves everyday’. ‘People similar today are most likely similar today’. So, we can potentially do this computation completely offline with a more sophisticated model.These are the 2 extreme cases in terms of ‘freshness’: Most examples fall into an intermediate category. For E.g. new jobs gets posted by the hour and they expire when they get filled up. All jobs posted today don’t expire the same day. Hence, we cache job recommendation for members for some time as it is OK to not recommend the absolute latest jobs instantly to all members.In solving the completely real-time Vs completely offline problem, we could have gone down the route of creating separate solutions optimized for the use-case. In the short run, that would have been a quicker solution.But we went down the platform route because we realized that we would churn out more and more such verticals as LinkedIn grows. Now, as a result of which we have the same code that computes recommendations online as well as offline. Moreover, in the production system caching and an expiry policy allows us to keep recommendations fresh irrespective of how we compute the recommendations. Now, as a result for newer verticals. We can easily get ‘freshness’ of recommendations irrespective of whether we compute recommendations online or offline.Another interesting trade-off, is choosing between content analysis and collaborative filtering.Historically speaking: ‘job posting has been a post-and-prey’ model. Where job posters post a job and pray that hopefully someone will apply for the job. But we at LinkedIn, believe in ‘post-and-produce’ so we go ahead and produce matches to the job poster in real-time right after the job gets posted. When someone posts a job, the job poster naturally expects the candidates to have a strong match between job and the profile. Hence, this type of recommendation is heavy on content analysis.On the other hand, we have a product called ‘Viewers of the profile also viewed ..’. When a member views more than 1 profile within a single session. We record it as a co-view. Then on aggregating these co-views for every member. We get the data for all profiles that get co-viewed, when someone visits any given profile. This is a classical collaborative filtering based recommendation much like Amaozon’s ‘people who viewed this item also viewed’Most other recommendations are hybrid. For e.g. for ‘SimilarJobs’ jobs that have high content overlap with each other are similar. Interestingly, jobs that get applied to or viewed by the same members are also Similar. So, Similar Jobs is a nice mix of content and collaborative filtering.Again, Because of a platform approach, we can re-use the content matching and the collaborative filtering components to come up with newer verticals without re-inventing the wheel.Finally, the last key trade-off is of precision vs recall.On our homepage, we suggest jobs that are good fit for our users with the motivation for them to be more aware of the possibilities out there. In some sense, we are pushing recommendations to you, as opposed to you actively looking for them.Even if a single job recommendation looks bad to the user either because of a lower seniority of the job or because the recommendation is for the company that the user is not fond of, our users might feel less than pleased.Here, getting the absolute best 3 jobs even at the cost of aggressively filtering out a lot of jobs is acceptable.On the other hand, We have another recommendation product called ‘Similar Profiles’ for hiring managers who are actively looking for candidates. Here, if one finds a candidate we suggest other candidates like the original one in terms of overall experience, specialty, education background and a host of other features.Since, the hiring manager is actively looking so in this case they are more open to getting a few bad ones as long as they get a lot of good ones too. So in essence, recall is more important here.Again, because of a platform approach and because we can re-use features, filters and code-base across verticals. So, tuning the knob of more precision vs more recall is mostly a matter of figuring out 1. how complicated the matching model should be and 2. how aggressively we want to apply filters for all recommendation verticals. Hence, our core message ‘Think Platform’. Now, we will discuss in some detail how our recommendations work.Lets see how we do recommendations by taking an example of ‘Similar Profiles’ that we just discussed. Given a member profile, the goal is to find other similar people for hiring. Lets try to find profiles similar to me.Here, we look at host of different features, such as1. User provided features like ‘Title, Specialty, Education, experience amongst others’2. Complex derived features like ‘Seniority’ and ‘Skills’, computed using machine learnt classifiers.3. Both these kinds of features help in precision, we also have features like ‘Related Titles’ and ‘Related Companies’ that help increase the recall.Intuitively, one might imagine that we use the following pair of features to compute Similar Profiles. In the next slide, we will discuss a more principled approach to figuring out pair of features to match against.Here in order to compute overall of similarity between me and Adil, we are first computing similarity between our specialties, our skills, our titles and other attribute.With this we get a ‘Similarity score vector’ for how Similar Adil is to me, Similarly we can get such a vector for other profiles.Now we somehow need to combine the similarity score in the vector to a single number s.t. profiles that have higher similarity score across more dimensions get ranked higher, as similar profiles for me.Moreover, the fact that our skills match might matter more for hiring than whether our education matches. Hence, there should be relative importance of one feature over the others.Once we get the topK recommendations, we also apply application specific filtering with the goal of leveraging domain knowledgeFor example, it could be the case that for a ‘Data Engineer role’ you as a hiring manager are looking for a candidate like one of your team member but who is local. Whereas, for all you know the ideal ‘Data Engineer most similar to the one you are looking for in terms of skills might be working somewhere in INDIA’ .To ensure our recommendation quality keeps improving as more and more people use our products, we use explicit and implicit user feedback combined with crowd-sourcing, to construct high quality training and test sets for constructing the ‘Importance weight vector’. Moreover, classifier with L1 regularization helps prune out the weakly correlated features. We use this for figuring out which features to match profiles against.We just discussed an example. However, the same concepts apply to all the recommendation verticals.And now the technologies that drives it all.The core our matching algorithm uses Lucene with our custom query implementation.We use Hadoop to scale our platform. It serves a variety of needs from computing Collaborative filtering features, building Lucene indices offline, doing quality analysis of recommendation and host of other exciting things that Adil will talk about in a bit.Lucene does not provide fast real-time indexing. To keep our indices up-to date, we use a real-time indexing library on top of Lucene called Zoie.We provide facets to our members for drilling down and exploring recommendation results. This is made possible by a Faceting Search library called Bobo.For storing features and for caching recommendation results, we use a key-value store Voldemort.For analyzing tracking and reporting data, we use a distributed messaging system called Kafka.Out of these Bobo, Zoie, Voldemort and Kafka are developed at LinkedIn and are open sourced. In fact, Kafka is an apache incubator project.Historically, we have used R for model training. We have recently started experimenting with Mahout for model training and are excited about it.All the above technologies, combined with great engineers powers LinkedIn’s Recommendation platform.Now Adil will talk about how we leverage Hadoop.In the second half of our talk we will present case studies on how Hadoop has helped us scale innovation for our Recommendation Platform. We will use the ‘Similar Profiles’ vertical which was discussed earlier as the example for each case study. As a quick reminder, similar profiles recommends profiles that are similar to the one a user is interested in. Some of its biggest customers are hiring managers and recruiters. For each of the case studies, we will lay out the solutions we tried before turning to Hadoop, analyze the pros and cons of the approaches before and after Hadoop, and finally derive some lessons that are applicable to folks working on large scale recommendation systems.When it comes to recommendations, relevance is the most important consideration. However, with over 120M members, and billions of recommendation computations, the latency of our recommendations becomes equally important. No mater how great our recommendations are, they wont be of utility to our members if we take too long to return recommendations. Among our many products, ‘Similar Profiles’ is a particularly challenging product to speed up. Our plain vanilla solution involved using a large number of features to mine the entire member index for the best matches. The latency of this solution was in the order of seconds. Clearly, no matter how relevant our recommendations were, with that type of latency, our members would not even wait for the results. So, something had to be done. We needed a solution that could pre-filter most of the irrelevant results while maintaining a high precision on the documents that survived the filter. One technique that meets these conditions is minhashing. On a very high level, minhashing involves running each document, in our case member profiles, through k hash functions to to construct a bit vector. One can play with ANDING/Oring of subsets of the bit vector, to get the right balance between recall and precision. As our second pass solution, we minhashed each document and stored the resulting bit array in the member index. At query time, we minhashed the query into a bit array, filtered out documents that did not have the exact same subsets of the bit array, and finally did advanced matching on documents that survived the filtration. This solution brought down the latency well below a second, however, minhashing did not give us the recall we had hoped for. This was a really disappointing result since we had spent significant engineering resources in productionalizing minhashing, yet it was all for nought.So, we went back to the drawing board and started thinking about how we can use Hadoop to solve this problem. The key breakthrough was when we realized that people do not reinvent themselves everyday. The folks I was similar to yesterday, are likely to be the same folks I am similar to today. This meant that we could serve ‘similar profiles’ recommendations from cache. When the cache would expire, we could compute fresh recommendations online and repopulate the cache. This meant that the user would almost always be served from cache. Great: but we still have to populate the cache somehow. This is where Hadoop comes into the picture. By opening an index shard per mapper, we can generate a portion of the recommendations in each mapper, and combine them into a final recommendation set in the reducers. With the distributed computation of Hadoop, we easily generate similar profiles for each member, and then copy over the results to online caches. So the three elements of:a) Offline batch computations on Hadoop copied to online caches.b) Aggressive online caching policyc) Online computation when the cache expiresHave scaled our similar profiles recommendations while maintaining a high precision of recommendations.So, the key takeaway from this case study of scaling is that if one is facing the problem of:a) High latency computationb )High qpsc) And not so stringent freshness requirementsThen one should leverage Hadoop and caching to scale the recommendation computation.With our scaling problems solved, we rolled out Similar Profiles to our members. The reception was amazing. However, we felt that we could do even better by going beyond content based features alone. One such feature that we wanted to experiment with was collaborative filtering. More specifically, if a member browses multiple member profiles within a single session, aka coviews, it is quite likely that those member profiles are very similar to each other. How we blend collaborative filtering with existing content based recommendations is the subject of our second case study: “blending multiple recommendation algorithms”.Our basic blending solution is this: While constructing the query for content based similar profiles, we fetch collaborative filtering recommendations and their scores, and attach them to the query. In the scoring of content based recommendations, we can use the collaborative filtering score as a boost. An alternative approach is a bag of models approach with content and collaborative filtering serving as two of the models in the bag.In either solution, we need a way to keep collaborative filtering results fresh. If two member profiles were coviewed yesterday, we should be able to use that knowledge today.We first sketched out a completely online solution. This online solution involved keeping track of the state of each member session, accumulating all the profile views within that session. At the end of each session, we updated the counts of the various coview pairs. As you can appreciate, such a stateful solution can get very complicated very quickly. As an example, we have to worry about machine failures, multi data center coordination just to name two challenges. In essence such a solution could introduce more problems than it solves. So, we scratched this solution even before implementing it.We thought more about this problem and realized two important aspects: 1) Coview counts can be updated in batch mode. 2) We can tolerate delay in updating our collaborative filtering results.These two properties, batch computation and tolerance for delay in impacting the online world, led us to leverage Hadoop to solve this problem. Our production servers can produce tracking events everytime a member profile is viewed. These tracking events are copied over to hdfs where everyday, we use these tracking events to batch compute a fresh set of collaborative filtering recommendations. These recommendations are then copied to online key value stores where we use the blending approaches outlined earlier to blend collaborative filtering and content based recommendations.Compared to the purely online solution, the Hadoop solution is simpler in complexity, less error prone, but introduces a lag between the time two profiles are coviewed and the time that coview has an impact on similar profiles. For us, this solution works great. The other great thing about this solution is that it can be easily extended to blend social or globally popular recommendations in addition to collaborative filtering.The lesson we derive from this case study is that by leveraging Hadoop, we were able to experiment with collaborative filtering in similar profiles without significant investment in an online system to keep collaborative filtering results fresh. Once our proof of concept was successful, we could always go back and see if reducing the lag between a profile coview and its impact on similar profile by building an online system would be useful. If it is, we could invest in a non-Hadoop system. However, by leveraging Hadoop, we were able to defer that decision till the point when we had data to backup our assumptions.A consistent feedback from hiring managers using Similar Profiles was that while the recommendations were highly relevant, often times the recommended members were not ready to take the next steps in their professional career. Such recommended members would thus respond negatively to a contact from the hiring manager, leading to a bad experience for the hiring manager. This feedback indicated a strong preference from our users that they would like a tradeoff between relevance of recommendations and the responsiveness of those recommendations. One can imagine a similar scenario playing out for a sales person looking for recommendations for potential clients.As our next case study, let’s take a look at how we approached solving this problem for our users. Let’s say we come up with an algorithm that assigns each LinkedIn member a ‘JobSeeker’ score which indicates how open is she to taking the next step in her career. As we said already, this feature would be very useful for ‘Similar Profiles’. However, the utility of this feature would be directly related to how many members have this score: aka coverage. The key challenge we faced was that since ‘Similar Profiles’ was already in production, we had to add this new feature while continuing to serve recommendations. We call this problem “grandfathering”.A naïve solution, could be to assign a ‘jobseeker’ score to a member next time she updates her profile. This approach will have minimal impact on the system serving traffic, however, we will not have all members tagged with such a score for a very long time, which impacts the utility of this feature for ‘Similar Profiles’.So, we scratch the naïve solution and look for a solution that will batch update all members with this score in all data centers while serving traffic.A second pass solution is to run a ‘batch’ feature extraction pipeline in parallel to the production feature extraction pipeline. This batch pipeline will query the db for all members and add a ‘job seeker score’ to every member. This solution ensures that we have an upper bound on the time it takes to grandfather all members with job seeker score. It will work great for small startups whose member base is in a few million range.However, the downside of this solution at LinkedIn scale is:1) It adds load on the production databases serving live traffic.2) To avoid the load, we end up throttling the batch solution which in turn makes the batch pipeline run for days or weeks. This slows down rate of batch update.3) Lastly, the two factors above combine to make grandfathering a ‘dreaded word’. You only end up grandfathering ‘once a quarter’ which is clearly not helpful in innovating faster.So, we clearly cannot use that solution either. However, one good aspect of this solution is the batch update which leads us into a Hadoop based solution.Using Hadoop, we take a snapshot of member profiles in production, move it to hdfs, grandfather members with a ‘job seeker score’ in a matter of hours, and copy this data back online. This way we can grandfather all members with a job seeker score in hours.The biggest advantage of using Hadoop here is that grandfathering is no longer a ‘dreaded word’. We can grandfather when ready instead of grandfather every quarter which speeds up innovation.So, in a nutshell, if one finds oneself slowed down due to constraints of updating features in the online world, consider batch updating the features offline using Hadoop and swapping them out online.With the first few versions of similar profiles out the door, we began to simultaneously investigate a number of avenues for improvement. Some of us investigated different model families, say logistic regression vs SVM, others investigated new features with the existing model. In this case study, we will talk about how we decided which one of these experiments would actually improve the online relevance of similar profiles so we could double down on getting them out to production. We are not concerned with ‘how’ we come up with these new models. For all that matters, we hand tuned a common sense model. The question is how to decide whether or not to move that new model to production.Now, as a base solution, we can always move every model to production. We can A/B test the model with real traffic and see which models sink and which ones float. The simplicity of this approach is very attractive, however, there are some major flaws with it:1) For each models we have to push code to production. This takes up valuable development resources.2) There is an upper limit on the number of A/B tests one can run at the same time. This can be due to user experience concerns and/or revenue concerns.3) Since online tests need to run for days before enough data is accumulated to make a call, this approach slows down rate of progress.Ideally, we would like to try all our ideas offline, evaluate them, and only push to production the best ones. Hadoop proves critical in the evaluation step. Overtime, using implicit and explicit feedback combined with crowdsourcing, we have accumulated a huge gold test set for ‘similar profiles’. We rank the gold set with each model on Hadoop, and use standard ranking metrics to evaluate which one performs best.As you can guess, Hadoop provides a very good sandbox for our ideas. We are able to filter many of the craziest ideas, and double down on only those few that show promise. Plus, it allows us to have relatively large gold sets which gives us strong confidence in our evaluation results.Now that we have learned a new model for ‘Similar Profiles’ that performs well in our offline evaluation framework, we need to test it online. An industry standard approach to this problem is known as A/B testing or bucket testing. Formally, AB testing involves partitioning real traffic between alternatives and then evaluating which alternative maximizes the desired objective. Typical desired objectives are CTR, revenue, or number of views.The key requirements of AB testing is that: time to evaluate which bucket to send traffic to should ideally be < 1ms, at-worst a few ms.Let’s discuss how we would do A/B testing for our new model. For simple partitioning requirements, one can use a mod-based scheme. This is very fast and very simple and can satisfy most use cases. However, if one wishes to partition traffic based on profile and member activity criteria for e.g. “Send 10% of members who have greater than 100 connections AND who have logged in the last one week AND who are based in Europe” then doing this online is too expensive. Keep in mind that deciding which bucket to send traffic to should be very fast, ideally less than a millisecond. In worst case scenario a few milliseconds. I am not going to even attempt an online solution for this problem.So, we go straight to Hadoop. For complex criteria like this, we run over our entire member base on Hadoop every couple of hours, assigning them to the appropriate bucket for each test. The results of this computation are pushed online, where the problem of A/B testing reduces to given a member and a test, fetching which bucket to send the traffic to from cache.The take home message here is: If you need complex targeting and A/B testing, leverage Hadoop.Our last case study involves the last step of a model deployment process: tracking and reporting. These two steps allow us to have an unbiased, data-driven way of saying whether or not a new model is successful in lifting our desired metrics: ctr or revenue or engagement or whatever else one is interested in. Our production servers generate tracking events every time a recommendation is impressed, clicked or rejected by the member.Before Hadoop, we used to have an online reporting tool that would listen to tracking events over a moving window of time, doing in-memory joins of different streams of events and reporting up-to-the minute performance of our models. The clear advantages of this approach were that we could see exactly how the model was performing online at this moment. However, there are a few downsides such asa) One cannot look at greater than certain amount of time window in the past..b) As the number of tracking streams increases, it becomes harder and harder to join them online.To increase the time window, we will have to spend significant engineering resources in architecting a scalable reporting system which would be an overkill. Instead we placed our bet on Hadoop. All tracking events at LinkedIn are stored on HDFS. Add to this data Pig or plain Map-Reduce, we can do arbitrary k-way joins across billions of rows to come up with reports that can look as far in the past as we want to.The advantages of this approach are quite clear. Complex joins are easy to compute and reporting is flexible on time windows. However, we cannot have up-to-the minute reports since we copy over tracking events in batch to hdfs. If we ever need that level of reporting, we can always use our online solution.We can say without any hesitation that Hadoop has now become an integral part of the whole life-cycle of our workflow starting from prototyping a new idea to eventually tracking the impact of that idea.By thinking about platform and not verticals, we are able to to come with newer verticals at a fast pace.By leveraging Hadoop, we were able to continuously improve the quality and scale the computations.Hence, these 2 ideas helped us ‘scale innovation’ at Linkedin.To conclude, we want to say that the data opportunity at LinkedIn is HUGE and so come work with us at LinkedIn!