Step 1 Pick Your Gradient: Fill & Download for Free

If you know what Gradient descent is, it is easy to think of Gradient Boosting as an approximation of it. So lets start with Gradient Descent. Note that I am presenting a simplified version of things. For rigor you can refer to the original paper or any of the books that cover it.Math alert! :)Gradient Descent (GD) - a short primerYou have a function [math]f(x)[/math] that you want to minimize. Assume [math]x[/math] to be a scalar. One way to iteratively minimize, and find the corresponding [math]x[/math] at the minima, is to follow this update rule at the [math]i^{th}[/math] iteration:[math]x^{(i)} = x^{(i-1)} - \eta \frac{df(x^{(i-1)})}{dx}[/math]Here [math]\eta[/math] is a positive constant.In effect, the value of [math]x[/math] found in the current iteration is its value in the previous iteration added to some fraction of the slope/gradient at this previous value. We stop when [math]x^{(i)} = x^{(i-1)}[/math]. We may start with an arbitrary value for [math]x^{(0)}[/math].The following fig shows how this works out:​When [math]x^{(i-1)}[/math] corresponds to A, the (negative) gradient is quite high, and consequently [math]x^{(i)}[/math] (at A') occurs farther ahead. At B, the gradient is considerably less, and hence B' occurs close to B. C and C' are the exact same point since the gradient here is [math]0[/math]. This is our stopping condition and C is where the minimum occurs.In effect, we seem to be moving our current estimate by an amount proportional to the gradient. This makes sense, since we expect the gradient to gradually become [math]0[/math] near the minima, and therefore, farther away we are from it, the absolute value of the gradient is higher. We don't want to spend a lot of iterations in these regions - so we move with longer "steps". Conversely, near the minima, we want to be cautious enough to not overshoot the minima, so we want to take smaller steps.In the case where [math]x[/math] is a vector, the principle remains the same. Now, we adjust every individual dimension of [math]x[/math] based on the slope along that direction. For the [math]i^{th}[/math] iteration and the [math]j^{th}[/math] dimension, this is what the update rule looks like:[math]x^{(i)}_j = [/math][math]x^{(i-1)}_j - \eta\frac{\partial f(x^{(i-1)})}{\partial x^{(i-1)}_j} [/math]At each iteration all dimensions are adjusted. The idea is, we want to move the vector [math]x[/math], as a whole, in a direction where each individual component minimizes [math]f(x)[/math]. This is important - this form would show up in the subsequent discussion.This is all we really need to know about gradient descent to understand gradient boosting.Gradient BoostingGradient Boosting carries over the previous technique to supervised learning. If you are careful with the notation its not difficult to see how it works.So we start with a function to minimize. We want a function whose value increases with how bad the classifier/regressor is. For a general treatment, we refer to this function as the loss function represented by [math]L[/math]. For Gradient Boosting loss functions must be differentiable. An example is the squared error between the actual and predicted value i.e. [math]L = (y_i - h(x_i))^2[/math].We want to minimize [math] f(x) = \sum_{i=1}^{N} L(y_i, h(x_i))[/math] i.e. the loss over all points [math](x_i, y_i)[/math]. Here [math]h(x)[/math] is the classifier/regressor; which for brevity we'll refer to as the predictor. [math]N[/math] is the total number of points.In the example of Gradient Descent above we minimized wrt [math]x[/math]. What are we minimizing wrt here? We are minimizing wrt the predictor function [math]h(x)[/math] since we want a predictor that minimizes the total loss f[math](x)[/math].Don't worry about minimizing wrt a function - that won't complicate matters much.Moving to the iterative world of gradient descent these are the steps we now take:Initialize [math]h^{(0)}(x) = c[/math], a constant, such that [math]c[/math] minimizes [math]f(x)[/math] i.e. pick [math]c[/math] that minimizes [math]\sum_{i=1}^N L(y_i,c)[/math].At the [math]i^{th}[/math] iteration, for [math]j=1,2,..., N[/math] compute [math]r_{ji}=-\frac{\partial L(y_j, h^{(i-1)}(x_j))}{\partial h^{(i-1)}(x_j)}[/math]. This is doable since we have assumed [math]L[/math] to be differentiable - for the ex of squared error, [math]\frac{\partial L}{\partial h}=-2(y-h)[/math] - we are only plugging in the values of [math]y_j[/math] and [math]h^{(i-1)}(x_j)[/math] in the differentiated expression. This is analogous to how we dealt with the components of the vector [math]x[/math] in the previous section.The previous step gives us a value [math]r_{ji}[/math] for each point [math]j[/math]. Thus we have a set of tuples [math](x_j, r_{ji})[/math]. We use this set of points as training data to construct a regression tree that can predict [math]r_{ji}[/math] given [math]x_j[/math]. This tree approximates the gradient. Think of the tree as a "black box" gradient expression which produces [math]r_{ji}[/math] given [math]x_j[/math] . This takes place of the [math]\frac{\partial f(x^{(i-1)})}{\partial x^{(i-1)}_j}[/math] expression we saw in GD, with this one tree sort of embodying the gradient for all points [math]x_j[/math]. We'll refer to this tree as [math]T^{(i)}_g[/math] (g for gradient, i is for the iteration). As before we want this gradient-tree to play a role in the update equation, but we are still left with the task of finding [math]\eta[/math] ...Assume that the tree [math]T^{(i)}_g[/math] has [math]K[/math] leaves. We know that the leaves of a tree fragment the feature space into disjoint exhaustive regions. Lets refer to these regions as [math]R_k[/math], for [math]k=1,2,...,K[/math]. If we send each point [math]x_j[/math] down the tree [math]T^{(i)}_g[/math], it will end up at some region [math]R_k[/math]. We now want to associate a constant [math]\eta_k[/math] for each such region [math]R_k[/math] such that the loss in a region, defined as: [math]\sum_{x_j \in R_k}[/math][math]L(y_j, h^{(i-1)}(x_j) + \eta_k)[/math] is minimized. These are solved as [math]k[/math] (simple) independent minimization problems for the [math]k[/math] regions. Note that now we have a tree providing well defined regions [math]R_k[/math], and we have constant values [math]\eta_k[/math], associated with each of these regions. In effect, this combination may be seen as another tree: structure given by [math]T^{(i)}_g[/math], but [math]\eta_k[/math] as predictions at the leaves.Finally, we come to the update step: [math]h^{(i)}(x) = h^{(i-1)}(x) + \sum_k \eta_k I(x \in R_k)[/math]. Here [math]I(x \in R_k)[/math] is an indicator function that has a value of [math]1[/math] when [math]x[/math] falls in the region [math]R_k[/math], [math]0[/math] otherwise. Don't let the indicator function confuse you - its just an elegant way of saying that for a point [math]x[/math] the second term is [math]\eta_k[/math] corresponding to the region is falls into. This second term, as mentioned in the previous step, is effectively a tree derived from [math]T^{(i)}_g[/math]. You can probably now see why [math]\eta_k[/math] was determined in the way it was: the minimization in the last step and the updation have the same form; thus the updated function has the minimum possible loss. Note how similar the update equation is to that of GD. You have your gradient and you have your [math]\eta[/math] (well, you've more than one now - [math]\eta_k[/math] - but this is a minor difference). Also note, and this is very interesting, there is actually no addition taking place in this updation step - what we are simply saying is, if you want to compute [math]h^{(i)}(x)[/math], compute [math]h^{(i-1)}(x)[/math], and add to it what ever [math]\eta_k[/math] you obtain by passing [math]x[/math] down the tree represented by the second term.Keep going back to step 2 till you have iterated the number of times - say [math]M[/math] - you want to.Finally return [math]h^{(M)}(x)[/math] as your predictor. Since at every iteration, your only update to the function at that point is adding a tree in step 5, what you finally return is a sum of trees. Or rather, you return a bunch of trees whose sum (plus [math]c[/math] from Step 1) is supposed to give you the function [math]h^{(M)}(x)[/math].(I personally don't like GOTO statements like Step 6, but my inability to created nested lists on Quora has stymied me here :) ).A few things to think about:Make sure you understand the role of calculating the per-point-gradient in Step 2. In the end we want the function [math]h(x)[/math] to minimize loss over all training points. As mentioned before, this is analogous to the case of a vector valued [math]x[/math] in GD.We had necessitated that the loss function be differentiable - we see this property being used in Step 2.Where are major computations happening in the algorithm? In constructing the "gradient trees" in Step 4 and determining the [math]\eta_k[/math] values in Step 5.In essence we have learnt a function [math]h^{(M)}(x)[/math] based on the values [math](x_j,y_j)[/math], that minimizes prediction errors [math]f(x)[/math]. The minimization is done in multiple steps: at every step we add a tree (Steps 4 and 5) that emulates adding a gradient based correction - very much like in GD. Using trees ensures that generalization of the gradient expression happens - because we need the gradient for an unseen/test point at each iteration as part of the calculation of [math]h^{(M)}(x).[/math]Probably trivial, but it is also interesting to note how our final function differs in form relative to the final point returned in GD. In GD, we have an updated point - one point - at any iteration. In the case of Gradient Boosting, we don't have a neat form in which the updated function exists; the function exists tangibly as a bunch of trees, with each tree representing the update in some iteration. In GD, you come to me after the [math]i^{th}[/math] iteration, I will give you a point [math]x_i[/math]; here, after the [math]i^{th}[/math] iteration, I will give you [math]i[/math] trees (plus, of course, the constant [math]c[/math] from Step 1): this is what [math]h^{(i)}(x)[/math] is.I find Gradient Boosting to be one of the cleverer algorithms out there due to the manner in which it adapts a well known optimization technique to the domain of learning.PS: Can someone help me out with the formatting here? - would really appreciate it. How do I include line breaks within a bullet point? I really want to break up points 4 and 5 - they look ungainly. I tried this In a Quora answer, how can I split one point in a bulleted list into two or more paragraphs or units of text? - but that didn't work. Is there a way to modify the HTML directly?

The chrome://newtab page is built with a lot of JavaScript, a line of which is responsible for those stripes:chrome.send('getFaviconDominantColor', [faviconUrl, this.id]); If we look in the Chromium source, this internal browser callback jumps around for a while before the color is determined and set on the page. If we trace it around for a while (see http://src.chromium.org/svn/trunk/src/chrome/browser/ui/webui/ntp/favicon_webui_handler.cc, http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.cc, and http://src.chromium.org/svn/trunk/src/ui/gfx/color_analysis.h), we finally get to CalculateKMeanColorOfPNG in the color analysis include (http://src.chromium.org/svn/trunk/src/ui/gfx/color_analysis.h).Here's the explanation currently included with that:// Returns an SkColor that represents the calculated dominant color in the png. // This uses a KMean clustering algorithm to find clusters of pixel colors in // RGB space. // |png| represents the data of a png encoded image. // |darkness_limit| represents the minimum sum of the RGB components that is // acceptable as a color choice. This can be from 0 to 765. // |brightness_limit| represents the maximum sum of the RGB components that is // acceptable as a color choice. This can be from 0 to 765. // // RGB KMean Algorithm (N clusters, M iterations): // TODO (dtrainor): Try moving most/some of this to HSV space? Better for // color comparisons/averages? // 1.Pick N starting colors by randomly sampling the pixels. If you see a // color you already saw keep sampling. After a certain number of tries // just remove the cluster and continue with N = N-1 clusters (for an image // with just one color this should devolve to N=1). These colors are the // centers of your N clusters. // TODO (dtrainor): Check to ignore colors with an alpha of 0? // 2.For each pixel in the image find the cluster that it is closest to in RGB // space. Add that pixel's color to that cluster (we keep a sum and a count // of all of the pixels added to the space, so just add it to the sum and // increment count). // 3.Calculate the new cluster centroids by getting the average color of all of // the pixels in each cluster (dividing the sum by the count). // 4.See if the new centroids are the same as the old centroids. // a) If this is the case for all N clusters than we have converged and // can move on. // b) If any centroid moved, repeat step 2 with the new centroids for up // to M iterations. // 5.Once the clusters have converged or M iterations have been tried, sort // the clusters by weight (where weight is the number of pixels that make up // this cluster). // 6.Going through the sorted list of clusters, pick the first cluster with the // largest weight that's centroid fulfills the equation // |darkness_limit| < SUM(R, G, B) < |brightness_limit|. Return that color. // If no color fulfills that requirement return the color with the largest // weight regardless of whether or not it fulfills the equation above. SkColor CalculateKMeanColorOfPNG(scoped_refptr<RefCountedMemory> png,  uint32_t darkness_limit,  uint32_t brightness_limit); So, basically, the k-means clustering algorithm converts the favicon into blocks of colors that are roughly the same, sometimes called superpixels. You can visualize the result of this similar to Voronoi cells:via: http://en.wikipedia.org/wiki/Voronoi_cellThis way, pixels next to each other will be grouped together if they are different only by an imperceptible amount. This happens often in image compression and when you have slight fades between colors in gradients or photos, so this step is necessary to have the precise algorithmic analysis get a more general idea of what the favicon is doing.Then, you return the largest block's color (calculated as the centroid of the block with the most pixels).

I imagine you are referring to the technical process Input-Throughput-Output. Here goes:Input: an email database with features including sender’s email, cc-emails, title, body, attachments, html codes,… and most importantly a classification ‘spam’ vs. ‘not-spam’ pre-determined by human beings.Throughput:Pick among the available algorithm - from something as simple as logistic regression or decision tree to something as complex as a deep neural network or gradient boosting random forest. Each option has strengths and weaknesses you need to evaluate based on available data (this is when you need expertise from a machine learning specialist)Dependent on step 1, wrangle and transform your provided data into the appropriate feedstock for your algorithm. Save at least 30 percent of the data for validation and testing.Feed the training data in, use validation data to optimise your algorithm and test data to measure its effectiveness. You might decide to further morph your available training data using techniques such as SVM or PCA or other datatype-specific manifold mappings. Trials and errors are your best friends here (this is why experts say each problem/industry is different)Output: You should be happy with your algorithm performance now. Deploy it and build a useful product/workflow around it. Ideally your application should, in real-time, be able to access the email-in-question, pre-process it into the correct form, feed it to your algorithm and serve up the classification ‘spam’ or ‘not-spam’ to you. This last step might require the programming assistance from software engineers.