Mn Dot Inspection: Fill & Download for Free

(This is the fourth answer in a 7 part series on Probabilistic Graphical Models ('PGMs').)So far, our running definition of inference has been:The task of using a given graphical model of a system, complete with fitted parameters, to answer certain questions regarding that system.Here, we'll discuss a general class designed to approximate these answers with simulations. That is, we'll get samples drawn from a distribution which approximates the distribution we're asking about.Due to one reason only, I enjoy these techniques the most. It's their generality. Exact inference algorithms demand the graphs are sufficiently simple and small. Variable Inference, a clever approximate approach, demands defining approximate distribution spaces and a means to search them effectively. Monte Carlo methods, however, demand no qualifying inspection of these graph - all graphs are fair game. This provides us with a much wider class of models to fit our reality.This isn't to say these methods are a cure-all. Yes, there are circumstances in which we fail to get answers. But the responsible reason, outside of the vague 'lack of convergence', is not well understood. So we accept these problem specific battles as the cost of supreme generality.But before we dive in, a short review will help.Refresher (Skip this if you’ve read answers 1 and 2!)In the first answer, we discovered why PGMs are useful for representing complex system. We defined a complex system as a set, [math]\mathcal{X}[/math], of [math]n[/math] random variables (‘RVs’) with a relationship we'd like to understand. We assume there exists some true but unknown joint distribution, [math]P[/math], which govern these RVs. We take it that a 'good understanding' means we can answer two types of questions regarding this [math]P[/math]:Probability Queries: Compute the probabilities [math]P(\mathbf{Y}|\mathbf{e})[/math]. This means: what is the distribution of [math]\mathbf{Y}[/math] given we have some observation of [math]\mathbf{E}[/math]?MAP Queries: Determine [math]\textrm{argmax}_\mathbf{Y}P(\mathbf{Y}|\mathbf{e})[/math]. That is, determine the most likely values of some RVs given an assignment of other RVs.(Where [math]\mathbf{E}[/math] and [math]\mathbf{Y}[/math] are two arbitrary subsets of [math]\mathcal{X}[/math]. If this notation is unfamiliar, see the 'Notation Guide' from the first answer).The idea behind PGMs is to estimate [math]P[/math] using two things:A graph: a set of nodes, one for each RV in [math]\mathcal{X}[/math], and a set of edges between them.Parameters: objects that, when paired with a graph and a certain rule, allow us to calculate probabilities of assignments of [math]\mathcal{X}[/math].Depending on these two, PGMs come in two main flavors: Bayesian Networks ('BNs') and Markov Networks ('MNs').A Bayesian Network involves a graph, denoted as [math]\mathcal{G}[/math], with directed edges and no directed cycles. The parameters are Conditional Probability Tables ('CPDs' or 'CPTs'), which are, as the naming suggests, select conditional probabilities from the BN. They give us the right hand side of the Chain Rule, which dictates we calculate probabilities this way:where [math]\textrm{Pa}_{X_i}^\mathcal{G}[/math] is the set of parent nodes of [math]X_i[/math] in the graph.A Markov Network's graph, denoted as [math]\mathcal{H}[/math], is different in that its edges are undirected and it may have cycles. The parameters are a set of functions (called ‘factors’) which map assignments of subsets of [math]\mathcal{X}[/math] to nonnegative numbers. Those subsets, which we'll call [math]\mathbf{D}_i[/math]'s, correspond to complete subgraphs of [math]\mathcal{H}[/math]. We can refer to this set as [math]\Phi=\{\phi_i(\cdots)\}_{i=1}^m[/math]. With that, we say that the Gibbs Rule for calculation probabilities is:where [math]Z[/math] is defined such that our probabilities sum to 1.To crystallize this idea, it's helpful to imagine the 'Gibbs Table', which lists that above product (without [math]Z[/math]) for each assignment. In the second answer, we pictured an example where [math]\mathcal{X}=\{C,D,I,G,S\}[/math] as:Lastly, we recall that the Gibbs Rule may express the Chain Rule. That is, we can always recreate the probabilities produced by a BN's Chain Rule with an another invented MN and its Gibbs Rule. Essentially, we define factors as those that reproduce looking up in a BN's CPDs. This equivalence allows us to reason solely in terms of the Gibbs Rule, while assured that whatever we discover will also hold for BNs. In other words, with regards to inference, if something works for [math]P_M[/math], then it works for [math]P_B[/math].Great, now… what's our starting point?We are handed a MN (might be the converted form of a BN). That is, we get a graph [math]\mathcal{H}[/math] and a set of factors [math]\Phi[/math]. We're interested in the distribution of a subset of RVs, [math]\mathbf{Y} \subset \mathcal{X}[/math], conditional on an observation of other RVs ([math]\mathbf{E}=\mathbf{e}[/math]). We'll have our answer presumably (to both queries) if we can generate samples, [math]\mathbf{y}[/math]'s, that come (approximately) from this distribution.The first step is to address conditioning. To do so, let's steal one idea from the second answer. That is, inference in a MN conditional on [math]\mathbf{E}=\mathbf{e}[/math] gives the same answer as unconditional inference in a specially defined MN. Let’s call its graph [math]\mathcal{H}_{|\mathbf{e}}[/math] and its factors [math]\Phi_{|\mathbf{e}}[/math]. It turns out [math]\mathcal{H}_{|\mathbf{e}}[/math] is [math]\mathcal{H}[/math], but with all [math]\mathbf{E}[/math] nodes and any edges involving them deleted. [math]\Phi_{|\mathbf{e}}[/math] is the set of factors [math]\Phi[/math], but with [math]\mathbf{E}=\mathbf{e}[/math] fixed as an input assignment. The point is if we can do unconditional inference, we can do conditional inference, granted we performed this change. For the sake of cleanliness, I'll drop the '[math]|\mathbf{e}[/math]' and assume we've already done this conditioning conversion.Finally, we're ready for the big idea.Markov Chain Monte Carlo (MCMC)In a nutshell, MCMC finds a way to sequentially sample [math]\mathbf{y}[/math]'s such that, eventually, these [math]\mathbf{y}[/math]'s are distributed as [math]P_M(\mathbf{Y}|\mathbf{e})[/math].To see this, we must first defined a Markov Chain. All this is is a set of states and transition probabilities defined between such states. These probabilities are the chances we transition to any other state given the current state. For our purposes, the set of states is [math]Val(\mathbf{X})[/math] where [math]\mathbf{X}=\mathcal{X}[/math] - all possible joint assignments of all variables. For any two states [math]\mathbf{x},\mathbf{x}' \in Val(\mathbf{X})[/math], we write the transition probability as [math]\mathcal{T}(\mathbf{x} \rightarrow \mathbf{x}')[/math]. We may refer to all such probabilities or the whole Markov Chain as [math]\mathcal{T}[/math].As a simple example, suppose our system was one RV, [math]X[/math], that could take 3 possible values (so [math]Val(X)=[x^1,x^2,x^3][/math]). Then we might have this Markov Chain:Thinking generally again, to 'sample' a Markov Chain means we sample a starting [math]\mathbf{x}^{(0)}\in Val(\mathbf{X})[/math] according to some starting distribution [math]P_\mathcal{T}^{(0)}[/math]. Then, we use our [math]\mathcal{T}(\mathbf{x} \rightarrow \mathbf{x}')[/math] probabilities to determine the next state, giving us [math]\mathbf{x}^{(1)}[/math]. Then we repeat, giving us a long series of [math]\mathbf{x}^{(t)}[/math]'s. If we were to restart the sampling procedure many times and select out the [math]t[/math]-th sample, we'd observe a distribution that we'll call [math]P_\mathcal{T}^{(t)}[/math].So, a sample of our toy examples might be: [math]x^1[/math] (33% chance) [math]\rightarrow x^3[/math] (75% chance) [math]\rightarrow x^2[/math] (50%) [math]\rightarrow x^2[/math] (70%). The first has a 33% chance because we assume [math]P_\mathcal{T}^{(0)}[/math] is uniform. Simple enough, right?By the nature of this procedure, we can figure this relation:This is to say: the probability of the next state one step from now is equal to the sum of the probabilities of being in another state and transitioning from there to that next state.Now, for a large [math]t[/math], it's reasonable to expect [math]P_\mathcal{T}^{(t)}[/math] to be very similar to [math]P_\mathcal{T}^{(t+1)}[/math]. Under some conditions, that's correct intuition. Whatever that common distribution is, we call it the stationary distribution of [math]\mathcal{T}[/math], and it's written as [math]\pi_\mathcal{T}[/math]. It is the single distribution that works for both [math]P_\mathcal{T}^{(t+1)}[/math] and [math]P_\mathcal{T}^{(t)}[/math] in that above relation. That is, it solves:In effect, [math]\pi_\mathcal{T}[/math] is the distribution that [math]P_\mathcal{T}^{(t)}[/math] converges too.With that, we're ready for the big insight:We may choose our Markov Chain, [math]\mathcal{T}[/math], such that [math]\pi_\mathcal{T} = P_M[/math]Now, conceivably, we could make our choice of [math]\mathcal{T}[/math] with [math]P_M[/math] in mind and solve for the stationary distribution to get our answer. However, in general, this isn't possible. Hence, we need our next 'MC'. That is, we'll use Monte Carlo simulations to solve our problem. Instead of trying to solve for [math]\pi_\mathcal{T}[/math], we execute the sampling procedure to produce a series of [math]\mathbf{x}^{(t)}[/math]'s, and then we observe an empirical approximation to [math]\pi_\mathcal{T}[/math] (and hence [math]P_M[/math]) after a number of iterations. If we are concerned with a subset [math]\mathbf{Y}[/math] from [math]\mathcal{X}[/math], we simply select out the [math]\mathbf{Y}[/math]-elements from our series of [math]\mathbf{x}^{(t)}[/math]'s and use those.OK, but...How do we choose [math]\mathcal{T}[/math]?This is where things get hairy. Fortunately, the algorithms make this decision for us, but to understand their relative advantages, we need to understand their common aim. None of them entirely nail that aim.In a nutshell, that aim is:We'd like a [math]\mathcal{T}[/math] for which sampling will converge quickly to a single [math]\pi_\mathcal{T}[/math], equal to [math]P_M[/math], from any starting [math]P^{(0)}_\mathcal{T}[/math].This is hard. Here are the major ways it may crash and burn:Imagine a [math]\mathcal{T}[/math] with two states where if you're on one, you always transition to the other. This makes for a heavy dependence on [math]P^{(0)}_\mathcal{T}[/math] and no stationary distribution [math]\pi_\mathcal{T}[/math]. Cyclic behavior like this means the Markov chain is periodic - we hate periodic chains.A [math]\mathcal{T}[/math] with a low conductance is one in which there are regions of the state space which are very hard to go between. This means that if you start in one, it'll take you a long time before you explore the other. Therefore, we have a near-dependency on [math]P^{(0)}_\mathcal{T}[/math]. Also, convergence will require traversing that narrow bridge, so it certainly won't be quick. If dotted lines imply small transition probabilities, this is an example of a low conductance [math]\mathcal{T}[/math]:If there exist two states such that if you're on one, you can never reach the other, that [math]\mathcal{T}[/math] is called reducible and the consequence is that there may be more than one stationary distribution.The protection against this unruly behavior are some theoretical properties you may demand of a [math]\mathcal{T}[/math]. You may demand it's aperiodic for example.Probably the most helpful one is called detailed balance. A [math]\mathcal{T}[/math] with this property has a [math]\pi_\mathcal{T}[/math] such that:for any pair of [math]\mathbf{x}, \mathbf{x}' \in Val(\mathbf{X})[/math].Compare this to the equation that defined the stationary distribution. The right side of that has many more terms, and as such, it has many more degrees of freedom for [math]\pi_\mathcal{T}(\mathbf{x})[/math] to sit within. Intuitively, detailed balance means the stationary distribution follows from single step transitions, and not from large cycles of many transitions. It's a kind of 'well connectedness'. If [math]\mathcal{T}[/math] implies any two states are reachable from each other, is aperoidic and has detailed balance, than we'll converge to the unique stationary distribution from any starting distribution.Enough with the theoretics! Let's see an algorithm.Gibbs Sampling - a type of MCMCIn this method, our first step is to uniformly sample a [math]\mathbf{x}^{(0)}[/math] from [math]Val(\mathcal{X})[/math]. Here, our transition probabilities will be such that we are forced to change this vector one element at a time to produce our samples. Specifically:Pick out [math]X_1[/math] from [math]\mathbf{X}[/math] and list out all values of [math]Val(X_1)[/math]. For example, [math]Val(X_1)=[x_1^1,x_1^2,x_1^3,x_1^4][/math].For each [math]x_1^i \in Val(X_1)[/math], create a new vector by subbing that [math]x_1^i[/math] into the [math]X_1[/math]-position of [math]\mathbf{x}^{(0)}[/math] (call it [math]\mathbf{x}^{(0)}_{i-subbed}[/math]).Plug each [math]\mathbf{x}^{(0)}_{i-subbed}[/math] into our Gibbs Rule[1], giving us 4 positive numbers and normalize them into probabilities.Use this size-4 probability vector to sample one of these [math]\mathbf{x}^{(0)}_{i-subbed}[/math]‘s. Use that as [math]\mathbf{x}^{(1)}[/math].Go back to step 1, but use [math]X_2[/math] this time to produce [math]\mathbf{x}^{(2)}[/math].And we keep cycling through these steps for as long as we'd like. In effect, steps 2-4 are sampling from a specially defined set of transition probabilities. As a result, if we pick a [math]t[/math] large enough, [math]\mathbf{x}^{(t)}[/math] will come from [math]P_M(\mathbf{X})[/math][2].But, you may have noticed an issue. [math]\mathbf{x}^{(t)}[/math] is only different from [math]\mathbf{x}^{(t-1)}[/math] by one element, so these sequences are seriously correlated over short distances. This series isn't a set of independent samples!To address this, we keep track of the effective sample size of our simulations. Imagine our sampling produced a series that is perfectly correlated - they are all the same. Clearly, this is effectively a single independent sample. At the other end, imagine there is no correlation - we have our independence and we have as many independent samples as samples (beyond that 'large enough' [math]t[/math]). So given the case where we have some correlation, the effective sample size is a number between these two extremes. There exists some heuristics to estimate this figure from lagged correlations. It's an important figure to keep handy, as it gives you a clue as to how good our approximated inference is.But correlation isn't the only issue. Since we update one [math]X_i[/math] at a time, it's also fairly slow.We need to get more general.The Metropolis-Hastings AlgorithmThe issue in Gibbs Sampling is that we move through the states very slowly. It would help if we could control how fast we explore this space. To guarantee that the resulting [math]\pi_\mathcal{T}[/math] equals our [math]P_M[/math], we'd have to apply a correction to compensate for that control. Roughly, this is the idea behind the Metropolis-Hasting algorithm.More specifically, we invent another Markov Chain, which we'll call [math]\mathcal{T}^Q[/math], which is our proposal Markov Chain. That is, it's defined over the same space ([math]Val(\mathbf{X})[/math]) and is responsible for proposing the next state, given whatever current state. This is our opportunity to invoke large leaps. Our correction will be to sample a yes-no event according to a certain acceptance probability. If we draw a yes, we transition to the proposed state. If we draw a no, we remain at the current state. This acceptance probability is specially design to ensure detailed balance. It is:This acceptance probability ensures that we will converge to [math]P_M[/math] from any starting distribution, given [math]\mathcal{T}^Q[/math] isn't especially badly behaved.For the intuiters, let's decompose [math]\mathcal{A}[/math]. It's effectively made up of [math]\frac{\tilde{P}_M(\mathbf{x}')}{\tilde{P}_M(\mathbf{x})}[/math] and [math]\frac{\mathcal{T}^Q(\mathbf{x}'\rightarrow\mathbf{x})}{\mathcal{T}^Q(\mathbf{x}\rightarrow\mathbf{x}')}[/math].[math]\frac{\tilde{P}_M(\mathbf{x}')}{\tilde{P}_M(\mathbf{x})}[/math] makes us more likely to reject states that are unfavorable according to [math]P_M[/math]. Notice it uses the unnormalized probabilities - no intractable [math]Z[/math] involved! Also, since it's a ratio involving the Gibbs Rule, if [math]\mathbf{x}'[/math] and [math]\mathbf{x}[/math] share some identical terms (maybe by design of our [math]\mathcal{T}^Q[/math]), it's possible some factors cancel. So we could save time by avoiding computation of the full Gibbs Rule.[math]\frac{\mathcal{T}^Q(\mathbf{x}'\rightarrow\mathbf{x})}{\mathcal{T}^Q(\mathbf{x}\rightarrow\mathbf{x}')}[/math] makes us unlikely to accept states that are easy to transition to (according to our proposal) but difficult to return from. Such behavior is at odds with detailed balance, so it's curtailing it for that sake of this property.Now, there are some practical considerations to keep in mind. First, [math]\mathcal{T}^Q[/math] should be able to propose everything in [math]Val(\mathbf{X})[/math], otherwise we'll give a zero probability to something [math]P_M[/math] might favor. Second, a rejected proposed sample is a waste of your computer’s time, so in this sense, we like high [math]\mathcal{A}[/math]'s. However, very high [math]\mathcal{A}[/math]'s might mean you aren't exploring the space quickly. Together, this means that [math]\mathcal{T}^Q[/math] must be tuned right to explore the space efficiently.Let's see it!I can't think of a better visual than the one I've seen in Kevin Murphy's text (Chapter 24, see source [2]). So, I’ll use that. First, let's pretend our 'intractable' [math]P_M[/math] is a mixture of two Gaussian distributions, like this:We'd like a set of samples which spent time along the horizontal axis in proportion to the height of this graph. To do so, we'll use a normal distribution centered on our current state: [math]\mathcal{T}^Q(\mathbf{x}\rightarrow\mathbf{x}') = \mathcal{N}(\mathbf{x}'|\mathbf{x},v)[/math] where [math]v[/math] is the variance (which we'll play with). We'll assume we can evaluate the probability ratio of two samples according to the intractable distribution. With that, we know how to generate proposals, accept/reject them and produce samples. From the book, this process can be represented as:As pointed out, there is a happy middle ground. There is a particular proposal variance that strikes the right balance of exploring the space and being accepted frequently.And that just about does it.MCMC is our most general tool for sampling from any given PGM. We need skill and intuition to shepherd these methods away from their failure points, but with that, we can perform inference on some rather exotic graphs. That shepherding becomes more difficult with the size and complexity of the PGM, so understanding these difficulties beforehand is the key to knowing when to use MCMC.What's next?At this point, especially if you've read answers 2 and 3 as well, you may have had your fill of inference. You might be curious as to how we actually learn the parameters of a BN or a MN. Well, you might enjoy:5. How are the parameters of a Bayesian Network learned?6. How are the parameters of a Markov Network learned?Footnotes[1] Actually, you don't need to compute the full Gibbs Rule product - you only need to consider the factors for which [math]X_1[/math] appears in. Just think about it - for all the factors for which [math]X_1[/math] doesn't appear, their product remains constant as you plug in different assignments of [math]X_1[/math]. When we normalize, this constant will be divided out, so we don't need to consider it. For large MNs, this is a huge efficiency gain![2] Well, [math]P_M(\mathbf{X}|\mathbf{e})[/math] in fact.Sources[1] Koller, Daphne; Friedman, Nir. Probabilistic Graphical Models: Principles and Techniques (Adaptive Computation and Machine Learning series). The MIT Press. The Markov Chain visuals are from this book, with Daphne’s permission.[2] Murphy, Kevin. Machine Learning: A Probabilistic Perspective. The MIT Press.

Ahhhhhh! Dear ANON, Nice troll try, but, sorry, no cigar.It’s actually too bad that you have no clue how engineering works, or how an engineering failure occurs. So, let me educate you just a teeny bit.First (and foremost), that bridge stood for 43 years. And, while many modern (1880s and later design) bridges can last almost 100 years (Lake Street-Marshall Bridge), they have serious problems as they age. So, let’s look at the problems that develop, shall we:Corrosion: This is the number one problem. It weakens all parts of the bridge.Metal fatigue: Bridges go through continuous expansion-shrinkage cycles. This places great stress on the components.Age: Steel is not a static thing. Components actually flow within steel. Also, as this happens, the parts become brittle with age.Quality of steel: This is critical as poor quality means that it cannot hold up to planned design specs.Engineering Design: This is a compromise between cost, durability, and safety. (More here in a bit.)After effect loading: This is where an engineered unit is used in a manner that exceeds it’s original design load parameters and, just maybe, its safety loading.Failure to identify a critical problem on inspection.OK, now that we have the basics, what happened to the I35W bridge in Minneapolis? Simple, really! A “gusset” plate failed. This caused an overload on the cord, which failed. The other cord, then could not handle the combined load & also failed. The bridge collapsed.So, what happened to cause the failure? 1, 2, 5, 6, and 7 above. Here’s the scoop:1, 2, & 7: The Gusset Plate had significant corrosion & was showing stress cracks upon inspection by MN DOT . It was deemed safe, multiple times.6: The bridge deck had been overlayed with a new surface some time before. This heavy deck was in the process of being replaced. the contractor had stored some equipment & replacement material on the bridge, itself.5: The original engineering firm, way back in the ’60s had designed the bridge with a thinner than before gusset plate system. The firm had not anticipated the extra loading, which pushed the safety margin. (The safety margin would have held, except for 1 & 2 above. It also looks like the extra load of the stored material assisted in the collapse due to overloading.)Thus, the bridge fell. The end result? The successor to the engineering firm was sued for not anticipating 53 years of corrosion, metal fatigue, overloading, and incompetent inspection.Note the Cement tankers, dump trucks, & etc. on the bridge in the lower left-hand center of the photo://Image from a City pages web page.Here’s a closer look://From: Workers Remember the I-35W Bridge Collapse, Fight for ChangeEvidence of serious overloading. And, it’s telling that the deck splintered under this load on impact with the river bed.You might also want to research the failure of:Silver Bridge - Wikipedia, Silver bridge collapse (37 years old)Tacoma Narrows Bridge (1940) - Wikipedia, Lessons From the Failure of a Great MachineBrooklyn Bridge - Wikipedia, https://erenow.com/common/the-great-bridge-the-epic-story-of-the-building-of-the-brooklyn-bridge/20.htmlEngineering is an art and the Engineers cannot see far into the future!PS. Since then, MN DOT and other safety inspectors have become *much* more cautious and closed more suspect bridges and designated them for repair, or replacement.Even a car park, when a large cement piece dropped from the ceiling & smashed a parked car. Falling chunk of concrete leads to closure of RiverCentre parking ramp. Corrosion and (smaller) falling cement chunks have been obvious for years. The bridge & roadways over & under the bridge between the car park & the River Center are probably next.More reading: The Tay Bridge Disaster