Sample R Sum S & Cover Letters - Porterville College: Fill & Download for Free

John Allen Paulos wrote a great column, Imagining a Hit Thriller With Number 'e', about "four mysterious appearances of e". All four yield methods for probabilistically approximating e. They are not equally efficient (statistically or computationally), and it would be interesting to create a fair way to compare their efficiencies.Here are the four mysterious appearances, written more concisely but less interestingly than in Paulos' article. I've also included R code for each.1. Empty boxes. Randomly put n balls into n boxes, where the boxes are labeled from 1 to n (for each ball, choose a uniformly random box, independently). Then n over the number of empty boxes should approximately be e.R code: n <- 10^4 s <- sample(1:n,n,replace=TRUE) t <- tabulate(s) n/sum(t==0)  2. Matching problem. Shuffle a deck of 52 cards labeled 1 through n, and count how many times it happens that the jth card in the deck has label j. Then the probability of there being no match is approximately 1/e.R code: m <- 52 n <- 10^4 r <- replicate(n,sum(sample(m)==(1:m))) n/sum(r==0)  3. Setting records. There are n runners, and they run a certain distance one at a time. A runner sets a record if his or her time is better than those of all the previous runners (so the first runner has the luxury of always setting a record). Assume that the times are independent and all have the same distribution. Let K count how many of the first n runners set records. Then for large n, the Kth root of n should approximately be e. Warning: This method requires a much larger value of n than the others to yield a good approximation.R code: n <- 10^4 u <- runif(n) r <- 0 k <- 0 for (j in 1:n) {  if (u[j]>r) {  r <- u[j]  k <- k+1  } } n^(1/k)  4. Adding random numbers until exceeding 1. Generate independent uniformly random numbers in the interval [0,1] until the total exceeds 1. On average, the number of numbers generated is e.R code: f <- function(){  s <- 0  j <- 0  while (s < 1) {  s <- s+runif(1)  j <- j+1  } j } n <- 10^4 r <- replicate(n,f()) sum(r)/n  

Great question!The basis of DQN is good ol’ Q-learning and so the reason DQN does not require importance sampling is the same reason classic Q-learning does not. Lets then review when you need importance sampling and why Q-learning is what it is. That should then make it clear why it’s not needed. If you feel really comfortable with why these techniques are what they are already, feel free to skip to the bottom.Expected Values and SamplingImportance sampling typically regards the common problem of wanting to estimate the expected value of some function on some random distribution.That is, we want to estimate: [math]E_{x\sim p}\left[f(x)\right][/math], where p is the random probability distribution. If we knew how the probability mass function and the function f were defined, then we could compute the expected value analytically:[math]E_{x\sim p}\left[f(x)\right] = \sum_x p(x)f(x)[/math]Often times, we don’t actually know the definition of f though, we only get observations. Or, other times still, the space of x is infinite, in which case the analytic solution requires an integral that we may not able to compute analytically depending on f and p:[math]E_{x\sim p}\left[f(x)\right] = \int_X p(x)f(x)dx[/math].For either of these reasons, we turn to sampling approaches, where we can obtain samples of f(x) that we know are drawn from p. Samples allows us to compute the empirical Monte Carlo estimate:[math]E_{x\sim p}\left[f(x)\right] \approx \frac{1}{n}\sum_i^n f(x_i)[/math]Here’s a bonus that will become relevant when we return to Q-learning. Another way to estimate an expected value from samples is via an iterative procedure. In this procedure, we start with a very bad estimate of the expected value; lets call it [math]\mu_0[/math] , which we can set to any arbitrary value we want. Then, with each new sample of [math]f(x)[/math] with x sampled from p, we generate a new estimate of [math]\mu[/math] by moving it slightly in the direction of the value we observed. That is:[math]\mu_{i+1} = \mu_i [/math][math][/math][math]+ \alpha_i \left( f(x_i) - \mu_i \right) [/math][math][/math][math][/math]where [math]\alpha_i[/math] is the “learning rate” that we slowly decrease from some initial value (with simple restrictions). Following this procedure, as we increase the number iterations, [math]\mu_i[/math] will converge to the true expected value.So while taking an average is one way we can handle an empirical estimate of an expected value, we can also always switch it to the above iterative procedure too.Importance SamplingNow with the basics out of the way we can get to when we need importance sampling. Suppose we wanted to estimate an expected value using the usual averaging method or the iterative approach, but we have a problem! Our samples are not drawn from p! Instead, lets say they’re drawn from some other distribution I. It seems like we’re hosed, but Importance Sampling comes to our rescue.We begin by doing something ridiculous. Let us remark the following obvious equivalence.[math]\int_X p(x)f(x)dx = \int_X \frac{I(x)}{I(x)} p(x)f(x)dx[/math]Of course this is true. Multiplying something and dividing by that same something does nothing because you cancel it out immediately.Lets rewrite the right hand side a bit more, this time by replacing some of the elements with a function g.[math]\int_X \frac{I(x)}{I(x)} p(x)f(x)dx = \int_X I(x)g(x)dx[/math]where[math]g(x) = \frac{1}{I(x)}p(x)f(x)[/math]Okay, we didn’t’ do anything right? We just put some of those terms into that function g. But look at the form of it now. It looks like we’re computing the expected value of g(x) from distribution I. That is,[math]E_{x\sim p}\left[ f(x) \right] = E_{x\sim I}\left[g(x) \right] = \int_X I(x)g(x)dx[/math]Consequently, we can apply usual Monte Carlo estimates:[math]E_{x\sim p}\left[ f(x) \right] = E_{x\sim I}\left[g(x) \right] \approx \frac{1}{n}\sum_i^n g_i[/math]Where if we expand g, we get:[math]E_{x\sim p}\left[ f(x) \right] = E_{x\sim I}\left[g(x) \right] \approx \frac{1}{n}\sum_i^n \frac{p(x_i)}{I(x_i)}f(x_i)[/math]So that is how we get to importance sampling. Armed with these expression, you’ll find you can similarly modify the iterative procedure as well.Q-learningOkay, so lets briefly review. We use importance sampling when we want to estimate an expected value of some function over a specific probability distribution (p), but the samples that we have are drawn from some other probability distribution (I).In RL, the policy you learn induces a probability distribution. In off-policy learning, the policy followed in the environment that generates the samples may be different than the policy being learned.So, if the RL algorithm requires an expected value over the policy distribution and we’re merely sampling from that distribution, you may need to incorporate importance sampling.In Q-learning, that latter part is not required. The learning objective does not require estimating an expected value over the policy distribution by sampling.To see why, lets start with the Bellman equation:[math]Q(s, a) = R(s, a) [/math][math][/math][math]+ \gamma \sum_{s'} T(s' [/math][math][/math][math]| [/math][math]s[/math][math], a)\max_{a'}Q(s', a')[/math]Where T(s’ | s, a) is the transition function which defines the probability of the environment transitioning to state s’ after taking action a in state s.Note then, that part of this equation is just asking for expected value over the probability distribution environment transitions. We could re-rewrite it as so to make that point more clear:[math]Q(s, a) = R(s, a) [/math][math][/math][math]+ \gamma E_{s' \sim T(\cdot [/math][math][/math][math]| [/math][math]s[/math][math], a)} \max_{a'}Q(s', a')[/math]In the original form with the sum, were just giving the exact analytic version of that expected value.Following the Value Iteration algorithm, we can compute these values in an iterative form by starting with arbitrary values for Q, and updating them by applying the Bellman equation everywhere.The trouble with the Bellman equation, however, is that it requires knowing the environment transition probabilities. But in RL, we do not know ahead of time how the environment works. But we can take actions to generate samples.Enter Q-learning: it’s just using Value Iteration in which we start with arbitrary estimates of Q and iteratively apply the Bellman equation, except instead of an analytic calculation of the expected value over the transition dynamics, we use samples from the environment and use the iterative expected value estimation approach I described earlier:[math]Q(s, a) = Q(s, a) [/math][math][/math][math]+ \alpha \left[r [/math][math][/math][math]+ \gamma \max_{a'}Q(s', a') - Q(s, a) \right][/math]where s’ is a sample from the environment.So, the important property here is the sampling Q-learning does is over the probability distribution of the environment state transitions, not over the policy distribution. Consequently, we don’t need to correct for different policy distributions!Bonus NotesIn regular Q-learning (which DQN follows), you estimate the expected value of the next state using the Q-function. But there are variants of Q-learning that will incorporate a multiple step reward return from some trajectory. In these cases, importance sampling can reappear, because now you do have a function (the return) which is sampled over the policy (and environment) distribution.It’s also worth noting that function approximation with Q-learning has some other critical challenges that has to do with the interaction of “bootstrapping” the Q-value estimates with function approximation that may result in results diverging. Interestingly, approaches to create more theoretically sound algorithms will often employ importance sampling on a different part of the problem. Understanding this subtly, however, is not important to understand why DQN, unlike other RL approaches, does not use importance sampling. The main reason for that difference is the reason I gave: because Q-learning does not make expected value estimates over the policy distribution, whereas other RL algorithms do.

Different people will have different things to say about what rigorous mathematical proof looks like to begin with so take the following experiment with a grain of salt.Let us investigate one question here and present a series of arguments sorted (subjectively) by rigour in an ascending order - that is we shall move from less rigorous to more rigorous gradually.Question: given a 2-space, unbounded, square lattice, is it possible to construct on it an equilateral triangle whose vertices are located at the nodes of the lattice?Take 1. Nah, not really. Look. Say, we take the side length of the proposed equilateral triangle to be [math]n[/math] integral units long. Then the height of such a triangle must be an irrational number:[math]\dfrac{n\sqrt{3}}{2} \tag*{}[/math]So if we place the base of the target triangle, say, horizontally, then the height of such a triangle must emanate from a node whose vertical coordinate, by definition, is an integer (perhaps even a natural number) - contradiction.Take 2. Nah. Place one vertex of the target triangle at a lattice’s node and claim that node to be the origin of an arbitrarily chosen rectangular coordinate system where the distance between any two immediate horizontal and vertical neighboring nodes is equal to unity.Place the second node of the target triangle along the system’s [math]x[/math]-axis. Then, one of the triangles’ sides and, hence, one of the triangle’s vertices must belong to a straight line [math]l[/math] with a certain, easily computable, slope:But that slope must be an irrational number which is clearly impossible by the very definition of an irrational number since such a number can not be expressed in the form of:[math]\dfrac{p}{q}, [/math][math][/math][math]\; p,q\in\mathbb{Z} \tag*{}[/math]Contradiction.Take 3. Nope. Assume for a moment that we somehow did manage to construct the required triangle.Surely, it is always possible to submerge such a triangle into a bounding rectangle whose side lengths are natural numbers and, therefore, whose area is a natural number also since natural numbers are closed under multiplication:But look at the highlighted (bounding) right triangles - their side lengths are also natural numbers, their areas are, in general, rational numbers and, therefore, the area of the target triangle must also be a rational number because rational numbers are closed under subtraction.But the area of an equilateral triangle with the side length of [math]a[/math] is:[math]a^2\cdot\dfrac{\sqrt{3}}{4} \tag*{}[/math]Wait a second.The side of the target triangle is actually a hypotenuse of the corresponding bounding right triangle with the integral side lengths of, say, [math]p[/math] and [math]q[/math]. Therefore by Euclid’s Book 1 Proposition 47, also known as the Pythagorean Theorem:[math]a^2 = p^2+q^2 \tag*{}[/math]Since [math]p[/math] and [math]q[/math] are natural numbers, their squares are also natural numbers (closure under multiplication). The sum of two natural numbers is also a natural number (closure under addition). But that would mean that [math]a^2[/math] must also be a natural number. Contradiction - we have a rational number on one side equal to a product of a rational number:[math]\dfrac{a^2}{4} \tag*{}[/math]and an irrational number, namely [math]\sqrt{3}[/math], on the other. Symbolically (for [math]a^2,b,c,d\in\mathbb{Z}[/math]):[math]\dfrac{c}{d} = \dfrac{a^2}{b}\cdot\sqrt{3} \tag*{}[/math]which is an impossibility.Take 4. No. Let the target equilateral triangle be placed as shown below:Then, the side length of the target triangle from the gray right triangle is:[math]\sqrt{p^2+q^2} \tag{1}[/math]The same side length from the blue right triangle [math]PAR[/math] is:[math]\sqrt{(r-p)^2+(q-s)^2} \tag{2}[/math]and the same side length from the green right triangle is:[math]\sqrt{r^2+s^2} \tag{3}[/math]where all the variables in this proof are taken to be natural numbers unless noted otherwise.The magnitudes captured in (1,2,3) are, by definition, the same:[math]p^2+q^2 = r^2+s^2 = (r-p)^2+(q-s)^2 \tag{4}[/math]and we can safely assume that all the integers [math]p,q,r,s[/math] are relatively prime since otherwise we could have simply cancelled out the square of their common factor in (4) and arrived at a new set of relatively prime integers which we rename accordingly.Therefore, it is not the case that all four integers [math]p,q,r,s[/math] are even.Therefore, the only possible mix of parities of the four integers is:three even/one oddtwo even/two oddone even/three oddzero even/four oddAs such, in theory, we have permutations of multisets with a finite supply of items. Designating even integers with [math]e[/math] and odd integers with [math]o[/math], we have, in a multiset notation, the following cases:[math]\{3*e, 1*o\} \tag{5}[/math][math]\{2*e, 2*o\} \tag{6}[/math][math]\{1*e, 3*o\} \tag{7}[/math][math]\{0*e, 4*o\} \tag{8}[/math]The number of possible permutations in this case is given by the multinomial coefficients and in the cases (5) and (7) we have:[math]\dfrac{4!}{3!\cdot 1!} = \dfrac{4!}{1!\cdot 3!} = 4 \tag*{}[/math]permutations, and in the case (6) we have:[math]\dfrac{4!}{2!\cdot 2!} = 6 \tag*{}[/math]permutations and in the last case we have only one possible permutation of items.It may be argued that a rigorous proof has to exhaust all these permutations where by one permutation we mean a particular distribution of parities over the four integers: say, [math]p,q,r[/math] are even and [math]s[/math][math][/math] is odd.We, however, are armed with[math]p^2+q^2 = r^2+s^2 \tag{9}[/math]from where it follows that the cases (5) and (7) must be rejected right away and, in theory, we must prove that - you’ve asked for it. We state that a parity of an integer is preserved over squaring: squares of even integers are even while squares of odd integers are odd. As an exercise you should now show that no matter the permutation of [math]e[/math]s and [math]o[/math]s over (9) the resultant parities will not agree. You may use [math]o+o=e[/math] and [math]e+e=e[/math] notation or a rigorous one. Example:[math]e+e = e \neq o = e+o \tag*{}[/math]Now we have only two scenarios to cover, namely (6) and (8).Consider (6): two integers are even and two are odd. Again, we do not exhaust all the possible permutations - only one as a sample.We shall show that it is never the case that [math]p[/math] and [math]q[/math] are both even - by contradiction. For assume that they are:[math]p=2p_1, [/math][math][/math][math]\; q=2q_1 \tag*{}[/math]but then necessarily:[math]r[/math][math] = 2r_1+1, [/math][math][/math][math]\; [/math][math]s[/math][math] = 2s_1+1 \tag*{}[/math]Putting these numbers into (9), we get:[math]4(p_1^2+q_1^2) = 4(r^2_1+s^2_1+r_1+s_1)+2 \tag*{}[/math]But clearly:[math](4(p_1^2+q_1^2)) \bmod 4 = 0 \tag*{}[/math]while[math](4(r^2_1+s^2_1+r_1+s_1)+2) \bmod 4 \neq 0 \tag*{}[/math]Contradiction. Therefore, the parities of the coordinates of the vertices [math]P[/math] and [math]Q[/math] must be opposite (per pair). Let us take it that:[math]p = 2p_1+1, [/math][math][/math][math]\; q = 2q_1 \tag*{}[/math][math]r[/math][math] = 2r_1, [/math][math][/math][math]\; [/math][math]s[/math][math] = 2s_1+1 \tag*{}[/math](but for a rigorous proof you should examine other permutations as well)From (4) then it follows that:[math]4(p_1^2+p_1+q_1)+1 = \tag*{}[/math][math]4(r_1^2+s_1^2 [/math][math][/math][math]+ s_1)+1 = \tag*{}[/math][math]4(r_1-p_1)^2+4(r_1-p_1)+4(q_1-s_1)^2-4(q_1-s_1)+2 \tag*{}[/math]and we have a disagreement in parities: the first two expressions above represent an odd integer while the last expression represents an even integer (and you should really spell that out).Hence, the case (6) is impossible.Now consider the last scenario, (8):[math]p = 2p_1+1, [/math][math][/math][math]\; q = 2q_1+1 \tag*{}[/math][math]r[/math][math] = 2r_1+1, [/math][math][/math][math]\; [/math][math]s[/math][math] = 2s_1+1 \tag*{}[/math]where from (9) we have:[math]4(p^2_1+q^2_1+p_1+q_1)+2 = \tag*{}[/math][math]4(r^2_1+s^2_1+r_1+s_1)+2 = \tag*{}[/math][math]4(r_1-p_1)^2+4(q_1-s_1)^2 \tag{10}[/math]which is a contradiction since the last expression in (10) is divisible by [math]4[/math] while the first two are not (and you should really use the [math]\bmod[/math] notation to spell that out also) and we are, finally, done![math]\blacksquare[/math]Extra for experts.It turns out that in 2-space the only regular [math]n[/math]-gon that can be constructed as required is just a square - no regular pentagons, hexagons, etc. For the proof of that statement ask a separate question.In 3-space, however, on an unbounded crystalline lattice of unit cubes it is possible to construct an equilateral triangle (whose vertices are located at the nodes of the cubic lattice).