File X 26 1/1: Fill & Download for Free

In addition to making awesome charts, the quantmod library offers a range of very helpful tools for using R code in finance. I use it so much, I’m not even sure I could do my job without it at this point.Also in the toolkit is PerformanceAnalytics (for which I did a quick intro on ouR Space), Rblpapi, the Bloomberg API for R.Rather than dig into all the various packages, let me give you a quick tour of the most powerful one, quantmodDATA GATHERINGBy far, the most powerful aspect of quantmod is its data-gathering ability. This is driven by the getSymbols() command, and can pull data from Yahoo Finance as well as the St. Louis Federal Reserve (FRED) database.For example, suppose you want to analyze the relationship between unemployment and price movements of the S&P 500. You can grab those to data structures using two lines of code:getSymbols('^GSPC') getSymbols('UNRATE', src='FRED') Of course, R is specifically built for statistical analysis, so it is that ability which so easily facilitates analysis. In fact, for econometric analysis, you can pull several data structures at once from the same database in the same command using the list operator c():getSymbols( c('UNRATE', 'ECBASSETS', 'T10Y3M'), src='FRED') This command pulled in the US unemployment rate, assets on the ECB balance sheet, and the 10-year minus 3-month US Treasury yield.You can also grab option chains and commodity prices:getOptionChain('IVV') yields$calls  Strike Last Chg Bid Ask Vol OI IVV181005C00285000 285.0 10.08 0.0 5.80 7.80 1 1 IVV181005C00290000 290.0 4.25 0.0 2.35 4.10 5 0 IVV181005C00292000 292.0 2.20 0.0 1.85 2.05 5 0 IVV181005C00292500 292.5 1.37 0.0 1.30 1.75 5 0 IVV181005C00294000 294.0 1.40 0.0 0.00 0.25 1 1 IVV181005C00295000 295.0 0.05 -0.7 0.00 0.15 1 119 IVV181005C00296000 296.0 0.05 0.0 0.00 0.10 2 8 IVV181005C00297000 297.0 0.14 0.0 0.05 0.30 2 3 IVV181005C00297500 297.5 0.29 0.0 0.00 0.20 3 14 IVV181005C00300000 300.0 0.75 0.0 0.40 1.40 1 1   $puts  Strike Last Chg Bid Ask Vol OI IVV181005P00275000 275.0 0.25 0 0.00 0.25 4 10 IVV181005P00282000 282.0 0.35 0 0.10 0.30 5 5 IVV181005P00285000 285.0 1.70 0 1.60 1.95 1 1 IVV181005P00289000 289.0 0.01 0 0.25 0.55 1 0 IVV181005P00290000 290.0 0.31 0 0.35 0.60 1 1 IVV181005P00292000 292.0 1.15 0 0.95 1.20 1 0 IVV181005P00292500 292.5 0.46 0 1.25 1.50 1 1 IVV181005P00293000 293.0 0.65 0 1.55 1.80 4 0 IVV181005P00294000 294.0 2.15 0 1.75 2.10 10 10 IVV181005P00296000 296.0 3.00 0 2.25 4.00 10 10 DATA PROCESSINGProbably my most-used functions are Delt(), Next(), SMA(), and diff(). Of course, diff() is Base R—which is why R is so powerful for what we do in finance—and Next() is basically the same as lag() (which is also Base R). Let’s walk through each.Notice that the data structure we downloaded for IVV is OHLC prices over time. Most of the time, we need to get return data, rather than raw price data, which is where the Delt() command comes in:ivv = Delt(IVV$IVV.Adjusted, k=20) In the code above, we are telling the function to grab the Adjusted price column, and then look at a rolling 20-period change. In other words, because this data is in daily format, when k=20, you are generating a rolling 20-day return for the Adjusted closing price.From here, we can do all sorts of things! Want a correlation table with a sexy visualization?ivv = Delt(IVV$IVV.Adjusted, k=20) # Convert price data to 20-day returns aapl = Delt(AAPL$AAPL.Adjusted, k=20) agg = Delt(AGG$AGG.Adjusted, k=20)   returns = data.frame(ivv, aapl, agg) # Put return data into a data frame colnames(returns) = c("IVV", "AAPL", "AGG") # Name the columns   cor_table = cor( na.omit(returns) ) # Generate a correlation table, but drop "NA" data   library(corrplot) # Load the corrplot library for visualizing correlation corrplot( cor_table ) # Generate the default correlation plot yields: Or, if you’d rather a bit more detail/bougie-ness,corrplot( cor_table, # Variable to plot  method="color", # Fills in boxes with relative colors  type="upper", # Returns only upper-half of correlation table  addCoef.col="black", # Prints the correlation number in the box in black  tl.col="black", # Changes x/y labels to black  tl.srt=45 # Turns top x labels 45 degrees ) When analyzing data in percentage form, I’ll need the change in percentage points relative to a previous figure. That is where diff() is useful. For example, let’s look at the yield curve movement from a year ago to today.getSymbols(   c('DGS1', 'DGS3', 'DGS5', 'DGS10', 'DGS20'), # US Treasury Yields at various maturities  src='FRED' )   change_1yr = diff( DGS1, 250 ) # Difference in yield from 250-days ago plot( change_1yr, type="l" ) # Plot the change abline( h = 0, col="red" ) # Add a red line at y = 0 And you can see how the 1-year US treasury yield has changed, in percentage points, over time. What about how the evolution of each point on the curve has affected the price change of AGG? Well, we can run a quick linear regression on this to find some answers. The hardest part is getting the data in a format we can readily use:getSymbols( c('DGS1', 'DGS3', 'DGS5', 'DGS10', 'DGS20'), src='FRED' )   # Find 20-day difference at each point on the yield curve dgs1 = diff( DGS1, k=20 )["2012::"] dgs1 = dgs1[xts:::startof(dgs1, "months")] # This converts to first value of each month   dgs3 = diff( DGS3, k=20 )["2012::"] dgs3 = dgs3[xts:::startof(dgs3, "months")]   dgs5 = diff( DGS5, k=20 )["2012::"] dgs5 = dgs5[xts:::startof(dgs5, "months")]   dgs10 = diff( DGS10, k=20 )["2012::"] dgs10 = dgs10[xts:::startof(dgs10, "months")]   dgs20 = diff( DGS20, k=20 )["2012::"] dgs20 = dgs20[xts:::startof(dgs20, "months")]   agg = Delt( AGG$AGG.Adjusted, k=20 )["2012::"] agg = agg[xts:::startof(agg, "months")]   data = data.frame( # Put into a data frame for easy access  dgs1,  dgs3,  dgs5,  dgs10,  dgs20,  agg ) colnames(data) = c("DGS1", "DGS3", "DGS5", "DGS10", "DGS20", "AGG") # Change column names for use in reg data = na.omit( data ) # Remove "NA"s   regression = lm( AGG ~ DGS1 + DGS3 + DGS5 + DGS10 + DGS20, data) # Run regression  We now have the regression stats:> summary(regression)   Call: lm(formula = AGG ~ DGS1 + DGS3 + DGS5 + DGS10 + DGS20, data = data)   Residuals:  Min 1Q Median 3Q Max  -0.027941 -0.004645 0.001518 0.004732 0.015993    Coefficients:  Estimate Std. Error t value Pr(>|t|)  (Intercept) 0.002040 0.001035 1.971 0.0531 . DGS1 0.042249 0.072034 0.587 0.5596  DGS3 -0.006899 0.099162 -0.070 0.9448  DGS5 0.080280 0.131954 0.608 0.5451  DGS10 -0.094172 0.148158 -0.636 0.5273  DGS20 -0.024033 0.094774 -0.254 0.8006  --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1   Residual standard error: 0.008054 on 63 degrees of freedom Multiple R-squared: 0.1356, Adjusted R-squared: 0.06698  F-statistic: 1.976 on 5 and 63 DF, p-value: 0.09436   >  Looks like this aspect of the yield curve is not very useful for predicting AGG 20-day returns!But, the point is, it took all of 3 minutes to get the data, wrangle the data, and regress the data. Just getting the data into Excel takes much longer than that! Not to mention, in python, getting up-to-date AGG price data is not even possible since the Yahoo Finance API stopped working there (it still works in R, for some unknown reason!). Indeed, unless you have a paid feed, getting data to python is not easy.TL;DRIn summary, quantmod combined with Base R (and maybe a few other libraries) is very powerful what what we do in finance. Indeed, in data gathering alone it is more powerful than python, and Excel doesn’t hold a candle!I would even recommend to those Excel ninjas out there that R can be used to push data to Excel quite easily using the write.csv() command. Using this, you can quickly download data and write it to a csv:library(quantmod) getSymbols("AAPL") write.table(   Delt(AAPL$APPL.Adjusted, k=20),   "C:/SAVE YOUR FILE HERE") At any rate, hopefully this is helpful, and I am open to any and all critiques and suggestions! I have only been using R for 18 months now, so there are plenty of places I can improve.—Code to generate the chart at the head of the post:library(quantmod) getSymbols('IVV') chartSeries(IVV['2018::']) addMACD(12, 26, 9,type='EMA', histogram=TRUE) addSMA(20, col='white') addRSI(n=14, maType='EMA', wilder=TRUE) addSMI(n=13, slow=25, fast=2, signal=9, ma.type='EMA') addBBands(n=20, sd=2, maType='SMA', draw='bands', on=1) 

The following tricks I find pretty useful in my daily Python work. I also added a few I stumbled upon lately.1. Use collectionsThis really makes your code more elegant and less verbose, a few examples I absorbed this week:Named tuples:>>> Point = collections.namedtuple('Point', ['x', 'y']) >>> p = Point(x=1.0, y=2.0) >>> p Point(x=1.0, y=2.0) Now you can index by keyword, much nicer than offset into tuple by number (less readable)>>> p.x 1.0 >>> p.y Elegantly used when looping through a csv:with open('stock.csv') as f:  f_csv = csv.reader(f)  headings = next(f_csv)  Row = namedtuple('Row', headings)  for r in f_csv:  row = Row(*r) # note the star extraction  # ... process row ... I like the unpacking star feature to throw away useless fields:line = 'nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false' >>> uname, *fields, homedir, sh = line.split(':') >>> uname 'nobody' >>> homedir '/var/empty' >>> sh '/usr/bin/false' Superconvenient: the defaultdict:from collections import defaultdict rows_by_date = defaultdict(list) for row in rows:  rows_by_date[row['date']].append(row)", Before I would init the list each time which leads to needless code:if row['date'] not in rows_by_date: rows_by_date[row['date']] = [] You can use OrderedDict to leave the order of inserted keys:>>> import collections >>> d = collections.OrderedDict() >>> d['a'] = 'A' >>> d['b'] = 'B' >>> d['c'] = 'C' >>> d['d'] = 'D' >>> d['e'] = 'E' >>> for k, v in d.items(): ... print k, v ...  a A b B c C d D e E Another nice one is Counter:from collections import Counterwords = [  'look', 'into', 'my', 'eyes', 'look', 'into', 'my', 'eyes',  'the', 'eyes', 'the', 'eyes', 'the', 'eyes', 'not', 'around', 'the',  'eyes', ""don't"", 'look', 'around', 'the', 'eyes', 'look', 'into',  'my', 'eyes', ""you're"", 'under' ] word_counts = Counter(words) top_three = word_counts.most_common(3) print(top_three) # Outputs [('eyes', 8), ('the', 5), ('look', 4)]", Again, before I would write most_common manually. Not necessary, this is all done already somewhere in the stdlib :)2. sorted() accepts a key arg which you can use to sort on something elseHere for example we sort on surname:>>> sorted(names, key=lambda name: name.split()[-1].lower()) ['Ned Batchelder', 'David Beazley', 'Raymond Hettinger', 'Brian Jones'] 3. Create XMl from dictCreating XML tags manually is usually a bad idea, I bookmarked this simple dict_to_xml helper:from xml.etree.ElementTree import Element def dict_to_xml(tag, d):  '''  Turn a simple dict of key/value pairs into XML  '''  elem = Element(tag)  for key, val in d.items():  child = Element(key)  child.text = str(val)  elem.append(child)  return elem" 4. Oneliner to see if there are any python files in a particular directorySometimes ‘any’ is pretty useful:import os files = os.listdir('dirname') if any(name.endswith('.py') for name in files): 5. Use set operations to match common items in lists >>> a = [1, 2, 3, 'a'] >>> b = ['a', 'b', 'c', 3, 4, 5] >>> set(a).intersection(b) {3, 'a'} 6. Use re.compileIf you are going to check a regular expression in a loop, don’t do this:for i in longlist:  if re.match(r'^...', i) yet define the regex once and use the pattern: p = re.compile(r'^...') for i in longlist:   if p.match(i) 7. Printing files with potential bad (Unicode) charactersThe book suggested to print filenames of unknown origin, use this convention to avoid errors:def bad_filename(filename):  return repr(filename)[1:-1] try:  print(filename) except UnicodeEncodeError:  print(bad_filename(filename)) Handling unicode chars in files can be nasty because they can blow up your script. However the logic behind it is not that hard to grasp. A good snippet to bookmark is the encoding / decoding of Unicode:>>> a 'pýtĥöñ is awesome\n' >>> b = unicodedata.normalize('NFD', a) >>> b.encode('ascii', 'ignore').decode('ascii') 'python is awesome\n' O’Reilly has a course on Working with Unicode in Python.8. Print is pretty cool (Python 3)I am probably not the only one writing this kind of join operations:>>> row = ["1", "bob", "developer", "python"] >>> print(','.join(str(x) for x in row)) 1,bob,developer,python Turns out you can just write it like this:>>> print(*row, sep=',') 1,bob,developer,python Note again the * unpacking. 9. Functions like sum() accept generators / use the right variable typeI wrote this at a conference to earn me a coffee mug ;)sum = 0 for i in range(1300):  if i % 3 == 0 or i % 5 == 0:  sum += i print(sum) Returns 394118, while handing it in I realized this could be written much shorter and efficiently:>>> sum(i for i in range(1300) if i % 3 == 0 or i % 5 == 0) 394118 A generator:lines = (line.strip() for line in f) is more memory efficient than:lines = [line.strip() for line in f] # loads whole list into memory at once And concatenating strings is inefficient:s = "line1\n" s += "line2\n" s += "line3\n" print(s) Better build up a list and join when printing:lines = [] lines.append("line1") lines.append("line2") lines.append("line3") print("\n".join(lines)) Another one I liked from the cookbook: portfolio = [  {'name':'GOOG', 'shares': 50},  {'name':'YHOO', 'shares': 75},  {'name':'AOL', 'shares': 20},  {'name':'SCOX', 'shares': 65} ] min_shares = min(s['shares'] for s in portfolio) One line to get the min of a numeric value in a nested data structure.10. Enumerate lines in for loopYou can number lines (or whatever you are looping over) and start with 1 (2nd arg), this is a nice debugging techniquefor lineno, line in enumerate(lines, 1): # start counting at 0  fields = line.split()  try:  count = int(fields[1])  ...  except ValueError as e:  print('Line {}: Parse error: {}'.format(lineno, e)) 11. PandasImport pandas and numpy:import pandas as pd import numpy as np 12. Make random dataframe with three columns:df = pd.DataFrame(np.random.rand(10,3), columns=list('ABC')) Select: # Boolean indexing (remember the parentheses) df[(df.A < 0.5) & (df.B > 0.5)] # Alternative, using query which depends on numexpr df.query('A < 0.5 & B > 0.5') Project: # One columns df.A # Multiple columns # there may be another shorter way, but I don't know it df.loc[:,list('AB')] Often used snippetsDates13. Difference (in days) between two dates:from datetime import dated1 = date(2013,1,1) d2 = date(2013,9,13) abs(d2-d1).days directory-of-script snippet os.path.dirname(os.path.realpath(__file__)) # combine with os.path.join(os.path.dirname(os.path.realpath(__file__)), 'foo','bar','baz.txt') 14. PostgreSQL-connect-query snippetimport psycopg2 conn = psycopg2.connect("host='localhost' user='xxx' password='yyy' dbname='zzz'") cur = conn.cursor() cur.execute("""SELECT * from foo;""") rows = cur.fetchall() for row in rows:  print " ", row[0] conn.close() Input parsing functions 15. Expand input-file args:# input_data: e.g. 'file.txt' or '*.txt' or 'foo/file.txt' 'bar/file.txt' filenames = [glob.glob(pathexpr) for pathexpr in input_data] filenames = [item for sublist in filenames for item in sublist] 15. Parse key-value pair strings like ‘x=42.0,y=1’:kvp = lambda elem,t,i: t(elem.split('=')[i]) parse_kvp_str = lambda args : dict([(kvp(elem,str,0), kvp(elem,float,1)) for elem in args.split(',')]) parse_kvp_str('x=42.0,y=1') Postgres database functions16. Upper case in Python (just for example):-- create extension plpythonu; CREATE OR REPLACE FUNCTION python_upper (  input text ) RETURNS text AS $$  return input.upper() $$ LANGUAGE plpythonu STRICT; 17. Convert IP address from text to integer:CREATE FUNCTION ip2int(input text) RETURNS integer LANGUAGE plpythonu AS $$ if 'struct' in SD:  struct = SD['struct'] else:  import struct  SD['struct'] = struct if 'socket' in SD:  socket = SD['socket'] else:  import socket  SD['socket'] = socket return struct.unpack("!I", socket.inet_aton(input))[0] $$; Convert IP address from integer to text: CREATE FUNCTION int2ip(input integer) RETURNS text LANGUAGE plpythonu AS $$ if 'struct' in SD:  struct = SD['struct'] else:  import struct  SD['struct'] = struct if 'socket' in SD:  socket = SD['socket'] else:  import socket  SD['socket'] = socket return socket.inet_ntoa(struct.pack("!I", input)) $$; 18. Commandline optionsoptparse-commandline-options snippet from optparse import OptionParser usage = "usage: %prog [options] arg " parser = OptionParser(usage=usage) parser.add_option("-x", "--some-option-x", dest="x", default=42.0, type="float",  help="a floating point option") (options, args) = parser.parse_args() print options.x print args[0] 19. print-in-place (progress bar) snippetimport time import sys for progress in range(100):  time.sleep(0.1)  sys.stdout.write("Download progress: %d%% \r" % (progress) )   sys.stdout.flush() Packaging snippets20. poor-mans-python-executable trickLearned this trick from voidspace. The trick uses two files (__main__.py and hashbang.txt):__main__.py: print 'Hello world' hashbang.txt (adding a newline after ‘python2.6’ is important): #!/usr/bin/env python2.6 Build an “executable”: zip main.zip __main__.py cat hashbang.txt main.zip > hello rm main.zip chmod u+x hello Run “executable”: $ ./hello Hello world 21. import-class-from-file trickImport class MyClass from a module file (adapted from stackoverflow):import imp mod = imp.load_source('name.of.module', 'path/to/module.py') obj = mod.MyClass() 22. Occusional-usage snippetsExtract words from stringwords = lambda text: ''.join(c if c.isalnum() else ' ' for c in text).split() words('Johnny.Appleseed!is:a*good&farmer') # ['Johnny', 'Appleseed', 'is', 'a', 'good', 'farmer'] 23. IP address to integer and backimport struct import socket def ip2int(addr):   return struct.unpack("!I", socket.inet_aton(addr))[0]  def int2ip(addr):   return socket.inet_ntoa(struct.pack("!I", addr)) 24. Fluent Python InterfaceCopied from riaanvddool.# Fluent Interface Definition class sql:  class select:  def __init__(self, dbcolumn, context=None):  self.dbcolumn = dbcolumn  self.context = context  def select(self, dbcolumn):  return self.__class__(dbcolumn,self) # Demo q = sql.select('foo').select('bar') print q.dbcolumn #bar print q.context.dbcolumn #foo Flatten a nested lists def flatten(elems):  """  [['a'], ['b','c',['d'],'e',['f','g']]]  """  stack = [elems]  top = stack.pop()  while top:   head, tail = top[0], top[1:]  if tail: stack.append(tail)  if not isinstance(head, list): yield head   else: stack.append(head)  if stack: top = stack.pop()  else: break snap rounding EPSILON = 0.000001 snap_ceil = lambda x: math.ceil(x) if abs(x - round(x)) > EPSILON else round(x) snap_floor = lambda x: math.floor(x) if abs(x - round(x)) > EPSILON else round(x) merge-two-dictionaries snippet x = {'a': 42} y = {'b': 127} z = dict(x.items() + y.items()) # z = {'a': 42, 'b': 127} 25. anonymous-object snippetAdapted from stackoverflow:class Anon(object):  def __new__(cls, **attrs):  result = object.__new__(cls)  result.__dict__ = attrs  return result 26. Alternative:class Anon(object):  def __init__(self, **kwargs):  self.__dict__.update(kwargs)  def __repr__(self):  return self.__str__()  def __str__(self):  return ", ".join(["%s=%s" % (key,value) for key,value in self.__dict__.items()]) 27. generate-random-word snippetFunction that returns a random word (could also use random.choicewith this list of words):import string, random randword = lambda n: "".join([random.choice(string.letters) for i in range(n)]) setdefault tricks Increment (and initialize) value: d = {} d[2] = d.setdefault(2,39) + 1 d[2] = d.setdefault(2,39) + 1 d[2] = d.setdefault(2,39) + 1 d[2] # value is 42 29. Append value to (possibly uninitialized) list stored under a key in dictionary:d = {} d.setdefault(2, []).append(42) d.setdefault(2, []).append(127) d[2] # value is [42, 127] Binary tricks30. add-integers-using-XOR snippetSwap two integer variables using the XOR swap algorithm:x = 42 y = 127 x = x ^ y y = y ^ x x = x ^ y x # value is 127 y # value is 42 I know that most of it has been mentioned already But I think you should find some new tricks as well.Greetings,

Friend, you're going to have a problem if these are strings, because strings need to be null terminated. To be able to hold the terminating null character, you're going to need char[26] instead of char[25].Try this for fun. Here's something that works:#include <stdio.h> #include <string.h>  int main() {  char str1[50];  char str2[26];   strcpy(str1, "abcdefghijklmnopqrstuvwxyABCDEFGHIJKLMNOPQRSTUVWX\0");  memset(str2, 0, 26);  strncpy(str2, str1, 25);   puts(str2);  return 0; } Now, comment out the memset, and change char str[26] to char str[25] and see what happens. (You'll get junk at the end of the output. That's because the null terminator isn't there.)When you change the size back up to 26, compile and run again, you'll still see junk at the end of the string. Uncomment the memset, compile and run again and you're golden.Try changing the string back down to 25, and changing the 26 in the memset to 25. Recompile and run again. You'll get junk on the end of the string again because you've overwritten the nulls we put in with memset.Another way of doing it (you might look at this after you've got char arrays down).#include <stdio.h> #include <stdlib.h> #include <string.h>  int main() {  const char *str1 = "abcdefghijklmnopqrstuvwxyABCDEFGHIJKLMNOPQRSTUVWXYZ\0";  char *str2 = calloc(26, 1);   strncpy(str2, str1, 25);   puts(str2);  free(str2);  return 0; } Notice I added another header file, stdlib.h. The calloc function allocates memory of (in this case) 26 units or 1 byte each, and initializes it to 0 itself so we don't need memset. We need one other thing, though, we should free str2 at the end to avoid memory leaks when we're done with str2.Hope this helps!