03 05 01: Fill & Download for Free

Without a doubt, my own.The worst piece of programming I ever did, which shames me to this day, was when a friend asked me to put together a countdown clock for the London 2012 Olympic committee. The announcement was due to be made whether London had won the bid, they were having a party, and they wanted a countdown on the wall.It wasn’t exactly a difficult job, but I put some work in to make it look nice. But I forgot one rather crucial detail. And so the next day I discovered people had watched the clock and chanted along as it counted down…00:0500:0400:0300:0200:0100:0000:-0100:-02…My cheeks burn every time I think of it.

I am happy to share about Pandas Time Series data analysis and I hope someone else will cover StatsModels. I use Pandas everyday, but I am not that familiar with StatsModels.Pandas Time Series DataThe Pandas package makes it easy to handle time series data (TS). Solving time series problems in Python is not so much fun sometimes. But, Pandas dedicated libraries contain many timeseries object optimizations. Thus, solving TS problems with Pandas is really quite simple.The Pandas library provides powerful tools to perform any data tasks. And, when it comes to time series data, Pandas is especially useful.The NumPy datatime64[ns] and timedelta64 dtypes definitely make my job a lot more fun.In this post, I will be covering 3 basic time series data manipulating functionalities.Date RangesDatetimeIndex & TimestampsResamplingNote: I am using Pandas version '0.19.1' for the code in these examples.Let’s go ahead and import the Pandas, NumPy and Matplotlib libraries:import pandas as pd import numpy as np import matplotlib.pyplot as plt %matplotlib inline plt.style.use('ggplot') import seaborn as sn pd.__version__ '0.19.1' Date RangesFirst, we will create a Datetimeindex with the frequency of one hour. The date range will be from January 1st. 2016 to the date of this post, November 30th. 2016.hours = pd.date_range('1/1/2016', '11/30/2016', freq='H') hours DatetimeIndex(['2016-01-01 00:00:00', '2016-01-01 01:00:00','2016-01-01 02:00:00', '2016-01-01 03:00:00','2016-01-01 04:00:00', '2016-01-01 05:00:00','2016-01-01 06:00:00', '2016-01-01 07:00:00','2016-01-01 08:00:00', '2016-01-01 09:00:00',...'2016-11-29 15:00:00', '2016-11-29 16:00:00','2016-11-29 17:00:00', '2016-11-29 18:00:00','2016-11-29 19:00:00', '2016-11-29 20:00:00','2016-11-29 21:00:00', '2016-11-29 22:00:00','2016-11-29 23:00:00', '2016-11-30 00:00:00'],dtype='datetime64[ns]', length=8017, freq='H')Next, we construct a DataFrame. We assign the DatetimeIndex values, to one of the DataFrame columns. We will fill the other column with random integers ranging from 1 to 100.np.random.seed(seed=1111) data = np.random.randint(1, high=100, size=len(hours)) df = pd.DataFrame({'col1': hours, 'col2': data}) df.head(10)  # First 10 rows   col1 col2 0 2016-01-01 00:00:00 29 1 2016-01-01 01:00:00 56 2 2016-01-01 02:00:00 82 3 2016-01-01 03:00:00 13 4 2016-01-01 04:00:00 35 5 2016-01-01 05:00:00 53 6 2016-01-01 06:00:00 25 7 2016-01-01 07:00:00 23 8 2016-01-01 08:00:00 21 9 2016-01-01 09:00:00 12  df.tail(10)  # Last 10 rows   col1 col2 8007 2016-11-29 15:00:00 6 8008 2016-11-29 16:00:00 3 8009 2016-11-29 17:00:00 3 8010 2016-11-29 18:00:00 89 8011 2016-11-29 19:00:00 82 8012 2016-11-29 20:00:00 91 8013 2016-11-29 21:00:00 81 8014 2016-11-29 22:00:00 42 8015 2016-11-29 23:00:00 17 8016 2016-11-30 00:00:00 78 Here is a short summary of a DataFrame:df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 8017 entries, 0 to 8016 Data columns (total 2 columns): col1 8017 non-null datetime64[ns] col2 8017 non-null int32 dtypes: datetime64[ns](1), int32(1) memory usage: 94.0 KB We can select a time period between 2 dates from our DataFrame creating a new object selected_period:selected_period = df[(df['col1'] > '2016-04-30 23:00:00') & (df['col1'] < '2016-05-02 01:00:00')] selected_period We selected a 24 hour period. col1 col2 2904 2016-05-01 00:00:00 49 2905 2016-05-01 01:00:00 19 2906 2016-05-01 02:00:00 94 2907 2016-05-01 03:00:00 8 2908 2016-05-01 04:00:00 69 2909 2016-05-01 05:00:00 32 2910 2016-05-01 06:00:00 42 2911 2016-05-01 07:00:00 54 2912 2016-05-01 08:00:00 19 2913 2016-05-01 09:00:00 64 2914 2016-05-01 10:00:00 61 2915 2016-05-01 11:00:00 37 2916 2016-05-01 12:00:00 8 2917 2016-05-01 13:00:00 27 2918 2016-05-01 14:00:00 19 2919 2016-05-01 15:00:00 4 2920 2016-05-01 16:00:00 36 2921 2016-05-01 17:00:00 88 2922 2016-05-01 18:00:00 17 2923 2016-05-01 19:00:00 67 2924 2016-05-01 20:00:00 81 2925 2016-05-01 21:00:00 74 2926 2016-05-01 22:00:00 51 2927 2016-05-01 23:00:00 59 2928 2016-05-02 00:00:00 1 Selecting a period is simpler, if we set 'col1' as the index.df = df.set_index('col1') df.head()  col2 col1  2016-01-01 00:00:00 29 2016-01-01 01:00:00 56 2016-01-01 02:00:00 82 2016-01-01 03:00:00 13 2016-01-01 04:00:00 35 As we can see in the first 5 rows, the index is now a DatetimeIndex not a RangeIndex.df.info() <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 8017 entries, 2016-01-01 00:00:00 to 2016-11-30 00:00:00 Data columns (total 1 columns): col2 8017 non-null int32 dtypes: int32(1) memory usage: 93.9 KB After we set our DatetimeIndex as the index, we select the same 24 hour period. We will use a label-location based indexer, DataFrame.ix().We will create a slice object with labels '2016-05-01 00:00:00':'2016-05-02 00:00:00'. Unlike in python slices, both the start and the stop are included here!To select the same 24 hours period, the slicing '2016-05-01 00:00:00' : '2016-05-02 00:00:00' must be different here from our previous selection df[(df['col1'] > '2016-04-30 23:00:00') & (df['col1'] < '2016-05-02 01:00:00')].one_day = df.ix['2016-05-01 00:00:00':'2016-05-02 00:00:00'] one_day   col2 col1  2016-05-01 00:00:00 49 2016-05-01 01:00:00 19 2016-05-01 02:00:00 94 2016-05-01 03:00:00 8 2016-05-01 04:00:00 69 2016-05-01 05:00:00 32 2016-05-01 06:00:00 42 2016-05-01 07:00:00 54 2016-05-01 08:00:00 19 2016-05-01 09:00:00 64 2016-05-01 10:00:00 61 2016-05-01 11:00:00 37 2016-05-01 12:00:00 8 2016-05-01 13:00:00 27 2016-05-01 14:00:00 19 2016-05-01 15:00:00 4 2016-05-01 16:00:00 36 2016-05-01 17:00:00 88 2016-05-01 18:00:00 17 2016-05-01 19:00:00 67 2016-05-01 20:00:00 81 2016-05-01 21:00:00 74 2016-05-01 22:00:00 51 2016-05-01 23:00:00 59 2016-05-02 00:00:00 1 With Pandas, we can easily visualize the data.one_day.plot(title= 'One Day Selection Graph (2016-05-01)') Now, that we selected a 24 hour period, let's have some fun with this. So, what can we do with this data...Let's say, we want to check the hourly fluctuation of the values during the selected 24 hours. Pandas makes this task simple.We will use Pandas DataFrame.shift() to shift the index by 1 hour (1 row).one_day.shift(1)  col2 col1  2016-05-01 00:00:00 NaN 2016-05-01 01:00:00 49.0 2016-05-01 02:00:00 19.0 2016-05-01 03:00:00 94.0 2016-05-01 04:00:00 8.0 2016-05-01 05:00:00 69.0 2016-05-01 06:00:00 32.0 2016-05-01 07:00:00 42.0 2016-05-01 08:00:00 54.0 2016-05-01 09:00:00 19.0 2016-05-01 10:00:00 64.0 2016-05-01 11:00:00 61.0 2016-05-01 12:00:00 37.0 2016-05-01 13:00:00 8.0 2016-05-01 14:00:00 27.0 2016-05-01 15:00:00 19.0 2016-05-01 16:00:00 4.0 2016-05-01 17:00:00 36.0 2016-05-01 18:00:00 88.0 2016-05-01 19:00:00 17.0 2016-05-01 20:00:00 67.0 2016-05-01 21:00:00 81.0 2016-05-01 22:00:00 74.0 2016-05-01 23:00:00 51.0 2016-05-02 00:00:00 59.0 All we have to do now is to subtract the shifted data frame from the original one_day data. Then, we will be able get the rise and fall of the values every hour of the day.Positive values show the rise and negative values show the fall for each hour of the selected period.fluctuation = one_day - one_day.shift(1) fluctuation   col2 col1  2016-05-01 00:00:00 NaN 2016-05-01 01:00:00 -30.0 2016-05-01 02:00:00 75.0 2016-05-01 03:00:00 -86.0 2016-05-01 04:00:00 61.0 2016-05-01 05:00:00 -37.0 2016-05-01 06:00:00 10.0 2016-05-01 07:00:00 12.0 2016-05-01 08:00:00 -35.0 2016-05-01 09:00:00 45.0 2016-05-01 10:00:00 -3.0 2016-05-01 11:00:00 -24.0 2016-05-01 12:00:00 -29.0 2016-05-01 13:00:00 19.0 2016-05-01 14:00:00 -8.0 2016-05-01 15:00:00 -15.0 2016-05-01 16:00:00 32.0 2016-05-01 17:00:00 52.0 2016-05-01 18:00:00 -71.0 2016-05-01 19:00:00 50.0 2016-05-01 20:00:00 14.0 2016-05-01 21:00:00 -7.0 2016-05-01 22:00:00 -23.0 2016-05-01 23:00:00 8.0 2016-05-02 00:00:00 -58.0 This is how it looks like when the original values and the fluctuation are side by side.fluctuation_plot = pd.DataFrame({'col2': one_day['col2'], 'Fluctuation': fluctuation['col2'] }) fluctuation_plot   Fluctuation col2 col1  2016-05-01 00:00:00 NaN 49 2016-05-01 01:00:00 -30.0 19 2016-05-01 02:00:00 75.0 94 2016-05-01 03:00:00 -86.0 8 2016-05-01 04:00:00 61.0 69 2016-05-01 05:00:00 -37.0 32 2016-05-01 06:00:00 10.0 42 2016-05-01 07:00:00 12.0 54 2016-05-01 08:00:00 -35.0 19 2016-05-01 09:00:00 45.0 64 2016-05-01 10:00:00 -3.0 61 2016-05-01 11:00:00 -24.0 37 2016-05-01 12:00:00 -29.0 8 2016-05-01 13:00:00 19.0 27 2016-05-01 14:00:00 -8.0 19 2016-05-01 15:00:00 -15.0 4 2016-05-01 16:00:00 32.0 36 2016-05-01 17:00:00 52.0 88 2016-05-01 18:00:00 -71.0 17 2016-05-01 19:00:00 50.0 67 2016-05-01 20:00:00 14.0 81 2016-05-01 21:00:00 -7.0 74 2016-05-01 22:00:00 -23.0 51 2016-05-01 23:00:00 8.0 59 2016-05-02 00:00:00 -58.0 1 We can also easily visualize the values and the fluctuation throughout the selected day.fluctuation_plot.plot(title= 'Data and Fluctuation on 2016-05-01') DatetimeIndex & TimestampsAnother type of timeseries data that's worth mentioning is time-stamped data.Creating a Timestamp couldn't be simpler.pd.Timestamp('2016-11-30') Timestamp('2016-11-30 00:00:00') Or,pd.Timestamp(2016, 11, 29) Timestamp('2016-11-29 00:00:00') To convert a Series to DatetimeIndex, we use the pandas.to_datetime() function. In the DataFrame below, the Date Time column contains 'date-like' values. But, when we check the type (.dtypes), we can see that the data type is object not datetime64[ns].d = pd.DataFrame({'Date Time': ['1/26/2016 07:03', '1/26/2016 07:00', '1/26/2016 14:46', '1/26/2016 17:46', '1/26/2016 14:48'], 'ColA': ['a','b','c','d','e']}) d  ColA Date Time 0 a 1/26/2016 07:03 1 b 1/26/2016 07:00 2 c 1/26/2016 14:46 3 d 1/26/2016 17:46 4 e 1/26/2016 14:48 d.dtypes ColA object Date Time object dtype: object All we will have to do to convert this object to datetime64[ns] is to use the to_datetime() function.d['Date Time'] = pd.to_datetime(d['Date Time'])  d   ColA Date Time 0 a 2016-01-26 07:03:00 1 b 2016-01-26 07:00:00 2 c 2016-01-26 14:46:00 3 d 2016-01-26 17:46:00 4 e 2016-01-26 14:48:00  d.dtypes  ColA object Date Time datetime64[ns] dtype: object The Date Time column values are now converted to datetime64[ns]. Yes, it is that simple. This is why we love Pandas.ResamplingResampling is a powerful functionality used for frequency conversion. We will apply this function to our one_day which we selected earlier for the 24 hour period.Remember, the frequency of our original data is one hour. We will apply DataFrame.resample() to convert the hourly frequency to a 6 hour frequency. Then, we will check the mean and sum for each 6 hour period within the selected 24 hours.Again, this is another situation where we only need to use a short, simple 'one-liner' code.Under the hood, Pandas .resample() is a time-based groupby. And, a reduction method is applied on each of the groups. Resampling comes in handy in financial applications but it can be useful in many other areas as well.The means for each 6 hour period:one_day.resample('6H').mean()[:-1]   col2 col1  2016-05-01 00:00:00 45.166667 2016-05-01 06:00:00 46.166667 2016-05-01 12:00:00 30.333333 2016-05-01 18:00:00 58.166667 We can easily plot this:one_day.resample('6H').mean()[:-1].plot() We can also check the sum of the values in each 6 hour period.one_day.resample('6H').sum()[:-1]   col2 col1  2016-05-01 00:00:00 271 2016-05-01 06:00:00 277 2016-05-01 12:00:00 182 2016-05-01 18:00:00 349  one_day.resample('6H').sum()[:-1].plot() There are, of course, many advanced Pandas TS functions. I just wanted to present a quick glimpse into the basics of Pandas TS with these simple examples.If you’re interested in playing around with this code, you can download the iPython Notebook from Github.

Q: Do I know I can't die? Assuming I don't know, it would take me much longer to get bored.00:00:01–00:00:03: adrenaline rush, immobility (freezing up)00:00:04–00:00:15 complete panic and fear, disorientation, trying to assess physical relations00:00:16–00:1:00 flailing, trying to grasp at anything nearby, desperately trying to think about ramifications of situation00:01:00–00:01:30 realization that I haven't hit the ground yet, resolution to calm self enough to count seconds to verify sense of time.00:01:31–00:04:30 struggle to count seconds overwhelmed by intervention of racing thoughts.00:04:31–00:04:59 realize that so much time must have passed while trying to count that counting is no longer necessary.00:05:00–00:05:05 Bafflement, mental review of possible reasons why I am still falling.00:05:06–12:00:00 intense thought about reasons and explanations for situation, vacillating between dream scenario, drugs, mind control, etc. Interspersed with sudden re-realization that I am falling, existential horror at inexplicability, wondering how long it will continue, dreading likelihood that it will continue forever.12:00:01 - Forever: Continue theorizing about situation, existence, etc. Attempting to sleep/get comfortable. Disassociation from inability to change current situation. Attempting to distract myself with thought experiments, mental games, etc. to calm down. Panicked boredom when attempts to entertain self are futile. Contemplating suicide, wondering if I could even successfully commit suicide in present situation.Tl;dr Please don't throw me down an endless hole. Or at least give me some pencil and paper first.