Monthly Calendar Template For Excel: Fill & Download for Free

Power BI Interview Questions – General Questions1). What are the parts of Microsoft self-service business intelligence solution?Ans: Microsoft has two parts for Self-Service BIParts of Self-Service BIExcel BI ToolkitIt allows users to create an interactive report by importing data from different sources and model data according to report requirement.Power BIIt is the online solution that enables you to share the interactive reports and queries that you have created using the Excel BI Toolkit.2). What is self-service business intelligence?Ans: Self-Service Business Intelligence (SSBI)SSBI is an approach to data analytics that enables business users to filter, segment, and, analyze their data, without the in-depth technical knowledge in statistical analysis, business intelligence (BI).SSBI has made it easier for end users to access their data and create various visuals to get better business insights.Anybody who has a basic understanding of the data can create reports to build intuitive and shareable dashboards.3). What is Power BI?Ans: Power BI is a cloud-based data sharing environment. Once you have developed reports using Power Query, Power Pivot and Power View, you can share your insights with your colleagues. This is where Power BI enters the equation. Power BI, which technically is an aspect of SharePoint online, lets you load Excel workbooks into the cloud and share them with a chosen group of co-workers. Not only that, but your colleagues can interact with your reports to apply filters and slicers to highlight data. They are completed by Power BI, a simple way of sharing your analysis and insights from the Microsoft cloud.Power BI features allow you to:Share presentations and queries with your colleagues.Update your Excel file from data sources that can be on-site or in the cloud.Display the output on multiple devices. This includes PCs, tablets, and HTML 5-enabled mobile devices that use the Power BI app.Query your data using natural language processing (or Q&A, as it is known).4). What is Power BI Desktop?Ans: Power BI Desktop is a free desktop application that can be installed right on your own computer. Power BI Desktop works cohesively with the Power BI service by providing advanced data exploration, shaping, modeling, and creating report with highly interactive visualizations. You can save your work to a file or publish your data and reports right to your Power BI site to share with others.5). What data sources can Power BI connect to?Ans: The list of data sources for Power BI is extensive, but it can be grouped into the following:Files: Data can be imported from Excel (.xlsx, xlxm), Power BI Desktop files (.pbix) and Comma Separated Value (.csv).Content Packs: It is a collection of related documents or files that are stored as a group. In Power BI, there are two types of content packs, firstly those from services providers like Google Analytics, Marketo or Salesforce and secondly those created and shared by other users in your organization.Connectors to databases and other datasets such as Azure SQL, Databaseand SQL, Server Analysis Services tabular data, etc.6). What are Building Blocks in Power BI?Ans: The following are the Building Blocks (or) key components of Power BI:Visualizations: Visualization is a visual representation of data.Example: Pie Chart, Line Graph, Side by Side Bar Charts, Graphical Presentation of the source data on top of Geographical Map, Tree Map, etc.Datasets: Dataset is a collection of data that Power BI uses to create its visualizations.Example: Excel sheets, Oracle or SQL server tables.Reports: Report is a collection of visualizations that appear together on one or more pages.Example: Sales by Country, State, City Report, Logistic Performance report, Profit by Products report etc.Dashboards: Dashboard is single layer presentation of multiple visualizations, i.e we can integrate one or more visualizations into one page layer.Example: Sales dashboard can have pie charts, geographical maps and bar charts.Tiles: Tile is a single visualization in a report or on a dashboard.Example: Pie Chart in Dashboard or Report.7). What are the different types of filters in Power BI Reports?Ans: Power BI provides variety of option to filter report, data and visualization. The following are the list of Filter types.Visual-level Filters: These filters work on only an individual visualization, reducing the amount of data that the visualization can see. Moreover, visual-level filters can filter both data and calculations.Page-level Filters: These filters work at the report-page level. Different pages in the same report can have different page-level filters.Report-level Filters: There filters work on the entire report, filtering all pages and visualizations included in the report.We know that Power BI visual have interactions feature, which makes filtering a report a breeze. Visual interactions are useful, but they come with some limitations:The filter is not saved as part of the report. Whenever you open a report, you can begin to play with visual filters but there is no way to store the filter in the saved report.The filter is always visible. Sometimes you want a filter for the entire report, but you do not want any visual indication of the filter being applied.8). What are content packs in Power BI?Ans: Content packs for services are pre-built solutions for popular services as part of the Power BI experience. A subscriber to a supported service, can quickly connect to their account from Power BI to see their data through live dashboards and interactive reports that have been pre-built for them. Microsoft has released content packs for popular services such as Salesforce.com: The Customer Success Platform To Grow Your Business, Marketo, Adobe Analytics, Azure Mobile Engagement, CircuitID, comScore Digital Analytix, Quickbooks Online, SQL Sentry and tyGraph.Organizational content packs provide users, BI professionals, and system integrator the tools to build their own content packs to share purpose-built dashboards, reports, and datasets within their organization.Power BI Interview Questions – DAX9). What is DAX?Ans: To do basic calculation and data analysis on data in power pivot, we use Data Analysis Expression (DAX). It is formula language used to compute calculated column and calculated field.DAX works on column values.DAX can not modify or insert data.We can create calculated column and measures with DAX but we can not calculate rows using DAX.Sample DAX formula syntax:For the measure named Total Sales, calculate (=) the SUM of values in the [SalesAmount] column in the Sales table.A- Measure NameB- = – indicate beginning of formulaC- DAX FunctionD- Parenthesis for Sum FunctionE- Referenced TableF- Referenced column name10). What are the most common DAX Functions used?Ans: Below are some of the most commonly used DAX function:SUM, MIN, MAX, AVG, COUNTROWS, DISTINCTCOUNTIF, AND, OR, SWITCHISBLANK, ISFILTERED, ISCROSSFILTEREDVALUES, ALL, FILTER, CALCULATE,UNION, INTERSECT, EXCEPT, NATURALINNERJOIN, NATURALLEFTEROUTERJOIN,SUMMARIZECOLUMNS, ISEMPTY,VAR (Variables)GEOMEAN, MEDIAN, DATEDIFF11). How is the FILTER function used?Ans: The FILTER function returns a table with a filter condition applied for each of its source table rows. The FILTER function is rarely used in isolation, it’s generally used as a parameter to other functions such as CALCULATE.FILTER is an iterator and thus can negatively impact performance over large source tables.Complex filtering logic can be applied such as referencing a measure in a filter expression.FILTER(MyTable,[SalesMetric] > 500)12). What is special or unique about the CALCULATE and CALCULATETABLE functions?Ans: These are the only functions that allow you modify filter context of measures or tables.Microsoft Power BI TrainingInstructor-led SessionsReal-life Case StudiesAssessmentsLifetime AccessExplore CurriculumAdd to existing filter context of queries.Override filter context from queries.Remove existing filter context from queries.Limitations:Filter parameters can only operate on a single column at a time.Filter parameters cannot reference a metric.13). What is the common table function for grouping data?Ans: SUMMARIZE()Main groupby function in SSAS.Recommended practice is to specify table and group by columns but not metrics.You can use ADDCOLUMNS function.SUMMARIZECOLUMNSNew group by function for SSAS and Power BI Desktop; more efficient.Specify group by columns, table, and expressions.14). What are some benefits of using Variables in DAX ?Ans: Below are some of the benefits:By declaring and evaluating a variable, the variable can be reused multiple times in a DAX expression, thus avoiding additional queries of the source database.Variables can make DAX expressions more intuitive/logical to interpret.Variables are only scoped to their measure or query, they cannot be shared among measures, queries or be defined at the model level.15). How would you create trailing X month metrics via DAX against a non-standard calendar?Ans: The solution will involve:CALCULATE function to control (take over) filter context of measures.ALL to remove existing filters on the date dimension.FILTER to identify which rows of the date dimension to use.Alternatively, CONTAINS may be used:CALCULATE(FILTER(ALL(‘DATE’),…….))16). What are the different Excel BI add-in?Ans: Below are the most important BI add-in to Excel:Power Query: It helps in finding, editing and loading external data.Power Pivot: Its mainly used for data modeling and analysis.Power View: It is used to design visual and interactively reports.Power Map: It helps to display insights on 3D Map.Power BI Interview Questions – Power Pivot17). What is Power Pivot?Ans: Power Pivot is an add-in for Microsoft Excel 2010 that enables you to import millions of rows of data from multiple data sources into a single Excel workbook. It lets you create relationships between heterogeneous data, create calculated columns and measures using formulas, build PivotTables and PivotCharts. You can then further analyze the data so that you can make timely business decisions without requiring IT assistance.18). What is Power Pivot Data Model?Ans: It is a model that is made up of data types, tables, columns, and table relations. These data tables are typically constructed for holding data for a business entity.19). What is xVelocity in-memory analytics engine used in Power Pivot?Ans: The main engine behind power pivot is the xVelocity in-memory analytics engine. It can handle large amount of data because it stores data in columnar databases, and in memory analytics which results in faster processing of data as it loads all data to RAM memory.21). What are some of differences in data modeling between Power BI Desktop and Power Pivot for Excel?Ans: Here are some of the differences:Power BI Desktop supports bi-directional cross filtering relationships, security, calculated tables, and Direct Query options.Power Pivot for Excel has single direction (one to many) relationships, calculated columns only, and supports import mode only. Security roles cannot be defined in Power Pivot for Excel.22). Can we have more than one active relationship between two tables in data model of power pivot?Ans: No, we cannot have more than one active relationship between two tables. However, can have more than one relationship between two tables but there will be only one active relationship and many inactive relationship. The dotted lines are inactive and continuous line are active.Power BI Interview Questions – Power Query23). What is Power Query?Ans: Power query is a ETL Tool used to shape, clean and transform data using intuitive interfaces without having to use coding. It helps the user to:Import Data from wide range of sources from files, databases, big data, social media data, etc.Join and append data from multiple data sources. Shape data as per requirement by removing and adding data.24). What are the data destinations for Power Queries?Ans: There are two destinations for output we get from power query:Load to a table in a worksheet.Load to the Excel Data Model.25). What is query folding in Power Query?Ans: Query folding is when steps defined in Power Query/Query Editor are translated into SQL and executed by the source database rather than the client machine. It’s important for processing performance and scalability, given limited resources on the client machine.26). What are some common Power Query/Editor Transforms?Ans: Changing Data Types, Filtering Rows, Choosing/Removing Columns, Grouping, Splitting a column into multiple columns, Adding new Columns ,etc.27). Can SQL and Power Query/Query Editor be used together?Ans: Yes, a SQL statement can be defined as the source of a Power Query/M function for additional processing/logic. This would be a good practice to ensure that an efficient database query is passed to the source and avoid unnecessary processing and complexityby the client machine and M function.28). What are query parameters and Power BI templates?Ans:Query parameters can be used to provide users of a local Power BI Desktop report with a prompt, to specify the values they’re interested in.The parameter selection can then be used by the query and calculations.PBIX files can be exported as Templates (PBIT files).Templates contain everything in the PBIX except the data itself.Parameters and templates can make it possible to share/email smaller template files and limit the amount of data loaded into the local PBIX files, improving processing time and experience .29). Which language is used in Power Query?Ans: A new programming language is used in power query called M-Code. It is easy to use and similar to other languages. M-code is case sensitive language.30). Why do we need Power Query when Power Pivot can import data from mostly used sources?Ans: Power Query is a self-service ETL (Extract, Transform, Load) tool which runs as an Excel add-in. It allows users to pull data from various sources, manipulate said data into a form that suits their needs and load it into Excel. It is most optimum to use Power Query over Power Pivot as it lets you not only load the data but also manipulate it as per the users needs while loading.Power BI Interview Questions – Power Map31). What is Power Map?Ans: Power Map is an Excel add-in that provides you with a powerful set of tools to help you visualize and gain insight into large sets of data that have a geo-coded component. It can help you produce 3D visualizations by plotting upto a million data points in the form of column, heat, and bubble maps on top of a Bing map. If the data is time stamped, it can also produce interactive views that display, how the data changes over space and time.32). What are the primary requirement for a table to be used in Power Map?Ans: For a data to be consumed in power map there should be location data like:Latitude/Longitude pairStreet, City, Country/Region, Zip Code/Postal Code, and State/Province, which can be geolocated by BingThe primary requirement for the table is that it contains unique rows. It must also contain location data, which can be in the form of a Latitude/Longitude pair, although this is not a requirement. You can use address fields instead, such as Street, City, Country/Region, Zip Code/Postal Code, and State/Province, which can be geolocated by Bing.33). What are the data sources for Power Map?Ans: The data can either be present in Excel or could be present externally. To prepare your data, make sure all of the data is in Excel table format, where each row represents a unique record. Your column headings or row headings should contain text instead of actual data, so that Power Map will interpret it correctly when it plots the geographic coordinates. Using meaningful labels also makes value and category fields available to you when you design your tour in the Power Map Tour Editor pane.To use a table structure which more accurately represents time and geography inside Power Map, include all of the data in the table rows and use descriptive text labels in the column headings, like this:BI and Visualization TrainingTABLEAU TRAINING AND CERTIFICATIONTableau Training and CertificationReviews5(29591)MICROSOFT POWER BI TRAININGMicrosoft Power BI TrainingReviews5(13668)ADVANCED MS EXCEL 2016 CERTIFICATION TRAININGAdvanced MS Excel 2016 Certification TrainingReviews5(8192)MICROSOFT BI CERTIFICATION TRAININGMicrosoft BI Certification TrainingReviews5(12569)MICROSOFT EXCEL VBA CERTIFICATION TRAININGMicrosoft Excel VBA Certification TrainingReviews5(6721)QLIKVIEW CERTIFICATION TRAININGQlikView Certification TrainingReviews5(9728)PENTAHO BI CERTIFICATION TRAININGPentaho BI Certification TrainingReviews4(3551)MICROSTRATEGY 10 CERTIFICATION TRAININGMicroStrategy 10 Certification TrainingReviews5(1870)NextIn case you wish to load your data from an external source:In Excel, click Data > the connection you want in the Get External Data group.Follow the steps in the wizard that starts.On the last step of the wizard, make sure Add this data to the Data Model is checked.Power BI Interview Questions – Additional Questions34). What is Power View?Ans: Power View is a data visualization technology that lets you create interactive charts, graphs, maps, and other visuals which bring your data to life. Power View is available in Excel, SharePoint, SQL Server, and Power BI.The following pages provide details about different visualizations available in Power View:ChartsLine chartsPie chartsMapsTilesCardsImagesTablesPower ViewMultiples VisualizationsBubble and scatter chartsKey performance indicators (KPIs)35). What is Power BI Designer?Ans: It is a stand alone application where we can make Power BI reports and then upload it to Data Visualization | Microsoft Power BI, it does not require Excel. Actually, it is a combination of Power Query, Power Pivot, and Power View.36). Can we refresh our Power BI reports once uploaded to cloud (Share point or http://Powebi.com)?Ans: Yes we can refresh our reports through Data Management gateway(for sharepoint), and Power BI Personal gateway(for Data Visualization | Microsoft Power BI)37). What are the different types of refreshing data for our published reports?Ans: There are four main types of refresh in Power BI. Package refresh, model or data refresh, tile refresh and visual container refresh.Package refreshThis synchronizes your Power BI Desktop, or Excel, file between the Power BI service and OneDrive, or SharePoint Online. However, this does not pull data from the original data source. The dataset in Power BI will only be updated with what is in the file within OneDrive, or SharePoint Online.Model/data refreshIt referrs to refreshing the dataset, within the Power BI service, with data from the original data source. This is done by either using scheduled refresh, or refresh now. This requires a gateway for on-premises data sources.Tile refreshTile refresh updates the cache for tile visuals, on the dashboard, once data changes. This happens about every fifteen minutes. You can also force a tile refresh by selecting the ellipsis (…) in the upper right of a dashboard and selecting Refresh dashboard tiles.Visual container refreshRefreshing the visual container updates the cached report visuals, within a report, once the data changes.To know more about data refresh and understand how to implement data refresh, you can check the following link.38). Is Power BI available on-premises?Ans: No, Power BI is not available as a private, internal cloud service. However, with Power BI and Power BI Desktop, you can securely connect to your own on-premises data sources. With the On-premises Data Gateway, you can connect live to your on-premises SQL Server Analysis Services, and other data sources. You can also scheduled refresh with a centralized gateway. If a gateway is not available, you can refresh data from on-premises data sources using the Power BI Gateway – Personal.39). What is data management gateway and Power BI personal gateway?Ans: Gateway acts a bridge between on-premises data sources and Azure cloud services.Personal Gateway:Import Only, Power BI Service Only, No central monitoring/managing.Can only be used by one person (personal); can’t allow others to use this gateway.On-Premises Gateway:Import and Direct Query supported.Multiple users of the gateway for developing content.Central monitoring and control.40). What is Power BI Q&A?Ans: Power BI Q&A is a natural language tool which helps in querying your data and get the results you need from it. You do this by typing into a dialog box on your Dashboard, which the engine instantaneously generates an answer similar to Power View. Q&A interprets your questions and shows you a restated query of what it is looking from your data. Q&A was developed by Server and Tools, Microsoft Research and the Bing teams to give you a complete feeling of truly exploring your data.41). What are some ways that Excel experience can be leveraged with Power BI?Ans: Below are some of the ways through which we can leverage Power BI:The Power BI Publisher for Excel:Can be used to pin Excel items (charts, ranges, pivot tables) to Power BI Service.Can be used to connect to datasets and reports stored in Power BI Service.Excel workbooks can be uploaded to Power BI and viewed in the browser like Excel Services.Excel reports in the Power BI service can be shared via Content Packs like other reports.Excel workbooks (model and tables) can be exported to service for PBI report creation.Excel workbook Power Pivot models can be imported to Power BI Desktop models.42). What is a calculated column in Power BI and why would you use them?Ans: Calculated Columns are DAX expressions that are computed during the model’s processing/refresh process for each row of the given column and can be used like any other column in the model.Calculated columns are not compressed and thus consume more memory and result in reduced query performance. They can also reduce processing/refresh performance if applied on large fact tables and can make a model more difficult to maintain/support giventhat the calculated column is not present in the source system.43). How is data security implemented in Power BI ?Ans: Power BI can apply Row Level Security roles to models.A DAX expression is applied on a table filtering its rows at query time.Dynamic security involves the use of USERNAME functions in security role definitions.Typically a table is created in the model that relates users to specific dimensions and a role.44). What are many-to-many relationships and how can they be addressed in Power BI ?Ans: Many to Many relationships involve a bridge or junction table reflecting the combinations of two dimensions (e.g. doctors and patients). Either all possible combinations or those combinations that have occurred.Bi-Directional Crossfiltering relationships can be used in PBIX.CROSSFILTER function can be used in Power Pivot for Excel.DAX can be used per metric to check and optionally modify the filter context.45). Why might you have a table in the model without any relationships to other tables?Ans: There are mainly 2 reasons why we would have tables without relations in our model:A disconnected table might be used to present the user with parameter values to be exposed and selected in slicers (e.g. growth assumption.)DAX metrics could retrieve this selection and use it with other calculations/metrics.A disconnected table may also be used as a placeholder for metrics in the user http://interface.It may not contain any rows of data and its columns could be hidden but all metrics are visible.46). What is the Power BI Publisher for Excel?Ans: You can use Power BI publisher for Excel to pin ranges, pivot tables and charts to Power BI.The user can manage the tiles – refresh them, remove them, in Excel.Pinned items must be removed from the dashboard in the service (removing in Excel only deletes the connection).The Power BI Publisher for Excel can also be used to connect from Excel to datasets that are hosted in the Power BI Service.An Excel pivot table is generated with a connection (ODC file) to the data in Azure.The Publisher installs all necessary drivers on local machine to establish connectivity .47). What are the differences between a Power BI Dataset, a Report, and a Dashboard?Ans: Dataset: The source used to create reports and visuals/tiles.A data model (local to PBIX or XLSX) or model in an Analysis Services ServerData could be inside of model (imported) or a Direct Query connection to a source.Report: An individual Power BI Desktop file (PBIX) containing one or more report pages.Built for deep, interactive analysis experience for a given dataset (filters, formatting).Each Report is connected to atleast one datasetEach page containing one or more visuals or tiles.Dashboard: a collection of visuals or tiles from different reports and, optionally, a pinned.Built to aggregate primary visuals and metrics from multiple datasets.48) What are the three Edit Interactions options of a visual tile in Power BI Desktop?Ans: The 3 edit interaction options are Filter, Highlight, and None.Filter: It completely filter a visual/tile based on the filter selection of another visual/tile.Highlight: It highlight only the related elements on the visual/tile, gray out the non-related items.Microsoft Power BI TrainingWeekday / Weekend BatchesSee Batch DetailsNone: It ignore the filter selection from another tile/visual.49). What are some of the differences in report authoring capabilities between using a live or direct query connection such as to an Analysis Services model, relative to working with a data model local to the Power BI Desktop file?Ans: With a data model local to the PBIX file (or Power Pivot workbook), the author has full control over the queries, the modeling/relationships, the metadata and the metrics.With a live connection to an Analysis Services database (cube) the user cannot create new metrics, import new data, change the formatting of the metrics, etc – the user can only use the visualization, analytics, and formatting available on the report canvas.With a direct query model in Power BI to SQL Server, for example, the author has access to the same features (and limitations) available to SSAS Direct Query mode.Only one data source (one database on one server) may be used, certain DAX functions are not optimized, and the user cannot use Query Editor functions that cannot be translated into SQL statements.50). How does SSRS integrate with Power BI?Ans: Below are some of the way through which SSRS can be integrated with Power BI:Certain SSRS Report items such as charts can be pinned to Power BI dashboards.Clicking the tile in Power BI dashboards will bring the user to the SSRS report.A subscription is created to keep the dashboard tile refreshed.Power BI reports will soon be able to be published to SSRS portal

Some people might think this positive change is exclusive to me, that perhaps I have some incredible determination, persistence or discipline to pull this off. I don’t want to disappoint, but it’s not. In fact, truth be told, I consider myself a very undisciplined person. What I do have though, are 6 specific tips that have been critical in enabling my lifestyle change. These have helped my new habits stick.If you have been trying to cultivate new habits with little success, then you might find these very useful. These habits are not rocket science – they are easy to understand, apply and have worked tremendously for me.Here they are:1. Know the Real Reason Why Your Habit Didn’t Stick PreviouslyAddress the root cause of the issue, not the effect. Desperately battling with yourself every morning to wake up at 5:30am is to address the effect. Understanding why you keep failing to wake up at 5:30am is to address the cause.For example, I couldn’t wake up early for the longest time ever, and all I kept doing is to keep trying and failing the next day. This would continue on for several months until I finally realized it was just going nowhere. I began to start analyzing my situation to understand why I couldn’t wake up early, through a self-questioning process. I probed into the situation, and asked myself “why” this was happening to drill down to the root cause.Below is an example of the drilling process:Why can’t I wake up early?Because I’m tired.Why am I tired?Because I didn’t have enough sleep.Why didn’t I have enough sleep?Because I slept late.Why did I sleep late?Because I had too many things to do.Why did I have so many things to do?Because I can’t finish them.Why can’t I finish them?Because I schedule more tasks than I can accomplish for the day.Getting down to this root cause helped me realize two things (1) All our habits are tied to one another (sleeping time, waking time, timeliness) (2) I underestimate the time taken to finish the tasks (and subsequently overestimate how fast I can do those tasks). Many times, I would target to finish multiple projects in 1 day, which wasn’t possible at all.This meant that to make my waking early habit stick, (1) I need to change habits that are related to waking early (see Tip #2) and (2) I have to be more realistic in my planning. Rather than stuff in so many tasks for a day and not finishing them, now I go for a challenging yet achievable schedule and complete my tasks accordingly.Keep asking why to drill down to the root reason. Once you get to the real cause, you can immediately resolve the issue.2. Pick Habits that Reinforce Each OtherOur habits are not standalone; they areinterlinked. Some habits have a stronger linkage with each other than others. For example, sleeping early and waking early are obviously linked to each other, while sleeping early and reading a book a day might not be so closely related. If you want to cultivate a habit, identify the other habits that are tied with it and make a holistic change. These habits will reinforce each other to help make the change seamless.For example, my new habits to: (a) Wake up early at 5am (b) Sleep before 12am (c) Be on time (d) Meditate (e) Have raw food diet are all interlinked.Waking up early means I more time to do my tasks, which helps me to sleep earlier in the night. This helps me to wake up early the next day.Being on time helps me to get my tasks completed on time, which helps me adhere to the day’s schedule. This means my sleeping time and subsequently my waking time does not get affected.Meditating clears out mental clutter and reduces the amount of sleep I need. Usually I sleep about pro6-10 hours, but on the nights I meditate, I require about 5-6 hours.Switching to a raw vegan diet has helped to increase my mental clarity, which meant I don’t need to sleep as much as before. I’m not saying that you need to go raw vegan just to cultivate a habit of sleeping/waking early, just that I noticed this particular benefit when I switched to this diet. You can sleep and wake up early perfectly fine by changing other habits.3. Plan For Your Habits (Right down to the timings)Having a schedule lets you know when you are on or off track for your habits. For the 1st day of my new lifestyle, I did a full-day planning and continued thereafter for all other days.What I do is this:On the night before, put together a list all the tasks I need to get done for the next day. This includes what’s on my calendar (I use Gcal).Batch them into (a) Major projects, (b) Medium sized tasks and (c) Small, administrative activitiesSlot them into my schedule for the day. Major projects would have most amount of time assigned. The principle I usually go by is 60-30-10 (% time spent) for a-b-c groups respectively.Be aware of how much time each task requires. If it helps, most of the time we underestimate the time we need. Make it a realistic yet challenging time to work towards. Usually I assign a 5-10 minutes buffer time in between tasks to account for the transition from 1 task to the next.Assign exact timings for when each task starts and ends. For example, 9am to 10:30am for Project A, 12:30-1:30pm for lunch, 6:30-7:30pm for commute.If there are more tasks to be done than my schedule allows, I’lldeprioritize the unimportant onesand put them off to another day.With all this planning done, when the next day comes all I have to do is to follow the schedule to a tee. I keep a close watch on the timing to ensure I’m on time. 5 minutes before it’s up, I do a wrap up and start transiting to the next task on the list.The beauty of having a precise schedule is it helps me know exactly when I’m taking more time than desired, and this helps me work on being more efficient. There are some timings which absolutely have to be protected, such as my sleeping/waking times and appointment times, so in that sense the time allocated on my tasks are fixed. That means I have to work more efficiently.It may seem like a hassle, but it really isn’t. It just takes me about 10 minutes to get each day’s schedule done. Not surprisingly, I have allocate time in my daily schedule to do my scheduling for the next day (11-11:10pm). All you have to do is create a template once, and then you can reapply this template for the other days. There will be similar items across all out days that can be reapplied, such as waking/breakfast/commuting/working/dinner/sleeping times, so it’s really very straight forward.If you don’t plan for when exactly to get the habit done and instead just arbitrarily say that you want it to be done sometime today, then there’s a very high chance it might not get done. This is why most people’s habits don’t stick. Other things will invariably keep popping in and you’d engage them without realizing it and throw your schedule off track. From there, other things get pushed back and you never get to carry out your habit.4. Stay Ahead of Your ScheduleI found it’s extremely motivating to stay ahead. Waking up early at 5am means I’m ahead of most people in the world (and myself too, if I were to stick to my old schedule), and that motivates me to work fast and stay ahead. What helps me continue this momentum is that I end my tasks earlier and start the next task before the scheduled timing. By ensuring I stay ahead of my schedule, I’m naturally motivated to work on all the things I have planned, including my habits. There’s no resistance to get them started at all.If a task is taking more time than needed, then I make a choice. Either I:Hurry up and get it doneDeprioritize the unnecessary orBorrow time out for my later tasks to continue working on the current one. This also means I have to work faster for the remainder of the day.This decision-making process is important, because otherwise you will end up playing catch-up for the rest of the day, which affects all your planned habits/activities. Subsequently, it also affects your will to maintain your habits. Stay ahead of your schedule and you will find it easier to stay motivated.5. Track Your HabitsTracking keeps you accountable to your habits. I have a whiteboard in my bedroom which I use to track my habits. On the whiteboard, I drew a large table, split by days (21 days to cultivate a new habit) and by habits. For the days where I do the habit, I will give it a check. For the days I don’t, I make a cross. It’s very satisfying to do the checks every time you finish a habit! You can also track your habits on paper or in your computer.Here are some great habit trackers online:HabitForge – Tracks new habits through a 21-day period. If you miss the habit for 1 day, it’ll restart.Rootein – Unlike Habit Forge, this is an ongoing habit tracker. There is also a mobile version for you to track your habits on the go.Joe’s Goals – Same as Rootein. There’s an option to place multiple checks on the same goal for extra-productive days.6. Engage People Around YouEngagement can occur on 2 levels – (a) Active engagement, where you inform your friends who might be interested in and cultivate the habit together with them or (b) Passive engagement, where you let others know about your plans and having them morally support you.I had both forms of support in my habit change. 2 days before I started my lifestyle revamp program, I posted an article on my blog, The Personal Excellence Blog, on the new 21-day Lifestyle Revamp Program I was taking on. I wrote in detail about the rationale behind the program, the benefits, the habits I was taking on and how I was going to achieve my goals. I also invited them to join me too in cultivating new habits. Much to my pleasant surprise, many readers responded in enthusiasm on new habits they wanted to cultivate and joined me in the 21-days of change.For my raw food diet, I told my mom that I’m just eating fruits and salads for the next 3 weeks, and she began to stock up the house with fruits like bananas, grapes and strawberries. In fact, I just finished a box of strawberries from typing this post. Yesterday, I went to watch How To Train A Dragon with my friend, and filled him in on my raw food diet. He then kept a look out for the restaurants we could dine in for that night. In the end I had warm baby spinach salad for dinner. My first time having it – can’t say I like it, but it’s nice for a change :D.Don’t feel that you’re alone in your habit change because you aren’t. There are always people around you who are more than willing to support you.Final WordsMy new habits have pretty much been integrated into my daily life now. Everything runs on auto-pilot and it feels like I’ve been doing this for a long while. My personal tips above have worked tremendously for myself, so while they may look simple and straightforward, don’t underestimate them. Try them out for yourself and let me know how your new habits are coming along for you

Getting Data From An Online SourceAs I’m sure many statisticians do, I keep a folder of “stock code”, or template scripts that do different things. This folder is always growing and the scripts are always improving, but there are a few in there that I think are worth sharing. Some of these are templates for common analyses, while others are just reminders of how to use a couple of commands to accomplish a practical task.This post is of the latter type. I’m going to discuss fetching data from a URL.Why might one need to fetch data from a URL?You want to share your code with someone who isn’t familiar with R and you want to avoid the inevitable explanation of how to change the file path at the beginning of the file. (“Make sure you only use forward slashes!”)The data at the URL is constantly changing and you want your analysis to use the latest each time you run it.You want the code to just work when it’s run from another machine with another directory tree.You want to post a completely repeatable analysis on your blog and you don’t want it to begin with “go to http://www.blahblahblah.com, download this data, and load it into R”.Whatever your reason may be, it’s a neat trick, but it’s not one I use so often that I can just rattle off the code for it from memory. So here’s my template. I hope it can help someone else.Caveat!!!This is only for data that is in tabular form already. This is not for web scraping (i.e. extracting a table of data from a Wikipedia page.) There areentire packages devoted to that. This is for the simplest of all cases where there is a .csv file or a .txt file (or similar) at a URL and you want to read it into R directly from that URL without the intermediate step of saving it somewhere on your computer.Using data.table’s fread()I love the data.table package. I use it every day, for almost every project I do. It’s an extension of the data.frame object class in R that makes many improvements. One of those improvements is in the function fread(). It’s data.table’s answer to base R’s read.csv(). It does many things better, but here I’m only going to address its ability to read data right from the web. As a primer, its typical use on a data file residing on your computer would look something like this:library(data.table)mydat <- fread('C://Some/File/Path.csv') Reading data from a source on the web is no different. The example the package authors give in the help file (?fread) is this:library(data.table)mydat <- fread('http://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat')head(mydat)  V1 V2 V3 V4 V51: 1 307 930 36.58 02: 2 307 940 36.73 03: 3 307 950 36.93 04: 4 307 1000 37.15 05: 5 307 1010 37.23 06: 6 307 1020 37.24 0 Now if you actually navigate to that link in your browser, you won’t see anything, but a download dialog should pop up. If you navigate to the parent directory of that address, http://www.stats.ox.ac.uk/pub/datasets/csb you will see some text further down the page you will see several links to data files. Each of these links launches a download dialog when clicked. To grab the URL of the data file to pass to fread(), right click the link and select “Copy link address”. Other data files online might appear in the browser instead of launching download dialog, like this one a professor of mine had us use for an assignment. fread() handles these URLs just the same.fread() makes smart decisions about how to read the data in (it detects column names and classes and so on), but the command has several arguments for specifying such things as well that you can use at your own discrimination. I find fread('filename') almost always just works, but sometimes there are reasons to be more explicit when reading data in.Using RStudioIf you’re not familiar with RStudio, you are a true R novice. If you know what it is, but don’t use it, skip ahead.In RStudio, you can click “Tools” -> “Import Dataset” -> “From Web URL” and a dialog will pop up asking you for a URL. Paste a URL into the dialog box (let’s just use the same one as before: http://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat) and click “OK”. A nice little window pops up and allows you to specify how the data should be read and what name the object should be given in R. When you click “Import”, the data is read in and some code appears in the console. What this interface does is download the data to a temporary file in a temporary folder and then read it in. The downloaded data file persists on your hard drive as long as your R session lasts, but disappears as soon as your R session ends.This is handy, but if you wanted to repeat the process, you would have to click through the menu again and supply the data URL again. This isn’t exactly “repeatable” in the Stack Overflow sense of the word.Using RCurl’s getURL()The RCurl package provides bindings to the cURL library. This is a C library for web connections. The cURL library does way more than we need for this task and frankly, I don’t understand a lot of it. I saved RCurl for last because iI usually try fread() first, and then if I get some sort of error, I resort to RCurl. Take for example the data set at this link: https://sakai.unc.edu/access/content/group/3d1eb92e-7848-4f55-90c3-7c72a54e7e43/public/data/bycatch.csv (also posted by a professor for an assignment of mine). If you try to fread() it, no dice. I have no idea what that error message means, but here’s how to get that data set in anyway.library(RCurl)myfile <- getURL('https://sakai.unc.edu/access/content/group/3d1eb92e-7848-4f55-90c3-7c72a54e7e43/public/data/bycatch.csv', ssl.verifyhost=FALSE, ssl.verifypeer=FALSE) What are the arguments ssl.verifyhost=F and ssl.verifypeer=F doing? To be quite honest, I don’t really know. But if I’m having trouble reading from a URL I try specifying these arguments and changing one or both to FALSE almost always circumvents whatever error I’m getting.This grabs the content residing at the specified URL, but doesn’t return a data.frame object. It has simply put the URL’s content into a string.class(myfile) [1] "character" So how to get this into a data.frame object? We’ll use textConnection() to open a “connection” with the string, much like you would open a connection with a file on your hard drive in order to read it. Then we’ll have read.csv() (or you could use read.table() or fread() or similar) to read the string object like a text file and create a data.frame object.mydat <- read.csv(textConnection(myfile), header=T)head(mydat)  Season Area Gear.Type Time Tows Bycatch1 1989-90 North Bottom Day 48 02 1989-90 North Bottom Night 6 03 1989-90 North Mid-Water Night 1 04 1989-90 South Bottom Day 139 05 1989-90 South Mid-Water Day 6 06 1989-90 South Bottom Night 6 0 And there you have it. The data from the URL is now in a data.frame and ready to go.Aside: read.csv() is just a vesion of read.table() with argument defaults such as sep = "," that make sense for reading .csv files.A Use CaseLet’s pretend I want to automate something having to do with weather in Chicago. Maybe it’s a knitr document that I have scheduled to re-knit every night on my server. Every time the script re-runs, it should somehow take into account recent weather in Chicago. Weather Undergroundoffers historic (and an hour ago counts as “historic”) hourly weather data for many different locations. Many of these locations are airports, which for obvious reasons, have several meteorological sensors on site. On the Weather Underground page you can select a location and a date and see hourly weather for that calendar day. At the bottom of the page, you can click “Comma Delimited File” to see the data in comma delimited format – perfect for reading into R.The URLs for such data are always of a predictable form:http://www.wunderground.com/history/airport/4LetterAirportCode/year/month/day/DailyHistory.html?some_junk_specifying_formatI see that the four letter airport code for Chicago is “KMDW” and after clicking through a few of these URLs, I see the stuff after “DailyHistory.html” doesn’t change. So if I know the date, I can construct the URL where the hourly Chicago airport wether for that date can be found in .csv format.First, I define the beginning and end of the URL, which never change.baseURL <- 'http://www.wunderground.com/history/airport/KMDW'suffixURL <- 'DailyHistory.html?HideSpecis=1&format=1' There is opportunity here to generalize this for many locations if one simply maps the four letter codes to other locations of interest usingswitch() or similar.Then I ask the system for todays date and from it produce a string having format year/month/day.Date <- Sys.Date()datestring <- format(Date, '%Y/%m/%d') Then I piece all of these strings together to get a URL which will lead to a .csv file of today’s weather in Chicago.url2fetch <- paste(baseURL, datestring, suffixURL, sep='/') Finally I grab the content of the webpage at that URL using the RCurl method described above. I choose getURL() instead of fread() for good reason; I’ll need to do some find-and-replace to clean up some html artifacts in the data and that is more efficient to do on one big string rather than on a bunch of individual values in a data.frame.url_content <- getURL(url2fetch) Now I have the content of the page in a string and I want to read that string into a data.frame object, but every line of the data ends with an html newline (“<br />”) and a text newline (“n”). read.csv() will recognize the “n” as a signal to start a new row of the data.frame, but the “<br />” isn’t recognized and will be appended to the value in the last column of every row. So let’s take care of this before read.csv() ever gets involved. I’ll do a simple find-and-replace where I find “<br />” and replace it with an empty string (""), aka nothing. This is the regex way of find-and-delete.url_content <- gsub('<br />', '', url_content) Finally I can “read” the data into a data.frame object with the help of read.csv() and textConnection().weather_data <- read.csv(textConnection(url_content))head(weather_data)  TimeCST TemperatureF Dew.PointF Humidity Sea.Level.PressureIn1 12:22 AM 21.9 17.1 82 30.022 12:53 AM 21.9 16.0 78 30.073 1:53 AM 21.9 15.1 75 30.094 2:24 AM 21.0 14.0 74 30.045 2:39 AM 21.0 14.0 74 30.046 2:53 AM 21.0 15.1 78 30.09 VisibilityMPH Wind.Direction Wind.SpeedMPH Gust.SpeedMPH PrecipitationIn1 1.0 NNE 13.8 - 0.012 1.0 NNE 15.0 - 0.013 4.0 NNE 11.5 - 0.004 2.5 NNE 16.1 - 0.005 1.5 NNE 12.7 - 0.006 1.8 NNE 12.7 - 0.00 Events Conditions WindDirDegrees DateUTC1 Snow Snow 30 2015-02-26 06:22:002 Snow Light Snow 30 2015-02-26 06:53:003 Snow Light Snow 30 2015-02-26 07:53:004 Snow Light Snow 30 2015-02-26 08:24:005 Snow Light Snow 30 2015-02-26 08:39:006 Snow Light Snow 30 2015-02-26 08:53:00