Integration Of Big Data And Relational Database: Fill & Download for Free

Eight years ago Hadoop was released. At that time, SQL was the lingua franca of data. If you had data, it was stored in a rigid relational database with few scaling options other than buying bigger servers. You needed to carefully design your data model and liberally apply indexes. Changes to ETL and schema were painful. On top of that, many workloads at the time were just not suited to SQL and relational databases.The Hadoop ecosystem took off like a rocket. It was viewed as the future of data. Why be limited by relational databases when Hadoop could do everything? Well, it turns out that many workloads were better suited to SQL and relational databases than Hadoop.SQL was integrated into Hadoop and other “no sql” platforms. Use the underlying engine of whatever technology you like, but query with SQL. This had a lot of success. But many challenges still existed. Could the engine support ANSI-complient SQL or did it require a new variant? Could the engine support relational modeling? How fast was it?People started looking at the problems with fresh eyes. Many workloads were suited to SQL and relational modeling, so why not just use relational databases for them? Vendors started implementing more features into their platforms. Support for semi-structured data like JSON was added. Better scalability and clustering. Integrations into other tools where SQL and relational databases were not the best tool for the job.SQL got hot again. So, what is the next big thing for SQL. I would look to technologies like Snowflake. SQL, without the hassles. Easy import for all data types. No need for indexes due to clever storage mechanisms. Simple horizontal and vertical scaling. Integration into virtually any data platform.The next big thing for SQL is cloud databases that support business with relational models, but include the flexibility to handle different types of workloads with minimal administrative overhead.

SQL is something that’s gonna be there till the end of the world. Even with the rise of Big Data Systems, relational database systems have not lost importance. Also RDBMS companies are rolling out Big Data flavored features like scaling, distributed processing, data warehousing to handle large scale structured data with the current implementations of Relational Databases. So the point is SQL is a very good place to start with and it has a great future. But I think you need to polish your skills and widen your scope to accommodate different requirements. So you should keeps these things in your mind:Start with a platform. SQL is an ANSI standard database scripting language, but that has been customized by large technology corps like Microsoft, Oracle, IBM, Sybase and so on. Pick up a platform and learn SQL on it. So if it is Microsoft, then install SQL Server and start exploring T-SQL. General statements are common on all platforms to some extent, but as you go deep, there changes at query, datatypes, execution levelsStart exploring more about the database engine of the database product. To write better queries, you need to know how they are evaluated inside the server, when you fire them.Get some grip on database administration.Having knowledge of reporting and data integration would be a plus point for you. So if it is SQL Server then try your hands on SSIS (SQL Server Integration Services) and SSRS (SQL Server Reporting Services).As a last resort, if you are going to give a dive into Big Data, then learn HIVE. It is similar to SQL.I am open to further questions.Happy Querying!!

Well there are a *bunch* of reasons that it isn’t generally a good idea. Especially when you are talking about *relational* databasesa binary blob (your image) is an opaque fact - you can’t do anything with it in the database except extract it (IO) or write new ones (IO) — you aren’t going to be using it to *index* anything or relate to anything. As far as the DB engine is concerned it isn’t much use and it takes up a bunch that you don’t want to waste.Relational databases tend to obsess about their structure. They pack in those fields into complex structures and do a bunch of arcane stuff to the underlying filesystem to try and speed up things - often engines have hidden length limits for rows and going over them causes strange things to happenYour relational database is either going to be doing one of two things:Being an essential part of a *transaction* flow - (i.e you are using it for it’s atomicity in an online process). In this mode the speed of the database is a *limiting factor* in how many transactions you can process in a given time. If the database is serving big binary blobs (images / videos / etc) then *it is NOT serving important transactions*Being some sort of source of truth (i.e you are using it because it offers strong guarantees and relational integrity) - in this case you will often export the data somewhere more useable for a presentation layer (like a cache or a nosql store) - in which case you may as well put the images in that layer too.Backing up and restoring your database takes longer / more resources because you are backing up blobs.Images tend to be edited with things like paint tools / photoshop / etc - having the images as *files* makes it a lot easier to work with these assets.It’s actually a lot easier to store versions of files in things like S3 — imagine if you store your images in some object store instead - then all you would need do in your database is store the path to the object.Generally speaking - the relational database is for information you want to run queries on / operate transactions on. Big blobs get in the way of that.