List Of Data Elements & Associated Complex Types: Fill & Download for Free

There are four main challenges when scaling a database: search, concurrency, consistency, and speed.Suppose you have a list of 10 names. To find someone, you just go down the list.But what if there are 1 million names? Now you need a strategy for finding something. A telephone book lists the names in alphabetical order so you can skip around. This is a solution to the search problem.What if 1 million people are trying to use the telephone book at the same time? This is the problem of concurrency. Everyone could wait in one very long line at City Hall, or you could print 1 million copies of the book -- a strategy called "replication". If you put them in people's homes -- a strategy called "distributed" -- you also get faster access.What if someone changes their phone number? The strategy of replication created a problem, which is that you now have to change all 1 million phone books. And when are you going to change them, because they are all in use? You could change them one at a time, but this would create a data consistency problem. You could take them all away and issue new ones, but now you have an availability problem while you are doing it.And what if thousands of people are changing their phone numbers every hour? Now you have a giant traffic jam called "contention for resources" which leads to "race conditions" (unpredictable outcomes) and "deadlocks" (database gridlock).All of these problems have solutions, but the solutions can get very complex. For example, you can issue addendums to the phone books (called "change logs") rather than reprinting all of them. But you have to make sure to check the addendums all the time. You can distribute new versions of the phone books with a cut-over date, so that everyone switches at the same time to get greater consistency, but now the phone books are always slightly out of date.Now scale this to billions of names in data centers distributed around the world accessed by millions of users.The basic goal of a database is to maintain the illusion that there is only one copy, only one person changes it at a time, everyone always sees the most current copy, and it is instantly fast. This is impossible to achieve at scale when millions of people are accessing and updating trillions of data elements from all over the world.The task of database design, therefore, is to come as close to this illusion as possible using hundreds of interlocking algorithmic tricks.

Skiplist is a data structure used to store sorted list of items, very much like Binary Search Tree. The main advantage of using Skiplists over BST is that it is best suited for concurrent access.One of the most common implementations of BST is Red Black Tree. The problem of accessing this tree concurrently comes when a data element has been modified and tree needs to rebalance, which would require mutex lock on large part of tree.On the other hand, when inserting/deleting/updating a node in a Skiplist, only nodes directly connected to the affected nodes needs locking.Herb Stutter once wrote a wonderful article on concurrent friendly data structures: http://www.drdobbs.com/parallel/choose-concurrency-friendly-data-structu/208801371

That depends on the problem you are trying to solve. If a problem is mapped to the most efficient data-structure which captures the essence of that problem, then it leads to an elegant solution of the problem.The "right" choice of data-structure would not only depend on the representation of the inputs, but the query it is supposed to be optimal for. E.g if asked to find a number among the list of number efficiently, then BST(Binary Search Tree) is a choice which would efficiently represent the input data for the set of all point search queries. If the query was for a range of numbers, and not just a single number, then BST is no longer the optimal choice but the data-structure to choose is maybe B+ Tree.For some kind of problems, it may be more efficient to choose even probablistic data-structures which offer an interesting trade-off between space, time, and correctness. Example of such data-structures are to name a few among the popular ones, bloom filters, and hyperloglog. Such data-structures may allow false positives or false negatives (with some error probability) depending on their design. These are respectively used to provide efficient solutions for determining membership of an element in a cache, and approximate counting. In the case of cache, if you mispredict the presence of the element in the cache, no big harm done, you just fetch the element from the higher-up hierarchy.To prove the case in point, here's an extremely simple data-structure (recently developed, specifically to solve a particular problem) which solves a not so easy problem regarding palindromes in strings elegantly. Trying to fit known data-structures to the problem results in solutions which are difficult to understand and implement.What is the most efficient data structure to represent palindromes in a given string? What is the range of problems that can be solved by this data structure?To summarize, you would need to study a problem, understand what constitutes an efficient solution in its context (memory, time complexity, error probability) and *google search* for the right data-structure (or creatively design it) to solve the problem elegantly.