Early Bird Patch: Fill & Download for Free

In general, large-eyed birds who hunt on the ground wake up first - as the saying goes: “The early bird gets the worm”. In a garden setting like ours, the robin, the blackbird and the thrush are first on the go. Their large eyes provide excellent low-light vision to help them snatch worms who are dallying too long about getting underground before light.The answer of course varies with habitat: seashore and mountain would wake to different voices.In an Irish early summer garden, the first blackbird announces his patch between 4.00 and 4.30 a.m. - “Blackbird singing in the dead of night”.

According to the data:Facebook serves 570 billion page views per month (according to Google Ad Planner).There are more photos on Facebook than all other photo sites combined (including sites like Flickr).More than 3 billion photos are uploaded every month.Facebook’s systems serve 1.2 million photos per second. This doesn’t include the images served by Facebook’s CDN.More than 25 billion pieces of content (status updates, comments, etc) are shared every month.Facebook has more than 30,000 servers (and this number is from last year!)Facebook still uses PHP and MySQL and LAMP .But it has optimized and revolutionized its software so that it can handle large audiences.Particularly it has segmented all its software for specific purposes.Below is the list of revolutionary software by facebook.MEMCACHEDIt is by now one of the most famous pieces of software on the internet which provides distributed memory caching service to be used as a caching layer between web server and mySQL avaoiding direct database access which is slow.HIPHOP FOR PHPPHP, being a scripting language, is relatively slow when compared to code that runs natively on a server. It converts PHP into C++ code which can then be compiled for better performance.HAYSTACKIt is Facebook’s high-performance photo storage/retrieval system (strictly speaking, Haystack is an object store, so it doesn’t necessarily have to store photos). It has a ton of work to do; there are more than 20 billion uploaded photos on Facebook, and each one is saved in four different resolutions, resulting in more than 80 billion photos.CASSANDRAIt is a distributed storage system with no single point of failure and it is mostly like NoSQLSCRIBEIt is flexible logging system of facebookHADOOP and HIVEFacebook uses it for data analysis and miningTHRIFTFacebook uses many languages for other system.Like Facebook uses php for frontend, erlang is used for chat. THIRFT makes interaction between the languages easy.VARNISHFacebook uses Varnish to serve photos and profile pictures, handling billions of requests every day. Like almost everything Facebook uses, Varnish is open sourceFacebook also uses a CDN to help serve static content. More or less each and every time they cross some milestone the engineers are going to redesign the whole application,develop some new and create patch for the older one.These are repetitive processes.There are lots of processes involved which helps Facebook to scale up and maintain the large audience.FOR TWITTERTwitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 SecondsPush Me Pull MePeople are creating content on Twitter all the time. The job of Twitter is to figure out how to syndicate the content out. How to send it to your followers.The real challenge is the real-time constraint. Goal is to have a message flow to a user in no more than 5 seconds.Delivery means gathering content and exerting pressure on the Internet to get it back out again as fast as possible.Delivery is to in-memory timeline clusters, push notifications, emails that are triggered, all the iOS notifications as well as Blackberry and Android, SMSs.Twitter is the largest generator of SMSs on a per active user basis of anyone in the world.Elections can be one of the biggest drivers of content coming in and fanouts of content going out.Two main types of timelines: user timeline and home timeline.A user timeline is all the tweets a particular user has sent.A home timeline is a temporal merge of all the user timelines of the people are you are following.Business rules are applied. @replies of people that you don’t follow are stripped out. Retweets from a user can be filtered out.Doing this at the scale of Twitter is challenging.Pull basedTargeted timeline. Things like Login or Sign up and home_timeline API. Tweets delivered to you because you asked to see them. Pull based delivery: you are requesting this data from Twitter via a REST API call.Query timeline. Search API. A query against the corpus. Return all the tweets that match a particular query as fast as you can.Push basedTwitter runs one of the largest real-time event systems pushing tweets at 22 MB/sec through the Firehose.Open a socket to Twitter and they will push all public tweets to you within 150 msec.At any given time there’s about 1 million sockets open to the push cluster.Goes to firehose clients like search engines. All public tweets go out these sockets.No, you can’t have it. (You can’t handle/afford the truth.)User stream connection. Powers TweetDeck and Twitter for Mac also goes through here. When you login they look at your social graph and only send messages out from people you follow, recreating the home timeline experience. Instead of polling you get the same timeline experience over a persistent connection.Query API. Issue a standing query against tweets. As tweets are created and found matching the the query they are routed out the registered sockets for the query.High Level for Pull Based TimelinesTweet comes in over a write API. It goes through load balancers and a TFE (Twitter Front End) and other stuff that won’t be addressed.This is a very directed path. Completely precomputed home timeline. All the business rules get executed as tweets come in.Immediately the fanout process occurs. Tweets that come in are placed into a massive Redis cluster. Each tweet is replicated 3 times on 3 different machines. At Twitter scale many machines fail a day.Fanout queries the social graph service that is based on Flock. Flock maintains the follower and followings lists.Flock returns the social graph for a recipient and starts iterating through all the timelines stored in the Redis cluster.The Redis cluster has a couple of terabytes of RAM.Pipelined 4k destinations at a timeNative list structure are used inside Redis.Let’s say you tweet and you have 20K followers. What the fanout daemon will do is look up the location of all 20K users inside the Redis cluster. Then it will start inserting the Tweet ID of the tweet into all those lists throughout the Redis cluster. So for every write of a tweet as many as 20K inserts are occurring across the Redis cluster.What is being stored is the tweet ID of the generated tweet, the user ID of the originator of the tweet, and 4 bytes of bits used to mark if it’s a retweet or a reply or something else.Your home timeline sits in a Redis cluster and is 800 entries long. If you page back long enough you’ll hit the limit. RAM is the limiting resource determining how long your current tweet set can be.Every active user is stored in RAM to keep latencies down.Active user is someone who has logged into Twitter within 30 days, which can change depending on cache capacity or Twitter’s usage.If you are not an active user then the tweet does not go into the cache.Only your home timelines hit disk.If you fall out of the Redis cluster then you go through a process called reconstruction.Query against the social graph service. Figure out who you follow. Hit disk for every single one of them and then shove them back into Redis.It’s MySQL handling disk storage via Gizzard, which abstracts away SQL transactions and provides global replication.By replicating 3 times if a machine has a problem then they won’t have to recreate the timelines for all the timelines on that machine per datacenter.If a tweet is actually a retweet then a pointer is stored to the original tweet.When you query for your home timeline the Timeline Service is queried. The Timeline Service then only has to find one machine that has your home timeline on it.Effectively running 3 different hash rings because your timeline is in 3 different places.They find the first one they can get to fastest and return it as fast as they can.The tradeoff is fanout takes a little longer, but the read process is fast. About 2 seconds from a cold cache to the browser. For an API call it’s about 400 msec.Since the timeline only contains tweet IDs they must “hydrate” those tweets, that is find the text of the tweets. Given an array of IDs they can do a multiget and get the tweets in parallel from T-bird.Gizmoduck is the user service and Tweetypie is the tweet object service. Each service has their own caches. The user cache is a memcache cluster that has the entire user base in cache. Tweetypie has about the last month and half of tweets stored in its memcache cluster. These are exposed to internal customers.Some read time filtering happens at the edge. For example, filtering out Nazi content in France, so there’s read time stripping of the content before it is sent out.High Level for SearchOpposite of pull. All computed on the read path which makes the write path simple.As a tweet comes in, the Ingester tokenizes and figures out everything they want to index against and stuffs it into a single Early Bird machine. Early Bird is a modified version of Lucene. The index is stored in RAM.In fanout a tweet may be stored in N home timelines of how many people are following you, in Early Bird a tweet is only stored in one Early Bird machine (except for replication).Blender creates the search timeline. It has to scatter-gather across the datacenter. It queries every Early Bird shard and asks do you have content that matches this query? If you ask for “New York Times” all shards are queried, the results are returned, sorted, merged, and reranked. Rerank is by social proof, which means looking at the number of retweets, favorites, and replies.The activity information is computed on a write basis, there’s an activity timeline. As you are favoriting and replying to tweets an activity timeline is maintained, similar to the home timeline, it is a series of IDs of pieces of activity, so there’s favorite ID, a reply ID, etc.All this is fed into the Blender. On the read path it recomputes, merges, and sorts. Returning what you see as the search timeline.Discovery is a customized search based on what they know about you. And they know a lot because you follow a lot of people, click on links, that information is used in the discovery search. It reranks based on the information it has gleaned about you.Search and Pull are InversesSearch and pull look remarkably similar but they have a property that is inverted from each other.On the home timeline:Write. when a tweet comes in there’s an O(n) process to write to Redis clusters, where n is the number of people following you. Painful for Lady Gaga and Barack Obama where they are doing 10s of millions of inserts across the cluster. All the Redis clusters are backing disk, the Flock cluster stores the user timeline to disk, but usually timelines are found in RAM in the Redis cluster.Read. Via API or the web it’s 0(1) to find the right Redis machine. Twitter is optimized to be highly available on the read path on the home timeline. Read path is in the 10s of milliseconds. Twitter is primarily a consumption mechanism, not a production mechanism. 300K requests per second for reading and 6000 RPS for writing.On the search timeline:Write. when a tweet comes in and hits the Ingester only one Early Bird machine is hit. Write time path is O(1). A single tweet is ingested in under 5 seconds between the queuing and processing to find the one Early Bird to write it to.Read. When a read comes in it must do an 0(n) read across the cluster. Most people don’t use search so they can be efficient on how to store tweets for search. But they pay for it in time. Reading is on the order of 100 msecs. Search never hits disk. The entire Lucene index is in RAM so scatter-gather reading is efficient as they never hit disk.Text of the tweet is almost irrelevant to most of the infrastructure. T-bird stores the entire corpus of tweets. Most of the text of a tweet is in RAM. If not then hit T-bird and do a select query to get them back out again. Text is almost irrelevant except perhaps on Search, Trends, or What’s Happening pipelines. The home timeline doesn’t care almost at all.MonitoringDashboards around the office show how the system is performing at any given time.If you have 1 million followers it takes a couple of seconds to fanout all the tweets.Tweet input statistics: 400m tweets per day; 5K/sec daily average; 7K/sec daily peak; >12K/sec during large events.Timeline delivery statistics: 30b deliveries / day (~21m / min); 3.5 seconds @ p50 (50th percentile) to deliver to 1m; 300k deliveries /sec; @ p99 it could take up to 5 minutesA system called VIZ monitors every cluster. Median request time to the Timeline Service to get data out of Scala cluster is 5msec. @ p99 it’s 100msec. And @ p99.9 is where they hit disk, so it takes a couple hundred of milliseconds.Zipkin is based on Google’s Dapper system. With it they can taint a request and see every single service it hits, with request times, so they can get a very detailed idea of performance for each request. You can then drill down and see every single request and understand all the different timings. A lot of time is spent debugging the system by looking at where time is being spent on requests. They can also present aggregate statistics by phase, to see how long fanout or delivery took, for example. It was a 2 year project to get the get for the activities user timeline down to 2 msec. A lot of time was spent fighting GC pauses, fighting memcache lookups, understanding what the topology of the datacenter looks like, and really setting up the clusters for this type of success.This is the whole story of two internet social networking giants. The scaling up process is very rigorous .Similarly all other large audience based platforms are using these type of techniques to manage. But stillWHERE THERE ARE ENGINEERS,YOU HAVE LOTS OF OPTIONS AND VERITIES.

My first impression of ‘America’ was formed more than 60 years ago. To be precise, it was February 7, 1957.I was a passenger on the 20,000-ton Cunard liner RMS Carinthia, a few months shy of my 14th birthday. With my mother and sister, I was en route from Liverpool to New York City where we would board a train to join my step-father in Toronto and start a new life in Canada.RMS Carinthia - Docking in MontrealOur passage had been a bit rough. We had run into a North Atlantic gale halfway across the Atlantic and many passengers had spent the rest of the voyage in their bunks. I had been quite sick myself, but Lenny the cabin steward who looked after the cabin I shared with three other gentlemen, had given me a tip on how to overcome it. The trick he said was to get up on deck, out in the fresh air where I could feel the wind in my face and watch the horizon. I tried it, not without some trepidation, and it worked. After watching Carinthia’s bows crashing into 30-foot waves for a few minutes, I was having too much fun to be seasick. It was better than the roller-coaster at the Blackpool Pleasure Beach.We had made landfall in Halifax, where most of our fellow passengers disembarked, but my mother had decided we would continue on to New York City. The night before we were to arrive, Lenny asked if we wanted to be woken up in time for the ship’s entry into New York harbour. I was excited at the prospect because I had read as much as I could about America and Canada in preparation for our journey. For months I had scrutinized maps of Ontario, New York, Nova Scotia and Quebec. I had pored over my Grandma’s collection of old “Life” and “Look” magazines for clues about the way people dressed, the cars they drove and the kind of food they ate.My sister and mother seemed totally indifferent to anything that would require them to get out of bed at 5:00 AM but I wasn’t going to miss any part of something I already knew would be one of those life-changing moments we often read about, but can all too easily miss. At a little before the appointed time, Lenny woke me up with a cup of tea and a small package he placed on the little shelf beside my bunk. It was toast and marmalade wrapped in a linen napkin.The tourist-class cabins on Carinthia weren’t quite as spacious as this picture from a Cunard brochure would suggest.I pulled on my new long trousers (they were my first pair, bought especially for the trip), and a thick woolly jumper over my liberty vest. I didn’t know back then that Americans called them ‘sweaters’ and ‘undershirts’. With a pair of warm socks, waterproof shoes and the Navy-pattern duffel coat my mother had bought “because the winter would be cold in Canada” I was ready for the new world. I stuck the toast in my pocket and headed for the observation deck.While I was learning how to defeat ‘mal de mer’ Lenny had told me that I should get as far for’ard as I could and I had found a perfect spot on the narrow observation deck just below the ship’s bridge. It was the perfect vantage point to watch the ship’s progress through the harbour. There was, alas, nothing much to see except for a lighter patch in the sky ahead and a sprinkling of stars glinting above us.I was shivering a bit, partly because I was excited, but mostly because I was bloody freezing! Winters in the North of England aren’t exactly balmy, but nothing in my reading had prepared me for the kind of cold that freezes the breath in your lungs and takes away your will to live. Much as I hated to give up my private panopticon, I trundled off to find somewhere warm.Lenny had told me that tea and cocoa would be available in the ship’s passenger lounges and nobody challenged me when I found myself in what was probably the First-Class Lounge. I was the only person there. I may not have got the worm, but it did seem that an ‘early bird’ would have no trouble scrounging a cuppa and a pocketful of biscuits.I found a spot where I could see what was happening outside. It was still very dark but through the windows, I began to see distant lights slowly sliding by on the starboard side of the ship. There were only a few at first and they were still distant, but I knew we were close to shore. The sky was clear with a little fog or mist on the water. I think I must have nodded off for a few minutes because, by the time I looked again, it was almost six. I could already make out the lights of Manhattan. There were so many that the very air seemed to glow.As the minutes passed the skyline became better defined and I could pick out individual buildings, though I had no idea what they all were. I wanted to ask someone, but as far as I could tell, I was the only passenger up early enough to brave the cold. As the ship slid into the inner harbour I could even see what looked like neon signs, but they were still too far off to read.Suddenly, I saw it.She wasn’t as big as I expected, but in her green robes, she was unmistakable. It was indeed, “Liberty Enlightening the World”. Actually, I thought, she was the one who needed enlightening. She certainly wasn’t as well-lit in those days as she is now. Not that it mattered. What counted was that, as drowsy as she was so early in the day, she was welcoming me!As the ship manoeuvred around the tip of the island and entered the Hudson, I began to see more and more… far too much to take in and far too much to remember clearly.The one thing I remember with absolute clarity was a huge sign that spelled out in blinking neon the word “AUTOMAT”. I thought there must be a spelling error. Maybe it was supposed to read “automatic” and was there a word missing?By then, I could make out individual vehicles. Even at 6:00 AM on a Sunday morning, there was more traffic than I usually saw at home in a week. A couple of tugs moved in to begin nudging our ship into place at the huge pier where it would tie-up. Looking up and down the river, there were at least a half-dozen other ships berthed at similar piers.Cunard’s Chelsea Piers - circa 1955I didn’t know it at the time, but the Chelsea Piers had been greeting Cunard ships and their passengers for more than half a century. They are gone now and until the site was redeveloped, for many years there was nothing left but a few pilings, like rotten teeth in the river’s mouth.But in 1957, the cavernous terminals where passengers met with customs and immigration officials as they claimed their baggage, were a going concern. I remember well the frenzied rush of disembarkation and the search for the steamer-trunks containing the odds and ends my mother believed were essential to immigrant life. There were people everywhere… and almost as many pigeons.By the time my mother had presented our papers and arranged for our baggage to be forwarded, it was mid-morning… a bright, windless day, but cold. We had hours and hours to kill before our train was scheduled to leave Pennsylvania Station. I couldn’t tell you how we got there but I remember walking up an almost empty Fifth Avenue, looking in store windows and trying to absorb it all. There was so much.I remember being taken into a restaurant where I was allowed to drink a cup of coffee with whatever it was we ordered from a menu longer than “War and Peace”. My sister was annoyed about something or other and Mum marvelled at how much everything cost.But the people we met were wonderful. In those days, or at least on that particular day, there were policemen on almost every corner we passed and they all smiled at us as we walked by. They had on their long, blue winter overcoats with a double row of shiny buttons and they all seemed to carry a night-stick. I thought of them as truncheons and when we asked a policeman how far we had to go before we got to Radio City, he twirled his stick and told us we were only “a few blocks” away. He had a different accent from the ones we had heard on the ship but he called my Mum “Ma’am” which I thought was very respectful. Looking back, I now realize that the waitress who looked after us at the restaurant was surprisingly patient as we asked what was the difference between a “hot dog” and a “hamburger” and if they did fish and chips. She called my sister, “honey” when she brought her food.Of course, I had no idea what “a block” was and thought “Radio City” had something to do with the Wizard of Oz. My Mum had been told about it by her sister who had married a GI during the war and lived in Huntsville, Alabama.She had decided that it would be the ideal place to spend our Sunday afternoon. My Grandma had given me some American money before we left Blackpool, the town in which I had been born and raised. It was only a few dollars but it was burning a hole in my pocket. I don’t remember much about Radio City except the concession stand. Like everything else I’d seen of America, it was huge and even a little intimidating. Unlike the snack bars in England, it offered an incredible range of exotic treats, all of which were utterly foreign.I had never seen popcorn but I didn’t like the sound of it and it smelled funny but I knew what chocolate was. I bought a Tootsie Roll, a Hershey’s chocolate bar, and a little plastic tube that offered up something called PEZ when you flipped up Mickey Mouse’s head. Best of all was a package labelled “M&M”. It contained peanuts coated in chocolate. My money was all gone but I didn’t care.I loved America! And I didn’t even know about the dancing girls!I don’t know how much the tickets cost either, but it was quite a show. There were jugglers and tumblers and two comedians who told jokes I didn’t get and then there were the Rockettes.I never did figure out how many of them there were. Every time I started counting, they would change formation or begin a new manoeuvre. I’m sure they were all lovely, but I wouldn’t know. Our seats were too far back to be able to pick out individual faces. All I remember is an endless blur of long and very glamorous legs.We had already got far more than we expected but there was more to come. There was a movie too! A John Wayne feature called “The Wings of Eagles”. It’s not terribly good, but I idolized John Wayne and it had aircraft carriers and airplanes.By the time we got out, it was late afternoon and already starting to get dark. I remember we walked over to Rockefeller Centre and watched the skaters and stopped somewhere for tea.Eventually, it was time for the train. Mum bundled us all into a big yellow taxi and asked the driver to take us to “Pennsylvania Station”. He promptly asked us where we were from.I don’t really remember much about the rest of the trip. I was asleep as soon as we found our seats on the train and when I woke up, we were in Toronto and my Dad was waiting for us at the station.Compared with New York, my first impression of Toronto was less than favourable. New York had been like the Circus, Christmas and my birthday all rolled up into one glorious day. By contrast, Toronto was more like the first day of term.