Hast thou entered into the treasures of the snow? Job 38:22, The Bible.

It is worth thinking about which “New Year” we are speaking of. Today we have the new calendar year, but there is also the new fiscal year, the new agricultural year, and probably others.

The situation we find in Bible times is similar: there is the regnal new year from which the length of the king’s reign is measured, the calendar new year, the religious new year (Exodus 12:2 – why isn’t Rosh Hashanah in April? Now you know.), the new year for trees (determines when you can start eating the fruit), and probably others.

Some of the New Year’s Day happenings are not necessarily or even likely the same type of year but I’m still going to appropriate them all for this post.

Noah and the End of the Great Flood

And it came to pass in the six hundredth and first year, in the first [month], the first [day] of the month, the waters were dried up from off the earth: and Noah removed the covering of the ark, and looked, and, behold, the face of the ground was dry. Genesis 8:13, The Bible

On the first day of the new year, Noah looked and found that the ground was dry and he was able to remove the covering from the ark. This reminds me that God is faithful to His promises. He had promised Noah His protection during the flood and now Noah is safely on the other side of the destructive deluge.

The new year is a good time to reflect on God’s goodness to each of us and to realize that He is always faithful to His promises, even when times and circumstances look dark and beyond hope.

Setting up the Hebrew Tabernacle

And it came to pass in the first month in the second year, on the first [day] of the month, [that] the tabernacle was reared up. Exodus 40:17, The Bible

The tabernacle serves as a reminder that God has chosen to be involved with life on this planet. He is not, as some have supposed, one who would up the universe in time past and since that initial infusion of energy left us all to our own devices. In describing the tabernacle and the furnishings that were to be provided for it, God indicates “And there I will meet with thee, and I will commune with thee from above the mercy seat” (Exodus 25:22, The Bible).

The new year is an appropriate time to remember that God has expressed His desire to have a relationship with us. In fact, he went as far as to send Jesus Christ into this world, knowing he would suffer and be cruelly mistreated so that he could righteously offer the forgiveness of sins and welcome His wayward creatures back into a personal relationship with Himself.

The Word of the Lord comes to Ezekiel

And it came to pass in the twenty-seventh year, in the first [month], on the first [day] of the month, [that] the word of the LORD came to me… Ezekiel 19:17, The Bible

The word of the LORD came to Ezekiel many times and the day and month are very often given. This particular occasion concerns one of the greatest prophetic events in history: The battle of Tyre. This prophecy, left partially fulfilled for hundreds of years, was completely fulfilled by Alexander the Great in 322 BC, serving as a lasting monument to the authority and reliability of God’s word. God gives us prophecy because He wants us to have confidence in Him and believe Him when we see things come to pass that He has indicated beforehand (Isaiah 43:10, John 14:29).

The new year is a great time to to understand that God wants to be believed. When people characterize my faith as “blind faith” and describe it as being without reason, they do so without really looking into the matter. True Biblical faith is based on well-placed confidence in the prophetic-historical statements of the Bible as well as our personal experience with the Living God in our own lives. There’s nothing blind or unreasoning about it and I don’t find God asking for something unreasonable. In fact, he encourages us to reason with Him:

“Come now, and let us reason together,” Says the LORD, “Though your sins are like scarlet, They shall be as white as snow; Though they are red like crimson, They shall be as wool. Isaiah 1:18, The Bible

So this New Year’s Day, reflect, remember and reason. God is good, He is faithful, He wants a relationship with us and He says Believe on the Lord Jesus Christ and you will be saved, Acts 16:31, The Bible.

I would be extremely hesitant to quote anything pithy or witty sounding from M. Muggeridge without pages of context around why it was necessary to do so. Perhaps even tricking a man on the street into reading the desired MM quote and then quoting the M.O.T.S. instead of MM would be preferable.

From the virgin birth

Seeing Through The Eye, page 207

To the resurrection

Seeing Through The Eye, page 22

The poison behind the nice sounding witticisms is laid bare.

• Is it Christmas – http://isitchristmas.com – Funny as it tracks the timezones across the globe, really only has much to do on one day per year though. But that one day is coming up soon, so start checking in daily.
• Is it Dark Outside – http://isitdarkoutside.com – Whopping funny. Works every day of the year.
• Umbrella Today – http://umbrellatoday.com – I guess they are hawking an iPhone app, but I really enjoyed how the interface pulls you in a little bit at a time. And if you visit for the first time on a rainy day (or just put in a Seattle, Washington zip code), that rainy-day graphic is stunning.

It’s all about hiding the complexity. Do something hard with your software but don’t make the users suffer just you did.

• Bad disktype code ORA-99xxx
• CTXUSER not enabled ORA-010xxx
• Flagellular misfire ORA-910xxx
• Nascent order lost code: ORA-82xxx
• Ferrule injector not found: ORA-14xxx

Of course it’s just a perl script and the system had nothing to do with Oracle. Just one of those little things that can be done to throw snoopers off the trail.

Some of the best stuff was just hearing about how other folks are using Hadoop. I also enjoyed hearing about the sizes of other people’s big-data problems. There were three tracks, so I only heard 1/3 of what took place, but here are some notes on what I did hear after the break.

It was a great day, a long day, glad I went.

Yahoo and Hadoop – Eric14

Yahoo search assist is now build with Hadoop based on 3 years of log data. It is a 20 step map-reduce job and it runs in about 20 minutes on their production cluster. This is down from a 26 day run (did I get that right?) in the pre-Hadoop days. Wow. Y! has several Hadoop clusters and the biggest is 4K nodes, 16Pb disk, 64 Tb RAM, 32K cores running Hadoop 18.3. Wow. I’m drooling over that.

Yahoo! folks are the biggest Hadoop committers and Y! sounds solidly behind the thing just like they have been since its inception. That was great to hear. Just about every zone of the Yahoo! home page has some contribution from a map-reduce job at some point.

Some stuff to keep an eye out for

• Zebra – a column oriented data store on hdfs
• Ooozie – a workflow and scheduling piece for map-reduce
• Mumak simulator – need to look that one up

Facebook – Ashish Thusoo

Fb gets 4Tb of compressed data per day with a 6-7x compression factor. That’s a lot of data. They separate production and ad-hoc jobs on to different clusters and use hive-replication to get the data in both places. This is something that is not publicly available yet but they have plans to open-source it when it is ready. Their big cluster is 4800 cores, 5.5 Pb disk, 12Tb disk per node. They run 7500 hive jobs per day, 95% of all jobs are hive, and this cluster runs 80,000 compute hours per day.

Visa – Joe Cunningham

Visa has had Hadoop in their research department for about 9 months and is doing some interesting things with it. They certainly have a big-data problem: 28 million merchants, 1.4 million ATM machines, 500 million accounts, 100-130 million transactions per day (8k per second). The Visa Net portion of a transaction approval happens in 50 ms and Visa Net has 2 seconds downtime per year.

Visa uses lots of FOSS in the enterprise so their interest in Hadoop came about in a perfectly natural way. One of the things the current research does is to produce fraud models that can be used in the real-time system. These models are produced off-line and it currently takes about 1 month to generate a fully-functional model. This involves sampling data, moving data, compressing data, etc. With Hadoop a fully-functional model can be generated in 13 minutes. Lots of the research happens on synthetic data and Visa has created map jobs to generate synthetic transaction data at the rate of 2 years of synthetic transactions in 6 hours.

Rackspace – Stu Hood

Stu talked most about the email and app hosting division of Rackspace but there are a lot of interesting things happening in Rackspace, both in the mail hosting division and in other divisions. The mail hosting division generates a lot of logs and needs to be very proactive in finding and surfacing delivery problems and dealing with spam and malware. They generate 300Gb/day of logs and have a 6 month window for analysis. There was some interesting talk of using map-reduce to generate a Lucene intermediate index format and then using Solr to merge it up into a searchable index. The time window target is to be able to query on data within 15 minutes of the event and to query on any dimension.

They either are or will be using Scribe to ingest data (syslog data?) directly into HDFS. I need to look into Scribe more as I had sort of forgotten about it. It could help solve some pressing issues.

JP Morgan Chase

One very interesting point the speakers brought out is that once your transactional data gets to be over 1 day old it is usually entirely static. But still it sits there provisioned and protected in your enterprise RDBMS transactional data store — read only. We can do better than that!

HDFS Security – Owen O’Malley, Yahoo!

There is lots of work going on to secure Hadoop Namenode, Jobtracker and Datanode. This is super important to our clients too and so we are very interested in seeing this succeed. Some of the major points:

• Namenode will give the user a Kerberos token to access blocks on a data node
• Only the user of a map-reduce job will be able to modify or kill it
• Map-reduce jobs run as the user
• Task working directory visibility limited to the user
• Only the right reduce tasks can see teh map outputs (secure shuffle)
• Encryption optional
• SPNEGO = permissions everywhere

In addition to the security work Owen talked some about how to make backward compatible APIs and how to mark them as such. The need for this grows as the project grows larger ecosystems. File system compatibility is also important. We know this right well as we got stuck on the bogus Hadoop 0.19 version that no one even mentions in public. We couldn’t go back to 0.18 and 0.20 wasn’t stable yet. Stuck stuck stuck.

VAIDYA

There are 165+ tunable parameters in Hadoop. Changing one usually has impact on the others. Vaidya uses a set of rules to analyze map-reduce history (after a job has run) using the user history and the job conf. It can measure things like reducer balance, check for compressed intermediate data, determine whether a combiner would be likely to help, and many other things. This is a lot of the stuff we do by hand when evaluating a map-reduce job. You could see something like this being used as a part of the certification that a job is ready to run on your production cluster. JIRA 4179.

eBay Streaming Architecture – Neel Sundaresan

eBay users spend $1400 per second. A vehicle is sold every 2 minutes. There are about 10 million new listings per day and 250 million users. The typical item on eBay is not catalogable (is that a word?) as most items are one-of-a-kind. So as compared to a Netflix type recommender system where you might need a 100k x 100k matrix, the eBay recommender would be an extremely large, sparse array. Neel called this a “long-tail” type system. There are terabytes of new transaction and user session data per day. eBay Research Labs studies this data to determine behavior in a number of dimensions, figures out how to perform A/B testing experiments, and does trending analysis.

The streaming architecture (Mobius) has a query language (MQL) that gives an SQL like interface but adds a start and end section to the query to give the window of data to operate on.

High Availability Hadoop – webContext

I had some problems understanding this talk at the beginning, but warmed up to it as it went along. There were lots of interesting products and tools mentioned that I should look into more.

• Hotslice to monitor job exit status
• Network bonding with LACP
• DRBD for disk mirroring
• Linux HA from linux-ha.org
• Spacewalk – an open source system like RedHat Satellite server

These guys have two machines running the Namenode using LinuxHA and DRBD on a virtual IP address. In addition to that they use the REST interface to the NameNode to /getimage and /getedit once per hour and make additional backups. They have had 6 NN failures in the last 18 months, 3 planned, 3 unplanned and have about a 15 second rollover to the backup. That is intense. Read more about this configuration on the cloudera blog.

RADFS

This is a set of patches in progress to make a low-latency HDFS. This is important work for things like hBase. There are some huge inefficiencies in sockets, file handles, and network usage for small positioned reads. There is still a lot of work to do on these patches before it could reasonably take over for the current HDFS approach. Interesting things to think about. JIRA HDFS-516.

Automatic Problem Diagnosis – CMU Team

The CMU team is using the M45 cluster that Y! has made available to several universities. They have some really nice visualizations of problems occurring in the cluster that come from looking at 64 different metrics in collected log files. One metric I thought especially interesting was heartbeat date skew between the master and slave. This work is being tracked in JIRA Chukwa – 94.

And be it ever remembered that Christians are not members of a club, of a sect or of an association; they are members of a body, each connected with allo, and all connected by the fact of the indwelling of the Holy Spirit with the risen and glorified Head in Heaven. This is an immense truth, and the practical carrying out of it will cost us not only all we have, but all we are. There is no place in all the universe where self will be so pulled to pieces as in the Assembly of God. And is it not well? Is it not a powerful proof of the divine ground on which that Assembly is gathered? Should we not be glad to have our hateful self thus pulled to pieces? Shall we or ought we run away from those who do it for us? Are we not glad–do we not often pray to get rid of self? And shall we quarrel with those who are God’s instruments in answering our prayers? True, they may do the work roughly and clumsily, but never mind that. Whoever helps me to crush and sink self does me a kind turn, however awkwardly he may do it. One thing is certain, no man can ever rob us of that which, after all, is the only thing worth having, namely Christ.

–C.H.M., Short Papers, Self Surrender, p. 18, Believers Bookshelf, pub 1995.
(capitalization his, emphasis mine)

I have probably read that several times before. It’s interesting how the experiences we go through color how (and how much) we understand a thing like that.

The big draw for us was the Tom Williams Quintet. I never really got a good picture of all five guys, but here’s some flavor, featuring the man himself.

We had been wanting to go and hear Tom play for quite some time, but this time the scheduling and baby-sitting just worked out and we were off. Tom, Cindy and I all went to high school together, were all in band, lived through the marching, playing in the rain and snow, and the other vagaries of high school. Some others from the old high school band group were there too: Betty (who tipped us off about this gig in the first place, Darren, and Jim. It was great to see everyone and several other folks in the audience figured out that we all knew each other and were asking questions about it. Lots of fun.

The TWQ was the event for the evening though and they played three very nice sets. The music was just great and they were really connecting with the crowd in the close quarters of the pavilion cafe. There was a very nice blend of tunes from “The Theme”, to Miles Davis, to a couple of Tom’s own compositions. Every set had a nice flow and we stayed for the whole deal.

Here’s a picture worthy of reminiscing over. Thinking about how “local guy makes good” and “we knew him back when…”. But truth be told, Tom “made good” a long time ago. If you don’t have his CDs, zip on over to amazon and check out Straight Street and Introducing Tom Williams, both of which are very nice.

raspberry harvest

We have been getting a tremendous harvest from our little row of raspberries this year.