Today when I reminded a friend not to lose hope in Indian cricket (after recent whitewash in England and Australia), another friend commented, 

प्रसून जी , यह भारतीय क्रिकेट है ..यहाँ पर हर विक्टरी दुसरे दिन पुरानी हो जाती है ...You have to perform at your best ... After all they are getting unexpected money. They should deliver the goods as per citizens expectations...

My reply to my friend was, 

Agreed, they need to perform... And here's my explanation of our recent performance... After Aussie dominance ended a few years ago with Warne/Gilchrist/McGrath/others retiring, we've hit a period where most top Test teams are equal (Eng/Aus/SA/SL/Ind)... As a result, the #1 Test team in last 2-3 years has been the team that played most games at HOME. And I expect this trend to continue. Here's some proof: India became #1 by playing all strong team at home in 2008-10 period... So did England in 2010-11 and became #1... now look at what Pak is doing to Eng outside Eng (down by 2-0)... Aussies won at home 4-0 against us in 2011/12 but India beat Aus 2-0 not too long ago (at HOME)...

---

Now, don't misunderstand me... I'm not saying India's terrible performance is okay. It is NOT. Indian fans deserve to be pissed. Recent performance is terrible. Several innings defeat. No one firing. Giving up so easily. Really bad. But I'm highlighting the fact that the current years will cause joy and heart break for fans depending on where their teams are playing (Home games will bring JOY and Away games will BREAK THEIR HEARTS). Cricket fans, let us prepare ourselves for this. Our team's Test cricket performance will more or less depend on where they're playing (checkout ICC's Future Tours Program). Test cricket ranking won't mean much! Wasim Akram feels the same way.


Here are some charts that explain my point of view... 

1) See how India played more tests at home between 2008 and 2010. In 2011/12, they've mostly AWAY games. Similarly, in 2011 England played mostly at HOME

---

2) Notice how we see more RED (losses) than GREEN (wins) in the charts below, India and England's performance in AWAY games

---

3) Notice how we see more GREEN (wins) than RED (losses) in the charts below, India and England's performance in HOME games

Rahul Dravid said this at the Bradman Oration. It stuck with me...

One of the things, Bradman said has stayed in my mind. That the finest of athletes had, along with skill, a few more essential qualities: to conduct their life with dignity, with integrity, with courage and modesty. All this he believed, were totally compatible with pride, ambition, determination and competitiveness. 

Many of my friends and family members ask me how I continue to follow Indian cricket when we are doing so bad (whitewash in Test Cricket in England in late 2011 and now in Australia (well almost there))...


My answer is that there are ups and downs in sports and my love for the team and the sport isn't based on convenience, it isn't based on my team doing well all the time. Sometimes we do well (hey, we just won the World Cup 9 months ago and were #1 Test team just a few months ago) and sometimes we don't (pathetic test cricket performance recently).






I am a fan of Indian cricket. I don't give up. I won't give up my faith, my love for the team, my love for the sport, my RELIGION.

I just hope that we get through this phase without much pain and get back up and running... Go India!

We are a global management consulting firm and are looking for data scientists in our team in New York/Washington DC and Gurgaon/Chennai (India). There are full-time and internship (New York) opportunities. There are multiple positions including developing complex models in healthcare, pricing and optimization. If interested, drop me an email at prasoonsharma at gmail dot com.

I want to learn the heavy-weight of Statistical softwares - SAS. It seems like the default choice for high-end statistics and I want to understand why.

I'm working in the healthcare practice in our firm and want to analyze claims and credit data (Terabytes, 50M+ records). Traditional ways (SQL) are limiting and desktop statistical softwares like R and Stata aren't suitable for such large data analysis. Other contenders (Matlab) don't seem to be in the same league.

So, its time to take a deep dive into SAS.

---

I'm looking for some advice to create a learning plan...

Good books

• I like learning by examples and found this on Amazon - Learning SAS by Example: A Programmer's Guide by Ronald P. Cody)
• I know some R, so this might be interesting - SAS and R: Data Management, Statistical Analysis, and Graphics by Ken Kleinman and Nicholas J. Horton
• Others?

Good tutorials

• I like video tutorials with examples e.g Statistics202.
• I also like tutorials from a programmer's perspective better
• Anything for SAS out there?

Good blogs

Premature optimization is the root of all evil

Donald Knuth

• England is at #3 and Sri Lanka are at #4 in ICC test ranking
• England plays well against Sri Lanka at home, and Sri Lanka has an edge in their home conditions. And the overall record is well balanced

Let the games begin!

Leave a comment if you're interested. I will contact you directly.

Team: Prasoon, Khaled, James

We've introduced R in the organization!

It is running along with the heavy weights of statistical analysis like SAS, SPSS, Matlab. Here's what we did and how we did it...
LESSONS LEARNED
Technical

1. You need to be careful in running it in a shared environment, where it can use all your CPU and memory if it runs for long
2. Don't forget to write unit tests using RUnit for your R code
3. Capturing exceptions from R and dealing with them properly (appropriate message to users)
4. rsruby installation documentation is good but needs a few tries depending on your Linux distribution
5. rsruby does not run on Windows (wasn't a problem for us as we run our web applications on LAMP)

Process

1. User acceptance testing: If you are transforming an Excel-based model into web-version, it is critical to have a fully working example of the Excel model to replicate it in R/other statistical packages
2. Overcoming the challenges of using new open source software in enterprise: Like most enterprise IT shops, we are used to commercial software as well and the idea of using open source software to do serious work is limited to the most popular open source frameworks like Drupal, Ruby on Rails, Linux. We positioned R as an add-on to our LAMP environment and got a separate virtual server dedicated to it as it is memory hungry.

Goal: Plot interesting charts from this year's Fortune Global 2000 list.

Download data in CSV format

I have big data problems.

I need to analyze 100s of millions of rows of data and tried hard for 2 weeks to see if I can use R for this. My assessment so far from the experiments...

1) R is best for data that fits a computer's RAM (so get more RAM if you can).

2) R can be used for datasets that don't fit into RAM using Bigmemory and ff packages. However, this technique works well for datasets less than 15 GB. This is in line with the excellent analysis done by Ryan. Another good tutorial for Bigmemory.

3) If we need to analyze datasets larger than 15 GB, then SAS, MapReduce and RDBMS :( seem like the only option as they store data on file system and access it as needed.

Since MapReduce implementations are clumsy and not business friendly yet, I wonder if its time to explore commercial analytics tools like SAS for big data analytics.


References
http://www.bytemining.com/2010/07/taking-r-to-the-limit-part-i-parallelization-in-r/
http://www.austinacl.blogspot.com (image)

Their love isn't apparent. You see it in their appreciation of each other's culture, entertainers and sportsmen, and even politicians e.g. Indian movies and movie stars are popular in Pakistan and Benazir was popular in India. You also see their love when Indian and Pakistani people meet in a 3rd country, where their media and politicians don't brainwash them. Their they get to know each other really well. I'm from India and one of my best friends is from Pakistan.

Their fights on the other hand are much more evident. Their fights have translated into wars (1965, 1971, 1999), political battles (UN, US/USSR, local politics), movies (Border) and sport rivalries (Cricket, Hockey). We compete in sporting events with fanaticism and millions follow it closely.

One such battle took place in Cricket World Cup 2011. 

It started with Pakistan's inspiring performance in preliminary rounds and then them knocking out West Indies (the weakest team in quarter finals) in quarter finals easily.

India was expected to make it to final eight and did. India had to play Australia (4 time world champions) in quarter finals. Australian cricket isn't as strong anymore with many superstars retiring Gilchrist, Hayden, Warne and McGrath). In a thrilling contest, India beat Australia in Motera, Ahmedabad. Indian bowlers restricted Aussies to 260 and the batsmen scored 261.

This setup up a thrilling contest between India and Pakistan for a place in finals.


I was in India during Cricket World Cup 2011 and got to witness this battle in Mohali, Chandigarh. This is my travelog for this particular contest.

---

Mar 29
- Baroda to Mumbai by air (12:30pm - 2pm)
- Pick brother from office at 6pm and head to the airport
- Dinner at Mumbai airport at 7pm while watching Sri Lanks vs. New Zealand semi-final match
- Mumbai to Delhi by air (9:30pm - 11:30pm)

Note: There were no direct flights from Baroda to Chandigarh, so I had to go to Mumbai. And then to Delhi from Mumbai as all direct flights from Mumbai to Chandigarh were sold out



Mar 30
- Delhi to Panipat by car (12 midnite to 2:30am)
- Dinner at dhaba (street restaurant) in Murthal - yummy food
- Slept for 2 hrs. Woke up at 5:30pm. Dressed in Indian team T-shirt and shorts. Grabbed my lucky Indian flag that I bought in Motera, Ahmedabad during India-Australia game
- Collected tickets at 10:30am
- Drove to Mohali stadium by car
- Found entrance 1C (11am) after driving around the stadium twice (need better instructions on the road for stadium gates)
- Easy checkin into stadium (20 mins in general first-level security and 2 mins in 1C gate security)
- Sat in block A Pavillion Terrace on south side (11:30am). Good luck with seats: no direct sunlight, beautiful weather
- Toss@2pm
- India started batting at 2:30pm
- We were snacking, hydrating thru out the innings
- Our section was filled with top-brass from Chandigarh, Delhi, Mumbai. Beautiful girls from Delhi, Mumbai, and Chandigarh
- Celebrity sighting: Arbaaz Khan was sitting in our section. Amir Khan, Preity Zinta, Indian PM Manmohan Singh and Indian businessmen were sitting in stands above us. Amir acknowledged the crowd, enjoyed the match thruout.


- Requested others with camera phones to click our pics and email it to us (eagerly awaiting those pics). I guess its time for me to upgrade my Blackberry and get one with a good camera.
- Duel with Pakistani fans sitting above us (noisy, beautiful girl, Afridi's brother, 4 adults, 4 - 6 kids). "Jeetega bhai jeetega India jeetega" would always overshadow "Jeetega bhai jeetega Pakistan jeetega".
- Good gesture from crowd: no booing, no Pakistan hai-hai, friendly, sporting, knowledgeable crowd
- Indian PM, Manmohan singh and Pak PM, Ghelani, met with the teams during opening. Everyone clapped. No one shouted anything bad! Indians love Sachin a lot and everything he does, so the focus was on him instead of the 2 head of states. Someone even shouted "Sachin for Prime Minister"
- India scored less than anticipated batting first (260 instead of 300+). Started well with Sehwag's blisterning (though short) knock but others could not capitalize. Even Tendulkar's inning wasn't impeccable (4 dropped catches)

Innings break

- Pakistanis start well but wickets keep falling afterwards to push them towards defeat
- Pakistani tails crumbles but not before giving the crowd a few anxious moments (Misbah's 6s)
- Exciting match, tense crowd. Anyone could've won till 90th over in the match!
- Sporting behavior on-field as well. No fights, no verbal duels. Nehra dismissed a close floored catch
- Great gestures off-field: India-Pakistan PMs; Players joking, laughing, shaking hands; Pakistan fan holding the Indian flag after Pakistan lost
- Match presentations: Sachin man of the match. Stadium acoustics suck. No one could hear the captains during the toss, interviews in the presentation ceremony (it was bad in Motera, Ahmedabad too). Punjab Cricket Association please fix it
- Peaceful exit at 11:30pm, decent, helpful crowd. Found car in 10 mins (quite amazing given the number of people and cars)



Mar 31
- Chandigarh to Panipat by car (12 midnite)
- Dinner@dhaba (line hotel) 2am - 2:40am. Yummy tandoori parathe (hail Punjab and Haryana for such great places to eat all along the road)
- Reached Panipat at 3:30am and crashed for 7 hours
- Panipat to Delhi by car (11:30am to 1:45pm)
- Watched highlights of the game at the airport (thx Hyundai for sponsoring big screens)
- Delhi to Mumbai by air (3pm to 5pm)
- Mumbai airport to home by car (5:30pm - 6:30pm)

---

Next up, is watching the finale between India and Sri Lanka at Wankhede stadium in Mumbai.

I'm introducing R to a few colleagues this week and want to share why learning a software like R is important... Here are a few articles that explain it well... Other reasons?

Importance of data science
- Couple years ago Google's Chief Economist Hal Varian said that the sexy job in the next ten years will be statisticians. Read the full article (requires registration)

The ability to take data - to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it's going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.
I think statisticians are part of it, but it's just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills - of being able to access, understand, and communicate the insights you get from data analysis - are going to be extremely important. Managers need to be able to access and understand the data themselves.

- Rise of data scientists

- Becoming a data scientist

- Essential skills for a data scientist

Where R fits?
R provides an environment for all tools needed for data science (see the data science process below from Benjamin Fry's thesis).


- R is ideal for small data analysis i.e. data that fits in a computer's RAM e.g. data < 10GB. Whereas SQL and search techniques seem good for larger data sets that can fit in one machine and techniques like Hadoop are good for BIG data sets that cannot fit in one machine.

- NY times article on R you ready for R?

- NY times article on R

- R is becoming popular

As we dig deeper into Stata or R debate, a few questions have come up.


Question 1: One of the things Stata does well is the way it constructs new variables (see example below). How to do this in R?



We can rewrite it as-is using for loops in R, which is slow and not elegant. What's the elegant way to write this in R? I haven't used plyr yet... Time to learn it?

Link to question on StackOverflow

I wrote some R code (linear optimization) to get the answer. Check it out. The code allows configurable constraints to let you create your own dream team. 

DISCLAIMER: Soccer isn't my favorite sport (its Cricket, if you're curious ;) and I don't play fantasy football.