Premature optimization is the root of all evil
Donald Knuth


England take on Sri Lanka in a test series starting today.
After watching Cricket world cup 2011 and witnessing India play in quarter final, semi final and the finale in stadiums all over India, I had little appetite for any of the IPL 2011 matches.
But now, I'm excited again. This test series will be a thrilling contest considering the two teams are more or less equal:
- England is at #3 and Sri Lanka are at #4 in ICC test ranking
- England plays well against Sri Lanka at home, and Sri Lanka has an edge in their home conditions. And the overall record is well balanced
Sri Lanka has won 2 of the last 3 test series against England with 1 draw. Clearly, they have a better track record. However, it is difficult to predict the winner as England has the home advantage and the momentum (Ashes tour and other tests in last 18 months). Sri Lanka has a new captain in Dilshan who will be under pressure. England will be under pressure too with the recent emergence of English cricket and the desire to prove their test cricket status.
Let the games begin!


Kochi has probably some of the best food I have tasted. Fresh, spicy, tangy and made with passion!! Happen to be taken to The Grand Hotel by our driver and turned out to be one of the best meals of the trip. Appam this good is not hard to find in Kerela, though does get [...]
While running "sunspot:solr:run" on Windows ["sunspot:solr:run" works with windows otherwise use "sunspot:solr:start"] if you are receiving error like -
Exception in thread "main" java.lang.NoClassDefFoundError:
'-Djava/util/logging/config/file=C:/DOCUME~1/pli/LOCALS~1/Temp/
logging/properties20100617-11284-furc4v-0'
Caused by: java.lang.ClassNotFoundException:
'-Djava.util.logging.config.file=C:.DOCUME~1.pli.LOCALS~1.Temp.
logging.properties20100617-11284-furc4v-0'
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
Could not find the main class:
'-Djava.util.logging.config.file=C:/DOCUME~1/pli/LOCALS~1/Temp/
logging.properties20100617-11284-furc4v-0'.
Program will exit.
Please change line no. 74 in server.rb from lib of sunspot gem,
"exec(Escape.shell_command(command))"
with "exec(Escape.shell_command(command).gsub("'",'"'))"For details refer source.
10. Can't crack that hard Sudoku problem?? Use R!
9. Want to pick a skill that will give you an early adopter advantage?? Learn R! It is the leading open source statistical and data analysis programming language, and is heating up!
8. Need to run statistical calculations in your software application?? Deploy R! It integrates with many programming languages like Java, Ruby, C++, Python
7. Looking for reusable libraries to solve a complex problem?? Get R! It has 2000+ free libraries to use in areas of finance, natural language processing, cluster analysis, optimization, prediction, high performance computing etc.
6. No Windows, No Doors - R runs on all the platforms. Just name it and you got it!! Windows PC, Mac, Linux to name a few
5. Did you know how much fun stats can be- Try R!!
4. Are you updated with the current trends?? Leading firms like NY Times, Google, Facebook, Bank of America, Pfizer, Merck are all using R, where are you??
3. Need to run your own analysis?? Need to solve an optimization problem?? Struggling with Excel or SQL in your model??..... just few statements away - Try R!!
2. Want to create a compelling chart?? Try R!
1. Want the coolest job in 2014?? Learn Statistics. It is the future. Data Scientists will be the sexy job in 2018.


This is a follow-up to a previous question on
VRP. I investigated R libraries and several other options to solve VRP and decided to build a custom desktop application using open source libraries from
COIN-OR. Screenshots attached below.
Leave a comment if you're interested. I will contact you directly.
Team: Prasoon, Khaled, James


Pavillion at ITC Maurya in Delhi has some great dining options. Even though its in Maurya (5-star hotel) its not too formal or too casual. If everyone dining with you is in the mood for a different cuisine, its possible to satisfy all their cravings here. I recommend a tasting platter to begin with.. What [...]
Started learning 'R'.
My first attempt was to plot data from Forbes 1000 list (refer to the exercise posted by Prasoon sharma)
Here is a bubble chart showing Forbes top 25 companies by Market Capitalization
Source code:
## read the csv file
FORBES.DF <- read.csv("forbes2000list_for_2011.csv")
## assign titles
names(FORBES.DF)<- c("Rank", "Company", "Country", "Industry", "Sales", "Profits", "Assets", "MarketCap")
## create a smaller vector
Forbes100ByMC <- FORBES.DF[order(-FORBES.DF$MarketCap),][1:100,]
Forbes25 <- Forbes100ByMC[1:25, ]
## plot the bubble chart using 'symbols'
radius <- sqrt(Forbes25$MarketCap/pi)
sales <- as.numeric(as.character(Forbes25$Sales))
profits <- as.numeric(as.character(Forbes25$Profits))
symbols(sales, profits, circles=radius, inches=0.9, fg="white", bg="light blue", xlab="Sales($'Billions)", ylab="Profits($'Billions)", main="Forbes 25 By Market Capitalization", xlim=c(min(range(sales))-50, max(range(sales))+50), ylim=c(min(range(profits))-2, max(range(profits))+2))
## print the names of companies
text(sales, profits, Forbes25$Company, cex=0.6, col="dark red")
Any feedback toward writting better 'R' code is welcome.
Check out how to generate stacked waterfall charts using highchart js library.
So you want to use re-captcha in your Rails application to avoid bots spamming your site, look no further - In your Rails 3 gem file add the gem “scaffoldhub” Do a – bundle install Lets say you want to …
Read more »
Update: the chart extension has evolved to produce waterfall charts with bubbles superimposed:
Just committed JS extension to create Waterfall charts
Look at
here for details
When I talk with CIOs these days, there's one question that comes up again and again: How does it all fit together? How does Social play with my Intranet? How does Social play with my document management system? How does...
By the end of this post you will be able to integrate your Web Hosted Application with QuickBooks Online Edition. You will understand below points.
Objectives:
1. Establish a communications channel with QuickBooks
2. Build a set of request messages
3. Send the request messages set to QuickBooks
4. Check the status codes & data in the response message set received from QuickBooks
5. Close the communications channel with QuickBooks
The Basic Communication Model is shown in Fig. 1
Fig. 1
Terminologies from Fig. 1
• Developer Application => Rails App
• Request => Https request generated using QBXML
• Response => Response over https in QBXML format.
• Company Owner’s Authorization => Authorization process for security purpose explained in detail in part-1. i.e. Establish a communications channel with QuickBooks
• Request Processor => Its nothing but a QBXML parser which process request from developer application and response from QuickBooks and sent in QBXML format to Developer application.
• QuickBooks => QuickBooks Account.
Now step by step:
1. Establish a communications channel with QuickBooksSteps to establish connection between developer application and QuickBooks.
1. Obtain a test QBOE (QuickBooks Online Edition) account from IDN for testing during development. These are available to paid memberships and participants in the NFR program.
Purchase an account from QuickBooks
As the quickbooks doesn’t support Multitenant platform, so the process is defined in step-7 for connecting specific account with specific company if you have multitenant rails application.
2. Register your application with IDN (Register your app at
appreg.intuit.com)
It’s an IDN Gateway Application Registration which is required to generate an Unique ID which is later used in https requests. You need to register valid https url of rails application with IDN.
URL: https://app_url
APPLICATION ID: Unique App Id generated- You will have some unique id generated eg: 169442259
Please make sure you have valid https ceritificate for the same url. Otherwise redirect it using proxies.
It also asks for some basic URLs which QuickBooks uses for further reference.
Subscribe URL: https://app_url/quickbooks/subscribe
Usually connection ticket is returned in this action when connection to QuickBooks account is created for first time with Rails App.
Change URL: https://app_url/quickbooks/change
Cancel URL: https://app_url/quickbooks/cancel
Login URL: https://app_url/login
3. Obtain a server certificate from a supported root certificate authority. This will be your https certificate.
4. Obtain a client certificate by generating a certificate signing request (CSR) and use the
appreg.intuit.com site to get it signed by Intuit.
a) Obtain a client certificate by generating a certificate signing request (CSR):
i. Generate a new key:
ii. $ openssl genrsa -out idn.key 1024
iii. Generate CSR:
iv. $ openssl req -new -key idn.key -out idn-foo.csr
v. IMPORTANT: Common Name (Your Name) needs to be EXACTLY
vi. your.url(THIS MUST BE AN IP ADDRESS OF YOUR HOSTED APP SERVER):your.application.name
b) Get CSR signed at
appreg.intuit.com Go to site, copy and paste CSR text and sign that bad boy!
c) Create PEM
The signing process will return a certificate. Copy the certificate to a text file and name it qboe-cert.pem. Now open the key you used to generate the CSR and paste that into the same text file above the certificate.
5. Understand and follow the QBOE security requirements for hosted web applications.
Your hosted web application (Developer Application) needs a server certificate in order to receive callbacks from QBOE. Your hosted web application also needs a client certificate signed by Intuit in order to POST QBOE requests to QBOE. (You need register your application with IDN before you can generate your certificate signing request.)
6 .In your code, using SSL, implement the presentation of the client certificate to be used when posting to QBOE.
The Intuit-signed client certificate must be presented to QBOE at every POST to QBOE sites. You have to use SSL at your web server in conjunction with the client certificate. The client certificate is presented using “pem” feature of httparty gem.
require 'httparty'
class Qboe
include HTTParty
base_uri 'https://webapps.quickbooks.com/j/AppGateway'
format :xml
headers 'Content-Type' => 'application/x-qbxml'
pem File.read("#{RAILS_ROOT}/qboe-cert.pem")
end
7. In your code, prompt your customer to grant a connection ticket authorizing your application to access the QBOE Company. Respond by sending the customer to the QBOE login page to get a connection ticket. QBOE POSTs the connection ticket back to the application subscription URL you specified when you registered your application. Handle this POST at that URL and store the connection ticket securely.
Sending the User to QBOE to Create a Connection Ticket
In your main form displayed in the user’s web browser, provide a means for the customer to start the authorization process, for example, a Subscribe button. Whatever the means, respond to the customer’s action by sending the customer to the QBOE login page at the following URL:
for production QBOE:
https://login.quickbooks.com/j/qbn/sdkapp/confirm?appid=&serviceid=2004&appdata=
Appdata = > company id (It is not required or mandatory but preferred to use if you have multitenant application.)
After your application gets the POST from QBOE, you need to extract the connection ticket
from the POST in subscribe action.
8. In your code, implement session ticket handling code. That is, prior to sending QBOE requests for a particular customer, get a session ticket by sending a SignonMsgsRq containing a SignonAppCertRq with that customer’s connection ticket. You’ll POST this to the QBOE data exchange URL to get the session ticket in the Signon AppCertRs response.
Eg: session.erb <?xml version="1.0"?>
<?qbxml version="6.0"?>
<QBXML>
<SignonMsgsRq>
<SignonAppCertRq>
<ClientDateTime><%=today%></ClientDateTime>
<ApplicationLogin><%=APPLICATION_LOGIN%></ApplicationLogin>
<ConnectionTicket><%=CONNECTION_TICKET%></ConnectionTicket>
<Language>English</Language>
<AppID><%=APPLICATION_ID%></AppID>
<AppVer>1</AppVer>
</SignonAppCertRq>
</SignonMsgsRq>
</QBXML>
APPLICATION_LOGIN = 'app_url'
CONNECTION_TICKET = 'TGT-68-wltE9aMgPpzDAC46PJwZvQ'
APPLICATION_ID = '169442259'
def self.get_session_ticket
today = Time.now.strftime("%Y-%m-%d")
xml_to_send = ERB.new(get_file_as_string("lib/quickbooks/session.erb")).result(binding)
result = post('/', :body => xml_to_send )
result["QBXML"]["SignonMsgsRs"]["SignonAppCertRs"]["SessionTicket"]
end
Response text:
{ "QBXML"=> { "SignonMsgsRs"=> { "SignonAppCertRs"=> {"ServerDateTime"=>"2010-03-17T05:29:26", "SessionTicket"=>"V1-211-BN1A_Bt_lRGjZQRgHeVIFQ:169714916", "statusSeverity"=>"INFO", "statusCode"=>"0" } } } }
2. Build a set of request messages
Below example explains a request created to create an invoice in QuickBooks.
<?xml version="1.0"?>
<?qbxml version="6.0"?>
<QBXML>
<SignonMsgsRq>
<SignonTicketRq>
<ClientDateTime><%= today %></ClientDateTime>
<SessionTicket><%= session %></SessionTicket>
<Language>English</Language>
<AppID><%= APPLICATION_ID %></AppID>
<AppVer>1</AppVer>
</SignonTicketRq>
</SignonMsgsRq>
<QBXMLMsgsRq onError="continueOnError">
<InvoiceAddRq>
<InvoiceAdd defMacro="TxnID:1234"><!-- required -->
<CustomerRef> <!-- required -->
<ListID>4</ListID>
</CustomerRef>
<TxnDate>2010-10-10</TxnDate>
<!-- BEGIN OR -->
<InvoiceLineAdd > <!-- optional -->
<ItemRef> <!-- optional -->
<FullName >Hours</FullName> <!-- optional -->
</ItemRef>
<Desc >Its added from rails app.</Desc> <!-- optional -->
<Quantity >2</Quantity> <!-- optional -->
<!-- BEGIN OR -->
<Rate >111</Rate> <!-- optional -->
<Amount >101.00</Amount> <!-- optional -->
<ServiceDate>2010-10-10</ServiceDate>
</InvoiceLineAdd>
<!-- OR -->
<!-- END OR -->
</InvoiceAdd>
</InvoiceAddRq>
</QBXMLMsgsRq>
</QBXML>
Following Fields of Rails App are mapped with QuickBooks Invoice using QBXML.
Rails App (Eg. Time/Expense Field Names) QuickBooks Invoice Field Names
Contact/Matter Contact (Full Name) => Client Name/Bill To
Time Entry Date/Expense Entry => Date Service Date
Activity Type/Expense Type => Product/Service
Description => Description
Duration (Hrs) => Qty
Rate/hr ($) => Rate
Final Bill Amount => Amount
Actual Mapping in QBXML for expense entry in invoice is explained below.
QBXML:
<InvoiceAddRq>
<InvoiceAdd defMacro="TxnID:1234">
<CustomerRef>
<ListID>4</ListID>
</CustomerRef>
<TxnDate>2010-10-10</TxnDate>
<InvoiceLineAdd >
<ItemRef>
<FullName >Fax</FullName>
</ItemRef>
<Desc >Fax to eric</Desc>
<Amount >101.00</Amount>
<ServiceDate>2010-10-10</ServiceDate>
</InvoiceLineAdd>
</InvoiceAddRq>
Eg: To add a client name in QuickBooks invoice we need to send ListID in QBXML request. ListID is acquired by executing a query to QuickBooks, Contact Full Name is the parameter passed to query.
3. Send the request messages set to QuickBooks
It’s done using “post” method from http arty gem.
4. Check the status codes & data in the response message set received from QuickBooks
Usually below set of parameters values exists in every request and response which we need to consider for every request and specifically for responses.
Attributes in Request and Response Messages
5. Close the communications channel with QuickBooks
Basically it not compulsory or needed while accessing QuickBooks Online Edition from Web Hosted application.
Tools & Software: Development Environment
QuickBooks SDK 8.0
QBXML Version 6.0
Ruby 1.8.6
Rails 2.3.5
Net Beans IDE 6.7
Gems: Http arty 0.5.2
References
1. Intuit QuickBooks SDK Technical Overview
2. Intuit QuickBooks SDK Programmer’s Guide
3. Onscreen Reference(OSR)
4. Web site for the Intuit Developer Network (IDN): http://developer.intuit.com
5. Blog: http://jnylund.typepad.com/joels_blog/2010/02/quickbooks-online-integration-from-ruby-on-rails-the-easy-way.html
6. Blog: http://www.depixelate.com/2007/05/03/rails-and-quickbooks-online-integration-part-1/
7. http://www.servicesidekick.com/ (Quickbooks Integration)
Points to remember while adding bundler to Rails < 3.x.x applications- When I had modified an application based rails 2.3.4 for bundler, it has taken my time to make it work as i forgot below points.
1. Please check if someone had made any changes inside plugins/gems which you have mentioned in gem file now.
2. Please note of gems if you are mentioning them :groups as they are not available to other groups. eg. Gems mentioned in :groups => :test are not available to :group => :development. Eg: if you run rake tasks which uses gem libraries in development and gems are mentioned in :tests group, it will give you errors for required gems.
3. Also make sure about gems and plugins versions which are being used in app and you mentioned Gemfile.
I hope it might save your time.
These are links/resources i referred for respective software installations.
memcached for Windows
How to install PhyMyAdmin on Centos 5.3
Installing RMagick on Linux, *BSD, or Cygwin systems
If you ever want to disable generating rdoc and ri when you install a ruby gem:
Add this to your
~/.gemrc:
install: --no-rdoc --no-ri
update: --no-rdoc --no-ri
Devise has become the de-facto library if you ever need Authentication. Tonnes of pluggable components which you can easily enable/disable.
For our application we had to reset session and easiest and cleanest way to do was to extend the devise - SessionsController:
For testing this piece:
With smiling face I ran the spec buuuuuuttttttttttttttttt arrrrggghhhh:
AbstractController::ActionNotFound
Checking the devise filters helped with some direction:
Have fun !!
I was working on migrating an existing Rails 2 app to Rails 3. And I came across a weird issue. All my controller specs started to fail with error - route not found even though all my routes were in place.
After some more reading in the rails routing module - found it overwrites the routes if you open the block and pass on extra routes
Hope this will save few hours of your development :))
How do you approach the most critical objective of your organization?
Probably you would start by qualifying this objective and start a project to accomplish it. Break it down to smaller goals, milestones and put some planning around it? May be you would further break down the goals into small actionable items and schedule some activities. Some of you would setup a rhythm to periodically accomplish the smaller tasks in order to realize your grand objective.
No matter whatever process or methodology you follow, you would certainly NOT leave your most critical objective to chance..certainly NOT by adopting an ad-hoc approach.
My moment of epiphany arrived when I realized "Continuous Self Learning" is the most critical aspect of my career. Probably the same applies to most of the readers of this blog.
So, how to you approach your personal 'Learning'? Is there a method? Or is it left to chance?
The folks who thought of Scrum were really smart guys. I don’t say this because Scrum has worked for me but I say this because the more I think of Scrum, the more I appreciate it. For example, why does …
Read more »
We've introduced R in the organization!
It is running along with the heavy weights of statistical analysis like SAS, SPSS, Matlab. Here's what we did and how we did it...
I started learning R last year and loved its simplicity and power. After using it primarily for personal projects, I came across a business problem in which R can be considered a good fit.
BUSINESS PROBLEM
The business need was to build a web-based tool for marketing budget optimization - Marketing RoI (Return on Investments) i.e. how should a company that has multiple advertisement channels allocate its marketing budget across multiple channels to maximize profit or customer loyalty or customer life time value (LTV).
1) Input: The input to the analysis is the company's historical marketing budget allocation, profit, customer loyalty and LTV.
2) Analysis: This analysis is done in 2 steps.
- Step 1) Our experts create a formula that relates the inputs given with RoI and LTV etc. It involves econometric techniques etc.
- Step 2) Optimization of the formula when the user conducts what-if analysis by varying total budget and/or spend across individual channels to see its effect on RoI and LTV. The desktop optimization model written in Excel using a commercial Excel plugin.
3) Output: Optimized spend across advertising channels and ability to evaluate multiple scenarios to determine optimum marketing mix
The initial version of the tool runs as an Excel model using a commercial Excel plugin. The business objective was to transform this Excel-based single-user application into multi-user web-based application.
TECHNICAL SOLUTION
A) Web application: The web forms needed to allow users to input data and run scenarios were simple. We develop web applications using Ruby on Rails on LAMP internally. Ruby on Rails gives us an agile environment to develop software by taking care of routine web application tasks like database connectivity.
B) Optimization: Since, the Excel model uses a commercial plugin for step 2, the stakeholders started with the hypothesis of using the same commercial plugin's server version for optimization in the web application too.
For this we had to prove a couple of things:
1) Optimization of formula from step 1
2) Integration with web application
Option 1: Commercial optimization engine
We did a quick spike to test optimization with the commercial optimization plugin's server version and also its integration with Ruby on Rails web application and it was successful. We had to use JRuby to integrate Ruby with plugin's server edition as it provides only Java and .NET API.
Option 2: R (Open source)
In parallel, we checked if R can be used. R is a leading open source statistical environment.
- To solve the optimization problem in R we found
a lot of R optimization packages and started testing packages like BB as the formula (from step 1) was non-linear, and had constraints and conditions. We tested BB's SPG function and also tried other generic algorithms. We got good optimization results from R (similar or better compared to commercial optimization engine).
- Now we had to check how to integrate R with our web application written in Ruby. We found a number of options like
integrating R with Apache (rApache) or
integrating R directly with Ruby (rsruby). We decided to use rsruby.
We ran a number of proof of concepts with R and shared results with stakeholders. The results were positive in terms of performance as well as the optimized results... So we got better results and that too for free!
LESSONS LEARNED
Technical
- You need to be careful in running it in a shared environment, where it can use all your CPU and memory if it runs for long
- Don't forget to write unit tests using RUnit for your R code
- Capturing exceptions from R and dealing with them properly (appropriate message to users)
- rsruby installation documentation is good but needs a few tries depending on your Linux distribution
- rsruby does not run on Windows (wasn't a problem for us as we run our web applications on LAMP)
Process
User acceptance testing: If you are transforming an Excel-based model into web-version, it is critical to have a fully working example of the Excel model to replicate it in R/other statistical packages
- Overcoming the challenges of using new open source software in enterprise: Like most enterprise IT shops, we are used to commercial software as well and the idea of using open source software to do serious work is limited to the most popular open source frameworks like Drupal, Ruby on Rails, Linux. We positioned R as an add-on to our LAMP environment and got a separate virtual server dedicated to it as it is memory hungry.


I wanted to quickly check the callbacks that my controller has in my new Rails 3 application. With more abstraction in Rails 3 (AbstractController) - the easiest way I found was:
Other ways to do ?
A few weeks back, I had the unique opportunity to visit my old employer, this time however as a customer. It was good to hear some old terms which I last heard when I worked there. It also got me …
Read more »
Busy busy last week. But still got a lot of time to think on my way back home. Only two things that you should aim for – happy teams and delighted customers. Choose any process that gives you these. Why …
Read more »