AnalyticBridge

Data Intelligence, Business Analytics

Seven questions about real time analytics

What types of data structures are routinely used for in-memory real time transaction scoring? I've used doubly circular linked lists to store (say) 20 most recent transactions with time stamp and other attributes, per merchant / per customer.
What kind of metrics work well in this context? Among many metrics, I've used last transaction or time to 5-th previous transaction.
Do you use a lot of rather small lookup tables that you can upload in memory, to store historical data, such as merchant summary statistics broken down per day, for the last 3 months (one entry per merchant per day)?
How do you optimize server performance? For instance, at 2am, when the volume of transactions is 5 times lower than at peak time, do you use the analytic servers for other tasks, such as end-of-day re-scoring?
At peak time (severe peaks), do you use a simplified model that requires less memory, if you lack bandwidth?
Have anybody used the Hadoop environment to feed into a true real time processing system (that is, with no latency), such as credit card processing?
For data science ROI to be positive, should advanced analytics / data science costs (in terms of people, extra hardware and software) represent less than 10% of the cost of general computer architecture (servers, engineers, basic data processing and reporting)? Is there a magic number, and if it is not 10%, what would it be?

Comment

Comment by Mike O'Neil on May 12, 2012 at 5:27pm: Re Q7, there is no magic other than the simple arithmetic that for any ROI to be > 1, R just needs to be bigger than I. (ROI=R/I)

Is this question really about how to attribute the contribution of data science relative to the other contributions to calculating I. What matters for that is establishing the marginal R on marginal I for each component contributing to total I. If marginal return on marginal I is greater for data science inputs, then you would spend the extra on it, if it was not, then clearly the marginal investment should be on the other inputs.

Welcome to
AnalyticBridge

Sign Up
or Sign In

Or sign in with:

1	Why is it so hard to count this way?
2	Yelp won't hire unemployed data scientists
3	Normalized odds-ratio
4	Reverse clustering for tactical warfare
5	Time Period for Analytical Positions Recruitment.
6	Seven Questions on Adopting Analytics Culture
7	Outliers : Capping rule
8	Correlation and R-Squared for Big Data
9	Cross-validation in R: a do-it-yourself and a black box approach
10	Do You Have What It Takes To Be a Data Scientist?