Let's say that you have a large number n of elements a, b, c, etc. and you want to group them into clusters. Each cluster is supposed to contain few elements, say O(1).

You have one similarity metric d(a,b) to compare any two elements a, b. Also, you have a list of all pairs where d(a,b) > threshold, or in other words, all pairs (a,b) where a and b belong to the same cluster. The n x…

See attached document, including the theorem, its proof and applications to business analytics (e.g. to produce model-free, data-driven confidence intervals for predictive scores). More explanations coming soon, in particular about how to leverage this deep statistical result when computing metrics against very large data sets.…

• Over-fitting.If you perform a regression with 200 predictors (with strong cross-correlations among predictors), use meta regression coefficients: that is, use coefficients of the form f[Corr(Var, Response), a,b, c] where a, b, c are three meta-parameters (e.g. priors in a Bayesian framework). This will reduce your number of parameters from 200 to 3, and eliminate most of the over-fitting
• Perform the right type of cross-validation. If your training set has…

If you have more than 100 friends on Facebook, you've probably noticed that Facebook always show up the same 20 friends on your profile page, day after day. FB actually shows up to 10 friends, but they rotate from a list of 20 friends that, according to FB data mining algorithms, are deemed to be your best friends.

What makes a connection become one of your FB best friend is how frequently she visits your profile. Your can influence this list to some extent, by posting comments…

ARMONK, N.Y., May 20, 2011 /PRNewswire/ -- As companies seek to gain real-time insight from diverse types of data, IBM (NYSE: IBM) today unveiled new software and services to help clients more effectively gain competitive insight, optimize infrastructure and better manage resources to address Internet-scale data. For the first time, organizations can…

Information Management Online, May 12, 2011 , Valerie Valentine

May 12, 2011 - Big data predictive analytics developerAlpine Data Labs got a $7.5 million funding boost this week, at the same time as it announced entry to the market after 15 months of product development of the Big Data…

The position would be located in Ebay’s Whitman Campus (Campbell, CA)

Business Title – Sr. Manager – Vertical Analytics

Position Type – Full Time Employee

Description - The Site Analytics group is responsible for delivering business insights and high impact analyses to the Global eBay Marketplace businesses (eg: eBay.com, eBay.co.uk).  Within this group, teams partner with business unit clients to address strategic and operational questions…

I came across an interesting read about prescriptive analytics that I wanted to share with the community. The author describes prescriptive as the next evolution after predictive analytics. This paragraph summarizes the differences between Predictive and Prescriptive:

While predictive analytics helps you model…

Hi I trying to build "influence score" for web sites and journals and I was wondering if someone can point me in a right direction as to the model and technique I should use.   Can I model the “influence score” simulate to credit score? Is the Klout score for measuring influence of bloggers a good example of “influence score”? Can I use logistic regression to try to model "influence score" or should I use other models? If anyone has experience in modeling "influence score"  or knows of any…

The American Statistical Association and CHANCE magazine have debuted The Statistics Forum, a blog to provide everyone the opportunity to participate in discussions about probability and statistics and their role in important and interesting topics. The blog, which is located on the CHANCE web site atchance.amstat.org, is edited by Andrew Gelman. Everyone is invited to read and comment on the…

The American Statistical Association (ASA), the nation's preeminent statistical society, urges members of the House of Representatives to support the Statistics Teaching, Aptitude and Training Act of 2011 (STAT Act of 2011), which was introduced today by Congressman Dave Loebsack (D-Iowa). A copy of the bill may be viewed at…

"Evidence of plagiarism and complaints about the peer-review process have led a statistics journal to retract a federally funded study that condemned scientific support for global warming.

The study, which appeared in 2008 in the journal Computational Statistics and Data Analysis, was headed by statistician Edward Wegman of George Mason University in Fairfax, Va. Its analysis was an outgrowth of a controversial congressional report that Wegman headed in 2006. The 'Wegman Report'…