Replies to This Discussion

Permalink Reply by Vincent Granville on November 17, 2010 at 6:49pm

It is a mandatory requirement for almost any kind of analytic job.You can learn SQL by creating your own MySQL database: it's free (open source). At this time, very few people use in-database analytics (such as SQL server data mining -- which still has many drawbacks compared to open source solutions).

• ▶ Reply

Permalink Reply by Ralph Winters on November 22, 2010 at 11:38am

In theory you are correct. You shouldn't need SQL for anything else other than managing data. However since most shops are SQL centric, you probably should learn it only so that you would be able to extract your own data when the need arised. In some cases a SQL query tool is the only tool that you will have at your disposal, and it will be the common language for speaking with IT.

-Ralph Winters

• ▶ Reply

Permalink Reply by Themos Kalafatis on November 22, 2010 at 3:08pm

I agree with Vincent and Ralph. Although your question is about using SQL during the analysis rather than using SQL to extract your data, here are some situations where i found SQL to be of great help :

1) When looking for outliers and marking them in a DB containing many records.
2) When doing transformations on Big Data.
3) Added bonus : Not relying all the time on -always- busy DBAs to give you what you need.

• ▶ Reply

Permalink Reply by eggman2001 on November 22, 2010 at 4:12pm

Thank you. That's all very helpful.

• ▶ Reply

Permalink Reply by Angela Waner on November 23, 2010 at 2:03pm

I agree with everyone.

Being able to write your own SQL, is valuable because of "downsized" IT departments with limited resources. And since I am writing my own SQL, I will sometimes clean data/generate new variables with the SQL. Maybe I want to extract just the domain part of an email address.

SELECT RIGHT(Email_Address, CHARINDEX('@', REVERSE(Email_Address))-1) AS [Domain Name]

• ▶ Reply

Permalink Reply by eggman2001 on November 26, 2010 at 6:29am

Thank you for the feedback. I understand much better now. I'm also wondering what some programs are to analyze the data that should be on my radar. I'm starting a masters program in statistics in the spring and am interested in what the most common programs are for analyzing data. R? SPSS? SAS? And do many people know just one program, or multiple ones?

Got one more question. Let's say I have some historic stock data including Date, Opening Price, High, Low, Closing Price and Volume. I've written a script (using Ruby) that goes row by row and performs different operations depending on the numbers in each row, meanwhile updating a set of variables as it does this. The values of the set of variables once the script has run is what I'm interested in. I know how to do this programically, obviously, but I'm wondering if this kind of task can be done (or be done better) with statistical programs.

• ▶ Reply

Permalink Reply by Ralph Winters on November 27, 2010 at 9:00am

All 3 of the programs are good choices. Both R & SAS have modules which integrate SQL calls within the environment.

• ▶ Reply

Permalink Reply by DataLLigence on December 6, 2010 at 12:21am

Knowing SQL is a huge plus in Analytics. Pulling data for my projects based on the business problem/requirement, processing the data, etc. using SQL is something I do everyday. Once I create the analysis or modeling dataset, I then import it into SAS or Clementine.

Don't think there are many differences amongst the data analysis programs. SAS is good in handling large datasets and they have been able to create this brand/image so successfully that SAS skills are very much in demand. Clementine is very good and easy to use, helps a lot when you don't have time and you don't want to write codes. But it's quite slow when you have large datasets. R requires a lot of coding skills and it's bigggest drawback is data handling capacity - it just cannot work on large datasets!

• ▶ Reply