Recent Big Data and Machine Learning Info Digest 2016/01/13

Michael Stonebraker: Big Data is (at least) Four Different Problems

• Stonebraker on "SQL Databases v. NoSQL Databases"
• Stonebraker on "NoSQL and Enterprises"
• Stonebraker on "mapReduce and Parallel DBmss"

10 More lessons learned from building real-life Machine Learning systems — Part I

Artificial Intelligence: Can Watson Save IBM?

Algorithms can predict how students will answer questions, and even explain why they would get questions wrong [paper]

Epic visualizations from FlowingData in 2015

Benchmarking streaming computation engines at Yahoo!

The untapped potential of health care APIs 

Microsoft researchers win Imagenet computer vision challenge

Learn python overtakes learn java

Data scientists keep forgetting the one rule every researcher should know by heart


Thanks Shaohua for above information sharing.

Starting R Programming course

Following our course list, I started my first one - R  Programming from Coursera today. 

According to the Wikipedia, R is a programming language and software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. The R language is widely used among statisticians and data miners for developing statistical software and data analysis. 

R  is an implementation of the S programming language combined with lexical scoping semantics inspired by Scheme. S was created by John Chambers while at Bell Labs. There are some important differences, but much of the code written for S runs unaltered.

The philosophy of S (so as R ) is what John Chambers said in the "Stages in the Evolution of S":

"...we wanted users to be able to begin in an interactive environment, where they did not consciously think of themselves as programming. Then as their needs became clearer and their sophistication increased, they should be able to slide gradually into programming, when the language and system aspects would become more important."

Choose Big Data Courses

To start our journey, we first need to choose the Big Data courses. 

There are so many courses (online and offline). Just google the 3 words, there are 121,000,000 results.

So, how should we start? Some people can help us with their recommendations:

• 10 Big Data Online Courses
• Latest MOOCs on Data Science
• New Year Resolutions for a Data Scientist

Based on our own research and experience, K and I come out the following courses first (we might adjust along the way).

What do you think? Want to study together?

New Year, New Journey.

Last year we have been organizing 3 Big Data related events (one of them with IBM Big Data University) through ACSIP.

When K and I started to explore the idea about Big Data education/training, we realized that we need to equip ourselves first. From May 5 to June 16 last year, we both took the MIT course- Tackling the Challenges of Big Data. We then together gave a try on one of Kaggle Competitions - Titanic: Machine Learning from Disaster. When Toronto Blue Jays was on their way to clinch a playoff berth and division championship in 2015, we worked together to make predictions of Jays' games by using A Markov Chain Approach to Baseball with MLB statistics data available online.

At the beginning of new year, we would like to get deeper into the Big Data field. We will start to learn more related courses together. 

It will be fun and challenge. If you like, you can follow our journey.

Big Data on Amazon - 1

Too much talk about the Big Data, I wrote another one after coming back from Oracle OpenWorld this year.

To learn a technical skill, it'd better to use it. Then how? In the companies, especially large companies, it's hard to do what you want (for new technologies) unless you happen to join the new projects or new established teams. Another way to learn by using is to join some open source projects outside your company work. 

But like most of IT professionals, you might not have enough spare time to join those open source projects if you have family to take care after daily work. So can we learn to use Big Data by testing in an existing platform with some examples?

I am sure you are thinking the same thing like myself. How about checking Amazon? It is still the main provider of PaaS and IaaS in the market. 

To test Big Data in Amazon, it offers the following through Amazon EMR, which simplifies running Hadoop and related big-data applications on AWS and can be used to manage and analyze vast amounts of data. 

• Hadoop Programming on Amazon EMR
• Data Analysis and Processing on Amazon EMR
• Data Storage on Amazon EMR
• Move Data with Amazon EMR

To know more about Amazon EMR, you can visit the FAQs. For the beginners like me, we can focus on the following areas:

Q: Where can I find code samples?

Check out the sample code in these Articles and Tutorials.

Q: How do I develop a data processing application?

You can develop a data processing job on your desktop, for example, using Eclipse or NetBeans plug-ins such as IBM MapReduce Tools for Eclipse (http://www.alphaworks.ibm.com/tech/mapreducetools). These tools make it easy to develop and debug MapReduce jobs and test them locally on your machine. Additionally, you can develop your cluster directly on Amazon EMR using one or more instances.

Microsoft and Big Data

While I was thinking about researching on Microsoft Big Data strategy, I just read an article about it - Microsoft's big data strategy: Democratize, in-memory, and the cloud - on ZDnet.

So what is Microsoft strategy on Big Data? In short, Microsoft promotes Big Data in 2 fronts: Business Intelligence formed over more than a decade of being in that market; and those of its other lines of business, including online services, gaming and cloud platforms.

In specific, there are following products related to Big Data:

SQL Server Parallel Data Warehouse

"Data Explorer" add-in for Excel

Hortwonworks Data Platform (HDP) for Windows

While the above products are all impressive and can compete with other companies, the Data Explorer is potentially adopted by more users since Excel is so popular in the office software market. 

As this Microsoft blog said, "Data Explorer enhances the self-service BI experience in Excel by simplifying data discovery and access to a broad range of public and enterprise data sources, enabling richer insights from data that has traditionally been difficult for users to get to. With Data Explorer, users can now quickly and easily import data from a variety of sources, including Web, Excel, Text, Database and Azure. Access to non-traditional sources such as Active Directory, Facebook and big data solutions like Hadoop are now within the reach of any user. Connecting directly to data from the web is easy and intuitive. Filtering and transforming your data can be done in just a few clicks and importing your final data into Excel is straightforward. "

You can check out the video below to see how Data Explorer works.

Computer Trading by using Big Data?

In the FT's article - FBI joins SEC in computer trading probe - on Mar.5, it told the following detail:

"Authorities are exploring potential holes in the system, including new algorithms referred to as “news aggregation” that search the internet, news sites and social media for selected keywords, and fire off orders in milliseconds. The trades are so quick, often before the information is widely disseminated, that authorities are debating whether they violate insider trading rules, the people familiar with the matter said."

It makes me wondering if above technology-driven trading trick was using Big Data technology. If some people or companies are able to use Big Data to generate benefits for other people or customers, there must be some guys who can take advantage of Big Data to make money for themselves.

Why did Big Data make a bad thing so easily?

• Big Data especially people's social network information from Internet are easily be accessed and consumed.
• There are so many Open Source technologies which can help on processing the Big Data.
• The cloud computing make Big Data crunching faster.
• The fast-growing IT technology including hardware and software drive the Big Data processing cost lower.

There is no way to stop people to access the Big Data just like no way to stop the data getting big.  Like fighting computer virus, people has to keep their anti-virus software updated. So for the above computer trading case, FBI and SEC have to get them equipped with Big Data technology to fight those guys who also know the Big Data.