BI Book - Reactive Business Intelligence

This book was for me one of the more original and interesting data mining books! It explains the several data mining tasks and techniques in a very intuitive and innovative way. It explain almost key points in the data mining field from the simple to complex topics. Even those persons that hate statistics and formulas will read and like this wonderful book. It has written in a “clean” language, with a lot of pictures that provides a better understanding about author’s message.

The book was written by the two persons of the picture above (R. Battiti and M. Brunato), and I truly suggest you to buy it. In the official book website, R. Battiti and M. Brunato added their description about it: “Reactive Business Intelligence is about integrating data mining, modeling and interactive visualization, into an end-to-end discovery and continuous innovation process powered by human and automated learning. This unifying goal requires collecting and integrating topics which are usually dissected into books dedicated to different areas” and also in the YouTube it’s possible to see a more personal description:

The concept of Reactive Business Intelligence introduced on the book is focused in four main areas: Data Mining, Models, Visualization and Feedback/Learning by software and people.

Throughout the book can be seen huge amount of intuitive pictures that support a fast learning, mainly in some topics often difficult to understand. Some examples can be seen in the official website, but I selected right now some of them for you:

Some points that I think could be improved/changed:

- Reduce the references to the visualization tool used (Grapheur) or add a new one more intuitive.

- More alignment between amazing book content to “real world” business cases

You can buy the book at official website!

I hope you enjoy it!

BI Conference - TechDays 2010 - After my session

Finally I have some time to describe what I experienced last week during the TechDays Portugal conferences. Today I’m writing my post nrº100 and could not do it better describing this fantastic experience

The facilities
The day before my session I went to see the event facilities and aclimatize myself to the room where I would present my session. There were many people working for this event, and it is important that everyone had the notion that organizing such an event is not easy. It requires lot of human labour and a lot of dedication, mainly from persons like Luis Martins, João Bilhim and Leonor (Microsoft) and several other amazing dedicated persons

The people
I met many people and also spoke with people who normally only contact via the Web. Is indeed a very interesting event, where people share knowledge and opinions that for sure is more important than if the lunch is good or bad! Jorge Paulino, Ricardo Peres, André Lage, Rodrigo Pinto, MVP guys, Jorge Moura, Nuno Batalha, Tiago Rente, and a lot of enthusiastic professionals

The Session
Most people, certainly should not have a notion that it was my first public session with so many people... and also, that a year ago never came into my head be presenting a session at TechDays. It was a very difficult challenge trying to pass a message on Data Mining for almost 300 people, mainly because the profiles of the audience was very diverse ... feedback received varied between easy and very difficult session contents. Thanks to all the persons that attended to my session. Well… if you're curious take a look into presentation below:

TechDays 2010 DAT202

I hope you enjoy it... I can translate to english if needed.

BI Conference - TechDays 2010 - Before my session

Dear friends,
There are only a few hours before presenting my session at TechDays. It will be a great challenge for me where I hope I can transmit the message I purposed to.

I don’t know yet, how many people will be watching my session, could be from 40 to 500 attendees… and for that reason I cannot measure right now the pression level that I’ll have today… :-)

There’ll be a lot of interesting sessions almost made by portuguese professionals…It’ll be a prove, that here in Portugal we have lot of expertise professionals.

I am sure that next year there will be lot of new speakers that are just watching this year. Mainly the master and pós-graduation BI Students… are your there!?!! :-)

Well… I'll leave now and I’ll wish a good luck to myself and a very good TechDays to all you. :-) I’ll be back after the event...

Pedro

Yes we can!...see Donald Farmer in Portugal!

For many years I have been following Donald Farmer work in Business Intelligence, mainly in Data Mining. Now and thanks to Microsoft Portugal we can see and make some questions to him in the first person.

"What do medieval archaeology, fish farming, Southwestern University of Chongqing and Microsoft Business Intelligence have in common? If you know, you should tell Donald Farmer, because he has been deeply involved in all of them at various times. Donald has worked in the Microsoft Business Intelligence team for 8 years covering many subject areas: data integration, information quality, metadata intelligence, master data management, OLAP, predictive analytics and self-service BI. He is a well-known speaker at Microsoft and other industry events, and the author of several books and articles" [Source: SQLBits]

In the scope of the Business Intelligence community (www.BIResort.net) Donald Farmer will come to Portugal for a meeting/session at Microsoft Auditorium on April 16, 2010 from 18h to 20h.

The session topic will be focused on Self-Service BI (Power Pivot) followed by some questions and answers. It’ll be a mix between public/private and all the attendees should have an invitation.

If you want to be on this FREE event, just send an email to BIResort@gmail.com

(Limited seats)

BI Conference - TechDays 2010 Portugal

Sorry for not writing for so many weeks, but last month was very hard to me… not only the weekly work as BI Consultant but also the work as invited teacher at Universidade Portucalense on a Business Intelligence pos-graduation. Additionally I was working on the event preparation which brings me to blog today: the TechDays 2010

Microsoft Portugal will organize the TechDays 2010 event next April (20/21/22) at Lisbon (Oeiras). For the first time, in spite of attending the event, I’ll be presenting a session on Data Mining with SQL 2008 R2… that I’m sure that will be a huge challenge for me.

My session description:
“The volume of business data grows exponentially and the technology made the world evolve rapidly. Bad decisions may be the result of the world be evolving more rapidly than the decision maker’s capability to analyze these data.

This session is intended to establish a first contact with multidimensional data structures (OLAP) and to recognize and to derive the exponential importance of knowledge discovery in databases through the data mining models in SQL Server 2008 R2”

Check my session link here

There is still a month for the event and there are more than 1200 participants confirmed. If you are near Portugal you cannot miss this!

Regards, Pedro.

BI News - Data Mining for Terrorists Is Urged

I found an interesting article that I want to share with you about Data Mining.

The article was Published at "The New York Times" on October 7, 2008 by Eric Lichtblau

WASHINGTON — "A federal panel of policy makers and scientific experts urged a government-wide evaluation Tuesday of programs that sift through databases looking for clues on terrorism, to determine whether the programs are effective and legal. The federal government has made aggressive use of so-called data-mining tools since the attacks of Sept. 11, 2001, as counterterrorism officials in many intelligence agencies have sought to analyze records on travel habits, calling patterns, e-mail use, financial transactions and other data to pinpoint possible terrorist activity.

The National Security Agency’s program for wiretapping terror suspects without warrants, the screening of suspicious airline passengers and the Pentagon’s ill-fated Total Information Awareness program, shut down by Congress in 2003 because of privacy concerns, have all relied on aspects of data mining.

But in a 352-page government study released on Tuesday, a committee of the National Research Council warned that successfully using these tools to deter terrorism “will be extremely difficult to achieve” because of legal, technological and logistical problems. It said a haphazard approach to using such tools threatened both Americans’ privacy rights and the country’s legitimate national security needs.

Mining through data patterns has been shown to work in commercial settings to predict what kind of toothpaste people may buy and what kind of movie they are likely to rent, or to detect casino card-counters or those engaged in credit card fraud.

But there is little evidence to confirm that the techniques work to actually find terrorists, despite the growing use in the last seven years, committee members said. Part of the problem, they said, is that the sample of known terrorists and actual attacks is so small that it is difficult to establish patterns of suspicious behavior.

The push to accumulate enormous amounts of information has also produced the risk of “a huge number of false leads” that could implicate people with no actual connections to terrorism, the committee said.

“More data does not mean better data,” said William J. Perry, the former defense secretary who was co-chairman of the panel, with Charles M. Vest, president of the National Academy of Engineering. "

Read entire article here

Data Mining and Neural Networks (NN)

Dear Friends,

This is my first post of 2009... happy new year!

I have been absent this month because I’m writing the second chapter of my master's thesis. Anyway, today I’ll write a bit of neural networks, which are on of the most important and useful Data Mining techniques.

But what is a neural network? A Neural Network (NN) is a system composed of many simple processing elements operating in parallel which can acquire, store, and utilize experiential knowledge. As I told you before, I’m very fascinated with Data Mining and the intelligence we can get from it. You can use it in Financial Markets (is the theme of my master dissertation), on insurance, or medicine, on retail, on all activities!!! The topics below explain better where you can apply NN, the learning methods, types and an Arquitecture that a NN could has.

NN Applications
-Classification
-Clustering
-Function approximation
-Prediction

NN Type
- Static (feed forward)
- Dynamic / Back-Propagation (feedback)

NN Arquitecture
- Single layer
- Multilayer
- Recurrent
- Self-organized

NN Learning Methods
- Supervised (network has inputs and their corresponding outputs)
- Unsupervised

In Data Mining classes directed by Professor Dr. Duarte Trigueiros, I did a practical excel exercise (homework) in order to better understand NN functionalities. I’ll share it with you today and feel free to contact me to get a copy of excel file. The excel files has the formulas behind and could be interesting if you think to improve your knowledge on this.


Initially, weights are random and the output of neural network will be compared to desired output. If same, reinforce patterns and if different, NN will adjust weights.

The dataset should be divided for training and testing, which in this case, because the NN is a supervised with back-propagation, the training dataset has the output classified and a better model will be created. After the model is created, the test dataset is considered as the input and the model could be evaluated.
Regards,
Pedro

BI Financial Markets with Data Mining

I have been a little absent due to the preparation of my Master's thesis in Decision Support Systems (DSS or BI), and it’s with a great pleasure that I share with you the theme that I will investigate during next year. This research project will be focused in the behaviour of financial markets through the use of Data Mining techniques. It’ll combine the technology through the creation of ETL processes, a Data Mart and Data Mining techniques, with the financial markets knowledge.

And the reasons that led me to choose this theme, are without doubt, the fascination for the mysterious world of financial markets, the fascination by data mining techniques (that are for me the true intelligence of a BI solution) and the need to create a DSS that enable the understand the impact of movements in financial markets in the budget of each one of us
Many of us invest money in the stock market through mutual funds, pension savings plans, through the brokerage firms and others ... but do we know what they are doing with our money? Could we trust on institutions that are investing our money? I think it's time to be ourselves the owners of our money and define what we want to do with it.

We have to know the right moment to subscribe, transfer or redeem financial products offered by various financial institutions, or even whether we should invest ourselves (in a long term and in a diversified way, we have high probability of good profits) in financial markets or put the money from under our beds.

And despite the banking sector is the most regulated in the world, still occurs several big frauds that could put in risk the survival of a company or even put in risk our savings... after several scandals in Portugal, France (Société General), and United States of America, today a new one…

Bernard Madoff, the former Nasdaq chairman and founder and president of a New York firm that invested funds for wealthy individuals, hedge funds and other institutions, was charged with operating what he told employees was a long-running $50 billion Ponzi scheme in what may be one of the largest frauds in history. Bernard Madoff was arrested today at 8:30 a.m… and some of his clients are BBVA, o HSBC, o Royal Bank of Scotland, Credit Agricole and Banesto.

How can a former chairman at the world's largest stock market with 20,000 listed companies have entered into schemes of fraud? Are these same markets efficient? There is much speculation and a lot of confidential information (inside trading) that is used to always generate profits for the same persons in the markets...

Aren’t you tired of being a spectator? I will use my arms (Data Mining) to understand better the financial markets and react to it with logic and not following the panic or the movements of the masses…

I never invested directly in stocks because I respect the stock market, however I was investing for few years in mutual funds that I earned about 20%... but I had the happy idea of leaving the market when I heard on the radio some advertising to subscribe mutual funds... which led me to think that people were entering into the market by “marketing” rather than the real value of companies... this one lesson that I learned by myself…

Data Mining - Rafal Event

In April 2008 Rafal Lukawiecki came to Portugal and unfortunately I didn’t attend it. I confess, that I’ve never heard about Rafal, but today, and after seeing the four webcats, I feel that I lost a fantastic event. So... do you know what I did? I link my laptop to my LCD television and see all the Rafal webcasts!! It was really fantastic... now I understand my friend Subhash!

But who is Rafal? See this YouTube video...
http://fr.youtube.com/watch?v=Vs46QZwnEcc

After my Data Mining class in my master BI degree, I’ve started to look to data mining with new eyes, and data mining is the main intelligence that a BI project could have... you could have an ETL, DW or Reporting module perfect, but the intelligence in BI is in the capability to predict the future, to analyse past and present behaviours and make the difference predicting on what it’ll happen next!

For those who really like to learn Data Mining, independently about the tools that usually use my suggestion is to start from here... from starting to see these four Rafal webcasts. Fortunately, in my master BI degree I had one of the most gurus of Data Mining here in Portugal... the teacher Dr. Duarte Trigueiros... He’s also fantastic and very comunicative as Rafal. From one hand Rafal speak more near the business logic and sometimes more intuitive and for other hand the teacher Dr. Duarte Trigueiros is more statistical. The conjuction of both approches is fantastic, and it was a privilege and a proud for me...

Introduction to Data Mining
Working with Data Mining
Using Data Mining in Your IT Systems (Part 1)
Using Data Mining in Your IT Systems (Part 2)
http://www.microsoft.com/emea/spotlight/event.aspx?id=99

After the Rafal webcasts, I dont have nothing more to write about it! But I’m currently doing a work for my master BI degree to predict the stock market behaviours... could be a good way to became rich fast!! I’ll tell about it in a next post...