Implementação de algoritmos de regras de associação nos arcabouços Hadoop-MapReduce e Spark

Abstract

In midst to the big amount of data constantly produced on computerized information systems, there are data mining algorithms able to find hidden information in this data. One of techniques implemented by this algorithms is known as association rules, which aims to find associations between items on same dataset. A recent proposal uses association rules to deal with product offer classification in online store. However, for big amount of data, the proposed algorithm runtime becomes unfeasible. There are frameworks enabling distributed algorithms implementation in computer cluster like Hadoop and Spark. Many data mining algorithms, such as Apriori Algorithm for association rules, has several implementation proposals using MapReduce. This work performed a study of proposed solutions of Apriori implementation on Hadoop-MapReduce. The algorithms was also adapted to Spark and a comparative was performed between frameworks. The results show that Spark implementations overcomes Hadoop-MapReduce implementations at runtime in most experiments. However, there is no single implementation that is the best in all the evaluated situations. An alternative to the product offer classification in online store problem on Hadoop-MapReduce and Spark was also carried out. The results show large capacity of adaptation to process big amount of data.