{"id":5180,"date":"2020-03-25T14:58:28","date_gmt":"2020-03-25T13:58:28","guid":{"rendered":"https:\/\/lcloud.pl\/?p=5180"},"modified":"2024-12-11T13:09:36","modified_gmt":"2024-12-11T12:09:36","slug":"quick-big-data-processing","status":"publish","type":"post","link":"https:\/\/lcloud.pl\/en\/quick-big-data-processing\/","title":{"rendered":"Quick Big Data processing | Amazon EMR"},"content":{"rendered":"
Amazon EMR is a service that allows cost-effective and fast processing of large amounts of data. It uses the Hadoop<\/a><\/span> (open-source data processing software) framework, based on Amazon EC2 and Amazon S3. <\/span><\/strong><\/span><\/h6>\n

It provides the ability to efficiently process large amounts of data in processes such as:<\/span><\/p>\n

    \n
  • indexing<\/span><\/li>\n
  • data mining<\/span><\/li>\n
  • machine learning<\/span><\/li>\n
  • financial analysis<\/span><\/li>\n<\/ul>\n

    Amazon EMR saves us time-consuming configuration, commissioning and management of Hadoop clusters and the computing power that we need. Thanks to this, we can freely build workflows and monitor the progress of big data analysis.<\/span><\/p>\n

    The main unit when using the service is a cluster, which consists of nodes that can perform different functions, i.e. they can be of different types. Amazon EMR, on each type of instance (node), installs other software components, thereby assigning a specific role to the framework (Hadoop Apache). There are 3 types of nodes (nodes):<\/span><\/p>\n

      \n
    • Master node<\/span><\/strong> – responsible for the distribution of data between all nodes. Also, it monitors the progress of the analysis and checks the condition of the entire cluster.<\/span><\/li>\n
    • Core node<\/span> <\/strong>– contains software components that launch task nodes.<\/span><\/li>\n
    • Task node<\/span><\/strong> – contains software components that perform the task and do not store data. This type of node is optional.<\/span><\/li>\n<\/ul>\n

      After building the cluster, we can proceed to commission it to work. The next step is data analysis.<\/span><\/p>\n