MapReduce and the aggregation pipeline are the two methods you’re able to use to deal with complex data processing in MongoDB. The aggregation framework is newer and known for its efficiency. But some developers still prefer to stick to MapReduce, which they consider more comfortable.
Practically, you want to pick one of these complex query methods since they achieve the same goal. But how do they work? How are they different, and which should you use?
How MapReduce Works in MongoDB
MapReduce in MongoDB allows you to run complex calculations on a large volume of data and aggregate the result into a more comprehensive chunk. The MapReduce method features two functions: map and reduce.
While working with MapReduce in MongoDB, you’ll specify the map and the reduce functions separately using JavaScript and insert each into the built-inmapReducequery.
The map function first splits the incoming data into key-value pairs—usually based on mapped grouping. This is where you specify how you want to group the data. The reduce function then runs custom calculations on the values in each data group and aggregates the result into a separate collection stored in the database.
How the Aggregation Pipeline Works in MongoDB
The aggregation pipeline in MongoDB is an improved alternative to MapReduce. Like MapReduce, it allows you to perform complex calculations and data transformations directly inside the database. But aggregation doesn’t require writing dedicated JavaScript functions that can reduce query performance.
Instead, it uses built-in MongoDB operators to manipulate, group, and compute data. It then aggregates the results after each query. Thus, the aggregation pipeline is more customizable since you can structure the output as you like.
How Queries Differ Between MapReduce and Aggregation
Assume you want to calculate the total sales of items based on product categories. In the case of MapReduce and aggregation, the product categories become the keys, while the sums of the items under each category become the corresponding values.
Take some example raw data for the described problem statement, that looks like this:
Let’s solve this problem scenario using MapReduce and an aggregation pipeline to differentiate between their queries and problem-solving methods.
The MapReduce Method
Using Python as the base programming language, themapReducequery of the previously described problem scenario looks like this:
If you run this against the original sample data, you’ll see output like this:
Look closely, and you should see that the map and reduce processors areJavaScript functionsinside Python variables. The code passes these to themapReducequery, which specifies a dedicated output collection (section_totals).
Using an Aggregation Pipeline
In addition to giving a smoother output, the aggregation pipeline query is more direct. Here’s what the previous operation looks like with the aggregation pipeline:
Running this aggregation query will give the following results, which are similar to the results from the MapReduce approach:
Query Performance and Speed
The aggregation pipeline is an updated version of MapReduce. MongoDB recommends using the aggregation pipeline instead of MapReduce, as the former is more efficient.
We tried to assert this claim while running the queries in the previous section. And when executed side-by-side on a 12GB RAM machine, the aggregation pipeline appeared to be faster, averaging 0.014 seconds during execution. It took the same machine an average of 0.058 seconds to run the MapReduce query.
That’s not a yardstick to conclude on their performances, but it appears to back up MongoDB’s recommendation. You might consider this time difference insignificant, but it will add up considerably across thousands or millions of queries.
The Pros and Cons of MapReduce
Consider the upsides and downsides of MapReduce to determine where it excels in data processing.
Pros and Cons of the Aggregation Pipeline
How about the aggregation pipeline? Considering its strengths and weaknesses provides more insight.
When Should You Use MapReduce or Aggregation Pipeline?
Generally, it’s best to consider your data processing requirements when choosing between MapReduce and the aggregation pipeline.
Ideally, if your data is more complex, requiring advanced logic and algorithms in a distributed file system, MapReduce can come in handy. This is because you can easily customize map-reduce functions and inject them into several nodes. Go for MapReduce if your data processing task requires horizontal scalability over efficiency.
On the other hand, the aggregation pipeline is more suitable for computing complex data that doesn’t require custom logic or algorithms. If your data resides in MongoDB only, it makes sense to use the aggregation pipeline since it features many built-in operators.
The aggregation pipeline is also best for real-time data processing. If your computation requirement prioritizes efficiency over other factors, you want to opt for the aggregation pipeline.
Run Complex Computations in MongoDB
Although both MongoDB methods are big data processing queries, they share a lot of differences. Instead of retrieving data before performing calculations, which can be slower, both methods directly perform calculations on the data stored in the database, making queries more efficient.
However, one supersedes the other in performance, and you guessed right. The aggregation pipeline trumps MapReduce in efficiency and performance. But while you might want to replace MapReduce with the aggregation pipeline at all costs, there are still specific areas of application where using MapReduce makes more sense.
Q: What Else Should I Know About the Aggregation Pipeline?
MongoDB’s aggregation pipeline is a multi-step processthat includes matching data, grouping it, and sorting it.
Q: What Queries and Commands Can I Use With MongoDB?
Although MongoDB isa NoSQL database, it still supportsmany of the operationsyou’ll be familiar with from traditional RDBMS programs.
Q: How Do the Map and Reduce Functions Work in JavaScript?
In JavaScript, map and reduce are methods of the Array class. They arehigher-order functionsthat you can use to build new functions for highly flexible, reusable code.
